Systems and methods for cropping images

Abstract

The system includes an imaging device and a processor. The imaging device has a field of view and is configured to operate in a normal state controlled by a first set of operating parameters for capturing a first sequence of images, and in a high-speed state controlled by a second set of operating parameters for capturing a second sequence of images. The processor is configured to estimate the movement path of an object based on data extracted from the first or second sequence of images and to control the imaging device in the high-speed state to crop the image in the second sequence of images that includes the region of interest containing the object. The processor controls the imaging device so that the second sequence of images includes the region of interest within the field of view and tracks the estimated movement path of the object.

Description

[Background technology]

[0001] <Claiming priority> This application claims priority to U.S. Nonprovisional Patent Application No. 18 / 334,764, filed on 14 June 2023, and the specification of the above-mentioned application is incorporated herein by reference.

[0002] Systems have been developed to track sports balls and analyze player movements (e.g., swings, throws, kicks, etc.) to enhance sports broadcasts and facilitate athlete training. These systems have included various tracking devices, such as radar and imaging devices, for tracking and analyzing the movements of the ball, athletes, and / or related equipment (e.g., rackets, bats, clubs, etc.). [Overview of the project]

[0003] This disclosure relates to a system including an imaging device configured with operating parameters for capturing a sequence of images, and a processor. The processor is configured to detect a ball in a first sequence of images captured by the imaging device configured with the current operating parameters, analyze one or more conditions related to the movement of the ball or ball-hitting device based on the position information of the ball or ball-hitting device determined from the first sequence of images, and when one or more conditions are met, trigger a change in the operating state of the imaging device and detect a ball in a second sequence of images captured by the imaging device configured with the adjusted operating parameters.

[0004] In one embodiment, the current operating parameters include a first operating state, and the adjusted operating parameters include a second operating state.

[0005] In one embodiment, the first operating state includes reduced operating parameters and power consumption compared to the second operating state.

[0006] In one embodiment, the operating parameters include the frame rate, resolution, or crop of a subset of images captured by the imaging device.

[0007] In one embodiment, the ball detection algorithm detects a ball, locates the ball in the image, and determines the position information of the ball in the image.

[0008] In one embodiment, the processor is further configured to track the ball across images in a sequence to determine whether the ball is moving or stationary.

[0009] In one embodiment, the first of one or more conditions is that the ball is stationary.

[0010] In one embodiment, the ball-hitting equipment detection algorithm detects the ball-hitting equipment, identifies its location within the image, and determines the location information of the ball-hitting equipment within the image.

[0011] In one embodiment, the processor is further configured to track the ball-hitting device across images in a sequence to determine whether the ball-hitting device is moving or stationary.

[0012] In one embodiment, the second of one or more conditions is that the ball-hitting device is within a predetermined distance from the ball.

[0013] In one embodiment, the third of one or more conditions is that the ball-hitting device moves away from the ball at an angle within a predetermined angular range.

[0014] In one embodiment, the second and third conditions are determined based on the type of ball-hitting equipment, and different types of ball-hitting equipment are associated with different distance and angle thresholds.

[0015] Furthermore, the present disclosure relates to a method that includes detecting a ball in a first sequence of images captured by an imaging device configured with current operating parameters, analyzing one or more conditions related to the movement of the ball or the movement of a ball striking implement based on the position information of the ball or the ball striking implement determined from the first sequence of images, triggering a change in the operating state of the imaging device when one or more conditions are met, and detecting a ball in a second sequence of images captured by an imaging device configured with adjusted operating parameters.

[0016] In one embodiment, the current operating parameters include a first operating state, and the adjusted operating parameters include a second operating state.

[0017] In one embodiment, the first operating state includes reduced operating parameters and power consumption compared to the second operating state.

[0018] In one embodiment, the operating parameters include the frame rate, resolution, or crop of a subset of the images captured by the imaging device.

[0019] In one embodiment, the ball detection algorithm detects the ball, locates the ball in the image, and determines the position information of the ball in the image.

[0020] In one embodiment, the method further includes tracking the ball across the images in the sequence to determine whether the ball is moving or stationary.

[0021] In one embodiment, a first condition among the one or more conditions is that the ball is stationary.

[0022] In one embodiment, the ball striking implement detection algorithm detects the ball striking implement, locates the ball striking implement in the image, and determines the position information of the ball striking implement in the image.

[0023] In one embodiment, the method further includes tracking a ball striking implement across images in a sequence to determine whether the ball striking implement is moving or stationary.

[0024] In one embodiment, a second condition among the one or more conditions is that the ball striking implement is within a predetermined distance from the ball.

[0025] In one embodiment, a third condition among the one or more conditions is that the ball striking implement is moving away from the ball at an angle within a predetermined angle range.

[0026] In one embodiment, the second and third conditions are determined based on the type of the ball striking implement, and different types of ball striking implements are associated with different distance and angle thresholds.

[0027] Furthermore, the present disclosure relates to a system including an imaging device that captures an image sequence of a ball in flight and a processor. The processor is configured to perform operations of detecting balls in a first image and a second image from the image sequence, implementing a dense optical flow (DOF) model to calculate pixel displacements across the first image and the second image, and calculating three-dimensional spin parameters of the ball based on the pixel displacements.

[0028] In one embodiment, the imaging device is configured to crop the first image and the second image to remove portions of the images that do not include the ball before passing the first image and the second image to the processor.

[0029] In one embodiment, the first image and the second image are cropped to center the ball in each of the first cropped image and the second cropped image.

[0030] In one embodiment, detecting the balls in the first image and the second image is based on a deep learning (DL)-based ball model.

[0031] In one embodiment, the DL-based ball model is based on a segmentation network.

[0032] In one embodiment, the processor is further configured to estimate the radius of the ball in each of the first and second images, and to reshape the first and second cropped images such that the radius of the ball in the first reshaped image matches the radius of the ball in the second reshaped image.

[0033] In one embodiment, the DOF model is implemented within DOF inference to generate flows for a pair of images, and the operation further includes detecting n balls in images from a sequence of images, and implementing the DOF model to generate flows for each pair of consecutive images such that n-1 flows are generated.

[0034] In one embodiment, the processor is further configured to perform the operations of applying spatial and temporal coherence filters to n-1 flows and filtering out any non-coherent flows.

[0035] In one embodiment, the processor is further configured to perform the operations of calculating the flow median and calculating the ball's three-dimensional spin parameters based solely on the flow median.

[0036] Furthermore, this disclosure relates to a method that includes detecting the ball in a first and second image from a sequence of images of a ball in flight, implementing a dense optical flow (DOF) model to calculate the pixel displacement across the first and second images, and calculating the ball's three-dimensional spin parameter based on the pixel displacement.

[0037] In one embodiment, the imaging device is configured to crop the first and second images to remove portions of the image that do not contain the ball, before passing the first and second images to the processor.

[0038] In one embodiment, the first and second images are cropped, and the ball is positioned at the center in each of the first and second cropped images.

[0039] In one embodiment, detecting balls in the first and second images is based on a deep learning (DL)-based ball model.

[0040] In one embodiment, the DL-based ball model is based on a segmentation network.

[0041] In one embodiment, the method further includes estimating the radius of a ball in a first image and a second image, and reshaping the first crop image and the second crop image such that the radius of the ball in the first reshaped image matches the radius of the ball in the second reshaped image.

[0042] In one embodiment, a DOF model is implemented within DOF inference to generate flows for a pair of images. The method further includes detecting n balls in images from a sequence of images and implementing a DOF model to generate flows for each pair of consecutive images so as to generate n-1 flows.

[0043] In one embodiment, the method further includes applying spatial and temporal coherence filters to n-1 flows and filtering out any non-coherent flows.

[0044] In one embodiment, the method further includes calculating the flow median and calculating the three-dimensional spin parameters of the ball based solely on the flow median.

[0045] Furthermore, this disclosure relates to a system including an imaging device and a processor. The imaging device has a field of view and is configured to operate in a normal state controlled by a first set of operating parameters for capturing a first sequence of images, and in a high-speed state controlled by a second set of operating parameters for capturing a second sequence of images. The processor is configured to estimate the movement path of an object based on data extracted from the first sequence of images or the second sequence of images, and to control the imaging device in the high-speed state to crop the image in the second sequence of images which includes a region of interest that includes the object. The processor controls the imaging device so that the second sequence of images includes a region of interest within the field of view and tracks the estimated movement path of the object.

[0046] In one embodiment, the processor is further configured to detect an impact event involving an object by analyzing a first sequence of images, and to change the state of the imaging device by applying a second set of operating parameters to operate the imaging device in a high-speed state when an impact event is detected.

[0047] In one embodiment, the object is a ball in flight, and the processor is further configured to estimate the trajectory of the ball in flight based on the impact event.

[0048] In one embodiment, the processor is further configured to apply data extracted from the cropped image to adjust the estimated trajectory of the flying ball for the subsequently cropped image in a second sequence of images.

[0049] In one embodiment, the object is a golf club, and the processor is further configured to estimate the trajectory of the golf club based on the impact event.

[0050] In one embodiment, the processor is further configured to apply data extracted from the cropped image to adjust the estimated movement path of the golf club for the subsequently cropped image in a second sequence of images.

[0051] In one embodiment, the object is a golf ball, and the high-speed processor is further configured to detect a strike event involving a golf club striking the golf ball by analyzing a second sequence of images, to control the imaging device in high speed to crop an image in the second sequence of images that includes a first region of interest including the estimated movement path of the golf club before the strike event, and to control the imaging device in high speed to crop an image in the second sequence of images that includes a second region of interest including the estimated movement path of the golf ball in flight after the strike event.

[0052] In one embodiment, the imaging device is fixedly positioned above the location where the impact event is expected, and the field of view of the imaging device includes at least a portion of the impact event and its movement path.

[0053] In one embodiment, the imaging device has a camera sensor chip having a plurality of discrete pixel elements that are read by a controller, and the controller in high-speed state is configured to handle a subset of the discrete pixel elements in order to read a cropped image that contains only the region of interest within the field of view.

[0054] In the embodiment, the processor is further configured to detect when a region of interest containing an object leaves the field of view and to change the state of the imaging device by applying a first set of operating parameters to operate the imaging device in a normal state until a further impact event is detected.

[0055] Furthermore, the disclosure provides an imaging device configured to operate in a normal state having a field of view and controlled by a first set of operating parameters for capturing a first sequence of images, and in a high-speed state controlled by a second set of operating parameters for capturing a second sequence of images; a method comprising: estimating the movement path of an object based on data extracted from the first sequence of images or the second sequence of images; and controlling the imaging device in the high-speed state to crop an image in the second sequence of images that includes a region of interest containing the object. The processor controls the imaging device so that the second sequence of images includes a region of interest within the field of view and tracks the estimated movement path of the object.

[0056] In one embodiment, the method further includes detecting an object collision event by analyzing a first sequence of images, and changing the state of the imaging device by applying a second set of operating parameters to operate the imaging device in a high-speed state when an impact event is detected.

[0057] In one embodiment, the object is a ball in flight, and the method further includes estimating the trajectory of the ball in flight based on a striking event.

[0058] In one embodiment, the method further includes applying data extracted from the cropped image to adjust the estimated trajectory of the ball in flight for the subsequently cropped image in a second sequence of images.

[0059] In one embodiment, the object is a golf club, and the method further includes estimating the trajectory of the golf club based on the impact event.

[0060] In one embodiment, the method further includes applying data extracted from the cropped image to adjust the estimated movement path of the golf club for the subsequently cropped image in a second sequence of images.

[0061] In one embodiment, the object is a golf ball, and the method further includes detecting a strike event involving a golf club striking a golf ball by analyzing a second sequence of images, controlling the imaging device at high speed to crop an image in the second sequence of images that includes a first region of interest including the estimated movement path of the golf club before the strike event, and controlling the imaging device at high speed to crop an image in the second sequence of images that includes a second region of interest including the estimated movement path of the golf ball in flight after the strike event.

[0062] In one embodiment, the imaging device is fixedly positioned above the location where an impact event involving an object is expected, and the field of view of the imaging device includes at least a portion of the impact event and its movement path.

[0063] In one embodiment, the imaging device has a camera sensor chip having a plurality of discrete pixel elements that are read by a controller, and the controller in high-speed state is configured to handle a subset of the discrete pixel elements in order to read a cropped image that contains only the region of interest within the field of view.

[0064] In embodiments, the method further includes detecting when a region of interest, including an object, leaves the field of view, and changing the state of the imaging device by applying a first set of operating parameters to operate the imaging device in a normal state until a further impact event is detected.

[0065] In one embodiment, the processor is further configured to control the imaging device in a high-speed state to crop images in a second sequence of images that include a first region of interest, including the estimated movement path of the golf club after a striking event.

[0066] In one embodiment, the method further includes controlling the imaging device in a high-speed state to crop images within a second sequence of images that include a first region of interest, which includes the estimated movement path of the golf club after a striking event. [Brief explanation of the drawing]

[0067] [Figure 1] Figure 1 shows an exemplary system according to this disclosure. [Figure 2] Figure 2 shows an exemplary image captured by the imaging device of the system shown in Figure 1. [Figure 3] Figure 3 shows a flowchart for changing the system state based on event detection. [Figure 4] Figure 4 shows a method for changing the state of the imaging system based on the detection of an event according to another exemplary embodiment. [Figure 5] Figure 5 shows a method for changing the state of the imaging system based on the detection of an event according to yet another exemplary embodiment. [Figure 6] Figure 6 shows a method for triggering a change in the operating state of an imaging device according to various exemplary embodiments. [Figure 7] Figure 7 shows a method for calculating the 3D spin axis (SA) of a moving sports ball from a sequence of images from a single imaging device, according to various exemplary embodiments. [Figure 8] Figure 8 shows a method for calculating the dense optical flow (DOF) of a sequence of image inputs describing a moving and spinning sports ball, according to various exemplary embodiments. [Figure 9] Figure 9 shows a method for calculating the three-dimensional (3D) spin axis from dense optical flow (DOF) for input pairs of sequential images describing a moving and spinning sports ball, according to various exemplary embodiments. [Figure 10] Figure 10 shows examples of implementations of the DeepVOG architecture 1000 in various exemplary embodiments. [Figure 11] Figure 11 shows methods for training deep learning (DL) dense optical flow (DOF) models according to various exemplary embodiments. [Figure 12] Figure 12 shows one embodiment of an overhead launch monitor system for a golf simulator. [Figure 13]Figure 13 shows a flowchart of the operations performed by the processor in Figure 12, which begin before the take-back event, during the flight of the golf ball, and end when the golf ball leaves the field of view of the imaging device. [Figure 14] Figure 14 shows a state diagram of the system shown in Figure 12. [Figure 15] Figure 15 shows an image captured under normal conditions by the imaging device shown in Figure 12, which has a field of view (FOV) that defines a visually observable scenario. [Modes for carrying out the invention]

[0068] Exemplary embodiments can also be further understood by referring to the following description and associated accompanying drawings, where similar elements are provided with the same reference numerals. The exemplary embodiments relate to systems and methods for event detection and / or parameter determination for sports game applications. In particular, these detections and / or determinations may be performed, for example, using image processing techniques on a sequence of images from an imaging device (e.g., from a single imaging device). In some aspects of these exemplary embodiments, the system can change the operating state of an imaging device based on the detection of a triggering event, for example, so that the imaging device transitions from a low-power state with relaxed operating parameters to a high-power state with more energy-intensive operating parameters.

[0069] In other embodiments, the high-power state of the imaging device relates to increasing the frame rate, increasing the resolution, and / or other operating parameters suitable for detecting specific events and / or deriving specific parameters. In one exemplary embodiment, the high-power state is used to capture images suitable for deriving parameters of the impact and / or launch of a sports ball. In one embodiment, the high-power state system can measure the three-dimensional (3D) spin axis of a sports ball. While some embodiments are described in relation to the launch of a golf ball, the exemplary embodiments can be applied to a variety of different sports, as will be described in more detail below.

[0070] In certain exemplary embodiments, a single imaging device is used to capture a sequence of images that can be processed by a computing device in a variety of operations, including, for example, object detection, object tracking, event detection, state detection, and parameter determination. These operations will be collectively referred to herein as “detection” or “determination.” These various operations require different image capture parameters and associated processing loads to perform detection or determination. In one non-limiting embodiment, certain detections may require high frame rates, while others may be performed at lower frame rates. Similarly, different types of detections may require or be suited to different resolutions or lighting conditions. Some detections may impose a high processing load on the computing device, while others may impose a relatively low processing load.

[0071] For example, various types of detection related to a golf game, such as object detection, state detection, or event detection, which may be performed based on a sequence of images (or a single image) from a single imaging device, include, for example, ball detection, tracking of ball movement, club detection (including identification of the type of club being used), tracking of club movement (including different aspects of a series of swings such as backswing and downswing), detection of a type of swing (e.g., putt, chip shot, flop shot, full swing, half swing, etc.), detection of impact between the club and the ball, and determination of the impact position on the club.

[0072] For example, various parameters related to a golf match that can be determined based on a sequence of images (or a single image) from a single imaging device include, for example, determining the initial velocity of the struck ball, identifying the initial direction of the struck ball's movement, club parameters such as the angle of incidence, dynamic loft, dynamic lie, club path, club speed, and face angle, and the distance from the imaging device to the ball.

[0073] Various image acquisition parameters related to acquiring a sequence of images (or a single image) from an imaging device include, for example, frame rate, resolution, zoom, illumination conditions, wavelength (e.g., visible spectrum, infrared spectrum, etc.), region of interest (ROI) identification (e.g., cropping), position, and orientation. Note that some imaging devices have different capabilities than others; for example, some imaging devices may be able to acquire images at a higher frame rate or resolution than others, and some imaging devices may have adjustable image acquisition parameters (up to a maximum value), such as a variable frame rate or resolution.

[0074] Some detections / determinations related to a golf match are preferably, or always, performed using specific image capture parameters, such as the minimum operating parameters required to capture an image that enables the derivation of the desired parameters, while other detections and / or determinations may be performed using more relaxed image capture parameters. In one exemplary embodiment, the determination of impact parameters for a golf club is preferably performed by capturing a number of images that are immediately before, immediately after, or both before and after the actual impact, including, for example, one or more images that are as close as possible to the time of impact.

[0075] In another exemplary example, the determination of ball launch parameters is performed by capturing a large number of images immediately after impact. Depending on the desired level of precision, the club and ball are relatively fast at the time of impact, so the imaging device needs to capture these images at a relatively high frame rate to provide a sufficient number of images to the computing device to enable the determination of these impact parameters. Conversely, in another embodiment, the detection of specific events related to a swing (e.g., backswing) may require fewer images (e.g., only two or three) captured at a relatively low frame rate.

[0076] Relatedly, different image acquisition parameters, detection types, and determination types can impose different processing requirements on the imaging device and / or computing device that processes the image data. These different imaging device settings and associated processing also lead to differences in power consumption. In one exemplary embodiment, higher frame rates, resolutions, and lighting conditions for the imaging device result in higher processing loads and power consumption than lower frame rates, resolutions, and lighting conditions. In another exemplary embodiment, some object detection algorithms (ball detection, club detection) can be executed sequentially on all images acquired by the imaging device without imposing a high processing load, while in yet another exemplary embodiment, the detection of impact parameters will impose a significant processing load due to the complexity of the calculations required.

[0077] U.S. Patent No. 10,953,303 describes, for example, a system and method for determining the impact characteristics of a sports ball at a sports ball striking element based on images from a single imaging device, and the entirety of which is incorporated herein by reference. U.S. Patent No. 11,452,911 describes, for example, a system and method for determining the launch characteristics of a sports ball based on images from a single imaging device, and the entirety of which is incorporated herein by reference.

[0078] A system (including at least an imaging device and associated computing devices) may be selected for use based on the type of detection / determination the system is intended to perform. Some systems may have limited imaging and / or processing capabilities and may be used for simpler detection / determination, while others may have advanced imaging and / or processing capabilities and may be used for more complex detection / determination. In one exemplary embodiment, an advanced imaging and processing unit is used to determine impact parameters, while a simpler imaging and processing unit, or the same imaging device operating in a less data-intensive manner, is used to analyze the swing motion. In another exemplary embodiment, a particular position and orientation of the imaging device is preferred or required to enable it to capture images suitable for the intended data-intensive detection / determination.

[0079] Naturally, when an advanced imaging / processing system is implemented, the system does not need to run continuously at its highest level. For example, if the system is intended to determine / detect various different parameters, events, or states, the system's operating parameters can be adjusted manually or automatically according to the current needs. Some image processing algorithms can run continuously or semi-continuously, while others run only occasionally. As mentioned above, some image capture parameters / settings are appropriate for specific detections / determinions, while other image capture parameters / settings are appropriate for other detections / determinions.

[0080] Therefore, those skilled in the art will understand that certain imaging systems, including an imaging device for capturing image sequences and a computing device for processing the captured images and deriving / detecting parameters / events from the images, can enter different operating modes or states based on the type of detection / decision performed in a given time. Such systems may operate at lower power levels when performing relatively simple detection / decisions and at higher power levels when performing more complex detection / decisions. Some systems may be capable of advanced imaging processes, such as high frame rates and / or high-resolution image acquisition, but may still be limited by the available processing power of the associated computing device. Thus, these systems may not always be able to operate to their maximum capacity, as an unacceptable delay in processing may occur if the system cannot process all the captured data in a timely manner.

[0081] Even when advanced imaging capabilities and significant processing resources are available, it may be desirable to limit the operation of the imaging device and / or computing device, particularly when certain operations are not required at a given time, to conserve power, prevent overheating, or for other reasons. Limiting the time the imaging device is operating in a high-power state, and / or limiting the CPU / GPU computational load whenever possible, especially if the system is configured to perform detection / decision that requires complex calculations.

[0082] According to various exemplary embodiments described herein, systems and methods relating to event detection and, for example, the determination of parameters from a sequence of images captured by a single imaging device are described, although additional imaging devices may be employed as desired. In some embodiments, methods are described for changing the state of one or more aspects / components of the system based on detection / determination made in advance of an anticipated future event. In particular, certain embodiments operate by changing the operating parameters and / or power state of the imaging device and / or computing device when a particular future event is anticipated. Detection of an object and / or event while the system is in a first state, for example, a lower power state including relaxed operating parameters for the imaging device, then triggers the system to transition to a second state, such as a higher power state including operating parameters for the imaging device suitable for further operation, detection, or determination, as will be described in more detail below.

[0083] In one embodiment of these exemplary embodiments, an operation is described for determining the spin parameters of a batted ball from a sequence of images captured by, for example, a single imaging device. Previous systems for determining spin parameters have involved multiple imaging devices and / or other sensors, such as radar, and no known existing system exists that can determine spin parameters from a single image stream from a single imaging device, as in this exemplary embodiment.

[0084] As will be described in more detail below, determining spin parameters using a single imaging device according to these exemplary embodiments is a computationally complex operation and is preferably performed under specific imaging device settings, including high frame rates, e.g., over 100 frames per second (fps), a crop centered on the ball (e.g., a region of interest (ROI)), and higher light intensity exposure. As will be understood by those skilled in the art, higher frame rates, light intensity, and processing requirements generate more heat and demand more processing power from the computing device. Therefore, it is preferable to minimize the duration that the system is in a high-power state.

[0085] Accordingly, in another aspect of these exemplary embodiments, an operation is described for detecting a trigger event that indicates a future event requiring high-power mode is likely to occur soon and / or within a calculated time, and for adjusting the system settings in response to the trigger event (i.e., anticipating the future event) (e.g., transitioning the imaging device to a high-power state with more intensive operating parameters). In some embodiments, trigger event detection immediately triggers the transition to the high-power state. In other embodiments, the transition to the high-power state is initiated only after a predetermined delay time has elapsed. The transition may be delayed, for example, if the timing of the expected future event is estimated to occur only after a predetermined time has elapsed since the detection of the trigger event.

[0086] With regard to golf matches, various embodiments of these exemplary models will be described in detail below. In one exemplary embodiment, the exemplary system includes an imaging device mounted on the ceiling and oriented so that the imaging device looks downwards at the scene area, for example, at the position where the golfer hits the ball, so that the imaging device has a field of view that includes the launch area. In these embodiments, the system is intended to capture the impact and launch of the ball with a very high degree of detail, and therefore the imaging device needs to operate in a high-power state with a high frame rate during the time that surrounds the launch of the ball in real time. Various trigger events, described in detail below, inform the system's decision on when and where to switch the imaging device to a higher intensity recording mode.

[0087] It should be understood that the principles described herein can be applied to a variety of use cases. The processing logic can be adapted to various scenarios in which future events can be predicted and the timing of these future events can be approximated. Various embodiments of methods for optimizing the operation of an imaging system to reduce or minimize the amount of time the system is in a high-power state of operation, while ensuring high accuracy in its parameter estimation, are provided below.

[0088] Figure 1 shows an exemplary system 100 including an imaging device 105, a computer 110 with memory 115 and a processor 120, and a display 125. System 100 can be implemented in a variety of different environments. In one exemplary embodiment, one or more components of system 100 are mounted on the ceiling or some other structure at a height above the scene visible from the imaging device 105, which is oriented so that its field of view faces downward in the scene area. In these embodiments, system 100 can be implemented in an indoor golf environment, such as a golf driving range or a golf simulator. However, exemplary embodiments are not limited to this arrangement. As will be described in more detail below, system 100 can be adapted to different golf environments or other sports environments, for example, where the imaging device 105 is in a different position / orientation. Furthermore, in some scenarios, system 100 may include additional sensors.

[0089] The imaging device 105 in this embodiment is configured to capture image data (images or frames) of the scene within its field of view (FOV), including the launch position of the golf ball and the area immediately preceding the launch position. The imaging device 105 can be configured for different operating settings, such as frame rate, resolution, zoom, crop, etc. The imaging device 105 in these embodiments is selected to have a maximum frame rate high enough to capture multiple images of the ball as it traverses its FOV in flight, for example, exceeding 100 frames per second (fps). However, the imaging device 105 can also be configured to capture images at lower frame rates.

[0090] The imaging device 105 may have a rolling shutter that captures image data one row (of pixels) at a time in a pattern that moves continuously across the photosensitive chip of the imaging device 105. In such a case, it is preferable that the imaging device 105 is oriented such that the direction in which pixels are exposed across the imaging device through the rolling shutter (i.e., the direction from the first row of exposed pixels to the next row of exposed pixels) is the same as the expected direction of movement of a ball across the field of view of the imaging device 105. At high frame rates, when using an electronic rolling shutter, the image sensor can continue to collect photons during the acquisition process and thus its sensitivity can be effectively increased. This effect is most pronounced when imaging motion under extreme conditions or rapid flashes of light. The imaging device 105 of this embodiment is configured to operate in a selected wavelength band. For example, the imaging device 105 may operate in the visible spectrum, the infrared spectrum, or the near-infrared spectrum. In an indoor environment, the field of view of the imaging device 105 may be illuminated by one or more light sources (not shown) to achieve the desired image quality.

[0091] In some embodiments, the computer 110 may be integrated with the imaging device 105, or the computer 110 may have a separate processing unit. The computer 110 may be connected to the imaging device 105 by a wired or wireless connection. For example, the imaging device 105 and the computer 110 may include respective transceivers (not shown) for sending and receiving data, as will be understood by those skilled in the art. The computer may store computer-readable data in the memory 115 for execution by the processor 120. For example, the memory 115 may include various image processing algorithms such as ball detection, object detection, event detection, deep learning (DL) models, dense optical flow (DOF) models, pre-processing, and post-processing, which will be described in more detail below. The results of the analysis, for example, parameter determination, may be presented to the user by the display 125. For example, the spin parameters of a launched ball may be provided by the display 125.

[0092] In some embodiments, the computer 110 may transmit commands to the imaging device 105 to adjust the operating state of the imaging device 105. For example, based on the output of certain image processing algorithms performed on the image stream from the imaging device 105, the computer 110 may determine that an event is approaching and cause the imaging device 105 to move to a higher power state, for example, to capture launch parameters. When this event is observed as necessary, the computer 110 may then transmit commands to the imaging device 105 to adjust the operating state of the imaging device back to a low power state.

[0093] In low-power mode, the imaging device 105 provides the computer 110 with a continuous stream of images at a low frame rate (e.g., 1 to 100 images / second) and / or alternatively, at a low resolution. In this low-power mode, the computer 110 executes one or more object detection algorithms continuously or semi-continuously for each frame received from the imaging device 105. The processor 120 detects all relevant objects in the scene using object detectors appropriate to its purpose and stores the information in memory 115. Object detection algorithms may include, for example, ball detectors and club detectors, as understood by those skilled in the art.

[0094] Various types of object detectors with varying complexities can be used according to this embodiment. For example, in some applications where high accuracy is desirable (e.g., for ball detection algorithms), it is possible to accurately determine the pixel position corresponding to a ball. In other embodiments, high accuracy may not be necessary or useful for ball detection algorithms. For example, object detectors such as YoloV3 or Faster-RCNN are commonly used in computer vision and / or AI / deep learning applications. As will be understood by those skilled in the art, an object detector may be designed to detect a particular type of object on which the object detector is trained, for example, using training data with detailed annotations. An object detector can be made from scratch or by retraining an already available model.

[0095] In some aspects of these exemplary embodiments, a ball detector is configured to detect a single ball and / or a group of balls, and a club detector is configured to detect a golf club and / or identify the type of golf club using, for example, a classifier, to determine the type of club detected in any image. Figure 2 shows an exemplary image 200 captured by the imaging device 105 described in Figure 1. Image 200 includes a golf club 205, a group of golf balls 210, and a single golf ball 215 separate from the group of golf balls 210. The ball detector detects the single golf ball 215 and the golf balls included in the group of golf balls 210, while the club detector detects the golf club 205.

[0096] Next, the movement of objects detected in previous images can be tracked by an object handler implemented on computer 110, which analyzes the previous and subsequent images. For example, the object handler may match objects detected in the current image with objects detected in previous images (based on, for example, the object type, classification, and location). This provides temporal information about each object, which can be used to determine whether each object is moving or stationary, and for objects determined to be moving, to determine the direction / trajectory and velocity of the object's motion. This association information between two objects of different object types is used to inform the system of the type of event being observed in the scene and whether the system needs to change its current state. A change in the system state will resolve the need for the system to change one or all of the following: for example, the frame rate of the imaging device from low (e.g., 1-100 fps) to high (e.g., 101-3,000 fps), the image resolution, and the image crop (if any). This information is provided by the object handler based on the type of event currently occurring, as identified by the system.

[0097] Figure 3 shows a flowchart 300 for changing the state of an imaging system based on the detection of an event and / or condition, according to one embodiment. The steps relating to the algorithm of flowchart 300 are generally described below, and a more detailed explanation is provided further below.

[0098] In 305, the current image is received from the imaging device. In 310, objects in the current image are located and classified. Information about these objects, such as type and location, is provided to the object handler. For example, the detected objects may include one or more balls and / or clubs.

[0099] In step 315, the object handler compares the object from the current image with previously detected objects. If the current object has not been previously detected, a new object is identified, and this information is stored by the object handler. The object handler then uses logic to determine whether an event of selected interest has occurred or will occur soon, based on the object's movement, for example, between consecutive images or across multiple previous images, as will be described in more detail below. In step 320, the object handler decides whether to change the state of the system based on an analysis of the detected movement of the object and the object's identifiability (i.e., having the trigger event that occurred). Based on the decision in step 320, the system can either change its state (325) or wait for the next frame (330). Steps 305-330 may then be repeated upon receiving the next frame.

[0100] The purpose of event-based detection may be to change the internal settings of the imaging device to enable, for example, high-speed image acquisition (e.g., high-speed image acquisition of an image of a certain time and position, including the time and position at which a golf ball will be struck by contact with the head of a moving golf club). For example, when the movement of the head of a golf club is detected near the location of a single golf ball (e.g., a golf ball separate from other golf balls) (or on a path that is determined to intersect with that location), the system may determine that the ball is about to be struck.

[0101] More specifically, when a single golf ball is stationary and the head of a golf club begins to move away from the single golf ball at a speed and / or acceleration within a predetermined range, the system will identify this as a trigger event (e.g., a backswing at a certain position or on a trajectory where the swing on the ball is predicted), which indicates that the ball will be struck soon, so the system needs to switch to impact detection / analysis mode, or a delay time needs to be determined after the system has switched to impact detection / analysis mode. In this impact detection / analysis mode, the system may change the operation of the imaging device from a low frame rate to a high frame rate and / or change the image resolution of the image. The system may also identify a region of interest (i.e., including the position of the single ball) so that the single golf ball is centered in a cropped image resized to include, for example, the head of the golf club at impact (i.e., reducing the size of the image to decrease the number of pixels to be analyzed), in order to obtain a clear and detailed image of the golf ball at or near the time of impact.

[0102] In one embodiment, when the imaging device captures images at 40 FPS in its normal operating state (e.g., low power state), each image is a still representation of the observed scene, and therefore the type of object present in the scene is known. In this embodiment, the time interval between consecutive images is 0.025 seconds. If the same object is identified at different positions in two or more consecutive images, the type and direction of motion can be determined.

[0103] For example, if the camera remains in the same position and pointed in the same direction, but the same golf ball is identified in multiple consecutive images, but at different image coordinates (u,v), the ball is determined to be moving. If the golf ball is determined to be stationary, but at the same time the golf club is moving slowly toward or away from the golf ball within a given distance from the golf ball, the system may determine that the golfer is addressing the golf ball and that the intention of the person holding the golf club is to begin the backswing before hitting the golf ball. Given the known physical size and shape of the golf ball, the conversion from pixels to meters can be calculated (for example, to determine the distance represented by other pixels located at the same or similar distance from the camera, based on the ball's known radius and the width of the ball's image within multiple pixels).

[0104] If the system detects that a golfer is addressing a golf ball, it analyzes subsequent images to identify a backswing (for example, by analyzing the images to determine whether the golf club has begun moving away from the golf ball at a constant or increasing speed over at least a given range of motion). When the golf club is located by the object detector, the type of golf club can also be classified by the classifier. The type of golf club may be, for example, a driver, wood, hybrid, iron, or a more specific club (e.g., a 4-iron, pitching wedge). Based on the classified club type, threshold requirements (e.g., the speed and range of motion required to identify the movement as a backswing) may vary. These variables may also vary based on historical data regarding the respective swing patterns of multiple golfers with multiple club types.

[0105] Figure 4 shows a flowchart 400 for changing the state of an imaging system based on the detection of an event, according to another exemplary embodiment. In 405, the current image is captured by the imaging device and transmitted to the processing unit. In 410, objects in the current image are detected and identified (i.e., classified). The object detector detects and localizes all relevant objects, i.e., the object detector identifies and localizes all golf balls and golf clubs visible in the image. In 415, the ball manager (e.g., object handler) matches objects from the current image with previously detected objects. If the current object has not been previously detected, a new object is identified, and this information is stored by the object handler. In this example, "n" objects 420 are identified by the object handler. If any club is identified, the club may be classified by type, for example, driver, iron, putter, etc.

[0106] In 425, the scene can be tracked based on information determined by the ball manager regarding objects in the scene. In one embodiment, if it is detected that a ball is separated from other balls / objects (and stationary objects) by a nearby club, the parameters may be set to determine whether a swing is occurring. As described above, various parameters relating to the swing motion may be applied based on the type of club. In 430, it is determined whether a swing, for example, the take-back portion of the swing, is occurring. As used herein, the term “take-back” refers to the portion of the backswing from the setup to the point where the club is substantially parallel to the ground. In 435, if no take-back is occurring, the system waits to receive another image. In 440, if a take-back is occurring, the system transmits a trigger event to adjust the operating state of the imaging device.

[0107] Figure 5 shows a flowchart 500 for changing the state of an imaging system based on the detection of an event according to yet another exemplary embodiment. Links in the flowchart 500 have the informational properties described in detail below, regardless of whether the link relates to further processing. As used herein, the term “link” refers to a branch in the flowchart 500 that includes a step of collecting data, a step of analyzing data, and a step of determining, based on the analysis, whether a state change event has occurred. The flowchart 500 includes links that, if fulfilled, lead to the detection of a certain state. The states in the system can be “not addressed,” “addressed,” “possible takeback,” and “takeback.” If there are no links passing through a state of the system, it is not set to “not addressed,” and no change is required in terms of the system. The processing logic for determining the “addressed” state is ultimately based on distance and angle thresholds for a given club type relative to the ball. However, in this embodiment, “addressed” should be determined after several required steps have been fulfilled.

[0108] In step 502, the current image is captured by the imaging device and transmitted to the processing unit. In step 504, objects in the current image are detected. The object detector detects and localizes all relevant objects, i.e., it localizes all golf balls and golf clubs visible in the image. In step 506, if a golf ball is present, its position is stored in memory, and the process proceeds to step 508. If no golf ball is present, the process proceeds to step 554. Image processing is considered complete, and the processor waits for the next image.

[0109] In 508, when one or more balls are detected, the ball manager determines whether the balls were located within a previous frame or if the detected ball is a new ball. If the ball is new, in 510, a new entry is created in the tracking log. If the ball is known, in 512, the ball's position is updated. If no previously tracked balls are detected, in 514, the entry for the ball in the tracking log is removed. The purpose of the ball manager is to keep track of all detected golf balls, to always know whether new balls have come into or out of view, and to update the positions of known golf balls.

[0110] In step 516, if the golf ball is stationary, the method proceeds to step 518. For example, if two or more consecutive images are captured in which the same ball is detected at the same position, it can be determined that the golf ball is stationary. If the golf ball is not stationary, the method proceeds to step 554, and the image processing is considered complete. In step 518, if a club is detected when the golf ball is stationary, the method proceeds to step 520. At this stage, the club can be detected anywhere in the image. If no club is detected, the method proceeds to step 554, and the image processing is considered complete.

[0111] In 520, the detected club is classified. Certain club types are important for event detection 530 and 540 described below, as different clubs can be swung differently. For example, a pitching wedge may be swung with a shorter and slower swing and backswing than expected with a driver. These considerations can therefore affect when the system enters high-power mode. In 522, a link is calculated between the club and the ball. For example, several distance thresholds may be set between the club and the ball. If the link passes in 524, the state may begin to change, and the method proceeds to 526. If the link does not pass, the method proceeds to 554, and the image processing is considered complete.

[0112] In 526, club information is stored, and based on the club type 528, the address state is calculated in 530. For example, the address state for a driver may differ from that for an iron or putter. To calculate both the address event and the take-back event, it is important to know the type of club that is specific. Based on the club type, various logical thresholds and specific movements will be included or ignored during the calculation of the event. For example, when swinging a driver, the user may address the golf ball with a swing that has a substantially constant circular motion from a stance far away from the golf ball, whereas with a putter, the golfer typically takes a stance very close to the golf ball during the swing and backswing, and addresses the ball with the movement of the putter head kept fairly constant in terms of speed.

[0113] In step 532, it is determined whether an address has occurred. If an address is detected, the process proceeds to step 534. If no address is detected, the process proceeds to step 554, and the image processing is considered complete. If an address event is determined, the processor determines whether the stored link sequence and its relationship to the positioning of the golf club and golf ball trigger the "possible backswing" event.

[0114] In step 536, it is determined whether a take-back is possible. If a take-back is possible, the method proceeds to step 540. If no take-back is detected, the method proceeds to step 552, where potential address events are analyzed. In step 540, the take-back condition is calculated considering the club type 528. In step 542, it is determined whether a "take-back" event is occurring. If a take-back is detected, the method proceeds to step 544. If no take-back is detected, the method proceeds to step 548. In step 548, if the club velocity is directed away from the ball, the method proceeds to step 550, where a take-back possibility event is detected. If the club velocity is not directed away from the ball, the method proceeds to step 552, where an address event is determined. If a "take-back" event is detected in step 544, in step 556, the processor instructs the system to change its state. This means the imaging device will change from a low frame rate and full field of view of the sensor to a high frame rate and a smaller ROI positioned so that the golf ball from the triggered link is centered.

[0115] Those skilled in the art will understand that many different types of event-based logic can be employed in various scenarios to modify the state of a sports analysis system according to exemplary embodiments, and that these exemplary embodiments are not limited to the exemplary scenarios described above or golf in general. Various sets of events in various sports and sports scenarios may follow known patterns or progressions. Algorithms designed to rely on these known patterns can then be used to optimize the operating state of the system, minimizing power consumption and / or processing load while maximizing the likelihood of detecting desired events by utilizing the most appropriate characteristics of the system.

[0116] Some exemplary embodiments are provided below. First, a desired event detection is identified, which is preferably performed by capturing images at a high frame rate, for example, under a high-power state. Next, various events that typically precede the desired event are identified. Finally, exemplary logic is described for triggering a high-power state at a specific point in time relative to the identified preceding events, to ensure that the desired event is captured by utilizing the characteristics of device components suitable for the desired event (e.g., enhanced imaging). Various considerations are described, including potential trade-offs between power consumption and the likelihood of capturing the desired event at the correct time. These embodiments are not limited to golf, and these exemplary embodiments can be extended to other sports.

[0117] Referring to the example above, in the context of a golf match, the detection of desirable events performed under high power conditions could be the impact between the club and the ball and the launch of the ball. The ball may be launched from any suitable hitting area, including a tee box on a golf course, a hitting bay at a golf driving range, an indoor hitting surface (e.g., within a golf simulator environment), or an unspecified area. Some hitting areas may allow for a preferred position and orientation of the imaging device, for example, mounted above the hitting area and oriented straight downwards. Some image processing algorithms may be adapted to the specific position / orientation of the imaging device. However, this is not required and, as will be understood by those skilled in the art, other image processing algorithms may be devised that operate effectively when the imaging device is placed / oriented in various different ways or in any way, as long as the field of view of the imaging device clearly covers the hitting surface (specifically, the ball before launch) without obstructions.

[0118] Some events that typically precede the launch of the ball include the ball moving away from a group of balls, the ball moving to a stationary position a minimum distance away from other objects, the ball being placed on a tee, the golfer holding the club at a specific position relative to the stationary ball (e.g., an approach shot), the backswing, the peak of the backswing, and especially the downswing when the trajectory of the downswing intersects with the position of the stationary ball. These events can be associated with approximate timing relative to impact. For example, when a golfer approaches a stationary ball, the launch can be expected to occur some minimum time after the approach (this amount of time can vary greatly, and in some cases, the launch may not occur after the approach; i.e., the golfer may move away from the ball after the approach without hitting it).

[0119] In another embodiment, when the system detects that the golfer has begun a backswing, a launch may be expected within a predetermined time corresponding to the expected forward swing of the club (which varies based on the type of club being swung, as described above). When the swing reaches its peak, a launch may be expected within a few seconds, a second, or a fraction of a second. When the downswing is detected, a launch may be expected immediately afterward. As previously described, this predetermined time may vary based on the detected club type, the position of the club at the furthest point of the backswing (for example, the point where the direction of the golfer's hand movement begins to reverse, or when the direction of the club head movement reverses), and historical data on the golfer's previous shots.

[0120] In one embodiment, a high-power state for capturing a launch may include, for example, a high frame rate, increased lighting, or higher resolution, and is generally triggered only at a specific point in time after certain essential conditions / events have been detected. For example, if a stationary ball is separated from other objects, the detection of a single ball at least a predetermined distance from all other balls in a given area may be set as essential to trigger a high-power state. Alternatively, the detection of a golfer in a specific posture (approach) and / or the detection of a club moving near a stationary ball may be set as essential to trigger a high-power state. If a club in swing motion is detected but there is no stationary ball nearby, a high-power state cannot be triggered. As will be understood, the system may alternatively operate to enter a high-power state only when a predetermined combination of these conditions is met simultaneously.

[0121] Since the system can analyze human movement, if it detects a golfer approaching the ball and positioning themselves in a swing position, the system may operate to define this as a trigger event indicating that the launch of the golf ball is expected within a predetermined time. However, the system may include information indicating that the time difference between the moment the trigger event is detected and the actual launch time has a duration of at least a predetermined length. In this case, the system may delay the transition to a high-power state for a time span determined based on this predetermined minimum time (i.e., less than or equal to the predetermined minimum time).

[0122] In addition, other power efficiencies may be achieved depending on this detection. For example, certain image processing algorithms may be executed before the approach, while others may be executed only after the golfer's approach has been detected. In this embodiment, a ball detection algorithm, a club detection algorithm, and / or a golfer posture detection algorithm may be executed to detect a golfer who is in a position near a stationary ball and in a posture indicating an approach to the ball. After this event is detected, a swing-related algorithm (i.e., an algorithm that analyzes the image to detect a movement indicating that the golfer is beginning a backswing in preparation to swing the club toward hitting the ball) may be triggered. In this case, the system may trigger a high-power mode only after a specific swing-related detection (e.g., backswing, apex, or start of downswing) has been performed.

[0123] The system can be fine-tuned to enter a high-power state only for the minimum duration necessary to capture the desired data. For example, in one embodiment, the system may enter a high-power state for a selected time range that includes only the desired duration immediately before the estimated launch time, which continues until impact, and for a specific length of time after impact required to establish the initial launch parameters. Alternatively, depending on the time required to transition to the high-power state, the system may immediately trigger the high-power state when a specific event is detected (e.g., the start of a downswing).

[0124] However, depending on the known or estimated duration from peak to impact (and further, depending on the processing delay to detect the event and / or initiate the change in power state), the high-power state may be entered at the optimal time immediately before impact. If the timing of future events cannot be predicted with high accuracy, the high-power state may be entered earlier (longer than the buffer time before the observed event), which may continue for a longer period after the expected future event to ensure that all time related to the event is captured.

[0125] Further logic can be implemented depending on different system setups. For example, the imaging device may not be placed / oriented in a predetermined position, may be placed / oriented manually, and may encounter obstacles. In some scenarios, the imaging device may have a field of view covering multiple hitting positions used by multiple golfers. In one exemplary embodiment, the movement of a first golfer closer to the imaging device may interfere with the detection of an event of a second golfer farther away from the imaging device. In this case, an event detected for the second golfer may trigger a high-power state early in the swing's progression under certain circumstances. For example, if an approach is detected for the second golfer and the target ball is clearly visible, but the actual swing motion is partially obstructed by another object (e.g., as the first golfer moves around the hitting position), a high-power state may be entered for a minimum amount of time after the second golfer's approach is detected, even if the actual swing is not detected.

[0126] In some embodiments, the system is designed to detect events and impacts of multiple golfers at various locations. For example, the imaging device may be positioned between two hitting bays such that the first half of the image includes the first hitting bay and the second half of the image includes the second hitting bay, and as a result, objects can be detected and tracked simultaneously by the system as they are launched from both hitting bays. If a future launch is expected for one of the golfers in the first hitting bay, the imaging device may temporarily enter a high-power state, allocating its resources to detect this launch, while the second golfer is temporarily ignored. From a practical standpoint, the system may define a region of interest around the ball in the first hitting bay, with the exception of analyzing activity in the second hitting bay. After a launch from the first hitting bay is detected, the imaging device may then resume object tracking for both hitting bays until another trigger event is detected indicating that a launch is expected in one or both hitting bays.

[0127] Therefore, although various embodiments of this disclosure have been described with respect to an imaging device mounted straight downward and above a single hitting area, those skilled in the art will understand that different positions / orientations of the imaging device can be used in conjunction with various arrangements in which the ball is expected to be launched.

[0128] Furthermore, it should be noted that high-power states may be desirable for detecting various types of shots in a golf match other than the initial impact / launch. For example, it may be desirable to detect and determine parameters related to bounce, such as the impact of the ball with a surface after flight. In this example, the prerequisite for entering a high-power state may be detected based on tracking data of the ball's flight indicating that impact with the ground is imminent. For example, the system may track the ball's trajectory (using any tracking device, such as an optical or radar-based tracking system) along with data that detects the geometric shape of the landscape into which the launched ball will land and / or utilizes stored information about the terrain of the area where the ball will land, and predict the time and location where the ball will land. Based on this information, the system may transition the imaging device to a high-power state immediately before the expected landing time. In this embodiment, the imaging device may be placed adjacent to the putting area (e.g., the green), facing the target pin or any other location where an incoming shot is expected to land.

[0129] It should be understood that the exemplary embodiments are described in relation to operations performed using images from a single imaging device. However, there are scenarios in which it may be beneficial to use additional imaging devices and / or other sensors to detect essential events that indicate anticipated upcoming events. In one embodiment, a high-power imaging device may cover multiple hitting areas on a golf driving range. Other sensors may be employed to detect any of the described trigger events that indicate an event to be observed is imminent (e.g., by tracking the position and / or movement of a ball, club, or golfer, or by tracking the golfer's posture and / or the orientation of a moving club), and this information may be used to control the system entering and exiting the high-power state. When it is determined that a launch or any other event to be observed is imminent, the high-power imaging device may be triggered to enter high-power mode and focus on the region of interest where the event is expected to occur.

[0130] In the context of a baseball game, desirable event detections performed under high power conditions may include the release of the pitch or the impact of the ball on the bat. Similar to the golf scenario described above, a baseball pitch is thrown from a clearly defined pitching area, for example, from a mound on a field or practice environment equipped with a pitching rubber (pitcher's plate). The system uses empirical information to establish the region of interest (for example, using the fact that baseball rules require the pitcher to maintain contact between the pitching rubber and their foot when starting the windup prior to every pitch, and that the ball can only be hit by a batter standing in a designated batter's box on the field), or if the user specifies the region of interest (for example, in a practice environment adjacent to home plate to which the pitch is aimed). Some pitching / hitting areas may allow for a preferred position and orientation of the imaging device, for example, mounted above the pitching / hitting area and oriented straight downwards, but this is not required. Therefore, if the system is trying to observe the release of the pitched ball and / or the impact of the pitched ball on the bat, this empirical information can significantly reduce the portion of the image that needs to be analyzed relative to the system, thereby reducing the computational load required to analyze the image generated by the system.

[0131] Some events that typically precede the pitching of the ball include the pitcher making contact with the pitching rubber and entering the pitching stance (when the pitcher is in the “set position”), the initial part of the pitching motion (stepping back, raising the leg), the pitcher pulling the arm back, and the forward movement of the pitcher’s arm. These events can be associated with the approximate timing relative to the release of the ball, and / or when the ball will reach a position where the batter can potentially hit it. Once the pitcher’s foot makes contact with the pitching rubber and the start of the windup is detected, the pitch may be anticipated within a short time (which will vary depending on the individual pitcher’s pitching motion). Therefore, once this windup is detected, an initial value regarding the time to the release of the ball may be adopted, and this value may then be modified, if desired, based on historical data (e.g., times from previous pitches, pitching sessions, or past performances of the pitcher during the same game).

[0132] As those skilled in the art will understand, the system may detect when the pitcher is using a full windup (generally, with no runners on base) or when pitching from a stretch position, and in these situations, use separate values ​​for the expected time from windup detection to pitch release. Alternatively, the system may first detect the start of the windup, and after this detection, detect the start of the pitcher's forward arm movement during the pitch, as this would indicate a shorter and more predictable time to pitch release. Briefly, the system may detect the start of the windup and use it as a trigger event to begin a countdown of a predetermined period (i.e., a countdown to the time expected to precede an event observed by a desired buffer amount) until the system switches to a high-power mode.

[0133] In the case of detecting contact between the ball and the bat, this time value is approximately equal to the time determined to be remaining until the release of the pitch and the time it takes for the thrown ball to travel to the area where it will reach the batter. The system may switch to high-power mode before the release of the thrown ball (if this is an event that is to be observed), or, if the only event that is to be observed is the ball being hit, the system may delay switching to high-power mode for an additional time approximately equal to the predicted travel time of the ball from the release point until the ball enters the zone where it can be hit.

[0134] If it is desired that both events be observed in high-power mode, the system may enter high-power mode immediately before the release of the pitch and exit high-power mode after the ball hits or passes home plate, or after the batter has passed the area where they are stationary. High-power conditions for capturing a pitch may include, for example, a high frame rate, increased lighting, and high resolution. Typically, some events that may precede the bat hitting the ball include the batter getting set in the batting position, the release of the pitch, the batter lifting their leg or otherwise preparing to swing, and the swing of the bat. As described above, the system may refuse to enter high-power mode if, even as the ball approaches the hitting zone (the area where the ball is within the batter's reach), no movement indicating the start of a swing is detected at the same time.

[0135] Similar to the golf-related examples described above, some event detections may imply anticipated future events but may not immediately trigger a high-power state. Various posture detection algorithms or arm / bat movement algorithms for pitchers or batters can be used to trigger a high-power state at a desired time.

[0136] In relation to a football (soccer) match, the desired event detection performed under high power conditions could be the kicker striking the ball. The ball may be kicked from any suitable launch area, e.g., from the position of a free kick or penalty kick. Some kicking areas may allow for a preferred position and orientation of the imaging device, for example, mounted above the kicking area and oriented straight downwards, but this is not required.

[0137] Some events that typically precede a kick include the ball moving to a stationary position a minimum distance away from other objects, the kicker positioning himself a minimum or maximum distance away from the stationary ball, the kicker approaching the stationary ball, and the kicker pulling his leg back in preparation for the kick.

[0138] Similar to the embodiments described above, some event detections may imply anticipated future events but may not immediately trigger a high-power state. Various posture detection algorithms for the kicker or motion detection algorithms for the kicker's legs can be used to trigger a high-power state at a desired time.

[0139] In relation to an American football game, a desired event detection performed under high power conditions could be, for example, a kicker striking the ball towards a field goal or a punt. The ball may be kicked from any suitable launching area, e.g., the position of the field goal. Some kicking areas may allow for a preferred position and orientation of the imaging device, for example, mounted above the kicking area and oriented straight downwards. Some image processing algorithms may be adapted to a specific position / orientation of the imaging device, but this is not required.

[0140] Some events that typically precede a football kick include, for example, the ball being moved to a stationary position and held upright by a device or a player; the kicker being positioned at a minimum or maximum distance from the stationary ball; the kicker approaching the stationary ball; and the kicker pulling their leg back in preparation for the kick.

[0141] Similar to the embodiments described above, some event detections may imply anticipated future events but may not immediately trigger a high-power state. Various posture detection algorithms for the kicker or motion detection algorithms for the kicker's legs can be used to trigger a high-power state at a desired time.

[0142] In the context of a tennis match, the events that are desirable to observe under high power conditions may be the striking of the ball by the racket. The ball can be struck from any area within the court and the surrounding area, but as will be understood by those skilled in the art, the ball's serve and the first return of the serve generally occur in areas that are more predictable than those that can be targeted by the system in the same manner as described above. In addition, by tracking the movement of the player and the ball, the position of substantially all shots can be predicted, and in the same manner as described above, the area of ​​interest can be reduced, thereby decreasing the computational load on the system. Depending on the construction of the structure surrounding the court, preferred positions and orientations for one or more imaging devices may be selected (e.g., mounted above the hitting area and oriented straight downward, or raised behind one or both ends of the court and pointed downward towards the court). The image processing algorithm can, if desired, be adapted to the specific position / orientation of the imaging device selected for each situation.

[0143] Typically, some events that may precede a tennis ball serve include the player taking the serving position, the ball being thrown upwards, the hitter pulling the racket back, and the hitter swinging the racket forward. Each or all of these actions can be detected to determine the time before the event is observed (e.g., hitting the ball with the racket for the serve), and as a result, a high-power mode may be entered before the observed event occurs (e.g., a high-power mode set to enter with a predetermined buffer time before the event is expected).

[0144] Similar to the embodiments described above, some event detections may imply anticipated future events but may not immediately trigger a high-power state. Using various posture detection algorithms for the tennis player or motion detection algorithms for the player's arm or racket, a high-power state can be triggered only at desired times (i.e., to minimize the time spent in high-power mode during which no action related to observed events occurs).

[0145] In the context of volleyball matches, a desirable event detection to perform under high-power conditions might be a volleyball player hitting the ball. For serves, the ball may be hit from a predictable area, such as behind the serve line, which can reduce the size of the region of interest and minimize computational resource consumption. For all other shots, tracking the ball and players allows for determining the likely location of the shot in the same way as described above for tennis. Depending on the construction of the structure surrounding the court, preferred positions and orientations for one or more imaging devices may be selected (e.g., mounted above the hitting area and oriented straight downwards, or raised behind one or both ends of the court and pointed downwards towards the court). Image processing algorithms can, if desired, be adapted to the specific position / orientation of the imaging device selected for each situation.

[0146] Typically, some events that may precede a volleyball serve include the player taking their serving position, the ball being thrown upwards, the hitter pulling their arm back, and the hitter swinging their arm forward, and at any time thereafter the ball's trajectory being within the range of any player's arm length (for example, in combination with one of the player's arms moving into a position related to getting ready to hit).

[0147] Similar to the embodiments described above, some event detections may imply anticipated future events but may not immediately trigger a high-power state. Various posture detection algorithms for volleyball players, or motion detection algorithms for the players' arms, can be used to trigger a high-power state only at desired times (i.e., to minimize the time spent in high-power mode during which no action related to observed events occurs).

[0148] It should be noted that certain detections may be possible with very low frame rates, resolutions, and lighting. Clearly, the accuracy and quality of these detections can be improved with higher frame rates. However, if accuracy can be adequately guaranteed with specific operating parameters for an imaging device that is less resource-dependent than those in high-power modes, these operating parameters may be used, for example, in scenarios where CPU resources are limited.

[0149] Different event detection processes and / or parameter determinations may be associated with different processing loads and durations over which those loads may be imposed. Some processes may run continuously, for example, detecting certain objects. Other processes may run only when a specific essential event is first detected, for example, a ball detected as moving may trigger various swing-related event detection processes.

[0150] Certain operations may only be available to computing devices with specific operational capabilities. System software can detect the capabilities of the camera and CPU, enabling the calculation of different events / parameters based on these capabilities. For example, if the camera is limited but the CPU is powerful, many basic event detection algorithms can potentially be run in tandem. Conversely, if the camera is powerful but the CPU is limited, complex processes using high resolution / high frame rates, for example, may only be run in the highest-priority scenarios, and decisions regarding parameters that determine when to enter high-speed mode may vary based on the CPU's current operating conditions.

[0151] In some embodiments, specific modules of the processing unit may be assigned to different tasks requiring varying amounts of processing power. For example, one module may be assigned to object detection and can run continuously. Another module may be assigned to motion detection and can run continuously. Yet another module may be assigned to event detection and can only run when, for example, essential conditions are met, and indicators of such conditions are received from modules that are already running. Each module may have some known processing / power consumption, and some may only be activated under specific conditions, such as when there is no risk of the system overheating.

[0152] Figure 6 shows a method 600 for triggering a change in the operating state of an imaging device according to various exemplary embodiments.

[0153] In 605, the ball is detected within a sequence of images captured by the imaging device. The imaging device may be configured with current operating parameters. In some embodiments, the ball may be a golf ball. In other embodiments, the ball may be another type of sports ball, as described above. Current operating parameters may include low-power states in which the imaging device captures images at a low frame rate, low resolution, or other relaxed parameters. The ball can be detected by implementing various types of ball detection algorithms. Further detection, such as club detection, may also be performed. In some embodiments, ball detection may be a prerequisite for performing further image processing analysis, such as club detection.

[0154] In 610, one or more conditions are analyzed regarding the movement of the ball and / or ball-hitting equipment. Initial conditions may include, for example, that a single ball is a certain distance away from other objects (such as a group of balls), that the distant ball is stationary, that the club is detected within the frame, and / or that the club is detected within a predetermined distance of the stationary ball. If one or more of these requirements are met, further conditions such as an approach or swing may be analyzed. The thresholds used to satisfy these conditions may vary based on the classification of the club type.

[0155] In 615, a change in the operating state of the imaging device is triggered when one or more conditions are met. For example, a condition may indicate that the ball is about to be launched. In some embodiments, the operating state of the imaging device may be triggered to transition from a low-power state to a high-power state. The high-power state may include operating parameters such as frame rate and resolution that are increased compared to the low-power state.

[0156] In the 620, the ball is detected within a further sequence of images captured by the imaging device using adjusted operating parameters. For example, the launch of the ball can be captured in great detail in a large number of images spaced very closely together in time. These images can be processed in various ways to derive the desired launch parameters.

[0157] In some embodiments, the desired launch parameters include spin parameters, including three-dimensional spin velocity and spin axis. Current techniques for analyzing ball movement in sports using a single imaging device are typically limited to measuring the velocity vector and speed of the moving ball. According to the technique described in detail below, the system can extract the complete 3D rotation vector in a shot of a sports ball by using only a few images acquired by a single camera.

[0158] Another aspect of these exemplary embodiments describes a method for calculating the spin parameters of a spinning spherical ball. In these embodiments, the complete three-dimensional (3D) rotation vector of a moving and rotating spherical sports ball is measured using images acquired by only a single camera. While some aspects of these embodiments are described in relation to golf, these methods can be applied to many other possible sports, including, for example, baseball, tennis, soccer, and volleyball.

[0159] The method described herein calculates the 3D spin axis (SA) of a moving ball by observing only a small portion of the shot's trajectory, for example, at launch. While designed to utilize the availability of a suitable number of frames (e.g., the first 15–30 of a shot), the algorithm can also be performed based on two frames. The algorithm is based on estimating the displacement of pixels within the ball between consecutive frames.

[0160] 3D rotation vector information is important in several applications. For example, if the 3D rotation vector is estimated using only the first few frames available in a sports ball shot, this information can be used in conjunction with launch angle, launch direction, and ball velocity information to provide the ball's complete trajectory. This is important in several applications, such as golf simulators, golf training, soccer training, and tennis training.

[0161] For these types of applications, it is desirable to provide output in less than 0.5 seconds by running on ordinary hardware such as a standard commercially available laptop. Therefore, the algorithms described in this document were developed for near real-time execution.

[0162] An algorithm for estimating 3D spatial awareness (SA) may be called a ball tracker (BT). Within this framework, this method can be used to analyze either optical (RGB) or infrared (IR) images in either 3-channel or 1-channel format.

[0163] BT works with optical or infrared images acquired inside a moving and spinning sports ball to provide 3D SA measurements. This is based on finding the exact displacement of every pixel belonging to the ball between consecutive frames. This is done by using the estimation of dense optical flow (DOF), which is a vector field that describes the pixel movement for every pixel in two input images. This is based on the recognition and tracking of all recognizable features on the ball's surface. For example, in the case of a golf ball, this could be the ball's label or a dent on the ball's surface. In this context, it is crucial to have high-resolution images acquired at fairly high video speeds and extremely short exposure times.

[0164] The primary requirement is determined by the maximum observable spin velocity (SR). Given that the algorithm relies on estimating pixel displacement, it is important that some of the same ball features are observable within two consecutive frames. To ensure this, features appearing on the edge of the ball in the first frame can at most be at the center of the ball in the next frame.

[0165] For example, consider a golf shot. According to golf literature, a typical golf shot averages several thousand RPM (revolutions per minute), although this varies depending on the type of club used. Examples of shots exceeding 10,000 RPM are difficult to find. Therefore, a conservative threshold of 15,000 RPM can be used. At this maximum value, the image would need to be acquired at a video speed of 1500 fps or higher.

[0166] It is preferable that 3-4 full rotations of the ball are available, which typically corresponds to 15-30 frames. However, the minimum threshold for the algorithm is just 2 images. Furthermore, the image resolution must be sufficient to correctly track the ball's features, such as images where the ball has a radius of 25 pixels or more. The images can be calibrated. Taking into account the ball radius information in meters, the sensor coordinates (u,v) can be converted to real coordinates (x,y,z).

[0167] Figure 7 shows a flowchart 700 for calculating the 3D spin axis (SA) of a moving sports ball from a sequence of images from a single imaging device, according to various exemplary embodiments. In flowchart 700, the operation of the algorithm is shown by boxes separated by continuous lines, the inputs and intermediate outputs of the algorithm are shown by boxes separated by dotted lines, and the deep learning (DL) model is shown by a cylindrical box. Flowchart 700 provides an overview of the ball tracker (BT) operation, but various aspects of flowchart 700 are described in more detail below with respect to Figures 8-11.

[0168] In 705, a set of "n" images is received as input. The set contains at least two images (n=2), but the accuracy of SA estimation can be improved with additional images, e.g., n=15-30. The set of images can include, for example, consecutive images captured at a high frame rate such as Xfps. The images can show the ball being launched. The images must show the ball in motion.

[0169] In 710, the ball detection algorithm is run on all input images. The ball detection algorithm is run considering the DL ball model 715, which is described in more detail below with respect to Figure 10. In the first step, the ball detection algorithm extracts very precise pixel and radius measurements of the ball. In the second step, a new crop of the input image is created where the ball is always the same size and in the same position.

[0170] In 720, dense optical flow (DOF) is estimated for every two consecutive images. Therefore, when using two input images, one DOF is estimated. For a set of "n" input images, n-1 DOFs are estimated. The DOFs are estimated considering the DL DOF model 725, which is explained in more detail below with respect to Figure 8. These n-1 DOFs are then post-processed, which is explained in more detail below with respect to Figure 9.

[0171] In step 730, the post-processed DOF is analyzed to calculate a 3D rotation vector, including the ball's spin velocity (SR) and spin axis (SA). The 3D rotation vector is calculated considering the scene's geometric shape 735, including camera and ball position information. In step 740, the 3D spin axis is output for display to the user.

[0172] According to one embodiment of these exemplary features, a procedure for extracting depth of field (DOF) from a sequence of image inputs describing a moving and spinning sports ball is described.

[0173] Figure 8 shows a flowchart 800 for calculating the dense optical flow (DOF) for a sequence of image inputs describing a moving and spinning sports ball, according to various exemplary embodiments. Figure 9 shows a flowchart 900 for calculating the three-dimensional (3D) spin axis from the dense optical flow (DOF) for a sequence of image inputs describing a moving and spinning sports ball, according to various exemplary embodiments. Similar to flowchart 700, the operation of the algorithm is shown by boxes separated by continuous lines, while the inputs and intermediate outputs of the algorithm are shown by boxes separated by dotted lines, and the deep learning (DL) model is shown by a cylindrical box. The steps of the algorithm are generally described below, and a more detailed description is provided further below.

[0174] A pair of images, for example, the first image 805a and the second image 805b, are received. The ball model 810 is implemented within a ball detection operation, for example, ball detection of the first and second images 805a and 805b. Ball parameters, including the ball's center and radius, are estimated for the first and second balls in 820a and 820b. These parameters are used to determine the crop 825a and 825b for the first and second images. The ball is at the center of the cropped image. The cropped image is then reshaped to a predetermined pixel size 830a and 830b, and the radius is calculated. From these image processing steps, the first and second modified images 835a and 835b are generated, having a center = (nd × nd) and a radius nd-1. The dense optical flow (DOF) model 840 is implemented within DOF inference 845. The first flow 850 is output.

[0175] DOF inference 845 can be performed on all pairs of consecutive images. For example, if the third image is received after the second image, DOF inference 845 can be performed on both the second and third images. For a set of n consecutive images capturing a spinning sports ball, n-1 flows can be generated.

[0176] Regarding Figure 9, at 905, multiple flows 905, for example, n-1 flows, are received. At 910, spatial and temporal coherence filters are applied. At 915, a median filter is applied. A flow median 920 is generated. Based on the ball characteristics 925 of the flow median, at 930, the 3D SA is calculated.

[0177] The steps of the algorithm described above rely on DOF estimation, as shown in Figure 8. For a pair of images, DOF estimation provides information about the shift of each pixel along the "u" and "v" axes from the first image to the second image. The DOF map contains information that enables the measurement of SR and 3D SA of the rotating ball.

[0178] If the DOF map is calculated from a video of a moving ball, the DOF will typically also provide information about the ball's translational movement. To remove this trajectory information from the analysis and generate a DOF that describes only the ball's rotation, the images are first reshaped in a preprocessing operation, referring to operations 830a,b described above with respect to Figure 8. The reshaped pair of images, for example, the first and second images 835a, 835b, have equivalent centers and radii with respect to the ball. This can be done if very accurate (sub-pixel precision) ball detection has been performed beforehand. This is done by using a DL-based ball detector, referring to operations 815a,b described above.

[0179] Subpixel-precision center and radius are used to crop and reshape all available input images. The crop is (2n d x 2n d ) has a size of and all crops are pixels (n d xn d Precisely centered within ) with radius n d You will have a ball with a -1.

[0180] Here, n d However, the following DOF method determines the pixel dimensions of the window in which it works. Using a larger window can achieve better performance, while using a smaller window can achieve faster inference. As a trade-off, n d You can use =64, but larger or smaller values ​​can also be used. Further information regarding the affine transformation used to crop and reshape the input image is provided below.

[0181] A DL algorithm called VCN (Volumetric Correspondence Networks) can be used for DOF estimation. Other DL algorithms such as LiteFlowNet or RAFT can also be used. Only a very accurate estimation of the displacement of pixels in the input image is required. The more important part is training the DL DOF model, which is shown above in Figure 8 and described in more detail below, referring to operation 840.

[0182] Ball detection algorithms can be implemented as object detection DL-based methods, such as standard SSDs or modern YOLO. Such algorithms can be applied to ball sports. Ball detection algorithms may also be based on segmentation networks, an example of which is provided below. However, various ball detectors with appropriate accuracy can be used.

[0183] For example, a hybrid architecture can be used that combines two well-known architectures, UNET and VNET. This architecture may be called DeepVOG.

[0184] Figure 10 shows implementation examples of the DeepVOG architecture 1000 in various exemplary embodiments. In this example, the input is a 128x128 pixel monochrome image (one input channel), and 16 intermediate channels are used. In this implementation, the segmentation network provides a 128x128 two-channel image as output, where each pixel in the two channels describes the probability that the pixel belongs to either the "background" class or the "ball" class. In the architecture 1000 shown in Figure 10, convolution symbols indicate a complete convolutional layer, downward arrows indicate a downsampling layer, upward arrows indicate an upsampling layer, and horizontal lines indicate connections along channel layers. For each block, the number of rows and columns is shown on the side, while the number of channels is shown at the top.

[0185] Considering having an estimated value of the center coordinates (c x , c y ) and the radius r, the acquisition of the ball crop is the floor integer representation of those estimated values

Number

Number

Number

Number

[0186] In any case, the DOF map estimated by the algorithm is desired to represent only the rotation of the ball. For this purpose, it is very important that all images are centered around the exact center of the ball. Also, it is very important that the ball size does not change between images.

[0187] Of course, in the original image, if the ball is moving, its apparent size will change slightly from frame to frame due to the change in distance from the camera. To solve the problem of placing the crop exactly centered within the floating estimate of its center, an affine transformation can be used. The algorithm obtains a slightly larger crop C1 with coordinates as follows.

Number

number

number

[0188] Figure 11 shows a flowchart 1100 for training a deep learning (DL) dense optical flow (DOF) model in various exemplary embodiments. This training may depend on the availability of simulation images of several shots. Simulations may be generated using software programs such as Blender or CAD. The simulation is created using a "reference ball" and a "true ball". That is, the reference ball is a ball with very well recognizable features and patterns that can be more easily tracked by any DOF algorithm, while the true ball is a much more realistic sports ball. The same shot is repeated twice in the simulator with the same launch parameters (ball velocity, launch angle, and launch direction, etc.) and the same 3D SA vector, the first time using the reference ball and the second time using the true ball. It is reasonable that the currently available models will be accurate by providing a good estimate of the DOF on the reference ball. An image of the reference ball 1105 is input, and the DOF model 1110 (e.g., untrained) is implemented within the first dense optical flow estimate 1115. The geometric shape 1120 of this scene is used to calculate the 3D spin parameters 1125. The true 3D spin parameters 1130 are used in the iterative flow adaptation 1135, and a test 1140 is performed to determine the ground truth 1145 of the true ball and reference ball. The true ball parameters 1150 and ground truth 1145 are used to determine the data augmentation 1155 for training the DOF model 1110. Using the trained DOF model 1110, the DOF analysis shown in Figure 8 can be performed.

[0189] Figure 9 shows a post-processing algorithm applied to n~1 available DOFs, extracting the intrinsic DOF of the shot and using it to extract the 3D SA by applying the formula described below.

[0190] In 905, the spatial and temporal coherence of available DOFs is analyzed. Each pair of input images must be characterized by a DOF that is spatially coherent; that is, the magnitude and direction of pixel displacement should not vary much between one pixel and its neighbors. If, instead, a significant change is detected between the displacements of some neighboring pixels, this DOF ​​is classified as not spatially coherent and is filtered out for the calculation of the 3D SA.

[0191] The same principle can be applied in the time direction. Every pixel displacement must be characterized by a small variation between one pair of input images and the next pair of input images. Otherwise, these two consecutive DOFs are assigned as not temporally coherent and are filtered out.

[0192] After filtering and removing this DOF ​​using spatial and temporal coherence analysis, the median DOF is calculated using the instantaneous 3D ball position. In fact, considering that the image is reshaped with a new crop and centered again, the DOF is a function of the 3D SA of the ball. In any case, it is also a function of the ball's original position in the image; that is, the same rotation will appear different from the camera depending on the position the ball occupies due to its projection. Thus, the temporal median DOF is calculated by taking projection into account.

[0193] As a result of a temporal median filter, the output is a single DOF. The formula for the SA of 3D is calculated only once for each DOF (median). However, this is not the only possible way to achieve the result. The SA of 3D can be calculated for each available DOF, and only then can the median of the resulting SA of 3D be evaluated. It is possible to provide instantaneous measurements of the SA of 3D for each input frame without evaluating the median at all. Even a median filter can be replaced by the mean or a weighted average. In the context of BT, a median filter can be used to obtain an acceptable trade-off between performance and the computational load of the algorithm so that BT can run near real time. More complex operations are possible on different hardware, or when the algorithm can spend more time providing an output. The calculation of the SA of 3D from a given DOF can use standard linear algebra techniques such as singular value decomposition (SVD).

[0194] Figure 12 shows an exemplary embodiment of an overhead launch monitor system 1200 for a golf simulator. The overhead launch monitor system 1200 may be for indoor use. The overhead launch monitor system 1200 according to the illustrated embodiment comprises a single imaging device 1210 positioned above a ground spot from which a golf player is expected to hit a golf ball 1270 with a golf club 1280. When playing indoors, the imaging device 1210 may be mounted on the ceiling or attached to a stand or rack 1260, and when playing outdoors, the imaging device 1210 may be fixed to a stand or rack 1260. The imaging device 1210 is preferably positioned 3 to 4.5 meters above the ground spot from which a golf player is expected to hit a golf ball 1270. The imaging device 1210 may include an optical system 1215 that defines the field axis 1216 and the aperture angle 1218 of the imaging device 1210. The imaging device 1210 has a field of view (FOV) that defines a visually observable scenario. The opening angle 1218 ensures that the field of view of the imaging device 1210 includes the golf ball 1270, at least a portion of the golf backswing path of the golf club 1280, and a portion of the path of the golf ball 1270 after it has been struck by the golf club 1280.

[0195] The imaging device 1210 may be a high-speed camera and can operate in a normal state with a resolution of 4096 x 2160 (4K or Ultra HD) and a frame rate of 40 fps (frames / second), as well as in a high-speed state with a frame rate of 2000 to 4000 fps, and a cropped image is output. The controller 1220 controls the state of the imaging device 1210 and manages the readings from the camera sensor chip 1212 of the imaging device 1210.

[0196] Under normal conditions, digital images are continuously supplied to the processor 1230 via connection 1250. The processor 1230 outputs the data stream to the monitor 1240 to display a video of the simulation or virtual golf course. The processor 1230 is further configured to analyze the captured images to identify the associated golf ball 1270 and golf club 1280. When the processor 1230 detects the backswing of the golf club as described above, the processor 1230 classifies it as a state change event and instructs the controller 1220 to change the state from normal to high speed. The instruction also includes a crop instruction indicating which portion of the normal image the processor 1230 wants to receive in high speed. There is no need to increase the speed of the processor 1230 as long as the data savings from cropping outweigh the data cost resulting from the higher frame rate.

[0197] The cropped image may have a resolution of 200 x 200, for example, that matches the region of interest (ROI). This means that only pixels present in the ROI are read from the imaging device 1210 and transferred to the processor 1230 via the connection 1250. This results in data savings of more than 100 times, and since the data cost due to higher frame rates is 50 to 100 times higher, the amount of data transferred from the imaging device 1210 to the processor 1230 is slightly reduced.

[0198] As the golfer prepares to hit the ball by taking the golf club back, the state changes to a high-speed state. The golf club 1280 then moves toward the golf ball 1270 as indicated by arrow 1282. When the golf club 1280 collides with the golf ball 1270, during the impact, the golf ball 1270 begins to roll on the hitting surface of the golf club 1280 as indicated by arrow 1274, and exits the hitting zone along the path indicated by arrow 1272. The flight of the golf ball 1270 along the path indicated by arrow 1272 is determined by the impact from the golf club 1280 and the spin resulting from the rotation indicated by arrow 1272, and the initial portion of the golf ball flight is recorded and analyzed by processor 1230 for use in the golf simulation to calculate and present the golf ball flight in a simulated or virtual golf course video streamed to monitor 1240.

[0199] The golf ball 1270 can leave the hitting zone at a speed of, for example, 60 m / s. The imaging device 1210 can track the golf ball 1270 for up to 1 meter before it leaves the field of view of the imaging device 1210. This means that the imaging device 1210 can track the golf ball 1270 for 1 / 60 of a second. When the imaging device 1210 is running at high speed, at 3000 fps, the camera can acquire approximately 50 consecutive images before the golf ball disappears from view.

[0200] Due to the hard crop ratio, it is important to recrop the camera image several times during the acquisition of the image sequence. In one embodiment, the impact of the club is used to predict the path of the golf ball 1270, and data from the cropped image is used to correct the predicted path of the golf ball 1270. The predicted path of the golf ball 1270 is used to recrop the captured camera image. In some cases, the processor 1230 sends a recrop command before each capture of the cropped image.

[0201] The sensor chip 1212 may be a charge-coupled device (CCD) or a CMOS active pixel sensor. Signal processing may be performed in a dedicated embedded circuit due to speed and power considerations. The sensor chip 1212 has multiple discrete pixel elements that are read by the controller 1220. In high-speed conditions, the controller 1220 may handle a subset of discrete pixel elements for reading a cropped image containing only the region of interest within the entire view of the imaging device.

[0202] The sensor chip 1212 preferably includes a global shutter. The global shutter applies an advanced method of reading data from the sensor chip 1212 so that the entire sensor is read at once. The global shutter operates by exposing the entire scene from top to bottom at once so that the sensor takes a snapshot of the scene contained in the field of view using all pixels at once. The controller 1220 includes a crop unit 1225. The crop unit 1225 is configured to receive data read from the sensor chip 1212 as a data input, receive data about the region of interest from the processor 1230 as a control input, and deliver cropped image data for the region of interest in the high-speed state of the imaging device 1210. The crop unit 1225 passes the image data for the region of interest (cropped image) to the processor 1230. In this way, the crop unit 1225 ignores all image data captured by the sensor chip 1212 outside the region of interest.

[0203] The crop unit 1225 may include logic circuits implemented using purpose-built hardware such as application-specific integrated circuits.

[0204] Reading the sensor multiple times at once offers several advantages. This minimizes the risk of warping or distortion when photographing fast-moving objects. The imaging device 1210 is also more robust against vibration. This also benefits video, as it eliminates the effects of blur when shooting in high-speed environments.

[0205] The imaging device 1210 does not require a mechanical shutter blade, which means fewer moving parts and a lower chance of camera damage. For indoor use, sufficient background light is required to obtain high-quality cropped images, which is easier with a global shutter because it has lower synchronization requirements compared to, for example, a roller shutter setup.

[0206] According to this aspect of the present invention, the processor 1230 receives data from the imaging device 1210 and controls the state of the imaging device 1210 based on the data acquired by the imaging device 1210 itself.

[0207] Figure 13 shows a flowchart of the operations performed by the system in Figure 12, which begin before the take-back event, during the flight of the golf ball, and end when the golf ball leaves the field of view of the imaging device. In step 1305, the imaging device 1210 captures an image of the ground spot from which the golf player is expected to hit the golf ball 1270. The image is captured at a normal video frame rate (e.g., 40fps). The captured image data is transferred to the processor 1230. In step 1310, the processor searches the image data for elements that may indicate the presence of a take-back event. Unless a take-back event is detected, the imaging device 1210 continues to capture images at a normal video frame rate in the normal state and transfer the image data to the processor 1230. If, in step 1310, the processor 1230 identifies elements in the image data that may indicate the presence of a take-back event, the processor 1230 instructs the imaging device 1210, via the controller 1220, to change its state from the normal state to the high-speed state in step 1315.

[0208] The processor 1230 determines the position of the object or golf ball 1270 and, in step 1320, predicts the position of the golf ball 1270 in the next image to be captured. If the golf ball 1270 has not yet been struck by the golf club 1280, the processor 1230 assumes that the golf ball 1270 remains in the same position. Once struck, the processor 1230 determines the position of the golf ball 1270 based on the geometric shape of the golf club 1280 and the way the club impacted the golf ball 1270. While in flight, the position of the golf ball 1270 is determined from its past position, its velocity, and its spin.

[0209] After the processor 1230 determines the position of the golf ball 1270 in the next image, in step 1325, the processor 1230 instructs the imaging device 1210 via the controller 1220 to crop the image to include a region of interest (ROI), the cropped image including the golf ball and the surrounding background. In one embodiment, the golf ball 1270 occupies at least 10% of the pixels in the cropped image. In another embodiment, the golf ball 1270 occupies at least 50% of the pixels in the cropped image. In step 1330, the processor 1230 instructs the imaging device 1210 via the controller 1220 to set the frame rate to 2000-4000 fps in high-speed mode.

[0210] In step 1335, the imaging device 1210 captures a cropped image according to the image crop command sent to the controller of the imaging device 1210 in step 1325. The cropped image contains the region of interest within the field of view of the imaging device 1210 and is intended to track the estimated trajectory of the flying ball. The cropped image is captured by dealing with only a subset of the discrete pixel elements of the sensor chip 1212 for transfer to the processor 1230.

[0211] In step 1340, the cropped image is transferred to the processor 1230. The processor 1230 uses the cropped image to adjust the estimated trajectory of the ball in flight for the image to be subsequently cropped. In step 1345, the processor 1230 predicts the next position of the object (e.g., the golf ball 1270). The processor 1230 further uses the cropped image to calculate flight parameters, including spin, which are used by the golf simulator. In step 1350, the processor 1230 checks whether the golf ball 1270 is still within the field of view of the imaging device 1210. In step 1355, if the golf ball 1270 is still within the field of view of the imaging device 1210, the processor 1230 instructs the imaging device 1210 to capture the next image of the second image sequence based on the next position of the object predicted in step 1345. Step 1355 defines the re-cropping of the image so that a new subset of discrete pixel elements is addressed as a region of interest within the field of view of the imaging device 1210, moving along the path of movement of the object or golf ball 1270.

[0212] As long as the golf ball 1270 remains within the field of view of the imaging device 1210, the processor 1230 executes steps 1335, 1340, 1345, 1350, and 1355 multiple times to capture multiple images that define a sequence of cropped images in the high-speed state of the imaging device 1210.

[0213] When the processor 1230 determines in step 1350 that the object or ball 1270 has left the field of view of the imaging device 1210, the processor 1230 proceeds to step 1360. In 1360, the tracking sequence is considered complete. The sequence of cropped images is complete, the fast state ends, and the processor 1230 instructs the controller 1220 to enter the normal state, thereby setting the frame rate according to the normal state, and the imaging device no longer crops images. At this point, the processor 1230 ensures that the imaging device 1210 enters the normal state and captures images of the ground spot from which the golf player is expected to hit the golf ball 1270, up to elements in the image data that may indicate the presence of a take-back event.

[0214] In one embodiment, the processor 1230 may receive support inputs for use in detecting state change events and for estimating the trajectory of a ball in flight from tracking radar-applied Doppler radar tracking.

[0215] Figure 14 shows a state diagram 1400 of the system shown in Figure 12. The imaging device 1210 can take on various operating states, including normal state, take-back state, and flight state.

[0216] Under normal conditions, the system takes an idle step 1410 in which the imaging device 1210 observes the entire scene in the field of view at a normal frame rate. In step 1415, when the image is received, the processor 1230 investigates for a state change event that would form as a club backswing. Unless a state change event is detected in step 1420, the imaging device 1210 remains in normal conditions and passes the image showing the entire scene in the field of view at a normal frame rate to the processor 1230, which is searching for a state change event such as a club backswing.

[0217] If the processor 1230 detects a first state change event, such as a club backswing, in step 1420 the processor 1230 commands the imaging device 1210 to enter the backswing state. Based on the image used to detect the first state change event, the processor 1230 can also communicate the predicted location and size of a region of interest (ROI) that includes, for example, a golf ball or a golf club. In step 1425, the imaging device 1210 starts operating at a high frame rate and begins cropping the image. The cropped image is then passed to the processor 1230 looking for another state change event, such as a golf ball strike. Unless a state change event is detected in step 1430, the imaging device 1210 remains in the backswing state and passes the cropped image showing the region of interest to the processor 1230. In one embodiment, if a golf ball is not struck within a predetermined period and / or the golf swing is abandoned without hitting the golf ball, the processor 1230 can detect a state change event and return to the normal state.

[0218] When the processor 1230 detects a second state change event, including a golf ball strike, in step 1430, the processor 1230 commands the imaging device 1210 to enter the flight state. Based on the cropped image used to detect the second state change event, the processor 1230 can also communicate the predicted position and size of a region of interest (ROI) including objects of interest, such as the golf ball and golf club. It tracks the golf ball to determine its spin and tracks the golf club to provide information on the golf club's path before and after the golf ball strike, which can be used to improve the golfer's technique.

[0219] When a second state change event is detected, the imaging device 1210 enters a flight state in step 1435 and operates at a high frame rate. In one embodiment, the first object of interest is a golf ball 1270, and the second object of interest is a golf club 1280. The imaging device 1210 captures a first sequence of cropped images including the first region of interest which includes the golf ball 1270, and a second sequence of cropped images including the second region of interest which includes the golf club 1280. As a result, the processor receives a first sequence of cropped images showing the golf ball in flight and a second sequence of cropped images showing the golf club after impact.

[0220] The image cropping changes image by image within a sequence of cropped images, as the golf ball and / or golf club move within the field of view of the imaging device 1210.

[0221] When the processor 1230 determines in state 1440 that the movement path of the golf ball within the field of view of the imaging device 1210 is complete, the processor 1230 considers this a state change event, and the system returns to the normal state in step 1410 and waits for a state change event to be detected.

[0222] Figure 15 shows an image 1500 captured under normal conditions by an imaging device 1210 having a field of view (FOV) that defines a visually observable scenario. A golf ball 1270 is visible inside image 1500. When a state change event, such as the backswing of the golf club, is detected, the state of the imaging device 1210 is changed to a high-speed state, and the image 1500 captured for the first image sequence is changed to a cropped image 1510 for the second image sequence. It can be seen that the golf ball 1270 occupies most of the cropped image 1510. The estimated movement path 1520 of the golf ball 1270 within the field of view of the imaging device 1210 is shown. Two further cropped images 1525 are illustrated, but as mentioned above, the imaging device 1210 can acquire approximately 50 consecutive images before the golf ball 1270 leaves the field of view of the imaging device 1210. The second image sequence will show a golf ball in the center of the image, while the background will change slightly from one image to the next.

[0223] Those skilled in the art will understand that various modifications can be made to the disclosed embodiments without departing from the teachings of this disclosure, and that these are intended to be limited only by the claims appended herein. For example, it should be noted that the features of the various embodiments can be combined in any way that does not specifically exclude or logically contradict any particular teaching of this disclosure.

Claims

1. A system comprising an imaging device and a processor, The imaging device is configured to operate in a normal state, which has a field of view and is controlled by a first set of operating parameters for capturing a first sequence of images, and a high-speed state, which is controlled by a second set of operating parameters for capturing a second sequence of images. The aforementioned processor, Based on the data extracted from the first sequence of the aforementioned images or the second sequence of the aforementioned images, the movement path of the object is estimated. The imaging device is controlled in the high-speed state and configured to crop the image in the second sequence of the image which includes the region of interest which includes the object. The processor is a system that controls the imaging device such that the second sequence of images includes a region of interest within the field of view and tracks the estimated movement path of the object.

2. The aforementioned processor, By analyzing the first sequence of the aforementioned images, a striking event involving the object is detected. The system according to claim 1, further configured to change the state of the imaging device by applying a second set of operating parameters for operating the imaging device in the high-speed state when the impact event is detected.

3. The system according to claim 2, wherein the object is a ball in flight, and the processor is further configured to estimate the trajectory of the ball in flight based on the impact event.

4. The system according to claim 3, wherein the processor is further configured to apply data extracted from the cropped image to adjust the estimated trajectory of the flying ball for the subsequently cropped image in the second sequence of the image.

5. The system according to claim 2, wherein the object is a golf club, and the processor is further configured to estimate the movement path of the golf club based on the impact event.

6. The system according to claim 5, wherein the processor is further configured to apply data extracted from the cropped image to adjust the estimated movement path of the golf club for the image subsequently cropped in the second sequence of images.

7. The object in question is a golf ball, The processor in the high-speed state is The system is further configured to detect a striking event involving a golf club striking the golf ball by analyzing the second sequence of the aforementioned images. The aforementioned processor, The imaging device is controlled in the high-speed state to crop the image in the second sequence of the image, which includes a first region of interest including the estimated movement path of the golf club before the impact event. The system according to claim 1, further configured to control the imaging device in the high-speed state to crop the images in the second sequence of images including a second region of interest including the estimated trajectory of the golf ball in flight after the impact event.

8. The system according to claim 2, wherein the imaging device is fixedly positioned above the location where the impact event is expected, and the field of view of the imaging device includes at least a portion of the impact event and its movement path.

9. The system according to claim 1, wherein the imaging device has a camera sensor chip having a plurality of discrete pixel elements read by a controller, and the controller in high-speed state is configured to deal with a subset of the discrete pixel elements in order to read a cropped image containing only the region of interest within the field of view.

10. The aforementioned processor, When the region of interest, including the object, leaves the field of view, The system according to claim 1, further configured to change the state of the imaging device by applying the first set of operating parameters for operating the imaging device in the normal state until a further impact event is detected.

11. To provide an imaging device configured to operate in a normal state, which has a field of view and is controlled by a first set of operating parameters for capturing a first sequence of images, and in a high-speed state, which is controlled by a second set of operating parameters for capturing a second sequence of images. Estimating the movement path of an object based on data extracted from the first sequence of the aforementioned images or the second sequence of the aforementioned images, This includes controlling the imaging device in the high-speed state to crop the image in the second sequence of the image that includes the region of interest including the object, A method in which the processor controls the imaging device such that the second sequence of the images includes the region of interest within the field of view and tracks the estimated movement path of the object.

12. By analyzing the first sequence of the aforementioned images, a striking event involving the object can be detected. The method according to claim 11, further comprising changing the state of the imaging device by applying a second set of operating parameters for operating the imaging device in the high-speed state when the impact event is detected.

13. The object is a ball in flight, and the method is The method according to claim 12, further comprising estimating the trajectory of the ball in flight based on the impact event.

14. The method according to claim 13, further comprising applying data extracted from the cropped image to adjust the estimated trajectory of the flying ball for the image subsequently cropped in the second sequence of the image.

15. The object is a golf club, and the method is The method according to claim 12, further comprising estimating the movement path of the golf club based on the impact event.

16. The method according to claim 15, further comprising applying data extracted from the cropped image to adjust the estimated movement path of the golf club for the image subsequently cropped in the second sequence of the image.

17. The object in question is a golf ball, The aforementioned method, By analyzing the second sequence of the aforementioned image, it is possible to detect a striking event involving a golf club striking the golf ball, Controlling the imaging device in the high-speed state to crop the image in the second sequence of the image, which includes a first region of interest including the estimated movement path of the golf club before the impact event, The method according to claim 11, further comprising controlling the imaging device in the high-speed state to crop the image in the second sequence of the image which includes a second region of interest which includes the estimated trajectory of the golf ball in flight after the impact event.

18. The method according to claim 12, wherein the imaging device is fixedly positioned above a location where a striking event involving the object is expected, and the field of view of the imaging device includes at least a portion of the striking event and its movement path.

19. The method according to claim 11, wherein the imaging device has a camera sensor chip having a plurality of discrete pixel elements read by a controller, and the controller in high-speed state is configured to deal with a subset of the discrete pixel elements in order to read a cropped image containing only the region of interest within the field of view.

20. To detect when the region of interest, including the object, leaves the field of view, The method according to claim 11, further comprising changing the state of the imaging device by applying the first set of operating parameters for operating the imaging device in the normal state until a further impact event is detected.

21. The aforementioned processor, The system according to claim 7, further configured to control the imaging device in the high-speed state to crop the image in the second sequence of the image which includes the first region of interest which includes the estimated movement path of the golf club after the impact event.

22. The method according to claim 17, further comprising controlling the imaging device in the high-speed state to crop the image in the second sequence of the image including the first region of interest including the estimated movement path of the golf club after the impact event.

Images