System and method for determining the position where an object crosses a target surface

Abstract

To provide a system and method for determining a position where an object crosses a target plane.SOLUTION: The system and method determines a position where an object moving along a trajectory crosses a target plane, by: obtaining a plurality of images of an object moving along a trajectory in an environment, the plurality of images capturing the object at different positions along the trajectory; and obtaining a plurality of images of an object moving along a trajectory in an environment, the plurality of images capturing the object at different positions along the trajectory, the plurality of images being obtained from two or more cameras positioned at different positions in the environment, each of the two or more cameras having a field of view (FOV) that includes the trajectory of the moving object and a target plane, the target plane being a plane that the moving object crosses during movement along the trajectory, each of the two or more cameras obtaining the images at a different time.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to object tracking, and more particularly to determining multiple positions of an object along a trajectory.

[0002] Image analysis can be useful in many technologies and fields to identify whether a tennis serve was on line, whether a foot was inbound, whether the ball touched the ground, whether illegal contact occurred, whether a baseball strike went over home plate, etc. An example of such image analysis can be used in the post-match review process that is now common in various sports.

[0003] In baseball, the strike zone is the area extending from the midpoint between the batter's shoulder and the top of his uniform pants to just below the kneecap, extending to home base, when the batter is ready to swing at the pitched ball. The strike zone is also defined as the volume of space through which a pitched ball must pass in order to be called a strike, even if the batter does not swing. The strike zone is used to determine whether a pitch is a strike or not, and is an important element in the game of baseball.

[0004] Many object tracking methods use two cameras to detect a baseball crossing the strike zone, but these methods are complex and generally require both cameras to find the object at the same time, i.e., hard synchronization. Therefore, it is necessary to provide a method to overcome the above limitations. Summary of the Invention

[0005] Embodiments of the present disclosure are directed to a method for determining a location where an object moving along a trajectory crosses a target plane. In some embodiments, the method includes acquiring multiple images of an object moving along a trajectory in an environment. In some embodiments, the multiple images capture the object at different locations along the trajectory. In some embodiments, the multiple images are acquired from two or more cameras positioned at various locations in the environment. In some embodiments, each of the two or more cameras acquires images at different times. In some embodiments, each of the two or more cameras has a field of view (FOV) that includes the trajectory of the moving object and the target plane. In some embodiments, the target plane is a plane that the moving object crosses while moving along the trajectory.

[0006] In some embodiments, each of the two or more cameras is in communication with at least one processor. In some embodiments, the method includes determining, by the at least one processor, a plurality of different positions of the object in respective trajectories in respective two-dimensional camera planes for each of the two or more cameras. In some embodiments, the method includes determining, by the at least one processor, a plurality of different positions of the object in respective trajectories for each of the two or more cameras in a two-dimensional common plane using the plurality of different positions of the object in respective trajectories in the two-dimensional camera planes for each of the two or more cameras. In some embodiments, the method includes determining, by the at least one processor, a position at which an object moving along the trajectory crosses a target surface using the plurality of different positions of the object in respective trajectories in the two-dimensional common plane for each of the two or more cameras. In some embodiments, the method may further include determining, by the at least one processor, a time at which the object moving along the trajectory crosses the target surface using the position at which the moving object crosses the target surface. In some embodiments, the object is a baseball pitched from a launch location toward a strike zone, the target surface is the strike zone, and the determined time is the time the baseball crosses the strike zone.

[0007] In some embodiments, the method may also include calibrating each of the two or more cameras in the environment. In some embodiments, the method may include obtaining a first homography matrix for each of the two or more cameras from the calibration. In some embodiments, the first homography matrix describes a projection from three-dimensional coordinates to two-dimensional coordinates, for example, if a three-dimensional calibration pattern is used. In some embodiments, after calibrating the cameras, the method may include obtaining a plurality of images capturing the object at various positions moving along a trajectory. In some embodiments, the method may include using the first homography matrix for each of the two or more cameras to determine coordinates of a plurality of different positions of the object in the respective trajectories in the two-dimensional camera plane for each of the two or more cameras.

[0008] In some embodiments, the method may further include transforming coordinates of the plurality of different positions of the object in the respective trajectories in the two-dimensional camera plane for each of the two or more cameras to the two-dimensional common plane using a second homography matrix for each of the two or more cameras to determine a plurality of different positions of the object in the respective trajectories in the two-dimensional common plane for each of the two or more cameras. In some embodiments, the second homography matrix describes a projection from the two-dimensional camera coordinates onto the two-dimensional common plane.

[0009] In some embodiments, the method may also include determining a position where the respective trajectories of each of the two or more cameras intersect in a two-dimensional commonality to obtain a position where an object moving along the trajectory crosses the target surface. In some embodiments, the method may also include back-projecting the position where the object moving along the trajectory crosses the target surface onto a two-dimensional camera plane for each of the two or more cameras. In some embodiments, the method may include back-projecting the position where the object moving along the trajectory crosses the target surface onto a two-dimensional camera plane for each of the two or more cameras using a second homography matrix. In some embodiments, the method may include determining a time where the object moving along the trajectory crosses the target surface using an intersection ratio between a plurality of different positions of the object and the back-projected position and corresponding timestamps of the plurality of different positions of the object and the back-projected position in the selected two-dimensional camera plane. In some embodiments, the time where the object moving along the trajectory crosses the target surface is a local camera time for each of the two or more cameras.

[0010] Embodiments of the present disclosure are directed to a system including one or more processors and a non-transitory computer-readable medium having instructions thereon that, in response to execution by the one or more processors, control execution of operations of the one or more processors to determine a location where an object moving along a trajectory crosses the target plane by: acquiring a plurality of images of an object moving along a trajectory in an environment, the plurality of images capturing the object at various positions along the trajectory; and acquiring a plurality of images of an object moving along a trajectory in an environment, the plurality of images capturing the object at various positions along the trajectory, the plurality of images being acquired from two or more cameras positioned at various positions within the environment, each of the two or more cameras having a field of view (FOV) that includes the trajectory of the moving object and a target plane, the target plane being a plane that the moving object crosses while moving along the trajectory, and each of the two or more cameras acquiring images at a different time.

[0011] The operation of the one or more processors is also controlled to determine a plurality of different positions of the object in respective trajectories in respective two-dimensional camera planes for each of the two or more cameras; determine a plurality of different positions of the object in respective trajectories for each of the two or more cameras in a two-dimensional common plane using the plurality of different positions of the object in respective trajectories in the two-dimensional camera planes for each of the two or more cameras; determine a position at which the object moving along the trajectory crosses a target surface using the plurality of different positions of the object in respective trajectories in the two-dimensional common plane for each of the two or more cameras; and determine a time at which the object moving along the trajectory crosses the target surface using the position at which the moving object crosses the target surface.

[0012] Further features, as well as the structure and operation of various embodiments, are described in detail below with reference to the accompanying drawings, where like reference numbers indicate identical or functionally similar elements. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a schematic diagram of an embodiment of a system for determining the position and corresponding time at which an object moving along a trajectory crosses a target surface; [Figure 2A] FIG. 2 is a schematic diagram illustrating the position of a camera device relative to the plane of the trajectory of a moving object. [Figure 2B] FIG. 2 is a schematic diagram illustrating the position of a camera device relative to the plane of the trajectory of a moving object. [Figure 3] FIG. 2 is a schematic diagram of one embodiment of a camera employed in the system of FIG. 1. [Figure 4] FIG. 1 is a schematic diagram of a camera plane and a two-dimensional virtual common plane for determining a position within the strike zone and strike time for a curved baseball trajectory in accordance with one embodiment of the present disclosure. [Figure 5] FIG. 1 is a flow diagram of an embodiment of a method for determining the time at which an object moving along a trajectory crosses a target surface. [Figure 6]FIG. 10 is a flow diagram of another embodiment of a method for determining the time at which an object moving along a trajectory crosses a target surface. [Figure 7] FIG. 1 illustrates a block diagram of an exemplary computing system for use with embodiments of the present disclosure.

[0014] Further features, as well as the structure and operation of various embodiments, are described in detail below with reference to the accompanying drawings, where like reference numbers indicate identical or functionally similar elements. DETAILED DESCRIPTION OF THE INVENTION

[0015] Embodiments of the present disclosure are directed to methods and systems for determining the time at which an object moving along a trajectory crosses a target surface. In the case of baseball, the methods and systems of the present disclosure can be implemented to determine the time at which a baseball crosses the strike zone. While the examples provided in this disclosure relate to a baseball environment, the present disclosure can also be implemented in other sports, including, but not limited to, football, cricket, tennis, basketball, and golf. For example, in cricket, the methods and systems described herein can be implemented in the beehive zone. There are also planes for football (goal posts) and soccer (goal net posts). In tennis and basketball, the ground plane can be considered a plane. Additionally, the methods and systems of the present disclosure can also be extended to any object, in addition to a ball, that crosses a plane (e.g., a plane above the ground). This would include golf clubs, baseball bats, cricket bats, and parts of a player's body. For objects other than a ball, a determination can be made, for example, in a race, when a player's legs cross the surface above the ground of the goal line, or in American football, when a player extends the football beyond the goal line.

[0016] In baseball, the disclosed methods and systems obtain accurate measurements of a baseball in and near the strike zone. From accurate measurements of the ball in the strike zone, an accurate measurement of strike time can be determined. Strike time can be defined as the time it takes for a pitched ball to cross the front of the strike zone. The strike zone and strike time can provide accurate measurements of the ball's trajectory and break count.

[0017] In some embodiments, the disclosed methods and systems are incorporated into pitch monitoring devices for evaluating player efficiency and performance. The devices may be used during actual games or for training. For example, in baseball, pitchers' performance is typically judged by traditional baseball metrics, such as earned runs, wins, and / or saves. Pitches are typically judged based on whether the ball is in the strike zone, whether the batter can hit the pitch, and / or the result of the pitch (e.g., strike, ball, foul ball, hit, etc.). It is also common to categorize pitches based on their trajectory and / or velocity, including, for example, fastball, four-seam fastball, two-seam fastball, cutter (or cut fastball), forkball, splitter, sinker, curveball, slider, changeup, etc. Pitch trajectory may be decomposed into components, such as launch, breaking point, final position, and total break. The trajectory metric may be a function that determines the quality of the pitch based on the pitch's launch, breaking point, final position, and total break.

[0018] In some implementations, the trajectory metric may be a mathematical combination (e.g., a linear combination) of these components. In some embodiments, pitch metrics may be independent of the batter facing the pitcher at the time of the pitch. For example, a standard strike zone may be used that is batter-independent (e.g., the strike zone is independent of the batter's height, batting stance, etc.). In some embodiments, the strike zone location of a baseball may be used to evaluate a pitch, because pitches in certain locations are more difficult for batters to hit than pitches in other locations.

[0019] In some embodiments, pitches may be characterized based on the location facing the pitcher when pitched and the particular batter. For example, the strike zone location of the trajectory may be used to define a desirable location for a particular batter, where the batter is more likely to get an out rather than a hit. This may take into account the batter's size, posture, and other factors. This may allow coaches and pitchers to determine the quality of pitches to throw to a particular batter, taking into account the batter's tendencies, preferences, weaknesses, etc.

[0020] In some embodiments, the methods and systems of the present disclosure include two or more cameras. In some embodiments, the two or more cameras may be arbitrarily positioned in the environment, for example, on the field, provided that the strike zone is within the field of view (FOV) of the cameras. In some embodiments, the cameras are positioned so that the cameras are spaced horizontally (i.e., one camera is above another) rather than vertically.

[0021] The disclosed methods and systems can track a baseball or other object moving in a straight line and / or a curved line, allowing for the use of a variety of trajectories and camera settings. Advantageously, the disclosed methods and systems do not require hard synchronization methods to accurately identify strike zone locations and strike times. Hard synchronization corresponds to when two or more devices are synchronized by a timer (using a common timer) and capture camera frames synchronously (sharing a hardwired external camera synchronization signal). The frames are captured synchronously, and coincidences between different camera frames are noted by a common timestamp. The disclosed methods and systems do not require the cameras to be synchronized. Instead, the cameras can be asynchronous cameras, and the disclosed methods and systems use event-based synchronization. Thus, the disclosed methods and systems can be applied without any of the two or more cameras having knowledge of what any of the other cameras are doing. In some embodiments, before using the cameras to track a moving object, calibration can be performed to calibrate each of the cameras relative to the strike zone plane, for example, using a standard two-dimensional calibration pattern on a small checkerboard or a three-dimensional calibration pattern. Furthermore, unlike conventional stereo imaging, the objects do not need to share a timestamp, and therefore the methods and systems of the present disclosure may be applied to systems that lack a common time reference.

[0022] The systems and methods of the present disclosure use event-based time synchronization because time synchronization (event-based) is recovered from the detection of moving objects and virtual plane crossings. Specifically, because the cameras used in the systems and methods of the present disclosure initially do not have information about other cameras, including their time synchronization (such as time delays) relative to each other, events such as a moving object crossing the virtual plane described herein can be used as a common event or reference for determining the time reference of each camera, which is thus useful for camera synchronization. Because the systems and methods of the present disclosure include synchronization of asynchronous cameras, i.e., event-based synchronization, the systems and methods of the present disclosure can be extended to generally synchronize two or more asynchronous cameras. Therefore, the systems and methods of the present disclosure can be used to synchronize unsynchronized cameras for other applications requiring synchronization. In the systems and methods of the present disclosure, a moving object crossing a virtual plane, i.e., a two-dimensional common plane, is used to synchronize two or more asynchronous cameras. Therefore, the disclosed methods and systems are not limited to determining strike times.

[0023] FIG. 1 is a schematic diagram of an object tracking system 10 employing two or more camera devices 100 for determining the time at which an object moving along a trajectory crosses a target surface. The embodiment of system 10 shown in FIG. 1 uses two camera devices 101, 102. However, as noted above, more than two camera devices 100 may also be employed. Camera device 100 may communicate wired or wirelessly with computing device 200 to perform object tracking and target surface detection operations. In some embodiments, camera device 100 and computing device 200 are implemented in a single monitoring device. In other embodiments, one or more camera devices 100 are implemented in a single monitoring device that includes computing device 200, and one or more other camera devices 100 are conventional cameras. For example, in one embodiment, camera device 101 and computing device 200 are implemented in a single monitoring device, and camera 102 is the only camera and communicates with computing device 200. Thus, images captured by camera 102 can be processed by computing device 200. In other embodiments, all camera devices 100 are regular cameras that communicate with computing device 200 .

[0024] Camera devices 101 and 102 capture multiple images of an object moving along a trajectory. In some embodiments, the system is disposed in an environment such as a baseball field. FIG. 1 illustrates, as an example, a baseball field environment 12 in which a pitcher (not shown) pitches a baseball 103 from a release point 104 toward home plate 106 along a trajectory 105. A strike zone 107 is an area on home plate 106 from the midpoint between the batter's shoulder and the top of his uniform pants to just below his kneecap. Strike zone 107 is a target plane 107. Camera devices 101 and 102 are disposed in environment 12 such that each has an image capture FOV that includes the trajectory 105 of the moving object, in this example, baseball 103, and target plane 107. As shown, camera device 101 has FOV 108, and camera device 102 has FOV 109. As shown, FOVs 108 and 109 both include strike zone 107 and baseball trajectory 105. In some embodiments, FOVs 108 and 109 may be substantially the same FOV. In some embodiments, the FOV of each of the cameras may also include baseball launch location or release point 104. Camera devices 101 and 102 each acquire multiple images capturing baseball 103 at various positions along trajectory 105. In some embodiments, each of camera devices 101 and 102 acquires images at different times. FIG. 1 illustrates a plurality of images captured at different times t A , t B , t C , t D , and t E Five ball positions A, B, C, D, and E are shown along trajectory 105 at the ball release point 104. Positions A, B, C, and D are between the release point 104 and the strike zone 107, and position E is where the baseball 103 crosses the strike zone 107.

[0025] In some embodiments, camera devices 100 are positioned relative to the trajectory of the moving object such that none of the camera devices are oriented coincident with the plane of the trajectory of the moving object. For example, as shown in FIG. 2A , camera device 101 and camera device 102 are not oriented coincident with the plane 115 of trajectory 105. In some embodiments, camera devices 100 are positioned relative to the trajectory of the moving object such that some of the camera devices are oriented coincident with the plane of the trajectory of the moving object and some of the camera devices are not oriented coincident with the plane of the trajectory of the moving object. For example, as shown in FIG. 2B , camera device 101 is not oriented coincident with the plane 115 of trajectory 105, but camera device 102 is oriented coincident with the plane 115 of trajectory 105.

[0026] Details of each camera device 100 are shown in Figure 3. In some embodiments of the present disclosure, camera device 100 may include any device, system, component, or collection of components configured to capture images and / or video. In some embodiments, the camera of camera device 100 may include an image capture unit 130, which may include a lens 111, an image sensor 112, a digital signal processor or controller 110, which may include one or more hardware microprocessors, a communication unit 120 connected to and controlled by controller 110, image capture unit 130 for capturing images and / or video, a calibration unit 140 for controlling camera calibration operations, memory 150, a tracking unit 160, and an image processing unit 170.

[0027] In some embodiments, the image capture unit 130 may include optical elements, such as additional filters. The image sensor 112, through which an image may be recorded, may include any device that converts an image represented by incident light into an electronic signal. The image sensor 112 may include multiple pixel elements that may be arranged as a pixel array (e.g., a grid of pixel elements) for acquiring an image. For example, the image sensor 112 may include a charge-coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS) image sensor. The pixel array may include a two-dimensional array having any aspect ratio. The image sensor 112 may be optically aligned with various optical elements, such as the lens 103, that focus light onto the pixel array. Any number of pixels may be included, such as hundreds or thousands of megapixels.

[0028] In one embodiment, image capture unit 130 captures images under the control of controller 110, and the captured images are stored in memory storage unit 150 under the control of controller 110. Additional data and information for the operation of camera 100 may be stored in memory 150. Images captured by image capture unit 130 are then stored in memory 150. Each captured image has a unique identification element, which may correspond to the time the image was captured. Additionally, other types of tracking information detected from at least one image may include pixel locations of the moving object being tracked. The pixel locations may be stored in memory 150. Hereinafter, the storage area of ​​memory 150 where images are stored may also be referred to as image storage unit 152, and the storage area of ​​memory 150 where “tracking information” related to target object tracking is stored will be referred to as tracking information storage unit 155. An image processing unit 170 is further provided to perform types of digital signal processing on raw camera image data, such as filtering, noise reduction, image sharpening, and the like.

[0029] 3, camera device 100 may include a communications unit 120 that enables the camera to receive remote control signals for performing camera operations in conjunction with object tracking and strike zone detection information, and includes at least one module for performing wired and / or wireless communications with a remote control processor or remote computing device 200. Under the control of controller 110, communications unit 120 may receive control signals from, and transfer captured images to, the remote control processor or computing device 200 through at least such modules.

[0030] Whether controlled remotely or locally using controller 110, each camera device 100 may operate at a particular frame rate or be capable of capturing a particular number of images at a given time. Camera 100 may operate at a frame rate of about 60 frames per second (fps) or greater, for example, about 100 to about 300 frames per second (fps). In some embodiments, a smaller subset of the available pixels in the pixel array may be utilized to enable the camera to operate at a higher frame rate.

[0031] Tracking unit 160 operates to detect moving target objects being tracked and to determine the location and time at which the target object crosses a target surface. In one embodiment, any number of different triggers may be used to cause camera device 100 to capture one or more images of a moving object. By way of non-limiting example, camera device 100 may be triggered upon receipt of a timed activation signal, or when a moving object is detected, recognized, or estimated to be within the camera's field of view, or when a moving object first begins or changes its flight (e.g., when a baseball is pitched, a baseball is hit, a golf ball is struck, a tennis ball is served, a cricket ball is launched, etc.), or when a moving object is detected at the top row of pixels in a pixel array, etc.

[0032] In one embodiment, a remote control processor or computing device 200 may control the operation of the entire camera device 100, which may be physically separated, to perform object tracking and detection according to embodiments of the present disclosure. The computing device 200 may configure the controller 110 to control the image capture unit 130 and acquire images according to control signals 175 received through the communication unit 120. The controller 110 may then store the acquired images on the device and / or transmit the stored images and associated image identification, tracking, and strike zone detection information to the computing device 200 for further processing. Further processing may include application of one or more algorithms to receive raw camera image data in a series of image frames. For example, according to embodiments herein, exemplary camera measurements of a moving object may include, but are not limited to, identifying one or more of the object radius, object center, velocity, elevation angle, and azimuth angle in each of the images, calculated, for example, based on the object radius, object center, and pre-measured camera alignment values.

[0033] FIG. 4 is a schematic diagram of a two-dimensional camera plane and a two-dimensional virtual common plane for determining strike zone position and strike time for a curved ball trajectory in one embodiment of the present disclosure. While the examples disclosed herein relate to an object moving on a curved trajectory, the concepts and equations disclosed may be implemented for a straight or linear trajectory as well. In this embodiment, there are two cameras 101 and 102, as described above in FIGS. 1-3. Whether synchronized or asynchronous cameras are used, a calibration must be performed to obtain a first homography matrix for each camera 101 and 102, i.e., a first homography matrix for camera 101 and a first homography matrix for camera 102. The first homography matrix for each camera 101 and 102 describes, for example, a projection from the three-dimensional coordinates of a three-dimensional calibration pattern to the two-dimensional coordinates of the camera plane. The homography matrix (H) obtained from the calibration is 3D) is a 4x3 matrix that describes the projection from 3D world coordinates to 2D camera coordinates, also known as the camera plane or 2D camera plane. It should be understood that the first homography matrix can also describe the projection from 2D coordinates to other 2D coordinates if a 2D calibration pattern is used instead of a 3D calibration pattern. Any camera calibration method may be used for this purpose, as long as the first homography matrix is ​​obtained from calibration.

[0034] An example of a 3D-to-2D homography matrix that may be implemented in some embodiments is described in USSN 18 / 595,592, the entire disclosure of which is incorporated herein by reference.

[0035] Another example of a 3D to 2D homography matrix is Projection constraints from the 3D world onto a 2D plane:

number

number

number

number

number

number

number

number

number

number

[0036] Once the cameras are calibrated, they are used to track a moving object. Each camera captures several images at different times and recovers two-dimensional camera coordinates within the two-dimensional camera plane. In the example shown in FIG. 4 , each camera 101 and 102 acquires four images of baseball 103 moving along trajectory 105. Each image captures baseball 103 at a different one of positions A, B, C, and D along trajectory 105. Tracking unit 160 determines, for each of the two cameras, various positions of baseball 103 along each trajectory within the respective two-dimensional camera plane. In some embodiments, tracking unit 160 uses a first homography matrix for each of the two or more cameras to determine the coordinates of multiple different positions of baseball A, B, C, and D along each trajectory within the two-dimensional camera plane for each of the two or more cameras. 4, tracking unit 160 determines baseball positions a1-b1-c1-d1 in trajectory 303 in two-dimensional camera plane 301 of camera 101. Additionally, tracking unit 160 determines baseball positions a2-b2-c2-d2 in trajectory 304 in two-dimensional camera plane 302 of camera 102.

[0037] Tracking unit 160 then uses the multiple different positions of baseball 103 in trajectories 303 and 304, respectively, in two-dimensional camera planes 301 and 302 for each of cameras 101 and 102 to determine multiple different positions of baseball 103 in trajectories 303 and 304, respectively, for each of cameras 101 and 102 in two-dimensional common plane 305. In some embodiments, tracking unit 160 transforms pixel coordinates of the multiple different positions of baseball 103 in trajectories 303 and 304, respectively, in two-dimensional camera planes 301 and 302 for each of cameras 101 and 102, to two-dimensional common plane 305. In some embodiments, tracking unit 160 transforms pixel coordinates of the multiple different positions of baseball 103 in trajectories 303 and 304, respectively, in two-dimensional camera planes 301 and 302 for each of cameras 101 and 102 ... 2D(n) ), the two-dimensional camera coordinates are transformed to a common two-dimensional plane 305 associated with the target plane, the strike zone 107. The common two-dimensional plane 305 can be a virtual plane, a real plane, a horizontally oriented plane, a vertically oriented plane, an angled or tilted plane, or a grid of planes. A second homography matrix (H 2D ) is a 3x3 matrix that describes the projection from the 2D camera coordinates onto the 2D virtual plane. (H 2D An example of a 3x3 matrix is ​​given in a technical report by Daniel Bardsley and Bai Li entitled "3D Reconstruction Using the Direct Linear Transform with a Gabor Wavelet-based Correspondence Measure."

[0038] 4, tracking unit 160 determines position A1-B1-C1-D1 of baseball 103 within trajectory 306 within common plane 305. Trajectory 306 and position A1-B1-C1-D1 are obtained by projecting trajectory 303 from two-dimensional camera plane 301 of camera 101 onto two-dimensional common plane 305. Similarly, tracking unit 160 determines position A2-B2-C2-D2 of baseball 103 within trajectory 307 within two-dimensional common plane 305. Trajectory 307 and position A2-B2-C2-D2 are obtained by projecting trajectory 304 from two-dimensional camera plane 302 of camera 102 onto two-dimensional common plane 305.

[0039] In some embodiments, tracking unit 160 uses a second homography matrix for each of cameras 101 and 102 to determine multiple distinct pixel locations of the baseball within respective trajectories 306 and 307 within two-dimensional common plane 305 for each of cameras 101 and 102. In some embodiments, the second homography matrix describes a projection from two-dimensional camera coordinates onto the two-dimensional common plane. Baseball trajectory 308 in three-dimensional space represents actual baseball trajectory 105 and is shown by circles representing positions A, B, C, and D.

[0040] After tracking unit 160 transforms the two-dimensional baseball positions a1-b1-c1-d1 and a2-b2-c2-d2 from the respective two-dimensional camera planes 301 and 302 to the two-dimensional common virtual plane 305, it fits lines to the respective positions A1-B1-C1-D1 and A2-B2-C2-D2 to form respective trajectories 306 and 307 for each camera 101 and 102. As can be seen in FIG. 4 , the two trajectories 306 and 307 on the two-dimensional common plane intersect at position E, which indicates the location where the baseball crosses strike zone 107 along baseball trajectory 308. Using this information, strike zone position E is obtained for baseball trajectory 308. Since the trajectory 306 and the trajectory 307 intersect at position E, the tracking unit 160 determines that the position E1 of the trajectory 306 and the position E2 of the trajectory 307 are equal to the position E of the baseball's trajectory 308. This strike zone position E on the two-dimensional common plane 305 is then transformed back to the two-dimensional camera planes 301 and 302 of the cameras 101 and 102, respectively. In some embodiments, the tracking unit 160 performs a two-dimensional inter-projection using a second homography matrix. Next, the projected points E1 and E2 (equal to E), i.e., e1 and e2, onto each of the respective camera planes 301 and 302 are determined. Once the projected points E (e1 and e2) are obtained using the intersection ratio on either the camera plane 301 or 302, the strike time t is determined because the intersection ratio on the two-dimensional virtual common plane 305 is the same as the intersection ratio on the two-dimensional camera plane 301 and the two-dimensional camera plane 302. EIn some embodiments, tracking unit 160 may use an intersection ratio between multiple different positions a1-b1-c1-d1 on two-dimensional camera plane 301 and back-projected position E1, and corresponding timestamps of the multiple different positions and back-projected positions on two-dimensional camera plane 301, to determine the strike zone position, i.e., the time when the baseball moving along trajectory 105 crosses target plane 107. Alternatively, tracking unit 160 may use an intersection ratio between multiple different positions or a2-b2-c2-d2 on two-dimensional camera plane 302 and back-projected position E2, and corresponding timestamps of the multiple different positions and back-projected positions on two-dimensional camera plane 302, to determine the strike zone position, or the time when the baseball moving along trajectory 105 crosses target plane 107.

[0041] As mentioned above, in some embodiments, a cross ratio is applied to determine strike time. After obtaining the strike zone position (e.g., point E) of the baseball ball trajectory, the strike zone position is projected onto each camera plane, i.e., 2D camera plane 301 and 2D camera plane 302, to obtain the corresponding pixel position (e1 and e2, respectively). The obtained positions are then used in the cross ratio equation to determine the strike time (t E Each of the points e1 and e2 has been determined by backprojection from the two-dimensional common plane 305 (point "E") onto each of the two-dimensional camera planes 301, 302 using the second homography matrix.

[0042] In some embodiments, the cross ratios described below may then be used to determine the times at the respective points e1 and e2 (i.e., t1 and te2). Because point "E" is a common event, the local camera times (i.e., te1 and te2) of cameras 101 and 102 at the respective points e1 and e2 are considered to be the time references of cameras 101 and 102, respectively. Therefore, the systems and methods described herein are referred to as event-based synchronization systems and methods.

[0043] For an object with a locus of 9p,

number

number

number

number

[0044] FIG. 5 is a flow diagram of an embodiment of a method for determining a time at which an object moving along a trajectory crosses a target plane. In some embodiments, the method includes step S10 of acquiring multiple images of an object moving along a trajectory in an environment. In some embodiments, the multiple images capture the object at different positions along the trajectory. In some embodiments, as described above, the multiple images are acquired from two or more cameras positioned at various positions in the environment. In some embodiments, each of the two or more cameras acquires images at different times, i.e., as asynchronous cameras. In some embodiments, each of the two or more cameras has a field of view (FOV) that includes the trajectory of the moving object and the target plane. In some embodiments, the FOV of each of the two or more cameras further includes the launch position of the moving object. In some embodiments, the target plane is a plane that the moving object crosses while moving along the trajectory. In some embodiments, the target plane is a virtual plane. In some embodiments, the target plane is a common virtual plane.

[0045] In some embodiments, the method includes step S12 of determining a plurality of different positions of the object in each trajectory in a respective two-dimensional camera plane for each of the two or more cameras. In some embodiments, the method includes step S14 of determining a plurality of different positions of the object in each trajectory in a respective two-dimensional camera plane for each of the two or more cameras in a two-dimensional common plane using the plurality of different positions of the object in each trajectory in the two-dimensional camera plane for each of the two or more cameras. In some embodiments, the method includes step S16 of determining a position at which the object moving along the trajectory intersects a target plane using the plurality of different positions of the object in each trajectory in the two-dimensional common plane for each of the two or more cameras. In some embodiments, the method includes step S18 of determining a time at which the object moving along the trajectory intersects the target plane using the position at which the moving object intersects the target plane. In some embodiments, the object is a baseball pitched from a launch position toward a strike zone, the target plane is the strike zone, and the determined time is a time at which the baseball intersects the strike zone.

[0046] 6 is a flow diagram of another embodiment of a method for determining the time at which an object moving along a trajectory crosses a target surface. In some embodiments, the method includes step S20 of calibrating each of two or more cameras in an environment. In some embodiments, the method includes step S24 of obtaining from the calibration a first homography matrix for each of the two or more cameras that describes a projection from three-dimensional coordinates (the three-dimensional calibration pattern) to two-dimensional coordinates (the two-dimensional camera plane). In some embodiments, after step S24, the method includes step S26 of obtaining a plurality of images capturing the object at various positions moving along the trajectory. In some embodiments, the method includes step S28 of using the first homography matrix for each of the two or more cameras to determine coordinates of a plurality of different positions of the object in the respective trajectories in the two-dimensional camera plane for each of the two or more cameras.

[0047] In some embodiments, the method includes step S30: transforming coordinates of a plurality of different positions of the object in each trajectory in the two-dimensional camera plane for each of the two or more cameras to the two-dimensional common plane using a second homography matrix for each of the two or more cameras. In step S30, the plurality of different positions of the object in each trajectory in the two-dimensional common plane for each of the two or more cameras is determined. In some embodiments, the second homography matrix describes a projection from the two-dimensional camera coordinates onto the two-dimensional common plane.

[0048] In some embodiments, the method includes step S32 of determining a position where the respective trajectories of each of the two or more cameras intersect in a two-dimensional common plane to obtain a position where an object moving along the trajectory crosses the target surface. In some embodiments, the method includes step S34 of back-projecting the position where the object moving along the trajectory crosses the target surface onto a two-dimensional camera plane for each of the two or more cameras. In some embodiments, step S34 includes back-projecting the position where the object moving along the trajectory crosses the target surface onto a two-dimensional camera plane for each of the two or more cameras using a second homography matrix. In some embodiments, the method may include step S36 of determining a time where the object moving along the trajectory crosses the target surface using an intersection ratio between a plurality of different positions of the object and the back-projected position and corresponding timestamps of the plurality of different positions of the object and the back-projected position in the selected two-dimensional camera plane. In some embodiments, once the above parameters are recovered, it may be possible to determine the ball's flight parameters using, for example, the "Trajectory Calculator" described in "Statcast and the Baseball Trajectory Calculator" by David Kagan and Alan M. Nathan, Phys. Teach. 55, 134 (2017). The methods described in this disclosure, including those of Figures 5 and 6, may be considered methods for synchronizing two or more cameras.

[0049] In some embodiments, two or more two-dimensional common planes may be constructed and used for object trajectory fitting. In an exemplary embodiment, the intersection of three or more consecutive two-dimensional virtual planes may be used to construct a ball trajectory that intersects these parallel planes. It should be understood that the number of two-dimensional virtual planes, their positions, and orientations are configurable. In some embodiments, a three-dimensional homography matrix for each of two or more cameras may be obtained after calibrating each of the two or more cameras. In some embodiments, after calibration, only a single three-dimensional homography matrix is ​​obtained for each calibrated camera. Multiple two-dimensional virtual planes for each camera may then be generated or constructed based on this three-dimensional homography matrix. After synchronizing two or more cameras using the plane intersection, that is, after the intersection of each virtual plane is obtained, a ball trajectory may be constructed. Because the multiple two-dimensional virtual planes are generated from the same three-dimensional homography matrix for each camera, calibration of each two-dimensional virtual plane is not required even if the selection of the two-dimensional virtual planes is changed (e.g., from a first set of two-dimensional virtual planes to a second set of two-dimensional virtual planes). In some embodiments, two-dimensional virtual planes may be constructed to cover the entire playground. As some two-dimensional virtual planes are more reliable or suitable for trajectory fitting purposes, the pitch or density of the virtual planes may need to be adjusted. Advantageously, as mentioned above, no further calibration of the new or adjusted two-dimensional virtual planes is required.

[0050] 7 illustrates an exemplary system 500 that may be used with computer system 200 of FIG. 1 in accordance with at least one embodiment described in the present disclosure. System 500 may include a processor 510, a memory 512, a communication unit 516, a display 518, a user interface unit 520, and peripheral devices 522, all of which may be communicatively coupled. In some embodiments, system 500 may be part of any of the systems or devices described in this disclosure. For example, system 500 may be part of controller 110 of FIG. 1. Additionally or alternatively, system 500 may be part of computing system 200 of FIG. 1.

[0051] In general, processor 510 may include any computer, computing entity, or processing device, including various computer hardware or software modules, and may be configured to execute instructions stored on any applicable computer-readable storage medium. For example, processor 510 may include a microprocessor, microcontroller, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or any other digital or analog circuit configured to interpret and / or execute program instructions and / or process data.

[0052] 7 as a single processor, it will be understood that processor 510 may include any number of processors distributed across any number of networks or physical locations configured to individually or collectively perform any number of operations described in this disclosure. In some embodiments, processor 510 may interpret and / or execute program instructions and / or process data stored in memory 512. In some embodiments, processor 510 may execute program instructions stored in memory 512.

[0053] For example, in some embodiments, processor 510 may execute program instructions stored in memory 512, e.g., related to simulating an object's trajectory, such that system 500 may perform or direct the performance of operations associated with the system as directed by the instructions. In these and other embodiments, the instructions may be used to perform one or more operations of the methods described in Figures 5 and 6 above.

[0054] Memory 512 may include a computer-readable storage medium or one or more computer-readable storage media for carrying or storing computer-executable instructions or data structures. Such computer-readable storage media may be any available medium that can be accessed by a computer, such as processor 510. By way of example, and not limitation, such computer-readable storage media may include non-transitory computer-readable storage media, including random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory devices (e.g., solid-state memory devices), or any other storage medium that can be used to store specific program code in the form of computer-executable instructions or data structures and that can be accessed by a computer. Combinations of the above may also be included within the scope of computer-readable storage media. Computer-executable instructions may include, for example, instructions and data configured to cause processor 510 to perform certain operations or groups of operations described in this disclosure. In these and other embodiments, the term "non-transitory" as described in this disclosure shall be construed to exclude only those types of transitory media that were held to be outside the scope of patentable subject matter in In re Nuijten, 500 F.3d 1346 (Fed.Cir.2007). Combinations of the above may also be included within the scope of computer-readable media.

[0055] The communications unit 516 may include any component, device, system, or combination thereof configured to transmit or receive information over a network. In some embodiments, the communications unit 516 may communicate with other devices at other locations, the same location, or, in some cases, other components within the same system. For example, the communications unit 516 may include a modem, a network card (wireless or wired), an infrared communications device, a wireless communications device (such as an antenna), and / or a chipset (such as a Bluetooth device, an 802.6 device (e.g., a metropolitan area network (MAN)), a WiFi device, a WiMax device, a cellular communications facility, etc.), etc. The communications unit 516 may enable data to be exchanged with a network and / or any other device or system described in this disclosure.

[0056] The display 518 may be configured as one or more displays, such as an LCD, LED, monitor, screen, or other type of display. The display 518 may be configured to output shot data, such as a simulated trajectory for an object, flight mapping, media content, and club identification information, as well as a schematic or representation of the environment through which the object traveled, at some point between the trajectory, a user interface, and other data as directed by the processor 510.

[0057] The user interface unit 520 may include any device that allows a user to interact with the system 500. For example, the user interface unit 520 may include a mouse, a trackpad, a keyboard, buttons, and / or a touchscreen, among other devices. The user interface unit 520 may receive input from a user and provide the input to the processor 510. In some embodiments, the user interface unit 520 and the display 518 may be combined. For example, a user may select a target area where placement of an object is desired via the user interface unit 520.

[0058] The peripheral devices 522 may include one or more devices. For example, the peripheral devices may include sensors, microphones, and / or speakers, among other peripheral devices. In these and other embodiments, the microphones may be configured to capture sound. The speakers may broadcast audio received by or otherwise generated by the system 500. The sensors may be configured to detect changes in light, pixels, sound, motion, rotation, position, orientation, magnetization, acceleration, tilt, vibration, force, speed, color, wind, etc.

[0059] Modifications, additions, or omissions may be made to system 500 without departing from the scope of the present disclosure. For example, in some embodiments, system 500 may include any number of other components that may not be expressly illustrated or described. Furthermore, depending on the particular implementation, system 500 may not include one or more of the illustrated and described components.

[0060] While the present invention has been particularly shown and described with respect to preferred embodiments thereof, it will be recognized by those skilled in the art that the foregoing and other changes in form and detail may be made therein without departing from the spirit and scope of the invention. It is therefore intended that the present invention not be limited to the exact forms and details described and illustrated, but fall within the scope of the appended claims.

Claims

1. 1. A method comprising: acquiring a plurality of images of an object moving along a trajectory in an environment, the plurality of images capturing the object at various positions along the trajectory, the plurality of images being acquired from two or more cameras positioned at various positions in the environment, each of the two or more cameras in communication with at least one processor, each of the two or more cameras having a field of view (FOV) that includes the trajectory of the moving object and a target plane, the target plane being a plane that the moving object traverses while moving along the trajectory, and each of the two or more cameras acquiring the images at a different time; determining, by the at least one processor, the plurality of different positions of the object within respective trajectories within respective two-dimensional camera planes for each of the two or more cameras; determining, by the at least one processor, a plurality of different positions of the object within the respective trajectories in the two-dimensional camera plane for each of the two or more cameras in a two-dimensional common plane using the plurality of different positions of the object within the respective trajectories in the two-dimensional camera plane for each of the two or more cameras; and determining, by the at least one processor, using the plurality of different positions of the object within the respective trajectories in the two-dimensional common plane for each of the two or more cameras, a position at which the object moving along the trajectory crosses the target surface.

2. 2. The method of claim 1, further comprising: determining, by the at least one processor, a time at which the object moving along the trajectory crosses the target surface using the location at which the moving object crosses the target surface.

3. 10. The method of claim 1, further comprising: determining coordinates of the plurality of different positions of the object in the respective trajectories in the two-dimensional camera plane for each of the two or more cameras using a first homography matrix obtained from calibrating each of the two or more cameras.

4. 4. The method of claim 3, wherein determining the plurality of different positions of the object within the respective trajectories within the two-dimensional common plane for each of the two or more cameras comprises transforming the coordinates of the plurality of different positions of the object within the respective trajectories within the two-dimensional camera plane for each of the two or more cameras to the common plane using a second homography matrix for each of the two or more cameras, the second homography matrix describing a projection from two-dimensional camera coordinates onto the two-dimensional common plane.

5. 5. The method of claim 4, wherein determining the location where the object moving along the trajectory intersects the target surface comprises determining the location where the respective trajectories of each of the two or more cameras intersect in the two-dimensional common plane.

6. 6. The method of claim 5, wherein determining the time at which the object moving along the trajectory crosses the target surface comprises back-projecting the position at which the object moving along the trajectory crosses the target surface onto the two-dimensional camera plane for each of the two or more cameras.

7. 7. The method of claim 6, further comprising using the second homography matrix to back-project the location where the object moving along the trajectory crosses the target surface onto the two-dimensional camera plane for each of the two or more cameras.

8. 7. The method of claim 6, further comprising: determining the time at which the object moving along the trajectory crosses the target surface using an intersection ratio between the plurality of different positions of the object and the back-projected position in a selected two-dimensional camera plane and corresponding timestamps of the plurality of different positions of the object and the back-projected position.

9. The method of claim 8 , wherein the time at which the object moving along the trajectory crosses the target surface is a local camera time for each of the two or more cameras.

10. 10. The method of claim 9, wherein the object is a baseball pitched from a launch location toward a strike zone, the target surface is the strike zone, and the determined time is the time when the baseball crosses the strike zone.

11. 1. A system comprising: two or more cameras positioned at different locations within the environment; one or more processors in communication with the two or more cameras; A non-transitory computer-readable medium containing instructions that, in response to execution by the one or more processors, control execution of operations of the one or more processors, the operations including: acquiring a plurality of images of an object moving along a trajectory in the environment, the plurality of images capturing the object at various positions along the trajectory, the plurality of images being acquired from the two or more cameras, each of the two or more cameras having a field of view (FOV) that includes the trajectory of the moving object and a target plane, the target plane being a plane that the moving object traverses while moving along the trajectory, and each of the two or more cameras acquiring the images at a different time; determining a plurality of different positions of the object within respective trajectories in respective two-dimensional camera planes for each of the two or more cameras; using the plurality of different positions of the object within the respective trajectories in the two-dimensional camera plane for each of the two or more cameras, to determine the plurality of different positions of the object within the respective trajectories for each of the two or more cameras within a two-dimensional common plane; and determining a position at which the object moving along the trajectory crosses the target surface using the plurality of different positions of the object within the respective trajectories in the two-dimensional common plane for each of the two or more cameras.

12. The system of claim 11 , wherein the operations further include using the location at which the moving object crosses the target surface to determine a time at which the object moving along the trajectory crosses the target surface.

13. 12. The system of claim 11 , further comprising: calibrating each of the two or more cameras to obtain a first homography matrix; and using the first homography matrix for each of the two or more cameras to determine coordinates of the plurality of different positions of the object in the respective trajectories in the two-dimensional camera plane for each of the two or more cameras.

14. 14. The system of claim 13, wherein determining the plurality of different positions of the object within the respective trajectories within the two-dimensional common plane for each of the two or more cameras comprises transforming the coordinates of the plurality of different positions of the object within the respective trajectories within the two-dimensional camera plane for each of the two or more cameras to the common plane using a second homography matrix for each of the two or more cameras, the second homography matrix describing a projection from two-dimensional camera coordinates onto the two-dimensional common plane.

15. 15. The system of claim 14, wherein determining the location where the object moving along the trajectory intersects the target surface comprises determining the location where the respective trajectories of each of the two or more cameras intersect in the two-dimensional common plane.

16. 16. The system of claim 15, wherein determining the time at which the object moving along the trajectory crosses the target surface comprises back-projecting the position at which the object moving along the trajectory crosses the target surface onto the two-dimensional camera plane for each of the two or more cameras using the second homography matrix.

17. 17. The system of claim 16, further comprising: determining the time at which the object moving along the trajectory crosses the target surface using an intersection ratio between the plurality of different positions of the object and the back-projected position in a selected two-dimensional camera plane and corresponding timestamps of the plurality of different positions of the object and the back-projected position.

18. 12. The system of claim 11, wherein the FOV further includes a launch location of the moving object, the object being a baseball pitched from the launch location toward a strike zone, the target plane being the strike zone, and the determined time being a time when the baseball crossed the strike zone.

19. A non-transitory computer-readable medium containing instructions that, in response to execution by one or more processors, control execution of operations of the one or more processors, the operations including: acquiring a plurality of images of an object moving along a trajectory in an environment, the plurality of images capturing the object at various positions along the trajectory, the plurality of images being acquired from two or more cameras positioned at various positions in the environment, each of the two or more cameras having a field of view (FOV) that includes the trajectory of the moving object and a target plane, the target plane being a plane that the moving object traverses while moving along the trajectory, and each of the two or more cameras acquiring the images at a different time; determining a plurality of different positions of the object within respective trajectories in respective two-dimensional camera planes for each of the two or more cameras; using the plurality of different positions of the object within the respective trajectories in the two-dimensional camera plane for each of the two or more cameras, to determine the plurality of different positions of the object within the respective trajectories for each of the two or more cameras within a two-dimensional common plane; and using the plurality of different positions of the object within the respective trajectories in the two-dimensional common plane for each of the two or more cameras to determine a position at which the object moving along the trajectory crosses the target surface.

20. 20. The non-transitory computer-readable medium of claim 19, wherein the operations further include using the location at which the moving object crosses the target surface to determine a time at which the object moving along the trajectory crosses the target surface.

21. 20. The non-transitory computer-readable medium of claim 19, wherein the operations further comprise: calibrating each of the two or more cameras in the environment using a calibration pattern to obtain a first homography matrix for each of the two or more cameras; and using the first homography matrix to determine coordinates of the plurality of different positions of the object on the respective trajectories in the two-dimensional camera plane for each of the two or more cameras; determining the plurality of different positions of the object within the respective trajectories within the two-dimensional common plane for each of the two or more cameras includes transforming the coordinates of the plurality of different positions of the object within the respective trajectories within the two-dimensional camera plane for each of the two or more cameras to the common plane using a second homography matrix for each of the two or more cameras, the second homography matrix describing a projection from two-dimensional camera coordinates onto the two-dimensional common plane; determining the position where the object moving along the trajectory intersects the target surface includes determining the position where the respective trajectories of each of the two or more cameras intersect in the two-dimensional common plane.

22. determining the time at which the object moving along the trajectory crosses the target surface includes back-projecting, using the second homography matrix, the position at which the object moving along the trajectory crosses the target surface onto the two-dimensional camera plane for each of the two or more cameras; 21. The non-transitory computer-readable medium of claim 20, wherein determining the time at which the object moving along the trajectory crosses the target surface comprises using an intersection ratio between the plurality of different positions of the object and the back-projected position in a selected two-dimensional camera plane and corresponding timestamps of the plurality of different positions of the object and the back-projected position to determine the time at which the object moving along the trajectory crosses the target surface.

Images