Method, device and program for detecting the position of a sports projectile within a scene

Abstract

Device for determining the position of a sports projectile within a scene during a time interval in which the sports projectile is obscured from the view of a camera, the device comprising: an interface capable of receiving a first sequence of images of the scene, captured by the camera at a first predetermined frame rate before the time interval, and a second sequence of images of the scene, captured by the camera at a second predetermined frame rate after the time interval; a sports projectile detection device capable of determining the position of the sports projectile within the scene for each image in the first and second sequences of images; and a velocity detection unit capable ofto determine the velocity of the sports projectile before the time interval based on the determined position of the sports projectile within the scene for each frame in the first sequence of frames and the first frame rate, and the velocity of the sports projectile immediately after the time interval based on the determined position of the sports projectile in the scene for each frame in the second sequence of frames and the second predetermined frame rate, and a sports projectile positioning unit that is operable to determine the position of the sports projectile during the time interval using the determined position of the sports projectile within the scene for each frame of the first and second sequences of frames and / or the determined velocity of the sports projectile immediately before and / or immediately after the time interval.

Description

[0001] The present invention relates to a method, a device and a program.

[0002] The description of the technical background serves the purpose of providing a general outline of the context of the present invention. The work of the inventors, as set forth in the technical background section, as well as aspects of the description, are not considered to be prior art, either directly or indirectly.

[0003] In ball sports, such as football, it is essential to know whether a goal has been scored. A goal is generally scored when the entire ball has crossed a predetermined goal line. Referees are responsible for observing the ball and deciding whether or not the ball has crossed the goal line and, consequently, whether or not a goal is to be scored. Such a procedure, although well established, can be quite unreliable. If the referees' view is obstructed or if they make an error, important match decisions may be made incorrectly. There is therefore a need to establish camera- and / or computer-based technology to determine whether a ball has crossed a line in order to assist referees in their decision-making.

[0004] One proposal for such technology involves equipping the match ball with a sensor that is detected by a number of detection units in such a way that the ball's position can be calculated at any given time. The sensor would therefore emit radio frequency (RF) pulses that are measured by the detection units at periodic intervals, with the time it takes for the pulse to reach each detection unit indicating the ball's distance from the respective sensor. A problem with this approach, however, is that the sensor inside the ball could noticeably change the ball's weight and / or balance. This is highly undesirable. Quality control is also difficult, as it is possible for the ball to be tampered with (e.g., kicked into the crowd), altering the sensor's characteristics.

[0005] Another proposal involves attaching sensors to the two goalposts, one at each end of the goal line. The sensors are then designed to detect the presence of objects that appear between them (e.g., using laser technology), thus allowing the detection of objects crossing the goal line. However, this system is quite unreliable because it is difficult to distinguish between a ball crossing the goal line and other objects crossing the goal line (e.g., the soccer players themselves). Also, the goalposts to which the sensors are attached are subject to noticeable movement during the soccer match and are therefore not perfectly aligned with the goal line itself, reducing the reliability and accuracy of the system.

[0006] WO 01 / 41884 A1 discloses a device for detecting the position of a sports projectile within a scene.

[0007] The present invention aims to solve the above-mentioned problems.

[0008] According to one aspect of the present invention, an apparatus is provided for determining the position of a sporting projectile within a scene during a time interval in which the sporting projectile is obscured from the view of a camera, the apparatus comprising an interface operable to receive a first sequence of images of the scene captured by the camera at a first predetermined frame / image rate before the time interval, and a second sequence of images of the scene captured by the camera at a second predetermined frame / image rate after the time interval, a sporting projectile detecting device operable to determine the position of the sporting projectile within the scene for each image in the first and second sequences of images,a velocity detection unit operable to determine the velocity of the sporting projectile before the time interval based on the determined position of the sporting projectile within the scene for each frame in the first sequence of frames and the first frame rate, and the velocity of the sporting projectile immediately after the time interval based on the determined position of the sporting projectile in the scene for each frame in the second sequence of frames and the second predetermined frame rate, and a sporting projectile position determination unit operable to determineto determine the position of the sports projectile during the time interval using the determined position of the sports projectile within the scene for each image of the first and second sequence of images and / or the determined speed of the sports projectile immediately before and / or immediately after the time interval.

[0009] The first sequence of images and the second sequence of images of the scene may comprise a predetermined line. The sports projectile position determination unit may be operable to generate an indication signal in the event that the position of the sports projectile is or has been determined to have crossed the predetermined line.

[0010] The sports projectile position determining unit may be operable or operable to generate the indication signal in the event that the position of the sports projectile is or has been determined to have crossed the predetermined line by a distance greater than the radius of the sports projectile.

[0011] The predetermined line may be a goal line.

[0012] The sports projectile determination unit may be further operable or operable to calculate the position of the sports projectile according to the deformation of the sports projectile in each image within the second sequence of images.

[0013] According to another aspect, a method is provided for determining the position of a sports projectile within a scene during a time interval during which the sports projectile is obscured from the view of a camera, the method comprising: receiving a first sequence of images of the scene taken by the camera at a first predetermined frame rate before the time interval, and a second sequence of images of the scene taken by the camera at a second predetermined frame rate after the time interval, determining the position of the sports projectile within the scene for each image in the first and second sequence of images,Determining the speed of the sports projectile before the time interval based on the determined position of the sports projectile within the scene for each frame in the first sequence of frames and the first frame rate, and the speed of the sports projectile immediately after the time interval based on the determined position of the sports projectile within the scene for each frame of the second sequence of frames and the second predetermined frame rate, and determining the position of the sports projectile during the time interval using the determined position of the sports projectile within the scene for each frame within the first and second sequences of frames and / or the determined speed of the sports projectile immediately before and / or immediately after the time interval.

[0014] The first sequence of images and the second sequence of images of the scene may comprise a predetermined line. The method includes generating an indication signal in the event that the position of the sports projectile is or has been determined to have crossed the predetermined line.

[0015] The method may comprise generating the indication signal if the position of the sports projectile is or has been determined to have crossed the predetermined line by a distance greater than the radius of the sports projectile.

[0016] The predetermined line may be a goal line.

[0017] The method may comprise calculating the position of the sports projectile according to the deformation of the sports projectile in each image within the second sequence of images.

[0018] The preceding paragraphs have been provided as a general introduction and are not intended to limit the scope of the following claims. The described embodiments, together with the further advantages thereof, may best be appreciated by reference to the following detailed description, taking into account the accompanying figures.

[0019] A more complete statement of the disclosure and many of the attendant advantages thereof will be more clearly apparent by reference to the following detailed description and the accompanying drawings. Fig. 1A schematically shows positions of a plurality of cameras relative to a goal line of a soccer field according to an embodiment of the invention. Fig. 1B shows schematically cameras and the goal line according to Fig. 1B, from a different perspective. Fig. Figure 2 schematically shows a system for detecting the position of a ball within a scene according to an embodiment of the invention. Fig. 3 schematically shows an apparatus for detecting the position of a ball within a scene according to an embodiment of the invention. Fig. 4A schematically shows the positions of camera calibration reference marks according to an embodiment of the invention. Fig. 4B schematically shows a first possible position of camera calibration reference marks according to an embodiment of the invention. Fig. 4C schematically shows a second possible position of camera calibration reference marks according to an embodiment of the invention. Fig. Figure 5A schematically shows a ball with a characteristic arrangement or pattern on its surface. Fig. Figure 5B shows a schematic image of a scene in which only a portion of the ball is Fig. 5A is visible. Fig. Figure 5C shows schematically the determination of the position of the center of the ball in the scene from Fig. 5B according to an embodiment of the invention. Fig. Figure 6A schematically shows two visible regions of a ball with a characteristic arrangement or pattern on its surface. Fig. Figure 6B schematically shows an enlarged view of the first visible area of Fig. 6A, wherein the first visible area comprises first orientation data according to an embodiment of the invention. Fig. Figure 6C schematically shows an enlarged view of the first visible area of Fig. 6A, wherein the second visible area comprises first orientation data according to an embodiment of the invention. Fig. 7 schematically shows the calculated trajectory of a ball during a time interval during which the ball is occluded from the view of a camera, according to an embodiment of the invention. Fig. Figure 8 schematically shows the operation of a system for determining the position of a ball within a scene according to an embodiment of the invention.

[0020] Reference will now be made to the figures, wherein like reference numerals designate identical or corresponding parts in the several views.

[0021] The Fig. 1A and Fig. 1B illustrate the positions of a plurality of cameras 100 arranged near or around a goal line 112 of a soccer field 118. Each of the cameras is configured to capture images of substantially the same scene from different fields of view or angles, where the scene includes the goal line 112 and the entire goal 122. In this embodiment, two cameras 100 are arranged behind the goal line, two cameras 100 are arranged in line with the goal line, and two cameras are arranged in front of the goal line. When a game is in progress, the cameras are operated to capture images of the scene when the ball is near the goal line.

[0022] In one embodiment, the cameras 100 are positioned far from the goal line such that the size of the ball does not change significantly in the captured images of the scene. For example, each of the cameras may be positioned approximately 50 meters from the goal line. This is advantageous because the size of the ball does not change within the angle of view or field of view of each camera, so that image processing techniques for detecting the ball within the captured images, e.g., in the sense of a block matching method, can be performed with greater efficiency. This is because the model of the ball used in the block matching method requires only limited scaling.

[0023] Although the cameras 100 can be arranged at any height relative to the football field, in the embodiments, the cameras are positioned such that objects other than the ball (e.g., spectators at the edge of the field) barely obstruct the camera's field of view. Where the cameras 100 are arranged, for example, in a stadium, the cameras 100 can be provided, for example, in the canopy that covers the crowd.

[0024] Furthermore, although six cameras are depicted in the embodiments, different numbers of cameras may be used. However, if the number of cameras is reduced, it becomes more difficult to accurately determine the position of the ball in a situation where the view of one or more cameras is obscured.

[0025] The cameras are positioned so that predetermined camera calibration reference markers are clearly within the field of view of each camera. As explained later, these markers are necessary for detecting and correcting for motion in the cameras. In one embodiment, the reference markers are attached to the goal net supports 116 and the goal posts 120. By positioning the reference markers at these locations, the likelihood of these points moving within the scene is very low. Additionally, by placing the reference markers near or at the top of the goal net supports 116, the likelihood of the markers being obscured is reduced.

[0026] To reduce the likelihood of the cameras being moved by wind (e.g., in the case of an outdoor football stadium), the cameras should be mounted in a sheltered location. The cameras can also be fitted with an aerodynamically designed housing to further reduce movement due to wind.

[0027] The Fig. Figure 2 schematically illustrates a system 200 for detecting the position of a ball within a scene according to an embodiment of the invention. Each camera 100 is operable to capture an image of the scene from a different field of view. As can be seen from the Fig. 2, each of the images I1 - I6 taken by the respective cameras 1 - 6 is then forwarded to a camera calibration computer 202.

[0028] The camera calibration computer 202 is operable to detect at least one of the camera calibration reference markers 402 in the captured image and compare the position of the reference marker with its position in a previous image. As will be described later, in the scene itself, the reference marker remains stationary, so any difference in the position of the reference marker between the current and previous images indicates that the camera itself has undergone movement (e.g., panning, swinging, tilting, rotating, or rolling). If movement in the position of a reference marker is detected, the camera calibration computer 202 is operable to apply a corrective transformation to the received image such that the movement of the camera is corrected. In other words, this means that the corrective transformation is applied to all pixels of the image.This counteracts any movement of the pixels in the image due to camera movement.

[0029] Any known prior art method for detecting a difference in the position of a reference mark between a first and second image and subsequently transforming the second image to offset the difference can be used here. One such technique involves applying a transformation matrix, which allows the relevant calibration variables to be adjusted to best match the pixel difference between the first and second images. Once the image processing by each of the camera calibration computers 202 is complete, the corrected captured images I1'-I6' are transmitted to the controller 204.

[0030] The controller 204 is operable or operable to accurately determine the position of the ball within the scene from the captured corrected images I1'-I6'. As explained later, the controller 204 achieves this by processing each of the images I1'-I6' to determine position and orientation data for the ball within that image. The controller 204 then combines the position and orientation data for each image and uses the resulting combined data, along with the known relative positions of each camera 100, to determine the positions of the ball within the scene. The controller 204 is also operable to determine the trajectory of the ball within the scene during periods in which the ball is occluded from the view of all cameras (e.g.,(e.g., when a large number of players are near the goal line and consequently obscure the ball from the view of all cameras between them). As described later, this is achieved by using the position, velocity, and impact characteristic data for the ball immediately before and immediately after the time interval during which the ball is obscured.

[0031] If the position of the ball within the scene has been determined by the controller 204, either directly from the captured images I1'-I6' (in the case that at least part of the ball is sufficiently visible by the cameras 100) or by the ball trajectory detection mechanism (in the case that the ball is obscured), the controller 204 determines whether or not the ball as a whole has crossed the entire goal line. This is determined by taking into account the center of the ball, in that if the center of the ball has crossed the entire goal line by a length equal to the radius of the ball, then a goal is counted. It is clear that the position of the ball near the scene is actually determined by this position of the center of the ball within the scene.

[0032] In a case where it is determined that the ball as a whole has crossed the goal line, a goal indication signal indicating that a goal is to be scored is transmitted from the controller 204 to a wireless transceiver 206. The wireless transceiver then transmits the goal indication signal wirelessly to a headset 208, which is worn by the referee. The wireless signal is transmitted over a secure channel. It is advantageous to transmit the goal indication signal over a secure channel to prevent false signals from rogue sources (e.g., other wireless transceivers owned by third parties attempting to influence the outcome of a soccer match by transmitting a false goal indication signal to the headset 208). The secure channel can, for example,be established by a handshaking authentication process between the headset 208 and the wireless transceiver 206, e.g. before the start of the football game.

[0033] It should be noted that, although Fig. 2 depicts the camera calibration computers 202 as separate from the controller 204, it would be possible to instead incorporate the camera calibration computers 202 within the controller 204. It would also be acceptable for the wireless transceivers 206 to be provided within the controller 204.

[0034] Although the example of a headset 208 has been presented, the wireless signal could also be transmitted to another type of receiver, e.g., a wristwatch or an earpiece or hearing aid worn by the referees. A plurality of such receivers may also be provided so that all referees can receive the goal-scoring signal simultaneously.

[0035] Controller 204 may also be operable to transmit more than just the goal-indicating signal. For example, in a case where the goal is scored or nearly scored, the detected position of the ball within the scene may be displayed as a computerized image or video. The image or video could then be transmitted to the television station along with the transmission of the goal-indicating signal to the referees for inclusion in the television broadcast of the soccer match.

[0036] Although the system 200 has been described as processing only one set of captured images I1-I6 in connection with the embodiments, the system may also process sequences of images captured by the cameras 100 at a predetermined frame rate. For example, images I11-I61 are captured at time t1, images I12-I62 are captured at time t2, images I1 n - I6 n at time t n and so on. This allows the determination of the position of the ball at times t1, t2, ..., t nand so on, so that the position of the ball can be tracked over the time the ball is in play. Any predetermined frame rate may be used, although in embodiments, the frame rate is chosen to be high enough to track a changing position of the ball, but not so high that any processing delay occurs in the controller 204 (i.e., real-time processing in the system is possible). For example, the frame rates may be 25, 30, 50, 60, or 100 frames per second per camera (corresponding to a total of 150, 180, 300, 360, or 600 frames per second for six cameras).

[0037] Fig. Figure 3 schematically illustrates the controller 204 in greater detail, according to one embodiment of the invention. The controller 204 includes an interface 300 through which each of the corrected captured images I1'-I6' from the camera calibration computers 202 enters the controller 204. The images are stored in memory 302. The controller 204 also includes a CPU 304 for executing the processing steps for determining the required ball position and trajectory. Furthermore, a ball pattern memory module 206 is provided for storing data indicating a characteristic arrangement or pattern of the ball's surface. Furthermore, an interface for the wireless transceiver(s) is provided, via which the goal indication signal can be transmitted to the headset 208.In addition to images I1'-I6', memory 302 may also store other sets of images captured by cameras 100 at different times or points in time than those at which images I1'-I6' were captured. Memory 302 may also store computer instructions executed by CPU 304.

[0038] To detect the position of the ball within the scene, each of the images I1'-I6' is processed by the CPU 304. The CPU 304 first attempts to identify a portion of the ball in each image by locating areas or portions of the image that match characteristic arrangements or patterns on the surface of the ball specified by data in the ball pattern storage module 306 or ball arrangement storage module 306. This may be achieved using a block matching technique, although any other suitable image processing known in the art may be used to select areas of an image that substantially match a predetermined pattern or arrangement.

[0039] If such a match occurs, the CPU 304 determines position data for the matching section of the image. For example, this position data may be a list of pixel positions that describe or define the edge of the matching section within the image. The position of pixels may be x- and y-coordinates established relative to a reference point in the image. The predetermined reference point may, for example, be one of the corners of the image. It may also be determined using the position of a predetermined object within the image.

[0040] The CPU 304 can also determine orientation data for the ball in the image. This orientation data is based on the areas or portions of the characteristic arrangement or pattern on the surface of the ball detected in the image and, as described later, enables the CPU 304 to determine the position of the center or centre of the ball within the scene even if, for example, only a portion or area of ​​the surface of the ball is visible within the field of view of each of the cameras 100.

[0041] The CPU 304 is also capable of determining the actual or real-life position of the center of the ball within the scene using the determined position and orientation data for the ball in each of the captured images (I1' - I6') along with the relative positions for each of the cameras 100.

[0042] For each captured image of the scene, the CPU 304 converts the determined image position data for a matching section in the image into scene position data for the matching section within the scene. For example, if the image position data is defined by the pixel positions (x1, y1), (x2, y2), ..., (x n , y n ), then the corresponding scene position data will be the position (real x1, real y1), (real x2, real y2), ..., (real x n , real y n). The positions within the scene are defined relative to the point within the scene that corresponds to the predetermined reference or datum point within the image. For example, if the predetermined reference or datum point within the image is defined by the position of a stationary object within the image, then the corresponding point within the scene will be the position of the object itself.

[0043] The image position data and the scene position data for an image are related by a scaling ratio (e.g., 0.5 cm in the scene equals one pixel), which depends on various parameter settings for the camera capturing the image, such as the camera's focal length and zoom. For example, for longer focal lengths and larger zoom values, the values ​​for the length of each of the pixels defining the image position data will correspond to smaller real-world distances within the scene than would be the case for shorter focal lengths and smaller zoom values, resulting in a lower scaling ratio.

[0044] Once scene position data has been determined for each of the captured images I1'-I6', the position of the ball within the scene is determined from the scene position data, the orientation data, and the relative positions of the cameras 100. The relative positions and various parameter settings for each of the cameras 100 are stored in memory 302 and can be appropriately changed for different camera positions and settings in football stadiums of different sizes. The relative positions of the cameras can be defined, for example, by defining the position of a first camera as the origin and by defining the positions of all other cameras relative to this origin.

[0045] The relative positions of the cameras 100 must be accessible to the CPU 304 so that the CPU 304 can determine the three-dimensional position of the ball within the scene from two-dimensional scene position data and orientation data for each of the cameras 100. For example, if the first camera is capable of determining the position of the ball in a first coordinate system defined by the parameters (x1, y1) with respect to a first predetermined reference point, and a second camera, positioned differently with respect to the first camera, is capable of determining the positions of the ball in a second coordinate system defined by the parameters (x2, y2) with respect to a second predetermined reference point, then the CPU 304 can only determine the three-dimensional position of the camera if it knows how the first and second coordinate systems relate to each other.Since this relationship between the first and second coordinate systems is dictated by the relative positions of the cameras 100, the relative positions of the cameras 100 must be accessible to the CPU 304 in order to determine the three-dimensional position of the ball.

[0046] Although the example described above was given for a simplified situation in which the ball was not occluded from the view of any camera, the relative positions of the cameras 100 must also be accessible to the CPU 304 in order for the CPU 304 to be able to determine the position of the ball when the ball is partially occluded from view. This is because, for each image within a captured set of images I1'-I6', the position data for the matching or matching regions or sections within the image are defined in a coordinate system specific to the camera that captured the image. In order for the CPU 304 to be able to determine the positions of the ball within the scene from the position and orientation data, the relationship between the coordinate systems for each camera 100 must be accessible to the CPU 304.Again, the relationship between the different coordinate systems is dictated by the relative positions of the cameras. Consequently, the relative positions of the cameras must be accessible or known to CPU 304.

[0047] The CPU 304 may generate a number of confidence values ​​when attempting to determine matching areas or sections within an image. The confidence values ​​may correspond, for example, to the color of the ball, the shape of the ball, or the arrangement or pattern of the ball, and would enable the CPU 304 to focus the ball arrangement detection or ball pattern detection process on areas or regions of the image where matching areas or sections are likely to be found. For example, if the CPU 304 knows from the data stored within the ball arrangement storage module 306 that the surface of the ball is largely white, the CPU 304 may search exclusively for light-colored areas or sections of the image.Alternatively, CPU 304 may search for specific colors that are not present on the surface of the ball, thereby preventing regions or areas of the image in which those colors appear from being processed. This is advantageous in that it reduces the processing overhead for CPU 304 in locating matching or passive regions or sections within an image.

[0048] The CPU 304 may perform the process of finding ball arrangements or ball patterns on the areas or surfaces of each image in a set of images I1 n ' - I6 n ' also focus on areas where the ball is likely to be located, based on the position of the ball in previously captured images stored in memory 302. For example, if the ball in image I1 n-1 ' was found at the coordinates (x, y), which at time t n-1was recorded by a particular camera 100, then the CPU 304 can start searching for the ball pattern or ball arrangement in a predetermined area which is in the vicinity of the coordinates (x, y) in the next image I1 n ' which at time t n was recorded by the special camera 100. The determined position of the ball within at least one other image within the same set of images I1 n ' - I6 n', the at least one image captured by another camera 100, may also be used. Alternatively, any other suitable and predictive image processing technique known in the art may be used. Again, this has the advantage of reducing the processing load on the CPU 304 when trying to locate matching or matching areas or sections within an image because an intelligent starting point is selected for this search. It also allows the position of the ball to be determined much more quickly.

[0049] The CPU 304 may also eliminate noise or interference and / or false balls from each of the images, for example, using the confidence values ​​or by using another suitable technique known in the art. Such techniques may include using data from only a single image or using data from a plurality of images. The plurality of images may be images within the same set I1. n ' - I6 n ' or include different sets of images taken at different times.

[0050] To determine the trajectory (sometimes referred to as velocity because velocity includes the value of the travel speed or velocity (speed) and the direction) of the ball within the scene during a time interval in which the ball is occluded from view by all cameras, a first sequence of images is taken by the cameras 100 at a first frame rate immediately before the time interval over which the ball is occluded. This first sequence of images is stored in memory 302. A second sequence of images is then taken by the cameras 100 at a second predetermined frame rate immediately after the time interval during which the ball is occluded. The second sequence of images is also stored in memory 302. For each separate set of images I1 n ' - I6 n ', which at time t nrecorded and stored in memory 302, the position of the ball within the scene is determined by CPU 304 as described above.

[0051] The CPU 304 then determines the speed and direction of the ball immediately before the time interval in which the ball is occluded, using the determined position of the ball for each set of images I1 n ' - I6 n ' within the first sequence of images along with the first predetermined frame rate. Also, the CPU 304 determines the velocity and direction of the ball immediately after the time interval during which the ball is obscured, using the determined position of the ball for each set of images I1 m ' - I6 m' within the second sequence of images together with the second predetermined frame rate. For a sequence of images, the CPU 204 can, for example, determine the speed of the ball at a time t n determine if the set of images I1 n ' - I6 n ' is recorded by calculating the difference between the position of the ball within the scene, which is determined from the set of images I1 n ' - I6 n ', which at time t n were recorded, and the position of the ball within the scene, which was determined from the previous set of images I1 n-1 ' - I6 n-1 ', which at time t n-1 The CPU 304 then divides this difference by the time interval t n - t n-1 which, for consecutively captured sets of images, is equal to the inverse of the predetermined image rate or frame rate.

[0052] The speed and direction of the ball can also be collectively referred to as the velocity of the ball.

[0053] Using impact characteristic data for the ball in conjunction with the detected velocities before and after the time interval during which the ball is occluded, the CPU 304 can determine the position of the ball during that time interval. The impact characteristic data is stored in memory 302 and contains all data useful for modeling the trajectory of the ball if the ball experiences an impulse while occluded (e.g., if it hits the goal post or is kicked). The impact characteristic data may include, for example, the pressure to which the ball is inflated, information regarding the aerodynamics of the ball, or how the elastic properties of the material from which the ball is constructed change with different temperatures and different weather conditions.The impact characteristic data are determined experimentally for the match ball before the start of the football match and, importantly, describe the amount of deformation that the ball experiences for a given force of impact or shock for a given time after the impact or shock.

[0054] The determination of the trajectory of the obscured or hidden ball can be performed automatically for time intervals in which the CPU 304 determines that the ball is not visible. If the determined trajectory then shows that the entire ball must have crossed the entire goal line, the goal indication signal can be sent to the headset 208. Alternatively, to reduce the processing effort, the detection of the trajectory of the obscured ball can be initiated manually in cases where the ball is obscured while very close to the goal line. Such manual access is possible because sets of corrected captured images I1 n ' - I6 n ', which were recorded at a predetermined frame rate, can be stored in the memory 302 for a predetermined period of time, e.g. for 30 seconds, in order to be available as needed.

[0055] The data contained in the ball arrangement storage module 306 or ball pattern storage module 306 indicating the characteristic arrangement or pattern of the surface of the ball can be changed or updated with regard to soccer balls with a deviating or different characteristic arrangement or pattern (e.g., for ordinary and highly visible patterned soccer balls). In addition, different manufacturers have different ball designs. Therefore, a library of ball arrangements or patterns can be stored within the storage module 306 for a number of different balls and ball designs. This changing or updating can, for example,This can be achieved by making the ball arrangement storage module or ball pattern storage module 306 physically removable and replaceable, or by making the ball pattern storage module 306 operable to receive new or updated data electronically from an external source, e.g., an external computer (not shown). If the ball pattern storage module 306 is configured to receive new or updated data electronically, this can be done either wired or wirelessly by means of an appropriate connection between the external source and the ball pattern storage module 306.

[0056] In order to update the memory 302 with data relating to, for example, the position of the cameras 100, the various parameters of the cameras 100, and the impact characteristic data for the ball, the controller 204 must have a user interface (not shown) so that such data can be entered into the controller 204. This user interface may include, for example, a traditional display, a keyboard and mouse, or a touchscreen interface. Since whether or not a goal is scored is determined by the CPU 304 based on whether the center of the ball has crossed the entire goal line by a distance greater than the radius of the ball, the data indicating the radius of the ball is contained either within the memory 302 or in the ball arrangement memory module or ball pattern memory module 306 of the controller 204.

[0057] The Fig. 4A-4C schematically illustrate the positions of camera calibration reference marks according to an embodiment of the present invention.

[0058] Fig. 4A schematically shows a view from the side of the goal. The camera calibration reference markers 402 or camera calibration fiducials 402 are formed on or near the top of each of the goal net supports 116 and on the bottom of each goal post 120. The reference markers 402 may be located at various locations. However, it is important that such locations be selected such that the reference markers 402 are stationary or immobile within the scene, so that any change in the position of the reference markers 402 between one captured image and the next is caused by the movement of the camera 100 and not by the movement of the reference marker.Each of the reference marks 402 is positioned so that it is visible by at least one of the cameras 100 (visible by at least one camera 100 to an extent that the camera calibration computer 202 can reliably determine the location of the reference mark 202).

[0059] In one embodiment, each of the cameras 100 may be able to clearly view a plurality of reference markers 402 such that the camera movement may be effectively canceled out or compensated for in the captured images even if the camera's field of view with respect to one or more reference markers 402 is obscured. For example, the reference markers 402 and the cameras 100 may be positioned such that each camera 100 is able to view a reference marker 402 at the bottom or lower portion of the goal post 120 and a reference marker 402 on each of the goal net supports 116. Indeed, by placing the reference markers 402 at the top or upper portion of each goal net support 116, the likelihood of a reference marker 402 moving or becoming obscured between captured images may be very low.

[0060] Fig. 4B schematically shows an enlarged view of the reference marker 402 on top or at the top of a goal net support 116. Two reference markers 402 are shown and arranged such that each reference marker 402 is clearly within the field of view of a different camera 100 (i.e., the cameras 100 are shown in the figure much closer to the reference markers than is actually the case). In one embodiment, a larger number of reference markers 402 may be provided on top or at the top of the goal net support 116 such that each is within the field of view of a different camera 100.

[0061] Fig. Figure 4C schematically illustrates an enlarged view of a reference mark on the ground or at the bottom of a goal post 120. In the Fig. 4C, the reference mark 402 is viewed from a frontal or frontal view. Each of the reference marks 402 corresponds to a specific camera 100, wherever they may be located. Thus, the reference marks should be oriented so that they are viewed in the camera's field of view from a substantially frontal perspective or view. This makes it easier for the camera calibration computer 202, which is connected to a camera 100, to determine the position of the reference mark 402. As can be seen from Fig. 4B, in a preferred embodiment, a larger number of reference markers 402 may be provided than shown. These may be arranged on the floor or lower portion of the goal net support 116 such that each of them lies within the field of view of a different camera 100.

[0062] As in the Fig. 4A-4C, in one embodiment, the reference marker 402 has a rectangular shape and is formed by a light-colored inner rectangle with a dark-colored rectangular wall. This makes it easier to dictate the position of the reference marker 402 in each image because there is a specific pattern or arrangement for the marker. Further, to more easily detect the reference marker 402 in each image, the reference marker 402 may be fluorescent and / or reflective. Positioning the reference markers 402 on top or at the top of the goal net supports 116 and at the bottom or bottom of the goal posts 120 is preferred because, unlike other parts or areas of the goal frame, these parts are subject to very little movement during the course of the soccer game.

[0063] Fig. 5A schematically illustrates a soccer ball 500 with a characteristic arrangement or pattern 502 on its surface. In this particular example, pattern 502 consists of dark-colored hexagonal shapes on a light-colored background. The data indicating characteristic pattern 502 is contained in ball arrangement module or ball pattern module 306.

[0064] Fig. 5B schematically shows a scene 504 during a soccer game in which the soccer ball 500 is partially obscured from view by an obstacle 506 (in this case, the obstacle is a soccer player's leg). In the case where an image of the scene is captured by a camera 100, it is correctively adjusted by the camera calibration computer 202 connected to the camera and then transmitted to the controller 204, where the CPU 304 within the controller processes the image to locate the position and orientation data for the ball within the image based on the data representing the characteristic arrangement or pattern 502 of the ball stored in the ball pattern storage module 306.

[0065] Fig. 5C schematically shows an enlarged view of the scene 504, which includes the ball 500 and the obstacle 506. Fig. 5C also shows areas or sections 508, 510 of the scene image that match the characteristic pattern 502 of the ball and are therefore detected by the CPU 304. These matching areas or sections 508, 510 correspond to the ball within the scene image. In many cases where the obstacle 506 is larger, only a single one of the matching areas or sections 508, 510 may be visible and therefore detectable by the CPU 304.

[0066] Furthermore, the lighting or exposure conditions within the scene may change during the game. This change in lighting may cause the colors of the ball to appear darker or lighter. If the playing field has artificial lighting that is turned on during the game, the color of the ball may appear brighter, with a whitened or brightened area where the artificial light is reflected off the ball. To account for this, the characteristic pattern 502 of the ball stored in the memory module 306 may include and apply an illumination or light algorithm to correct for these changes. Such algorithms are known in the art of image processing and will not be described in detail here.

[0067] As previously described, the CPU 304 is capable of determining the position and orientation data for the ball within each image within the set of images I1 n ' - I6 n ' to determine the actual position of the center or centre 512 of the ball 500 within the scene.

[0068] The Fig. 6A - 6C schematically illustrate an example of orientation data for an image of a ball within a scene. Fig. Figure 6A again shows the soccer ball 500 with a characteristic pattern 502. The visible portions 600 and 602 are shown, each of which has characteristic pattern portions 604 and 606, respectively. It is assumed that for a particular image of the scene within a captured set of images I1 n ' - I6 n' the visible areas or sections 600, 602 are the only visible sections of the ball within the image. Accordingly, as already described, the characteristic pattern areas or sections 604 and 606 within the visible areas or sections 600, 602 are detected by the CPU 304, and position data is determined for each of the visible sections 600, 602. In this example, the characteristic pattern sections 604 and 606 will form the orientation data for the ball within the image.

[0069] The Fig. 6B and Fig. 6C schematically show enlarged views of the visible areas 600 and 602. The CPU 304 maps each of the characteristic pattern areas 604, 606 to the corresponding portions of the data indicative of the characteristic pattern 502 stored within the ball pattern storage module 306. In combination with the position data for each of the visible areas 600, 602, the CPU 304 then has sufficient information to determine the position of the center or center 512 of the ball 500 within the image. This is because, given the position data for each of the visible areas 600 and 602 within the image, only one position of the center or center 512 of the ball 500 is capable of producing the specific characteristic pattern areas 604 and 606 within the respective visible areas or portions 600 and 602.

[0070] For simplification, the Fig. 6A - 6C, how the CPU 304 determines the position of the center or centre of the ball within a single image within a captured set of images I1 n ' - I6 n ' using the position and orientation data for the ball within the image. However, it would not be possible to determine the 3-dimensional position of the center of the ball within the scene for a single image, since the captured image itself is only 2-dimensional. For example, if only the visible area 600 were visible within the captured image, it would be very difficult for the CPU 304 to accurately determine the position of the center or middle of the ball within the scene, because multiple positions of the ball are likely conceivable, which can essentially evoke the characteristic pattern section 604 within the visible area 600.

[0071] In reality, the CPU 304 is therefore operated to obtain position and orientation data for the ball for each image within a captured set of images I1 n ' - I6 n ' is used, together with the positions and various parameter settings of the cameras 100, to determine the 3-dimensional p1 of the center of the ball within the scene. For example, if a first visible area or section with a first characteristic pattern section exists in a first image together with a second visible area or section with a second characteristic pattern section in a second image, the CPU 304 is operated to determine the position of the center of the ball within the scene based on the position data and on the basis of the characteristic pattern section for each of the first and second visible areas using the same principle as described in connection with the Fig. 6A - 6C. Since two separate images (from two separate cameras) were used, it is possible to determine the 3-dimensional position of the center of the ball within the scene.

[0072] Fig. Figure 7 schematically shows a trajectory of the ball, which was determined by means of the CPU 304 during a time interval while the ball was hidden from the view of the cameras 10. It is assumed that the ball at time t n is obscured when the positions of the ball within the scene are determined by the CPU 304 from the set of images I1 n ' - I6 n ', which is recorded by the cameras 100 at time t nrecorded can no longer be accurately determined. For example, the ball may be surrounded by a large number of obstacles such that no part of the ball is visible to any of the cameras 100. Alternatively, the ball may only be visible to a single camera 100, in which case the CPU 304 cannot be provided with sufficient position and orientation data to determine the positions of the ball within the scene.

[0073] Fig. 7 shows how the ball 500 enters the hidden area 700 with the center or middle 512 via the incoming trajectory 702 and leaves the hidden area 700 again via the outgoing trajectory 704. The CPU 304 is capable of calculating the velocity v1 of the ball at time t1 immediately before being hidden by the hidden area 700, as well as the velocity v2 of the ball at time t2 immediately after being hidden in the hidden area 700, which was previously described. The CPU 304 is then capable of determining the occluded trajectory 708 of the ball within the occluded region 700 using at least one aspect of the determined position of the ball within the scene at times t1 and t2, the determined velocity of the ball at times t1 and t2, and the impact characteristic data for the ball stored in the memory 302.

[0074] In the Fig. In the example illustrated in Figure 7, the goal line 706 is located within the occluded area 700. The occluded trajectory 708 determined by the CPU 304 indicates that the center 512 of the ball 500 crossed the goal line 708 during the time interval during which the ball was occluded. Therefore, the goal was counted. Embodiments of the present invention therefore enable a determination of whether or not the ball crossed the goal line, even during time intervals during which the ball is occluded from view by the cameras 100.

[0075] By knowing the accelerations, velocities, positions, and / or relative times for the incoming and outgoing trajectories, the system is able to interpolate the position / time information by calculating the force acting on the ball during the change of trajectory. Knowing the force acting on the ball enables the compression on the ball to be calculated, as well as the position / time of the object that came into contact with the ball.

[0076] So if before the ball is hidden the ball is moving with speed v1, the angle of the trajectory and the speed or velocity of the ball are known. If after the ball is seen again the ball is moving with a speed v2, the angle of the trajectory and the speed or velocity of the ball are known. Since it is assumed that there is a force acting on the ball during the hidden period (either because the ball is deflected by an object or because, for example, a player kicks the ball) such that the ball is moving with speed v2, the position at which the force must have acted can be assumed to be the furthest distance the ball has moved.

[0077] Since the change in the ball's velocity is a result of the force acting on it, it is possible to calculate the force acting on the ball. Furthermore, since the ball's velocity v2 (i.e., its speed and direction) is known, it is possible to extrapolate when the force must have acted on the ball. Accordingly, the ball's furthest position can be calculated.

[0078] Additionally, by knowing the compression of the ball, it is possible to determine when the force must have been applied to the ball. For example, if the force was applied just before the ball became visible again, the ball will appear deformed due to the applied force. However, if the force was applied shortly after the ball was hidden, the ball would have regained its shape and thus appear less deformed. The degree to which the ball is deformed is calculated based on the pressure on the ball (and other relevant shock or impact characteristics) and the force applied to the ball.

[0079] A brief description of the operation or operation of the system according to Fig. 8. During the game, all cameras 100 capture images from different fields of view or angles of the game, as illustrated in connection with step 800. The cameras 800 also capture the reference markers 402 located at the base of the goalpost and at the top of the goal net supports 116. For each image, the position in the image associated with at least one of the reference markers 402 is determined, as illustrated in connection with step 802. Since the reference markers 402 are assumed to be at a fixed location within the scene, any movement of the reference markers 402 between successive images is due to movement of the cameras 100.In step 804, the difference in position between the at least one reference marker 402 captured by each camera 100 in consecutive images is used to determine a motion transformation. In other words, this means that all other pixels in the captured image are moved by the same amount, since the movement of the reference markers 402 in the image is due to the movement of the camera 100. Consequently, the motion transformation is applied to the position of each pixel in the captured image.

[0080] After applying the motion transformation to each pixel, regions or sections of each image that match the characteristic pattern of the ball's surface are detected. Position and orientation data are determined for each of these matching regions, as shown in step 806. The position and orientation data for each image are then used to determine the position of the ball within the scene according to step 808. The determination of the matching regions or sections, the determination of the position and orientation data for the matching regions, and the subsequent determination of the position of the ball within the scene are described in conjunction with the Fig. 3 and 5A - 6C.

[0081] In step 810, a decision is made based on the determined position of the ball within the scene as to whether the entire ball has crossed the entire goal line. If it can be assumed that the entire ball has crossed the entire goal line, a goal indication signal is generated, as shown in connection with step 812. This goal indication signal is then transmitted to the referee's headset.

[0082] Although the foregoing description has been given in connection with goal lines, the present invention is not limited to this aspect. This means, in particular, that in embodiments, the referee(s) may receive an indication regarding the ball crossing a line within the playing field, e.g., the touchline (throw-in line) or the goal line or goal kick line, or the like.

[0083] Although the previous description referred to balls, any other sports projectiles can also be considered, e.g. shuttlecocks or ice hockey pucks.

[0084] To the extent that embodiments of the present invention are subject to implementation, at least in part, using software-controlled data processing devices, a computer program for providing such software control and a transmission, storage, or other media through which a computer program is provided also form aspects of the present invention.

[0085] Numerous modifications and variations of the present disclosure are conceivable in light of the above teachings. It is therefore understood that, within the scope of the claims, the present invention may be modified accordingly.

[0086] Insofar as embodiments of the present invention have been presented in terms of an implementation, at least partially by means of a software-controlled data processing device, it will be understood that non-volatile and machine-readable storage media carrying such software, e.g., an optical disk, a magnetic disk, a semiconductor memory, or the like, are also subject matter of the present invention.

Claims

[1] A device for determining a position of a sports projectile (500) within a scene during a time interval in which the sports projectile (500) is obscured from a view of a camera (100), the device comprising: an interface (300) operable to receive a first sequence of images (I1 - I6) of the scene captured by the camera (100) at a first predetermined frame rate before the time interval, and a second sequence of images (I1 - I6) of the scene captured by the camera (100) at a second predetermined frame rate after the time interval, a sports projectile detection device operable to determine the position of the sports projectile (500) within the scene for each image in the first and second sequences of images (I1 - I6), a speed detection unit operable to determine the speed of the sports projectile (500) before the time interval based on the determined position of the sports projectile (500) within the scene for each frame in the first sequence of frames (I1 - I6) and the first predetermined frame rate, and the speed of the sports projectile (500) immediately after the time interval based on the determined position of the sports projectile (500) in the scene for each frame in the second sequence of frames (I1 - I6) and the second predetermined frame rate, and a sports projectile position determination unit operable to determine the position of the sports projectile (500) during the time interval using the determined position of the sports projectile (500) within the scene for each image of the first and second sequence of images (I1 - I6) and / or the determined velocity of the sports projectile (500) immediately before and / or immediately after the time interval. [2] Device according to claim 1, in which the first sequence of images (I1 - I6) and the second sequence of images (I1 - I6) of the scene have a predetermined line and wherein the sports projectile position determining unit is operable to generate an indication signal in the event that the position of the sports projectile (500) is or has been determined to have crossed the predetermined line. [3] Device according to claim 1, wherein the sports projectile position determining unit is operable to generate an indication signal in the event that the position of the sports projectile (500) is or has been determined to have crossed a predetermined line by a distance greater than a radius of the sports projectile (500). [4] Apparatus according to any one of claims 2 to 3, wherein the predetermined line is a goal line. [5] The apparatus of any one of claims 1 to 4, wherein the sports projectile position determining unit is further operable to calculate the position of the sports projectile (500) according to a deformation of the sports projectile (500) in each image within the second sequence of images (I1 - I6). [6] A method for determining a position of a sports projectile (500) within a scene during a time interval during which the sports projectile (500) is obscured from a view of a camera (100), the method comprising: Receiving a first sequence of images (I1 - I6) of the scene taken by the camera (100) at a first predetermined frame rate before the time interval, and a second sequence of images (I1 - I6) of the scene taken by the camera (100) at a second predetermined frame rate after the time interval, Determining the position of the sports projectile (500) within the scene for each image in the first and second sequence of images (I1 - I6), Determining a speed of the sports projectile (500) before the time interval based on the determined position of the sports projectile (500) within the scene for each frame in the first sequence of frames (I1 - I6) and the first predetermined frame rate, and the speed of the sports projectile (500) immediately after the time interval based on the determined position of the sports projectile (500) within the scene for each frame of the second sequence of frames (I1 - I6) and the second predetermined frame rate, and Determining the position of the sports projectile (500) during the time interval using the determined position of the sports projectile (500) within the scene for each image within the first and second sequence of images (I1 - I6) and / or the determined speed of the sports projectile (500) immediately before and / or immediately after the time interval. [7] Method according to claim 6, in which the first sequence of images (I1 - I6) and the second sequence of images (I1 - I6) of the scene have a predetermined line and in which the method comprises generating an indication signal in the event that the position of the sports projectile (500) is or has been determined to have crossed the predetermined line. [8] Method according to claim 7, comprising generating the indication signal if the position of the sports projectile (500) is or has been determined to have crossed the predetermined line by a distance which is greater than a radius of the sports projectile (500). [9] A method according to any one of claims 7 to 8, wherein the predetermined line is a goal line. [10] Method according to one of claims 6 to 9, comprising calculating the position of the sports projectile (500) according to a deformation of the sports projectile (500) in each image within the second sequence of images (I1 - I6). [11] A computer program comprising computer-readable instructions which, when loaded onto a computer, cause the computer to carry out a method according to any one of claims 6 to 10. [12] A computer program product adapted to store the computer program according to claim 11 therein or thereon.

Images