GAME FOLLOWING

Description

Brief description of the invention

[0001] Sporting activities are an integral part of physical fitness in society. Many athletes, however, want not only to participate in sports but also to improve their performance in their chosen sport to be competitive in events. In many sports, the lack of meaningful data about one's own performance poses a significant problem, as it makes it difficult to identify errors and areas for improvement.

[0002] Activity monitoring systems, for example for ball sports such as football, are known and include, for example, sensors embedded in the ball that detect its movements. These sensors can, for example, use a magnetic field to determine the trajectory and / or position of the ball, as shown, for example, in EP 2 657 924 A1.

[0003] Other monitoring and / or information systems used in ball sports, particularly racket sports, are known as smart court systems. Tracking the movement of the often fast-moving objects on the court is very difficult and time-consuming, which is why complex systems are often used that enable tracking of the object on the court and in 3D space. This allows, among other things, real-time decisions to be made during a game, such as line calls in tennis, or for statistics to be collected on the course of a sporting event. These complex systems often include multiple and / or movable cameras that enable the tracking of an object, such as the ball.

[0004] The Smartcourt system of WO 2013 / 124856 A1, for example, comprises a large number of video cameras mounted in a fixed position. Using image data from the various camera perspectives, a 3D model of the court, the ball, and the players is created. Based on the movements captured within this model, conclusions are drawn about the type of stroke, the ball speed, and the points of impact, and these can be viewed by the athletes.

[0005] US9737784B1 is an electronic referee system in which several camera arrays are mounted at different heights on the net post. Objects can be tracked because they cross the fields of view of the individual arrays at different times, allowing conclusions to be drawn about their movements.

[0006] WO 2017 / 100465 A1 discloses a system with a camera positioned near the goalpost and covering both sides of the playing field. An auto-calibration process creates a playing field map that includes the positioning of the playing field, the net, and / or the camera. This allows the 2D coordinates from the video image generated during the game to be converted into 3D coordinates on the playing field map.

[0007] US 2015 / 018990 A1 describes a smart court system that adapts to a confined sports environment to enable real-time analysis and debriefing of sporting activities. The smart court system consists of: (i) an automatic recording system comprising multiple video cameras located within a playing field, configured for real-time (RT) recording of a sports session and utilizing an automatic calibration and stabilization module; and (ii) a data processing system comprising: (a) a recording module for capturing a video stream; (b) an object detection module positioned to extract foreground objects from each frame during the RT sports session;(c) an event module for automatically analyzing the movement and activities of the tracked objects in order to automatically identify and classify events, create a synchronized event log, and calculate statistics that occurred during the RT sports session; and (d) a presentation module that enables an immediate debriefing, a combined biomechanical and tactical analysis of the video.

[0008] US Patent 2018 / 154232A1 discloses a line-calling device that uses a single stationary camera to identify object state conditions, solving object tracking problems that would otherwise require solutions with multiple or moving cameras. Although the line-calling device is described in the context of an automated tennis line-calling device, it is also applicable to other object tracking problems that can be solved by identifying object states without the use of moving or multiple cameras. The tennis line-calling device is mounted on the net post of a tennis court, providing a close-up view of both static (court lines) and moving (ball, player, etc.) objects.In one embodiment, the integration of a video camera system with a line display function offers additional advantages, including the installation of a single turnkey device on any tennis net post to enable an automatic line display function without the need for external video cameras or processing functions.

[0009] Current smart court solutions for ball and / or racket sports are therefore complex and expensive, leaving athletes who do not train or play professionally without the ability to analyze their matches. Furthermore, systems with a camera perspective do not allow for the analysis of different shot types, such as in tennis. Additionally, existing systems are unable to determine the correct score if the system's recording of the game differs from the players' actual decisions.

[0010] The object of the present invention is therefore to provide a system, a method and a computer program product that eliminates the aforementioned disadvantages and also offers amateur players a way to track and analyze their games.

[0011] The present invention therefore relates to a method for game tracking as specified in claim 1. In one embodiment, the playing field is determined by calibration, wherein individual points in a two-dimensional (2D) image are assigned to a specific point in a two-dimensional (2D) model of the playing field. In a preferred embodiment, the calibration is performed using a homography matrix in which at least four points are used that correlate in the image and the model. In a particularly preferred embodiment, these are four corner points of the playing field.

[0012] In one embodiment, the detection of people on the playing field comprises the following steps: a) Generating at least one background image of a playing field; b) Generating a background model; c) Separating each background image from each currently captured image that includes a person by: i) determining an absolute difference between the background model and the currently captured image, and ii) generating a foreground mask, The foreground mask is characterized in that all pixels belonging to the foreground have the value 255 and the pixels of the background have the value 0. In a preferred embodiment, pixels where the difference is greater than a defined threshold are defined as the foreground and pixels with a smaller difference are defined as the background.

[0013] In a further embodiment of the present invention, the object on the playing field is identified by the movement of each captured image. In a preferred embodiment, the determination of the movement comprises the following steps: a) Creating difference images of successive images; b) Calculating absolute difference images where movement is visible at two points in the difference image, preferably a first position where the object was located in the previous image and a second position where the object is located in the current image; c) Generating a first motion mask for the first difference image and a second motion mask for the second difference image; and d) Generating a final motion mask that includes the movements shown in the first motion mask and the second motion mask.

[0014] In a particularly preferred embodiment, the conjunction of the individual pixels of both motion masks is performed using a bitwise AND operator.

[0015] In one embodiment, the trajectory or flight path of the object is determined by performing the following steps: a) Determining a first and a second starting point of a potential curve; b) Combining the potentially determined starting points with all potential objects in the subsequent image; c) Checking whether each object continues an existing curve; d) Determining the object's position in successive images; e) Comparing the curves from a) and d); f) Determining a curve's trajectory; and g) Classifying the curve by determining a second curve following the first.

[0016] In a preferred embodiment, pixels of objects that correlate with pixels of people are not taken into account.

[0017] In another embodiment, the identification of events during the course of the game comprises the following steps: a) Recording a trigger event; b) Comparing the position of at least one person and one object with the playing field model; c) Comparison with a database.

[0018] In a preferred embodiment, the events have been recorded in a categorized data set in a previous step in order to perform the comparison using an artificial intelligence trained on this data set.

[0019] In the present invention, a correction of the game score calculated during the game comprises the following steps: a) Monitoring of possible game scores during the game; b) Checking the compatibility of all possible game scores with the detected events; c) Checking the plausibility of the remaining game scores.

[0020] The present invention further relates to a device for tracking at least one object and player on a playing field as specified in claim 8.

[0021] Furthermore, the present invention relates to a computer program for carrying out a previously described method when the computer program is executed on a computer.

[0022] The present invention also relates to a computer program product a) with a storage medium on which a previously described computer program is stored; and / or b) that can be directly loaded into the internal memory of a digital computer and includes sections of software code that perform the steps according to the procedure described above when the product is running on a computer.

[0023] The task is further characterized by the embodiments in the claims and described in more detail by the explanations in the description, the examples and the drawings. Images

[0024] Fig. 1 Schematic representation of a homography transformation within field detection. Fig. 2 Schematic representation of curve determination. a) Difference image t; b) Difference image t+1; c) Point of impact; d) Difference image t+2; e) Difference image t+3; f) Impact. Fig. 3 Schematic representation of a program flow for a tennis match. Fig. 4 Schematic check for compatibility during the course of play using a scoring tree. Detailed description of the invention

[0025] The present invention relates to a method, a device, a computer program, and a computer program product for tracking an object on a static playing field. The object can be any conceivable object used in a game, such as a handball, a soccer ball, a tennis ball, a shuttlecock, or similar. In a preferred embodiment of the present invention, the object is a ball used in racket sports. In a particularly preferred embodiment, the ball game is tennis.

[0026] For the sake of simplicity, the invention is described below in relation to a tennis game comprising a tennis court, a tennis ball, a tennis player, and a tennis racket. However, this is representative of other sports that include a playing field, at least one player, and an object such as a ball. Furthermore, the use of "a" or "an" is employed to describe one or more elements, components, steps, modules, and / or things. These terms should be interpreted as encompassing one or at least one, and the singular includes the plural unless otherwise clearly intended.

[0027] The invention relates to a method for tracking a person and / or object on a playing field. The method comprises the steps of field detection, person detection, object detection, curve detection, event detection, and backward scoring. These steps are explained in more detail below and are carried out using suitable means, such as a data processing system or a computer.

[0028] Field recognition, i.e., the determination or initialization of the playing field, serves for calibration so that information from two-dimensional (2D) image acquisitions can be projected onto another two-dimensional (2D) image, the playing field model. This information includes, for example, points of impact and / or the positions of at least one player. In a preferred embodiment of the present invention, calibration is performed using a homography matrix. This matrix describes the translation, rotation, and distortion of one and the same planar surface, e.g., the playing field, in any two images. Preferably, at least four points from the image acquisition are used to calculate the homography matrix, which correlate with four points in the model image. Preferably, the four corner points of a single court, for example in tennis, are used, which are either pre-selected by the user or calculated by image processing methods.These methods can be standard image processing techniques or those from machine learning. An example from image processing would be edge detection, where various mathematical operators (edge ​​filters) such as the Laplace filter, Sobel operator, and Scharr operator are used to identify boundaries between different areas (e.g., lines on a playing field) within an image. By calculating the intersection points of the detected lines, the corner points of the playing field can then be determined. An example of a machine learning method would be a Convolutional Neural Network (CNN), which in turn requires a previously classified dataset. A Convolutional Neural Network is a network of neurons modeled on the biological functioning of the brain, implemented at the machine level.In this process, localized features are extracted from input images, and these image fields are convolutionally transformed using filters. The input to a convolution layer is an mxmxr image, where m is the height and width of the image and r is the number of channels. For example, an RGB image has r = 3 channels. This data is passed through multiple layers and repeatedly filtered and subsampled.

[0029] In a further step of the present procedure, persons located on the playing field are identified. The terms player, participant, person, and / or human are used interchangeably in the description and refer to all humans or non-humans, such as animals, located on the playing field. This is in contrast to objects, which are inanimate objects. The detection of at least one player or other participant is carried out using a so-called background subtraction procedure. This subtractor serves to separate objects in the foreground from objects in the background of an image and to check the objects with regard to their shape. To create a background model for the background subtraction, at least one image of the empty playing field is taken. In the next step, this background image, i.e.,The empty playing field is initialized using the well-known "Mixture of Gaussians Background Subtractor," and a background model is calculated. This technique assumes that the intensity values ​​of each pixel in the video can be modeled using the "Mixture of Gaussians" method. A simple heuristic determines which intensities are most likely to be in the background. The pixels that do not match these values ​​then become the foreground pixels. Preferably, multiple images captured by the image system are used to calculate and model the background and update it. In a preferred embodiment, an update occurs every 20 captured image data points, with continuous differences between the current image and the background model being incorporated into the background model.This ensures that constant changes in the background are not detected as foreground elements and therefore potentially as people.

[0030] For person recognition, in one embodiment of the present invention, a separation of the foreground and background images is performed for each image generated by the image acquisition system. This creates a foreground mask, which is particularly preferably characterized in that all pixels belonging to the foreground have the value 255 and the pixels of the background have the value 0. To achieve this, a computer calculates the absolute difference between the background model and the currently captured image of the image acquisition system. Using the absolute difference, the differences between two images can be represented, with the absolute difference indicating the absolute value of the difference between the two images. In a preferred embodiment, pixels where the difference is greater than a defined threshold are considered foreground pixels. Pixels with a smaller difference are considered background pixels.The extracted objects from the foreground mask are potential human beings. Using a data collection stored on the computer or other medium and / or comparison with data on the internet, the objects are examined and / or compared based on their height, width, area and / or position in the image and classified as human or non-human.

[0031] In the following procedure, the position of the object in the 2D model of the playing field is determined using the homography matrix generated in the field recognition and the lowest pixels of a human or non-human classified in the object recognition.

[0032] In addition to object recognition, a further step of the present invention involves detecting objects on the playing field using the image acquisition system. The term "objects" is used interchangeably with the term "ball" or "ball object" and includes all suitable objects that are used in the same way as a ball, for example, in racket sports, goal games, and / or bat-and-ball games. The object does not require a specific shape but depends on its suitability. Objects or balls include, but are not limited to, the following list: tennis balls, shuttlecocks, flying discs, soccer balls, volleyballs, etc. Accordingly, the ball can be round, oval, disc-shaped, or ring-shaped, equipped with handles or flight stabilizers (feathers), braided, perforated, with movable filling, hollow, or solid.

[0033] To detect the movement of the balls, movements in images from the image acquisition system are examined. For this purpose, difference images are created from successive images from the image acquisition system, allowing motion-induced changes between the images to be detected. In one embodiment, to enable the object to be recognized against different backgrounds, absolute difference images are created between successive images. As the ball moves through the field of view of the image acquisition system, it passes by different backgrounds that can be brighter, such as lamps or sunlight, or darker, such as the ground or walls, than the ball. Using the absolute differences, movement is always visualized at two points in the difference image.These locations preferably consist of a first position where the ball was located in the previous image and a second position where the ball is located in the current image. Accordingly, the disappearance of the ball in the first image and its reappearance are detected.

[0034] In one embodiment, preferably only the movement on a current image at time t = 1 (hereinafter referred to as image 1) is detected. For this purpose, all movements, i.e., the appearance and disappearance of the ball, are first determined. This determination is carried out in a first step by generating a difference image between an initial image (hereinafter referred to as image 0) and image 1. This detects the disappearance of the ball from a first position in image 0, where the ball was located, and the appearance of the ball in image 1 in a second position. In a further embodiment, a second difference image (difference image 2) is generated between image 1 and a second image (hereinafter referred to as image 2). Differential image 2 detects the disappearance of the ball from image 1 to image 2 and its appearance in image 2.

[0035] In a further step of the object recognition process, motion masks are generated for the first and second difference images. These are referred to as Motion Mask 1 and Motion Mask 2. In these motion masks, all pixels where the absolute difference value exceeds a defined threshold are assigned the value 255. All pixels below the defined threshold are assigned the value 0. A predefined value, such as a predefined threshold, is either a fixed value or a value determined at any given time before performing a calculation that compares a specific value with the predefined value. Accordingly, movements in the motion masks and their corresponding pixels have the value 255. In the next step, a final motion mask (Motion Mask 3) is generated.This final motion mask 3 contains the movements that appear in both the first motion mask 1 and the second motion mask 2. In a preferred embodiment of the present invention, the final motion mask is created using a logical conjunction of the individual pixels of both motion masks, preferably with a bitwise AND operator. Such a final motion mask 3 enables the extraction of potential objects, such as balls. In a preferred embodiment, a distinction is made between the extracted ball objects and whether they are located inside or outside a detected person. This is particularly important in curve detection, since the movement of potential objects inside a detected person is usually caused by the person and not by the object, for example, the ball.

[0036] In a further embodiment of the present invention, the trajectory or flight path of the object is determined. It is assumed that the object can both initiate and continue a curve. All potential objects that are not detected within a recognized person by the image acquisition system and / or the image processing system are defined as potential starting points for a new curve. In a next step, these potentially defined starting points are combined with all potential objects in the subsequent image. For example, the ball objects A and B recorded in a first image are identified. Accordingly, the corresponding curves 1(A) and 2(B) are generated. In a second image, the two potential ball objects C and D are identified. Based on the curves generated in the second image, the following curves are generated: 1(A, C), 2(A, D), 3(B, C), 4(B, D).In a particularly preferred embodiment, two further curves 5(C) and 6(D) are generated, since, as previously described, each object may start a new curve.

[0037] In a subsequent step of trajectory detection, each potential ball object is checked to see if it continues an existing curve involving two or more balls. To perform this trajectory check, the ball's position in successive images is determined and compared with the existing curve, which is defined by a first and second starting point. This check is preferably performed using additional data on the ball's direction of flight and / or its acceleration. This data—the direction of flight and acceleration—can be calculated by combining three object points, thus enabling a prediction of the next ball object's position in the image. Each ball located near the determined predicted position is added to the calculated curve until it has a start and end point.

[0038] In a particularly preferred embodiment, ball objects detected within a person, i.e., whose pixels correlate with the pixels of a person, are not used to generate new curves because, as previously explained, any movement located there may be caused by the person themselves and their movement, and not by the object, i.e., the ball, itself. However, ball objects in such positions, i.e., within the correlated area of ​​detected people, can be used in curve determination, i.e., for curve continuation and prediction.

[0039] The creation of new curves for each identified potential ball object results in numerous potential curves, many of which are duplicated. To prevent this redundancy, one embodiment of the present invention performs a check after each processed image. During such a check, identical or similar curves are identified, recognized, and then deleted from the computational data. In a preferred embodiment of the present invention, curves with fewer than eight determined ball positions, preferably fewer than six, and particularly preferably fewer than four determined ball positions, are also deleted from the computational data. Deleting the data sets with few actually determined ball positions reduces noise that could potentially lead to a distorted actual curve determination.

[0040] After the complete detection of several curves, which may differ in type, these are classified in one embodiment of the present invention. The classification is based on the sport. Preferably, data sets of curves and trajectories typical for the sport are stored and compared with the curves detected by the image acquisition system. In the example of tennis, the curves are categorized into different classes such as shot, serve, point of impact, over the net, end of the rally, etc. The terms "classification" and "characterization" are used interchangeably here and describe the identification of a curve according to its type, trajectory, and, if applicable, the event, such as a serve, a penalty kick, or a free throw. The classification of a curve (hereinafter K1) is achieved by determining the curve that follows it.To determine the curve following K1, at least one curve identified after K1 is checked, and the curve identified as the corresponding successor curve (K2) is checked. Preferably, all curves following K1 are checked as potential corresponding successor curves (K2') of K1. In a preferred embodiment, all curves (K2') that begin at a later time, but less than 30 frames later, are checked. In a particularly preferred embodiment, the curve that has the smallest spatial and temporal distance to K1 is defined as the corresponding successor curve K2.

[0041] In one embodiment, a corresponding predecessor curve (K1) is additionally determined for the successor curve (K2) using the same criteria as previously used for the successor curve K2. Such a comparison of the curves prevents curves generated on other playing fields from being characterized as the curve of the correct playing field.

[0042] After a corresponding comparison of curves K1 and K2—that is, determining K1 as the predecessor of K2 and vice versa—directions, changes in direction, and / or angles between the curves are recorded for the following characterization and compared with predefined patterns from a database. This database can be a compilation of internal (local) data stored on the system or stored externally on storage media. Furthermore, such a database can also be accessed via cloud-based storage, a network, or the internet.

[0043] In one embodiment of the present invention, the curves determined and defined by the image acquisition and processing systems are used to identify events in the game. Individual events throughout the entire game are identified and categorized. These events include, but are not limited to, the following: the beginning and end of each rally or game, change of serve or player ends, end of a set and beginning of a tie-break, free kick, penalty kick, free throw, pitch, tie, foul, etc. As previously explained, for the sake of simplicity, event recognition is illustrated using a tennis match as an example. These events are also referred to as triggering events. They are stored and categorized within a database.

[0044] The start of a rally is characterized by a player's serve, which is why the event detection system waits for a curve identified as a serve. Upon the occurrence of this serve curve, the system first checks the player's position to ensure the serve was detected on the correct tennis court and then determines the positions of both players at the time of the serve. This player position detection is achieved using the image capture system and the previously described object recognition. In a further embodiment, this player location information then provides information regarding the score, indicating whether the current score can be categorized as even or odd games and, for statistical purposes, whether it is a second serve (two consecutive serves from the same position).After detecting such a rally start, further events in the game are identified in a preferred embodiment. These could, for example, be the point of impact after a serve. In tennis, such a point of impact must be located in a different part of the court depending on the position of the serving player. This point of impact, or this event, indicates whether a second serve is likely to follow. In a preferred embodiment, for each identified event, it is checked whether this event occurred on the correct court and / or whether it occurred within the court boundaries. To determine whether the event took place inside or outside the court area, the ball and / or player position is compared with the identified court data.In a preferred embodiment, a probability is additionally determined based on the distance of the detected point of impact from the lines bounding the playing field, indicating that the system's decision was correct. The closer the point of impact is to the lines, the lower the probability.

[0045] In one embodiment of the present invention, the detection of events, as well as the positions of players and balls, takes place in the court model, whereby the points of impact and standing positions are transformed into the plane of the tennis court model using the homography matrix determined during calibration. Events, such as balls flying over the net and the end of a rally, preferably also undergo a check to determine whether the corresponding player is in the correct court. Furthermore, in a preferred embodiment, shots are characterized according to a data set. This data set for shots can comprise general local databases, external information from networks or the internet, or a self-generating data collection. Preferably, the data set for shots is a collection of data that has been extracted and categorized from previous recordings.Data extraction can be performed on local storage media as well as on storage media in local networks or the Internet.

[0046] Such a data collection also stores events, such as a change of serve, which always occurs at the beginning of a new game and constitutes a game-ending event. This event can be further confirmed if a player who did not serve previously performs several consecutive serves. These game-influencing events are also referred to as triggering events. The same applies to change of ends, which also constitute a game-ending event. Additionally, in one embodiment of the present invention, all events can be independently identified and characterized by a player or a third party through input. Such user input can automatically trigger an update of the data collections stored in the database.Furthermore, another embodiment may include elements that allow manual input of an event, such as the end of a match. Remote controls, smartphones, buttons, touchscreens, or other devices may be used for this purpose.

[0047] In one embodiment of the present invention, a scoring tree is generated. This scoring tree is created using data from events in the game and the game's progress. The scoring tree is generated by a computer and stored on a suitable storage medium. Suitable storage media include internal and external storage devices, such as cloud storage, network storage, and / or hard disk storage. Each event generates a corresponding data record in the scoring tree that defines the event. Based on this event, and preferably taking into account the surrounding circumstances, the potential further course of the game is determined. This can be displayed on suitable devices such as smartphones, screens, computers, laptops, tablets, and similar devices.After the match has ended, the scoring tree contains all potential scores at every point in the game, with each score having a probability that describes its plausibility. A check is then performed to determine the compatibility of all possible scores with the detected events. This check verifies whether a score is even possible in light of the detected events. Fig. 4Using a tennis match as an example, this section demonstrates how the existing results in the scoring tree are first checked for compatibility with the event "change of ends," and then the plausibility of the remaining scores is calculated based on the remaining probability. Plausibility describes the consistency or accuracy of the statements and serves as an evaluation criterion between the actually determined values ​​and the calculated values. Based on the course of the match and the events occurring within it, the potentially possible scores are narrowed down in real time, and the final result is adjusted accordingly. For example, in tennis, only an odd number of completed games are possible for events related to the change of ends, such as 0:1, 2:1, 0:3, etc.In contrast, when the serve changes sides, only an even number of completed games is possible, such as 0:2, 1:1, 4:0, etc. Therefore, in a preferred embodiment of the present invention, potential scores that do not meet the conditions of an actual event on the court are deleted from the scoring tree. Corresponding further potential scores are adjusted after the deletion process in proportion to their probabilities, so that the overall probability is 100%. The advantage of this method is that the game result no longer depends on individual decisions at the point level, but can be corrected retrospectively by key events. In a particularly preferred embodiment, the game sequence is only recorded as "incorrect" by the system if the majority of individual decisions are incorrect.A majority means over 50% of the decisions, preferably over 60% of the decisions, and most preferably over 70%.

[0048] In one embodiment, with regard to the detection of the playing field, the at least one person, the at least one object, the curves and events on the playing field, as well as backward scoring, the automatic tracking method captures precise scores, from which associated statistics, such as error, winner, ace, or double fault, are recognized and can be linked to the player's position and the type of shot. In one embodiment of the invention, the method comprises commands and the following steps: a) determining an event A; b) comparing the event with the player's position and / or the type of shot; c) transmitting the data to an imaging device.

[0049] In one embodiment of the present invention, all image information data is extracted from a single video image, meaning that each image information point is captured by a single camera. A three-dimensional model is not necessary for this. To reliably recognize the diverse events, such as shots in tennis, various images and / or perspectives are usually required to determine a specific type, such as forehand, backhand, serve, volley, topspin, flat, or slice. To circumvent this requirement, one embodiment creates a database of images of already categorized events, such as tennis shots, from a camera with a predetermined position. The image points and data obtained from the predetermined positioning, along with the corresponding shots, are then categorized in a subsequent step.The data generated is used to create a so-called Convolutional Neural Network (CNN), i.e., an artificial neural network. In this CNN, input to the network comes in the form of pixels stored at a specific point within a particular neuron. These artificially generated neurons are interconnected to form an artificial neural network, enabling them to exchange messages. The connections between the neurons and / or networks have a numerical weight that is adjusted during the training process, ensuring that a properly trained network responds correctly to a recognized image or pattern. See, for example, Bengio, Y. & Lecun, Yann. (1997). Convolutional Networks for Images, Speech, and Time-Series, as described previously.

[0050] Using the generated data and simple recognition via a single image or video from a single camera position, networked playing fields can be created in a very short time, allowing players to receive match statistics, point-by-point video analysis, and / or personal coaching based on the actual course of the game. To accurately reflect the course of the game, as previously described, decisions depend on the player's behavior and the corresponding object or ball, and are not simply calculated by a system. The described system is therefore capable of generating accurate, point-based statistics, even if the player's and the system's decisions initially differ.

[0051] In one embodiment of the present invention, the generated data is transferred to one or more devices, which may include, but are not limited to, smartphones, tablets, screens, monitors, televisions, and / or computers. To create individualized training, in a preferred embodiment of the present invention, each playing field has specific codes that can be scanned, for example, using a smartphone or a corresponding application (app) on the device. The player can then log in to the field, use it, and have their data analyzed. In one embodiment, the data is uploaded to an external storage device after the match, where the player can view it.This storage can be a cloud-based service, a network, a smartphone, a computer, or similar devices or combinations suitable for storing and reproducing the data.

[0052] The data for the method described above are generated using a device comprising an image acquisition system. This image acquisition system consists of at least one camera positioned to capture the entire playing field. This allows image data of the entire playing field to be generated and determined. In a preferred embodiment, image data is extracted from two cameras, one directed towards a first side of the playing field, for example, the left side, and the second camera directed towards a second side, for example, the right half of the court. The at least one camera is mounted at a predetermined location. This location is preferably the net post, in which the camera is integrated. In a further embodiment, an additional camera, such as a network camera, can be provided. This can, for example, be positioned behind the baseline.This additional video footage can be collected for later analysis by players and coaches.

[0053] The data from at least one camera is transmitted to a processing module and / or a storage device via a transmission module. The transmission module can be wired or wireless. In a preferred embodiment, it is a wired version.

[0054] The processing unit comprises modules that execute the steps of the procedure described above. Therefore, the processing unit includes at least one module for determining the playing field, at least one module for person detection, at least one module for object identification, at least one module for curve detection, at least one module for event identification, and at least one module for event-based result determination.

[0055] The at least one module for determining the playing field defines its position, size, and dimensions. For this purpose, image data from at least one camera is first transferred to the module by a transmission unit. In a further step, a calibration is performed within the field recognition module, whereby individual points in a two-dimensional (2D) image are assigned to a specific point in a two-dimensional (2D) model of the playing field, such as the... Fig. 1 The calibration is preferably carried out using a homography matrix in which at least four points are used that correlate in the image and the model; these are particularly preferably four corner points of the playing field.

[0056] The at least one module for detecting people on the playing field identifies players or other objects on the field, as already described in the procedure. For this purpose, image data from at least one camera is first transferred to the module from a transmission unit. In a further step, at least one background image of an empty playing field is generated. In the following step, a background model is created. For this, at least one image of the empty playing field is first captured. In the next step, this background image, i.e., the empty playing field, is initialized using the well-known "Mixture of Gaussian's Background Subtractor," and a background model is calculated that updates regularly over the course of the procedure. This subtractor is used to separate objects in the foreground and background of an image and to perform a shape check.Each newly acquired image is separated by calculating its absolute difference with the background model. The generated data is used to create a foreground mask. In a preferred embodiment, this mask has the property that all pixels belonging to the foreground have the value 255 and the pixels of the background have the value 0, with pixels where the difference is greater than a defined threshold preferably being designated as the foreground and pixels with a smaller difference being designated as the background.

[0057] The at least one object detection module, such as for balls, on the playing field identifies movements in each captured image. First, image data from a camera must be transmitted to the object detection module via the transmission module or a previous module. In the following steps, difference images are created from successive images, and absolute difference images are calculated from these, in which movement is visible at two points in the difference image. In a preferred embodiment, these two points are a first position where the ball was located in the previous image and a second position where the ball is located in the current image. Subsequently, a first motion mask is generated for the first difference image, and a second motion mask is generated for the second difference image.These are used to determine a final motion mask that combines both movements. Preferably, the conjunction of the individual pixels of both motion masks is performed using a bitwise AND operator.

[0058] The curve detection module, like the previous modules, uses image data from at least one camera, transmitted to it via a previous module or the transmission module. First, potential starting points of a curve are defined. These are combined with all potential objects in the subsequent image, and for each object, it is checked whether an existing curve is continued. Based on this data, the object's position in successive images is determined. In the next step, the two curves are compared, and the curve's shape is determined. The curve is then classified based on this shape, preferably taking into account the object's next curve. In a preferred embodiment, pixels of objects that correlate with pixels of people are not considered.

[0059] The module for identifying events during gameplay compares the position of at least one person and one object with the game field model. For this purpose, the module communicates with one of the preceding modules and / or a memory to which the data determined by one of the preceding modules has been transferred and stored. Using this data, the event identification module compares it with databases to identify the event. The module can communicate with a variety of databases or information storage devices to which it is connected. In one embodiment, pre-trained artificial intelligences are additionally used to classify the events. In a preferred embodiment, this is a convolutional neural network (CNN).

[0060] The event-oriented result acquisition module of the present device corrects the game score calculated during the course of the game by checking the possibilities of all possible game scores against the actual events detected by the previous modules. For this purpose, the modules are directly or indirectly interconnected and / or the data acquisition for the required information is carried out via a memory that saves all generated data from one of the previous modules. In a preferred embodiment, the adjustment of potential game scores is made proportionally to the probabilities, so that the overall probability results in 100%.

[0061] On the computing module, such as a computer, the previously described method, or at least a part thereof, for object tracking on a playing field, comprising field detection, object detection, item detection, curve detection, event detection, and backward scoring, is implemented. Therefore, the present invention further relates to a computer program for carrying out a previously described method when the computer program is executed on a computer.

[0062] When implementing the method, at least some of the steps belonging to the method are carried out by a processor through the execution of instructions. In a further embodiment, instructions or a part of the instructions for executing the described method and / or for implementing the described method in a system can be stored on a non-transitory, computer-readable data carrier.

[0063] The device of the present invention can comprise a plurality of identical and / or different modules. Modules are also referred to as "components" or "functional units." Furthermore, modules and / or components can also be "computer-executed" and / or "computer-implemented." The modules are implemented within a computer system, which typically includes a processor and memory. In general, a module is a component of a system that performs specific operations to implement a particular functionality. Examples of functionalities include receiving measured values ​​(such as image data) or calculating the field using a computation module. However, the modules can also possess other functionalities described in the embodiments of the method above.

[0064] The term "module" here encompasses a tangible entity that is physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a particular way or to perform certain operations described herein. In embodiments where the modules are temporarily configured (e.g., programmed), not every module needs to be configured or instantiated at every moment. For example, a general-purpose processor may be configured to execute different modules at different times. In some embodiments, a processor implements a module by executing instructions that implement at least part of the module's functionality. Optionally, memory may store the instructions (e.g.,(as computer code) which are read and processed by the processor and cause the processor to perform at least some operations involved in implementing the functionality of the module.

[0065] Additionally or alternatively, a memory, which may comprise one or more memory devices, can in one embodiment store data that is read and processed by the processor to implement at least part of the module's functionality. In another embodiment, the memory can comprise one or more hardware elements capable of storing information accessible to a processor. In one embodiment, the memory can be at least partially integrated into the processor, located on the same chip as the processor, and / or be a physical element separate from the processor.

[0066] In one embodiment, the at least one processor executes instructions stored in memory that perform operations involved in implementing the functionality of a specific module. The at least one processor can also operate in such a way that the performance of the relevant operations is supported in a cloud computing environment or as Software-as-a-Service (SaaS). For example, at least some of the operations involved in implementing a module can be performed by a group of computers accessible via a network, such as the internet, and / or via one or more appropriate interfaces, such as application programming interfaces (APIs). Optionally, some of the modules can be executed in a distributed manner across multiple processors.The at least one processor can be located at a single geographical location or distributed across multiple geographical locations. Optionally, some modules can include the execution of instructions on devices belonging to the users and / or located near the players or spectators.

[0067] For example, processes that include, for instance, the presentation of results, can be partially or completely executed on processors belonging to the players' devices. These devices include, for example, laptops, tablets, and smartphones; however, this list is not limited to those mentioned and can include any known device. Furthermore, in one embodiment, data can be uploaded to cloud-based servers. In some embodiments, modules can provide information to other modules and / or receive information from other modules. Accordingly, such modules can be considered communicatively coupled. If several such modules are present simultaneously, communication can be achieved through signal transmission.In embodiments where modules are configured or instantiated at different times, communication between such modules can be achieved, for example, by storing and retrieving information in memory structures that multiple modules can access. In one embodiment, a module can perform an operation and store the output of that operation on a storage device with which it is communicatively coupled. Another module can then access the storage device at a later time to retrieve and process the stored output.

[0068] In this context, the present invention further relates to a computer program product comprising a storage medium on which a computer program is stored that includes the previously described game tracking method, comprising field detection, person detection, object detection, curve detection, event detection, and backward scoring. In a preferred embodiment, the computer product is loaded directly into the internal memory of a digital computer and comprises software code sections that execute the game tracking steps when the product is running on a computer.The term computer program product here encompasses a computer program stored on a medium, such as RAM, ROM, CD, devices and similar equipment; an embedded system as a comprehensive system with a computer program, such as an electronic device with a computer program; a network of computer-implemented computer programs, such as server systems, client systems, cloud computing systems and the like; and / or computers on which a computer program is loaded, running, stored, executed or being developed.

[0069] Although the methods disclosed herein may be described and presented with reference to certain steps performed in a specific sequence, it is understood that these steps may be combined, subdivided, and / or rearranged to form an equivalent method without deviation from the teachings of the embodiments. Accordingly, unless expressly stated herein, the sequence and grouping of the steps do not constitute a limitation of the embodiments. Furthermore, for the sake of clarity, the methods and mechanisms of the embodiments are in some cases described in the singular. However, unless otherwise stated, some embodiments may include multiple iterations of a method or multiple instantiations of a mechanism.For example, if a processor is disclosed in one embodiment, the scope of that embodiment should also cover the use of multiple processors. Certain features of the embodiments, which may have been described in the context of separate embodiments for clarity, may also be provided in various combinations within a single embodiment. Conversely, various features of the embodiments, which may have been described in the context of a single embodiment for reasons of space, may also be provided separately or in any suitable subcombination.

[0070] In another embodiment, the methods and programs can be executed with various computer system configurations. These computer systems include, but are not limited to, cloud computing, client-server models, grid computing, peer-to-peer, handheld devices, multiprocessor systems, microprocessor-based systems, programmable consumer electronics, minicomputers, and / or mainframes. Additionally or alternatively, some of the embodiments can be performed in a distributed computing environment where the tasks are carried out by remote processing devices connected via a communication network. In a distributed computing environment, program components can be located on both local and remote computers and / or storage devices.Additionally or alternatively, some of the implementations can be carried out in the form of a service such as Infrastructure-as-a-Service (IaaS), Platform-as-a-Service (PaaS), Software-as-a-Service (SaaS) and / or Network-as-a-Service (NaaS).

[0071] These and other embodiments of the present invention are disclosed in and encompassed by the description and examples. The features mentioned in the preceding description and in the claims can be combined in any combination, and combinations of features that are meaningful within the scope of the invention are to be considered disclosed. Further literature on any of the known materials, processes, and applications that can be used in accordance with the present invention can be accessed from public libraries and databases, for example, using electronic devices. A more complete understanding of the invention can be obtained by referring to the figures and examples provided for illustrative purposes, which are not intended to limit the scope of the invention. Examples Example: Program flow for backward scoring using tennis as an example

[0072] In Fig. 3This illustrates a program flow for backward scoring using a tennis match as an example. First, a list of detected events is provided (101), which was previously generated through data acquisition. In the next step (102), an empty scoring tree is initialized. The individual events are processed within this tree (103). This is followed by the determination of a rally (104). If a rally occurred, all potential scores resulting from that rally are determined in the following step (105). The scoring tree is then updated according to the corresponding score, and new "leaves" are added (106).

[0073] If no rally occurred, it is determined whether a rally end can be established (107). If a rally ended, the probability of the rally's outcome is calculated in the following step (108). The calculated probability is recorded on the leaves of the scoring tree (109), and the overall probability for each score is updated (110).

[0074] If no rally occurred, it is determined whether the game has ended (111). If an end is detected, all scores with incomplete games are removed from the scoring tree (112). In the next step, the change of ends is identified (113). If a change of ends occurred, all scores with an even number of games are removed from the scoring tree (114). However, if no change of ends is detected, all scores with an odd number of games are removed from the scoring tree (115).

[0075] If no end to the game has been determined, it is checked whether the match is over (116). If so, all scores with incomplete games (117) and sets (118) are removed from the scoring tree. In the final step, the score with the highest probability of being the final score is determined (119) and displayed as the game score (120).

Claims

1. Method for game monitoring, comprising the following steps: a) determination of the playing field based on image data from an image recording system, by a computing unit; b) recognition of at least one person on the playing field based on the image data from the image recording system, by the computing unit; c) identification of objects on the playing field based on the image data from the image recording system, by the computing unit; d) determination of at least one flight curve of an identified object based on the image data from the image recording system, by the computing unit; characterised in that the computing unit, using the flight curve, identifies at least one event by means of which the scores calculated in the course of the game are corrected by means of backward scoring, event-based result determination, and the real score is output; the correction of the score calculated in the course of the game comprising the following steps: a) monitoring the possible scores during the game whereby a results tree is generated on the basis of data from events in the game and the course of the game, which, after completion of the game, contains the potential scores of the game, each score having a probability describing its plausibility; b) checking the compatibility of all possible scores with the detected events, whereby checks are performed to ascertain whether a score is possible or not against the background of the detected events; c) checking the plausibility of the remaining scores on the basis of the probabilities.

2. Method according to claim 1, wherein the determination of the playing field is carried out by means of calibration, wherein individual points in a two-dimensional (2D) image are assigned to a specific point in a two-dimensional (2D) model of the playing field, wherein the calibration is preferably carried out via a homography matrix in which at least four points are used which correlate in the image and the model, particularly preferably four corner points of the playing field.

3. Method according to claim 1 or 2, wherein the detection of persons on the playing field comprises the following steps: a) generation of at least one background image of an empty playing field; b) generation of a background model; c) separation of each background image from each currently recorded image that comprises a person, and in which an object has thus been classified as a human on the basis of height and / or width and / or position in the image; d) determination of an absolute difference between the background model and the currently recorded image, and e) generation a foreground mask, the foreground mask being characterised in that all pixels belonging to the foreground have the value 255 and the pixels of the background have the value 0, wherein preferably pixels in which the difference is greater than a fixed limit value are determined as the foreground and pixels with a smaller difference are determined as the background.

4. Method according to any of claims 1 to 3, wherein the object is identified on the playing field by movements of each recorded image, preferably wherein the determination of the movement comprises the following steps: a) creation of differential images of successive images; b) calculation of absolute difference images in which a movement is visible at two points in the difference image, preferably a first position in which the ball was located in the previous image and a second position in which the ball is located in the current image; c) generation of a first movement mask for the first difference image and a second movement mask for the second difference image; and d) generation of a final movement mask comprising the movements occurring in the first movement mask and the second movement mask; preferably, the individual pixels of both movement masks are conjugated with a bitwise AND operator.

5. Method according to any of claims 1 to 4, wherein the flight path or flight curve of the object is determined, comprising the following steps: a) setting a first and second starting point of a potential curve; b) combining the potentially set starting points with all potential objects of the next image to follow; c) checking each object to ascertain whether an existing curve is continued; d) determining positions of the object on consecutive images; e) comparing the curves from a) and d); f) determining a curve progression; and g) classifying the curve by determining a second curve following the first curve, wherein preferably image points of objects that correlate with image points of persons are not taken into account.

6. Method according to any of claims 1 to 5, wherein the identification of events in the course of the game comprises the following steps: a) capture of a triggering event; b) comparison of the position of at least one person and one object with the playing field model; c) reconciliation with a database, wherein the events have preferably been captured in a categorised data set in a previous step in order to carry out the reconciliation using an artificial intelligence trained with this data set.

7. Method according to any of claims 1 to 6, wherein potential scores that do not meet the conditions of an event on the playing field are deleted from the events tree, and the further potential scores after the deletion process are adjusted in relation to the probabilities, so that the total probability is 100%.

8. Device for monitoring at least one object and player on a playing field, comprising: a) an image recording system, comprising i) at least one camera arranged in such a way that the entire playing field is captured; ii) at least one transmission module designed to store the image data recorded by the camera and / or to transmit it to a computing unit, a memory, or system; wherein the at least one camera and the at least one transmission module are attached to at least one net post; b) a computing unit, comprising modules for iii) playing field determination based on the image data; iv) person recognition based on the image data; v) object determination based on the image data; vi) curve recognition based on the object determination; vii) event identification based on the curve recognition; wherein the computing unit is characterised by a module for event-based result determination, which is designed to perform the following steps when correcting the score calculated in the course of the game: 1) monitoring the possible scores during the game whereby a results tree is generated on the basis of data from events in the game and the course of the game, which, after completion of the game, contains the potential scores of the game, each score having a probability describing its plausibility; 2) checking the compatibility of all possible scores with the detected events, whereby checks are performed to ascertain whether a score is possible or not against the background of the detected events; 3) checking the plausibility of the remaining scores on the basis of the probabilities.

9. Computer program for performing a method according to any of claims 1 to 7 when said computer program is run on a computer.

10. Computer program product a) with a storage medium on which a computer program according to claim 9 is stored; and / or b) that can be loaded directly into the internal memory of a digital computer and comprises software code sections with which the steps according to claims 1 to 7 are performed when the product runs on a computer.