Method and system for extracting athlete data in match video, electronic equipment and medium

Abstract

The embodiment of the invention provides a method and system for extracting athlete data in a match video, electronic equipment and a medium, and belongs to the field of computer vision and sports data analysis. The method comprises the following steps: acquiring a match video, performing target detection on athletes in the video, and determining a target athlete; tracking the target athlete by using a DeepSORT algorithm to obtain the appearance description of the target athlete according with the motion information; and recognizing the action of the target athlete by using a SlowFast algorithm to obtain the movement behavior data of the target athlete. According to the method, the tracks and the positions of the athletes can be obtained in real time through video positioning and tracking in combination with DeepSORT and SlowFast algorithms, the efficiency is high, and accurate data support can be provided for sports competition analysis, tactical formulation and athlete training by recognizing the positions of the athletes on the ground and analyzing motion behavior data between the athletes.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and sports data analysis, and in particular to a method, system, electronic equipment and medium for extracting athlete data from a game video. Background Art

[0002] Extracting player data from match recordings is the most intuitive way to analyze football matches. Through the video, each event in the match is manually marked in detail, including the initiator, recipient, time point, coordinates, event, and event attributes. After the match event is marked, a targeted match report is issued through data analysis to help managers and coaches analyze the match and help fans objectively understand the current status of the team.

[0003] Currently, the existing methods mainly extract data through manual video labeling. This method is highly subjective, does not cover all content, and is costly and inefficient. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide a method, system, electronic device and medium for extracting athlete data from a game video, which are used to completely or at least partially solve the technical problems existing in the above-mentioned prior art.

[0005] In order to achieve the above object, an embodiment of the present invention provides a method for extracting athlete data from a game video, comprising: Obtaining a video of a match, and performing target detection on athletes in the video of the match to determine a target athlete; Tracking the target athlete using the DeepSORT algorithm to obtain an appearance description of the target athlete that matches the motion information; The SlowFast algorithm is used to identify the target athlete's movements and obtain the target athlete's movement behavior data, wherein the SlowFast algorithm includes a Slow path for capturing spatial semantics and a Fast path for capturing temporal resolution motion information.

[0006] Optionally, obtaining a video of a match, and performing target detection on athletes in the video of the match to determine a target athlete includes: Use the YOLOv8 algorithm to detect the athletes in the game video and obtain the bounding box of each target: Collect and annotate the image data of each video frame in the game video, and organize it into the format required by the YOLOv8 algorithm; The YOLOv8 model is trained using the image data of each video frame in the annotated game video, so that the YOLOv8 model outputs the category, position and confidence information of the target athlete; The image data of each video frame in the game video is input into the YOLOv8 model to generate the bounding box of each athlete. The IOU threshold is set to measure the degree of overlap of the bounding boxes, and whether to delete the bounding box is determined based on the degree of overlap. Each bounding box contains the location information and confidence score of the target athlete.

[0007] Optionally, the target athlete is tracked using a DeepSORT algorithm to obtain an appearance description of the target athlete that matches the motion information, including: Use the YOLOv8 algorithm to detect the athletes in the game video and obtain the bounding box of each target; Extract features in each target bounding box through a pre-trained CNN convolutional neural network; wherein the features include but are not limited to appearance features and motion features of the target athlete; The Kalman filter is used to predict the state of the target athlete in the current frame based on the motion state of the target athlete in the previous frame. For the successfully matched target, its state is updated using the update equation of the Kalman filter. For the newly appeared target, the Kalman filter is initialized and added to the tracking list. For the disappeared target, it is removed from the tracking list.

[0008] Optionally, the update equation of the Kalman filter is:

[0009] In the formula, express Time information pair The prediction of the state at the moment, A represents the state transfer matrix, express The optimal estimate at the moment, B represents the control input matrix, express Control input at any time.

[0010] Optionally, the SlowFast algorithm is used to identify the target athlete's movements to obtain the target athlete's movement behavior data, including: Inputting a video of a match containing a sports scene of an athlete into a SlowFast model with a dual-path structure, wherein the SlowFast model with a dual-path structure includes a slow path and a fast path, wherein the slow path captures spatial information in the video at a low frame rate, including but not limited to the static features of the athlete's body posture and position; and the fast path captures temporal information at a high frame rate, including but not limited to the changes and dynamic features of the athlete's movements; The feature maps extracted from the slow and fast paths are fused by element-by-element addition or concatenation to combine spatial and temporal information to describe the athlete’s movements; The fused feature maps are classified to identify specific action categories, where the action categories include but are not limited to running, jumping, shooting, and defensive actions.

[0011] On the other hand, the present invention also provides a system for extracting athlete data from a game video, comprising: An acquisition unit is used to acquire a video of a match, and to perform target detection on athletes in the video of the match to determine a target athlete; A tracking unit, used to track the target athlete using a DeepSORT algorithm to obtain an appearance description of the target athlete that matches the motion information; The recognition unit is used to recognize the movements of the target athlete using the SlowFast algorithm to obtain the movement behavior data of the target athlete, wherein the SlowFast algorithm includes a Slow path for capturing spatial semantics and a Fast path for capturing temporal resolution movement information.

[0012] Optionally, the acquiring unit is specifically used for: Use the YOLOv8 algorithm to detect the athletes in the game video and obtain the bounding box of each target: Collect and annotate the image data of each video frame in the game video, and organize it into the format required by the YOLOv8 algorithm; The YOLOv8 model is trained using the image data of each video frame in the annotated game video, so that the YOLOv8 model outputs the category, position and confidence information of the target athlete; The image data of each video frame in the game video is input into the YOLOv8 model to generate the bounding box of each athlete. The IOU threshold is set to measure the degree of overlap of the bounding boxes, and whether to delete the bounding box is determined based on the degree of overlap. Each bounding box contains the location information and confidence score of the target athlete.

[0013] Optionally, the tracking unit is specifically used for: Use the YOLOv8 algorithm to detect the athletes in the game video and obtain the bounding box of each target; Extract features in each target bounding box through a pre-trained CNN convolutional neural network; wherein the features include but are not limited to appearance features and motion features of the target athlete; The Kalman filter is used to predict the state of the target athlete in the current frame based on the motion state of the target athlete in the previous frame. For the successfully matched target, its state is updated using the update equation of the Kalman filter. For the newly appeared target, the Kalman filter is initialized and added to the tracking list. For the disappeared target, it is removed from the tracking list.

[0014] On the other hand, the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned method for extracting athlete data from a game video when executing the program.

[0015] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-mentioned method for extracting athlete data from a game video.

[0016] Through the above technical solution, combined with the video positioning tracking of DeepSORT and SlowFast algorithms, the trajectory and position of athletes can be obtained in real time with high efficiency. In addition, by identifying the positions of athletes on the field and analyzing the sports behavior data between athletes, accurate data support can be provided for sports game analysis, tactical formulation and athlete training.

[0017] Other features and advantages of the embodiments of the present invention will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present invention, but do not constitute a limitation on the embodiments of the present invention. In the accompanying drawings: Figure 1 It is a flow chart of an implementation of a method for extracting athlete data from a game video provided by an embodiment of the present invention; Figure 2 The present invention is a schematic diagram of a system for extracting athlete data from a video game provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The specific implementation of the embodiment of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the embodiment of the present invention, and is not used to limit the embodiment of the present invention.

[0020] See also Figure 1 FIG. 1 is a flowchart of an implementation method for extracting athlete data from a game video provided by an embodiment of the present invention, comprising the following execution steps: Step 100: Obtain a video of a match, and perform target detection on athletes in the video of the match to determine a target athlete.

[0021] Specifically, when executing step 100, the following steps may be specifically performed: The YOLOv8 algorithm is used to detect the athletes in the game video and obtain the bounding box of each target. The specific steps include the following: S1: Collect and annotate the image data of each video frame in the game video, and organize it into the format required by the YOLOv8 algorithm.

[0022] S2: Train the YOLOv8 model using the image data of each video frame in the annotated game video, so that the YOLOv8 model outputs the category, position and confidence information of the target athlete.

[0023] S3: Input the image data of each video frame in the game video into the YOLOv8 model to generate a bounding box for each athlete. The IOU threshold is set to measure the degree of overlap of the bounding boxes, and whether to delete the bounding box is determined based on the degree of overlap. Each bounding box contains the location information and confidence score of the target athlete.

[0024] In some embodiments, non-maximum suppression is applied to remove overlapping bounding boxes. During the target detection process, the model generates multiple bounding boxes (Bounding Boxes), each of which contains the location information of the object (such as coordinates) and a confidence score (i.e., the probability of the object's existence). Set the IOU threshold: IOU (Intersection over Union) is an indicator that measures the degree of overlap between two bounding boxes, and its value range is between 0 and 1. An IOU threshold is preset. If the threshold is greater than 0.7, the bounding box is judged to be redundant and needs to be suppressed. After deleting the redundant bounding boxes, if there are still bounding boxes in the list, select the next bounding box with the highest confidence as the new reference bounding box, and repeat the steps until all bounding boxes in the list have been processed.

[0025] Step 101: Track the target athlete using the DeepSORT algorithm to obtain an appearance description of the target athlete that matches the motion information.

[0026] Specifically, when executing step 101, the following steps may be specifically performed: S1010: Use the YOLOv8 algorithm to detect the athletes in the game video and obtain the bounding box of each target.

[0027] S1011: Extract features in each target bounding box through a pre-trained CNN convolutional neural network; wherein the features include but are not limited to appearance features and motion features of the target athlete.

[0028] S1012: Use the Kalman filter to predict the state of the target athlete in the current frame based on the motion state of the target athlete in the previous frame. For the successfully matched target, use the update equation of the Kalman filter to update its state. For the newly appeared target, initialize the Kalman filter and add it to the tracking list. For the disappeared target, remove it from the tracking list.

[0029] The updated status includes but is not limited to position and speed information.

[0030] In some embodiments, the update equation of the Kalman filter is:

[0031] In the formula, express Time information pair The prediction of the state at the moment, A represents the state transfer matrix, express The optimal estimate at the moment, B represents the control input matrix, express Control input at any time.

[0032] Step 102: using the SlowFast algorithm to identify the target athlete's movements and obtain the target athlete's movement behavior data, wherein the SlowFast algorithm includes a Slow path for capturing spatial semantics and a Fast path for capturing temporal resolution movement information.

[0033] Specifically, when executing step 102, the following steps may be specifically performed: S1020: Inputting a video of a game recording of a sports scene containing athletes into a SlowFast model with a dual-path structure, wherein the SlowFast model with a dual-path structure includes a slow path and a fast path, the slow path captures spatial information in the video at a low frame rate, including but not limited to the athlete's body posture and position static features; the fast path captures temporal information at a high frame rate, including but not limited to the changes and dynamic features of the athlete's movements.

[0034] S1021: The feature maps extracted from the slow path and the fast path are fused by element-by-element addition or concatenation, combining spatial and temporal information to describe the athlete's movements.

[0035] In some embodiments, the accuracy of action recognition is improved by feature fusion optimization: at the end of the model, the feature maps extracted by the slow branch and the fast branch are fused. The fusion method usually includes sending the features of the fast branch to the slow branch through lateral connections for mixing. Data transformation: Since the feature dimensions of the two branches are inconsistent (the fast branch is {αT, S², βC}, and the slow branch is {T, S², αβC}), the results of the fast branch need to be transformed to match the feature dimensions of the slow branch. Commonly used data transformation techniques include Time-to-channel, Time-strided sampling, and Time-strided convolution.

[0036] S1022: Classify the fused feature map to identify specific action categories, where the action categories include but are not limited to running, jumping, shooting, and defensive actions.

[0037] In some implementations, in order to improve the recognition accuracy of the SlowFast model, the training strategy is optimized by adjusting the parameters of the learning rate, optimizer, and loss function, and data enhancement, regularization, and other techniques are used to prevent overfitting. After the model training is completed, the player data is exported.

[0038]

[0039] In the formula, n represents the number of samples, Indicates The true value of the samples, Indicates The predicted value of a sample.

[0040] By combining video positioning and tracking using YOLOv8, DeepSORT, and SlowFast algorithms, we can obtain personnel trajectories and position coordinates in real time, identify the positions of athletes on the field, and analyze passing data and position data between athletes, providing accurate data support for sports game analysis, tactical formulation, and athlete training.

[0041] See also Figure 2 FIG. 1 is a schematic diagram of a system for extracting athlete data from a game video provided by an embodiment of the present invention, comprising: The acquisition unit 20 is used to acquire the game video and perform target detection on the athletes in the game video to determine the target athletes; A tracking unit 21, used to track the target athlete using a DeepSORT algorithm to obtain an appearance description of the target athlete that matches the motion information; The recognition unit 22 is used to recognize the movement of the target athlete by using the SlowFast algorithm to obtain the movement behavior data of the target athlete, wherein the SlowFast algorithm includes a Slow path for capturing spatial semantics and a Fast path for capturing temporal resolution movement information.

[0042] In some implementations, the acquisition unit is specifically configured to: Use the YOLOv8 algorithm to detect the athletes in the game video and obtain the bounding box of each target: Collect and annotate the image data of each video frame in the game video, and organize it into the format required by the YOLOv8 algorithm; The YOLOv8 model is trained using the image data of each video frame in the annotated game video, so that the YOLOv8 model outputs the category, position and confidence information of the target athlete; The image data of each video frame in the game video is input into the YOLOv8 model to generate the bounding box of each athlete. The IOU threshold is set to measure the degree of overlap of the bounding boxes, and whether to delete the bounding box is determined based on the degree of overlap. Each bounding box contains the location information and confidence score of the target athlete.

[0043] In some embodiments, the tracking unit is specifically used to: Use the YOLOv8 algorithm to detect the athletes in the game video and obtain the bounding box of each target; Extract features in each target bounding box through a pre-trained CNN convolutional neural network; wherein the features include but are not limited to appearance features and motion features of the target athlete; The Kalman filter is used to predict the state of the target athlete in the current frame based on the motion state of the target athlete in the previous frame. For the successfully matched target, its state is updated using the update equation of the Kalman filter. For the newly appeared target, the Kalman filter is initialized and added to the tracking list. For the disappeared target, it is removed from the tracking list.

[0044] On the other hand, the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the method for extracting athlete data from a game video as described in any one of the above-mentioned embodiments are implemented.

[0045] On the other hand, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method for extracting athlete data from a game video as described in any of the above embodiments.

[0046] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0047] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0048] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0049] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0050] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0051] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0052] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0053] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0054] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for extracting athlete data from a game video, characterized in that: include: Obtaining a video of a match, and performing target detection on athletes in the video of the match to determine a target athlete; Tracking the target athlete using the DeepSORT algorithm to obtain an appearance description of the target athlete that matches the motion information; The SlowFast algorithm is used to identify the target athlete's movements and obtain the target athlete's movement behavior data, wherein the SlowFast algorithm includes a Slow path for capturing spatial semantics and a Fast path for capturing temporal resolution motion information.

2. The method for extracting athlete data from a game video according to claim 1, characterized in that: Obtain the game video, and perform target detection on the athletes in the game video to determine the target athletes, including: Use the YOLOv8 algorithm to detect the athletes in the game video and obtain the bounding box of each target: Collect and annotate the image data of each video frame in the game video, and organize it into the format required by the YOLOv8 algorithm; The YOLOv8 model is trained using the image data of each video frame in the annotated game video, so that the YOLOv8 model outputs the category, position and confidence information of the target athlete; The image data of each video frame in the game video is input into the YOLOv8 model to generate the bounding box of each athlete. The IOU threshold is set to measure the degree of overlap of the bounding boxes, and whether to delete the bounding box is determined based on the degree of overlap. Each bounding box contains the location information and confidence score of the target athlete.

3. The method for extracting athlete data from a game video according to claim 1, characterized in that: The target athlete is tracked using the DeepSORT algorithm to obtain an appearance description of the target athlete that matches the motion information, including: Use the YOLOv8 algorithm to detect the athletes in the game video and obtain the bounding box of each target; Extract features in each target bounding box through a pre-trained CNN convolutional neural network; wherein the features include but are not limited to appearance features and motion features of the target athlete; The Kalman filter is used to predict the state of the target athlete in the current frame based on the motion state of the target athlete in the previous frame. For the successfully matched target, its state is updated using the update equation of the Kalman filter. For the newly appeared target, the Kalman filter is initialized and added to the tracking list. For the disappeared target, it is removed from the tracking list.

4. The method for extracting athlete data from a game video according to claim 3, characterized in that: The update equation of the Kalman filter is: In the formula, express Time information pair The prediction of the state at the moment, A represents the state transfer matrix, express The optimal estimate at the moment, B represents the control input matrix, express Control input at any time.

5. The method for extracting athlete data from a game video according to claim 1, characterized in that: The SlowFast algorithm is used to identify the target athlete's movements and obtain the target athlete's sports behavior data, including: Inputting a video of a game containing a sports scene of an athlete into a SlowFast model with a dual-path structure, wherein the SlowFast model with a dual-path structure includes a slow path and a fast path, wherein the slow path captures spatial information in the video at a low frame rate, including but not limited to the static features of the athlete's body posture and position; and the fast path captures temporal information at a high frame rate, including but not limited to the changes and dynamic features of the athlete's movements; The feature maps extracted from the slow and fast paths are fused by element-by-element addition or concatenation to combine spatial and temporal information to describe the athlete’s movements; The fused feature maps are classified to identify specific action categories, where the action categories include but are not limited to running, jumping, shooting, and defensive actions.

6. A system for extracting athlete data from a game video, characterized in that: include: An acquisition unit is used to acquire a video of a match, and perform target detection on athletes in the video of the match to determine a target athlete; A tracking unit, used to track the target athlete using a DeepSORT algorithm to obtain an appearance description of the target athlete that matches the motion information; The recognition unit is used to recognize the movements of the target athlete using the SlowFast algorithm to obtain the movement behavior data of the target athlete, wherein the SlowFast algorithm includes a Slow path for capturing spatial semantics and a Fast path for capturing temporal resolution movement information.

7. The system for extracting athlete data from a game video according to claim 6, characterized in that: The acquisition unit is specifically used for: Use the YOLOv8 algorithm to detect the athletes in the game video and obtain the bounding box of each target: Collect and annotate the image data of each video frame in the game video, and organize it into the format required by the YOLOv8 algorithm; The YOLOv8 model is trained using the image data of each video frame in the annotated game video, so that the YOLOv8 model outputs the category, position and confidence information of the target athlete; The image data of each video frame in the game video is input into the YOLOv8 model to generate the bounding box of each athlete. The IOU threshold is set to measure the degree of overlap of the bounding boxes, and whether to delete the bounding box is determined based on the degree of overlap. Each bounding box contains the location information and confidence score of the target athlete.

8. The system for extracting athlete data from a game video according to claim 6, characterized in that: The tracking unit is specifically used for: Use the YOLOv8 algorithm to detect the athletes in the game video and obtain the bounding box of each target; Extract features in each target bounding box through a pre-trained CNN convolutional neural network; wherein the features include but are not limited to appearance features and motion features of the target athlete; The Kalman filter is used to predict the state of the target athlete in the current frame based on the motion state of the target athlete in the previous frame. For the successfully matched target, its state is updated using the update equation of the Kalman filter. For the newly appeared target, the Kalman filter is initialized and added to the tracking list. For the disappeared target, it is removed from the tracking list.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method for extracting athlete data from a game video as described in any one of claims 1 to 5 are implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for extracting athlete data from a game video as described in any one of claims 1 to 5 are implemented.

Images