Tennis ball detection method applied to video stream and based on deep learning

Abstract

The invention discloses a deep learning-based tennis ball detection method applied to a video stream. The method comprises the following steps: S10, a data preprocessing stage: performing data preprocessing on a time sequence video stream to obtain training data; s20, a model training stage: stacking a continuous picture sequence and a background frame image in the training data pair through a stacking module to form network training input, performing multi-layer down-sampling through an encoder structure to obtain a feature vector fused with time sequence information, performing processing through a time attention module to improve the enhancement of a small target object, and performing model training; restoring a corresponding thermodynamic diagram of the input time sequence through a decoder structure; through comparison with a thermodynamic diagram of a real label, a loss value is obtained in a weighted loss calculation module, so that model parameters are solved through back propagation; and S30, a model reasoning stage, wherein after network training is completed, video streams are input into the network for reasoning operation to detect the position of the tennis ball.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a tennis ball detection method based on deep learning applied to video streams. Background Art

[0002] In recent years, tennis has become an increasingly popular competitive sport, attracting a large number of viewers and sponsors, and has extremely high commercial value. The tennis ball detection and tracking technology in tennis can not only provide a scientific basis for event analysis, but also show more detailed sports details in the broadcast and enhance the viewing experience. In addition, the precise detection and tracking of the tennis ball position can directly support functions such as shot speed analysis and landing point prediction, and play an important role in applications such as tactical analysis and referee assistance.

[0003] However, tennis detection and tracking technology still faces many challenges. In long-distance shooting, which is usually the main perspective of the broadcast stream, the tennis ball is very small relative to the background scene and other objects, usually only 6 to 13 pixels, which makes its texture features difficult to be clearly captured by traditional image processing methods; the color of the tennis ball may be very close to the surrounding environment (such as the field, the audience), which increases the complexity of detection. In addition, the speed of tennis in the game is very fast, and motion blur often occurs due to high-speed movement. The tennis ball in the image may appear elliptical or smeared, which further increases the difficulty of detection. In addition, the rapid change of direction of the tennis ball after hitting and the sudden change of the rebound position also make the tracking process more complicated.

[0004] Traditional methods usually construct a significant feature map of tennis balls based on color and brightness differences, combine simple shape, size and other information to preliminarily identify potential tennis ball areas, and then use post-processing steps to eliminate false detection areas. These methods rely on manual feature extraction and rule design. Although they are effective under specific lighting and simple backgrounds, they have a high false detection rate in complex game environments and cannot meet real-time detection requirements. Existing deep learning-based detection methods are helpful in improving detection accuracy, but they are computationally intensive and cannot achieve efficient real-time processing, especially for small and fast-moving tennis balls. The detection effect is not ideal.

[0005] Manual assisted detection is currently used in scenarios such as match replays, but the high labor costs make it difficult to support long-term applications in continuous matches. Therefore, in order to solve the above problems, an efficient and accurate tennis ball detection and tracking method suitable for video streams is urgently needed, which not only has a low false detection rate, but also meets the real-time requirements, so as to better serve the broadcasting and analysis needs of modern tennis matches. Summary of the invention

[0006] In view of the above problems, the present invention provides a tennis ball detection method based on deep learning applied to video streams, which is used to detect the ball position in tennis matches in real time and with high quality.

[0007] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0008] A deep learning based tennis ball detection method applied to a video stream comprises the following steps:

[0009] S10, data preprocessing stage: performing data preprocessing on the time-series video stream to obtain training data, wherein the image in the training data and the annotation file of the tennis ball position in the annotation image are a pair of data, which meet the time-series continuity;

[0010] S20, model training stage: stack the continuous image sequence and background frame image in the training data pair through the stacking module to form the network training input, obtain the feature vector of fused time series information through the encoder structure multi-layer downsampling, and after the enhancement of small target objects is improved through the time attention module, restore the corresponding heat map of the input time series through the decoder structure; by comparing with the heat map of the real label, the loss value is obtained in the weighted loss calculation module, and the model parameters are solved by back propagation;

[0011] S30, model reasoning stage: after completing the network training, the video stream is input into the network for reasoning operation to detect the position of the tennis ball.

[0012] In a possible implementation, the data preprocessing of the time-series video stream to obtain training data in S10 includes: collecting tennis match videos from public domains and authorized domains for video annotation, selecting multiple matches from different periods and different stadiums to produce training data, performing rough screening based on video information, counting the coverage of various events, manually screening, marking the main event time points that meet the viewing angle, and slicing and classifying the video through a computer program; at the same time, performing sample cleaning on the obtained video data, and punctuating the position of the tennis ball in the continuous video sequence; constructing an online data enhancement module, including enhancement of brightness, contrast, screen size, and occlusion mode, constructing a training background image for video training, obtaining a background frame image by directly fusing the pixels of the video sequence, and finally forming a data pair combination of the image sequence and the annotation file sequence for training.

[0013] In a possible implementation manner, the data pair finally formed is:

[0014] <X,Y> =[ <x1,x2,…,x n ,x background >, <y1,y2,…,y n >]

[0015] Among them, x i ,y i are the corresponding image sequence and annotation file sequence, respectively, background The background frame image.

[0016] In one possible implementation, the encoder structure in S20 includes a stacking module, a first feature extraction layer, a second feature extraction layer, a third feature extraction layer and a fourth feature extraction layer connected in sequence, and the data pairs obtained by data preprocessing are combined and stacked through the stacking module. The stacked data passes through the first feature extraction layer, the second feature extraction layer, the third feature extraction layer and the fourth feature extraction layer in sequence, and the data resolution is converted into 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the original image size in sequence, and finally a feature map with high-level feature information is obtained.

[0017] In one possible implementation, the decoder structure includes a first fusion module, a second fusion module, and a third fusion module, each fusion module including an upsampling operation and a feature extraction layer for fusing feature maps from the corresponding encoder feature extraction layer and the output of the preceding module. First, the input map resolution of the preceding module is increased by upsampling and then connected to the corresponding feature map of the encoder. The generated feature map is fused using the feature extraction layer. The feature map enhanced by the temporal attention module passes through the first fusion module, the second fusion module, and the third fusion module in sequence, and is fused with the third feature extraction layer, the second feature extraction layer, and the first feature extraction layer of the encoder, respectively. Finally, a stacked heat map that fuses shallow and deep features of the network is obtained. The heat map is input into a weighted loss calculation module, and the loss value is obtained through a weighted loss function, thereby back-propagating and learning accurate parameters.

[0018] In a possible implementation, the heat map includes the response position of the sphere in the image, and the weighted loss function is used to increase the weight of the sphere response area, so that the network can more accurately focus on and predict the position of the sphere; after processing the heat map of the training module and the heat map generated by the true label value, the absolute error loss function is calculated, and the loss function used in the training is the weighted BCE Loss, and the specific formula is as follows:

[0019]

[0020] y and The predicted heat map value and the real label heat map value are identified respectively. w is the adjustment weight, which can be set to y. The optimization goal is to solve the model parameter θ to minimize the loss function, that is:

[0021]

[0022] Where θ represents the model parameters to be optimized, M is the number of training samples, and BCE Loss is the weighted loss function. Finally, the training of the entire model is completed.

[0023] In a possible implementation, after completing the network training in S30, the video stream is input into the network for inference operation to detect the tennis position. Specifically, the sliding window is used to perform inference on consecutive frame images. In a window of inference, the detection confidence weight of the image at the center is the highest, and decreases to both sides of the window in sequence. In this way, when the sliding distance is less than the window length, the image will have multiple tennis position detections, and the final detection value is obtained based on the weight. The window size is 2k+1, with k frames on the left and right of the center frame, and the final tennis detection value S final The calculation formula is expressed as:

[0024]

[0025] w j represents the weight of the jth frame in the window, s j represents the confidence of the tennis ball position detection in the jth frame. ,The weight formula is:

[0026]

[0027] σ is a hyperparameter that controls how fast the weights decay, in such a way that the frame at the center contributes the most to the confidence in the window, and the influence of the frames on both sides gradually decreases.

[0028] The present invention has the following beneficial effects:

[0029] (1) Different from the single-frame detection model of other algorithms, this method is based on temporal information and can achieve more stable detection of single-frame blur or occlusion in fast motion.

[0030] (2) By combining image and temporal context information, this method is more robust to background noise and interference from similar objects than existing algorithms. It can achieve stable and effective detection of spheres in non-green backgrounds, which is a pain point of traditional algorithms.

[0031] (3) This method integrates detection and tracking information, reduces the post-processing process, and is beneficial to the real-time operation of downstream algorithms and engineering maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A flowchart of a tennis ball detection method based on deep learning applied to a video stream according to an embodiment of the present invention;

[0033] Figure 2 The present invention is a block diagram of a training module device in a tennis ball detection method based on deep learning applied to a video stream according to an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] See also Figure 1 , which is a flowchart of a tennis ball detection method based on deep learning applied to a video stream according to an embodiment of the present invention, comprising the following steps:

[0036] S10, data preprocessing stage: performing data preprocessing on the time series video stream to obtain training data, wherein the image in the training data and the annotation file of the tennis ball position in the annotation image are a pair of data, which meet the time series continuity;

[0037] S20, model training stage: stack the continuous image sequence and background frame image in the training data pair through the stacking module to form the network training input, obtain the feature vector of fused time series information through the encoder structure multi-layer downsampling, and after the enhancement of small target objects is improved through the time attention module, restore the corresponding heat map of the input time series through the decoder structure; by comparing with the heat map of the real label, the loss value is obtained in the weighted loss calculation module, and the model parameters are solved by back propagation;

[0038] S30, model reasoning stage: after completing the network training, the video stream is input into the network for reasoning operation to detect the position of the tennis ball.

[0039] In a specific application example, the data preprocessing of the time-series video stream in S10 to obtain training data includes: collecting tennis event videos from the public domain and authorized domain for video annotation. In order to ensure the diversity of samples, multiple games in different periods and different stadiums are selected to produce training data to ensure the generalization performance of the model. Based on the video information, a rough screening is performed, and the coverage of various events is counted. Through manual screening, the main event time points that meet the perspective are marked, and the video is sliced ​​and classified through a computer program; at the same time, the obtained video data is sample cleaned, and the video is blurred, low in clarity, too small in pixels, irrelevant videos, etc. are cleaned, and the position of the tennis ball is punctuated in the continuous video sequence; an online data enhancement module is constructed, including enhancement of brightness, contrast, screen size, and occlusion mode, and a training background image is constructed for video training. The background frame image is obtained by directly fusing the video sequence by extracting frames, and the typical value is 15 frames. Finally, a data pair combination of image sequence and annotation file sequence is formed for training. The data pair finally formed is:

[0040] <X,Y>=[ <x1,x2,…,x n ,x background >, <y1,y2,…,y n >]

[0041] Among them, x i ,y i are the corresponding image sequence and annotation file sequence, respectively, background The background frame image.

[0042] Figure 2 It is a principle block diagram of the model training device. Different from the traditional mode of single-frame detection plus tracking algorithm of the traditional neural network method, the embodiment of the present invention locates the position of the tennis ball by detecting a continuous video frame sequence, aiming to use background modeling information to filter static misdetected objects and the information of the previous and next frames to locate the specific position of the ball, with excellent recall performance and accuracy. The structure of the encoder in S20 includes a stacking module, a first feature extraction layer, a second feature extraction layer, a third feature extraction layer and a fourth feature extraction layer connected in sequence, and the data pairs obtained by data preprocessing are combined and stacked through the stacking module, and the stacked data passes through the first feature extraction layer, the second feature extraction layer, the third feature extraction layer and the fourth feature extraction layer in sequence, and the data resolution is converted to 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the original image size in sequence, and finally a feature map with high-level feature information is obtained. Furthermore, the structure of the decoder includes a first fusion module, a second fusion module and a third fusion module, each fusion module includes an upsampling operation and a feature extraction layer, which is used to fuse the feature maps from the corresponding encoder feature extraction layer and the output of the previous module. First, the input map resolution of the previous module is increased by upsampling and connected with the corresponding feature map of the encoder, and the generated feature map is fused using the feature extraction layer. The feature map enhanced by the temporal attention module passes through the first fusion module, the second fusion module and the third fusion module in turn, and is fused with the third feature extraction layer, the second feature extraction layer and the first feature extraction layer of the encoder respectively. Finally, a stacked heat map that fuses the shallow and deep features of the network is obtained. The heat map is input into the weighted loss calculation module, and the loss value is obtained through the weighted loss function, so as to learn accurate parameters through back propagation.

[0043] Specifically, each feature extraction layer consists of a convolution layer, a batch normalization layer, and a nonlinear activation function. The convolution layer is used to extract local features of the input image. The convolution kernel size is 3x3 to obtain finer-grained spatial information. The batch normalization layer is used to standardize the convolution output, which helps to speed up training and stabilize model convergence. Substituting x for the network input, the nonlinear activation function is expressed as:

[0044] ReLU(x)=max(0,x)

[0045] The function of the feature extraction layer is to extract the temporal information and spatial position information of the object so that the network can estimate the response position of the sphere in the continuous image sequence by using the motion information of the image sequence and the image information of the background modeling picture.

[0046] Specifically, since the sphere occupies a small area in the image, using the temporal attention module allows the network to focus more on the dynamic changes between each frame, thereby ignoring other irrelevant information. The formula of the attention mechanism can be expressed as:

[0047]

[0048] Where Q, K, and V represent the query matrix, key matrix, and value matrix, respectively, and d : It is the dimension of the key, which is used to scale its module to make the network focus more on the moving sphere information and ignore irrelevant information.

[0049] Furthermore, the heat map includes the response position of the sphere in the image. The weighted loss function is used to increase the weight of the sphere response area, so that the network can focus on and predict the position of the sphere more accurately. After processing the heat map of the training module and the heat map generated by the true label value, the absolute error loss function is calculated. The loss function used in training is weighted BCE Loss. The specific formula is as follows:

[0050]

[0051] y and The predicted heat map value and the real label heat map value are identified respectively. w is the adjustment weight, which can be set to y. The optimization goal is to solve the model parameter θ to minimize the loss function, that is:

[0052]

[0053] Where θ represents the model parameters to be optimized, such as network weights, biases, etc., M is the number of training samples, and BCELoss is the weighted loss function. Finally, the training of the entire model is completed.

[0054] In a specific application example, after completing the network training in S30, the video stream is input into the network for inference operation to detect the position of the tennis ball. Specifically, during inference, a sliding window is used to infer the continuous frame images. In a window of inference, the detection confidence weight of the image at the center is the highest, and it decreases to both sides of the window in sequence. In this way, when the sliding distance is less than the window length, the image will have multiple tennis ball position detections, and the final detection value is obtained based on the weight. The window size is 2k+1, with k frames on the left and right of the center frame, and the final tennis ball detection value S final The calculation formula is expressed as:

[0055]

[0056] w j represents the weight of the jth frame in the window, s j represents the confidence of the tennis ball position detection in the jth frame. ,The weight formula is:

[0057]

[0058] σ is a hyperparameter that controls how fast the weights decay, in such a way that the frame at the center contributes the most to the confidence in the window, and the influence of the frames on both sides gradually decreases.

[0059] Through the above-mentioned tennis ball detection method based on deep learning applied to video streams, the accuracy of tennis ball position detection in tennis matches can be effectively improved, and high-speed moving balls can be effectively detected. Based on end-to-end training and real-time operation, it can be used for downstream tasks such as broadcasting and distribution.

[0060] It should be understood that the exemplary embodiments described herein are illustrative rather than restrictive. Although one or more embodiments of the present invention have been described in conjunction with the accompanying drawings, it should be understood by those skilled in the art that various changes in form and detail may be made without departing from the spirit and scope of the present invention as defined by the appended claims.

Claims

1. A tennis ball detection method based on deep learning applied to video streams, characterized in that: The following steps are involved: S10, data preprocessing stage: performing data preprocessing on the time-series video stream to obtain training data, wherein the image in the training data and the annotation file of the tennis ball position in the annotation image are a pair of data, which meet the time-series continuity; S20, model training stage: the continuous image sequence and background frame image in the training data pair are stacked through the stacking module to form the network training input, and the feature vector of the fused time series information is obtained through multi-layer downsampling of the encoder structure. After the enhancement of small target objects is improved through the time attention module, the corresponding heat map of the input time series is restored through the decoder structure; By comparing with the heat map of the real label, the loss value is obtained in the weighted loss calculation module, and the model parameters are solved by back propagation; S30, model reasoning stage: after completing the network training, the video stream is input into the network for reasoning operation to detect the position of the tennis ball.

2. The tennis ball detection method based on deep learning applied to video stream according to claim 1, characterized in that: The data preprocessing of the time-series video stream in S10 to obtain training data includes: collecting tennis game videos from public domains and authorized domains for video annotation, selecting multiple games in different periods and on different courts to produce training data, performing rough screening based on video information, counting the coverage of various games, manually screening, marking the main game time points that meet the viewing angle, and slicing and classifying the video through a computer program; at the same time, performing sample cleaning on the obtained video data, punctuating the position of the tennis ball in the continuous video sequence; constructing an online data enhancement module, including enhancement of brightness, contrast, picture size, and occlusion mode, constructing a training background image for video training, obtaining a background frame image by directly fusing the pixels of the video sequence, and finally forming a data pair combination of the image sequence and the annotation file sequence for training.

3. The tennis ball detection method based on deep learning applied to video stream according to claim 2, characterized in that: The final data pair is: <X,Y>=[<x1,x2,…,x n ,x background >,<y1,y2,…,y n >] Among them, x i ,y i are the corresponding image sequence and annotation file sequence, respectively, background The background frame image.

4. The tennis ball detection method based on deep learning applied to video stream according to claim 3, characterized in that: The encoder structure in S20 includes a stacking module, a first feature extraction layer, a second feature extraction layer, a third feature extraction layer and a fourth feature extraction layer connected in sequence. The data pairs obtained by data preprocessing are combined and stacked through the stacking module. The stacked data passes through the first feature extraction layer, the second feature extraction layer, the third feature extraction layer and the fourth feature extraction layer in sequence. The data resolution is converted into 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the original image size in sequence, and finally a feature map with high-level feature information is obtained.

5. The tennis ball detection method based on deep learning applied to video stream according to claim 4, characterized in that: The decoder structure includes a first fusion module, a second fusion module and a third fusion module. Each fusion module includes an upsampling operation and a feature extraction layer, which is used to fuse the feature maps from the corresponding encoder feature extraction layer and the output of the previous module. First, the input map resolution of the previous module is increased by upsampling and then connected with the corresponding feature map of the encoder. The feature extraction layer is used to fuse the generated feature map. The feature map enhanced by the temporal attention module passes through the first fusion module, the second fusion module and the third fusion module in turn, and is fused with the third feature extraction layer, the second feature extraction layer and the first feature extraction layer of the encoder respectively. Finally, a stacked heat map that fuses the shallow and deep features of the network is obtained. The heat map is input into the weighted loss calculation module, and the loss value is obtained through the weighted loss function, so as to learn accurate parameters through back propagation.

6. The tennis ball detection method based on deep learning applied to video stream according to claim 5, characterized in that: The heat map includes the response position of the sphere in the image, and the weighted loss function is used to increase the weight of the sphere response area, so that the network can focus on and predict the position of the sphere more accurately; After processing the heat map of the training module and the heat map generated by the true label value, the absolute error loss function is calculated. The loss function used in training is weighted BCE Loss. The specific formula is as follows: Weighting y and The predicted heat map value and the real label heat map value are identified respectively. w is the adjustment weight, which can be set to y. The optimization goal is to solve the model parameter θ to minimize the loss function, that is: Where θ represents the model parameters to be optimized, M is the number of training samples, and BCE Loss is the weighted loss function. Finally, the training of the entire model is completed.

7. The tennis ball detection method based on deep learning applied to video stream according to claim 6, characterized in that: After the network training is completed in S30, the video stream is input into the network for inference operation to detect the position of the tennis ball. Specifically, the sliding window is used to infer the continuous frame images. In a window of inference, the detection confidence weight of the image at the center is the highest, and it decreases to both sides of the window. In this way, when the sliding distance is less than the window length, the image will have multiple tennis position detections, and the final detection value is obtained based on the weight. The window size is 2k+1, and there are k frames on the left and right of the center frame. The final tennis ball detection value S 12345 The calculation formula is expressed as: w6 represents the weight of the jth frame in the window, and s6 represents the confidence of the tennis position detection in the jth frame. The weight formula is: σ is a hyperparameter that controls how fast the weights decay, in such a way that the frame at the center contributes the most to the confidence in the window, and the influence of the frames on both sides gradually decreases.

Images