Track tracking method for table tennis ball moving at high speed

Abstract

The invention relates to a high-motion table tennis trajectory tracking method, which comprises the following steps of: performing motion blur correction on an original table tennis competition video, reducing motion blur, and enabling the edge and the texture of a table tennis to be clearer; then super-resolution reconstruction processing is carried out, the pixels of the table tennis balls in the frame images are increased, and the omission ratio is reduced when the improved YOLOv7 network model is adopted to carry out multi-scale small target detection on the frame images after super-resolution reconstruction subsequently; next, an improved DeepSORT algorithm is adopted to track the bounding box video frame sequence detected in the second step to generate continuous two-dimensional coordinates, tracking position errors are reduced, then a three-dimensional track is obtained through monocular depth estimation and table plane constraint reconstruction, the whole process is subjected to assembly line design, error accumulation of traditional staged processing is avoided, and the tracking accuracy is improved. Four difficulties of small targets, motion blur, shielding and complex backgrounds are jointly solved, and the average success rate gt of the whole process is obtained; 90%.

Description

Technical Field

[0001] The present invention relates to the technical fields of image processing and machine vision, and more particularly, to a method for tracking the trajectory of a table tennis ball in high-speed motion. Background Art

[0002] High-speed moving targets are commonly found in fields such as sports events, biomedicine, intelligent transportation, and military applications. They are characterized by high motion speed, wide motion range, variable environment, and poor regularity. Measuring the speed and plotting the trajectory of high-speed moving objects are important techniques and methods in fields such as physics and engineering. In fields such as industry, transportation, and national defense, it is often necessary to accurately measure and analyze the motion trajectory and speed of high-speed moving objects for related research and applications.

[0003] The speed measurement and trajectory plotting of high-speed moving objects often use technologies such as high-speed photography and lidar. High-speed photography technology captures images during the motion process by placing high-speed cameras on the motion trajectory of the moving object, and then calculates the speed and trajectory by analyzing these images. Lidar technology emits laser beams onto the moving object and then measures the time and distance of the laser beam's return to calculate the position and speed of the object.

[0004] Defects of the prior art: (1) High miss rate of small targets: The detection recall rate of table tennis balls with a pixel ratio <0.5% by traditional detection algorithms (such as YOLOv3) is lower than 60%; (2) Feature loss caused by motion blur: In ordinary videos (30 FPS), the motion displacement of the table tennis ball can reach 10 pixels / frame, resulting in the failure of traditional optical flow methods (such as Lucas-Kanade) to track; (3) Interference from complex backgrounds: The texture of the table tennis table and the dynamic background of the audience result in a false detection rate >15%; (4) Poor occlusion recovery ability: When occluded by the racket, the prediction error based on Kalman filtering exceeds 20 pixels. Summary of the Invention

[0005] The technical problem to be solved by the present invention is how to improve the target detection rate, reduce the tracking position error, and meet the requirements of game analysis in high-speed motion scenarios.

[0006] The present invention provides a method for tracking the trajectory of a table tennis ball in high-speed motion, including: Step 1, obtaining the original table tennis game video, performing motion blur correction processing and super-resolution reconstruction processing on the original table tennis game video to obtain a clear video frame sequence; Step 2: Build an improved YOLOv7 network model to perform multi-scale small target detection on the video frame sequence, and obtain a bounding box video frame sequence containing the bounding box coordinates, class information, and confidence information of the table tennis ball in each frame; Step 3: Use the improved DeepSORT algorithm to track and manage the bounding box video frame sequence, and obtain a complete trajectory composed of continuous two-dimensional coordinates of the target; Step 4: Reconstruct the complete trajectory into a three-dimensional trajectory through monocular depth estimation and table plane constraint.

[0007] Compared with the prior art, the present application has the following advantages: Motion blur correction is performed on the original table tennis game video to reduce motion blur and make the edges and textures of the table tennis ball clearer; then, super-resolution reconstruction processing is carried out to increase the pixels of the table tennis ball in the frame image. When the improved YOLOv7 network model is subsequently used to perform multi-scale small target detection on the frame image after super-resolution reconstruction, the missed detection rate is reduced; then, the improved DeepSORT algorithm is used to track the bounding box video frame sequence detected in Step 2 to generate continuous two-dimensional coordinates, reducing the tracking position error, and then a three-dimensional trajectory is obtained through monocular depth estimation and table plane constraint reconstruction. The pipeline design of the whole process avoids the error accumulation of traditional stage-by-stage processing, jointly solves the four major difficulties of small targets, motion blur, occlusion, and complex background, and the average success rate of the whole process > 90%.

[0008] In a possible implementation manner, the motion blur correction process for the original table tennis game video in Step 1 specifically includes: Step 101A: Generate a motion blur kernel based on the video parameters of the original table tennis game video and the moving speed of the table tennis ball , and the calculation formula is: ; In the formula, , represents the standard deviation, , represents the center position of the motion blur kernel, , represents the length of the blur kernel, , where represents the moving speed of the table tennis ball; , represents the frame interval time; represents the width of the video frame of the original table tennis game video, represents the width of the standard table tennis table; Step 102A: Build an improved DeblurGANv2 network model. The improved DeblurGANv2 network model includes a DeblurGANv2 network generator and a DeblurGANv2 discriminator. The DeblurGANv2 network generator uses a multi-scale residual network for motion blur correction. The input of the DeblurGANv2 network generator is the generated motion blur kernel and the blurred frame image of the original table tennis match video , and outputs a de-blurred low-resolution frame image . The DeblurGANv2 discriminator is used to judge the authenticity of the low-resolution frame image generated by the DeblurGANv2 network generator and assist in generating the loss function of the improved DeblurGANv2 network model

[0009] Compared with the prior art, by first calculating the motion blur kernel and using the motion blur kernel and the improved DeblurGANv2 network model, compared with the traditional pure data-driven de-blurring method, the method of the present application can reduce motion blur and make the edges and textures of the table tennis ball clearer

[0010] In a possible implementation manner, the motion blur correction performed by the DeblurGANv2 network generator in step 102A specifically includes A1: Concatenate the blurred frame image of the original table tennis match video and the motion blur kernel along the channel dimension to form an input tensor ; A2: Perform a first-layer downsampling operation on the input tensor to obtain a feature map . The expression is ; In the formula, represents a convolution operation with a convolution kernel size of , represents the number of channels; IN represents instance normalization operation; represents an activation function; A3: Perform a second-layer downsampling operation on the feature map to obtain a feature map . The expression is ; A4: Use 9 cascaded residual blocks for deep residual learning to extract global features, and add the motion blur kernel as channel attention in the 4th and 7th residual blocks to perform weighted adjustment of features. Specifically, it includes ; In the formula, represents performing residual learning to extract features, represents performing weighted adjustment by adding channel attention to the residual block, where, The expression of ; The expression of ; In the formula, represents the sigmoid function, represents global average pooling, represents the output of fully connected and activation, represents the output after weighted adjustment by adding channel attention; represents the size of the feature, represents the feature map in the pixel coordinates; represents the learned weight matrix; A5, for the feature map output by the cascaded residual block perform the first upsampling operation to obtain the feature map , and the expression is: ; A6, the feature map perform the second upsampling operation to obtain the feature map , and the expression is: ; A7, perform normalization on the feature map to obtain the deblurred low-resolution frame image , and the expression is: ; In the formula, represents normalizing the pixel value to [-1, 1]; The loss function for generating the improved DeblurGANv2 network model includes perceptual loss and adversarial loss of the DeblurGANv2 discriminator, and the calculation formula is: ; In the formula, represents the total loss of the improved DeblurGANv2 network model, represents the perceptual loss generated by using the ReLU3_3 layer feature extractor of the pre-trained VGG-19, represents the adversarial loss of the DeblurGANv2 discriminator, where, ; ; Wherein, the ReLU3_3 layer feature of the pre-trained VGG-19 is a real and clear image; denotes the DeblurGANv2 discriminator, the ReLU3_3 layer feature extractor of the pre-trained VGG-19.

[0011] In a possible implementation manner, the super-resolution reconstruction process in step 1 specifically includes: Step 101B, building an ESRGAN network model, adding a spatial attention mechanism to the RRDB module of the ESRGAN network model to obtain an improved ESRGAN network model; Step 102B, constructing a loss function for the improved ESRGAN network model; Step 103B, inputting the low-resolution frame image into the improved ESRGAN network model for multi-scale feature fusion, and outputting a super-resolution frame image , to form a clear video frame sequence.

[0012] In a possible implementation manner, the calculation formula of the loss function of the improved ESRGAN network model constructed in step 102B is: ; Wherein, denotes the loss function of the improved ESRGAN network model, denotes the perceptual loss generated by using the ReLU5_4 layer feature extractor of VGG-19, denotes the adversarial loss generated by the discriminant network of the improved ESRGAN network model, denotes the pixel loss, and the calculation formula is: ; ; ; Wherein, denotes the super-resolution frame image output by the improved ESRGAN network model, denotes the real super-resolution frame image; denotes the ReLU5_4 layer feature extractor of VGG-19; The multi-scale feature fusion in the improved ESRGAN network model in step 103B specifically includes: B1, normalizing the pixel values of the low-resolution frame image to [-1, 1] to obtain a normalized input block , the calculation formula is: ; B2, the improved ESRGAN network model first uses a shallow convolutional layer to extract the initial feature map , ; represents a convolutional operation with an output channel number of 16 and a convolutional kernel of ; B3, a spatial attention mechanism is embedded in each RRDB module for multi-scale feature extraction and fusion. Each RRDB module includes multiple densely connected convolutional layers to extract features layer by layer. The feature representation of the th convolutional layer is the feature map , and the expression is: ; In the formula, represents the feature concatenation operation; B4, perform spatial attention weighting on the features output by each densely connected convolutional layer to enhance the features of the target area. The expression is: ; In the formula, represents performing spatial attention weighting on the feature map , , represents the learned weight matrix; B4, add the feature map after spatial attention weighting to the feature map to form a residual connection and obtain the output feature map , and the expression is ; B5, fuse the output features of 16 RRDB modules to obtain a multi-scale feature map with high-frequency and low-frequency information. The expression is: ; In the formula, represents the output feature map of the th RRDB module; B6, perform sub-pixel convolution operation on the multi-scale feature map to obtain a high-resolution super-resolution frame image , and the expression is: ; In the formula, represents the operation of rearranging the channel number of the multi-scale feature map into pixels in the spatial dimension.

[0013] Compared with the prior art, the improved ESRGAN network model is used for multi-scale feature extraction and fusion. By combining the spatial attention mechanism, the edge features of small targets are enhanced, the loss of details in the sampling process is avoided, and the missed detection rate of small targets is reduced.

[0014] In a possible implementation manner, the improved YOLOv7 network model constructed in step 2 includes an input end, a backbone network, a neck network, and a detection head. The backbone network is CSPDarknet53, and a spatial pyramid decomposition convolutional layer is added at the end of the backbone network to replace the original downsampling layer; the neck network uses an improved PANet for multi-scale feature fusion, and the detection head uses a dynamic label assignment algorithm to dynamically adjust the positive sample threshold and improve the detection weight of small targets. Specifically, it includes: Step 201, input the super-resolution frame image from the input end, and perform initial downsampling extraction through the backbone network to obtain a feature map ; Step 202, the spatial pyramid decomposition convolutional layer decomposes the feature map into subgraphs at a preset step size ; The expression is: ; In the formula, represents decomposing the feature map into subgraphs at step size Step 203, perform independent convolution on each subgraph to extract a local feature map , and the expression is: ; Step 204, reorganize the local features into the original resolution, retain the spatial details, and output a feature map : ; In the formula, represents the operation of restoring the original resolution through pixel rearrangement; Step 205, the input of the neck network includes the large-scale feature map output by the second CSP of the backbone network , the medium-scale feature map output by the spatial pyramid decomposition convolutional layer, and the small-scale feature map output by the fourth CSP of the backbone network; the neck network first upsamples the small-scale feature map by 2 times and splices it with the medium-scale feature map ; The expression is: In the formula, represents the upsampling operation; Step 206: The neck network downsamples the medium-scale feature map by a factor of 2 along the bottom-up path and then concatenates it with the small-scale feature map . The expression is: ; In the formula, represents the downsampling operation; Step 207: Input the large-scale feature map , the medium-scale fused feature map , and the small-scale fused feature map into the detection head. The decoupled head of the detection head outputs the class information and prediction boxes corresponding to each size. Then, a dynamic label assignment strategy is used to define the positive / negative sample features for object tracking, and the super-resolution frame image The bounding box coordinates, class information, and confidence information of the table tennis ball in the image; Step 208: Construct a small target weighted loss function for the improved YOLOv7 network model. The calculation formula is: ; In the formula, , , represents an improved IoU metric considering the overlapping region, center point distance, and aspect ratio, , represents the Euclidean distance between the center of the predicted detection box and the ground truth box, represents the length of the diagonal of the minimum bounding box, represents the aspect ratio consistency metric; represents the dynamic weighted classification loss, , represents the predicted class probability, represents the weight of difficult samples, , represents the loss weight adjustment, ; represents the dynamic threshold parameter of the dynamic assignment strategy.

[0015] Compared with the prior art, through spatial pyramid decomposition convolution, the loss of small target features caused by traditional downsampling is avoided, and a winter label assignment strategy is used to assign more positive samples to the table tennis ball, improving the detection accuracy of small targets and reducing the false detection rate.

[0016] In a possible implementation manner, the specific steps of step 3 include: Step 301: Input the video frame sequence of bounding boxes, create independent tracks for each detection box in the initial frame image; initialize the Kalman filter state vector ; Step 302: Determine whether there are detection boxes in the current frame image. If there are, form a detection box set , and enter Step 303; if not, it means the current frame image is occluded, and enter Step 304; Step 303: Use the DeepSort algorithm to match the detection box set with the tracks in the historical tracks ; output all the complete tracks with matching associations; Step 304: Use the B-spline interpolation method to perform occlusion processing and track recovery on the current frame image, and output the complete tracks.

[0017] In a possible implementation manner, the specific steps of Step 303 include: C1: Use the lightweight OSNet network to extract the 128-dimensional feature vector of the detection box : ; C2: Construct a state transition matrix based on the extended Kalman filter , and use the state transition matrix to predict the state vector of the current frame . The calculation formula is: ; represents the state vector of the th track at the frame; represents the predicted state vector of the current frame ; represents the frame interval time; C3: Calculate the cost matrix based on the feature vector and the state vector . The calculation formula is: ; In the formula, represents the observation threshold, represents the historical feature mean stored for the th track at the , where represents the error covariance, represents the observation matrix, ; represents the measurement noise covariance; C4, ; if the output is 1, it means that the detection box matches and associates successfully with the historical trajectory until the detection boxes of all frames of images match the trajectories, and all the complete trajectories of matching and association; if the output is 0, it means that the matching is unsuccessful, and all the complete trajectories of matching and association.

[0018] In a possible implementation manner, step 304 specifically includes: D1, if the detection box is occluded, based on the historical trajectory points perform cubic B-spline fitting three times to obtain the trajectory points of the current frame : ; In the formula, represents the B-spline basis function, represents the weight; D2, when the entire appearance feature is missing, use the state transition matrix based on the extended Kalman filter to make predictions, and the calculation formula is: .

[0019] Compared with the prior art, using the lightweight OSNet to achieve real-time tracking improves the accuracy of cross-frame target association; and through B-spline interpolation processing, the occluded table tennis trajectory is restored, and the trajectory prediction accuracy is improved.

[0020] In a possible implementation manner, step 4 specifically includes: Step 401, use unit depth estimation for the continuous two-dimensional coordinates of the table tennis obtained in step 3 to generate a disparity map D; and combine the middle coordinates of the detection box to calculate the three-dimensional coordinates, and the calculation formula is: ; ; ; Step 402, use the table plane constraint to optimize the depth value through the RANSAC algorithm to eliminate jitter; BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a schematic flowchart of the present invention. Detailed implementation manners

[0022] First of all, those skilled in the art should understand that these implementation manners are only used to explain the technical principles of the embodiments of the present application, and are not intended to limit the protection scope of the embodiments of the present application. Those skilled in the art can make adjustments according to needs to adapt to specific application scenarios.

[0023] In the description of the embodiments of the present application, it should be noted that unless otherwise clearly specified and limited, the terms "connected" and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium. For those of ordinary skill in the art, the specific meanings of the above terms in the embodiments of the present application can be understood according to specific situations.

[0024] In the embodiments of the present application, unless otherwise clearly specified and limited, the first feature being "on" or "under" the second feature can be that the first and second features are in direct contact, or the first and second features are indirectly in contact through an intermediate medium. Moreover, the first feature being "above", "over" and "on" the second feature can be that the first feature is directly above or obliquely above the second feature, or simply means that the first feature has a higher horizontal height than the second feature. The first feature being "under", "beneath" and "under" the second feature can be that the first feature is directly below or obliquely below the second feature, or simply means that the first feature has a lower horizontal height than the second feature.

[0025] The present application will be further described in detail below with reference to the drawings and specific embodiments.

[0026] See Figure 1 As shown, the embodiments of the present application disclose a method for tracking the trajectory of a table tennis ball moving at high speed, including: Step 1, obtaining the original table tennis game video, performing motion blur correction processing and super-resolution reconstruction processing on the original table tennis game video to obtain a clear video frame sequence; the parameter information of the original table tennis game video in this embodiment includes: video frame width (pixels) , actual width of the table , video frame rate = 30, frame interval time ; table tennis ball movement speed ; Performing motion blur correction processing on the original table tennis game video: Step 101A, generating a motion blur kernel , and the calculation formula is: ; In the formula, , represents the standard deviation, , represents the central position of the motion blur kernel, , represents the length of the blur kernel, , where represents the moving speed of the table tennis ball; , represents the frame interval time; represents the video frame width of the original table tennis match video, represents the width of the standard table tennis table; in this embodiment, the table fills the horizontal field of view in the video, and the blur kernel length of this embodiment is .

[0027] Step 102A, build an improved DeblurGANv2 network model, the improved DeblurGANv2 network model includes a DeblurGANv2 network generator and a DeblurGANv2 discriminator. The DeblurGANv2 network generator uses a multi-scale residual network for motion blur correction. The input of the DeblurGANv2 network generator is the generated motion blur kernel and the blurred frame image of the original table tennis match video , and outputs a de-blurred low-resolution frame image . The DeblurGANv2 discriminator is used to judge the authenticity of the low-resolution frame image generated by the DeblurGANv2 network generator, and assist in generating the loss function of the improved DeblurGANv2 network model.

[0028] Step 102A specifically includes: A1, splice the blurred frame image of the original table tennis match video with the motion blur kernel along the channel dimension to form an input tensor ; in this embodiment, the blurred frame image .

[0029] A2, perform the first layer of downsampling operation on the input tensor through 2 convolutional layers to obtain a feature map ; the expression is: ; In the formula, represents the convolution operation with a convolution kernel size of , represents the number of channels; IN represents the instance normalization operation; Denote the activation function; the output feature map .

[0030] A3 performs a second-layer downsampling operation on the feature map to obtain the feature map , and the expression is: ; The output feature map .

[0031] A4 uses 9 cascaded residual blocks for deep residual learning to extract global features, and adds a motion blur kernel to the 4th and 7th residual blocks as channel attention to perform weighted adjustment of features and enhance the features of the motion blur kernel. Specifically, it includes: ; In the formula, denotes performing residual learning to extract features, denotes performing weighted adjustment by adding channel attention to the residual block. Among them, The expression of is: ; The expression of is: ; In the formula, denotes the sigmoid function, denotes global average pooling, denotes the output of fully connected and activation, denotes the output after adding channel attention for weighted adjustment; denotes the size of the feature, denotes the feature map in the pixel coordinates; denotes the learned weight matrix.

[0032] A5 performs a first-layer upsampling operation on the feature map output by the cascaded residual blocks through the first transposed convolutional layer to restore it to the original resolution to obtain the feature map , and the expression is: .

[0033] A6 performs a second-layer upsampling operation on the feature map through the second transposed convolutional layer to obtain the feature map , and the expression is: .

[0034] A7, for the feature map Perform normalization operation through a convolutional layer to obtain a deblurred low-resolution frame image , and the expression is: ; In the formula, means normalizing the pixel value to [-1, 1] .

[0035] The loss function for generating the improved DeblurGANv2 network model includes perceptual loss and adversarial loss of the DeblurGANv2 discriminator, and the calculation formula is: ; In the formula, represents the total loss of the improved DeblurGANv2 network model, represents the perceptual loss generated by using the ReLU3_3 layer feature extractor of the pre-trained VGG-19, represents the adversarial loss of the DeblurGANv2 discriminator, where ; ; In the formula, the ReLU3_3 layer feature of the pre-trained VGG-19 for the real clear image; represents the DeblurGANv2 discriminator, the ReLU3_3 layer feature extractor of the pre-trained VGG-19.

[0036] The training set data in this step includes synthetic data and real data. Among them, the synthetic data is generated by applying motion blur to clear images (GoPro dataset + self-built table tennis images) using the motion blur kernel , and the real data is the blurred frames collected from actual game videos, paired with the clear frames (calibrated by a high-speed camera).

[0037] The super-resolution reconstruction process in step 1 specifically includes: Step 101B, build an ESRGAN network model, and add a spatial attention mechanism to the RRDB module of the ESRGAN network model to enhance the edge region features of the table tennis ball and enhance the detail recovery ability of the table tennis ball, obtaining an improved ESRGAN network model.

[0038] Step 102B, construct the loss function of the improved ESRGAN network model; the calculation formula of the loss function of the improved ESRGAN network model is: ; In the formula, Represents the loss function of the improved ESRGAN network model. Represents the perceptual loss generated by the ReLU5_4 layer feature extractor of VGG-19, and represents the adversarial loss generated by the discriminator network of the improved ESRGAN network model. Represents the pixel loss, and the calculation formula is: ; ; ; In the formula, Represents the super-resolution frame image output by the improved ESRGAN network model. Represents the real super-resolution frame image; Represents the ReLU5_4 layer feature extractor of VGG-19.

[0039] Step 103B, input the low-resolution frame image into the improved ESRGAN network model for multi-scale feature fusion, and output the super-resolution frame image , to form a clear video frame sequence, specifically including: B1, Normalize the pixel values of the low-resolution frame image to [-1, 1] to obtain the normalized input block , and the calculation formula is: .

[0040] B2, The improved ESRGAN network model first uses a shallow convolutional layer to extract the initial feature map , ; Represents a convolutional operation with an output channel number of 16 and a convolutional kernel of .

[0041] B3, A spatial attention mechanism is embedded in each RRDB module for multi-scale feature extraction and fusion. Each RRDB module includes multiple densely connected convolutional layers, and features are extracted layer by layer. The feature representation of the th convolutional layer is the feature map , and the expression is: ; In the formula, Represents the feature concatenation operation.

[0042] B4, Perform spatial attention weighting on the features output by each densely connected convolutional layer to enhance the features of the target area, and the expression is: ; In the formula, Represents the feature map Perform spatial attention weighting, , denotes the learned weight matrix.

[0043] B4, the feature map after spatial attention weighting is added to the feature map to form a residual connection, obtaining the output feature map , and the expression is .

[0044] B5, fuse the output features of 16 RRDB modules to obtain a multi-scale feature map with high-frequency and low-frequency information , and the expression is: ; In the formula, denotes the th output feature map of the RRDB module.

[0045] B6, perform sub-pixel convolution operation on the multi-scale feature map to obtain a high-resolution super-resolution frame image , and the expression is: ; In the formula, denotes the operation of rearranging the number of channels of the multi-scale feature map into pixels in the spatial dimension.

[0046] Compare the performance of the traditional ESRGAN network model and the improved ESRGAN network model of this application, as shown in Table 1: Table 1 Performance comparison table of traditional ESRGAN network model and improved ESRGAN network model ;

[0047] As shown in Table 1, by introducing the spatial attention mechanism, multi-scale feature fusion, and optimized loss function, the improved ESRGAN significantly improves the accuracy of super-resolution reconstruction and the small target detection ability, meeting the requirements of high-speed table tennis trajectory tracking.

[0048] Step 2: Build an improved YOLOv7 network model to perform multi-scale small target detection on the video frame sequence, obtaining a bounding box video frame sequence containing the bounding box coordinates, class information, and confidence information of the table tennis ball in each frame. The built improved YOLOv7 network model includes an input end, a backbone network, a neck network, and a detection head. The backbone network is CSPDarknet53. A spatial pyramid decomposition convolutional layer is added at the end of the backbone network to replace the original downsampling layer, avoiding the loss of high-frequency details of small targets (table tennis balls) caused by the traditional downsampling layer. The neck network uses an improved PANet for multi-scale feature fusion, and the detection head uses a dynamic label assignment algorithm to dynamically adjust the positive sample threshold, enhancing the detection weight of small targets.

[0049] Step 2 specifically includes: Step 201: Input the super-resolution frame image from the input end, and perform initial downsampling extraction through the backbone network to obtain a feature map .

[0050] Step 202: The spatial pyramid decomposition convolutional layer decomposes the feature map at a preset stride to obtain subgraphs ; The expression is: ; In the formula, represents splitting the feature map into subgraphs at stride , and the dimension of the output subgraph is .

[0051] Step 203: Perform independent convolution on each subgraph to extract a local feature map , and the expression is: .

[0052] Step 204: Recombine the local features into the original resolution, retain the high-frequency details, and output the feature map : ; represents the operation of restoring the original resolution through pixel rearrangement to enhance the features of small targets.

[0053] Step 205: The input of the neck network includes the large-scale feature map output by the second CSP of the backbone network, the medium-scale feature map output by the spatial pyramid decomposition convolutional layer, and the small-scale feature map output by the fourth CSP of the backbone network; The neck network first arranges the small-scale feature map along the top-down path After upsampling by a factor of 2, it is concatenated with the medium-scale feature map The expression is: ; In the formula, represents the upsampling operation.

[0054] Step 206, the neck network down-samples the medium-scale feature map by a factor of 2 along the bottom-up path and concatenates it with the small-scale feature map The expression is: ; In the formula, represents the downsampling operation.

[0055] Step 207, input the large-scale feature map , the medium-scale fused feature map , and the small-scale fused feature map into the detection head. The decoupled head of the detection head outputs the class information and prediction boxes corresponding to each size. Then, a dynamic label assignment strategy is adopted to define the positive / negative sample features for object tracking, enhance the detection weight of small objects, and output the super-resolution frame image The bounding box coordinates, class information, and confidence information of the table tennis ball in the image; specifically including: For the prediction box of each detection head, calculate its IoU matrix with the ground truth box: ; ; ; In the formula, represents the th prediction box, represents the th ground truth box; Calculate the target normalized size ; represents the width and height of the bounding box; represents the width and height of the video frame image; Calculate the dynamic positive sample matching threshold: , , to ensure that the matching threshold for small objects (table tennis balls) is lower; in the dynamic label assignment strategy, select the anchor boxes with IoU greater than as positive samples. When multiple anchor boxes meet the conditions, preferentially select the smaller-sized anchor boxes and preferentially assign more positive samples to small objects; as a result, the threshold of the small object ( in this embodiment drops to 0.2, and the threshold of the large object ( ) The threshold is increased to 0.3.

[0056] In this embodiment, the prediction boxes that meet the following conditions are forcibly assigned as positive samples: and ; Final positive sample label: .

[0057] Step 208: Construct a weighted loss function for small targets of the improved YOLOv7 network model. The calculation formula is: ; In the formula, , The localization loss weight of small targets is doubled. represents an improved IoU metric considering the overlapping area, center point distance, and aspect ratio. , represents the Euclidean distance between the center of the predicted detection box and the ground truth box. represents the length of the diagonal of the minimum bounding box. represents the aspect ratio consistency metric; represents the dynamic weighted classification loss. , represents the predicted class probability. represents the weight of difficult samples. , represents the loss weight adjustment. ; represents the normalized size of the target.

[0058] In this embodiment, the effect of using the traditional YOLOv7 network model for small target detection is compared with the effect of using the improved YOLOv7 network model for multi-scale small target detection in this application. The result is shown in Table 2: Table 2 Comparison of the effects of the traditional YOLOv7 network model and the improved YOLOv7 network model ; It can be seen that in this application, by using the improved YOLOv7 network model, the recall rate of small targets is increased by 23%, and the matching accuracy of high-quality prediction boxes is increased by 15%. Therefore, the improved YOLOv7 network model in this application can handle targets of different sizes more fairly and significantly improve the small target detection performance (such as the mAP@0.5 of table tennis detection is increased by 23%).

[0059] The improved YOLOv7 network model adopted in this embodiment first normalizes to eliminate the difference in image resolution, enabling the model to adapt to multi-scale inputs. Secondly, through dynamic thresholds and weighting, it alleviates the problem of small object missed detections, avoids large objects dominating the loss function, and balances the training weights of objects of different sizes. Finally, a dynamic label assignment strategy is adopted, where the prediction boxes output by the small-scale feature map (120×67) obtain more positive samples, alleviating missed detections. The large scale detects large objects, and the small scale focuses on small objects. After fusion, it covers the full size. The NMS for each scale is executed in parallel, and the total time consumption only increases by 1.2 ms (Tesla T4 test).

[0060] Step 3: Use the improved DeepSORT algorithm to perform trajectory tracking and management on the sequence of bounding box video frames, obtaining a complete trajectory composed of the continuous two-dimensional coordinates of the targets.

[0061] Step 3 specifically includes:[[]] Step 301: Input the sequence of bounding box video frames, create independent trajectories for each detection box in the initial frame image; initialize the Kalman filter state vector .

[0062] Step 302: Determine whether there are detection boxes in the current frame image. If there are, form a detection box set , and enter Step 303; if not, there is occlusion in the current frame image, and enter Step 304.

[0063] Step 303: Use the DeepSort algorithm to match the detection box set with the trajectories in the historical trajectory ; output all the complete trajectories of the matching associations.

[0064] Step 303 specifically includes:[[]] C1: Use the lightweight OSNet network to extract the 128-dimensional feature vector of the detection box : .

[0065] C2: Construct a state transition matrix based on the extended Kalman filter , and use the state transition matrix to predict the state vector of the current frame , and the calculation formula is:[[]] ; represents the state vector of the th trajectory at the Represents the state vector of the current frame predicted ; ; Represents the frame interval time.

[0066] C3, based on the feature vector and the state vector to calculate the cost matrix, and the calculation formula is: ; In the formula, represents the observation threshold, represents the mean value of the historical features stored in the th trajectory at the frame; where, represents the error covariance, represents the observation matrix, ; represents the measurement noise covariance.

[0067] C4, ; if the output is 1, it means that the detection box is successfully matched and associated with the historical trajectory , until the detection boxes of all frames of images are matched with the trajectories, all completely matched and associated trajectories; if the output is 0, it means that the matching is unsuccessful, all completely matched and associated trajectories.

[0068] Step 304, use the B-spline interpolation method to perform occlusion processing and trajectory recovery on the current frame image, and output the complete trajectory.

[0069] Step 304 specifically includes: D1, if occlusion occurs in the detection box, based on the historical trajectory points to perform cubic B-spline fitting to obtain the trajectory points of the current frame : ; In the formula, represents the B-spline basis function, represents the weight.

[0070] D2, when the entire appearance feature is missing, use the state transition matrix based on the extended Kalman filter to make predictions, and the calculation formula is: .

[0071] Compared with the prior art, lightweight OSNet is adopted to achieve real-time tracking, improving the accuracy of cross-frame target association; and through B-spline interpolation processing, the occluded table tennis trajectory is restored to enhance the trajectory prediction accuracy.

[0072] Step 4: Reconstruct the complete trajectory into a three-dimensional trajectory through monocular depth estimation and the table plane constraint, specifically including: Step 401: Use unit depth estimation for the continuous two-dimensional coordinates of the table tennis ball obtained in Step 3 to generate a disparity map D; And combine the intermediate coordinates of the detection box Calculate the three-dimensional coordinates, and the calculation formula is: ; ; ; Step 402: Utilize the table plane constraint to optimize the depth value through the RANSAC algorithm to eliminate jitter; Step 403: Connect the three-dimensional coordinate points in chronological order to form a complete three-dimensional trajectory.

[0073] In this application, motion blur correction is performed on the original table tennis match video to reduce motion blur and make the edges and textures of the table tennis ball clearer; then super-resolution reconstruction processing is carried out to increase the pixels of the table tennis ball in the frame image. When the improved YOLOv7 network model is subsequently used for multi-scale small target detection of the frame image after super-resolution reconstruction, the missed detection rate is reduced; then, the improved DeepSORT algorithm is used to track the sequence of bounding box video frames detected in Step 2 to generate continuous two-dimensional coordinates, reducing the tracking position error, and then a three-dimensional trajectory is reconstructed through monocular depth estimation and the table plane constraint. The pipeline design of the whole process avoids the error accumulation of traditional stage-by-stage processing, jointly solves the four major difficulties of small targets, motion blur, occlusion, and complex backgrounds, and the average success rate of the whole process > 90%.

[0074] In the description of the embodiments of this application, it should be noted that in the description of this application, terms such as "inside" and "outside" indicating directions or position relationships are based on the directions or position relationships shown in the drawings. This is only for convenience of description and does not indicate or imply that the device or component must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application.

[0075] In the description of the present application, the descriptions with reference to terms such as "one embodiment", "some embodiments", "in this embodiment", "specific example", or "some examples" mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0076] As described above, it is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed in the present application should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for tracking the trajectory of a table tennis ball moving at high speed, characterized in that, Including: Step 1: Obtain the original table tennis match video, perform motion blur correction and super-resolution reconstruction on the original table tennis match video to obtain a clear video frame sequence; Step 2: Build an improved YOLOv7 network model, perform multi-scale small target detection on the video frame sequence to obtain a bounding box video frame sequence containing the bounding box coordinates, class information, and confidence information of the table tennis ball in each frame; Step 3: Use the improved DeepSORT algorithm to perform trajectory tracking and management on the bounding box video frame sequence to obtain a complete trajectory composed of continuous two-dimensional coordinates of the target; Step 4: Reconstruct the complete trajectory into a three-dimensional trajectory through monocular depth estimation and table plane constraint.

2. The method for tracking the trajectory of a table tennis ball moving at high speed according to claim 1, characterized in that, The motion blur correction process for the original table tennis match video in Step 1 specifically includes: Step 101A, generate a motion blur kernel based on the video parameters of the original table tennis match video and the moving speed of the table tennis ball, and the calculation formula is: , the calculation formula is: ； In the formula, , represents the standard deviation, , represents the central position of the motion blur kernel, , represents the length of the blur kernel, , where represents the moving speed of the table tennis ball; , represents the frame interval time; represents the video frame width of the original table tennis match video, represents the width of the standard table tennis table; Step 102A, build an improved DeblurGANv2 network model. The improved DeblurGANv2 network model includes a DeblurGANv2 network generator and a DeblurGANv2 discriminator. The DeblurGANv2 network generator uses a multi-scale residual network for motion blur correction. The input of the DeblurGANv2 network generator is the generated motion blur kernel and the blurred frame image of the original table tennis match video , and outputs the de-blurred low-definition frame image . The DeblurGANv2 discriminator is used to judge the authenticity of the low-definition frame image generated by the DeblurGANv2 network generator , and assist in generating the loss function of the improved DeblurGANv2 network model.

3. The method for tracking the trajectory of a table tennis ball moving at high speed according to claim 2, wherein, The motion blur correction by the DeblurGANv2 network generator in Step 102A specifically includes: A1, concatenate the blurred frame images of the original table tennis match video with the motion blur kernel along the channel dimension to form an input tensor ; A2, for the input tensor perform the first layer of downsampling operation to obtain the feature map ; the expression is: ； In the formula, represents a convolution operation with a convolution kernel size of , represents the number of channels; IN represents the instance normalization operation; represents the activation function; A3, for the feature map perform the second-layer downsampling operation to obtain the feature map , and the expression is: ； A4 uses 9 cascaded residual blocks for deep residual learning to extract global features, and motion blur kernels are added to the 4th and 7th residual blocks. As channel attention, it performs weighted adjustment of features, specifically including: ； In the formula, represents performing residual learning to extract features, represents performing weighted adjustment by adding channel attention to the residual block, where The expression of is: ； The expression is: ； In the formula, represents the sigmoid function, represents global average pooling, represents the output of fully connected and activation, represents the output after weighted adjustment by adding channel attention; represents the size of the feature, represents the feature map the pixel coordinates in; represents the learned weight matrix; A5, the feature map output by the cascaded residual blocks performs the first upsampling operation to obtain the feature map , and the expression is: ； A6, Feature Map Perform the second upsampling operation to obtain the feature map , and the expression is: ； A7, perform a normalization operation on the feature map to obtain a deblurred low-resolution frame image , and the expression is: ； In the formula, denotes normalizing the pixel value to [-1, 1]; The loss function of the improved DeblurGANv2 network model includes perceptual loss and adversarial loss of the DeblurGANv2 discriminator, and the calculation formula is: ； In the formula, represents the total loss of the improved DeblurGANv2 network model, represents the perceptual loss generated by using the ReLU3_3 layer feature extractor of the pre-trained VGG-19, represents the adversarial loss of the DeblurGANv2 discriminator, where ； ； Wherein, the ReLU3_3 layer feature of the pre-trained VGG-19 is a real and clear image; denotes the DeblurGANv2 discriminator, and the ReLU3_3 layer feature extractor of the pre-trained VGG-19.

4. The method for tracking the trajectory of a table tennis ball moving at high speed according to claim 2, wherein The super-resolution reconstruction process in Step 1 specifically includes: Step 101B: Build an ESRGAN network model, add a spatial attention mechanism to the RRDB module of the ESRGAN network model to obtain an improved ESRGAN network model; Step 102B: Construct the loss function of the improved ESRGAN network model; Step 103B, input the low-resolution frame image into the improved ESRGAN network model for multi-scale feature fusion, and output the super-resolution frame image , and form a clear video frame sequence.

5. The method for tracking the trajectory of a table tennis ball moving at high speed according to claim 4, wherein The calculation formula of the loss function of the improved ESRGAN network model constructed in Step 102B is: ； In the formula, represents the loss function of the improved ESRGAN network model, represents the perceptual loss generated by the ReLU5_4 layer feature extractor of VGG-19, represents the adversarial loss generated by the discriminator network of the improved ESRGAN network model, represents the pixel loss, and its calculation formula is: ； ； ； In the formula, represents the super-resolution frame image output by the improved ESRGAN network model, represents the real super-resolution frame image; represents the feature extractor of the ReLU5_4 layer of VGG-19; The multi-scale feature fusion in the improved ESRGAN network model in Step 103B specifically includes: B1, normalize the pixel values of the low-resolution frame image to [-1, 1] to obtain the normalized input block , and the calculation formula is: ; B2, the improved ESRGAN network model first uses a shallow convolutional layer to extract the initial feature map , ; denotes a convolutional operation with an output channel number of 16 and a convolutional kernel of ; B3, a spatial attention mechanism is embedded in each RRDB module for the extraction and fusion of multi-scale features. Each RRDB module includes multiple densely connected convolutional layers to extract features layer by layer. The feature representation of the -th convolutional layer is expressed as the feature map , and the expression is: ； In the formula, represents the feature splicing operation; B4: Perform spatial attention weighting on the features output by each densely connected convolutional layer to enhance the features of the target area, and the expression is: ； In the formula, represents spatial attention weighting for the feature map , , represents the learned weight matrix; The feature map after spatial attention weighting, is added to the feature map to form a residual connection and obtain the output feature map , and the expression is ; B5 fuses the output features of 16 RRDB modules to obtain a multi-scale feature map with high-frequency and low-frequency information. , and the expression is: ； In the formula, represents the output feature map of the th RRDB module; B6, perform sub-pixel convolution operation on the multi-scale feature map to obtain a super-resolution frame image with high resolution , and the expression is: ； In the formula, represents a pixel operation that rearranges the number of channels of the multi-scale feature map into the spatial dimension.

6. The method for tracking the trajectory of a table tennis ball moving at high speed according to claim 1, characterized in that, The improved YOLOv7 network model built in Step 2 includes an input end, a backbone network, a neck network, and a detection head. The backbone network is CSPDarknet53, and a spatial pyramid decomposition convolutional layer is added at the end of the backbone network to replace the original downsampling layer; the neck network uses an improved PANet for multi-scale feature fusion, and the detection head uses a dynamic label assignment algorithm to dynamically adjust the positive sample threshold to enhance the detection weight of small targets; specifically including: Step 201, input the super-resolution frame image from the input end, and perform initial downsampling extraction through the backbone network to obtain the feature map ; Step 202, the spatial pyramid decomposition convolutional layer decomposes the feature map with a preset stride to obtain sub - graphs ; The expression is: ; In the formula, represents dividing the feature map into sub - graphs with a stride of ; Step 203, for each sub-graph perform independent convolution to extract a local feature map , and the expression is: ； Step 204, reorganize the local features into the original resolution, retain the spatial details, and output the feature map : ； In the formula, represents the operation of restoring the original resolution by pixel rearrangement; Step 205, the input of the neck network includes the large-scale feature map output by the second layer CSP of the backbone network , the medium-scale feature map output by the spatial pyramid decomposition convolutional layer and the small-scale feature map output by the fourth layer CSP of the backbone network ; the neck network first upsamples the small-scale feature map by 2 times along the top-down path and then concatenates it with the medium-scale feature map . The expression is: ； In the formula, represents an upsampling operation; Step 206, the neck network downsamples the medium-scale feature map by a factor of 2 along the bottom-up path and then concatenates it with the small-scale feature map. The expression is as follows: after downsampling by a factor of 2 and concatenating with the small-scale feature map The expression is: ； In the formula, represents a downsampling operation; Step 207: Input the large-scale feature map , the medium-scale fused feature map , and the small-scale fused feature map into the detection head. The decoupled head of the detection head outputs the class information and prediction boxes corresponding to each size, and then a dynamic label assignment strategy is adopted to define the positive / negative sample features for object tracking, and a super-resolution frame image is output, including the bounding box coordinates, class information, and confidence information of the table tennis ball in the image; Step 208: Construct the small target weighted loss function of the improved YOLOv7 network model, and the calculation formula is: ； In the formula, , , represents an improved IoU metric that takes into account the overlapping region, the distance between the center points, and the aspect ratio. , represents the Euclidean distance between the center of the predicted detection box and the ground truth box. represents the length of the diagonal of the minimum bounding box. represents the aspect ratio consistency metric. represents the dynamic weighted classification loss. , represents the predicted class probability. represents the weight of difficult samples. , represents the loss weight adjustment. ; represents the dynamic threshold parameter of the dynamic allocation strategy.

7. The method for tracking the trajectory of a table tennis ball moving at high speed according to claim 1, characterized in that, Step 3 specifically includes: Step 301: Input the video frame sequence of bounding boxes, create independent trajectories for each detection box in the initial frame image; initialize the Kalman filter state vector ; Step 302, determine whether there is a detection box in the current frame image. If so, form a detection box set , and proceed to step 303; if not, in the current frame image, there is occlusion, and proceed to step 304; Step 303, using the DeepSort algorithm, the set of detection boxes is matched with the historical trajectories in the historical trajectories ; all the complete trajectories of the matching associations are output; Step 304, perform occlusion processing and trajectory recovery on the current frame image using B-spline interpolation method, and output the complete trajectory.

8. The method for tracking the trajectory of a table tennis ball moving at high speed according to claim 7, wherein Step 303 specifically includes: C1. Extract the detection box using the lightweight OSNet network 128-dimensional feature vector : ； C2, construct a state transition matrix based on the extended Kalman filter , and use the state transition matrix to predict the state vector of the current frame . The calculation formula is as follows: ； Indicates the state vector of the n-th trajectory at the frame; indicates the state vector of the predicted current frame ； Indicates the frame interval time; C3, based on the feature vector and the state vector to calculate the cost matrix, and the calculation formula is: ； In the formula, represents the observation threshold, express Frame time The historical feature mean stored in each track; ,in, represents the error covariance, represents the observation matrix, ; represents the measurement noise covariance; C4, ; If the output is 1, it means that the detection box matches and associates successfully with the historical trajectory until the detection boxes of all frames of images match the trajectories, and all the complete trajectories of matching and association; if the output is 0, it means that the matching is unsuccessful, and all the complete trajectories of matching and association.

9. The method for tracking the trajectory of a table tennis ball moving at high speed according to claim 7, characterized in that, Step 304 specifically includes: D1, if occlusion occurs in the detection box, based on historical trajectory points Perform cubic B-spline fitting three times to obtain the trajectory points of the current frame : ； In the formula, represents the B-spline basis function, represents the weight; D2. When the entire appearance feature is missing, the state transition matrix based on the extended Kalman filter is used for prediction. The calculation formula is: .

10. The method for tracking the trajectory of a table tennis ball moving at high speed according to claim 1, wherein Step 4 specifically includes: Step 401: Use unit depth estimation for the continuous two-dimensional coordinates of the table tennis ball obtained in Step 3 to generate a disparity map D; Combined with the middle coordinates of the detection box Calculate the three-dimensional coordinates, and the calculation formula is: ； ； ； Step 402: Optimize the depth value through the RANSAC algorithm using the billiard table plane constraint to eliminate jitter. Step 403: Connect the three-dimensional coordinate points in chronological order to form a complete three-dimensional trajectory.

Images