A badminton match video line violation intelligent visual detection method

Abstract

The present application relates to the field of intelligent image processing and computer vision, and particularly relates to a badminton match video line violation intelligent visual detection method, comprising: constructing a badminton match image dataset containing foot contour and field key point annotation; designing and training a DFSNet foot segmentation network, which introduces an attention mechanism in the decoder part to realize accurate segmentation of left and right foot contours; training a key point detection model based on the annotated key points, and generating a bottom line, a service line and a center line binary mask that conforms to the actual white line in the image through secondary refinement; finally, performing pixel-level overlap detection on the foot contour output by the DFSNet and the refined field line mask, and if there are overlapping pixels, it is determined that the foot has violated the line. The present application takes foot contour analysis as the core, combines deep learning feature extraction with geometric prior optimization, realizes automatic and accurate detection of badminton line violation, and can be widely applied to sports event auxiliary judgment and sports analysis.

Description

Technical Field

[0001] This invention relates to the fields of intelligent image processing and computer vision technology, and in particular to an intelligent visual detection method for stepping out of bounds in badminton match videos. Background Technology

[0002] In badminton, stepping on the line (i.e., an athlete's foot touching the court line) is a common and crucial penalty. Currently, referees mainly rely on visual observation for judgment, but due to the fast pace of the game, rapid athlete movement, and the thinness of the court lines, human judgment is prone to errors and controversies. With the development of computer vision technology, some studies have attempted to use deep learning to assist in judging sports events, such as locating the athlete's foot position through human pose estimation or object detection. However, these methods have the following drawbacks: First, athletes' foot postures change frequently during rapid movement, and the left and right feet can easily obscure each other, making it difficult for keypoint detection-based methods to accurately extract foot contours; second, traditional methods usually rely solely on deep learning models and do not fully consider the geometric prior information of the court lines, making it difficult to guarantee detection accuracy in complex environments such as changes in lighting and blurred court lines. Summary of the Invention

[0003] The purpose of this invention is to provide an intelligent visual detection method for badminton match video violations involving stepping on the line, in order to improve the problems existing in the prior art.

[0004] This invention is implemented as follows: a smart visual detection method for stepping out of bounds in badminton match videos, comprising the following steps:

[0005] S1. Collect badminton match images, perform pixel-level left and right foot contour annotations on the athletes' feet in the images, generate single-channel mask images, and simultaneously annotate 30 intersection points of the court lines as court key points and record their pixel coordinates. Associate the foot mask images with the court key point annotation information to obtain a comprehensive dataset, and randomly divide the comprehensive dataset into training set, test set and validation set according to the proportion; the court lines include the baseline, service line and center line;

[0006] S2. Construct the DFSNet network model. The DFSNet network model adopts an encoder-decoder architecture. The encoder is a deep residual convolutional neural network that extracts low-level detail features and high-level semantic features. The high-level semantic features are input into the hollow spatial pyramid pooling (ASPP) module to obtain enhanced high-level features. The decoder fuses the low-level detail features with the enhanced high-level features output by the ASPP module. An attention module is introduced to enhance foot edge and region attention. Finally, the three-channel segmentation results corresponding to the background, left foot, and right foot are output.

[0007] S3. Construct a key point detection model and a secondary refinement module. The key point detection model is based on a deep convolutional neural network and a multi-level deconvolutional upsampling module. It predicts the preliminary coordinates of 30 key points on the field through heat map regression. The secondary refinement module refines the preliminary coordinates to obtain accurate field line parameters. Based on these parameters, it generates binary masks of the baseline, service line, and center line that match the actual white lines on the field.

[0008] S4: The DFSNet network model is trained using Dice Loss as the loss function to obtain a trained DFSNet network model. The Dice Loss loss function is used to overcome the class imbalance problem where background pixels dominate, so that the DFSNet network model can focus on accurately segmenting the foot contour.

[0009] S5: Input the image to be detected into the trained DFSNet network model. After forward propagation, take the class with the highest probability of each pixel and output a binary mask image that distinguishes the left and right feet.

[0010] S6: Input the image to be detected into the key point detection model, obtain the preliminary coordinates, and then process it through the secondary refinement module to obtain the binary mask of the court line. Perform pixel-level overlap detection between the binary mask images of the left and right feet and the binary mask of the court line. If there are overlapping pixels, it is determined that the corresponding foot is stepping on the line and committing a foul. Output the judgment result and generate a visualization image.

[0011] More preferably, step S1 specifically includes:

[0012] S101: Collect badminton match images, use annotation tools to annotate the athletes' feet in the images at the pixel level, draw the polygonal outlines of the left and right feet respectively, and generate corresponding single-channel mask images, where pixel value 0 represents the background, 1 represents the left foot, and 2 represents the right foot.

[0013] S102: For the same image, label 30 predefined key points of the court, including the intersection of the baseline and the sideline, the intersection of the service line and the sideline, and the intersection of the center line and the baseline. Assign an index from 0 to 29 to each key point and record its pixel coordinates.

[0014] S103: Based on the image file name, associate the single-channel mask images of the left and right feet of the same image with the annotation information of the site key points to construct a comprehensive dataset containing foot segmentation labels and key point labels;

[0015] S104: The comprehensive dataset is randomly divided into training, testing and validation sets according to the proportion, and a fixed random seed is used to ensure the repeatability of the division.

[0016] More preferably, in step S2, the encoder is a deep residual convolutional neural network pre-trained on ImageNet, containing 5 convolutional stages. The first layer outputs 256 channels of low-level detail features, and the fourth layer outputs 2048 channels of high-level semantic features. The ASPP module contains a 1×1 convolution, three 3×3 dilated convolutions with dilation rates of 6, 12, and 18, and a global average pooling parallel branch. Each branch outputs 256 channels, and after concatenation, the dimensionality is reduced to 256 channels by a 1×1 convolution to obtain enhanced high-level features.

[0017] The decoder process is as follows: the low-level detail features are reduced to 48 channels by 1×1 convolution, and after batch normalization and ReLU activation, they are concatenated with the enhanced high-level feature channels after 4x bilinear upsampling to obtain 304-channel fused features; the fused features are reduced to 256 channels by 3×3 convolution, and after batch normalization and ReLU activation, they are input into the attention module, and finally output as three-channel segmentation results by 1×1 convolution.

[0018] More preferably, the attention module is a CBAM attention module, including a channel attention submodule and a spatial attention submodule; the channel attention submodule performs global average pooling and global max pooling on the input feature map respectively, and after processing through a shared fully connected layer, the two are summed and activated by Sigmoid to generate channel weights; the spatial attention submodule performs average pooling and max pooling on the channel-weighted feature map in the channel dimension, and after concatenation, it is processed by a 7×7 convolution and activated by Sigmoid to generate spatial weights; the channel weights and spatial weights are applied to the input features in sequence to achieve feature recalibration.

[0019] More preferably, in step S3, the feature extraction backbone of the key point detection model is a deep convolutional neural network, which outputs a feature map of 1 / 32 of the input image size. After three deconvolutional layers with a stride of 2, the feature map is restored to 1 / 4 of the input image size. After each deconvolutional layer, batch normalization and ReLU activation are performed. Finally, a 30-channel heatmap is output after a 1×1 convolution. The model training uses a weighted mean square error loss function, which assigns higher weights to positive samples in the Gaussian region near the key points. The optimizer is Adam, the initial learning rate is 0.001, the batch size is 2, and the training iterations are 30,000.

[0020] More preferably, in step S3, the specific process of secondary refinement is as follows:

[0021] S301: Obtain key points on a single field line. If the number of valid key points on the current line is less than 2, generate an empty mask directly; otherwise, fit an initial straight line based on the key points on the line and calculate its normal vector.

[0022] S302: Extract the white pixel with the largest gray value in the direction perpendicular to the initial line with an adaptive search radius. The sampling position is determined according to the projection parameters of the key point on the initial line, and the interval length is extended proportionally to both ends during sampling.

[0023] S303: Different refinement strategies are adopted according to the type of court lines. The baseline is determined by dynamic weighted search and robust fitting with RANSAC through multiple rounds of iteration, and the refined line parameters are determined by combining the gray score. The service line and center line are determined by iterative optimization and robust fitting with RANSAC, and the refined line parameters are determined by combining the gray score.

[0024] S304: Calculate and compare the grayscale scores of the refined line and the initial line. If the refined line has a better score, the refined line is adopted; otherwise, the initial line is retained to obtain the final line parameters.

[0025] S305: Based on the final determined straight line parameters, generate the center line and expand it according to the adaptively estimated line width to generate a binary mask of the court lines. The adaptively estimated line widths of the baseline and service line are based on the percentile distance of key points, and the adaptively estimated line width of the center line is based on gradient measurement.

[0026] More preferably, the formula for calculating grayscale score is: ;

[0027] in, For grayscale scoring, The average gray level of the sampling points on the straight line; The average distance from the key point to the line; This is the grayscale weight, with a value ranging from 0.5 to 1.5. For the service line and center line, the distance weights are calculated based on distance. The value ranges from 2.0 to 6.0; for the baseline, the distance weight... The value ranges from 3.0 to 8.0.

[0028] More preferably, in step S4, the expression for the DiceLoss loss function is: in, Denotes the DiceLoss loss function. This represents the index of a pixel in the image, with a value ranging from 1 to N, where N is the total number of pixels in the image. Predict probabilities for the model; This is a real label; The smoothing term is set to 1.0; the SGD optimizer is used during training, with an initial learning rate of 0.001, momentum of 0.9, weight decay of 1e-4, batch size of 2, 20,000 training iterations, and a poly strategy with a power-law decay of 0.9.

[0029] More preferably, in step S6, the pixel-level overlap detection involves traversing all non-zero pixels. If the left foot mask overlaps with any court line mask, the left foot is judged to be in violation of the line. If the right foot mask overlaps with any court line mask, the right foot is judged to be in violation of the line. In the visualized image, the left foot outline is drawn in blue, the right foot outline is drawn in red, the court line area is filled with semi-transparent green, and the line-crossing status is marked in the image.

[0030] Compared with existing technologies, this invention has the following advantages: By constructing a DFSNet network model and introducing an attention mechanism, this invention achieves accurate extraction of the left and right foot contours, effectively addressing foot occlusion and rapid movement issues. Simultaneously, by utilizing a secondary refinement method combining keypoint detection and geometric priors, a binary mask of the court lines is generated that closely matches the actual white lines in the image, significantly improving positioning accuracy in complex environments. Finally, through pixel-level overlap detection between the foot contour mask and the court line mask, automatic and accurate line-crossing foul judgment is achieved. This invention combines deep learning feature extraction with geometric prior optimization and can be widely applied in badminton match auxiliary judging, motion analysis, and sports technology, possessing significant practical value. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 This is a flowchart of a method according to an embodiment of the present invention;

[0033] Figure 2 This is a schematic diagram of the structure of the DFSNet network model in an embodiment of the present invention;

[0034] Figure 3 This is a schematic diagram of the CBAM attention module in an embodiment of the present invention;

[0035] Figure 4 This is a schematic diagram of the key point detection network in an embodiment of the present invention;

[0036] Figure 5 This is a schematic diagram of the secondary refining process in an embodiment of the present invention;

[0037] Figure 6 This is a comparative diagram of segmentation results based on this application and existing methods;

[0038] Figure 7 This is a schematic diagram illustrating the visualization effect of line-stepping judgment in an embodiment of the present invention. Detailed Implementation

[0039] To better understand the above-described objects, features, and advantages of the present invention, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Many specific details are set forth in the following description to provide a thorough understanding of the invention; however, the invention may be practiced in other ways different from those described herein, and therefore, the invention is not limited to the specific embodiments disclosed below.

[0040] Example 1: This example provides an intelligent visual detection method for stepping out of bounds in badminton match videos, such as... Figure 1 As shown, it includes the following steps:

[0041] S1: Constructing the original image dataset of badminton matches: Obtain badminton match videos from various video websites and public dataset websites, decompose the videos into continuous image frames, filter out images with clear athlete feet and 30 unobstructed key points on the court, and construct the original image dataset.

[0042] For each image in the original image dataset, two types of annotations are performed: left and right foot contour annotations and site keypoint annotations, resulting in a comprehensive dataset. This comprehensive dataset is then randomly divided into training, testing, and validation sets in a 6:2:2 ratio. The specific steps are as follows:

[0043] S101: Use the Labelme annotation tool to annotate the athlete's feet in the image at the pixel level, and draw the polygonal outlines of the left and right feet respectively; save the annotation information in JSON format, with each outline corresponding to a label, where the label for the left foot is "left_foot" and the label for the right foot is "right_foot"; then, write a script to convert the JSON annotations into a single-channel mask image, where a pixel value of 0 represents the background, 1 represents the left foot, and 2 represents the right foot.

[0044] S102: For the same image, label 30 predefined key points of the court, including the intersection of the baseline and sideline, the intersection of the service line and sideline, and the intersection of the center line and baseline; assign an index from 0 to 29 to each key point and record its pixel coordinates; the labeling information is also saved in JSON format, and the label corresponding to each key point is a numeric index.

[0045] S103: Based on the image file name, associate the single-channel mask images of the left and right feet of the same image with the site key point annotation information to construct a comprehensive dataset containing foot segmentation labels and key point labels.

[0046] S104: The comprehensive dataset is randomly divided into training, testing, and validation sets in a ratio of 6:2:2. To ensure the repeatability of the division, the random seed is fixed at 42 in the code, so that the same division result is obtained every time it is run.

[0047] S2: Construct the DFSNet network model Figure 2 The diagram shows the structure of the DFSNet network model. In the diagram, s refers to the convolution stride, k refers to the kernel size, p refers to padding, and s=2 (1) indicates that the layer uses a stride of 2 for downsampling, and this only applies to the first convolutional block of the layer. The DFSNet network model adopts an encoder-decoder architecture. The encoder uses a deep residual convolutional neural network pre-trained on ImageNet to extract multi-scale features from the input image. It contains 5 convolutional stages, with layer 1 outputting 256 channels of low-level detail features and layer 4 outputting 2048 channels of high-level semantic features.

[0048] The high-level feature input is a Spatial Pyramid Pooling (ASPP) module, which contains five parallel branches: a 1×1 convolution, three 3×3 dilated convolutions (with dilation rates of 6, 12, and 18, respectively), and a global average pooling branch. Each branch outputs 256 channels. After concatenating the outputs of each branch along the channel dimension, the dimensionality is reduced to 256 channels through a 1×1 convolution, resulting in the enhanced high-level feature.

[0049] The decoder consists of: a 1×1 convolutional layer for reducing low-level detail features to 48 channels and performing batch normalization and ReLU activation; a 4x bilinear upsampling layer for scaling up the enhanced high-level features output by the ASPP module to the same size as the reduced low-level detail features; a feature concatenation layer for concatenating the two features along the channel dimension to obtain a 304-channel fused feature; a 3×3 convolutional layer for reducing the fused feature to 256 channels and performing batch normalization and ReLU activation; a CBAM attention module for recalibrating the features along the channel and spatial dimensions; and finally, a 1×1 convolutional layer outputs a 3-channel segmentation result, corresponding to the background, left foot, and right foot, respectively.

[0050] Figure 3This is a diagram of the CBAM module structure, where H and W represent the height and width of the input image, respectively, s refers to the convolution stride, k refers to the kernel size, and p refers to padding. The CBAM module includes a channel attention submodule and a spatial attention submodule. The channel attention submodule performs global average pooling and global max pooling on the input feature map, processes them through a shared fully connected layer, adds them together, and then activates them with a sigmoid function to generate channel weights. The spatial attention submodule performs average pooling and max pooling on the channel-weighted feature map along the channel dimension, concatenates them, and then activates them with a 7×7 convolution and a sigmoid function to generate spatial weights. The channel weights and spatial weights are applied to the input features in sequence to achieve feature recalibration.

[0051] S3: Construct a keypoint detection model and a secondary refinement module, such as Figure 4 As shown in the figure, s refers to the convolution stride, and s=2 (1) indicates that the layer uses a stride of 2 for downsampling, and it only takes effect in the first convolution block of the layer; the key point detection model adopts a deconvolution structure based on a deep convolutional neural network. In this embodiment of the invention, the feature extraction backbone adopts a pre-trained deep convolutional neural network, and outputs a feature map with a spatial resolution of 1 / 32 of the input image. Then, the spatial resolution is gradually restored through multiple deconvolution layers with a stride of 2. In this embodiment of the invention, three deconvolution layers are used, and each deconvolution layer is followed by batch normalization and ReLU activation. Finally, the feature map size is enlarged to 1 / 4 of the input image. Finally, a 1×1 convolution layer is used to output a heat map corresponding to the number of key points. In this embodiment of the invention, there are 30 channels, and each channel corresponds to one key point.

[0052] When training the keypoint detection model, a weighted mean squared error loss function is used, the optimizer is Adam, the initial learning rate is 0.001, the batch size is 2, and the number of training iterations is 30,000. The positive sample area in the heatmap, i.e. the Gaussian region near the keypoint, is given higher weight, so that the model pays more attention to the accurate prediction of the keypoint position.

[0053] The preliminary coordinates of the predicted key points are then refined to obtain a precise site line; for example... Figure 5 As shown, the secondary refining process includes:

[0054] First, key points are obtained on a single court line. If the number of valid key points on the current line (valid key points are defined as those with a peak confidence score ≥ 0.9 in the heatmap output by the key point detection model; otherwise, they are considered invalid key points) is less than 2, an empty mask (a blank mask without valid areas) is directly generated. Otherwise, an initial straight line is fitted based on the key points on the line, and its normal vector is calculated. The points on the baseline include 9, 10, 11, 12, and 13; the points on the service line include 7, 22, 29, 26, and 15; and the points on the center line include 2, 19, 28, 29, 24, and 11.

[0055] In the direction perpendicular to the initial straight line, white pixels in the image are extracted using an adaptive search radius, which is dynamically adjusted according to the line type and key point distribution. In this embodiment, the search radius for the baseline is 2.5 times the line width, the search radius for the service line is 2.0 times the line width, and the search radius for the center line is 1.2 times the line width. The line width is adaptively estimated as follows: For the baseline and service line, it is calculated based on the percentile distance from key points on the line to the initial straight line. In this embodiment, the 95th percentile is used for the baseline and the 70th percentile for the service line, respectively limited to the range of 3-8 pixels and 3-6 pixels. For the center line, the line width is measured using a gradient method: 30 points are uniformly sampled along the center line direction. At each sampling point, grayscale values ​​are collected along a direction perpendicular to the center line with a scanning radius of 2.0 times the current line width. The gradient is calculated, and positive and negative peaks are detected. The difference between the peak positions is taken as the line width of that point. The median of the line width values ​​of all sampling points is multiplied by 0.9, and the result is limited to the range of 4 to 15 pixels to obtain the final line width of the center line. Specifically, white pixels are searched for in the grayscale image along a direction perpendicular to the initial line, using an adaptive search radius. In the grayscale image, for each sampling position, the pixel with the largest grayscale value within the search radius along the normal direction is searched as a candidate point. The sampling position is determined based on the projection parameters of the key points on the initial line: all key points on the line are projected onto the initial line to obtain the projection parameter value t for each key point; the minimum and maximum values ​​are then used to form the initial projection interval [t]. min , t max During sampling, the interval length is extended by 10% at both ends to cover the full length of the line, avoiding the omission of the end area of ​​the line segment due to the narrow projection range of the key point.

[0056] Next, different refinement strategies are adopted according to the type of court lines: For the baseline, a dynamic weighted search strategy is adopted, while pre-calculated doubles baseline positions are used to eliminate interference. The dynamic weighted search strategy is as follows: First, the pre-calculated doubles baseline positions are used to determine whether the current extraction point is interfered with. During the search, for each extraction point, it is detected upward along the opposite direction of the normal vector. If there is a pixel whose distance from the pre-calculated doubles baseline line is less than 1.2 times the half-width, it is determined to be interfered with. If it is interfered with, it is only searched downward, and the grayscale value of the white pixel found is multiplied by a weight of 1.8; if there is no interference, it is searched both upward and downward, but the grayscale value of the white pixel found in the downward direction is multiplied by a weight of 1.2. Finally, the candidate points were robustly fitted using iterative optimization and the RANSAC method. The refined line parameters were then determined by combining the mean grayscale value and the distance from the keypoint to the line as a comprehensive score. For the service line and center line, the selected white pixels were robustly fitted using iterative optimization and the RANSAC method. The refined line parameters were then obtained by combining the mean grayscale value and the distance from the keypoint to the line as a comprehensive score. The pre-calculation method for the doubles baseline position was as follows: a line was fitted using five keypoints (indices 8, 23, 24, 25, and 14) of the doubles baseline, and the distance from these five points to the line was calculated. The median distance was taken as the half-width of the doubles baseline.

[0057] The formula for calculating grayscale score is:

[0058]

[0059] in, For grayscale scoring, The average gray level of the sampling points on the straight line; The average distance from the key point to the line; and For preset weighting coefficients, The value range is 0.5 to 1.5; for different lines, The range of values ​​for is as follows: for the service line and the center line, The value range is 2.0 to 6.0; for the bottom line, The value range is 3.0 to 8.0. In this embodiment of the invention, for the service line and the center line, the grayscale weight... The value is 1.0, representing the distance weight. The value is 3.0; for the baseline, the grayscale weight is... The value is 1.0, representing the distance weight. The value is 5.0.

[0060] Simultaneously, the refinement of the straight line is verified to ensure it meets expectations. The final straight line is determined by a comprehensive evaluation: for the baseline, service line, and center line, a comprehensive score is calculated based on the average grayscale value and the average distance from the keypoints to the line. If the refined line scores higher, the refined result is adopted; otherwise, the initial straight line is retained. Finally, based on the final determined straight line parameters, a single-pixel-wide center line is generated within the keypoint projection range. This center line is then expanded according to the aforementioned adaptively estimated line width to obtain binary masks for the baseline, service line, and center line that match the actual white lines in the image.

[0061] S4: Train the DFSNet network model using Dice Loss as the loss function, the expression of which is:

[0062] ;

[0063] in, Denotes the DiceLoss loss function. This represents the index of a pixel in the image, with a value ranging from 1 to N, where N is the total number of pixels in the image. Predict probabilities for the model; This is a real label; The smoothing term is set to 1.0. The optimizer uses SGD with an initial learning rate of 0.001, momentum of 0.9, and weight decay of 1e-4. The batch size is set to 2, and the number of training iterations is 20,000. The learning rate uses a poly decay policy with a power exponent of 0.9.

[0064] S5: The trained DFSNet network model is used for foot segmentation inference. The image to be detected is input into the trained DFSNet network model. After forward propagation, the class with the highest probability in the three channels (background, left foot, right foot) for each pixel is taken as the prediction result for that pixel. Finally, a binary mask image that accurately distinguishes the left and right feet is output. Specifically, the model loading function is first called to obtain a DFSNet network model instance. Then, the input image size is uniformly scaled to 720 pixels high and 1280 pixels wide, and normalized. The mean of the normalization is set to 0.485, 0.456, and 0.406, and the standard deviation is set to 0.229, 0.224, and 0.225. Finally, the image is fed into the DFSNet network model to obtain the left and right foot masks.

[0065] To verify the superiority of the DFSNet method in foot contour segmentation, foot detection was performed on the same batch of badminton match images using both the YOLO object detection method and the DFSNet method. The results are as follows: Figure 6 As shown. Figure 6This diagram illustrates a comparison of foot detection results in an embodiment of the present invention. The upper part shows the YOLO detection result, which only outputs rectangular bounding boxes for both feet and cannot obtain precise contours. The lower part shows the DFSNet segmentation result of the present invention, which can accurately delineate the independent contours of the left and right feet. Experimental results show that the method of the present invention is significantly superior to traditional target detection methods in terms of the precision of foot contour segmentation.

[0066] S6: Line violation judgment and result output. Input the image to be detected into the key point detection model to obtain the preliminary coordinates of 30 key points. After secondary refinement module processing, obtain the baseline, service line and center line binary mask that matches the actual white line in the image.

[0067] The line violation determination process specifically includes: performing pixel-level comparisons between the left and right foot binary masks output by DFSNet and the binary masks of each court line generated by secondary refinement; traversing all non-zero pixels; if the left foot mask and the line mask have any overlapping pixels, the left foot is determined to be a line violation; if the right foot mask and the line mask have any overlapping pixels, the right foot is determined to be a line violation.

[0068] Visualization results as follows Figure 7 As shown, the outlines of the left and right feet are drawn in blue and red respectively, the court line area is filled with semi-transparent green, and the current judgment result is marked in the upper left corner of the image. In this embodiment, the format "Left Foot: FOUL / OK |Right Foot: FOUL / OK" is used to indicate whether the left foot and right foot stepped on the line and committed a foul.

[0069] Example 2: Building upon Example 1, this example saves the weight files of the trained DFSNet and keypoint detection model in a specified path, and implements inference functionality by loading the model weights. The system supports single-image input; the user specifies the path of the image to be detected via command-line parameters. The program sequentially completes image preprocessing, foot segmentation, keypoint detection, secondary refinement, pixel-level overlap detection, and visualization output, finally saving the annotation results to the specified output directory. The method described in this example is also applicable to batch image processing and competition video analysis: for batch images, simply traverse the image files in the specified folder and repeat the above process for each image to automatically complete the line violation detection for multiple images; for competition videos, read the video images frame by frame and execute the above process to achieve frame-by-frame line violation analysis, and summarize and output the statistical results. During processing, all intermediate results and final judgment images can be saved as needed for subsequent analysis and penalty reference.

[0070] This invention focuses on offline frame analysis of badminton match videos and can be widely applied to match video replay and judgment, athlete footwork training analysis, and other fields. It has the advantages of simple deployment and accurate detection.

[0071] Under a hardware environment with an AMD Ryzen 7 7735H processor with Radeon Graphics, an NVIDIA GeForce RTX 4060 Laptop GPU, 16GB of RAM, and 8GB of video memory, the effectiveness and advancement of the method described in the embodiments of the present invention were tested, and the results are as follows:

[0072] (1) Foot segmentation performance test

[0073] The foot segmentation performance of DFSNet was evaluated on the test set. Mean Intersection over Union (Mean IoU) and pixel accuracy, commonly used in semantic segmentation tasks, were employed as evaluation metrics. Test results show that the IoU for the background class reached 99.99%, the IoU for the left foot class reached 68.20%, the IoU for the right foot class reached 66.95%, the Mean IoU reached 78.38%, and the overall pixel accuracy reached 99.99%.

[0074] (2) Foot section profile accuracy test

[0075] The average symmetric surface distance between the predicted mask and the real mask was calculated using a skeletonization method. The test results on the test set are shown in Table 1. The median error for the left foot was 1.02 pixels, with the first quartile (Q1) at 0.75 pixels and the third quartile (Q3) at 1.51 pixels; the median error for the right foot was 1.06 pixels, with Q1 at 0.79 pixels and Q3 at 1.50 pixels; the overall median error was 1.04 pixels, and 75% of the samples had an error less than 1.5 pixels. The test results demonstrate that the method of this invention can achieve pixel-level accurate foot segmentation on the vast majority of images. A few samples showed larger errors due to extremely small or severely occluded feet in the image, but this did not affect the overall performance evaluation.

[0076] Table 1. Foot segmentation contour accuracy test results (pixels)

[0077] category median First quartile (Q1) Third quartile (Q3) left foot 1.02 0.75 1.51 Right foot 1.06 0.79 1.50 overall 1.04 - -

[0078] (3) Accuracy test of line stepping judgment

[0079] To verify the detection capability of the method of this invention for line-crossing fouls, a test set was constructed by randomly selecting images from the competition images. The left and right foot line-crossing states were manually labeled for each image. The method of this invention was used to automatically detect line-crossing fouls on the test set, and the results were compared with manually labeled images. Accuracy, precision, recall, and F1 score were calculated, and the results are shown in Table 2. Table 2 shows that the recall rate for left foot line-crossing reached 100%, indicating that the system can effectively detect all left foot line-crossing situations; the precision rate for right foot line-crossing reached 100%, indicating that all right foot samples judged as line-crossing were correct. The overall accuracy reached 94.78% and 95.65%, respectively, verifying the effectiveness and reliability of the method of this invention. The difference in performance between left and right feet mainly stems from the imbalance in the number of left and right foot line-crossing samples in the training data. Further optimization can be achieved through data augmentation or adjusting the judgment threshold to make the detection performance more balanced.

[0080] Table 2 Performance Test Results for Line-Stepping Judgment

[0081] category accuracy Accuracy Recall rate F1 score left foot 94.78% 64.71% 100.00% 0.78 Right foot 95.65% 100.00% 58.33% 0.73

[0082] While embodiments of the present invention have been described in detail above, it will be apparent to those skilled in the art that various modifications and variations can be made to these embodiments. However, it should be understood that such modifications and variations fall within the scope and spirit of the invention as defined in the claims. Furthermore, the invention described herein may have other embodiments and can be implemented or carried out in various ways.

Claims

1. A smart visual detection method for stepping out of bounds in badminton match videos, characterized in that, Includes the following steps: S1. Collect badminton match images, perform pixel-level left and right foot contour annotations on the athletes' feet in the images, generate single-channel mask images, and simultaneously annotate 30 intersection points of the court lines as court key points and record their pixel coordinates. Associate the foot mask images with the court key point annotation information to obtain a comprehensive dataset, and randomly divide the comprehensive dataset into training set, test set and validation set according to the proportion; the court lines include the baseline, service line and center line; S2. Construct the DFSNet network model. The DFSNet network model adopts an encoder-decoder architecture. The encoder is a deep residual convolutional neural network that extracts low-level detail features and high-level semantic features. The high-level semantic features are input into the hollow spatial pyramid pooling (ASPP) module to obtain enhanced high-level features. The decoder fuses the low-level detail features with the enhanced high-level features output by the ASPP module. An attention module is introduced to enhance foot edge and region attention. Finally, the three-channel segmentation results corresponding to the background, left foot, and right foot are output. S3. Construct a key point detection model and a secondary refinement module. The key point detection model is based on a deep convolutional neural network and a multi-level deconvolutional upsampling module. It predicts the preliminary coordinates of 30 key points on the field through heat map regression. The secondary refinement module refines the preliminary coordinates to obtain accurate field line parameters. Based on these parameters, it generates binary masks of the baseline, service line, and center line that match the actual white lines on the field. S4: The DFSNet network model is trained using Dice Loss as the loss function to obtain a trained DFSNet network model. The Dice Loss loss function is used to overcome the class imbalance problem where background pixels dominate, so that the DFSNet network model can focus on accurately segmenting the foot contour. S5: Input the image to be detected into the trained DFSNet network model. After forward propagation, take the class with the highest probability of each pixel and output a binary mask image that distinguishes the left and right feet. S6: Input the image to be detected into the key point detection model, obtain the preliminary coordinates, and then process it through the secondary refinement module to obtain the binary mask of the court line. Perform pixel-level overlap detection between the binary mask images of the left and right feet and the binary mask of the court line. If there are overlapping pixels, it is determined that the corresponding foot is stepping on the line and committing a foul. Output the judgment result and generate a visualization image.

2. The detection method according to claim 1, characterized in that, Step S1 specifically includes: S101: Collect badminton match images, use annotation tools to annotate the athletes' feet in the images at the pixel level, draw the polygonal outlines of the left and right feet respectively, and generate corresponding single-channel mask images, where pixel value 0 represents the background, 1 represents the left foot, and 2 represents the right foot. S102: For the same image, label 30 predefined key points of the court, including the intersection of the baseline and the sideline, the intersection of the service line and the sideline, and the intersection of the center line and the baseline. Assign an index from 0 to 29 to each key point and record its pixel coordinates. S103: Based on the image file name, associate the single-channel mask images of the left and right feet of the same image with the annotation information of the site key points to construct a comprehensive dataset containing foot segmentation labels and key point labels; S104: The comprehensive dataset is randomly divided into training, testing and validation sets according to the proportion, and a fixed random seed is used to ensure the repeatability of the division.

3. The detection method according to claim 2, characterized in that, In step S2, the encoder is a deep residual convolutional neural network pre-trained on ImageNet, which contains 5 convolutional stages. The first layer outputs 256 channels of low-level detail features, and the fourth layer outputs 2048 channels of high-level semantic features. The ASPP module contains a 1×1 convolution, three 3×3 dilated convolutions with dilation rates of 6, 12, and 18, and a global average pooling parallel branch. Each branch outputs 256 channels, which are then concatenated and reduced to 256 channels by a 1×1 convolution to obtain enhanced high-level features. The decoder process is as follows: the low-level detail features are reduced to 48 channels by 1×1 convolution, and after batch normalization and ReLU activation, they are concatenated with the enhanced high-level feature channels after 4x bilinear upsampling to obtain 304-channel fused features; the fused features are reduced to 256 channels by 3×3 convolution, and after batch normalization and ReLU activation, they are input into the attention module, and finally output as three-channel segmentation results by 1×1 convolution.

4. The detection method according to claim 3, characterized in that, The attention module is the CBAM attention module, which includes a channel attention submodule and a spatial attention submodule. The channel attention submodule performs global average pooling and global max pooling on the input feature map respectively, and then adds them after processing through a shared fully connected layer. Sigmoid activation is used to generate channel weights. The spatial attention submodule performs average pooling and max pooling on the channel-weighted feature map along the channel dimension, and after concatenation, it generates spatial weights through 7×7 convolution and Sigmoid activation; the channel weights and spatial weights are applied to the input features in sequence to achieve feature recalibration.

5. The detection method according to claim 4, characterized in that, In step S3, the feature extraction backbone of the keypoint detection model is a deep convolutional neural network, which outputs a feature map of 1 / 32 of the input image size. After three deconvolutional layers with a stride of 2, the feature map is restored to 1 / 4 of the input image size. Each deconvolutional layer is followed by batch normalization and ReLU activation. Finally, a 30-channel heatmap is output after a 1×1 convolution. The model training uses a weighted mean square error loss function, which assigns higher weights to positive samples in the Gaussian region near the keypoint. The optimizer is Adam, with an initial learning rate of 0.001, a batch size of 2, and 30,000 training iterations.

6. The detection method according to claim 5, characterized in that, In step S3, the specific process of secondary refinement is as follows: S301: Obtain key points on a single field line. If the number of valid key points on the current line is less than 2, generate an empty mask directly; otherwise, fit an initial straight line based on the key points on the line and calculate its normal vector. S302: Extract the white pixel with the largest gray value in the direction perpendicular to the initial line with an adaptive search radius. The sampling position is determined according to the projection parameters of the key point on the initial line, and the interval length is extended proportionally to both ends during sampling. S303: Different refinement strategies are adopted according to the type of court lines. The baseline is determined by dynamic weighted search and robust fitting with RANSAC through multiple rounds of iteration, and the refined line parameters are determined by combining the gray score. The service line and center line are determined by iterative optimization and robust fitting with RANSAC, and the refined line parameters are determined by combining the gray score. S304: Calculate and compare the grayscale scores of the refined line and the initial line. If the refined line has a better score, the refined line is adopted; otherwise, the initial line is retained to obtain the final line parameters. S305: Based on the final determined straight line parameters, generate the center line and expand it according to the adaptively estimated line width to generate a binary mask of the court lines. The adaptively estimated line widths of the baseline and service line are based on the percentile distance of key points, and the adaptively estimated line width of the center line is based on gradient measurement.

7. The detection method according to claim 6, characterized in that, In step S303, the formula for calculating the grayscale score is as follows: ; in, For grayscale scoring, The average gray level of the sampling points on the straight line; This represents the average distance from the key point to the line. This is the grayscale weight, with a value ranging from 0.5 to 1.

5. For the service line and center line, the distance weights are calculated based on distance. The value ranges from 2.0 to 6.0; for the baseline, the distance weight... The value ranges from 3.0 to 8.

0.

8. The detection method according to claim 1, characterized in that, In step S4, the expression for the DiceLoss loss function is: in, This represents the DiceLoss loss function. This represents the index of a pixel in the image, with a value ranging from 1 to N, where N is the total number of pixels in the image. Predict probabilities for the model; This is a real label; The smoothing term is set to 1.0; the SGD optimizer is used during training, with an initial learning rate of 0.001, momentum of 0.9, weight decay of 1e-4, batch size of 2, 20,000 training iterations, and a poly strategy with a power-law decay of 0.

9.

9. The detection method according to claim 1, characterized in that, In step S6, pixel-level overlap detection involves traversing all non-zero pixels. If the left foot mask overlaps with any court line mask, the left foot is judged to be in violation of the line. If the right foot mask overlaps with any court line mask, the right foot is judged to be in violation of the line. In the visualized image, the left foot outline is drawn in blue, the right foot outline is drawn in red, the court line area is filled with semi-transparent green, and the line-crossing status is marked in the image.

Images