A basketball player posture detection method and system based on deep learning

Abstract

The application discloses a kind of based on deep learning's basketball player posture detection method and system, it is related to sports science technology field, to solve the accuracy problem of player posture detection in basketball game;The method collects basketball game video from multiple perspectives, and pre-processes video, calculates video quality score, and classifies video.For A type video and B type video, different key point optimization detection methods are used, further, muscle and brain electrical signals are fused, the physiological state score of player is calculated, the posture key point confidence is dynamically adjusted, and the detection accuracy is improved.Finally, the matching degree of posture key point sequence in video is calculated by dynamic time warping.The application can detect the posture of basketball player with high precision in complex environment, has strong robustness, and is suitable for sports training and tactical analysis.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of sports science, and more particularly, to a basketball player posture detection method and system based on deep learning. BACKGROUND

[0002] With the development of computer vision technology, player posture detection has become an important technical means in sports analysis and training. Traditional basketball player posture detection methods rely on basic image processing techniques, which have problems such as insufficient capture of player posture details and difficulty in handling occlusion. In addition, existing technologies are mostly focused on detection through a single perspective or fixed equipment, lacking real-time monitoring and optimization of complex dynamic movements of players, resulting in low detection accuracy. Although deep learning has made some progress in image recognition, how to deal with fast and high dynamic movement states in basketball, as well as multi-angle occlusion and blur, is still a technical problem to be solved. Therefore, a basketball player posture detection method based on multi-perspective video analysis and physiological signal fusion provides higher precision technical support for improving player training and competition performance, and becomes the focus of research in this field.

[0003] To solve the above problems, the present application provides a solution. SUMMARY

[0004] In order to overcome the above-mentioned defects of the prior art, embodiments of the present application provide a basketball player posture detection method and system based on deep learning to solve the problems raised in the background art.

[0005] To achieve the above-mentioned purpose, the present application provides the following technical solutions:

[0006] In one preferred embodiment, it comprises:

[0007] Step 1: Obtain basketball game video and classify and label posture key points;

[0008] Step 2: Use a deep learning model to optimize the detection of basketball player posture key points;

[0009] Step 3: Temporal and spatial modeling to identify movement actions.

[0010] In one preferred embodiment, in step 1, multi-angle basketball game videos of basketball players are obtained; posture key points of basketball players in the basketball game video and confidence values of each posture key point are detected.

[0011] In a preferred embodiment, in step 1, the player occlusion rate Zv and the video dynamic blur degree Sd in the basketball video are calculated; the player occlusion rate and the video dynamic blur degree of the basketball video are weighted and summed to determine the quality score Q of the basketball video; the quality threshold Yz is set, when the quality score of the basketball video is greater than or equal to the quality threshold Yz, it indicates that the basketball video belongs to the A type high quality basketball video; when the quality score of the basketball video is less than the quality threshold Yz, it indicates that the basketball video belongs to the B type low quality basketball video; the pose key points of the A type and B type basketball videos are labeled.

[0012] In a preferred embodiment, in step 2, the A type basketball video is parsed to generate image data; the skeletal features of the basketball players in the image data are extracted to generate the heat map Ha of the pose key points; the coordinates of each pose key point and the best connection mode between the pose key points are calculated.

[0013] In a preferred embodiment, in step 2, the B type basketball video is parsed to generate image data; the skeletal features of the basketball players in the image data are extracted, and the feature extraction accuracy is enhanced to generate the heat map Hb of the pose key points; the coordinates of each pose key point are calculated and compensated; the best connection mode between the pose key points is calculated.

[0014] In a preferred embodiment, in step 2, the muscle and nerve electrical signals of the basketball players are collected; the comprehensive muscle activity index Ma and the motor cortex activation index Ca of the basketball players are calculated; the comprehensive muscle activity index Ma and the motor cortex activation index Ca of the basketball players are weighted and summed to determine the physiological state score Sf of the basketball players.

[0015] In a preferred embodiment, in step 2, the physiological state score Sf is used as a weight to adjust the confidence value Ci of the pose key points of the basketball players in the A and B type basketball videos; the confidence threshold Yc is set, when Sf is greater than or equal to the confidence threshold Yc, the confidence of the pose key points of the basketball players in the A and B type basketball videos is improved; when Sf is less than the confidence threshold Yc, the influence of the pose key points of the basketball players in the A and B type basketball videos is reduced; the pose key point sequence data of the basketball players in the A and B type basketball videos are fused.

[0016] In a preferred embodiment, in step 3, the pose key point sequence data of the basketball players is converted into time series data, the spatio-temporal features of the pose key points of the basketball players are obtained by using the model, the spatio-temporal features are mapped to the action categories, and the probability values of each category of action are calculated.

[0017] In a preferred embodiment, comprising: data acquisition and quality assessment module, data optimization module, motion analysis module and data storage module, signal connection between each module;

[0018] The data acquisition and quality assessment module is mainly used for collecting multi-angle basketball player video, performing standardized processing on the basketball video, and evaluating the quality of the basketball video, so as to classify the basketball video;

[0019] The data optimization module is mainly used for extracting key point information of the basketball player and optimizing the posture key point data according to different quality of the basketball video;

[0020] The motion analysis module is mainly used for analyzing the posture change of the basketball player, extracting key posture features, and determining the posture category;

[0021] The data storage module is mainly used for storing all data in the processing process.

[0022] The application discloses a basketball player posture detection method and system based on deep learning, relates to the field of sports science and technology, and aims to solve the accuracy of player posture detection in basketball games. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 It is a structure schematic diagram of the basketball player posture detection system based on deep learning.

[0024] Figure 2 It is an operation flowchart of the basketball player posture detection method based on deep learning. DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the application will be clearly and completely described with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the application.

[0026] EMBODIMENT

[0027] The application discloses a basketball player posture detection method based on deep learning, which comprises the following steps: Figure 2 As shown in the figure, comprising:

[0028] Step 1: Construct a basketball player posture dataset and preprocess it;

[0029] Place a high-definition camera at the front, side and top three angles respectively, set the high-definition camera frame rate to be greater than 60 FPS, the resolution to be 1920x1080, and the format to be MP4, and obtain multi-angle basketball video of a basketball player;

[0030] Then, use the FFmpeg tool to call the fps option to uniformly adjust the frame rate of all basketball videos to 30 FPS, ensure the time step is consistent, use bilinear interpolation to scale the resolution of all basketball videos, ensure all videos are adjusted to 1920x1080 resolution, traverse the frame data of the basketball video, use a sliding window method, and intercept 30 frames as a slice each time, intercept the next slice after sliding back 15 frames, and process the entire video.

[0031] After the basketball video processing is completed, the COCO pre-trained model is used to detect the posture key points of the basketball player in the basketball video, such as head, shoulder, elbow, wrist, hip, knee and ankle posture key points; the OpenPose API is used to analyze the basketball video frame, and the confidence value of each posture key point is output.

[0032] Since the player occlusion rate and video dynamic blur degree in the basketball video will cause the quality of the obtained basketball video to be different, the average value T of the confidence value of the posture key points in each frame of data in the basketball video is calculated, all key points are traversed, the key points with a confidence value lower than the average value are counted, and the player occlusion rate Zv is calculated according to the formula:

[0033]

[0034] Wherein, N represents the number of all key points, and Ci represents the confidence value of the posture key point;

[0035] The Laplace transform is used to calculate the video dynamic blur degree Sd of the basketball video according to the formula:

[0036]

[0037] Wherein, M represents the number of video frames, I j(x1,y1) is the Laplace value of the pixel point (x1,y1) of the jth frame, for Laplace operator;

[0038] After that, the player occlusion rate and the video dynamic blur degree of the basketball game video are weighted and summed to determine the quality score Q of the basketball game video, according to the formula: Q = Q1 x Zv + Q2 x Sd, wherein Q1 represents the weight of the player occlusion rate Zv, and Q2 represents the weight of the video dynamic blur degree Sd; and a quality threshold Yz is set, and the quality score of the basketball game video is compared with the quality threshold Yz, when the quality score of the basketball game video is greater than or equal to the quality threshold Yz, it indicates that the basketball game video belongs to the A type high-quality basketball game video; when the quality score of the basketball game video is less than the quality threshold Yz, it indicates that the basketball game video belongs to the B type low-quality basketball game video;

[0039] Further, for the posture key points in the A type high-quality basketball game video, OpenPose is directly used for labeling; for the posture key points in the B type low-quality basketball game video, a key point tracking algorithm based on optical flow method is used to optimize the posture key point loss problem, and a SVM training classifier is used to remove the misdetected posture key points;

[0040] Step 2: using a deep learning model to optimize the detection of basketball player posture key points;

[0041] For the A type basketball game video:

[0042] A posture estimation algorithm model HRNet suitable for key point detection of high-resolution images based on deep learning is used to detect the skeletal key points of the basketball player;

[0043] The specific process is as follows:

[0044] OpenCV or FFmpeg tools are used to parse the A type basketball game video obtained in step 1 frame by frame, and each frame generates an RGB format image, which is stored as image data; the image data is scaled and normalized by using PIL; the RGB image is uniformly adjusted to the input size of the HRNet model, and an input batch is constructed, and N1 images are input each time, and the HRNet model is input according to the batch;

[0045] The HRNet model is used to extract the feature mapping layer of the input RGB image at different resolutions; the hierarchical feature extraction is performed through the ResNet deep convolutional network, the skeletal features of the basketball player in the image data are extracted, and the 2D Gaussian distribution block is used to combine the multi-scale features to fit the heat map Ha of the posture key points;

[0046] After that, on the heat map, the (x1, y1) coordinates of each posture key point are calculated by using the Gaussian peak detection method, the best connection mode between the posture key points is calculated by using the Hungarian matching algorithm, and the topological structure is checked to ensure that the connection relationship conforms to the human body posture, then the skeleton structure is assembled to generate a complete human skeleton sequence;

[0047] Finally, the Kalman filter is used to time sequence smooth the (x1, y1) posture key point coordinates of each frame to optimize the continuity of the key point motion.

[0048] For the basketball video of type B:

[0049] The posture detection method based on the lightweight posture estimation algorithm and the time sequence information optimization is used to detect the skeleton key points of the basketball player;

[0050] The specific process is as follows:

[0051] The OpenCV or FFmpeg tool is used to parse the basketball video of type B obtained in step 1 frame by frame, and an RGB format image is generated for each frame, which is stored as image data; after scaling and normalizing the image data by using the PIL, it is input into the lightweight posture estimation algorithm MobileNetV2+PoseNet;

[0052] The lightweight posture estimation algorithm MobileNetV2+PoseNet is used to extract the basketball player skeleton features in the image data, and the multi-scale feature extraction method is used to enhance the feature extraction accuracy; then, the Gaussian distribution fitting is used to extract the peak points of the heat map, and the heat map Hb of the posture key points is generated by fitting;

[0053] After that, on the heat map, the (x2, y2) coordinates of each posture key point are calculated by using the Gaussian peak detection method, and the key point motion trajectory between the front and rear frames is calculated based on the optical flow tracking method RAFT to track and compensate the key points;

[0054] Based on the (x3, y3) coordinates of the compensated posture key points, the best connection mode between the posture key points is calculated by using the Hungarian matching algorithm to form a complete skeleton structure, and then a complete human skeleton sequence is generated;

[0055] Since the posture of the basketball player is not only controlled by the skeleton, but also affected by the muscle and nerve, therefore, the low EMG electromyographic sensor is installed on the key muscle groups of the basketball player such as quadriceps femoris, gastrocnemius, deltoid and biceps brachii to collect the muscle electric signal of the basketball player in real time; the EEG device Muse2 is installed on the frontal motor cortex area of the basketball player to collect the neural electric signal of the basketball player in real time;

[0056] After that, the characteristic value extraction is carried out on the collected muscle electrical signal of the basketball player, and the specific steps are as follows:

[0057] Root mean square RMS calculation:

[0058]

[0059] Integral electromyogram i EMG calculation:

[0060]

[0061] Median frequency MF calculation:

[0062]

[0063] Wherein, P(f) represents the power spectral density;

[0064] The comprehensive muscle activity index Ma of multiple muscle groups is calculated, and the specific formula is: Ma=Qr×RMS+Qi×i EMG+Qm×MF, wherein Qr represents the weight of the root mean square RMS, Q i represents the weight of the integral electromyogram i EMG, and Qm represents the weight of the median frequency MF;

[0065] For the collected neural electrical signal of the basketball player, the alpha, beta and gamma band powers are calculated according to the power spectral density, and the motor cortex activation index Ca is calculated based on the alpha, beta and gamma band powers, and the specific formula is: Ca=(Pβ-Pα) / (Pα+Pβ+Pγ), wherein Pα represents the alpha band power, Pβ represents the beta band power, and Pγ represents the gamma band power;

[0066] Further, the weighted sum of the comprehensive muscle activity index Ma and the motor cortex activation index Ca of the basketball player is used to determine the physiological state score Sf of the basketball player, and the specific formula is: Sf=Qj×Ma+Qyd×Ca, wherein Qj represents the weight of the comprehensive muscle activity index Ma, and Qyd represents the weight of the motor cortex activation index Ca; The physiological state score Sf of the basketball player is normalized by using the Min-Max method;

[0067] The physiological state score Sf is used as the weight to adjust the confidence value Ci of the posture key point of the basketball player in the A and B type basketball video, and the specific formula is: newCi=C i×Sf, and the confidence threshold Yc is set, when Sf is greater than or equal to the confidence threshold Yc, the confidence of the posture key point of the basketball player in the A and B type basketball video is improved; When Sf is less than the confidence threshold Yc, the influence of the posture key point of the basketball player in the A and B type basketball video is reduced;

[0068] After that, the matching degree of the key point sequences of the A and B type videos is calculated by dynamic time warping (DTW): D(PA, PB) = DTW(PA, PB), wherein PA and PB are the posture key point sequences of the A and B type videos; and D represents the similarity of the posture key point sequences of the A and B type videos.

[0069] The fusion weight is calculated as follows:

[0070]

[0071] WB = 1 - WA

[0072] wherein WA represents the weight of the posture key point sequence of the A type basketball video, WB represents the weight of the posture key point sequence of the B type basketball video, and the Softmax normalization is adopted to ensure that WA + WB = 1.

[0073] After that, the posture key point coordinates are updated by weighted fusion, and the fused posture key point sequence of the basketball player is output.

[0074] Step 3: Temporal-spatial modeling to identify the motion action.

[0075] The key point sequence data of the basketball player output in step 2 is converted into time series data.

[0076] A fixed time window T1 is set, and the key point sequence of continuous T1 frames is extracted from the basketball video stream, wherein the key point sequence of each frame contains N key points (such as 25), and each key point is composed of its (x, y) coordinates and confidence Ci.

[0077] The key point sequence of each time window is organized into a three-dimensional tensor X e R^(T x N x C), wherein C represents the number of channels, and the tensor represents the skeleton data of the basketball player in T1 frames.

[0078] ST-GCN modeling is adopted: the adjacency matrix A of the skeleton graph is defined based on the connection relationship of the human body joints, each node v i in the skeleton graph corresponds to a key point and an edge, which represents the connection relationship between two key points; the K-order neighborhood matrix is adopted to represent the topological relationship between the key points, and the topological structure required for the graph convolution operation is constructed; the graph convolution operation is performed on the input three-dimensional tensor X to learn the spatial relationship between different key points: Xo = s(W x A x X), wherein W represents the weight of the graph convolution layer, and s represents a nonlinear activation function.

[0079] A one-dimensional time series convolution layer is adopted to process the time dimension to capture the change characteristics of the posture over time: Xoo = Conv1D(Xo, kernel = k), wherein k represents the size of the convolution kernel.

[0080] After ST-GCN processing, the spatio-temporal feature representation of T1 frame is obtained: F = [F1, F2,..., FT], wherein each Ft represents the motion feature within the T1 frame window; the spatio-temporal feature is mapped to the action category through a fully connected layer: Y = Softmax (Wf x F + b), wherein Wf represents the fully connected layer weight, b represents the bias term, and Softmax outputs the probability value of each category.

[0081] The application also provides a basketball player posture detection system based on deep learning, as shown in the accompanying drawings, comprising a data acquisition and quality evaluation module, a data optimization module, a motion analysis module and a data storage module, and the modules are signal connected; Figure 1

[0082] The data acquisition and quality evaluation module is mainly used for acquiring multi-angle basketball video of basketball players, performing standardized processing on the basketball video, evaluating the quality of the basketball video, and classifying the basketball video;

[0083] The data optimization module is mainly used for extracting key point information of the basketball players and optimizing posture key point data for basketball videos of different qualities.

[0084] The motion analysis module is mainly used for analyzing the posture change of the basketball players, extracting key posture features, and determining the posture category.

[0085] The data storage module is mainly used for storing all data in the processing process.

[0086] The above formulas are all dimensionless numerical calculations, the formula is obtained by software simulation of a large amount of data to obtain the latest real situation, and the preset parameters in the formula are set by the person skilled in the art according to the actual situation.

[0087] The above embodiments can be realized by software, hardware, firmware or any combination thereof, in whole or in part. When realized by software, the above embodiments can be realized in the form of a computer program product in whole or in part.

[0088] Those skilled in the art can realize that the modules and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application of the technical solution and the constraints of the application. Professional technicians can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0089] ​In addition, each function module in each embodiment of the present application can be integrated in one processing module, or each module can be physically present alone, or two or more modules can be integrated in one module.

[0090] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0091] Finally: the above is only a preferred embodiment of the present application, and is not used to limit the present application, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A deep learning-based basketball player pose detection method, The application is characterized in that it comprises Step 1: Obtain basketball video and classify and label posture key points; Step 2: Optimize the detection of basketball player posture key points by using a deep learning model; Step 3: Temporal and spatial modeling to identify movement actions; In step 1, the player occlusion rate Zv and the video dynamic blur degree Sd in the basketball video are calculated; the player occlusion rate and the video dynamic blur degree of the basketball video are weighted and summed to determine the quality score Q of the basketball video; a quality threshold Yz is set, when the quality score of the basketball video is greater than or equal to the quality threshold Yz, it indicates that the basketball video belongs to the A type high-quality basketball video; when the quality score of the basketball video is less than the quality threshold Yz, it indicates that the basketball video belongs to the B type low-quality basketball video; the posture key points of the A type and B type basketball videos are labeled; In step 2, the muscle and nerve electrical signals of the basketball player are collected; the comprehensive muscle activity index Ma and the motor cortex activation index Ca of the basketball player are calculated; the comprehensive muscle activity index Ma and the motor cortex activation index Ca of the basketball player are weighted and summed to determine the physiological state score Sf of the basketball player; In step 2, the physiological state score Sf is used as a weight to adjust the confidence value Ci of the posture key points of the basketball player in the A and B type basketball videos, and a confidence threshold Yc is set, when Sf is greater than or equal to the confidence threshold Yc, the reliability of the posture key points of the basketball player in the A and B type basketball videos is improved; when Sf is less than the confidence threshold Yc, the influence of the posture key points of the basketball player in the A and B type basketball videos is reduced; the posture key point sequence data of the basketball player in the A and B type basketball videos are fused.

2. The basketball player pose detection method based on deep learning according to claim 1, characterized in that: In step 1, multi-angle basketball video of the basketball player is obtained; the posture key points of the basketball player in the basketball video and the confidence value of each posture key point are detected. 3.The method of claim 2, wherein; In step 2, the A type basketball video is analyzed to generate image data; the skeletal features of the basketball player in the image data are extracted to generate a heat map Ha of the posture key points; the coordinates of each posture key point and the best connection mode between the posture key points are calculated.

4. The basketball player pose detection method based on deep learning according to claim 3, characterized in that: In step 2, the B type basketball video is analyzed to generate image data; the skeletal features of the basketball player in the image data are extracted, and the feature extraction accuracy is enhanced to generate a heat map Hb of the posture key points; the coordinates of each posture key point are calculated and compensated; the best connection mode between the posture key points is calculated.

5. The basketball player pose detection method based on deep learning according to claim 1, characterized in that: In step 3, the posture key point sequence data of the basketball player is converted into time series data, the spatio-temporal features of the basketball player posture key points are obtained by using a model, the spatio-temporal features are mapped to action categories, and the probability values of each category of action are calculated. 6.A basketball player pose detection system based on deep learning, characterized in that, It comprises The data acquisition and quality evaluation module, the data optimization module, the motion analysis module and the data storage module are signal connected; The data acquisition and quality evaluation module is used to obtain basketball video and classify and label posture key points; The data optimization module is used to optimize the detection of basketball player posture key points by using a deep learning model; The motion analysis module is used for spatiotemporal modeling to identify motion actions. In the data acquisition and quality evaluation module, the player occlusion rate Zv and the video dynamic blur degree Sd in the basketball video are calculated; the player occlusion rate and the video dynamic blur degree of the basketball video are weighted and summed to determine the quality score Q of the basketball video; the quality threshold Yz is set, when the quality score of the basketball video is greater than or equal to the quality threshold Yz, it indicates that the basketball video belongs to the A type high-quality basketball video; when the quality score of the basketball video is less than the quality threshold Yz, it indicates that the basketball video belongs to the B type low-quality basketball video; the posture key points of the A type and B type basketball videos are labeled; In the data optimization module, the muscle and nerve electrical signals of the basketball player are collected; The comprehensive muscle activity index Ma and the motor cortex activation index Ca of the basketball player are calculated; The comprehensive muscle activity index Ma and the motor cortex activation index Ca of the basketball player are weighted and summed to determine the physiological state score Sf of the basketball player; In the data optimization module, the physiological state score Sf is used as a weight to adjust the confidence value Ci of the posture key points of the basketball player in the A and B type basketball videos, and the confidence threshold Yc is set, when Sf is greater than or equal to the confidence threshold Yc, the reliability of the posture key points of the basketball player in the A and B type basketball videos is improved; when Sf is less than the confidence threshold Yc, the influence of the posture key points of the basketball player in the A and B type basketball videos is reduced; the posture key point sequence data of the basketball player in the A and B type basketball videos are fused; The data storage module is mainly used for storing all data in the processing process.

Images