A martial arts action recognition and action quality evaluation method based on artificial intelligence
By integrating convolutional neural network processing of visible light and infrared images with skeleton extraction from point cloud data, the problem of low accuracy in martial arts movement recognition and evaluation was solved, realizing intelligent and quantitative evaluation of martial arts movements and improving accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HARBIN INST OF TECH
- Filing Date
- 2025-04-23
- Publication Date
- 2026-05-15
AI Technical Summary
Existing methods for identifying and evaluating martial arts movements have low accuracy, are greatly influenced by the subjective factors of the evaluators, and lack accurate quantitative assessment methods.
Visible light and infrared images are fused together, features are extracted using a convolutional neural network, and human skeletons are extracted using point cloud data. A GRU network is used for action recognition and evaluation, and attention mechanisms and smoothing processes are employed to improve accuracy.
It enables intelligent, quantitative, and accurate evaluation of martial arts movements, improving the accuracy of movement recognition and evaluation while reducing human intervention.
Smart Images
Figure CN120375474B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of martial arts movement recognition and movement quality evaluation, and specifically relates to a method for martial arts movement recognition and movement quality evaluation based on artificial intelligence. Background Technology
[0002] Exercise is a primary way for residents to maintain their physical fitness in daily life. At the same time, the teaching of sports techniques is an important component of physical education in universities. Through instruction, students can effectively master the techniques, skills, and knowledge of different sports, comprehensively improve their physical fitness, cultivate exercise habits, and thus form a healthy lifestyle. With the vigorous promotion of martial arts in recent years, the number of martial arts enthusiasts has been continuously increasing. Simultaneously, as my country vigorously promotes martial arts education in schools to support the inheritance of national traditions, martial arts has gradually entered university physical education classes, becoming a very common university physical education course. Therefore, how to comprehensively, scientifically, and reasonably evaluate the quality of martial arts movements has become a focal point.
[0003] However, traditional methods of martial arts movement recognition and evaluation are too heavily influenced by the subjective factors of the evaluator and lack an accurate benchmark. Therefore, there is an urgent need for a more precise, quantitative, effective, and objective method like computers to assist in the evaluation of martial arts movements. To achieve intelligent, quantitative, and accurate evaluation of the quality of martial arts routines, it is necessary to conduct intelligent recognition and evaluation of martial arts movements.
[0004] In the field of action recognition, with the development of deep learning technology, deep learning-based methods for martial arts action recognition and evaluation have gradually emerged, including:
[0005] Convolutional neural networks (CNNs) obtain images containing human bodies from videos, then feed these images into the CNN for feature extraction to obtain action recognition results. For example, patent application CN114419505A proposes a martial arts action recognition method based on human posture estimation.
[0006] Recurrent neural networks (RNNs) model the acquired human body key point data into a sequence, using the time dimension as the sequence dimension and the key point dimension as the feature dimension. The RNN is then used to extract features and finally classify them to obtain the recognition result.
[0007] In terms of motion evaluation, existing methods can extract the human skeleton based on deep learning technology and then evaluate motion based on the characteristics of the human skeleton. However, overall, the accuracy of existing martial arts motion recognition and evaluation methods is still low and needs to be further improved. Summary of the Invention
[0008] The purpose of this invention is to solve the problem of low accuracy in existing martial arts movement recognition and evaluation methods, and to propose a martial arts movement recognition and movement quality evaluation method based on artificial intelligence.
[0009] The technical solution adopted by this invention to solve the above-mentioned technical problems is: a method for martial arts movement recognition and movement quality evaluation based on artificial intelligence, the method specifically including the following steps:
[0010] Step 1: At each acquisition time point, visible light images, infrared images, and point cloud data of martial arts movements are acquired simultaneously.
[0011] Step 2: For any time point t, the visible light image and infrared image acquired synchronously are fused to obtain the fused image corresponding to time point t.
[0012] Similarly, the fused image corresponding to each acquisition time point is obtained;
[0013] Step 3: Use the fused image from the first acquisition time point to the fused image from the Tth acquisition time point as input to the action recognition network, and output the martial arts action recognition result through the action recognition network;
[0014] Step 4: Extract the human skeleton from the point cloud data obtained at each acquisition time point, and smooth the extracted skeleton to obtain the human skeleton corresponding to each acquisition time point.
[0015] Step 5: Evaluate the quality of the martial arts movements based on the final human skeleton obtained in Step 4 and the martial arts movement recognition results in Step 3.
[0016] Furthermore, the fusion of the visible light image and the infrared image yields the fused image corresponding to time point t; the specific process is as follows:
[0017] The visible light image and infrared image acquired synchronously at time point t are used as inputs to the image fusion network. The working process within the image fusion network is as follows:
[0018] The infrared image is used as the input to the first convolutional layer with a kernel size of 3×3, and the output of the first convolutional layer is used as the input to the second convolutional layer with a kernel size of 1×1.
[0019] The output of the second convolutional layer is used as the input of the first max pooling layer, and the output of the first max pooling layer is used as the input of the third convolutional layer with a kernel size of 3×3.
[0020] The output of the third convolutional layer is used as the input of the fourth convolutional layer with a kernel size of 1×1. The output of the first max pooling layer is added to the output of the fourth convolutional layer, and the result of the addition is used as the input of the second max pooling layer.
[0021] The output of the second max pooling layer is used as the input of the fifth convolutional layer with a kernel size of 3×3, and the output of the fifth convolutional layer is used as the input of the sixth convolutional layer with a kernel size of 1×1.
[0022] The output of the second max pooling layer is added to the output of the sixth convolutional layer, and the result is used as the input of the third max pooling layer.
[0023] The output of the third max pooling layer is used as the input of the seventh convolutional layer with a kernel size of 3×3, and the output of the seventh convolutional layer is used as the input of the eighth convolutional layer with a kernel size of 1×1.
[0024] The visible light image is used as the input to the ninth convolutional layer with a kernel size of 3×3, and the output of the ninth convolutional layer is used as the input to the tenth convolutional layer with a kernel size of 1×1.
[0025] The output of the tenth convolutional layer is used as the input of the fourth max pooling layer, and the output of the fourth max pooling layer is used as the input of the eleventh convolutional layer with a kernel size of 3×3.
[0026] The output of the eleventh convolutional layer is used as the input of the twelfth convolutional layer with a kernel size of 1×1. The output of the fourth max pooling layer is added to the output of the twelfth convolutional layer, and the result of the addition is used as the input of the fifth max pooling layer.
[0027] The output of the fifth max pooling layer is used as the input of the thirteenth convolutional layer with a kernel size of 3×3, and the output of the thirteenth convolutional layer is used as the input of the fourteenth convolutional layer with a kernel size of 1×1.
[0028] The output of the fifth max pooling layer is added to the output of the fourteenth convolutional layer, and the result is used as the input of the sixth max pooling layer.
[0029] The output of the sixth max pooling layer is used as the input of the fifteenth convolutional layer with a kernel size of 3×3, and the output of the fifteenth convolutional layer is used as the input of the sixteenth convolutional layer with a kernel size of 1×1.
[0030] The outputs of the eighth and sixteenth convolutional layers are used as inputs to the attention-based fusion module. The output of the attention-based fusion module is used as input to the seventeenth convolutional layer with a kernel size of 1×1. The output of the seventeenth convolutional layer is then used as input to the first feature extractor.
[0031] The infrared image is used as the input to the second feature extractor, the visible light image is used as the input to the third feature extractor, the outputs of the first feature extractor, the second feature extractor, and the third feature extractor are used as the input to the CNN network, and the binary image output by the CNN network is used as the fused image of the infrared image and the visible light image.
[0032] Furthermore, the working process of the attention-based fusion module is as follows:
[0033] The output of the eighth convolutional layer is weighted and summed with the output of the sixteenth convolutional layer to obtain the weighted sum result a;
[0034] The output of the eighth convolutional layer is concatenated with the weighted summation result a to obtain the concatenation result a1;
[0035] The connection result a1 is used as the input of the eighteenth convolutional layer with a kernel size of 1×1, and the output of the eighteenth convolutional layer is used as the input of the first sigmoid activation function layer.
[0036] The connection result a1 is used as the input of the first spatial attention unit. The output of the first sigmoid activation function layer is added to the output of the first spatial attention unit to obtain the sum result b. Then, the sum result b is multiplied by the output of the eighth convolutional layer to obtain the multiplication result b1.
[0037] The output of the sixteenth convolutional layer is concatenated with the weighted summation result a to obtain the concatenation result a2.
[0038] The connection result a2 is used as the input of the nineteenth convolutional layer with a kernel size of 1×1, and the output of the nineteenth convolutional layer is used as the input of the second sigmoid activation function layer.
[0039] The connection result a2 is used as the input of the second spatial attention unit. The output of the second sigmoid activation function layer is added to the output of the second spatial attention unit to obtain the sum result c. Then, the sum result c is multiplied by the output of the sixteenth convolutional layer to obtain the multiplication result c1.
[0040] Then add the multiplication result b1 to the multiplication result c1, and use the sum as the output of the attention-based fusion module.
[0041] Furthermore, the working process of the first feature extractor is as follows:
[0042] Within the first feature extractor, the inputs of the first feature extractor are used as the inputs of the first branch and the second branch, respectively, where:
[0043] The first branch includes, in sequence, the twentieth convolutional layer with a kernel size of 3×3, the first ReLU activation function layer, the twenty-first convolutional layer with a kernel size of 3×3, the second ReLU activation function layer, the twenty-second convolutional layer with a kernel size of 3×3, and the third ReLU activation function layer. That is, the output of the third ReLU activation function layer is used as the output of the first branch.
[0044] The second branch includes a 23rd convolutional layer with a kernel size of 3×3 and a 24th convolutional layer with a kernel size of 3×3, with the output of the 24th convolutional layer serving as the output of the second branch.
[0045] Add the outputs of the first and second branches, and use the sum as the inputs to the third and fourth branches, respectively.
[0046] The third branch includes, in sequence, the 25th convolutional layer with a kernel size of 3×3, the 4th ReLU activation function layer, the 26th convolutional layer with a kernel size of 3×3, the 5th ReLU activation function layer, the 27th convolutional layer with a kernel size of 3×3, and the 6th ReLU activation function layer. The output of the 6th ReLU activation function layer is used as the output of the third branch.
[0047] The fourth branch includes the twenty-eighth convolutional layer with a kernel size of 3×3 and the twenty-ninth convolutional layer with a kernel size of 3×3, with the output of the twenty-ninth convolutional layer serving as the output of the fourth branch.
[0048] The outputs of the third branch and the fourth branch are added together, and the sum is used as the output of the first feature extractor.
[0049] Furthermore, the action recognition network includes T spatiotemporal feature extraction modules, and the working process of the action recognition network is as follows:
[0050] The fused image at the t-th acquisition time point is used as the input of the t-th spatiotemporal feature extraction module, where t = 1, 2, ..., T; and the output of the 1st spatiotemporal feature extraction module is used as the input of the 1st GRU module.
[0051] The output of the t-th spatiotemporal feature extraction module and the output of the (t-1)-th GRU module are used as the input of the t-th GRU module;
[0052] The outputs of the first GRU module to the Tth GRU module are averaged, and the averaged result is then passed through a fully connected layer. Finally, the output of the fully connected layer is used as the input of the Softmax layer, and the action recognition result is output through the Softmax layer.
[0053] Furthermore, the working process of the first spatiotemporal feature extraction module is as follows:
[0054] In the first spatiotemporal feature extraction module, the input fused image is first used as the input of the thirtieth convolutional layer, then the output of the thirtieth convolutional layer is used as the input of the first spatiotemporal feature extraction unit, then the output of the first spatiotemporal feature extraction unit is used as the input of the second spatiotemporal feature extraction unit, and then the output of the second spatiotemporal feature extraction unit is used as the input of the third spatiotemporal feature extraction unit.
[0055] The output of the thirtieth convolutional layer is added to the output of the third spatiotemporal feature extraction unit, and the result is used as the output of the first spatiotemporal feature extraction module.
[0056] Furthermore, the working process of the first spatiotemporal feature extraction unit is as follows:
[0057] Within the first spatiotemporal feature extraction unit, the input of the first spatiotemporal feature extraction unit is used as the input of the thirty-first convolutional layer, and the output of the thirty-first convolutional layer is used as the input of the first BN layer, and the output of the first BN layer is used as the input of the seventh ReLU activation function layer.
[0058] The output of the seventh ReLU activation function layer is used as the input of the thirty-second convolutional layer, the output of the thirty-second convolutional layer is used as the input of the second BN layer, the output of the second BN layer is used as the input of the eighth ReLU activation function layer, and the output of the eighth ReLU activation function layer is used as the input of the first channel attention unit.
[0059] The input of the first spatiotemporal feature extraction unit is used as the input of the thirty-third convolutional layer, the output of the thirty-third convolutional layer is used as the input of the third BN layer, and the output of the third BN layer is used as the input of the ninth ReLU activation function layer.
[0060] The output of the ninth ReLU activation function layer is used as the input of the thirty-fourth convolutional layer, the output of the thirty-fourth convolutional layer is used as the input of the fourth BN layer, and the output of the fourth BN layer is used as the input of the tenth ReLU activation function layer.
[0061] The input of the first spatiotemporal feature extraction unit is used as the input of the thirty-fifth convolutional layer, and the output of the thirty-fifth convolutional layer is used as the input of the fifth BN layer. The output of the fifth BN layer is used as the input of the eleventh ReLU activation function layer.
[0062] The output of the eleventh ReLU activation function layer is used as the input of the thirty-sixth convolutional layer, the output of the thirty-sixth convolutional layer is used as the input of the sixth BN layer, the output of the sixth BN layer is used as the input of the twelfth ReLU activation function layer, and the output of the twelfth ReLU activation function layer is used as the input of the third spatial attention unit.
[0063] Multiply the output of the tenth ReLU activation function layer with the output of the first channel attention unit to obtain the multiplication result e;
[0064] The output of the tenth ReLU activation function layer is multiplied by the output of the third spatial attention unit to obtain the multiplication result f;
[0065] Add the product e to the product f to get the sum g;
[0066] The output of the third spatial attention unit is multiplied by the output of the first channel attention unit to obtain the multiplication result h;
[0067] Add g and h to obtain the sum k, and use the sum k as the output of the first spatiotemporal feature extraction unit.
[0068] Furthermore, the extraction of the human skeleton from the point cloud data obtained at each acquisition time point specifically involves:
[0069] Taking point cloud data obtained at any acquisition time point as an example
[0070] Step 1: Initialize the number of iterations n = 1;
[0071] Step 2: Initialize l = 1;
[0072] Step 3: For the l-th point in the point cloud data, calculate the search radius r centered on the l-th point:
[0073]
[0074] Among them, l k Let dist(l,l) be the k-th neighbor of point l. k ) represents the relationship between point l and point l k The distance, k = 1, 2, ..., K;
[0075] Search for a set of points within a region centered at point l and with radius r, calculate the center of all points within the set, and then move point l to the calculated center position.
[0076] Step 4: Determine whether l has traversed all points in the point cloud data;
[0077] If not all points in the point cloud data have been traversed, then let l = l + 1 and return to step three;
[0078] If all points in the point cloud data have been traversed, proceed to step five;
[0079] Step 5: Determine if n = N, where N represents the maximum number of iterations.
[0080] If n < N, then let n = n + 1 and return to step two;
[0081] If n = N, then continue to return to step six;
[0082] Step 6: Extract the human skeleton from the point cloud data obtained after the Nth iteration.
[0083] Furthermore, the extracted skeleton is smoothed to obtain the human skeleton corresponding to each acquisition time point; the specific process is as follows:
[0084]
[0085] Where, x t′,j Let x represent the coordinates of the j-th joint in the human skeleton at the t'-th acquisition time point. t′,j This represents the smoothed coordinates of the j-th joint in the human skeleton corresponding to the t'-th acquisition time point.
[0086] Furthermore, the specific process of step 5 is as follows:
[0087] Step 51: Calculate the direction vector of each joint based on the coordinates of each joint point of the human skeleton, and obtain the standard movements from the database based on the martial arts movement recognition results in Step 3.
[0088] Step 52: Calculate the cosine similarity between the joint direction vector calculated in Step 51 and the direction vector of the standard movement;
[0089] Step 53: Obtain the final evaluation result of the martial arts movements based on the calculation results of Step 52.
[0090] The beneficial effects of this invention are:
[0091] This invention combines visible light and infrared images to identify martial arts movements, allowing for more comprehensive information to be considered during the recognition process. By acquiring visible light and infrared images at multiple consecutive time points, the continuous nature of martial arts movements can be perfectly matched. Through the fusion of various image information and the combined application of image data from multiple acquisition time points, this invention ensures the accuracy of martial arts movement recognition. Based on the martial arts movement recognition results, the standard movements to be compared with the point cloud data can be determined. Then, the skeleton extraction method designed in this invention is used to process the point cloud data, accurately extracting the human skeleton. The extracted human skeleton is then compared with the standard movements to improve the accuracy of evaluating the quality of martial arts movements. Attached Figure Description
[0092] Figure 1 This is a flowchart of a martial arts movement recognition and movement quality evaluation method based on artificial intelligence according to the present invention;
[0093] Figure 2(a) is a visible light image;
[0094] Figure 2(b) is the infrared image corresponding to Figure 2(a);
[0095] Figure 2(c) shows the point cloud data corresponding to Figure 2(a);
[0096] Figure 2(d) is the human skeleton diagram corresponding to Figure 2(c);
[0097] Figure 3(a) is the second visible light image;
[0098] Figure 3(b) is the infrared image corresponding to Figure 3(a);
[0099] Figure 3(c) shows the point cloud data corresponding to Figure 3(a);
[0100] Figure 3(d) is the human skeleton diagram corresponding to Figure 3(c);
[0101] Figure 4(a) is a visible light image;
[0102] Figure 4(b) is the infrared image corresponding to Figure 4(a);
[0103] Figure 4(c) shows the point cloud data corresponding to Figure 4(a);
[0104] Figure 4(d) is the human skeleton diagram corresponding to Figure 4(c). Detailed Implementation
[0105] Specific implementation method one: Combining Figure 1 This embodiment describes an artificial intelligence-based method for martial arts movement recognition and quality evaluation, which specifically includes the following steps:
[0106] Step 1: At each acquisition time point, visible light images, infrared images, and point cloud data of martial arts movements are acquired simultaneously.
[0107] Since martial arts movements are not instantaneous but continuous over time, this invention sets multiple consecutive acquisition time points within the duration of a martial arts movement, and acquires visible light images, infrared images, and point cloud data at each acquisition time point.
[0108] Step 2: For any time point t, the visible light image and infrared image acquired synchronously are used as inputs to the image fusion network. The working process within the image fusion network is as follows:
[0109] The infrared image is used as the input to the first convolutional layer with a kernel size of 3×3, and the output of the first convolutional layer is used as the input to the second convolutional layer with a kernel size of 1×1.
[0110] The output of the second convolutional layer is used as the input of the first max pooling layer, and the output of the first max pooling layer is used as the input of the third convolutional layer with a kernel size of 3×3.
[0111] The output of the third convolutional layer is used as the input of the fourth convolutional layer with a kernel size of 1×1. The output of the first max pooling layer is added to the output of the fourth convolutional layer, and the result of the addition is used as the input of the second max pooling layer.
[0112] The output of the second max pooling layer is used as the input of the fifth convolutional layer with a kernel size of 3×3, and the output of the fifth convolutional layer is used as the input of the sixth convolutional layer with a kernel size of 1×1.
[0113] The output of the second max pooling layer is added to the output of the sixth convolutional layer, and the result is used as the input of the third max pooling layer.
[0114] The output of the third max pooling layer is used as the input of the seventh convolutional layer with a kernel size of 3×3, and the output of the seventh convolutional layer is used as the input of the eighth convolutional layer with a kernel size of 1×1.
[0115] The visible light image is used as the input to the ninth convolutional layer with a kernel size of 3×3, and the output of the ninth convolutional layer is used as the input to the tenth convolutional layer with a kernel size of 1×1.
[0116] The output of the tenth convolutional layer is used as the input of the fourth max pooling layer, and the output of the fourth max pooling layer is used as the input of the eleventh convolutional layer with a kernel size of 3×3.
[0117] The output of the eleventh convolutional layer is used as the input of the twelfth convolutional layer with a kernel size of 1×1. The output of the fourth max pooling layer is added to the output of the twelfth convolutional layer, and the result of the addition is used as the input of the fifth max pooling layer.
[0118] The output of the fifth max pooling layer is used as the input of the thirteenth convolutional layer with a kernel size of 3×3, and the output of the thirteenth convolutional layer is used as the input of the fourteenth convolutional layer with a kernel size of 1×1.
[0119] The output of the fifth max pooling layer is added to the output of the fourteenth convolutional layer, and the result is used as the input of the sixth max pooling layer.
[0120] The output of the sixth max pooling layer is used as the input of the fifteenth convolutional layer with a kernel size of 3×3, and the output of the fifteenth convolutional layer is used as the input of the sixteenth convolutional layer with a kernel size of 1×1.
[0121] The outputs of the eighth and sixteenth convolutional layers are used as inputs to the attention-based fusion module. The working process of the attention-based fusion module is as follows:
[0122] The output of the eighth convolutional layer is weighted and summed with the output of the sixteenth convolutional layer to obtain the weighted sum result a;
[0123] The output of the eighth convolutional layer is concatenated with the weighted summation result a to obtain the concatenation result a1;
[0124] The connection result a1 is used as the input of the eighteenth convolutional layer with a kernel size of 1×1, and the output of the eighteenth convolutional layer is used as the input of the first sigmoid activation function layer.
[0125] The connection result a1 is used as the input of the first spatial attention unit. The output of the first sigmoid activation function layer is added to the output of the first spatial attention unit to obtain the sum result b. Then, the sum result b is multiplied by the output of the eighth convolutional layer to obtain the multiplication result b1.
[0126] The output of the sixteenth convolutional layer is concatenated with the weighted summation result a to obtain the concatenation result a2.
[0127] The connection result a2 is used as the input of the nineteenth convolutional layer with a kernel size of 1×1, and the output of the nineteenth convolutional layer is used as the input of the second sigmoid activation function layer.
[0128] The connection result a2 is used as the input of the second spatial attention unit. The output of the second sigmoid activation function layer is added to the output of the second spatial attention unit to obtain the sum result c. Then, the sum result c is multiplied by the output of the sixteenth convolutional layer to obtain the multiplication result c1.
[0129] Then add the multiplication result b1 to the multiplication result c1, and use the sum as the output of the attention-based fusion module;
[0130] The output of the attention-based fusion module is used as the input of the seventeenth convolutional layer with a kernel size of 1×1, and the output of the seventeenth convolutional layer is used as the input of the first feature extractor (the first feature extractor can extract the common features of infrared images and visible light images).
[0131] Infrared images are used as input to a second feature extractor (which extracts features specific to infrared images), and visible light images are used as input to a third feature extractor (which extracts features specific to visible light images). The outputs of the first, second, and third feature extractors are used as input to a CNN network. The binary image output by the CNN network is used as a fused image of the infrared and visible light images (in the binary image, the pixel value corresponding to the human body region is 1, and the pixel value corresponding to the background region is 0). When performing subsequent classification tasks based on the fused image, since the fused image already reflects the accurate position and posture of the human body, the accuracy of martial arts movement recognition can be significantly improved by using fused images from multiple consecutive time points.
[0132] The working process of the first feature extractor is as follows:
[0133] Within the first feature extractor, the inputs of the first feature extractor are used as the inputs of the first branch and the second branch, respectively, where:
[0134] The first branch includes, in sequence, the twentieth convolutional layer with a kernel size of 3×3, the first ReLU activation function layer, the twenty-first convolutional layer with a kernel size of 3×3, the second ReLU activation function layer, the twenty-second convolutional layer with a kernel size of 3×3, and the third ReLU activation function layer. That is, the output of the third ReLU activation function layer is used as the output of the first branch.
[0135] The second branch includes a 23rd convolutional layer with a kernel size of 3×3 and a 24th convolutional layer with a kernel size of 3×3, with the output of the 24th convolutional layer serving as the output of the second branch.
[0136] Add the outputs of the first and second branches, and use the sum as the inputs to the third and fourth branches, respectively.
[0137] The third branch includes, in sequence, the 25th convolutional layer with a kernel size of 3×3, the 4th ReLU activation function layer, the 26th convolutional layer with a kernel size of 3×3, the 5th ReLU activation function layer, the 27th convolutional layer with a kernel size of 3×3, and the 6th ReLU activation function layer. The output of the 6th ReLU activation function layer is used as the output of the third branch.
[0138] The fourth branch includes the twenty-eighth convolutional layer with a kernel size of 3×3 and the twenty-ninth convolutional layer with a kernel size of 3×3, with the output of the twenty-ninth convolutional layer serving as the output of the fourth branch.
[0139] The outputs of the third branch and the fourth branch are added together, and the sum is used as the output of the first feature extractor.
[0140] In this invention, the structures of the second feature extractor and the third feature extractor are the same as those of the first feature extractor.
[0141] Step 3: Use the fused image from the first acquisition time point to the fused image from the Tth acquisition time point as input to the action recognition network, and output the martial arts action recognition result through the action recognition network;
[0142] The action recognition network includes T spatiotemporal feature extraction modules. The working process of the action recognition network is as follows:
[0143] The fused image at the t-th acquisition time point is used as the input of the t-th spatiotemporal feature extraction module, where t = 1, 2, ..., T; and the output of the 1st spatiotemporal feature extraction module is used as the input of the 1st GRU module.
[0144] The working process of the first spatiotemporal feature extraction module is as follows:
[0145] In the first spatiotemporal feature extraction module, the input fused image is first used as the input of the thirtieth convolutional layer, and then the output of the thirtieth convolutional layer is used as the input of the first spatiotemporal feature extraction unit. In the first spatiotemporal feature extraction unit, the input of the first spatiotemporal feature extraction unit is used as the input of the thirty-first convolutional layer, and then the output of the thirty-first convolutional layer is used as the input of the first BN layer. The output of the first BN layer is used as the input of the seventh ReLU activation function layer.
[0146] The output of the seventh ReLU activation function layer is used as the input of the thirty-second convolutional layer, the output of the thirty-second convolutional layer is used as the input of the second BN layer, the output of the second BN layer is used as the input of the eighth ReLU activation function layer, and the output of the eighth ReLU activation function layer is used as the input of the first channel attention unit.
[0147] The input of the first spatiotemporal feature extraction unit is used as the input of the thirty-third convolutional layer, the output of the thirty-third convolutional layer is used as the input of the third BN layer, and the output of the third BN layer is used as the input of the ninth ReLU activation function layer.
[0148] The output of the ninth ReLU activation function layer is used as the input of the thirty-fourth convolutional layer, the output of the thirty-fourth convolutional layer is used as the input of the fourth BN layer, and the output of the fourth BN layer is used as the input of the tenth ReLU activation function layer.
[0149] The input of the first spatiotemporal feature extraction unit is used as the input of the thirty-fifth convolutional layer, and the output of the thirty-fifth convolutional layer is used as the input of the fifth BN layer. The output of the fifth BN layer is used as the input of the eleventh ReLU activation function layer.
[0150] The output of the eleventh ReLU activation function layer is used as the input of the thirty-sixth convolutional layer, the output of the thirty-sixth convolutional layer is used as the input of the sixth BN layer, the output of the sixth BN layer is used as the input of the twelfth ReLU activation function layer, and the output of the twelfth ReLU activation function layer is used as the input of the third spatial attention unit.
[0151] Multiply the output of the tenth ReLU activation function layer with the output of the first channel attention unit to obtain the multiplication result e;
[0152] The output of the tenth ReLU activation function layer is multiplied by the output of the third spatial attention unit to obtain the multiplication result f;
[0153] Add the product e to the product f to get the sum g;
[0154] The output of the third spatial attention unit is multiplied by the output of the first channel attention unit to obtain the multiplication result h;
[0155] Add g and h to obtain the sum k, and use the sum k as the output of the first spatiotemporal feature extraction unit.
[0156] The output of the first spatiotemporal feature extraction unit is then used as the input of the second spatiotemporal feature extraction unit, and the output of the second spatiotemporal feature extraction unit is used as the input of the third spatiotemporal feature extraction unit.
[0157] The output of the thirtieth convolutional layer is added to the output of the third spatiotemporal feature extraction unit, and the result is used as the output of the first spatiotemporal feature extraction module. Through the continuous processing of three spatiotemporal feature extraction units, this invention can extract spatiotemporal features at different scales, ensuring the accuracy of martial arts movement recognition.
[0158] The output of the t-th spatiotemporal feature extraction module and the output of the (t-1)-th GRU module are used as the input of the t-th GRU module;
[0159] The outputs of the first to the Tth GRU modules are averaged, and the average result is then passed through a fully connected layer. Finally, the output of the fully connected layer is used as the input to the Softmax layer, which outputs the action recognition result. This invention performs martial arts action recognition based on multiple images during a continuous action process, which can further ensure the accuracy of martial arts action recognition.
[0160] Step 4: Extract the human skeleton from the point cloud data obtained at each acquisition time point, and smooth the extracted skeleton to obtain the human skeleton corresponding to each acquisition time point. Specifically:
[0161] Taking point cloud data obtained at any acquisition time point as an example
[0162] Step 1: Initialize the number of iterations n = 1;
[0163] Step 2: Initialize l = 1;
[0164] Step 3: For the l-th point in the point cloud data, calculate the search radius r centered on the l-th point:
[0165]
[0166] Among them, l k Let dist(l,l) be the k-th neighbor of point l. k ) represents the relationship between point l and point l k The distance, k = 1, 2, ..., K;
[0167] In this invention, the distance between each of the remaining points and the l-th point is calculated, and then the K points with the smallest distance are selected. The selected points are then used to calculate the search radius.
[0168] Search for a set of points within a region centered at point l and with radius r, calculate the center of all points within the set, and then move point l to the calculated center position.
[0169] Step 4: Determine whether l has traversed all points in the point cloud data;
[0170] If not all points in the point cloud data have been traversed, then let l = l + 1 and return to step three;
[0171] If all points in the point cloud data have been traversed, proceed to step five;
[0172] Step 5: Determine if n = N, where N represents the maximum number of iterations.
[0173] If n < N, then let n = n + 1 and return to step two;
[0174] If n = N, then continue to return to step six;
[0175] Step Six: By moving the point cloud data as close as possible to the final skeleton, the accuracy of human skeleton extraction can be improved. The specific process of smoothing the extracted skeleton is as follows:
[0176]
[0177] Where, x t′,j Let x represent the coordinates of the j-th joint in the human skeleton at the t'-th acquisition time point. t′,j This represents the smoothed coordinates of the j-th joint in the human skeleton corresponding to the t'-th acquisition time point.
[0178] By smoothing the joints of the human skeleton, the joint coordinate error caused by single-frame point cloud data can be minimized, thereby improving the accuracy of martial arts movement evaluation.
[0179] Step 5: Evaluate the quality of the martial arts movements based on the final human skeleton obtained in Step 4 and the martial arts movement recognition results in Step 3.
[0180] Step 51: Calculate the direction vector of each joint based on the coordinates of each joint point of the human skeleton, and obtain the standard movements from the database based on the martial arts movement recognition results in Step 3.
[0181] Step 52: Calculate the cosine similarity between the joint direction vector calculated in Step 51 and the direction vector of the standard movement;
[0182] Step 53: Obtain the final evaluation result of the martial arts movements based on the calculation results of Step 52.
[0183] In martial arts movements, to ensure the standardization of the movements, fixed angles need to be formed between joints. Therefore, this invention uses direction as the primary evaluation index. A movement at a single point in time may correspond to multiple joint direction vectors; in this case, the average cosine similarity of the multiple joint direction vectors can be used as the evaluation index. The entire recognition and evaluation method of this invention requires no human intervention and can automatically output the recognition and evaluation results of martial arts movements.
[0184] Experimental Section
[0185] Figures 2(a), 2(b), and 2(c) show the first set of visible light images, infrared images, and point cloud data collected, with the extracted human skeleton shown in Figure 2(d). Figures 3(a), 3(b), and 3(c) show the second set of visible light images, infrared images, and point cloud data collected, with the extracted human skeleton shown in Figure 3(d). Figures 4(a), 4(b), and 4(c) show the third set of visible light images, infrared images, and point cloud data collected, with the extracted human skeleton shown in Figure 4(d).
[0186] The above examples of the present invention are merely illustrative of the computational model and process of the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is impossible to exhaustively list all possible implementations here. Any obvious variations or modifications derived from the technical solutions of the present invention are still within the scope of protection of the present invention.
Claims
1. A method for martial arts movement recognition and movement quality evaluation based on artificial intelligence, characterized in that, The method specifically includes the following steps: Step 1: At each acquisition time point, visible light images, infrared images, and point cloud data of martial arts movements are acquired simultaneously. Step 2: For any given time point Visible light and infrared images were acquired simultaneously at the location, and the time points were obtained by fusing the visible light and infrared images. The corresponding fused image; Similarly, the fused image corresponding to each acquisition time point is obtained; The time point is obtained by fusing visible light images and infrared images. The corresponding fused image; the specific process is as follows: time point The visible light and infrared images acquired simultaneously are used as inputs to the image fusion network. The working process within the image fusion network is as follows: The infrared image is used as the input to the first convolutional layer with a kernel size of 3×3, and the output of the first convolutional layer is used as the input to the second convolutional layer with a kernel size of 1×1. The output of the second convolutional layer is used as the input of the first max pooling layer, and the output of the first max pooling layer is used as the input of the third convolutional layer with a kernel size of 3×3. The output of the third convolutional layer is used as the input of the fourth convolutional layer with a kernel size of 1×1. The output of the first max pooling layer is added to the output of the fourth convolutional layer, and the result of the addition is used as the input of the second max pooling layer. The output of the second max pooling layer is used as the input of the fifth convolutional layer with a kernel size of 3×3, and the output of the fifth convolutional layer is used as the input of the sixth convolutional layer with a kernel size of 1×1. The output of the second max pooling layer is added to the output of the sixth convolutional layer, and the result is used as the input of the third max pooling layer. The output of the third max pooling layer is used as the input of the seventh convolutional layer with a kernel size of 3×3, and the output of the seventh convolutional layer is used as the input of the eighth convolutional layer with a kernel size of 1×1. The visible light image is used as the input to the ninth convolutional layer with a kernel size of 3×3, and the output of the ninth convolutional layer is used as the input to the tenth convolutional layer with a kernel size of 1×1. The output of the tenth convolutional layer is used as the input of the fourth max pooling layer, and the output of the fourth max pooling layer is used as the input of the eleventh convolutional layer with a kernel size of 3×3. The output of the eleventh convolutional layer is used as the input of the twelfth convolutional layer with a kernel size of 1×1. The output of the fourth max pooling layer is added to the output of the twelfth convolutional layer, and the result of the addition is used as the input of the fifth max pooling layer. The output of the fifth max pooling layer is used as the input of the thirteenth convolutional layer with a kernel size of 3×3, and the output of the thirteenth convolutional layer is used as the input of the fourteenth convolutional layer with a kernel size of 1×1. The output of the fifth max pooling layer is added to the output of the fourteenth convolutional layer, and the result is used as the input of the sixth max pooling layer. The output of the sixth max pooling layer is used as the input of the fifteenth convolutional layer with a kernel size of 3×3, and the output of the fifteenth convolutional layer is used as the input of the sixteenth convolutional layer with a kernel size of 1×1. The outputs of the eighth and sixteenth convolutional layers are used as inputs to the attention-based fusion module. The output of the attention-based fusion module is used as input to the seventeenth convolutional layer with a kernel size of 1×1. The output of the seventeenth convolutional layer is then used as input to the first feature extractor. The infrared image is used as the input to the second feature extractor, the visible light image is used as the input to the third feature extractor, the outputs of the first feature extractor, the second feature extractor, and the third feature extractor are used as the input to the CNN network, and the binary image output by the CNN network is used as the fused image of the infrared image and the visible light image. The working process of the attention-based fusion module is as follows: The output of the eighth convolutional layer is weighted and summed with the output of the sixteenth convolutional layer to obtain the weighted sum result a; The output of the eighth convolutional layer is concatenated with the weighted summation result a to obtain the concatenation result a1; The connection result a1 is used as the input of the eighteenth convolutional layer with a kernel size of 1×1, and the output of the eighteenth convolutional layer is used as the input of the first sigmoid activation function layer. The connection result a1 is used as the input of the first spatial attention unit. The output of the first sigmoid activation function layer is added to the output of the first spatial attention unit to obtain the sum result b. Then, the sum result b is multiplied by the output of the eighth convolutional layer to obtain the multiplication result b1. The output of the sixteenth convolutional layer is concatenated with the weighted summation result a to obtain the concatenation result a2; The connection result a2 is used as the input of the nineteenth convolutional layer with a kernel size of 1×1, and the output of the nineteenth convolutional layer is used as the input of the second sigmoid activation function layer. The connection result a2 is used as the input of the second spatial attention unit. The output of the second sigmoid activation function layer is added to the output of the second spatial attention unit to obtain the sum result c. Then, the sum result c is multiplied by the output of the sixteenth convolutional layer to obtain the multiplication result c1. Then add the multiplication result b1 to the multiplication result c1, and use the sum as the output of the attention-based fusion module; Step 3, place the first The fused images at the acquisition time points up to the first... The fused images from each acquisition time point are used as input to the action recognition network, which then outputs martial arts action recognition results. Step 4: Extract the human skeleton from the point cloud data obtained at each acquisition time point, and smooth the extracted skeleton to obtain the human skeleton corresponding to each acquisition time point. Step 5: Evaluate the quality of the martial arts movements based on the final human skeleton obtained in Step 4 and the martial arts movement recognition results in Step 3.
2. The method for martial arts movement recognition and movement quality evaluation based on artificial intelligence according to claim 1, characterized in that, The working process of the first feature extractor is as follows: Within the first feature extractor, the inputs of the first feature extractor are used as the inputs of the first branch and the second branch, respectively, where: The first branch includes, in sequence, the twentieth convolutional layer with a kernel size of 3×3, the first ReLU activation function layer, the twenty-first convolutional layer with a kernel size of 3×3, the second ReLU activation function layer, the twenty-second convolutional layer with a kernel size of 3×3, and the third ReLU activation function layer. That is, the output of the third ReLU activation function layer is used as the output of the first branch. The second branch includes a 23rd convolutional layer with a kernel size of 3×3 and a 24th convolutional layer with a kernel size of 3×3, with the output of the 24th convolutional layer serving as the output of the second branch. Add the outputs of the first and second branches, and use the sum as the inputs to the third and fourth branches, respectively. The third branch includes, in sequence, the 25th convolutional layer with a kernel size of 3×3, the 4th ReLU activation function layer, the 26th convolutional layer with a kernel size of 3×3, the 5th ReLU activation function layer, the 27th convolutional layer with a kernel size of 3×3, and the 6th ReLU activation function layer. The output of the 6th ReLU activation function layer is used as the output of the third branch. The fourth branch includes the twenty-eighth convolutional layer with a kernel size of 3×3 and the twenty-ninth convolutional layer with a kernel size of 3×3, with the output of the twenty-ninth convolutional layer serving as the output of the fourth branch. The outputs of the third branch and the fourth branch are added together, and the sum is used as the output of the first feature extractor.
3. The method for martial arts movement recognition and movement quality evaluation based on artificial intelligence according to claim 2, characterized in that, The action recognition network includes T spatiotemporal feature extraction modules. The working process of the action recognition network is as follows: The first The fused image at the acquisition time point is used as the first... The input to each spatiotemporal feature extraction module, =1,2,…,T; and use the output of the first spatiotemporal feature extraction module as the input of the first GRU module; The first The output of the first spatiotemporal feature extraction module and the first The output of -1 GRU module is used as the first... The input of each GRU module; The outputs of the first GRU module to the Tth GRU module are averaged, and the averaged result is then passed through a fully connected layer. Finally, the output of the fully connected layer is used as the input of the Softmax layer, and the action recognition result is output through the Softmax layer.
4. The method for martial arts movement recognition and movement quality evaluation based on artificial intelligence according to claim 3, characterized in that, The working process of the first spatiotemporal feature extraction module is as follows: In the first spatiotemporal feature extraction module, the input fused image is first used as the input of the thirtieth convolutional layer, then the output of the thirtieth convolutional layer is used as the input of the first spatiotemporal feature extraction unit, then the output of the first spatiotemporal feature extraction unit is used as the input of the second spatiotemporal feature extraction unit, and then the output of the second spatiotemporal feature extraction unit is used as the input of the third spatiotemporal feature extraction unit. The output of the thirtieth convolutional layer is added to the output of the third spatiotemporal feature extraction unit, and the result is used as the output of the first spatiotemporal feature extraction module.
5. The method for martial arts movement recognition and movement quality evaluation based on artificial intelligence according to claim 4, characterized in that, The working process of the first spatiotemporal feature extraction unit is as follows: Within the first spatiotemporal feature extraction unit, the input of the first spatiotemporal feature extraction unit is used as the input of the thirty-first convolutional layer, and the output of the thirty-first convolutional layer is used as the input of the first BN layer, and the output of the first BN layer is used as the input of the seventh ReLU activation function layer. The output of the seventh ReLU activation function layer is used as the input of the thirty-second convolutional layer, the output of the thirty-second convolutional layer is used as the input of the second BN layer, the output of the second BN layer is used as the input of the eighth ReLU activation function layer, and the output of the eighth ReLU activation function layer is used as the input of the first channel attention unit. The input of the first spatiotemporal feature extraction unit is used as the input of the thirty-third convolutional layer, the output of the thirty-third convolutional layer is used as the input of the third BN layer, and the output of the third BN layer is used as the input of the ninth ReLU activation function layer. The output of the ninth ReLU activation function layer is used as the input of the thirty-fourth convolutional layer, the output of the thirty-fourth convolutional layer is used as the input of the fourth BN layer, and the output of the fourth BN layer is used as the input of the tenth ReLU activation function layer. The input of the first spatiotemporal feature extraction unit is used as the input of the thirty-fifth convolutional layer, and the output of the thirty-fifth convolutional layer is used as the input of the fifth BN layer. The output of the fifth BN layer is used as the input of the eleventh ReLU activation function layer. The output of the eleventh ReLU activation function layer is used as the input of the thirty-sixth convolutional layer, the output of the thirty-sixth convolutional layer is used as the input of the sixth BN layer, the output of the sixth BN layer is used as the input of the twelfth ReLU activation function layer, and the output of the twelfth ReLU activation function layer is used as the input of the third spatial attention unit. Multiply the output of the tenth ReLU activation function layer with the output of the first channel attention unit to obtain the multiplication result e; The output of the tenth ReLU activation function layer is multiplied by the output of the third spatial attention unit to obtain the multiplication result f; Add the product e to the product f to get the sum g; The output of the third spatial attention unit is multiplied by the output of the first channel attention unit to obtain the multiplication result h; Add g and h to obtain the sum k, and use the sum k as the output of the first spatiotemporal feature extraction unit.
6. The method for martial arts movement recognition and movement quality evaluation based on artificial intelligence according to claim 5, characterized in that, The extraction of the human skeleton from the point cloud data obtained at each acquisition time point is specifically as follows: Taking point cloud data obtained at any acquisition time point as an example Step 1: Initialize the number of iterations ; Step 2: Initialization ; Step 3: For the first point cloud data... The point is calculated with the first point as the basis. Search radius centered on point : in, Point The Neighboring points, Point With point distance, ; Search by point Centered on, with Given a set of points within a radius, calculate the center of all points within the set, and then... Move to the calculated center position; Step 4: Judgment Whether to iterate through all points in the point cloud data; If not all points in the point cloud data have been traversed, then let Return to step three; If all points in the point cloud data have been traversed, proceed to step five; Step 5: Determine if the condition is met. , This indicates the maximum number of iterations set. like Then let Return to step two; like If so, proceed to step six. Step Six: For the first The point cloud data obtained after the next iteration is used to extract the human skeleton.
7. The method for martial arts movement recognition and movement quality evaluation based on artificial intelligence according to claim 6, characterized in that, The extracted skeleton is smoothed to obtain the human skeleton corresponding to each acquisition time point; the specific process is as follows: in, Indicates the first The first human skeleton corresponding to the first collection time point Coordinates of each joint point Indicates the first The first human skeleton corresponding to the first collection time point The coordinates of each joint after smoothing.
8. The method for martial arts movement recognition and movement quality evaluation based on artificial intelligence according to claim 7, characterized in that, The specific process of step 5 is as follows: Step 51: Calculate the direction vector of each joint based on the coordinates of each joint point of the human skeleton, and obtain the standard movements from the database based on the martial arts movement recognition results in Step 3. Step 52: Calculate the cosine similarity between the joint direction vector calculated in Step 51 and the direction vector of the standard movement; Step 53: Obtain the final evaluation result of the martial arts movements based on the calculation results of Step 52.