Lightweight shadowboxing scoring method based on attention mechanism improvement and related equipment

Abstract

The invention provides an attention mechanism improvement-based lightweight Taijiquan scoring method and related equipment, and the method comprises the steps: carrying out the preprocessing of a Taijiquan video, inputting the preprocessed Taijiquan video into a skeleton point extraction network, extracting the skeleton point features in each frame image, and obtaining a skeleton point sequence file; inputting the skeleton point sequence file into a long short-term memory network, identifying a Tai Chi style, distributing a weight for each skeleton point, and generating a weighted skeleton point sequence; obtaining a standard shadowboxing video, and calculating the score of the shadowboxing video based on the standard shadowboxing video; with the adoption of the method, the time sequence dependence in the shadowboxing type can be captured by utilizing the long-short-term memory network, and the weight is distributed to each skeleton point by utilizing the attention mechanism, so that the system can dynamically adjust the focus point according to the characteristics of different actions, and the accuracy of action recognition and analysis is improved; and the accuracy and objectivity of shadowboxing scoring are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of action recognition, and more specifically, to a lightweight Tai Chi scoring method and related equipment based on an improved attention mechanism. Background Art

[0002] Tai Chi is a treasure of Chinese traditional culture. It is a traditional sport with a wide audience demand in China. It has a deep mass base and is one of the important ways for the people to keep fit. In the practice of Tai Chi, objective scoring can help Tai Chi practitioners get specific feedback and understand their own shortcomings, so as to make targeted improvements and improvements to better understand and master the essence of Tai Chi and its moves. At the same time, Tai Chi scoring can assist referees in scoring in martial arts competitions as a scoring reference, which can improve scoring efficiency and reduce human errors. In addition, scoring helps to maintain the standardization and normalization of Tai Chi movements, and helps to inherit and promote Tai Chi culture. Therefore, the standardized scoring of Tai Chi movements is of great significance.

[0003] However, the current scoring and judging of Tai Chi mainly relies on the personal experience of the referee or coach to score the movements. The evaluation system lacks systematicity and objectivity, which will inevitably lead to scoring bias. Therefore, in the face of the problems existing in Tai Chi in public participation and competition, it is necessary to design an intelligent movement quality analysis method and give trainees reasonable feedback to improve the posture of the movements.

[0004] The current Tai Chi scoring algorithm often considers factors that are too general and single. The Tai Chi action scoring dimensions should at least include standardization, coherence, coordination, breath, etc. At the same time, the focus areas should also be different for different moves.

[0005] The Chinese invention patent application (patent number CN118053201A) discloses a Tai Chi action scoring method based on deep learning, a storage medium and an electronic device. It uses a human 3D key point detection model to process the video stream data of Tai Chi movement, obtain the coordinates of the key points of the human skeleton, and then use the time series to normalize the coordinates of the key points of the human skeleton. The adjacency relationship of the key points of the human body is constructed through the adjacency matrix, and finally input into the ST-GCNbackbone network, and then input into the classification head and regression head respectively. The classification head outputs the Tai Chi action classification; the regression head will output the score value of the corresponding Tai Chi action; provide a more accurate sports human posture score evaluation; so that the rhythm and posture of each Tai Chi action have a corresponding score evaluation.

[0006] However, in this invention patent application, the traditional deep learning feature extraction algorithm takes into account only a few factors, and the effect of directly extracting Tai Chi action features is poor. In addition, the extracted video features are poorly interpretable and difficult to be directly used for user feedback. In addition, the GCN spatiotemporal feature extraction algorithm has low computational efficiency and poor real-time performance. Summary of the invention

[0007] In view of the shortcomings of the prior art, the purpose of the present invention is to provide a lightweight Tai Chi scoring method and related equipment based on an improved attention mechanism, so as to overcome the shortcomings of the prior art in that the feature extraction algorithm considers only a single factor, the computational efficiency of the GCN spatiotemporal feature extraction algorithm is low, and the algorithm has poor real-time performance.

[0008] The above technical objectives of the present invention are achieved through the following technical solutions: a lightweight Tai Chi scoring method based on an improved attention mechanism, comprising: Preprocess the Tai Chi video, input the preprocessed Tai Chi video into the skeleton point extraction network, extract the skeleton point features in each frame, and obtain the skeleton point sequence file; Inputting the skeleton point sequence file into the long short-term memory network, identifying Tai Chi moves, and assigning weights to each skeleton point to generate a weighted skeleton point sequence; Obtain a standard Tai Chi video, and extract a standard skeleton point sequence based on the standard Tai Chi video; The similarity between the weighted skeleton point sequence and the standard skeleton point sequence is calculated, and the score of the Tai Chi video is calculated based on the similarity.

[0009] In one embodiment, the preprocessing of the Tai Chi video specifically includes: using a Gaussian filter to filter each frame of the Tai Chi video.

[0010] In one embodiment, the preprocessed Tai Chi video is input into the skeleton point extraction network, and the skeleton point features in each frame are extracted respectively to obtain a skeleton point sequence file, which specifically includes: inputting the preprocessed Tai Chi video into the skeleton point extraction network frame by frame, and identifying the skeleton point coordinates and the skeleton point connection relationship in each frame respectively; counting the frame number, skeleton point coordinates and skeleton point connection relationship corresponding to each frame, and generating a skeleton point sequence file.

[0011] In one embodiment, the skeleton point sequence file is input into a long short-term memory network, Tai Chi moves are identified, and weights are assigned to each skeleton point to generate a weighted skeleton point sequence, which specifically includes: using the long short-term memory network to identify the temporal changes of the skeleton points in the skeleton point sequence file to identify Tai Chi moves; using the attention mechanism in the long short-term memory network to assign weights to each skeleton point and output a weighted skeleton point sequence.

[0012] In one embodiment, the use of the long short-term memory network to identify the temporal changes of the skeleton points of the skeleton point sequence file to identify Tai Chi moves specifically includes: dividing the skeleton point sequence file into a plurality of subsequences; and using the long short-term memory network to respectively identify the Tai Chi moves corresponding to each subsequence.

[0013] In one embodiment, the use of the long short-term memory network to respectively identify the Tai Chi moves corresponding to each subsequence specifically includes: generating a first cell state; using the long short-term memory network to process the first bone point coordinates in the subsequence, and updating the first cell state to generate a second cell state according to the processing result; determining whether there are still unprocessed bone point coordinates, and if so, marking the second bone point coordinates adjacent to the first bone point coordinates as the first bone point coordinates, marking the second cell state as the first cell state, and re-executing this step; if not, executing the next step; inputting the hidden state in the second cell state into the fully connected layer, classifying the subsequence, and obtaining the Tai Chi moves corresponding to the subsequence.

[0014] In one embodiment, the method of using the long short-term memory network to process the first bone point coordinates in the subsequence and updating the first cell state to generate the second cell state according to the processing result specifically includes: using the forget gate in the long short-term memory network to process the first bone point coordinates to generate a forgetting vector; using the forgetting vector to update the first cell state to generate a forgotten cell state; using the input gate in the long short-term memory network to process the first bone point coordinates to generate an input vector and a candidate cell state; using the input vector to update the first cell state to generate an input cell state; and calculating the sum of the forgotten cell state and the input cell state to generate a second cell state.

[0015] In one embodiment, the method of using the attention mechanism in the long short-term memory network to assign weights to each skeleton point and output a weighted skeleton point sequence specifically includes: Get the bone point feature vector of each frame's bone point coordinates :

[0016] in, Indicates Tai Chi moves; Calculate the query vector of the attention mechanism separately , key vector and the value vector ,include:

[0017]

[0018]

[0019] in, , and All are learnable weight parameter matrices; Calculating the query vector With key vector The dot product of is used to measure the similarity between them, and the Softmax function is used to normalize the dot product result to obtain the attention weight of each bone point. The obtained attention weight is combined with the value vector Multiply and sum to get the weight vector :

[0020] in, is the key vector The dimension of represents the scaling factor used to smooth the distribution of attention weights; Calculate the weight vector and the bone point feature vector The product of , get the weighted bone point sequence ,include:

[0021] In one embodiment, the method of using the attention mechanism in the long short-term memory network to assign weights to each skeleton point and output a weighted skeleton point sequence also includes: After format conversion, the following skeleton point sequence format is obtained:

[0022] in, Indicated in Among the Tai Chi moves, In the frame, The horizontal coordinate of the bone point containing weight information corresponding to the bone point, Indicated in Among the Tai Chi moves, In the frame, The vertical coordinate of the bone point containing weight information corresponding to each bone point.

[0023] In one embodiment, the calculating the similarity between the weighted skeleton point sequence and the standard skeleton point sequence, and calculating the score of the Tai Chi video based on the similarity, specifically includes: Calculating the accumulated distance between the weighted skeleton point sequence and the standard skeleton point sequence using a dynamic time warping algorithm; The accumulated distance is converted into a score of the Tai Chi video.

[0024] A lightweight Tai Chi scoring device based on an improved attention mechanism, comprising: The skeleton point extraction unit is used to pre-process the Tai Chi video, input the pre-processed Tai Chi video into the skeleton point extraction network, extract the skeleton point features in each frame, and obtain a skeleton point sequence file; A skeleton point weighting unit, used to input the skeleton point sequence file into the long short-term memory network, identify Tai Chi moves, and assign weights to each skeleton point to generate a weighted skeleton point sequence; A standard skeleton point extraction unit, used for acquiring a standard Tai Chi video, and extracting a standard skeleton point sequence based on the standard Tai Chi video; A scoring unit is used to calculate the similarity between the weighted skeleton point sequence and the standard skeleton point sequence, and calculate the score of the Tai Chi video based on the similarity.

[0025] A computer-readable storage medium stores a computer program, which implements the steps of the above method when executed by a processor.

[0026] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0027] In summary, the present invention has the following beneficial effects: a lightweight Tai Chi scoring method based on an improved attention mechanism, comprising: preprocessing a Tai Chi video, inputting the preprocessed Tai Chi video into a skeleton point extraction network, extracting the skeleton point features in each frame, and obtaining a skeleton point sequence file; inputting the skeleton point sequence file into a long short-term memory network, identifying Tai Chi moves, and assigning weights to each skeleton point to generate a weighted skeleton point sequence; obtaining a standard Tai Chi video, and extracting a standard skeleton point sequence based on the standard Tai Chi video; calculating the similarity between the weighted skeleton point sequence and the standard skeleton point sequence, and calculating the score of the Tai Chi video based on the similarity; adopting the method of the present invention, the long short-term memory network can be used to capture the temporal dependency in Tai Chi moves, and the attention mechanism is used to assign weights to each skeleton point so that the system can dynamically adjust the focus point according to the characteristics of different actions. Important skeleton points will be given higher weights, thereby improving the accuracy of action recognition and analysis, and improving the accuracy and objectivity of Tai Chi scoring; in addition, this application uses long short-term memory networks to identify Tai Chi moves, and can score each move in the Tai Chi video, thereby improving the interpretability of the final score; by improving the LSTM algorithm, the problems of low computational efficiency and poor real-time performance of the traditional GCN model are overcome, the computing resource requirements of the model are reduced, the model can be made lightweight, and the model can be deployed on small computing devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A flowchart of a lightweight Tai Chi scoring method based on an improved attention mechanism of the present invention; Figure 2 It is a structural diagram of a lightweight Tai Chi scoring device improved based on the attention mechanism in an embodiment of the present invention; Figure 3 is an internal structure diagram of a computer device in an embodiment of the present invention; In the figure, 1. Skeleton point extraction unit; 2. Skeleton point weighting unit; 3. Standard skeleton point extraction unit; 4. Scoring unit. DETAILED DESCRIPTION

[0029] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings. Several embodiments of the present invention are shown in the accompanying drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein.

[0030] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal connection of two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances. The terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.

[0031] In the present invention, unless otherwise clearly specified and limited, the first feature being "above" or "below" the second feature may include the first and second features being in direct contact, or may include the first and second features being in contact not directly but through another feature between them. Moreover, the first feature being "above", "above" and "above" the second feature includes the first feature being directly above and obliquely above the second feature, or simply indicates that the first feature is higher in level than the second feature. The first feature being "below", "below" and "below" the second feature includes the first feature being directly below and obliquely below the second feature, or simply indicates that the first feature is lower in level than the second feature. The terms "vertical", "horizontal", "left", "right", "above", "below" and similar expressions are for illustrative purposes only, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0032] Embodiment 1 In order to solve the above problems, the present invention provides a lightweight Tai Chi scoring method based on an improved attention mechanism. Figure 1 As shown, including: S1, preprocessing the Tai Chi video, inputting the preprocessed Tai Chi video into the skeleton point extraction network, extracting the skeleton point features in each frame, and obtaining a skeleton point sequence file; S2, inputting the skeleton point sequence file into the long short-term memory network, identifying Tai Chi moves, and assigning weights to each skeleton point to generate a weighted skeleton point sequence; S3, obtaining a standard Tai Chi video, and extracting a standard skeleton point sequence based on the standard Tai Chi video; S4. Calculate the similarity between the weighted skeleton point sequence and the standard skeleton point sequence, and calculate the score of the Tai Chi video based on the similarity.

[0033] In practical applications, by preprocessing Tai Chi videos (such as noise reduction, background removal, etc.), the accuracy of subsequent skeleton point extraction can be improved, and the skeleton point features in each frame can be extracted to accurately capture the key postures of the human body. The generation of skeleton point sequence files simplifies the complex original video data into standardized skeleton point coordinates. This structured data is convenient for subsequent processing and analysis. The long short-term memory network (LSTM) can capture the temporal dependencies in Tai Chi moves, and assign weights to each skeleton point so that the system can dynamically adjust the focus according to the characteristics of different actions, improving the accuracy of action recognition and analysis; the skeleton point sequence extracted from the standard Tai Chi video can be used as a reference benchmark to provide an objective criterion for the actions of different practitioners; by calculating the similarity between the weighted skeleton point sequence and the standard skeleton point sequence, the difference between the actual action and the standard action can be accurately measured; the scoring system based on similarity calculation can provide quantitative feedback for Tai Chi practitioners, which helps to guide and improve their actions.

[0034] In summary, the technical solution of the present application has higher scoring accuracy than the Tai Chi scoring method in the prior art and can more accurately obtain the weight information of different bone points.

[0035] In one embodiment, the preprocessing of the Tai Chi video specifically includes: using a Gaussian filter to filter each frame of the Tai Chi video. In practical applications, Gaussian filters are used to remove noise from videos. They can smooth images, reduce details and high-frequency components, and thus eliminate random noise in images. In Tai Chi video processing, the use of Gaussian filters can weaken the fine background noise in the video by smoothing the image, making the subject's movements more prominent. Although Gaussian filters can cause the edges of the image to become blurred, in skeleton point detection, this smoothing can help reduce false detections caused by overly complex edges or details.

[0036] In one embodiment, the preprocessed Tai Chi video is input into the skeleton point extraction network, and the skeleton point features in each frame are extracted respectively to obtain a skeleton point sequence file, which specifically includes: inputting the preprocessed Tai Chi video into the skeleton point extraction network frame by frame, and identifying the skeleton point coordinates and the skeleton point connection relationship in each frame respectively; counting the frame number, skeleton point coordinates and skeleton point connection relationship corresponding to each frame, and generating a skeleton point sequence file.

[0037] In practical applications, by extracting the skeleton point features of each frame in the video and generating a sequence file, the complex dynamic information in the video is converted into a structured and easy-to-process data form. The skeleton point sequence file contains the coordinates and connection relationships of each key point of the human body in the video at different time points, which is convenient for subsequent analysis and processing. By converting video data into a skeleton point sequence file, the system can focus on analyzing the key point features of human movements without processing other irrelevant visual information. This helps to improve the accuracy of action recognition, especially under complex backgrounds or lighting conditions.

[0038] In one embodiment, the skeleton point sequence file is input into a long short-term memory network, Tai Chi moves are identified, and weights are assigned to each skeleton point to generate a weighted skeleton point sequence, which specifically includes: using the long short-term memory network to identify the temporal changes of the skeleton points in the skeleton point sequence file to identify Tai Chi moves; using the attention mechanism in the long short-term memory network to assign weights to each skeleton point and output a weighted skeleton point sequence.

[0039] In practical applications, the attention mechanism can assign different weights to different bone points. Since different Tai Chi moves focus on different parts of the body, the attention mechanism allows the network to pay more attention to those key bone points when identifying a specific move. For example, in some moves, hand movements may be more important, while in other moves, the overall posture of the feet or the body may be more critical. By dynamically assigning weights, the system can capture the characteristics of each movement more accurately. Using the attention mechanism to dynamically assign weights makes the system more robust in the face of different movements and individual differences. There may be slight differences in the movements of different individuals, but through appropriate weight adjustments, the system can adapt to these differences and make reasonable recognition and scoring.

[0040] In one embodiment, the use of the long short-term memory network to identify the temporal changes of the skeleton points of the skeleton point sequence file to identify Tai Chi moves specifically includes: dividing the skeleton point sequence file into a plurality of subsequences; and using the long short-term memory network to respectively identify the Tai Chi moves corresponding to each subsequence.

[0041] In practical applications, the transitions between Tai Chi movements are often coherent and natural. By dividing the sequence into subsequences, the system can better identify and process these transition segments. LSTM can identify the start and end of the action in each subsequence, and then more accurately capture the continuity and transitions between actions, thereby improving the understanding and recognition of the overall action flow. Processing long sequences usually requires higher computing resources and time, especially when the sequence contains a large amount of skeletal point information. After dividing the sequence into subsequences, LSTM can process these subsequences in parallel or segment by segment, reducing the complexity of a single calculation, thereby improving the overall computing efficiency.

[0042] In one embodiment, the step of using the long short-term memory network to respectively identify the Tai Chi moves corresponding to each subsequence specifically includes: generating a first cell state; Processing the first bone point coordinates in the subsequence using the long short-term memory network, updating the first cell state to generate a second cell state according to the processing result; determining whether there are still unprocessed bone point coordinates, if so, marking the second bone point coordinates adjacent to the first bone point coordinates as the first bone point coordinates, marking the second cell state as the first cell state, and re-performing this step; if not, executing the next step; The hidden state in the second cell state is input into the fully connected layer, the subsequence is classified, and the Tai Chi moves corresponding to the subsequence are obtained.

[0043] In practical applications, some Tai Chi moves may involve complex movement changes, such as turning or jumping, which are usually accompanied by rapid changes in multiple skeletal points. LSTM can better capture and understand these complex movement changes by gradually processing each skeletal point and dynamically adjusting the cell state according to these changes, thereby making more accurate recognition. The hidden state is an important output of LSTM, which contains comprehensive information at the current moment. After processing the subsequence, the hidden state is input into the fully connected layer for classification, which ensures that the classifier obtains sufficient contextual information, thereby improving the accuracy of classification. In this way, LSTM can more accurately map the processed skeletal point information to specific Tai Chi moves.

[0044] In one embodiment, the using the long short-term memory network to process the first bone point coordinates in the subsequence and updating the first cell state to generate the second cell state according to the processing result specifically includes: Processing the first skeleton point coordinates using the forget gate in the long short-term memory network to generate a forget vector; updating the first cell state using the forget vector to generate a forget cell state; Processing the first skeleton point coordinates using the input gate in the long short-term memory network to generate an input vector and a candidate cell state; updating the first cell state using the input vector to generate an input cell state; The sum of the forgotten cell state and the input cell state is calculated to generate a second cell state.

[0045] Specifically, the forget gate is based on the coordinates of the current input bone point and the hidden state at the previous moment , generates a vector between 0 and 1 , represents the “degree of forgetting” of each element in the current cell state,

[0046] In the formula, represents the Sigmoid function, for The hidden state at the moment, that is, the output value of the previous cell; is the input value of the current cell; is the weight of the forget gate, is the bias of the forget gate.

[0047] Output of forget gate will be consistent with the cell state at the previous moment Multiply. This operation determines which information in the cell state at the previous moment needs to be retained and which needs to be forgotten or discarded:

[0048] in, If an element in is close to 1, it means that the corresponding cell state element will be retained; If an element in is close to 0, it means that the corresponding cell state element will be forgotten.

[0049] The input gate is based on the coordinates of the current input bone point and the hidden state at the previous moment , generates a vector between 0 and 1 , indicating the degree of influence of the current input on the cell state.

[0050]

[0051] in, is the weight of the input gate; is the bias of the input gate; is the activation function; At the same time, LSTM calculates the candidate cell state generated by the current input bone point coordinates , this candidate state represents new information that may be added to the cell state.

[0052]

[0053] is the hyperbolic tangent activation function; is the candidate value weight, Bias for candidate values.

[0054] Output of the input gate With candidate cell states Multiplication determines the input information at the current moment Contribution to cell state:

[0055] In summary, after passing through the input gate of the forget gate, the second cell state can be obtained:

[0056] in, Represents the multiplication of corresponding elements in the matrix.

[0057] Finally, after a The output gate of the layer determines the output cell state. Then the cell state is passed through a The output of the layer is multiplied by the output of the output gate to ultimately determine the cell output

[0058]

[0059] Hidden State Contains key information extracted from the cell state, which represents the comprehensive information of the current time step and can be used in subsequent cycles.

[0060] In one embodiment, the method of using the attention mechanism in the long short-term memory network to assign weights to each skeleton point and output a weighted skeleton point sequence specifically includes: Get the bone point feature vector of each frame's bone point coordinates :

[0061] in, Indicates Tai Chi moves; Calculate the query vector of the attention mechanism separately , key vector and the value vector ,include:

[0062]

[0063]

[0064] in, , and All are learnable weight parameter matrices; Calculating the query vector With key vector The dot product of is used to measure the similarity between them, and the Softmax function is used to normalize the dot product result to obtain the attention weight of each bone point. The obtained attention weight is combined with the value vector Multiply and sum to get the weight vector :

[0065] in, is the key vector The dimension of represents the scaling factor used to smooth the distribution of attention weights; Calculate the weight vector and the bone point feature vector The product of , get the weighted bone point sequence ,include:

[0066] In one embodiment, the method of using the attention mechanism in the long short-term memory network to assign weights to each skeleton point and output a weighted skeleton point sequence also includes: After format conversion, the following skeleton point sequence format is obtained:

[0067] in, Indicated in Among the Tai Chi moves, In the frame, The horizontal coordinate of the bone point containing weight information corresponding to the bone point, Indicated in Among the Tai Chi moves, In the frame, The vertical coordinate of the bone point containing weight information corresponding to each bone point.

[0068] In practical applications, the attention mechanism allows the system to dynamically assign weights according to different Tai Chi moves, focusing on the skeletal points that are most critical to action recognition. The key parts of each move may be different. The attention mechanism can automatically identify the most important skeletal points in each frame by calculating the similarity of the query vector, key vector, and value vector, and assign them higher weights, thereby improving the accuracy of action recognition. Due to individual differences, there may be slight differences in the movements of different practitioners. The attention mechanism can tolerate these slight differences by assigning higher weights to key skeletal points, so that the system can still maintain a high level of recognition and scoring accuracy when facing the movements of different individuals.

[0069] In one embodiment, the calculating the similarity between the weighted skeleton point sequence and the standard skeleton point sequence, and calculating the score of the Tai Chi video based on the similarity, specifically includes: Calculating the accumulated distance between the weighted skeleton point sequence and the standard skeleton point sequence using a dynamic time warping algorithm; The accumulated distance is converted into a score of the Tai Chi video.

[0070] Specifically, the dynamic time warping algorithm can align two variables with different lengths or speeds in the time axis in an optimal way, and then find the minimum distance between the two. It specifically includes the following steps: first, calculate the Euclidean distance between the corresponding bone points in the standard bone point sequence and the weighted bone point sequence; suppose there are m frames in the actual action sequence and n frames in the standard action sequence, then calculate the first The sum of the Euclidean distances between the coordinates of each weighted bone point in the frame and the coordinates of each standard bone point in the frame can generate an m*n size distance matrix, where the distance matrix elements represent the distances between the corresponding two frames:

[0071] in, Indicates The first The horizontal coordinate of the weighted bone point; Indicates The first The horizontal coordinate of the standard bone point, Indicates The first The ordinate of the weighted bone point; Indicates The first The vertical coordinates of the standard bone points are calculated respectively. After calculating the distances between the corresponding bone points, the sum of all Euclidean distances is taken as the first The distance between all weighted skeleton points in the first frame and all standard skeleton points in the lth frame.

[0072] According to the above calculation results, after constructing the distance matrix D of size m*n, the distance matrix D is used to calculate the cumulative distance matrix C, where each element Indicates the element from the upper left corner To element The minimum cumulative distance between, including:

[0073] in, , : indicates the matching path from above, : Indicates the matching path from the left; : Through this recursive formula, DTW gradually fills the cumulative distance matrix , thus finding arrive The minimum cumulative distance.

[0074] The above process is illustrated by using numbers to represent the features of the bone points in the bone point sequence: standard bone point sequence A = [1, 2, 3, 4]; weighted bone point sequence B = [1, 3, 4, 5]; The same numbers in the sequence indicate that the features of the two skeleton points are closest. The absolute value of the difference between the two elements is used as the Euclidean distance to obtain the following distance matrix D:

[0075] According to the above matrix, from the upper left corner Start looking for the minimum cumulative distance of each point, and construct the cumulative distance matrix C:

[0076] To find the best matching path, start from the bottom right corner of the matrix Start by following the path back to the upper left corner In path selection, the direction with the smallest cumulative distance is selected each time.

[0077] from , the previous step of the minimum is from the left . Path selection: From Left to .

[0078] from , the previous step of the minimum is from above . Path selection: From To the upper left corner .

[0079] from , the previous step of the minimum is from the diagonal . Path selection: From Up to .

[0080] from , the previous step of the minimum is from the diagonal . Path selection: From To the upper left corner . The optimal path: .

[0081] (1,1): The first element 1 of sequence A matches the first element 1 of sequence B.

[0082] (2,2): The second element 2 of sequence A matches the second element 3 of sequence B.

[0083] (3,2): The third element 3 of sequence A matches the second element 3 of sequence B.

[0084] (4,3): The 4th element 4 of sequence A matches the 3rd element 4 of sequence B.

[0085] (4,4): The 4th element 4 of sequence A matches the 4th element 5 of sequence B.

[0086] Based on the above content, the difference between the skeleton point sequence containing weights and the skeleton point sequence of the standard Tai Chi video can be calculated, and then based on the difference, the content in the Tai Chi video can be scored, for example, the Tai Chi video can be converted into a percentage system.

[0087] In summary, this application assigns weights to each skeleton point so that the system can dynamically adjust the focus point according to the characteristics of different actions, thereby improving the accuracy of action recognition and analysis; the skeleton point sequence extracted from the standard Tai Chi video is used as a reference benchmark to provide an objective evaluation standard for the actions of different practitioners; by calculating the similarity between the weighted skeleton point sequence and the standard skeleton point sequence, the difference between the actual action and the standard action can be accurately measured; the scoring system based on similarity calculation can provide quantitative feedback for Tai Chi practitioners, which helps to guide and improve actions.

[0088] Embodiment 2 See also Figure 2 , a lightweight Tai Chi scoring device based on an improved attention mechanism, the lightweight Tai Chi scoring device based on an improved attention mechanism comprises: The skeleton point extraction unit 1 is used to pre-process the Tai Chi video, input the pre-processed Tai Chi video into the skeleton point extraction network, extract the skeleton point features in each frame, and obtain a skeleton point sequence file; Skeleton point weighting unit 2, used for inputting the skeleton point sequence file into the long short-term memory network, identifying Tai Chi moves, and assigning weights to each skeleton point to generate a weighted skeleton point sequence; A standard skeleton point extraction unit 3 is used to obtain a standard Tai Chi video and extract a standard skeleton point sequence based on the standard Tai Chi video; The scoring unit 4 is used to calculate the similarity between the weighted skeleton point sequence and the standard skeleton point sequence, and calculate the score of the Tai Chi video based on the similarity.

[0089] For the specific limitations of the improved lightweight Tai Chi scoring device based on the attention mechanism, please refer to the limitations of the improved lightweight Tai Chi scoring method based on the attention mechanism above, which will not be repeated here. Each module in the above-mentioned improved lightweight Tai Chi scoring device based on the attention mechanism can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0090] Those skilled in the art will understand that Figure 2 The structure shown in the figure is only a block diagram of a part of the structure related to the scheme of the present application, and does not constitute a limitation of the scheme of the present application. The specific lightweight Tai Chi scoring device based on the improvement of the attention mechanism may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0091] Embodiment 3 A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the improved lightweight Tai Chi scoring method based on the attention mechanism as described in Example 1 is implemented.

[0092] Embodiment 4 In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 3As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. When the computer program is executed by the processor, a lightweight Tai Chi scoring method based on an improved attention mechanism is implemented.

[0093] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0094] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented: Preprocess the Tai Chi video, input the preprocessed Tai Chi video into the skeleton point extraction network, extract the skeleton point features in each frame, and obtain the skeleton point sequence file; Inputting the skeleton point sequence file into the long short-term memory network, identifying Tai Chi moves, and assigning weights to each skeleton point to generate a weighted skeleton point sequence; Obtain a standard Tai Chi video, and extract a standard skeleton point sequence based on the standard Tai Chi video; The similarity between the weighted skeleton point sequence and the standard skeleton point sequence is calculated, and the score of the Tai Chi video is calculated based on the similarity.

[0095] In one embodiment, the preprocessing of the Tai Chi video specifically includes: A Gaussian filter is used to filter each frame in the Tai Chi video.

[0096] In one embodiment, the pre-processed Tai Chi video is input into the skeleton point extraction network, and the skeleton point features in each frame are extracted respectively to obtain the skeleton point sequence file, which specifically includes: The pre-processed Tai Chi video is input into the skeleton point extraction network frame by frame, and the coordinates of the skeleton points and the connection relationship of the skeleton points in each frame are identified respectively; The frame number, skeleton point coordinates and skeleton point connection relationship corresponding to each frame are counted to generate a skeleton point sequence file.

[0097] In one embodiment, the step of inputting the skeleton point sequence file into a long short-term memory network, identifying Tai Chi moves, and assigning weights to each skeleton point to generate a weighted skeleton point sequence specifically includes: Using the long short-term memory network to identify the time sequence changes of the skeleton points in the skeleton point sequence file, and identifying the Tai Chi moves; The attention mechanism in the long short-term memory network is used to assign weights to each skeleton point and output a weighted skeleton point sequence.

[0098] In one embodiment, the step of using the long short-term memory network to identify the temporal changes of the skeleton points in the skeleton point sequence file and identifying the Tai Chi moves specifically includes: The skeleton point sequence file is divided into a plurality of subsequences; and the long short-term memory network is used to respectively identify the Tai Chi moves corresponding to each subsequence.

[0099] In one embodiment, the step of using the long short-term memory network to respectively identify the Tai Chi moves corresponding to each subsequence specifically includes: generating a first cell state; Processing the first bone point coordinates in the subsequence using the long short-term memory network, updating the first cell state to generate a second cell state according to the processing result; determining whether there are still unprocessed bone point coordinates, if so, marking the second bone point coordinates adjacent to the first bone point coordinates as the first bone point coordinates, marking the second cell state as the first cell state, and re-performing this step; if not, executing the next step; The hidden state in the second cell state is input into the fully connected layer, the subsequence is classified, and the Tai Chi moves corresponding to the subsequence are obtained.

[0100] In one embodiment, the using the long short-term memory network to process the first bone point coordinates in the subsequence and updating the first cell state to generate the second cell state according to the processing result specifically includes: Processing the first skeleton point coordinates using the forget gate in the long short-term memory network to generate a forget vector; updating the first cell state using the forget vector to generate a forget cell state; Processing the first skeleton point coordinates using the input gate in the long short-term memory network to generate an input vector and a candidate cell state; updating the first cell state using the input vector to generate an input cell state; The sum of the forgotten cell state and the input cell state is calculated to generate a second cell state.

[0101] In one embodiment, the method of using the attention mechanism in the long short-term memory network to assign weights to each skeleton point and output a weighted skeleton point sequence specifically includes: Get the bone point feature vector of each frame's bone point coordinates :

[0102] in, Indicates Tai Chi moves; Calculate the query vector of the attention mechanism separately , key vector and the value vector ,include:

[0103]

[0104]

[0105] in, , and All are learnable weight parameter matrices; Calculating the query vector With key vector The dot product of is used to measure the similarity between them, and the Softmax function is used to normalize the dot product result to obtain the attention weight of each bone point. The obtained attention weight is combined with the value vector Multiply and sum to get the weight vector :

[0106] in, is the key vector The dimension of represents the scaling factor used to smooth the distribution of attention weights; Calculate the weight vector and the bone point feature vector The product of , get the weighted bone point sequence ,include:

[0107] In one embodiment, the method of using the attention mechanism in the long short-term memory network to assign weights to each skeleton point and output a weighted skeleton point sequence also includes: After format conversion, the following skeleton point sequence format is obtained:

[0108] in, Indicated in Among the Tai Chi moves, In the frame, The horizontal coordinate of the bone point containing weight information corresponding to the bone point, Indicated in Among the Tai Chi moves, In the frame, The vertical coordinate of the bone point corresponding to each bone point contains weight information.

[0109] In one embodiment, the calculating the similarity between the weighted skeleton point sequence and the standard skeleton point sequence, and calculating the score of the Tai Chi video based on the similarity, specifically includes: Calculating the accumulated distance between the weighted skeleton point sequence and the standard skeleton point sequence using a dynamic time warping algorithm; The accumulated distance is converted into a score of the Tai Chi video.

[0110] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0111] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0112] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A lightweight Tai Chi scoring method based on an improved attention mechanism, characterized in that: include: Preprocess the Tai Chi video, input the preprocessed Tai Chi video into the skeleton point extraction network, extract the skeleton point features in each frame, and obtain the skeleton point sequence file; Inputting the skeleton point sequence file into the long short-term memory network, identifying Tai Chi moves, and assigning weights to each skeleton point to generate a weighted skeleton point sequence; Obtain a standard Tai Chi video, and extract a standard skeleton point sequence based on the standard Tai Chi video; The similarity between the weighted skeleton point sequence and the standard skeleton point sequence is calculated, and the score of the Tai Chi video is calculated based on the similarity.

2. According to claim 1, a lightweight Tai Chi scoring method based on an improved attention mechanism is characterized in that: The preprocessing of the Tai Chi video specifically includes: A Gaussian filter is used to filter each frame in the Tai Chi video.

3. The lightweight Tai Chi scoring method based on the improved attention mechanism according to claim 1 is characterized in that: The pre-processed Tai Chi video is input into the skeleton point extraction network, and the skeleton point features in each frame are extracted to obtain the skeleton point sequence file, which specifically includes: The pre-processed Tai Chi video is input into the skeleton point extraction network frame by frame, and the coordinates of the skeleton points and the connection relationship of the skeleton points in each frame are identified respectively; The frame number, skeleton point coordinates and skeleton point connection relationship corresponding to each frame are counted to generate a skeleton point sequence file.

4. The lightweight Tai Chi scoring method based on the improved attention mechanism according to claim 1 is characterized in that: The step of inputting the skeleton point sequence file into the long short-term memory network, identifying Tai Chi moves, and assigning weights to each skeleton point to generate a weighted skeleton point sequence specifically includes: Using the long short-term memory network to identify the time sequence changes of the skeleton points in the skeleton point sequence file, and identifying the Tai Chi moves; The attention mechanism in the long short-term memory network is used to assign weights to each skeleton point and output a weighted skeleton point sequence.

5. The lightweight Tai Chi scoring method based on the improved attention mechanism according to claim 4 is characterized in that: The method of using the long short-term memory network to identify the time sequence changes of the skeleton points in the skeleton point sequence file and identifying the Tai Chi moves specifically includes: The skeleton point sequence file is divided into a plurality of subsequences; and the long short-term memory network is used to respectively identify the Tai Chi moves corresponding to each subsequence.

6. The lightweight Tai Chi scoring method based on the improved attention mechanism according to claim 5 is characterized in that: The step of using the long short-term memory network to respectively identify the Tai Chi moves corresponding to each subsequence specifically includes: generating a first cell state; Processing the first bone point coordinates in the subsequence using the long short-term memory network, updating the first cell state to generate a second cell state according to the processing result; determining whether there are still unprocessed bone point coordinates, if so, marking the second bone point coordinates adjacent to the first bone point coordinates as the first bone point coordinates, marking the second cell state as the first cell state, and re-performing this step; if not, executing the next step; The hidden state in the second cell state is input into the fully connected layer, the subsequence is classified, and the Tai Chi moves corresponding to the subsequence are obtained.

7. The lightweight Tai Chi scoring method based on the improved attention mechanism according to claim 1 is characterized in that: The calculating the similarity between the weighted skeleton point sequence and the standard skeleton point sequence, and calculating the score of the Tai Chi video based on the similarity, specifically includes: Calculating the accumulated distance between the weighted skeleton point sequence and the standard skeleton point sequence using a dynamic time warping algorithm; The accumulated distance is converted into a score of the Tai Chi video.

8. A lightweight Tai Chi scoring device based on an improved attention mechanism, characterized in that: The lightweight Tai Chi scoring device improved based on the attention mechanism includes: The skeleton point extraction unit is used to pre-process the Tai Chi video, input the pre-processed Tai Chi video into the skeleton point extraction network, extract the skeleton point features in each frame, and obtain a skeleton point sequence file; A skeleton point weighting unit, used to input the skeleton point sequence file into the long short-term memory network, identify Tai Chi moves, and assign weights to each skeleton point to generate a weighted skeleton point sequence; A standard skeleton point extraction unit, used for acquiring a standard Tai Chi video, and extracting a standard skeleton point sequence based on the standard Tai Chi video; A scoring unit is used to calculate the similarity between the weighted skeleton point sequence and the standard skeleton point sequence, and calculate the score of the Tai Chi video based on the similarity.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the improved lightweight Tai Chi scoring method based on the attention mechanism as described in any one of claims 1 to 7 is implemented.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the improved lightweight Tai Chi scoring method based on the attention mechanism as described in any one of claims 1-7.

Images