Flower sword action recognition method and system based on attitude estimation

Abstract

The invention discloses a fancy sword action recognition method and system based on posture estimation, and relates to the technical field of human body action recognition. The identification method comprises the following steps: step 1, establishing a flower sword action data set; 2, designing a network model; 3, designing a sword movement auxiliary training system; according to the invention, by establishing the flower sword action data set, the learning difficulty is reduced, and network training, verification and testing are facilitated; six different flower sword actions are selected, firstly, OpenPose is utilized to carry out human body posture estimation, then an estimation result is processed into a feature sequence, the feature sequence is input into a fusion model CNN-LSTM to be recognized, and finally, the flower sword actions are evaluated and analyzed by utilizing a human body posture estimation result.

Description

Technical Field

[0001] The invention belongs to the technical field of human motion recognition, and in particular relates to a foil motion recognition method and system based on posture estimation. Background Art

[0002] In recent years, human action recognition has become a hot topic in artificial intelligence visual applications, especially in athlete performance evaluation and training optimization. Foil emphasizes the training of basic technical movements and steps. In traditional teaching, coaches often judge and correct students' movements, but the number of fencing coaches is scarce and their levels are uneven. Beginners' levels improve slowly and even risk injury. Combining the foil action recognition technology in artificial intelligence can provide more scientific guidance and personalized teaching for foil learners. Furthermore, action recognition mostly uses video or image sequences to directly input into the model for training and verification, but the image data is relatively complex and there is background noise interference, which makes the network model design complex. It is impossible to achieve a high recognition accuracy for actions with large changes. The collection of foil datasets is also challenging. The video datasets in the fencing field of UCF (UCF SportsAction Dataset, University of Central Florida Sports Action Dataset) are mainly from competitions. The movements change rapidly and are difficult to learn. However, the datasets for basic technical movements of foil lack standardized samples. Summary of the invention

[0003] To solve the existing problems, the present invention aims to provide a method and system for foil action recognition based on posture estimation.

[0004] A foil action recognition method based on posture estimation of the present invention, its recognition method is as follows:

[0005] Step 1: Establish a foil action dataset: Based on the characteristics of foil actions, a foil action dataset in video format containing 4446 clips of six basic technical actions was established;

[0006] Step 2: Design of network model: Combining the human pose estimation algorithm OpenPose with convolutional neural network and long short-term memory network, the key points of the human body extracted by OpenPose are processed into digital feature sequences, which reduces the dimension of input data and eliminates the background noise interference of image data. Based on this, the CNN-LSTM hybrid network model is innovatively optimized to reduce the number of convolutional layers and fully connected layers, reduce the complexity of network calculation, and effectively improve the training efficiency of the model. Finally, the hybrid features are passed to the Softmax classifier for final classification;

[0007] Step 3: Design an auxiliary training system for foil sports: The system realizes the interaction between the mobile phone and the large screen for function selection, foil action video following, practice video recording, and recognition and motion analysis of basic foil technical movements in practice videos.

[0008] Preferably, the establishment of the foil action data set is divided into video acquisition, video data preprocessing, key point detection of the video, generation of action continuous frames and data set generation, and establishment of the foil data set.

[0009] Preferably, the design of the network model is divided into convolutional neural network design, LSTM network design, fully connected layer and output layer design, and optimization of the CNN-LSTM network model. The optimization of the CNN-LSTM network model is divided into adjustment of the network architecture, selection of hyperparameters, and optimization of algorithm model recognition accuracy.

[0010] Preferably, the foil sports auxiliary training system includes an interactive module, a data acquisition module and a background database module, which realizes the functions of collecting, saving, analyzing data and outputting results, multiple information interactions, smooth operation of the system and convenient data management.

[0011] Preferably, the interaction module realizes human-computer interaction through Python QT5 and MQTT communication, uses PyQT for interface design, and realizes mobile phone remote control interface switching through MQTT, which is convenient for user operation.

[0012] Preferably, the data acquisition module extracts required data from a background database, uses the data to perform key point detection, generates feature sequences, and performs action recognition, key angle calculation, key point movement speed analysis, and longitudinal comparison data analysis.

[0013] Preferably, the background database module obtains and stores data from the interactive interface, provides information to the data processing module, and receives and stores processed data.

[0014] Compared with the prior art, the present invention has the following beneficial effects:

[0015] First, by establishing a foil action dataset, the learning difficulty is reduced, making it easier to use for network training, verification, and testing.

[0016] Second, six different foil movements were selected. OpenPose was first used to estimate the human posture. The estimated results were then processed into feature sequences and input into the fusion model CNN-LSTM for recognition. Finally, the human posture estimation results were used to evaluate and analyze the foil movements. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] For ease of explanation, the present invention is described in detail with reference to the following specific implementations and the accompanying drawings.

[0018] Figure 1 It is a schematic diagram of the foil action sequence of the present invention;

[0019] Figure 2 It is a schematic diagram of the framework of the foil technique action recognition algorithm in the present invention;

[0020] Figure 3 This is a schematic diagram of the LSTM network structure in the present invention;

[0021] Figure 4 This is a schematic diagram of the CNN-LSTM network structure in the present invention;

[0022] Figure 5 This is a schematic diagram of user demand analysis in the present invention;

[0023] Figure 6 It is a system framework diagram in the present invention;

[0024] Figure 7 The schematic diagram of the three main modules of the system design in the present invention;

[0025] Figure 8 is a flow chart of the interactive module in the present invention;

[0026] Fig. 9 This is a working diagram of the data processing module in the present invention;

[0027] Fig.10 It is a motion recognition and analysis workflow diagram in the present invention;

[0028] Fig.11 This is a functional diagram of the database module in the present invention;

[0029] Fig.12 It is a schematic diagram of the basic structure of the hybrid model in the present invention. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is described below by the specific embodiments shown in the accompanying drawings. However, it should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. The structures, proportions, sizes, etc. illustrated in the drawings of this specification are only used to match the contents disclosed in the specification for people familiar with this technology to understand and read, and are not used to limit the limiting conditions that the present invention can be implemented, so they have no technical substantive significance. Any structural modification, change in proportional relationship or adjustment of size, without affecting the effects that the present invention can produce and the purposes that can be achieved, should still fall within the scope of the technical content disclosed by the present invention. In addition, in the following description, the description of known structures and technologies is omitted to avoid unnecessary confusion of the concept of the present invention.

[0031] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0032] Specific implementation method 1: This specific implementation method adopts the following technical solutions:

[0033] Part I: Dataset Collection and Creation:

[0034] 1.1. Description of basic foil techniques:

[0035] Since there is no public foil dataset that can be used for foil action recognition, a dataset of seven people performing six different foil actions was collected. These six very basic technical actions include: holding the sword back and forth, defense, stepping forward and stabbing, lunging and stabbing, attacking and counterattacking, and sprinting.

[0036] 1.2. Creation of experimental data set:

[0037] Through the definition and analysis of the basic technical movements of foil sports, as well as the related dynamic performances in foil sports related training videos and recordings, this data set will classify several common basic technical movements in foil sports, and collect dynamic video data of basic technical movements of foil according to six technical dynamic videos in foil movements, such as sword holding, defense, straight sprint, and lunge. After the video collection is completed and the data is initially screened, the Jianying software is used to complete the video editing work required before the experiment to meet the video processing requirements. And by using its multi-threaded method, the speed of video editing, transcoding and other operations is greatly improved, and the preliminary collection and processing of the video data required for the experiment is completed.

[0038] 1.2.1 Video acquisition:

[0039] The data was collected using an ordinary high-definition 1080p camera to collect foil action videos with a frame rate of 30 frames per second. The system uses Windows 10 system, with an RTX2070 GPU and 32G memory. During the collection, the optimal installation position of the camera was selected according to the actual scene requirements; the camera was placed in front of the experimental area, and the focus was adjusted to shoot, so that the whole body of the subject could be captured completely and clearly during the collection process. The camera was about 3.5 meters away from the shooting action station. Each participant in the shooting repeated a single foil action in front of the camera many times, and returned to the starting point after each completion to start the next action again. The video of each action was collected indoors. After collecting and editing the video, a total of 4446 video clips were obtained, each of which was between 2 seconds and 6 seconds long, and the data set size was 11.6G.

[0040] 1.2.2 Video data preprocessing:

[0041] After the source video is collected, generally, except when the athlete maintains a static action or performs slow motion exercises, the time to complete a complete set of technical movements in the actual foil sport is very short, and it does not exceed 5 seconds from the appearance of an action to the completion. Therefore, in order to ensure the effect of the homemade data set, it is necessary to carefully distinguish and extract the actions that appear in the video. In this specific implementation method, each video is edited first, and a video clip containing a complete foil technical action is intercepted using video editing software, and each video is basically kept within 5 seconds (including intercepting a longer video into multiple short videos); only a complete set of video clips of the target action is retained, which is equivalent to extracting the action that needs to be identified separately. The standard requirements for video editing are: in the intercepted video, the interference factors to the technical action are as small as possible, it must be ensured that there is no obvious obstruction to the human body, and the entire intercepted clip must have a complete set of completed technical actions.

[0042] 1.2.3. Detect key points in the video:

[0043] After collecting and preliminarily processing the original video, further operations are required to form a standardized data set. Since the video image is interfered by background and lighting, and the image data is relatively complex and has high data dimensions, the key points of the video are detected first. The specific method is to use the OpenPose algorithm to detect the key points of each frame of each video of six different foil actions: holding the sword back and forth, sprinting, defending, attacking and counterattacking, stabbing and attacking, and obtaining the coordinate data of the athlete's skeletal key points in each frame. In this way, the complex image features are transformed into simple, sparse, and low-dimensional digital features. In addition, the key points directly reflect the skeleton structure, which is easy to capture the movements and postures of the human body and is not easily affected by additional interference. In the key point detection, only the extraction of the human skeleton coordinates is involved, and the different changes in movements do not affect the extraction effect.

[0044] 1.2.4. Generate action continuous frames and generate data sets:

[0045] The number of selected frames will be processed uniformly. For each different category of foil technical action, such as holding the sword back and forth and defensive actions, the body posture and characteristics of the action are different, and the time to complete the whole set of actions of different categories is also different. Therefore, after video preprocessing, the number of frames will be inconsistent between individual videos of different categories, and even some frames will differ greatly. The goal of creating a data set is to ensure that each video image sequence can completely cover a whole set of technical actions of this category. The frame number rules and intervals must be set according to certain calculation logic. The strategy adopted in this specific implementation method uniformly extracts a fixed number of frames, which specifically includes: first counting the number of frames of each video and determining the number of frames to be extracted (such as extracting 45 frames from each video), then calculating the frame extraction interval based on the number of frames, and finally extracting the corresponding frames according to the frame extraction interval as data preparation for the action sequence. This uniform frame extraction method can quickly obtain the desired action sequence and ensure that a complete set of target technical actions is obtained within a continuous forty-five frame image. Figure 1 As shown in the figure, taking the counterattack action as an example, the action sequence image is drawn using the key point coordinates of continuous frames. For the convenience of display, the coordinate points and connecting lines of the left eye, left ear, right eye, and right ear are omitted when drawing the image. It can be seen from the sequence image that the action sequence contains the complete technical action process, and the changes in the action can also be clearly presented.

[0046] 1.2.5、Establishment of Foil Dataset:

[0047] After generating a continuous digital feature sequence for each video clip, the data needs to be divided into training set, validation set and test set in a ratio of 8:2:2 according to the action. Each data set contains different action types. After data processing, 8692, 2282, and 2367 feature sequences were obtained for network training, validation, and testing respectively.

[0048] Part II: Design of network model:

[0049] like Figure 2As shown in the figure, first, the input is a video containing a complete process of a single foil action. After the input, the video data is preprocessed, including cropping, frame extraction, background noise reduction, etc., to form a frame sequence. The OpenPose model is used to continuously detect key points of the frame sequence of the basic foil technical action to obtain the digital features of 18 human skeleton key points in the video frame, and the digital features of the video frame are combined into a feature sequence containing a complete foil action. Then, the digital feature sequence is sent to the CNN-LSTM network model. In the CNN-LSTM network model, first, CNN learns the spatial relationship between different postures and postures through the relative position between coordinates, identifies specific postures or action patterns, and extracts key changes such as running, raising hands and stabbing, and specific steps; next, the CNN action representation values ​​in the digital feature sequence are respectively input into the LSTM units and used to learn the action and change signals associated with the digital feature sequence; then, the output value of each LSTM unit is linearly transformed, and finally the output of the classifier Softmax layer and the corresponding action behavior between the digital feature sequence are used to determine the final action recognition accuracy probability.

[0050] Attitude estimation algorithm selection:

[0051] This specific implementation proposes an innovative solution, which is to convert image data into skeletal key point data that reflects the athlete throughout the entire movement process, and use digital feature sequences instead of traditional image sequences. This method not only significantly simplifies the calculation process, but also effectively eliminates unnecessary interference factors, thereby improving the accuracy and efficiency of motion recognition. By extracting the data sequence of 18 key skeletal points, it is possible to capture and analyze motion postures more accurately, thereby reducing the data dimension and ensuring the intuitiveness and effectiveness of feature extraction. This simplification not only reduces the computing resources required for model training, but also improves real-time performance, making its application in complex scenarios more feasible.

[0052] The comparison of various human posture estimation algorithms in terms of accuracy is as follows:

[0053] Table 1 Analysis and comparison of various posture estimation algorithms

[0054]

[0055] Among them, AP (Average Precision) stands for average precision, which indicates the overall performance of the model under different IoU (Intersection over Union) thresholds. AP is an important indicator for evaluating detection algorithms. The higher the value, the higher the overall recognition accuracy of the model. AP50 represents the average precision when the IoU threshold is set to 0.50, indicating the performance under looser conditions. AP75 represents the average precision when the IoU threshold is set to 0.75, indicating the performance under stricter conditions, and generally reflects the robustness and accuracy of the model. APM represents the average precision of medium-sized targets, which is mainly used to evaluate the model's recognition ability for medium-sized targets. APL represents the average precision of large-sized targets, which evaluates the model's detection performance for large targets.

[0056] From the perspective of the overall performance of AP, the CPN algorithm ranks the highest with an AP of 72.1. The AP50 and AP75 values ​​are also high, 91.4 and 80.0 respectively, showing that its overall recognition ability of the target is relatively strong. It also performs well in medium targets (APM68.7) and large targets (APL 77.2). In contrast, the Hourglass algorithm performs the worst in all indicators, with an AP of only 56.6, and both APM and APL are below 60, indicating that it faces great challenges in posture estimation. As for the OpenPose algorithm, although the AP of OpenPose is not as high as that of CPN, it performs well on AP50 and AP75, showing that it has good target recognition ability in general, especially under looser IoU conditions; its performance on medium targets (APM 57.1) and large targets (APL 68.2) shows that it is relatively adaptable to various targets and is suitable for processing human pose estimation tasks of various sizes. For the implementation of OpenPose, compared with some other algorithms, such as Mask-RCNN, OpenPose usually has lower requirements for computing resources and can run in a relatively ordinary hardware environment. In addition, the OpenPose algorithm is relatively mature, with strong community support, easy to use, and can be quickly integrated into existing applications. In scenarios with high real-time requirements, the implementation of OpenPose can often provide a faster response speed, which is suitable for applications that require instant feedback, such as human-computer interaction and sports motion analysis.

[0057] Since different posture estimation algorithms are applicable to different tasks, it is not possible to measure whether these posture estimation algorithms are suitable for fencing action recognition tasks solely based on detection accuracy. In the field of action recognition and behavior analysis, the bottom-up method focuses on local features and can effectively capture the dynamic changes of the human body. It shows greater flexibility in dealing with new postures or new actions. When new human actions or postures appear, the bottom-up algorithm can adapt by learning new key point patterns without redesigning the global model. At the same time, the bottom-up algorithm is more computationally efficient because it does not have to perform global processing on the entire image. In addition, the bottom-up posture estimation method can be better combined with other computer vision tasks to help improve the overall recognition effect.

[0058] In summary, choosing OpenPose as the key point detection algorithm can achieve a good balance in accuracy, adaptability and ease of implementation, taking into account both accuracy and feasibility for the future development of real-time application scenarios. Using OpenPose to obtain the key points of each frame of the human body during the foil process is the first step in realizing action recognition. OpenPose is an open source library for multi-person key point detection, which can perform joint detection of human body, hand and facial key points. The 18 key points tracked by OpenPose are the positions of ears, eyes, nose, neck, shoulders, hips, knees, ankles, elbows and wrists. After key point detection, the key point data of a single frame is saved in a json file.

[0059] Action recognition network design:

[0060] 1. Convolutional Neural Network Design:

[0061] The model first obtains the spatial information of the digital feature sequence through a three-layer CNN architecture, thereby forming the representation characteristics of human body movements.

[0062] Let the input be a matrix of shape N×18, where N is the time step (number of frames) and each row represents the coordinates (x, y) of 18 key points in a frame, as shown in formula (2-1):

[0063]

[0064] Among them, x ij Represents the coordinates of the jth key point in the i-th frame.

[0065] In a specific implementation, since the data input to the CNN network is not a video image, but a digital feature containing the coordinates of key points, a sliding window method is not used, but a direct convolution method is used to extract sequence features. For the digital features of a single frame in the digital feature sequence, the convolution kernel is set to W and the size is set to K×18. In this case, the convolution kernel slides on the sequence of key points to extract local temporal features from continuous frames.

[0066]

[0067] Among them, Y[i] is the value of the feature map at the i-th position after convolution.

[0068] The convolution method uses "padding = SAME" to add zero padding to the edges of the input data to maintain the size of the output features. Let P be the number of paddings, then:

[0069]

[0070] The convolution operation will capture the posture changes of key points in time series as follows:

[0071] F[i,j]=f(Y[i,j]) (2-4)

[0072] Among them, f is the activation function, which is the Relu function in this article, and F[i,j] is the feature map after activation.

[0073] To improve the stability and efficiency of training, batch normalization is used within each time step of the convolutional layer output. In addition, a dropout layer with a rate of 0.5 is used to prevent overfitting by randomly setting half of the input units to zero during training. After the convolution and normalization stages, another batch normalization layer is utilized to normalize the entire time series of features, ensuring consistent scaling and facilitating subsequent processing stages.

[0074] After these initial layers, the model employs a temporal distribution flattening operation to convert the spatial features into a single vector representation at each time step, preparing the data for processing by the LSTM network.

[0075] 2. LSTM network design:

[0076] An output R of the CNN layer corresponds to an LSTM input at time t. Figure 3 The internal structure diagram of the LSTM unit is shown in the figure. From the structure diagram, we can find that there is a memory unit C throughout the LSTM, which is jointly constrained by the input gate, forget gate, output gate and the previous memory unit.

[0077] For a certain time t, the forget gate determines which information should be forgotten or retained from the memory unit at each time step, and works together with the input gate to ultimately obtain the historical information retained at the current time node. The formula is as follows:

[0078] f t =σ(W f ·[h t-1 ,x t ]+b f ) (2-5)

[0079] Where σ is the logistic activation function, W f and b f are the weight matrix and bias term of the forget gate respectively, [h t-1 ,x t ] is the hidden state of the previous time step and the input of the current time step.

[0080] The input gate determines the current situation by adjusting the amount of information input at the current moment. In extreme cases, if the input information at the current moment has no practical value for the existing information in the neuron, the input gate will not allow the input information at the current time point to merge with the existing information. In general, the input information will more or less merge with the existing information through the input gate. The input gate consists of two parts: a sigmoid layer that determines which values ​​will be updated, and a tanh layer that creates a new candidate value vector that will be added to the state. The formula is as follows:

[0081] i t =σ(W i ·[h t-1 ,x t ]+b i ) (2-6)

[0082]

[0083] Here, I t is the output of the input gate, is the candidate memory cell state, W i , W C and b i , b C are the relevant weights and biases respectively. The update of the cell state combines the information of the forget gate and the input gate:

[0084]

[0085] The output gate determines which part of the memory cell state will be output to the hidden state. At the same time, the output value will be scaled by the tanh function. For different tasks, the amount of information transmitted is also different, and it is not necessary to transmit all the information. LSTM uses the output gate and the forget gate to determine whether the information at each moment can be retained or forgotten from the memory cell. The formula is as follows:

[0086] o t =σ(W o ·[h t-1 ,x t ]+b o ) (2-9)

[0087] h t =o t *tanh(C t ) (2-10)

[0088] Among them, σ represents the sigmoid function, · represents the dot product of a vector or matrix, and o t is the output of the output gate, x t is the input at time t, h t is the hidden state output at time t, C t is the cell state at time t, W and b are the weight and bias respectively.

[0089] 3. Design of fully connected layer and output layer:

[0090] The prediction results of this specific implementation are generated through two fully connected layers. The nodes of the first layer are connected to all the time series data of the LSTM unit to integrate local features. Each node of the second layer is fully connected to the nodes of the first layer, and the mixed features are counted. The results are then input into the Softmax classifier to predict the category probability. The calculation results are shown in formula (2-11):

[0091] y t =W t ×h t (2-11)

[0092] 4. Optimization of CNN-LSTM network model:

[0093] The training performance of the model is usually affected by multiple factors such as computing architecture and hyperparameter settings. In order to ensure the training accuracy of the model and reduce the training time, this specific implementation method optimizes the speed and recognition performance of the network model by means of reasonably designing hyperparameters and adjusting the network architecture.

[0094] 1. Adjustment of network architecture:

[0095] In this specific implementation, by adjusting and optimizing the selected network structure, the training speed of the CNN-LSTM hybrid neural network is effectively improved. The CNN network algorithm part of the CNN-LSTM model architecture constructed in this specific implementation includes:

[0096] (1) Since the input data type of the algorithm model selected in this specific implementation is the coordinate data of 18 skeleton key points instead of video image sequence in the CNN network, the spatial information of the data is relatively simple, so a small number of convolutional network layers can extract sufficient posture features. Therefore, the convolutional network is reduced to one layer;

[0097] (2) At the end of the network, the fully connected layer is removed and the output of the LSTM network is directly input into the softmax activation function to convert it into a probability distribution so that the sum of all outputs is 1. Finally, the probability of each category is output for decision making.

[0098] Through the adjustment of the above network architecture, the training parameters can be greatly reduced in the process of extracting spatial feature vectors from the convolutional neural network and inputting them into the LSTM architecture, thereby reducing the training time.

[0099] 2. Reasonable design and selection of hyperparameters:

[0100] This specific implementation method selects the appropriate batch-size parameter through comparative experiments to ensure that the model has a faster training speed while achieving the best performance. Since the batch size of a single iteration is a key factor affecting the processing speed, the batch hyperparameter usually includes three values: 32, 64, and 128. This specific implementation method tests these three batch-sizes by training the model, and selects the smallest possible batch value to determine the most appropriate setting while achieving similar performance.

[0101] This specific implementation adopts the strategy of selecting the number of iterations (epochs) and the learning rate. When selecting the two parameters of the number of iterations and the learning rate, it is necessary to save the model once every 100 iterations, and then draw images of the recognition accuracy change curves, loss changes and the current learning rate of these models and analyze and discuss them. After comparative analysis, the number of epochs at which the performance of the corresponding model tends to be stable and the loss no longer decreases under the condition of setting a specific learning rate is selected, and the number of epochs corresponding to the learning rate reaching the equilibrium point is recorded, and then the learning rate is continued to be reduced, and the training comparison is continued according to the same steps as before, until the number of iterations and learning rate corresponding to the optimal performance of the model are finally obtained. Based on the experience of training and experimental operations, it is concluded that: under normal circumstances, the smaller the learning rate, the smaller the number of epochs required. Therefore, the strategy of modifying the learning rate and the number of training iterations is used to reduce the training time while achieving the same accuracy and improve the training efficiency of the model.

[0102] 3. Optimization of algorithm model recognition accuracy:

[0103] This specific implementation method fully utilizes the technical advantages of the constructed CNN and LSTM hybrid neural network model through in-depth optimization and design. First, the local connection characteristics and weight sharing characteristics of the CNN network make the model more efficient when processing high-dimensional data. These characteristics not only reduce the number of parameters, but also increase the sensitivity of the model to local features, thereby simplifying the complexity of feature abstraction and enabling the network to quickly acquire and process important information. On the other hand, the LSTM network, with its powerful memory ability, can effectively capture the temporal features in long time series. This ability enables the model to integrate the digital feature sequence of foil technical movements output by CNN, especially the dynamic changes in the time dimension. Through sequence learning and feature training, the model is more accurate in the use of time information, thereby improving the final prediction accuracy. Combining the advantages of these two networks, this specific implementation method constructs a powerful model that can both extract spatial features and process temporal information. This design concept provides a more effective solution for processing complex action recognition tasks. The specific optimization design is as follows:

[0104] In the network structure of this specific embodiment, the initial layer configuration includes a time-division wrapper around a one-dimensional convolutional neural network (Conv1D) with 16 convolution kernels, each with a kernel size of 3. This layer uses a rectified linear unit (ReLU) activation to introduce nonlinearity and operates in a "same" padding mode to maintain the temporal dimension of the input sequence.

[0105] To improve the stability and efficiency of training, batch normalization is applied within each time step of the convolutional layer output. In addition, a dropout layer with a rate of 0.5 is used to prevent overfitting by randomly setting half of the input units to zero during training. After the convolution and normalization stages, another batch normalization layer is utilized to normalize the entire time series of features, ensuring consistent scaling and facilitating subsequent processing stages.

[0106] After these initial layers, the model employs a temporal distribution flattening operation to transform the spatial features into a single vector representation at each time step, preparing the data for sequential processing by a long short-term memory (LSTM) network.

[0107] The LSTM layer has 20 units and a unit forgetting bias set to 0.5. It is configured to retain the memory of the previous time step and propagate relevant temporal information through time. By setting return_sequences to True, the LSTM layer outputs the complete sequence, which is crucial for capturing temporal dependencies in action recognition tasks. Finally, to determine the prediction process, a final TimeDistributed Dense layer with a softmax activation function is used to produce a probability distribution over predefined action classes. This layer ensures that the sum of the output probabilities of the model in all possible action classes is 1, which facilitates direct classification. For model parameter updates, the Adam optimizer with a learning rate of 0.0001 is used to effectively handle the high-dimensional parameter space and minimize the classification cross entropy loss function. Finally, the CNN-LSTM network model designed in this specific embodiment is as follows Figure 4 shown.

[0108] Part III: System Design:

[0109] 1. Design of auxiliary training system for foil sports:

[0110] Based on the deep learning action recognition algorithm model that has been experimentally verified before, this specific implementation method provides high recognition accuracy and training speed for the recognition of basic technical actions of foil. On this basis, this specific implementation method proposes a design scheme for an auxiliary training system for foil sports. The system can recognize the six basic technical actions in foil sports, including sword holding back and forth, defense, step forward and stab attack, bow step stab attack, counterattack and sprint. By citing the posture estimation action recognition algorithm under the above-mentioned CNN-LSTM hybrid neural network mode, the system analyzes the technical classification of athletes in the foil technical action video and conducts a detailed analysis and explanation. At the same time, the key point data obtained is used to assist the practice of basic actions, providing support for the comprehensive rational design of the competition and training plans of foil athletes and the formulation of competition strategies.

[0111] 2. System design goals:

[0112] This system uses the in-depth study of the CNN-LSTM action recognition algorithm of OpenPose key point detection to train a network model for basic technical action recognition in foil, and applies it to foil auxiliary training. Through the basic data collection, processing and comparative analysis of club and athlete information, users can upload personal training videos and interpret their actions through algorithm extraction technology. This interpretation and analysis will help athletes complete foil training with standard actions. In addition, after the introduction of the action recognition algorithm, the intelligent analysis mechanism can provide real-time feedback on the real action data made by athletes on the field, so that coaches can provide timely tactical guidance. Before the design and development of the system is officially completed, its feasibility must be initially analyzed, including technical and economic feasibility, practical usability and other dimensions. The following is the specific analysis content.

[0113] In the feasibility analysis of actual operation, the design and development of this system focused on the simplicity of the user interface, so that users do not need to deeply understand the background operation and processing flow of the system. For general users, the complexity and difficulty of the complex background data processing and operation process can be almost ignored. In addition, the system can be directly deployed and installed on a pure Windows operating system, and does not have excessive requirements for computer configuration. Therefore, the system shows good feasibility in actual operation.

[0114] 3. System Requirements Analysis:

[0115] The system conducts a detailed investigation based on actual demand conditions, carefully analyzes the user needs in the investigation, and uses clear, concise and standardized demand documents to determine the core functions and basic design of the system, such as Figure 5 As shown, the main functional requirements of the system are briefly described below:

[0116] (1) Users can query athletes’ personal information and compare athletes’ training records;

[0117] (2) Users can upload sports videos and use the system’s recognition algorithm to classify the athletes’ movements in the videos and analyze the standard of the players’ movements. A sports analysis report is generated based on the results, and athletes can be assisted in conducting comparative analysis from a vertical perspective:

[0118] (3) Administrators can add, delete, modify, and query user data information and grant users rights. Users with different role rights can obtain different rights to query and modify their personal data information.

[0119] (4) The motion video, the processed dynamic recognition information, the recognized annotated video, etc. are stored in a database, including the storage location of the video, the motion type of the video, and the name of the video.

[0120] 4. System software design framework:

[0121] During the overall design phase, various demand analysis tasks must be integrated into the software system to achieve a general software design framework. In the overall design of the software system, in order to reduce logical problems in the development process, the system will be deployed based on the Windows operating system and set a unified data and device interface standard for the entire management system. Through verification and addition by the background administrator, the application can access the training data and analysis reports of foil athletes through these interfaces. The core functions of the system cover data collection and storage, sports performance analysis, and data visualization. The main architecture of the system is as follows: Figure 6 shown.

[0122] like Figure 7 As shown in the figure, this design includes three modules: interaction module, data acquisition module and background database module. The system includes the functions of collecting, saving, analyzing data and outputting results, and also ensures the interaction of multiple information, smooth operation of the system and convenient data management. The interaction of the three main modules of interaction module, data processing module and background database is as follows:

[0123] (1) Interaction module:

[0124] like Figure 8 As shown, the interactive module provides athletes with an interactive interface as an interactive carrier. On the interactive interface, athletes can register personal information, log in to the interface, choose a learning method, upload their own sports videos, check their own training records, and watch their own sports analysis reports, etc.; and through the interactive interface, the system can also collect users' personal information, training videos and other data as a calculation basis for providing them with sports analysis.

[0125] The module realizes human-computer interaction through Python QT5 and MQTT communication, uses PyQT for interface design, and realizes mobile phone remote control interface switching through MQTT, which is convenient for user operation.

[0126] PyQt provides a complete set of window controls, and each PyQt control corresponds to a Qt control. By creating the main layout, adding title labels, setting the scoreboard and button layout, the interface layout is designed, and the corresponding logical functions are added to realize the display of the foil auxiliary motion system.

[0127] MQTT communication is used to switch the remote control interface of the mobile phone. First, develop the mobile phone program, use the MQTT client library to connect to the configured MQTT Broker, and design the user interface, including buttons, sliders and other controls to trigger remote control instructions. When the user operates on the interface, such as clicking a button or adjusting a slider, the application converts these operations into MQTT messages and publishes them to the defined topic.

[0128] Develop the controlled device side, use the same MQTT client library to connect to the same MQTT Broker, and ensure that both the mobile phone and the controlled device are connected to the same Broker. Subscribe to the same MQTT topic as the mobile app on the controlled device to receive remote control instructions. After receiving the MQTT message, the device will parse the message and perform corresponding operations, such as changing the state or performing a specific action.

[0129] On the TV side, the interface is divided into three parts: the nine-square grid on the left is used to select foil movements, and initially the blue selection prompt stays on the first icon in the first row; the large black screen on the right waits for the camera image to be displayed; the function buttons at the bottom are used to control the playback, pause and stop of the learning video, and are responsible for turning on the camera, recording video and turning off the camera during practice. After the practice is completed, it is used for action recognition and motion analysis.

[0130] In addition to the common "power off", "volume +", "volume -" and "return" functions on the mobile phone, the middle ring button is used for major operations. The four arc-shaped buttons on the top, bottom, left and right represent "up", "down", "left" and "right" respectively, and the middle round button is "OK".

[0131] When collecting data in the interactive interface, the video is acquired through an ordinary camera with a frame rate of 30 frames per second and a resolution of 1080x1440.

[0132] (2) Data processing module:

[0133] like Fig. 9 As shown in the figure, in the data processing module, the main function is to extract the required data from the background database, such as personal videos uploaded by users and digital feature sequences generated by key point detection. The system uses this data to detect key points, generate feature sequences, and perform action recognition, key angle calculation, key point movement speed analysis, and longitudinal comparative data analysis. This information will help coaches and athletes evaluate game performance and skill characteristics, thereby providing a basis for formulating game strategies and plans. In addition, the system also supports uploading and querying athletes' physical reports, including test data such as height and vertical jump height. After the calculation is completed, the recognition results and related calculation results will be stored in the background database.

[0134] In the data processing module, the motion recognition / analysis part is particularly important. In the motion analysis part, the main task is to obtain the corresponding data and then calculate the various analysis items according to the calculation formula. In motion recognition, it is necessary to obtain the digital feature sequence of the athlete and obtain the type of action through the trained network model. The workflow is as follows: Fig.10 shown.

[0135] Data preprocessing aims to convert the motion videos uploaded by users into digital feature sequences containing motion information, in preparation for subsequent motion recognition and analysis. The specific process is as follows: First, the actual foil action clips are cut out from the video. The system automatically completes this process without the need for manual editing by the user. Next, frame extraction is performed, and the pre-trained model of OpenPose is used to detect key points for each frame, and the coordinate data of the 18 key points detected are saved in a JSON file of the specified path. Finally, the coordinate data of all frames are merged into a JSON file to complete the production of a complete motion sequence data, which is then uploaded to the backend database.

[0136] Action recognition and motion analysis are the core parts of the data processing module. After data preprocessing, the complete action feature sequence will be input into the trained CNN-LSTM model to generate six possible action recognition results: sword holding back and forth, second defense and sixth defense, lunge attack, step forward and straight attack, straight sprint and counterattack. The system will ultimately determine the unique action corresponding to a single video based on the probability values ​​of different actions.

[0137] In motion analysis, the acquired human key point sequence is used in combination with coordinate data, video frame rate, video length and frame extraction rules to calculate the speed changes of key points such as the practitioner's right wrist, right ankle and left ankle, as well as the angle changes between the right arm and right leg, and draw corresponding curves. Then, these results are compared with standard movements and combined with professional foil knowledge to provide motion analysis. Finally, the motion recognition and analysis results will be uploaded to the backend database for storage.

[0138] After data preprocessing, the recorded video is saved in the specified path, and then the video is extracted. OpenPose is used to detect the key points of the human body in the extracted frames. Finally, the data is merged as a whole and input into the trained parameter model for recognition. In addition, the key point data obtained by OpenPose is used to calculate the changes in the movement speed of the focus points and the changes in the angle formed during the movement process, which can determine whether the foil action is standard.

[0139] After completing the video recording, control the phone to turn off the camera and select "Generate Motion Analysis Report". After selecting, a window will pop up on the interface, prompting you to start motion recognition and motion analysis. Finally, a pop-up window will display the results of the motion recognition, inform you of the save path of the motion analysis report, and generate the motion analysis report in the corresponding path.

[0140] (3) Backend database:

[0141] like Fig.11 As shown in the figure, the backend database is mainly responsible for acquiring and storing data from the interactive interface, providing information for the data processing module, and receiving and storing processed data. The stored information includes the personal information, sports data, and comparative analysis data of the foil athletes. The personal information includes the user's surname, native place, training location, and club name; the sports data and analysis data include the athlete's sports video at a specific time, digital feature sequence, action category, movement speed, and body angle changes.

[0142] Specific implementation method 2: In this specific implementation method, the combination of convolutional neural network (CNN) and long short-term memory network (LSTM) is an innovative direction in the field of deep learning. CNN can extract latent features and spatial features from sample data by using convolution kernels, and has a strong ability to capture local features of data such as images and texts, while LSTM is good at processing long-term dependencies in sequence data and can effectively remember and forget information. Combining the advantages of the two, feature fusion and hierarchical fusion can be achieved and applied to multimodal data. The basic structure of the hybrid model is as follows: Fig.12 shown.

[0143] In the CNN-LSTM network, the data is processed by the convolutional neural network, and all features are fused to obtain the feature description of the convolutional neural network, that is, the CNN sliding window and weight sharing are used to obtain the local spatial features of the video sequence and the output result is used as the input of the reserved layer, and then the data is passed to the LSTM. Under normal circumstances, the input data at this time needs to be shaped into the type of LSTM processing. After the LSTM obtains the new input, it determines what needs to be kept and discarded. The retention and discarding are completed with the help of the sigmoid activation function. Any data multiplied by 1 is itself, and any function multiplied by 0 is 0, so you can choose to forget and save important data. When the data is multiplied by 1, it means that it is retained, and the data obtained from the inputs updates the state of the entire model. Finally, the output gate is used to determine the information carried, and the new state and hidden state are transferred to the next time step.

[0144] In a specific implementation, CNN-LSTM is used to realize the recognition of foil action. The input is a digital feature sequence composed of the coordinates of the key points of the human body obtained after key point detection by OpenPose on multiple frames of pictures. Then, it is input into CNN to extract spatial features from the input data, and then these feature sequences are passed to LSTM to capture time dependency. Finally, the softmax activation function is used to output the probability distribution to realize this multi-classification task.

[0145] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, the embodiments should be considered exemplary and non-restrictive in all respects, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims be included in the present invention.

[0146] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A method for foil action recognition based on posture estimation, characterized in that: Its identification method is as follows: Step 1: Establish a foil action dataset: Based on the characteristics of foil actions, a foil action dataset in video format containing 4446 clips of six basic technical actions was established; Step 2: Design of network model: Combining the human pose estimation algorithm OpenPose with convolutional neural network and long short-term memory network, the key points of the human body extracted by OpenPose are processed into digital feature sequences, which reduces the dimension of input data and eliminates the background noise interference of image data. Based on this, the CNN-LSTM hybrid network model is innovatively optimized to reduce the number of convolutional layers and fully connected layers, reduce the complexity of network calculation, and effectively improve the training efficiency of the model. Finally, the hybrid features are passed to the Softmax classifier for final classification; Step 3: Design an auxiliary training system for foil sports: The system realizes the interaction between the mobile phone and the large screen for function selection, foil action video following, practice video recording, and recognition and motion analysis of basic foil technical movements in practice videos.

2. The method for foil action recognition based on posture estimation according to claim 1, characterized in that: The establishment of the foil action data set is divided into video acquisition, video data preprocessing, key point detection of the video, generation of action continuous frames and data sets, and establishment of the foil data set.

3. The method for foil action recognition based on posture estimation according to claim 1, characterized in that: The design of the network model is divided into convolutional neural network design, LSTM network design, fully connected layer and output layer design, and optimization of the CNN-LSTM network model. The optimization of the CNN-LSTM network model is divided into adjustment of the network architecture, selection of hyperparameters, and optimization of algorithm model recognition accuracy.

4. The method for foil action recognition based on posture estimation according to claim 1, characterized in that: The foil sports auxiliary training system comprises an interactive module, a data acquisition module and a background database module, which realizes the functions of collecting, saving, analyzing data and outputting results, multiple information interactions, smooth operation of the system and convenient data management.

5. The method for foil action recognition based on posture estimation according to claim 4, characterized in that: The interactive module realizes human-computer interaction through Python QT5 and MQTT communication, uses PyQT for interface design, and realizes mobile phone remote control interface switching through MQTT, which is convenient for user operation.

6. The method for foil action recognition based on posture estimation according to claim 4, characterized in that: The data acquisition module extracts the required data from the background database, uses the data to perform key point detection, generates feature sequences, and performs action recognition, key angle calculation, key point movement speed analysis, and longitudinal comparison data analysis.

7. The method for foil action recognition based on posture estimation according to claim 4, characterized in that: The backend database module acquires and stores data from the interactive interface, provides information to the data processing module, and receives and stores processed data.

Images