Action recognition method, device, electronic device and computer-readable storage medium

Abstract

The present application provides a motion recognition method, device, electronic device and computer-readable storage medium, the method comprising: inputting a data stream to be tested into a motion recognition model to obtain a motion recognition result of the data stream to be tested; the training process of the motion recognition model comprises: obtaining a training set, the training set comprising multiple training data, each training data comprising at least one sample data stream and annotated data of the motion recognition result of the sample data stream; inputting the sample data stream in the training data into a preset deep learning model to obtain predicted data of the motion recognition result of the sample data stream; updating the model parameters of the deep learning model; detecting whether a preset training end condition is met; if so, using the trained deep learning model as the motion recognition model; if not, continuing to train the deep learning model using the next training data. This method overcomes the defects of traditional skeleton motion recognition and improves recognition accuracy.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a motion recognition method, device, electronic device, and computer-readable storage medium. Background Art

[0002] Human skeleton action recognition typically involves using human skeleton data obtained through depth cameras like Microsoft Kinect or advanced human pose estimation algorithms like OpenPose as input, followed by classification and recognition using deep learning algorithms. Human skeleton action recognition has broad research prospects, with applications in public security, security monitoring, smart homes, and daily entertainment.

[0003] The existing technology has low accuracy and poor action recognition effect. Based on this, the present application provides an action recognition method, device, electronic device and computer-readable storage medium to improve the existing technology. Summary of the Invention

[0004] The purpose of this application is to provide a motion recognition method, device, electronic device and computer-readable storage medium, which overcomes the defects of traditional skeleton motion recognition, improves recognition accuracy and has good recognition effect.

[0005] The purpose of this application is achieved by the following technical solutions:

[0006] In a first aspect, the present application provides a method for motion recognition for a human skeleton, the method comprising:

[0007] Inputting a data stream to be tested into a motion recognition model to obtain a motion recognition result of the data stream to be tested, wherein the data stream to be tested includes a sample skeleton data stream and a preset motion data stream, and the motion recognition result of the data stream to be tested is used to indicate at least one motion type corresponding to the data stream to be tested;

[0008] The training process of the action recognition model includes:

[0009] Obtaining a training set, the training set including a plurality of training data, each of the training data including at least one sample data stream and annotated data of an action recognition result of the sample data stream, the action recognition result of the sample data stream being used to indicate at least one action type corresponding to the sample data stream;

[0010] For each training data in the training set, perform the following processing:

[0011] Inputting the sample data stream in the training data into a preset deep learning model to obtain prediction data of the action recognition result of the sample data stream;

[0012] Updating model parameters of the deep learning model based on the predicted data and the labeled data of the action recognition result of the sample data stream;

[0013] Detect whether the preset training end condition is met; if so, use the trained deep learning model as the action recognition model; if not, continue training the deep learning model using the next training data.

[0014] The beneficial effect of this technical solution is that the step of performing motion recognition on the human skeleton through the motion recognition model may be to input the data stream to be tested into the motion recognition model to obtain the motion recognition result of the data stream to be tested. Among them, the training process may include: first obtaining a training set, the training set includes multiple training data, and the motion recognition result of the data stream to be tested is used to indicate at least one motion type corresponding to the data stream to be tested. The motion type corresponding to the data stream to be tested can be displayed intuitively and efficiently, with a high degree of intelligence. The sample data in the training data can be input into a preset deep learning model, and the motion recognition model is obtained through training to perform recognition. This method is based on computer technology and uses a deep learning model to better fit the data and actual situation. The deep learning model has a strong fitting ability and can approximate complex functions to reach infinite dimensions, further improving the accuracy of the test. The method overcomes the defects of traditional skeleton motion recognition and improves recognition accuracy.

[0015] In some optional implementations, obtaining a training set includes:

[0016] Using a recording device to obtain multiple sample skeleton data streams;

[0017] performing data preprocessing on the plurality of sample skeleton data streams to remove the sample skeleton data streams with recording failure and / or poor recording quality from the plurality of sample skeleton data streams;

[0018] The pre-processed plurality of sample skeleton data streams and the preset motion data stream are made into a compressed file as the training set.

[0019] The beneficial effects of this technical solution are: using a recording device to record multiple sample skeleton data streams and perform data preprocessing on the sample skeleton data streams, removing sample skeleton data streams that have failed to record and / or have poor recording quality, retaining sample skeleton data streams that have been successfully recorded and / or have good recording quality, and making the multiple sample skeleton data streams after preprocessing and the preset motion data stream into a compressed file as a training set. Data preprocessing helps to improve the quality of the data, reduce redundant sample skeleton data streams of poor quality, thereby reducing the training time, making the subsequent data processing, analysis and model training process easier and more effective, improving the accuracy of the data, ensuring the validity of the data to the greatest extent, and avoiding wasting computing resources; it can also bring stronger generalization capabilities to the model, making the training data as close as possible to the test data, thereby improving the prediction accuracy, and also allowing the model to learn more robust features. Data preprocessing can highlight certain characteristics of the target object in the sample data stream, characteristic parameters of the action type, etc., all of which are conducive to the recognition, tracking and understanding of the target in the sample data stream. In this application, the use of recording equipment to obtain multiple sample skeleton data streams can greatly save labor costs and reduce the safety hazards that exist when manually collecting data. It has the effect of comprehensive, efficient, and high-quality image collection. It can also realize real-time automatic recording and acquisition, and at the same time realize the planning, monitoring of skeleton movement and compression and automatic transmission of sample data streams.

[0020] In some optional embodiments, the deep learning model includes a shallow feature extraction network, a first spatiotemporal fusion module, a deep feature extraction network, and a second spatiotemporal fusion module;

[0021] The step of inputting the sample data stream in the training data into a preset deep learning model to obtain prediction data of the action recognition result of the sample data stream includes:

[0022] Performing shallow feature extraction on the sample skeleton data stream and the preset motion data stream using the shallow feature extraction network to obtain skeleton feature information and motion feature information;

[0023] Performing a first spatiotemporal fusion on the skeleton feature information and the motion feature information using the first spatiotemporal fusion module to obtain a skeleton-motion fusion data stream and a motion-skeleton fusion data stream;

[0024] Performing deep feature extraction on the skeleton-motion fusion data stream using the deep feature extraction network to obtain first fusion feature information;

[0025] Performing deep feature extraction on the motion-skeleton fusion data stream using the deep feature extraction network to obtain second fusion feature information;

[0026] The first fused feature information and the second fused feature information are fused using the second spatiotemporal fusion module to obtain prediction data of the action recognition result of the skeleton data stream.

[0027] The beneficial effects of this technical solution are as follows: a sample data stream is input into a preset deep learning model, the deep learning model comprising a shallow feature extraction network, a first spatiotemporal fusion module, a deep feature extraction network, and a second spatiotemporal fusion module; the shallow feature extraction network is used to perform shallow feature extraction on the sample skeleton data stream to obtain skeleton feature information, and the shallow feature extraction network is used to perform shallow feature extraction on the preset motion data stream to obtain motion feature information; the skeleton feature information and the motion feature information are fused in a first spatiotemporal fusion module to obtain a skeleton-motion fusion data stream and a motion-skeleton fusion data stream; the deep feature extraction network is then used to perform deep feature extraction on the skeleton-motion fusion data stream and the motion-skeleton fusion data stream, respectively, to obtain first fused feature information and second fused feature information; finally, the first fused feature information and the second fused feature information are fused in a second spatiotemporal fusion module to obtain predicted data for the action recognition result of the skeleton data stream. By processing the sample data stream using the shallow feature extraction network, the first spatiotemporal fusion module, the deep feature extraction network, and the second spatiotemporal fusion module, the predicted data for the action recognition result can be efficiently obtained, with a high degree of intelligence. Skeleton-based action recognition models require networks and modules with higher average precision and improved fusion efficiency. This allows for more targeted connections between skeletons and actions, identifying key information and locating the action type corresponding to the skeleton in a single step. The use of spatiotemporal fusion modules (the first and second spatiotemporal fusion modules) effectively integrates the features of the sample skeleton data stream with the preset motion data stream, resulting in improved performance in the final prediction.

[0028] In some optional implementations, the process of obtaining the skeleton-motion fusion data stream includes:

[0029] Downsampling the skeleton feature information to obtain low-dimensional skeleton data;

[0030] Obtaining a first self-attention weight based on the low-dimensional skeleton data and a first activation function;

[0031] Acquire the skeleton-motion fusion data stream based on the skeleton feature information, the first self-attention weight, and the motion feature information;

[0032] The process of obtaining the motion-skeleton fusion data stream includes:

[0033] Downsampling the motion feature information to obtain low-dimensional motion data;

[0034] Obtaining a second self-attention weight based on the low-dimensional motion data and the second activation function;

[0035] Based on the motion feature information, the second self-attention weight and the skeleton feature information, the motion-skeleton fusion data stream is obtained.

[0036] The beneficial effects of this technical solution are: the skeleton-motion fusion data stream and the motion-skeleton fusion data stream are respectively obtained through the first spatiotemporal fusion module, the skeleton feature information is downsampled (i.e., average pooling operation) to obtain low-dimensional skeleton data, and the first self-attention weight is extracted using the first activation function, and then based on the skeleton feature information, the first self-attention weight and the motion feature information, the skeleton-motion fusion data stream is obtained (the first self-attention weight is used to perform a dot product operation with the skeleton feature information to restore the original input size, and then it is added to the motion feature information to obtain the skeleton-motion fusion data stream); the motion feature information is downsampled (i.e., average pooling operation) to obtain low-dimensional motion data, and the second self-attention weight is extracted using the second activation function, and then based on the motion feature information, the second self-attention weight and the skeleton feature information, the motion-skeleton fusion data stream is obtained (the second self-attention weight is used to perform a dot product operation with the motion feature information to restore the original input size, and then it is added to the skeleton feature information to obtain the motion-skeleton fusion data stream). By utilizing self-attention weights, we can better focus on extracting features based on the skeleton and motion type, which enables the deep learning model to more accurately locate and identify, thereby generating a more accurate and comprehensive fused data stream, which brings a qualitative improvement to the performance of the deep learning model.

[0037] In some optional embodiments, the deep feature extraction network includes multiple CNN layers and multiple Transformer layers;

[0038] The step of performing deep feature extraction on the skeleton-motion fusion data stream using the deep feature extraction network to obtain first fusion feature information includes:

[0039] Inputting the skeleton-motion fusion data stream into the plurality of CNN layers to obtain a plurality of first feature maps corresponding to the skeleton-motion fusion data stream;

[0040] Inputting the plurality of first feature maps into the plurality of Transformer layers to obtain the first fused feature information;

[0041] The step of performing deep feature extraction on the motion-skeleton fusion data stream using the deep feature extraction network to obtain second fusion feature information includes:

[0042] Inputting the motion-skeleton fusion data stream into a plurality of CNN layers to obtain a plurality of second feature maps corresponding to the motion-skeleton fusion data stream;

[0043] Input the plurality of second feature maps into the plurality of Transformer layers to obtain the second fused feature information.

[0044] The beneficial effect of this technical solution is that the skeleton-motion fusion data stream is input into multiple CNN layers to obtain multiple first feature maps corresponding to the skeleton-motion fusion data stream, and the multiple first feature maps are input into multiple Transformer layers to obtain first fusion feature information. Similarly, the motion-skeleton fusion data stream is input into multiple CNN layers to obtain multiple second feature maps corresponding to the motion-skeleton fusion data stream, and the multiple second feature maps are input into multiple Transformer layers to obtain second fusion feature information. In view of the fact that the skeleton is not affected by light and the sample skeleton data stream has strong robustness, CNN is innovatively used to capture the spatial information of the data stream and Transformer is used to capture the temporal information of the data stream, which reduces the amount of calculation and uses a smaller model to achieve accuracy comparable to that of a large model.

[0045] In some optional embodiments, the deep learning model further includes a fusion prediction module, and the training data in the training set includes multi-person training data including multiple people and single-person training data including a single person;

[0046] The step of inputting the sample data stream in the training data into a preset deep learning model to obtain prediction data of the action recognition result of the sample data stream includes:

[0047] For each single person training data, perform the following processing:

[0048] Performing supplementary processing and rearrangement processing on a single-person sample data stream in the single-person training data to obtain a processed data stream, wherein the data format of the processed data stream is the same as the data format of the multi-person sample data stream in the multi-person training data;

[0049] Inputting the processed data stream into the fusion prediction module to obtain prediction data of the action recognition result of the processed data stream;

[0050] Restoring the data format of the prediction data of the action recognition result of the processed data stream to obtain the action types and confidence levels of multiple people;

[0051] The action type corresponding to the highest confidence level is obtained as prediction data of the action recognition result of the single-person sample data stream.

[0052] The beneficial effect of this technical solution is that the input data can include a single-person sample data stream containing a single person and a multi-person sample data stream containing multiple people. For each single-person training data, the single-person sample data stream in the single-person training data is supplemented (the second, third, fourth, etc. people in the single-person training data are supplemented with 0), and then the single-person sample data stream in the single-person training data is rearranged; the rearranged single-person sample data stream is input into the fusion prediction module to obtain the prediction data of the action recognition result of the processed data stream, and the data format of the prediction data is restored to obtain the action type and confidence of multiple people, and the action type with the highest confidence is selected as the prediction data of the action recognition result of the single-person sample data stream. The advantage of this is that the number of input people can be flexibly handled. When the number of input people changes, only the data structure needs to be adjusted without adjusting the network structure. That is, single-person and multi-person situations can be processed and action recognized, so as to obtain the action type with the highest confidence as the prediction data of the action recognition result of the single-person sample data stream.

[0053] In some optional embodiments, the method further comprises:

[0054] Obtaining a model file corresponding to the action recognition model;

[0055] The model file is tested to obtain a test result.

[0056] The beneficial effect of this technical solution lies in: after model training is completed, the model file corresponding to the action recognition model is obtained and then tested to obtain test results. This method can evaluate the model's generalization ability, thereby verifying its effectiveness. It can also predict various indicators in actual application, specify psychological thresholds for real-world application scenarios, and select application scenarios with appropriate accuracy, thereby reducing trial and error costs during application.

[0057] In a second aspect, the present application provides a motion recognition device for performing motion recognition on a human skeleton, the device comprising:

[0058] An action recognition unit is configured to input a data stream to be tested into an action recognition model to obtain an action recognition result of the data stream to be tested, wherein the data stream to be tested includes a sample skeleton data stream and a preset motion data stream, and the action recognition result of the data stream to be tested is used to indicate at least one action type corresponding to the data stream to be tested;

[0059] The training process of the action recognition model includes:

[0060] Obtaining a training set, the training set including a plurality of training data, each of the training data including at least one sample data stream and annotated data of an action recognition result of the sample data stream, the action recognition result of the sample data stream being used to indicate at least one action type corresponding to the sample data stream;

[0061] For each training data in the training set, perform the following processing:

[0062] Inputting the sample data stream in the training data into a preset deep learning model to obtain prediction data of the action recognition result of the sample data stream;

[0063] Updating model parameters of the deep learning model based on the predicted data and the labeled data of the action recognition result of the sample data stream;

[0064] Detect whether the preset training end condition is met; if so, use the trained deep learning model as the action recognition model; if not, continue training the deep learning model using the next training data.

[0065] In some optional implementations, obtaining a training set includes:

[0066] Using a recording device to obtain multiple sample skeleton data streams;

[0067] performing data preprocessing on the plurality of sample skeleton data streams to remove the sample skeleton data streams with recording failure and / or poor recording quality from the plurality of sample skeleton data streams;

[0068] The pre-processed plurality of sample skeleton data streams and the preset motion data stream are made into a compressed file as the training set.

[0069] In some optional embodiments, the deep learning model includes a shallow feature extraction network, a first spatiotemporal fusion module, a deep feature extraction network, and a second spatiotemporal fusion module;

[0070] The step of inputting the sample data stream in the training data into a preset deep learning model to obtain prediction data of the action recognition result of the sample data stream includes:

[0071] Performing shallow feature extraction on the sample skeleton data stream and the preset motion data stream using the shallow feature extraction network to obtain skeleton feature information and motion feature information;

[0072] Performing a first spatiotemporal fusion on the skeleton feature information and the motion feature information using the first spatiotemporal fusion module to obtain a skeleton-motion fusion data stream and a motion-skeleton fusion data stream;

[0073] Performing deep feature extraction on the skeleton-motion fusion data stream using the deep feature extraction network to obtain first fusion feature information;

[0074] Performing deep feature extraction on the motion-skeleton fusion data stream using the deep feature extraction network to obtain second fusion feature information;

[0075] The first fused feature information and the second fused feature information are fused using the second spatiotemporal fusion module to obtain prediction data of the action recognition result of the skeleton data stream.

[0076] In some optional implementations, the process of obtaining the skeleton-motion fusion data stream includes:

[0077] Downsampling the skeleton feature information to obtain low-dimensional skeleton data;

[0078] Obtaining a first self-attention weight based on the low-dimensional skeleton data and a first activation function;

[0079] Acquire the skeleton-motion fusion data stream based on the skeleton feature information, the first self-attention weight, and the motion feature information;

[0080] The process of obtaining the motion-skeleton fusion data stream includes:

[0081] Downsampling the motion feature information to obtain low-dimensional motion data;

[0082] Obtaining a second self-attention weight based on the low-dimensional motion data and the second activation function;

[0083] Based on the motion feature information, the second self-attention weight and the skeleton feature information, the motion-skeleton fusion data stream is obtained.

[0084] In some optional embodiments, the deep feature extraction network includes multiple CNN layers and multiple Transformer layers;

[0085] The step of performing deep feature extraction on the skeleton-motion fusion data stream using the deep feature extraction network to obtain first fusion feature information includes:

[0086] Inputting the skeleton-motion fusion data stream into the plurality of CNN layers to obtain a plurality of first feature maps corresponding to the skeleton-motion fusion data stream;

[0087] Inputting the plurality of first feature maps into the plurality of Transformer layers to obtain the first fused feature information;

[0088] The step of performing deep feature extraction on the motion-skeleton fusion data stream using the deep feature extraction network to obtain second fusion feature information includes:

[0089] Inputting the motion-skeleton fusion data stream into a plurality of CNN layers to obtain a plurality of second feature maps corresponding to the motion-skeleton fusion data stream;

[0090] Input the plurality of second feature maps into the plurality of Transformer layers to obtain the second fused feature information.

[0091] In some optional embodiments, the deep learning model further includes a fusion prediction module, the training data in the training set includes multi-person training data including multiple people and single-person training data including a single person; inputting the sample data stream in the training data into the preset deep learning model to obtain prediction data of the action recognition result of the sample data stream includes:

[0092] For each single person training data, perform the following processing:

[0093] Performing supplementary processing and rearrangement processing on a single-person sample data stream in the single-person training data to obtain a processed data stream, wherein the data format of the processed data stream is the same as the data format of the multi-person sample data stream in the multi-person training data;

[0094] Inputting the processed data stream into the fusion prediction module to obtain prediction data of the action recognition result of the processed data stream;

[0095] Restoring the data format of the prediction data of the action recognition result of the processed data stream to obtain the action types and confidence levels of multiple people;

[0096] The action type corresponding to the highest confidence level is obtained as prediction data of the action recognition result of the single-person sample data stream.

[0097] In some optional embodiments, the device is further used to:

[0098] Obtaining a model file corresponding to the action recognition model;

[0099] The model file is tested to obtain a test result.

[0100] In a third aspect, the present application provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any of the above methods when executing the computer program.

[0101] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above methods are implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0102] The present application is further described below with reference to the accompanying drawings and implementation methods.

[0103] Figure 1 A flow chart of an action recognition method provided in an embodiment of the present application is shown.

[0104] Figure 2 A schematic diagram of a process for obtaining a training set provided in an embodiment of the present application is shown.

[0105] Figure 3 A schematic diagram of a process for obtaining prediction data provided in an embodiment of the present application is shown.

[0106] Figure 4 A schematic diagram of the structure of a shallow feature extraction network provided in an embodiment of the present application is shown.

[0107] Figure 5 A schematic structural diagram of a spatiotemporal fusion module provided in an embodiment of the present application is shown.

[0108] Figure 6 A schematic diagram of the structure of a deep feature extraction network provided in an embodiment of the present application is shown.

[0109] Figure 7 A schematic diagram of another process for obtaining prediction data provided in an embodiment of the present application is shown.

[0110] Figure 8 A structural diagram of a fusion prediction module provided in an embodiment of the present application is shown.

[0111] Figure 9 A schematic structural diagram of a motion recognition device provided by the present application is shown.

[0112] Figure 10 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown.

[0113] Figure 11 A schematic diagram of the structure of a program product provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0114] Below, the embodiments of the present application are further described in conjunction with the accompanying drawings and specific implementation methods. It should be noted that, under the premise of no conflict, the various embodiments or technical features described below can be arbitrarily combined to form a new implementation method.

[0115] In the embodiments of the present application, "at least one" refers to one or more, and "more" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can represent: a, b, c, a and b, a and c, b and c, a and b and c, where a, b and c can be single or multiple. It is worth noting that "at least one" can also be interpreted as "one or more items".

[0116] It should also be noted that in the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any implementation or design scheme described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other implementations or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0117] Below, we first briefly describe one of the technical fields of this application (skeleton action recognition).

[0118] Skeletal action recognition can be divided into two stages. The first stage is the manual feature design stage. In this stage, manually designed feature extraction methods are usually used in combination with traditional machine learning methods for classification. This method cannot well describe motion relationships, resulting in low accuracy and impractical application. The second stage is the feature extraction stage using deep learning methods. This stage can be divided into three research directions:

[0119] (1) Based on Convolutional Neural Networks (CNN), this method usually converts skeleton data or extracted features into pseudo images and then uses CNN for classification. This type of method cannot effectively extract the temporal relationship between skeleton data;

[0120] (2) Based on Recurrent Neural Network (RNN), this method usually treats skeleton data as sequence input, so it can capture temporal relationships well, but its ability to capture spatial relationships within a single frame is poor;

[0121] (3) Based on Graph Convolution Neural Networks (GCNNs), this method usually regards skeleton data as a graph, with joint points as vertices and bone connections as edges. Although this method has achieved good results in terms of accuracy, the model parameters of this type of method are large and the calculation is complex, and it cannot be applied in actual scenarios.

[0122] Patent CN115205980A discloses a human action recognition network based on a self-attention mechanism, which includes a linear processing module, a first-level self-attention processing unit, a second-level self-attention processing unit, an average pooling processing unit, and a classification unit that are cascaded in sequence;

[0123] The human skeleton data sequence X in After inputting the linear processing module, a high-dimensional skeleton data sequence X1 is obtained, and then after passing through the first-level self-attention processing unit, the first-level skeleton joint feature sequence X2 is output;

[0124] The first-level skeleton joint feature sequence X2 is divided into two branches in parallel, and is input into the second-level self-attention processing unit for processing, and the second-level skeleton joint feature sequence X2 is output in parallel. J 3, and the second-level body part feature sequence X P 3. The second-level skeleton joint feature sequence X J 3 and the second-level body part feature sequence X P 3. Input them into the average pooling processing unit respectively to obtain the result feature data X out , and then the classification unit recognizes it and outputs the recognition result. This method can only handle the action recognition of a single skeleton.

[0125] Based on this, the present application provides a motion recognition method, device, electronic device and computer-readable storage medium to improve the existing technology.

[0126] Method Example

[0127] See also Figure 1 , Figure 1 A flow chart of an action recognition method provided in an embodiment of the present application is shown.

[0128] The present invention provides a method for motion recognition for a human skeleton. The method includes:

[0129] Step S101: Inputting a data stream to be tested into a motion recognition model to obtain a motion recognition result of the data stream to be tested, wherein the data stream to be tested includes a sample skeleton data stream and a preset motion data stream, and the motion recognition result of the data stream to be tested is used to indicate at least one motion type corresponding to the data stream to be tested;

[0130] The training process of the action recognition model includes:

[0131] Obtaining a training set, the training set including a plurality of training data, each of the training data including at least one sample data stream and annotated data of an action recognition result of the sample data stream, the action recognition result of the sample data stream being used to indicate at least one action type corresponding to the sample data stream;

[0132] For each training data in the training set, perform the following processing:

[0133] Inputting the sample data stream in the training data into a preset deep learning model to obtain prediction data of the action recognition result of the sample data stream;

[0134] Updating model parameters of the deep learning model based on the predicted data and the labeled data of the action recognition result of the sample data stream;

[0135] Detect whether the preset training end condition is met; if so, use the trained deep learning model as the action recognition model; if not, continue training the deep learning model using the next training data.

[0136] Thus, the step of performing motion recognition on a human skeleton using a motion recognition model may include inputting a test data stream into the motion recognition model to obtain a motion recognition result for the test data stream. The training process may include first obtaining a training set, the training set including a plurality of training data, and the motion recognition result for the test data stream being used to indicate at least one motion type corresponding to the test data stream.

[0137] The action type corresponding to the data stream to be tested can be displayed intuitively and efficiently, with a high degree of intelligence. The sample data in the training data can be input into a preset deep learning model, and an action recognition model is obtained through training to perform recognition.

[0138] This method, based on computer technology and leveraging deep learning models, can better fit data and real-world conditions. Deep learning models possess powerful fitting capabilities and can approximate complex functions up to infinite dimensions, further improving test accuracy. This method overcomes the shortcomings of traditional skeletal action recognition and improves recognition accuracy.

[0139] The embodiments of the present application do not limit the method for obtaining the labeled data. For example, manual labeling, automatic labeling, or semi-automatic labeling can be used. The test data stream can be, for example, a real sample skeleton data stream and a preset motion data stream, or it can be automatically generated using the generative network of the GAN model.

[0140] The embodiments of the present application do not limit the model training process. For example, it can adopt a supervised learning training method, a semi-supervised learning training method, or an unsupervised learning training method.

[0141] The embodiments of the present application do not limit the conditions for the end of training. For example, it can be that the number of training times reaches a preset number (the preset number is, for example, 1 time, 3 times, 10 times, 100 times, 1000 times, 10,000 times, etc.), or it can be that the training data in the corresponding training set completes one or more training times, or it can be that the total loss value obtained from this training is not greater than the preset loss value.

[0142] The embodiment of the present application does not limit the labeling data, which can be, for example, one or more of Chinese characters, letters, numbers, symbols, shapes, and colors.

[0143] The embodiment of the present application does not limit the format of the data stream to be tested, which can be, for example, MP4, MPEG, AVI, nAVI, ASF, MOV, 3GP, WMV, DivX, XviD, RM, RMVB, FLV / F4V, etc.

[0144] The embodiment of the present application does not limit the size of the data stream to be tested, which can be, for example, 10KB, 11KB, 15KB, 1MB, 7MB, 10MB, 20MB, 50MB, 100MB, 200MB, etc.

[0145] The embodiment of the present application does not limit the size of the sample skeleton data stream, which can be, for example, 10KB, 11KB, 15KB, 1MB, 7MB, 10MB, 20MB, 50MB, 100MB, etc.

[0146] The embodiment of the present application does not limit the size of the preset motion data stream, which can be, for example, 10KB, 11KB, 15KB, 1MB, 7MB, 10MB, 20MB, 50MB, 100MB, etc.

[0147] The embodiment of the present application does not limit the action recognition results. The action recognition results of the sample data stream are used to indicate at least one action type corresponding to the sample data stream. The action recognition results (corresponding action types) can be squatting, standing, walking, jumping, turning, raising the upper arm, etc.

[0148] The embodiment of the present application does not limit the preset motion data stream. The preset motion data stream can be collected and produced by using an acquisition device, or it can be an existing motion data stream.

[0149] In some optional embodiments, the method for producing the preset motion data stream may be: first, the original joint point of each subject's motion in the preset motion data stream is expressed as J=(x, y, z), and the motion sequence of each subject in the preset motion data stream is expressed as S=(J1, J2, J3...J N ), where N is the number of joints. The preset motion data stream M is calculated from two consecutive frames and can be expressed as Wherein t is the number of frames of the preset motion data stream.

[0150] See also Figure 2 , Figure 2 A schematic diagram of a process for obtaining a training set provided in an embodiment of the present application is shown.

[0151] In some optional implementations, obtaining a training set includes:

[0152] Step S201: using a recording device to obtain multiple sample skeleton data streams;

[0153] Step S202: performing data preprocessing on the plurality of sample skeleton data streams to remove the sample skeleton data streams with recording failure and / or poor recording quality from the plurality of sample skeleton data streams;

[0154] Step S203: The pre-processed plurality of sample skeleton data streams and the preset motion data stream are made into a compressed file as the training set.

[0155] Therefore, a recording device is used to record multiple sample skeleton data streams and perform data preprocessing on the sample skeleton data streams, removing sample skeleton data streams that have failed to be recorded and / or have poor recording quality, retaining sample skeleton data streams that have been successfully recorded and / or have good recording quality, and making the preprocessed multiple sample skeleton data streams and preset motion data streams into a compressed file as a training set.

[0156] Data preprocessing helps improve data quality, reduce redundant and poor-quality sample skeleton data streams, thereby reducing training time, making subsequent data processing, analysis, and model training easier and more effective, improving data accuracy, maximizing data validity, and avoiding wasting computing resources; it can also give the model stronger generalization capabilities, making the training data as close as possible to the test data, thereby improving prediction accuracy, and enabling the model to learn more robust features.

[0157] Data preprocessing can highlight certain characteristics of the target object in the sample data stream, characteristic parameters of the action type, etc., which are conducive to the recognition, tracking and understanding of the target in the sample data stream.

[0158] In this application, the use of recording equipment to obtain multiple sample skeleton data streams can greatly save labor costs and reduce the safety hazards that exist when manually collecting data. It has the effect of comprehensive, efficient, and high-quality image collection. It can also realize real-time automatic recording and acquisition, and at the same time realize the planning, monitoring of skeleton movement and compression and automatic transmission of sample data streams.

[0159] The embodiment of the present application does not limit the recording device, which can be, for example, a depth camera such as Kinect.

[0160] In the embodiment of the present application, sample skeleton data streams with recording failure and / or poor recording quality include sample skeleton data streams with unsuccessful recording, blurred recording images, excessively long recording times, or excessively short recording times. The embodiment of the present application does not limit the format of the compressed file, which may be, for example, npy, npz, etc.

[0161] In some optional embodiments, the step of creating the compressed file may include using a computer program to extract only the three-dimensional coordinates of the joint points from the data stream to be tested, and then writing the motion category labels corresponding to the three-dimensional coordinates of the joint points into the compressed file. In this application, the parameters of the data set to be tested may include the number of frames, the number of people, the number of joints, and the three-dimensional coordinates of the corresponding number of joint points.

[0162] See also Figure 3 , Figure 3 A schematic diagram of a process for obtaining prediction data provided in an embodiment of the present application is shown.

[0163] In some optional embodiments, the deep learning model includes a shallow feature extraction network, a first spatiotemporal fusion module, a deep feature extraction network, and a second spatiotemporal fusion module;

[0164] The step of inputting the sample data stream in the training data into a preset deep learning model to obtain prediction data of the action recognition result of the sample data stream includes:

[0165] Step S301: using the shallow feature extraction network to perform shallow feature extraction on the sample skeleton data stream and the preset motion data stream to obtain skeleton feature information and motion feature information;

[0166] Step S302: using the first spatiotemporal fusion module to perform a first spatiotemporal fusion on the skeleton feature information and the motion feature information to obtain a skeleton-motion fusion data stream and a motion-skeleton fusion data stream;

[0167] Step S303: performing deep feature extraction on the skeleton-motion fusion data stream using the deep feature extraction network to obtain first fusion feature information;

[0168] Step S304: performing deep feature extraction on the motion-skeleton fusion data stream using the deep feature extraction network to obtain second fusion feature information;

[0169] Step S305: using the second spatiotemporal fusion module to fuse the first fused feature information and the second fused feature information to obtain prediction data of the action recognition result of the skeleton data stream.

[0170] Thus, the sample data stream is input into a preset deep learning model, which includes a shallow feature extraction network, a first spatiotemporal fusion module, a deep feature extraction network, and a second spatiotemporal fusion module; the shallow feature extraction network is used to perform shallow feature extraction on the sample skeleton data stream to obtain skeleton feature information, and the shallow feature extraction network is used to perform shallow feature extraction on the preset motion data stream to obtain motion feature information;

[0171] The skeleton feature information and the motion feature information are subjected to a first spatiotemporal fusion using a first spatiotemporal fusion module, thereby obtaining a skeleton-motion fusion data stream and a motion-skeleton fusion data stream; and a deep feature extraction network is then used to perform deep feature extraction on the skeleton-motion fusion data stream and the motion-skeleton fusion data stream, respectively, thereby obtaining first fusion feature information and second fusion feature information;

[0172] Finally, the first fused feature information and the second fused feature information are fused using a second spatiotemporal fusion module to obtain prediction data of the action recognition result of the skeleton data stream.

[0173] The sample data stream is processed using a shallow feature extraction network, a first spatiotemporal fusion module, a deep feature extraction network, and a second spatiotemporal fusion module, respectively, which can efficiently obtain prediction data for action recognition results with a high degree of intelligence.

[0174] In the skeleton-based action recognition model, networks and modules with higher average accuracy and better fusion efficiency should be used, so that the connection between the skeleton and the action can be obtained more specifically, key information can be found, and the action type corresponding to the skeleton can be located in one step.

[0175] Using the spatiotemporal fusion modules (the first spatiotemporal fusion module and the second spatiotemporal fusion module), the features of the sample skeleton data stream and the preset motion data stream are effectively fused, showing better performance in the final prediction.

[0176] See also Figure 4 、 5 , 6, Figure 4 A schematic diagram of the structure of a shallow feature extraction network provided in an embodiment of the present application is shown; Figure 5 A schematic structural diagram of a spatiotemporal fusion module provided in an embodiment of the present application is shown; Figure 6 A schematic diagram of the structure of a deep feature extraction network provided in an embodiment of the present application is shown.

[0177] In some optional embodiments, a shallow feature extraction network, such as Figure 4 As shown in the figure, a four-layer CNN structure (Conv) is used, the convolution kernel size is 3*3, the stride is 2, the number of input channels is 3, 64, 32, 64 respectively, and the number of output channels is 64, 32, 64, 64 respectively.

[0178] The space-time fusion module (including the first space-time fusion module and the second space-time fusion module) is as follows: Figure 5 In the spatiotemporal fusion module, X1 undergoes average pooling (AvgPool) and a Sigmoid activation function to extract the self-attention weights (including the first and second self-attention weights). This is then dot-multiplied with X1 to restore the original input size, and finally added to X2 to obtain the fused output Out. When X1 is skeleton feature information, X2 is motion feature information; when X1 is motion feature information, X2 is skeleton feature information.

[0179] Deep feature extraction networks, such as Figure 6 As shown in the figure, the deep feature fusion network consists of a feature extraction module composed of N1 CNN layers and N2 Transformer layers. The CNN layer consists of a 3*3 convolutional layer (Conv 3*3), a BN layer (BNLayer), a Relu activation function (Relu), and a 1*1 convolutional layer (Conv1*1). The Transformer module first divides the feature map obtained by the CNN layer into blocks and encodes the divided feature map with an encoding dimension of 768. After encoding, the parameters Q, K, and V are calculated respectively, and then multi-head self-attention is performed. The features calculated by multi-head self-attention are the features extracted from the spatiotemporal relationship.

[0180] The embodiment of the present application does not limit the first space-time fusion module and the second space-time fusion module, wherein the first space-time fusion module can be the same as the second space-time fusion module (that is, the two are the same space-time fusion module), or it can be different from the second space-time fusion module (that is, different space-time fusion modules are used).

[0181] In some optional implementations, the process of obtaining the skeleton-motion fusion data stream includes:

[0182] Downsampling the skeleton feature information to obtain low-dimensional skeleton data;

[0183] Obtaining a first self-attention weight based on the low-dimensional skeleton data and a first activation function;

[0184] Acquire the skeleton-motion fusion data stream based on the skeleton feature information, the first self-attention weight, and the motion feature information;

[0185] The process of obtaining the motion-skeleton fusion data stream includes:

[0186] Downsampling the motion feature information to obtain low-dimensional motion data;

[0187] Obtaining a second self-attention weight based on the low-dimensional motion data and the second activation function;

[0188] Based on the motion feature information, the second self-attention weight and the skeleton feature information, the motion-skeleton fusion data stream is obtained.

[0189] Thus, a skeleton-motion fusion data stream and a motion-skeleton fusion data stream are respectively obtained through the first spatiotemporal fusion module, the skeleton feature information is downsampled (i.e., an average pooling operation) to obtain low-dimensional skeleton data, and the first self-attention weight is extracted using the first activation function. Then, based on the skeleton feature information, the first self-attention weight and the motion feature information, the skeleton-motion fusion data stream is obtained (the first self-attention weight is used to perform a dot product operation with the skeleton feature information to restore the original input size, and then added to the motion feature information to obtain the skeleton-motion fusion data stream);

[0190] The motion feature information is downsampled (i.e., average pooling operation) to obtain low-dimensional motion data, and the second self-attention weight is extracted using the second activation function. Then, based on the motion feature information, the second self-attention weight and the skeleton feature information, the motion-skeleton fusion data stream is obtained (the second self-attention weight is used to perform a dot product operation with the motion feature information to restore the original input size, and then added to the skeleton feature information to obtain the motion-skeleton fusion data stream).

[0191] By utilizing self-attention weights, we can better focus on extracting features based on the skeleton and motion type, which enables the deep learning model to more accurately locate and identify, thereby generating a more accurate and comprehensive fused data stream, which brings a qualitative improvement to the performance of the deep learning model.

[0192] In some optional embodiments, the skeleton feature information is average pooled (AvgPool) to obtain low-dimensional skeleton data, and then the Sigmoid activation function is used to extract the first self-attention weight, the first self-attention weight is multiplied with the skeleton feature information to restore the original input size, and finally it is added to the motion feature information to obtain a skeleton-motion fusion data stream; the motion feature information is average pooled (AvgPool) to obtain low-dimensional motion data, and then the Sigmoid activation function is used to extract the second self-attention weight, the second self-attention weight is multiplied with the motion feature information to restore the original input size, and finally it is added to the skeleton feature information to obtain a motion-skeleton fusion data stream.

[0193] In some optional embodiments, the deep feature extraction network includes multiple CNN layers and multiple Transformer layers;

[0194] The step of performing deep feature extraction on the skeleton-motion fusion data stream using the deep feature extraction network to obtain first fusion feature information includes:

[0195] Inputting the skeleton-motion fusion data stream into the plurality of CNN layers to obtain a plurality of first feature maps corresponding to the skeleton-motion fusion data stream;

[0196] Inputting the plurality of first feature maps into the plurality of Transformer layers to obtain the first fused feature information;

[0197] The step of performing deep feature extraction on the motion-skeleton fusion data stream using the deep feature extraction network to obtain second fusion feature information includes:

[0198] Inputting the motion-skeleton fusion data stream into a plurality of CNN layers to obtain a plurality of second feature maps corresponding to the motion-skeleton fusion data stream;

[0199] Input the plurality of second feature maps into the plurality of Transformer layers to obtain the second fused feature information.

[0200] Thus, the skeleton-motion fusion data stream is input into multiple CNN layers to obtain multiple first feature maps corresponding to the skeleton-motion fusion data stream, and the multiple first feature maps are input into multiple Transformer layers to obtain first fusion feature information.

[0201] Similarly, the motion-skeleton fusion data stream is input into multiple CNN layers to obtain multiple second feature maps corresponding to the motion-skeleton fusion data stream, and the multiple second feature maps are input into multiple Transformer layers to obtain second fusion feature information.

[0202] Taking into account the characteristics that skeletons are not affected by light and the sample skeleton data stream is very robust, CNN is innovatively used to capture the spatial information of the data stream, and Transformer is used to capture the temporal information of the data stream, which reduces the amount of calculation and uses a smaller model to achieve accuracy comparable to that of a large model.

[0203] See also Figure 7 , Figure 7 A schematic diagram of another process for obtaining prediction data provided in an embodiment of the present application is shown.

[0204] In some optional embodiments, the deep learning model further includes a fusion prediction module, and the training data in the training set includes multi-person training data including multiple people and single-person training data including a single person;

[0205] The step of inputting the sample data stream in the training data into a preset deep learning model to obtain prediction data of the action recognition result of the sample data stream includes:

[0206] For each single person training data, perform the following processing:

[0207] Step S401: supplementing and rearranging the single-person sample data stream in the single-person training data to obtain a processed data stream, wherein the data format of the processed data stream is the same as the data format of the multi-person sample data stream in the multi-person training data;

[0208] Step S402: inputting the processed data stream into the fusion prediction module to obtain prediction data of the action recognition result of the processed data stream;

[0209] Step S403: restoring the data format of the prediction data of the action recognition result of the processed data stream to obtain the action types and confidence levels of multiple people;

[0210] Step S404: obtaining the action type corresponding to the highest confidence level as prediction data of the action recognition result of the single-person sample data stream.

[0211] Therefore, the input data may include a single-person sample data stream containing a single person and a multi-person sample data stream containing multiple people. For each single-person training data, the single-person sample data stream in the single-person training data is supplemented (the second, third, fourth, etc. of the single-person training data are supplemented with 0), and then the single-person sample data stream in the single-person training data is rearranged;

[0212] The rearranged single-person sample data stream is input into the fusion prediction module to obtain the predicted data of the action recognition result of the processed data stream. The data format of the predicted data is restored to obtain the action types and confidence levels of multiple people. The action type with the highest confidence level is selected as the predicted data of the action recognition result of the single-person sample data stream.

[0213] The advantage of doing this is that it can flexibly handle the number of input people. When the number of input people changes, only the data structure needs to be adjusted without adjusting the network structure. That is, single-person and multi-person situations can be processed and action recognition can be performed, thereby obtaining the action type with the highest confidence as the prediction data of the action recognition result of the single-person sample data stream.

[0214] In some optional embodiments, the fusion prediction module, such as Figure 8 As shown in the figure, in real-world applications, multiple people may interact in the same scene. The fusion prediction module addresses this problem. The input data is formatted as [B, H, W, C, P], where P is the number of people. Therefore, this data needs to be processed. First, the remaining people in the single-person sample data stream are padded with zeros. Secondly, when feeding the data into the network, the number of people P and the batch size are rearranged, so that the data input shape becomes [(BP), H, W, C]. After the fusion prediction module completes, the output shape is [(BP), num_classes], and the restored output format is [B, num_classes, P]. The Max function then uses the highest confidence level among the multi-person actions as the final prediction output.

[0215] See also Figure 8 , Figure 8 A structural diagram of a fusion prediction module provided in an embodiment of the present application is shown.

[0216] In some optional embodiments, the method further comprises:

[0217] Obtaining a model file corresponding to the action recognition model;

[0218] The model file is tested to obtain a test result.

[0219] Therefore, after the model training is completed, the model file corresponding to the action recognition model is obtained, and then the model file is tested to obtain the test results.

[0220] This method can evaluate the generalization ability of a model, thereby verifying its effectiveness. It can also predict various indicators in actual applications, specify psychological thresholds for real-world application scenarios, and select application scenarios with appropriate accuracy, thereby reducing trial and error costs during application.

[0221] In some optional implementations, after the action recognition model is obtained through training, a model file corresponding to the action recognition model is obtained; and the action recognition model file is tested using a test set.

[0222] In a specific embodiment, a depth camera may be used to first collect multiple sample skeleton data streams, perform data preprocessing on the sample skeleton data streams, remove sample skeleton data streams with failed recording and / or poor recording quality, retain sample skeleton data streams with successful recording and / or good recording quality, and obtain a preset motion data stream. The method for making the preset motion data stream may be to first determine that the original joint point of each subject's movement in the preset motion data stream is represented as J=(x, y, z), and the motion sequence of each subject in the preset motion data stream is represented as S=(J1, J2, J3...J N ), where N is the number of joints. The preset motion data stream M is calculated from two consecutive frames and can be expressed as Where t is the number of frames of the preset motion data stream; the two are then compressed into a compressed file as the test data stream, and the test data stream is input into the motion recognition model to obtain a motion recognition result of the test data stream. The training process may include: first obtaining a training set, wherein the training set includes multiple training data, and the motion recognition result of the test data stream is used to indicate at least one motion type corresponding to the test data stream.

[0223] The sample data stream is input into a preset deep learning model, which includes a shallow feature extraction network, a first spatiotemporal fusion module, a deep feature extraction network and a second spatiotemporal fusion module; the shallow feature extraction network is used to perform shallow feature extraction on the sample skeleton data stream to obtain skeleton feature information, and the shallow feature extraction network is used to perform shallow feature extraction on the preset motion data stream to obtain motion feature information, and the skeleton-motion fusion data stream and the motion-skeleton fusion data stream are respectively obtained through the first spatiotemporal fusion module, and the skeleton feature information is downsampled (i.e., average pooling operation) to obtain low-dimensional skeleton data, and the first activation function is used to extract the first self-attention weight, and then based on the skeleton feature information, The first self-attention weight and motion feature information are used to obtain the skeleton-motion fusion data stream (the first self-attention weight is used to perform a dot product operation with the skeleton feature information to restore the original input size, and then the data is added to the motion feature information to obtain the skeleton-motion fusion data stream); the motion feature information is downsampled (i.e., an average pooling operation) to obtain low-dimensional motion data, and the second self-attention weight is extracted using the second activation function, and then based on the motion feature information, the second self-attention weight and the skeleton feature information, the motion-skeleton fusion data stream is obtained (the second self-attention weight is used to perform a dot product operation with the motion feature information to restore the original input size, and then the data is added to the skeleton feature information to obtain the motion-skeleton fusion data stream).

[0224] The step of using a deep feature extraction network to perform deep feature extraction on the skeleton-motion fusion data stream to obtain the first fusion feature information may be to input the skeleton-motion fusion data stream into multiple CNN layers to obtain multiple first feature maps corresponding to the skeleton-motion fusion data stream, and input the multiple first feature maps into multiple Transformer layers to obtain the first fusion feature information; the step of using a deep feature extraction network to perform deep feature extraction on the motion-skeleton fusion data stream to obtain the second fusion feature information may be to input the motion-skeleton fusion data stream into multiple CNN layers to obtain multiple second feature maps corresponding to the motion-skeleton fusion data stream, and input the multiple second feature maps into multiple Transformer layers to obtain the second fusion feature information. Finally, the first fusion feature information and the second fusion feature information are fused using a second spatiotemporal fusion module to obtain predicted data of the action recognition result of the skeleton data stream.

[0225] In practice, the input data may include a single-person sample data stream containing a single person and a multi-person sample data stream containing multiple people. For each single-person training data set, the single-person sample data stream in the single-person training data is padded (the second, third, and fourth people in the single-person training data are padded with 0s), and then the single-person sample data stream in the single-person training data is rearranged. The rearranged single-person sample data stream is input into the fusion prediction module to obtain the predicted data for the action recognition results of the processed data stream. The data format of the predicted data is restored to obtain the action types and confidence levels of multiple people. The action type with the highest confidence level is selected as the predicted data for the action recognition results of the single-person sample data stream. That is, at input, the data format is [B, H, W, C, P], where P is the number of people. Therefore, it is necessary to process this part of the data. First, the remaining people in the single-person sample data stream are padded with 0s. Second, when inputting the data into the network, the number of people P and the batch size are rearranged, so that the data input shape becomes [(BP), H, W, C]. After the fusion prediction module completes, the output format is [(BP), num_classes], and the restored output format is [B, num_classes, P]. The Max function then uses the most confident action among multiple people as the final prediction output. This approach offers flexibility in handling the number of people in the input. When the number of people changes, only the data structure needs to be adjusted, without adjusting the network structure. This allows for processing and action recognition for both single and multi-person scenarios, resulting in the highest-confidence action type as the prediction for the action recognition result in the single-person sample data stream.

[0226] After the action recognition model training is completed, the model file corresponding to the action recognition model is obtained, and then the model file is tested using the test set to obtain the test results. The test set includes multiple sample skeleton data streams and preset motion data streams.

[0227] Device embodiment

[0228] See also Figure 9 , Figure 9 A schematic structural diagram of a motion recognition device provided by the present application is shown.

[0229] The embodiment of the present application also provides a motion recognition device, the specific implementation of which is consistent with the implementation and technical effects recorded in the above method embodiment, and some contents will not be repeated here.

[0230] An embodiment of the present application provides a motion recognition device for performing motion recognition on a human skeleton, the device comprising:

[0231] The action recognition unit 101 is configured to input a data stream to be tested into a action recognition model to obtain an action recognition result of the data stream to be tested, wherein the data stream to be tested includes a sample skeleton data stream and a preset motion data stream, and the action recognition result of the data stream to be tested is used to indicate at least one action type corresponding to the data stream to be tested;

[0232] The training process of the action recognition model includes:

[0233] Obtaining a training set, the training set including a plurality of training data, each of the training data including at least one sample data stream and annotated data of an action recognition result of the sample data stream, the action recognition result of the sample data stream being used to indicate at least one action type corresponding to the sample data stream;

[0234] For each training data in the training set, perform the following processing:

[0235] Inputting the sample data stream in the training data into a preset deep learning model to obtain prediction data of the action recognition result of the sample data stream;

[0236] Updating model parameters of the deep learning model based on the predicted data and the labeled data of the action recognition result of the sample data stream;

[0237] Detect whether the preset training end condition is met; if so, use the trained deep learning model as the action recognition model; if not, continue training the deep learning model using the next training data.

[0238] In some optional implementations, obtaining a training set includes:

[0239] Using a recording device to obtain multiple sample skeleton data streams;

[0240] performing data preprocessing on the plurality of sample skeleton data streams to remove the sample skeleton data streams with recording failure and / or poor recording quality from the plurality of sample skeleton data streams;

[0241] The pre-processed plurality of sample skeleton data streams and the preset motion data stream are made into a compressed file as the training set.

[0242] In some optional embodiments, the deep learning model includes a shallow feature extraction network, a first spatiotemporal fusion module, a deep feature extraction network, and a second spatiotemporal fusion module;

[0243] The step of inputting the sample data stream in the training data into a preset deep learning model to obtain prediction data of the action recognition result of the sample data stream includes:

[0244] Performing shallow feature extraction on the sample skeleton data stream and the preset motion data stream using the shallow feature extraction network to obtain skeleton feature information and motion feature information;

[0245] Performing a first spatiotemporal fusion on the skeleton feature information and the motion feature information using the first spatiotemporal fusion module to obtain a skeleton-motion fusion data stream and a motion-skeleton fusion data stream;

[0246] Performing deep feature extraction on the skeleton-motion fusion data stream using the deep feature extraction network to obtain first fusion feature information;

[0247] Performing deep feature extraction on the motion-skeleton fusion data stream using the deep feature extraction network to obtain second fusion feature information;

[0248] The first fused feature information and the second fused feature information are fused using the second spatiotemporal fusion module to obtain prediction data of the action recognition result of the skeleton data stream.

[0249] In some optional implementations, the process of obtaining the skeleton-motion fusion data stream includes:

[0250] Downsampling the skeleton feature information to obtain low-dimensional skeleton data;

[0251] Obtaining a first self-attention weight based on the low-dimensional skeleton data and a first activation function;

[0252] Acquire the skeleton-motion fusion data stream based on the skeleton feature information, the first self-attention weight, and the motion feature information;

[0253] The process of obtaining the motion-skeleton fusion data stream includes:

[0254] Downsampling the motion feature information to obtain low-dimensional motion data;

[0255] Obtaining a second self-attention weight based on the low-dimensional motion data and the second activation function;

[0256] Based on the motion feature information, the second self-attention weight and the skeleton feature information, the motion-skeleton fusion data stream is obtained.

[0257] In some optional embodiments, the deep feature extraction network includes multiple CNN layers and multiple Transformer layers;

[0258] The step of performing deep feature extraction on the skeleton-motion fusion data stream using the deep feature extraction network to obtain first fusion feature information includes:

[0259] Inputting the skeleton-motion fusion data stream into the plurality of CNN layers to obtain a plurality of first feature maps corresponding to the skeleton-motion fusion data stream;

[0260] Inputting the plurality of first feature maps into the plurality of Transformer layers to obtain the first fused feature information;

[0261] The step of performing deep feature extraction on the motion-skeleton fusion data stream using the deep feature extraction network to obtain second fusion feature information includes:

[0262] Inputting the motion-skeleton fusion data stream into a plurality of CNN layers to obtain a plurality of second feature maps corresponding to the motion-skeleton fusion data stream;

[0263] Input the plurality of second feature maps into the plurality of Transformer layers to obtain the second fused feature information.

[0264] In some optional embodiments, the deep learning model further includes a fusion prediction module, and the training data in the training set includes multi-person training data including multiple people and single-person training data including a single person;

[0265] The sample data stream in the training data is input into a preset deep learning model to obtain prediction data of the action recognition result of the sample data stream:

[0266] For each single person training data, perform the following processing:

[0267] Performing supplementary processing and rearrangement processing on a single-person sample data stream in the single-person training data to obtain a processed data stream, wherein the data format of the processed data stream is the same as the data format of the multi-person sample data stream in the multi-person training data;

[0268] Inputting the processed data stream into the fusion prediction module to obtain prediction data of the action recognition result of the processed data stream;

[0269] Restoring the data format of the prediction data of the action recognition result of the processed data stream to obtain the action types and confidence levels of multiple people;

[0270] The action type corresponding to the highest confidence level is obtained as prediction data of the action recognition result of the single-person sample data stream.

[0271] In some optional embodiments, the device is further used to:

[0272] Obtaining a model file corresponding to the action recognition model;

[0273] The model file is tested to obtain a test result.

[0274] Device Example

[0275] The present application also provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program. The steps of any of the above methods are consistent with the implementation methods and technical effects recorded in the implementation methods of the above methods, and some contents are not repeated here.

[0276] See also Figure 10 , Figure 10 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown.

[0277] The electronic device includes at least one memory 210, at least one processor 220, and a bus 230 connecting different platform systems.

[0278] The memory 210 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 211 and / or a cache memory 212 , and may further include a read-only memory (ROM) 213 .

[0279] The memory 210 also stores a computer program, which can be executed by the processor 220, so that the processor 220 implements the steps of any of the above methods.

[0280] The memory 210 may also include a utility 214 having at least one program module 215, such program module 215 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0281] Accordingly, the processor 220 may execute the aforementioned computer program and the utility 214 .

[0282] The processor 220 may be implemented as one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), or other electronic components.

[0283] Bus 230 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures.

[0284] The electronic device may also communicate with one or more external devices 240, such as a keyboard, pointing device, Bluetooth device, etc., and may also communicate with one or more devices capable of interacting with the electronic device, and / or any device that enables the electronic device to communicate with one or more other computing devices (e.g., a router, modem, etc.). Such communication may be performed via input / output interface 250. Furthermore, the electronic device may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via network adapter 260. The network adapter 260 may communicate with other modules of the electronic device via bus 230. It should be understood that, although not shown in the figures, other hardware and / or software modules may be used in conjunction with the electronic device, including but not limited to microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0285] Media Examples

[0286] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of any of the above methods are implemented. Its specific implementation method is consistent with the implementation method and the technical effect achieved in the above method embodiment, and some contents will not be repeated here.

[0287] See also Figure 11 , Figure 11 A schematic diagram of the structure of a program product provided in an embodiment of the present application is shown.

[0288] The program product is used to implement any of the above methods. The program product can use a portable compact disc read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited to this. In the embodiment of the present application, the readable storage medium can be any tangible medium that contains or stores a program, and the program can be used by or in combination with an instruction execution system, device or device. The program product can use any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0289] A computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, or any suitable combination thereof. The program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java and C++, as well as conventional procedural programming languages such as C, Python, or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a standalone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. Where a remote computing device is involved, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0290] This application is explained from the perspectives of purpose of use, effectiveness, progress and novelty, and has met the functional enhancement and use requirements emphasized by the Patent Law. The above description and drawings of this application are only preferred embodiments of this application and are not intended to limit this application. Therefore, all structures, devices, features, etc. that are similar or identical to those of this application, that is, all equivalent replacements or modifications made in accordance with the scope of the patent application of this application, should fall within the scope of protection of the patent application of this application.

Claims

1. A method for motion recognition, characterized in that: For performing action recognition on a human skeleton, the method comprises: Inputting a data stream to be tested into a motion recognition model to obtain a motion recognition result of the data stream to be tested, wherein the data stream to be tested includes a sample skeleton data stream and a preset motion data stream, and the motion recognition result of the data stream to be tested is used to indicate at least one motion type corresponding to the data stream to be tested; The training process of the action recognition model includes: Obtaining a training set, the training set including a plurality of training data, each of the training data including at least one sample data stream and annotated data of an action recognition result of the sample data stream, the action recognition result of the sample data stream being used to indicate at least one action type corresponding to the sample data stream; For each training data in the training set, perform the following processing: Inputting the sample data stream in the training data into a preset deep learning model to obtain prediction data of the action recognition result of the sample data stream; the deep learning model includes a shallow feature extraction network, a first spatiotemporal fusion module, a second spatiotemporal fusion module and a deep feature extraction network; inputting the sample data stream in the training data into the preset deep learning model to obtain prediction data of the action recognition result of the sample data stream, including: using the shallow feature extraction network to perform shallow feature extraction on the sample skeleton data stream and the preset motion data stream to obtain skeleton feature information and motion feature information; using the first spatiotemporal fusion module to perform first spatiotemporal fusion on the skeleton feature information and the motion feature information to obtain a skeleton-motion fusion data stream and a motion-skeleton fusion data stream; using the deep feature extraction network to perform deep feature extraction on the skeleton-motion fusion data stream to obtain first fusion feature information; using the deep feature extraction network to perform deep feature extraction on the motion-skeleton fusion data stream to obtain second fusion feature information; using the second spatiotemporal fusion module to fuse the first fusion feature information and the second fusion feature information to obtain prediction data of the action recognition result of the skeleton data stream; Updating model parameters of the deep learning model based on the predicted data and the labeled data of the action recognition result of the sample data stream; Check whether the preset training end condition is met; if so, use the trained deep learning model as the action recognition model; if not, continue training the deep learning model using the next training data.

2. The motion recognition method according to claim 1, characterized in that The obtaining of the training set comprises: Using a recording device to obtain multiple sample skeleton data streams; performing data preprocessing on the plurality of sample skeleton data streams to remove the sample skeleton data streams that have failed to be recorded and / or have poor recording quality from the plurality of sample skeleton data streams; The pre-processed plurality of sample skeleton data streams and the preset motion data stream are made into a compressed file as the training set.

3. The motion recognition method according to claim 1, wherein: The process of obtaining the skeleton-motion fusion data stream includes: Downsampling the skeleton feature information to obtain low-dimensional skeleton data; Obtaining a first self-attention weight based on the low-dimensional skeleton data and a first activation function; Acquire the skeleton-motion fusion data stream based on the skeleton feature information, the first self-attention weight, and the motion feature information; The process of obtaining the motion-skeleton fusion data stream includes: Downsampling the motion feature information to obtain low-dimensional motion data; Obtaining a second self-attention weight based on the low-dimensional motion data and the second activation function; Based on the motion feature information, the second self-attention weight and the skeleton feature information, the motion-skeleton fusion data stream is obtained.

4. The motion recognition method according to claim 3, characterized in that The deep feature extraction network includes multiple CNN layers and multiple Transformer layers; The step of performing deep feature extraction on the skeleton-motion fusion data stream using the deep feature extraction network to obtain first fusion feature information includes: Inputting the skeleton-motion fusion data stream into the plurality of CNN layers to obtain a plurality of first feature maps corresponding to the skeleton-motion fusion data stream; Inputting the plurality of first feature maps into the plurality of Transformer layers to obtain the first fused feature information; The step of performing deep feature extraction on the motion-skeleton fusion data stream using the deep feature extraction network to obtain second fusion feature information includes: Inputting the motion-skeleton fusion data stream into a plurality of CNN layers to obtain a plurality of second feature maps corresponding to the motion-skeleton fusion data stream; Input the plurality of second feature maps into the plurality of Transformer layers to obtain the second fused feature information.

5. The motion recognition method according to claim 1, wherein: The deep learning model further includes a fusion prediction module, and the training data in the training set includes multi-person training data including multiple people and single-person training data including a single person; The step of inputting the sample data stream in the training data into a preset deep learning model to obtain prediction data of the action recognition result of the sample data stream includes: For each single person training data, perform the following processing: Performing supplementary processing and rearrangement processing on a single-person sample data stream in the single-person training data to obtain a processed data stream, wherein the data format of the processed data stream is the same as the data format of the multi-person sample data stream in the multi-person training data; Inputting the processed data stream into the fusion prediction module to obtain prediction data of the action recognition result of the processed data stream; Restoring the data format of the prediction data of the action recognition result of the processed data stream to obtain the action types and confidence levels of multiple people; The action type corresponding to the highest confidence level is obtained as prediction data of the action recognition result of the single-person sample data stream.

6. The motion recognition method according to claim 1, characterized in that: The method further comprises: Obtaining a model file corresponding to the action recognition model; The model file is tested to obtain a test result.

7. A motion recognition device, characterized in that: For performing motion recognition on a human skeleton, the device comprises: An action recognition unit is configured to input a data stream to be tested into an action recognition model to obtain an action recognition result of the data stream to be tested, wherein the data stream to be tested includes a sample skeleton data stream and a preset motion data stream, and the action recognition result of the data stream to be tested is used to indicate at least one action type corresponding to the data stream to be tested; The training process of the action recognition model includes: Obtaining a training set, the training set including a plurality of training data, each of the training data including at least one sample data stream and annotated data of an action recognition result of the sample data stream, the action recognition result of the sample data stream being used to indicate at least one action type corresponding to the sample data stream; For each training data in the training set, perform the following processing: Inputting the sample data stream in the training data into a preset deep learning model to obtain prediction data of the action recognition result of the sample data stream; the deep learning model includes a shallow feature extraction network, a first spatiotemporal fusion module, a second spatiotemporal fusion module and a deep feature extraction network; inputting the sample data stream in the training data into the preset deep learning model to obtain prediction data of the action recognition result of the sample data stream, including: using the shallow feature extraction network to perform shallow feature extraction on the sample skeleton data stream and the preset motion data stream to obtain skeleton feature information and motion feature information; using the first spatiotemporal fusion module to perform first spatiotemporal fusion on the skeleton feature information and the motion feature information to obtain a skeleton-motion fusion data stream and a motion-skeleton fusion data stream; using the deep feature extraction network to perform deep feature extraction on the skeleton-motion fusion data stream to obtain first fusion feature information; using the deep feature extraction network to perform deep feature extraction on the motion-skeleton fusion data stream to obtain second fusion feature information; using the second spatiotemporal fusion module to fuse the first fusion feature information and the second fusion feature information to obtain prediction data of the action recognition result of the skeleton data stream; Updating model parameters of the deep learning model based on the predicted data and the labeled data of the action recognition result of the sample data stream; Check whether the preset training end condition is met; if so, use the trained deep learning model as the action recognition model; if not, continue training the deep learning model using the next training data.

8. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor is configured to implement the steps of the method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images