Human body action recognition method and device, computer equipment and storage medium

Abstract

The invention relates to a human body action recognition method and device, computer equipment and a storage medium. The method comprises the following steps: acquiring a human body skeleton action sequence and corresponding acceleration information of a to-be-detected human body; generating a multi-modal information data set according to the human skeleton action sequence and the corresponding acceleration information; inputting the multi-modal information data set into a pre-trained human body action recognition model, and outputting a recognition action category result of the to-be-detected human body; wherein the human body action recognition model comprises a spatial-temporal feature extraction module, a motion feature extraction module, a multi-source heterogeneous information fusion module and an action recognition and classification module. By adopting the method, the action recognition accuracy and the action recognition efficiency are improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, computer equipment and storage medium for human motion recognition. Background Art

[0002] With the continuous development of artificial intelligence and the increasing popularity of smart wearable devices, in many scenarios, it is necessary to identify human movements in videos to determine the types of human movements. For example, in a monitoring system, in order to perceive human intentions, it is necessary to identify human movements in videos obtained by the monitoring system. In a human-computer interaction system, in order to understand human behavior, it is necessary to identify human movements in videos obtained by the human-computer interaction system.

[0003] However, traditional human motion recognition methods have problems such as low motion recognition accuracy or low motion recognition efficiency. Summary of the invention

[0004] Based on this, it is necessary to provide a human motion recognition method, device, computer equipment and storage medium that can improve the low accuracy and efficiency of motion recognition in order to address the above technical problems.

[0005] In a first aspect, a method for human motion recognition is provided, the method comprising:

[0006] Obtaining a human skeleton motion sequence and corresponding acceleration information of a human body to be tested;

[0007] Generate a multimodal information data set according to the human skeleton motion sequence and the corresponding acceleration information;

[0008] The multimodal information data set is input into a pre-trained human motion recognition model, and the recognition motion category result of the human body to be tested is output; wherein the human motion recognition model includes a spatiotemporal feature extraction module, a motion feature extraction module, a multi-source heterogeneous information fusion module and a motion recognition classification module; wherein the spatiotemporal feature extraction module is used to extract the spatiotemporal features of the skeletal spatiotemporal graph of the human skeletal motion sequence to obtain the spatiotemporal feature information; the motion feature extraction module is used to extract the motion features of the acceleration information to obtain the motion feature information; the multi-source heterogeneous information fusion module is used to perform feature fusion processing on the spatiotemporal feature information and the motion feature information to obtain the fused feature information; the motion recognition classification module is used to perform action recognition classification on the fused feature information to obtain the recognition action category result.

[0009] In one of the embodiments, a multimodal information data set is input into a pre-trained human motion recognition model, and the recognition motion category result of the human body to be tested is output, including: constructing a spatiotemporal graph according to the human skeleton motion sequence in the multimodal information data set to obtain a skeleton spatiotemporal graph; extracting spatiotemporal features from the skeleton spatiotemporal graph based on a spatiotemporal feature extraction module to obtain spatiotemporal feature information; wherein the spatiotemporal feature extraction module adopts an ST-GCN spatiotemporal graph convolutional network; extracting motion features from acceleration information in the multimodal information data set based on a motion feature extraction module to obtain motion feature information; wherein the motion feature extraction module adopts a CNN-Transformer network; based on a multi-source heterogeneous information fusion module, performing feature fusion processing on the spatiotemporal feature information and the motion feature information to obtain fused feature information; wherein the multi-source heterogeneous information fusion module adopts an MSH IF multi-source heterogeneous information fuser; based on a motion recognition classification module, performing motion recognition classification on the fused feature information to obtain a recognition motion category result; wherein the motion recognition classification module adopts a classifier.

[0010] In one of the embodiments, obtaining a human skeleton motion sequence and corresponding acceleration information of a human body to be tested includes: obtaining a target video signal and a corresponding target inertial signal of the human body to be tested; the target video signal includes a video signal of multiple motions of the human body to be tested continuously acquired by an ordinary camera; the target inertial signal includes acceleration signals of multiple motions of the human body to be tested acquired by each inertial sensor; the acceleration signal includes an X-axis acceleration signal, a Y-axis acceleration signal and a Z-axis acceleration signal; the inertial sensor is installed at a corresponding part to be tested of the human body to be tested; performing time alignment processing on the target video signal and the corresponding target inertial signal to obtain a target video signal after the time alignment processing and a corresponding target inertial signal after the time alignment processing; performing signal processing on the target video signal after the time alignment processing and the corresponding target inertial signal after the time alignment processing to obtain a human skeleton motion sequence and corresponding acceleration information.

[0011] In one of the embodiments, signal processing is performed on the target video signal after time alignment processing and the corresponding target inertial signal after time alignment processing to obtain a human skeletal motion sequence and corresponding acceleration information, including: inputting the target video signal after time alignment processing into a human posture estimation algorithm to obtain a human skeletal motion sequence; wherein the human posture estimation algorithm includes an Openpose algorithm and an Alphapose algorithm.

[0012] In one of the embodiments, signal processing is performed on the target video signal after time alignment processing and the corresponding target inertial signal after time alignment processing to obtain a human skeleton motion sequence and corresponding acceleration information, including: performing a two-norm transformation processing on the X-axis acceleration signal, the Y-axis acceleration signal and the Z-axis acceleration signal in each acceleration signal of the target inertial signal after time alignment processing to obtain acceleration information.

[0013] In one embodiment, the parts to be tested include the upper arms, waist and thighs; and the recognized action category results include sitting, walking, running, jumping and standing.

[0014] In one of the embodiments, the method includes: obtaining a preset number of historical human skeletal motion sequences, corresponding historical acceleration information and corresponding historical motion type labels of historical human bodies to be tested, and generating a corresponding historical multimodal information data sample set according to the historical human skeletal motion sequences, corresponding historical acceleration information and corresponding historical motion type labels of each historical human body to be tested; randomly dividing each historical multimodal information data sample set to generate a training sample set and a test sample set; training a preset human motion recognition initial model according to the training sample set, and testing the human motion recognition initial model according to the test sample set, adjusting the model parameters of the human motion recognition initial model based on the indicators obtained from the training and testing until the indicators meet the preset requirements, and generating a human motion recognition model to output the recognition action category results based on the human motion recognition model.

[0015] In a second aspect, a human motion recognition device is provided, the device comprising an information acquisition module, a set generation module and a motion recognition module.

[0016] Among them, the information acquisition module is used to obtain the human skeleton motion sequence and the corresponding acceleration information of the human body to be tested; the set generation module is used to generate a multimodal information data set according to the human skeleton motion sequence and the corresponding acceleration information; the action recognition module is used to input the multimodal information data set into a pre-trained human motion recognition model, and output the recognition action category result of the human body to be tested; wherein the human motion recognition model includes a spatiotemporal feature extraction module, a motion feature extraction module, a multi-source heterogeneous information fusion module and a motion recognition classification module; wherein the spatiotemporal feature extraction module is used to extract the spatiotemporal features of the skeleton spatiotemporal graph of the human skeleton motion sequence to obtain the spatiotemporal feature information; the motion feature extraction module is used to extract the motion features of the acceleration information to obtain the motion feature information; the multi-source heterogeneous information fusion module is used to perform feature fusion processing on the spatiotemporal feature information and the motion feature information to obtain the fused feature information; the action recognition classification module is used to perform action recognition classification on the fused feature information to obtain the recognition action category result.

[0017] In a third aspect, a computer device is provided. The computer device includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the steps of any method in the above method embodiments are implemented.

[0018] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of any method in the above method embodiments are implemented.

[0019] The above-mentioned human motion recognition method, device, computer equipment and storage medium obtain the human skeleton motion sequence and corresponding acceleration information of the human body to be tested; then, a multimodal information data set is generated according to the human skeleton motion sequence and the corresponding acceleration information; then, the multimodal information data set is input into a pre-trained human motion recognition model, and the recognition motion category result of the human body to be tested is output; wherein the human motion recognition model includes a spatiotemporal feature extraction module, a motion feature extraction module, a multi-source heterogeneous information fusion module and a motion recognition classification module; wherein the spatiotemporal feature extraction module is used to extract the spatiotemporal features of the skeleton spatiotemporal graph of the human skeleton motion sequence to obtain the spatiotemporal feature information; the motion feature extraction module is used to extract the motion features of the acceleration information to obtain the motion feature information; the multi-source heterogeneous information fusion module is used to perform feature fusion processing on the spatiotemporal feature information and the motion feature information to obtain the fused feature information; the motion recognition classification module is used to perform motion recognition classification on the fused feature information to obtain the recognition action category result, which improves the accuracy and convenience of human motion recognition and also improves the motion recognition efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A diagram of an application environment of a human motion recognition method in one embodiment;

[0021] Figure 2 is a schematic flow chart of a human motion recognition method in one embodiment;

[0022] Figure 3 A schematic diagram of a flow chart of obtaining a human skeleton motion sequence and corresponding acceleration information of a human body to be tested in one embodiment;

[0023] Figure 4 A schematic diagram of a flow chart of inputting a multimodal information data set into a pre-trained human action recognition model and outputting a recognition action category result of a human body to be tested in an embodiment;

[0024] Figure 5 A schematic diagram of inputting a multimodal information data set into a pre-trained human action recognition model and outputting a recognition action category result of a human body to be tested in an embodiment;

[0025] Figure 6 Schematic diagram of the architecture of the CNN-Transformer network;

[0026] Figure 7 This is a schematic diagram of the structure of the multi-head attention mechanism;

[0027] Figure 8 A schematic diagram of a process for generating a human action recognition model in one embodiment;

[0028] Fig. 9 is a structural block diagram of a human motion recognition device in one embodiment;

[0029] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0031] In order to facilitate understanding of the present application, the present application will be described more fully below with reference to the relevant drawings. Embodiments of the present application are provided in the drawings. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive.

[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0033] It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, without departing from the scope of this application, a first resistor may be referred to as a second resistor, and similarly, a second resistor may be referred to as a first resistor. Both the first resistor and the second resistor are resistors, but they are not the same resistor.

[0034] It can be understood that the “connection” in the following embodiments should be understood as “electrical connection”, “communication connection”, etc. if the connected circuits, modules, units, etc. have electrical signals or data transmission between each other.

[0035] When used herein, the singular forms "a", "an", and "said / the" may also include plural forms, unless the context clearly indicates otherwise. It should also be understood that the terms "include / comprise" or "have" etc. specify the presence of stated features, wholes, steps, operations, components, parts or combinations thereof, but do not exclude the possibility of the presence or addition of one or more other features, wholes, steps, operations, components, parts or combinations thereof.

[0036] The human action recognition method provided in this application can be applied to Figure 1 In the application environment shown, the terminal 102 communicates with the server 104 through a network. The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices, and the server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0037] In one embodiment, Figure 2 As shown, a human action recognition method is provided, and the method is applied to Figure 1 Taking the server 104 in the example as an example, the following steps are included:

[0038] Step 201, obtaining a human skeleton motion sequence and corresponding acceleration information of a human body to be tested.

[0039] The human skeleton motion sequence is determined based on the video signals of multiple motions of the human body to be tested continuously collected by an ordinary camera; the acceleration information corresponding to the human skeleton motion sequence of the human body to be tested is determined based on the acceleration signals of multiple motions of the human body to be tested collected by each inertial sensor. Specifically, the server 104 obtains the human skeleton motion sequence of the human body to be tested and the acceleration information corresponding to the human skeleton motion sequence of the human body to be tested.

[0040] In one embodiment, if Figure 3 As shown, obtaining the human skeleton motion sequence and corresponding acceleration information of the human body to be tested includes steps 301 to 303.

[0041] Step 301, obtaining a target video signal and a corresponding target inertial signal of a human body to be tested.

[0042] Among them, the target video signal includes video signals of multiple movements of the human body to be measured continuously collected by an ordinary camera; the target inertial signal includes acceleration signals of multiple movements of the human body to be measured collected by each inertial sensor; the acceleration signal includes X-axis acceleration signal, Y-axis acceleration signal and Z-axis acceleration signal; the inertial sensor is installed at the corresponding part to be measured of the human body to be measured.

[0043] In one embodiment, the parts to be measured include upper arms, waist and thighs. It is understandable that installing inertial sensors on the upper arms, waist and thighs of the human body to be measured can efficiently and accurately collect acceleration signals of multiple movements of the human body to be measured, saving costs.

[0044] Step 302 , time-aligning the target video signal and the corresponding target inertial signal to obtain the time-aligned target video signal and the corresponding time-aligned target inertial signal.

[0045] Specifically, the server 104 performs time alignment processing on the target video signal and the corresponding target inertial signal to obtain a target video signal after time alignment processing and a target inertial signal after time alignment processing, thereby avoiding time asynchrony between the target video signal and the corresponding target inertial signal and improving the accuracy of human motion recognition.

[0046] Step 303 , performing signal processing on the time-aligned target video signal and the corresponding time-aligned target inertial signal to obtain a human skeleton motion sequence and corresponding acceleration information.

[0047] Specifically, the server 104 performs signal processing on the target video signal after time alignment processing and the corresponding target inertial signal after time alignment processing to obtain the human skeletal motion sequence and the acceleration information corresponding to the human skeletal motion sequence, thereby improving the efficiency and convenience of obtaining the human skeletal motion sequence and the corresponding acceleration information.

[0048] In one embodiment, signal processing is performed on the time-aligned target video signal and the corresponding time-aligned target inertial signal to obtain a human skeleton motion sequence and corresponding acceleration information, including:

[0049] The target video signal after time alignment is input into the human posture estimation algorithm to obtain the human skeleton action sequence.

[0050] The human body posture estimation algorithm includes the Openpose algorithm and the Alphapose algorithm. The server 104 inputs the target video signal after time alignment processing into the human body posture estimation algorithm to obtain the human body skeleton action sequence, thereby improving the efficiency and convenience of obtaining the human body skeleton action sequence.

[0051] In a specific example, the human skeleton action sequence obtained by the Openpose algorithm may include, but is not limited to, data of 18 key skeleton points or 25 key skeleton points. The above is only a specific example, which can be flexibly set according to user needs in actual applications and is not limited here.

[0052] In this embodiment, the server 104 inputs the target video signal after time alignment processing into the human body posture estimation algorithm to obtain the human body skeleton motion sequence, thereby improving the convenience and efficiency of obtaining the human body skeleton motion sequence.

[0053] In one embodiment, signal processing is performed on the time-aligned target video signal and the corresponding time-aligned target inertial signal to obtain a human skeleton motion sequence and corresponding acceleration information, including:

[0054] The X-axis acceleration signal, the Y-axis acceleration signal and the Z-axis acceleration signal of each acceleration signal of the target inertial signal after the time alignment processing are subjected to a two-norm transformation processing to obtain acceleration information.

[0055] Specifically, the server 104 may perform a two-norm conversion process on the X-axis acceleration signal, the Y-axis acceleration signal, and the Z-axis acceleration signal of each acceleration signal of the target inertial signal after the time alignment process to obtain acceleration information.

[0056] In a specific example, the acceleration information can be obtained based on the following expression:

[0057]

[0058] Among them, A is the acceleration information; A x is the X-axis acceleration signal; A y A is the Y-axis acceleration signal; z is the Z-axis acceleration signal; in addition, the X-axis, Y-axis and Z-axis form a spatial rectangular coordinate system with the inertial sensor as the origin. The above is only a specific example, which can be flexibly set according to user needs in actual applications and is not limited here.

[0059] In this embodiment, the X-axis acceleration signal, the Y-axis acceleration signal, and the Z-axis acceleration signal of each acceleration signal of the target inertial signal after time alignment are subjected to a two-norm transformation process to obtain acceleration information, thereby improving the efficiency and convenience of obtaining the acceleration information.

[0060] In this embodiment, a target video signal and a corresponding target inertial signal of a human body to be tested are obtained; then, the target video signal and the corresponding target inertial signal are time-aligned to obtain a target video signal after the time alignment processing and a corresponding target inertial signal after the time alignment processing; then, signal processing is performed on the target video signal after the time alignment processing and the corresponding target inertial signal after the time alignment processing to obtain a human skeleton motion sequence and corresponding acceleration information, thereby improving the efficiency and convenience of obtaining the human skeleton motion sequence and the corresponding acceleration information.

[0061] Step 202: Generate a multimodal information data set based on the human skeleton motion sequence and the corresponding acceleration information.

[0062] Specifically, the server 104 can generate a multimodal information data set according to the human skeleton motion sequence of the human body to be tested and the acceleration information corresponding to the human skeleton motion sequence.

[0063] Step 203: input the multimodal information data set into a pre-trained human action recognition model, and output the recognition action category result of the human body to be tested.

[0064] Among them, the human motion recognition model includes a spatiotemporal feature extraction module, a motion feature extraction module, a multi-source heterogeneous information fusion module and a motion recognition classification module; among them, the spatiotemporal feature extraction module is used to extract the spatiotemporal features of the skeletal spatiotemporal graph of the human skeletal motion sequence to obtain the spatiotemporal feature information; the motion feature extraction module is used to extract the motion features of the acceleration information to obtain the motion feature information; the multi-source heterogeneous information fusion module is used to perform feature fusion processing on the spatiotemporal feature information and the motion feature information to obtain the fused feature information; the motion recognition classification module is used to perform action recognition classification on the fused feature information to obtain the recognized action category result.

[0065] The recognition action category results include sitting, walking, running, jumping and standing. Specifically, the server 104 inputs the multimodal information data set into the pre-trained human action recognition model to output accurate recognition action category results of the human body to be tested, thereby improving the accuracy and convenience of human action recognition and the efficiency of action recognition.

[0066] In one embodiment, if Figure 4 and Figure 5 As shown, a multimodal information data set is input into a pre-trained human action recognition model, and a recognition action category result of a human body to be tested is output, including steps 401 to 405.

[0067] Step 401, constructing a spatiotemporal graph according to the human skeleton motion sequence in the multimodal information data set to obtain a skeleton spatiotemporal graph.

[0068] Specifically, the server 104 can construct a spatiotemporal graph based on the human skeleton motion sequence in the multimodal information data set to obtain a skeleton spatiotemporal graph, thereby facilitating the extraction of corresponding spatiotemporal feature information through the skeleton spatiotemporal graph.

[0069] In a specific example, the human skeleton action sequence may include, but is not limited to, 18 key skeleton points or 25 key skeleton points. The skeleton spatiotemporal graph may be represented as G = {V, E}. Where V = {v ti|t=1,...,T;i=1,...,N};v ti It is represented as the i-th joint node in the t-th frame, N is the maximum number of joint nodes, T is the maximum number of frames, and V is the node matrix set; the node matrix set includes all joint nodes in the human skeleton action sequence, and the number of joint nodes set in the node matrix set can be but not limited to 18 or 25. The joint nodes can also be understood as key bone points. The expression of the i-th joint node in the t-th frame is composed of the joint nodes and the corresponding joint node confidences. ti ), it is worth noting that the construction of the skeletal space-time graph consists of two natural link structures. First, a spatial graph is constructed in each frame according to the natural connection relationship of the human skeleton; then, the temporal relationship of the joint nodes, i.e., the key skeletal points, is represented by the edges between frames. Figure 5 The above is only a specific example, which can be flexibly set according to user needs in actual applications and is not limited here.

[0070] Step 402: extracting spatiotemporal features from the skeletal spatiotemporal graph based on the spatiotemporal feature extraction module to obtain spatiotemporal feature information.

[0071] The spatiotemporal feature extraction module uses the ST-GCN spatiotemporal graph convolutional network. Specifically, the server 104 extracts spatiotemporal features from the skeletal spatiotemporal graph based on the spatiotemporal feature extraction module to obtain spatiotemporal feature information.

[0072] In a specific example, the spatiotemporal feature extraction module divides the spatial bone points in the skeletal spatiotemporal graph into three subsets through the adjacency matrix of the graph convolution: the root node itself, the set of neighboring nodes close to the center of gravity of the skeleton, and the set of neighboring nodes far away from the center of gravity of the skeleton. The spatiotemporal feature extraction module uses the ST-GCN spatiotemporal graph convolution network to fuse the input adjacency matrix subset with the skeletal spatiotemporal graph with the help of the graph convolution module to obtain the spatiotemporal feature information. The temporal convolution network TCN can also be used to effectively model the temporal information of human skeletal action sequences in different frames.

[0073] Specifically, the expression of the spatiotemporal feature extraction module is:

[0074]

[0075] Among them, f out (v ti ) is the output feature information of the spatiotemporal feature extraction module at the i-th joint node in the t-th frame, f in (v tj ) is the input feature of the jth joint node in the tth frame, B(v ti ) is the neighboring node set of the i-th joint node in the t-th frame, l ti (vtj ) is the label mapping of the i-th joint node in the t-th frame, Z ti (v ti ) is a normalization term. In addition, Z ti (v tj )=|{v tk |l ti (v tk )=l ti (v tj )}| is equal to the cardinality of the corresponding subset.

[0076] Extending the concept of neighborhood to include nodes connected in time, the expression of neighborhood is

[0077]

[0078] Where Γ is the size of the temporal convolution kernel, and the label mapping l is modified ST The expression is:

[0079]

[0080] The expression of the simplified spatiotemporal feature extraction module is:

[0081]

[0082] Wherein, A is the adjacency matrix, Λ is the degree matrix, and W is the weight matrix. The above are only specific examples, which can be flexibly set according to user needs in actual applications and are not limited here.

[0083] Step 403: extract motion features from the acceleration information in the multimodal information data set based on the motion feature extraction module to obtain motion feature information.

[0084] The motion feature extraction module uses a CNN-Transformer network. Based on the motion feature extraction module, the server 104 can extract motion features from the acceleration information in the multimodal information data set to obtain motion feature information, thereby overcoming the shortcoming of insufficient encoding of human motion learning information of the human body to be tested through a single path of the ST-GCN spatiotemporal graph convolutional network.

[0085] In a specific example, the self-attention mechanism in the CNN-Transformer network architecture is very effective in capturing long-range dependencies in the data, making it highly effective in various fields, such as natural language processing (NLP), large models, and traffic flow prediction. Figure 6As shown in the figure, CNN learns the spatial features in the multi-dimensional synthetic acceleration signal, uses Transformer to model the temporal features in the acceleration signal, and builds a CNN-Transformer network model to learn the motion feature information in the habitual data.

[0086] Transformer mainly consists of three basic components: multi-head self-attention mechanism, feedforward layer and residual connection module. Figure 7 The structure diagram of the multi-head attention mechanism is shown, in which the single-head self-attention mechanism head i It is expressed as:

[0087]

[0088] The query vectors can be obtained through the linear transformation matrix. Query vector The value obtained by the query

[0089] Among them, the multi-head self-attention mechanism MultiHeadAttention consists of multiple single-head self-attention mechanisms:

[0090] MultiHeadAttention(X q ,X k ,X v )=Concat(head1,head2,...,head n )W 0

[0091] Concat is a concatenation operation that concatenates all attention values. 0 is the weight matrix.

[0092] Specifically, the output feature vector output of the i-th layer Transformer encoder model Res for:

[0093] output Res =LN(input+DP(MultiHeadAttention(input)))

[0094] output=LN(output Res +DP(FeedForward(output Res )))

[0095] Among them, input is the input feature vector of the i-th layer Transformer encoder, output Resis the intermediate feature vector of the i-th layer Transformer encoder, output is the output feature vector of the i-th layer Transformer encoder, DP is the discard operation, LN is the layer normalization operation, and FeedForward is the forward fully connected layer. The above are only specific examples. In actual applications, they can be flexibly set according to user needs and are not limited here.

[0096] Step 404: Based on the multi-source heterogeneous information fusion module, feature fusion processing is performed on the spatiotemporal feature information and the motion feature information to obtain fused feature information.

[0097] The multi-source heterogeneous information fusion module adopts MSH IF multi-source heterogeneous information fusion device. Based on the multi-source heterogeneous information fusion module, the server 104 performs feature fusion processing on the spatiotemporal feature information and the motion feature information to obtain fused feature information.

[0098] In a specific example, the abstract expression of the MSH IF multi-source heterogeneous information fusion device is:

[0099] O F =MSHIF(Output GCN ,Output TF )

[0100] =MultiHeadAttention(Output GCN ,Output TF ,Output TF )

[0101] Among them, Output GCN Output is the spatiotemporal feature information output by the ST-GCN spatiotemporal graph convolutional network. TF The motion feature information output by the CNN-Transformer network. The above is only a specific example, which can be flexibly set according to user needs in actual applications and is not limited here.

[0102] Step 405: Based on the action recognition classification module, action recognition classification is performed on the fused feature information to obtain an action classification result.

[0103] Among them, the action recognition classification module adopts a classifier. Based on the action recognition classification module, the server 104 performs action recognition classification on the fused feature information to obtain the recognition action category result, thereby achieving the goal of building a dual-branch model of the ST-GCN spatiotemporal graph convolutional network and the CNN-Transformer network, thereby overcoming the disadvantage of insufficient encoding of human motion learning information of the human body to be tested through a single path of the ST-GCN spatiotemporal graph convolutional network, using the attention mechanism to fuse multi-source heterogeneous feature information, making full use of the advantages of multimodal information, improving the accuracy and convenience of human motion recognition, and also improving the efficiency of action recognition.

[0104] In this embodiment, a spatiotemporal graph is constructed according to the human skeleton action sequence in the multimodal information data set to obtain a skeletal spatiotemporal graph; then, based on the spatiotemporal feature extraction module, the spatiotemporal features of the skeletal spatiotemporal graph are extracted to obtain spatiotemporal feature information; wherein the spatiotemporal feature extraction module adopts the ST-GCN spatiotemporal graph convolutional network; then, based on the motion feature extraction module, the motion features of the acceleration information in the multimodal information data set are extracted to obtain motion feature information; wherein the motion feature extraction module adopts the CNN-Transformer network; and, based on the multi-source heterogeneous information fusion module, the spatiotemporal feature information and the motion feature information are subjected to feature fusion processing to obtain fused feature information; wherein the multi-source heterogeneous information fusion module adopts the MSH IF multi-source heterogeneous information fusion device; finally, based on the action recognition classification module, the fused feature information is classified for action recognition to obtain the recognized action category result; among them, the action recognition classification module adopts a classifier to achieve the dual-branch model of ST-GCN spatiotemporal graph convolutional network and CNN-Transformer network, thereby overcoming the shortcomings of insufficient encoding of human motion learning information of the human body to be tested through a single path of ST-GCN spatiotemporal graph convolutional network, and using the attention mechanism to fuse multi-source heterogeneous feature information, making full use of the advantages of multimodal information, improving the accuracy and convenience of human motion recognition, and also improving the efficiency of action recognition.

[0105] In a specific example, the pre-trained human action recognition model consists of a 6-layer ST-GCN spatiotemporal graph convolutional network. The input skeletal spatiotemporal graph is first normalized through a BN layer. The ST-GCN layer includes a GCN network and a TCN network. The output channels of the 6-layer GCN network are 64, 64, 128, 128, 256, and 256 respectively. The drop rate is set to 0.6 after each layer of the ST-GCN spatiotemporal graph convolutional network to avoid overfitting. Then, the CNN-Transformer network is composed of a CNN and a Transformer model in series. First, the input acceleration information is positionally encoded, and then the spatial information of the acceleration information is modeled using a CNN network. A single-layer CNN is used, and its convolution kernel size is set to 4 and the step size is set to 2. Finally, a 3-layer Transformer model is used to learn the temporal features in the synthetic acceleration signal. In addition, the human action recognition model is an end-to-end training process. The human skeletal action sequence in the multimodal information data set is a tensor of N×C×T×V, where N is the number of batches, C represents the number of feature channels, T is the number of frames, and V is the number of joint nodes. The acceleration sequence shape of the CNN-Transformer network input is N×C×L, where N is the batch size, C is the number of feature channels, and L is the sequence length. In the human action recognition model, 100 training iterations are set, the learning rate is set to 0.01, and the learning rate is decayed at 45 epochs and 85 epochs. The cross entropy loss function and Adam optimizer are used for gradient optimization, and error back propagation and optimization of network parameters and weights at each layer are implemented. The above are only specific examples. In actual applications, they are flexibly set according to user needs and are not limited here.

[0106] Based on this, the above-mentioned human motion recognition method obtains the human skeleton motion sequence and corresponding acceleration information of the human body to be tested; then, a multimodal information data set is generated according to the human skeleton motion sequence and the corresponding acceleration information; then, the multimodal information data set is input into a pre-trained human motion recognition model, and the recognition motion category result of the human body to be tested is output; wherein, the human motion recognition model includes a spatiotemporal feature extraction module, a motion feature extraction module, a multi-source heterogeneous information fusion module and a motion recognition classification module; wherein, the spatiotemporal feature extraction module is used to extract the spatiotemporal features of the skeleton spatiotemporal graph of the human skeleton motion sequence to obtain the spatiotemporal feature information; the motion feature extraction module is used to extract the motion features of the acceleration information to obtain the motion feature information; the multi-source heterogeneous information fusion module is used to perform feature fusion processing on the spatiotemporal feature information and the motion feature information to obtain the fused feature information; the motion recognition classification module is used to perform motion recognition classification on the fused feature information to obtain the recognition action category result, which improves the accuracy and convenience of human motion recognition, and also improves the motion recognition efficiency.

[0107] In one embodiment, if Figure 8 As shown, the method includes steps 801 to 803.

[0108] Step 801, obtaining a preset number of historical human skeleton motion sequences, corresponding historical acceleration information, and corresponding historical motion type labels of the human bodies to be tested, and generating a corresponding historical multimodal information data sample set according to the historical human skeleton motion sequences, corresponding historical acceleration information, and corresponding historical motion type labels of each historical human body to be tested;

[0109] Step 802, performing random partitioning processing on each historical multimodal information data sample set to generate a training sample set and a test sample set;

[0110] Step 803, train the preset initial model of human motion recognition according to the training sample set, and test the initial model of human motion recognition according to the test sample set, adjust the model parameters of the initial model of human motion recognition based on the indicators obtained from the training and testing, until the indicators meet the preset requirements, generate a human motion recognition model, and output the recognition action category result based on the human motion recognition model.

[0111] Specifically, the server 104 obtains a preset number of historical human skeletal motion sequences, corresponding historical acceleration information, and corresponding historical motion type labels of the historical human bodies to be tested, and generates a corresponding historical multimodal information data sample set according to the historical human skeletal motion sequences, corresponding historical acceleration information, and corresponding historical motion type labels of each historical human body to be tested; then, each historical multimodal information data sample set is randomly divided to generate a training sample set and a test sample set; then, a preset human motion recognition initial model is trained according to the training sample set, and the human motion recognition initial model is tested according to the test sample set, and the model parameters of the human motion recognition initial model are adjusted based on the indicators obtained from the training and testing until the indicators meet the preset requirements, and a human motion recognition model is generated, so as to output the recognition action category result based on the human motion recognition model, thereby improving the efficiency and convenience of generating a human motion recognition model.

[0112] In this embodiment, a preset number of historical human skeletal motion sequences, corresponding historical acceleration information and corresponding historical motion type labels of the human bodies to be tested are obtained, and a corresponding historical multimodal information data sample set is generated according to the historical human skeletal motion sequences, corresponding historical acceleration information and corresponding historical motion type labels of each historical human body to be tested; then, each historical multimodal information data sample set is randomly divided to generate a training sample set and a test sample set; then, a preset human motion recognition initial model is trained according to the training sample set, and the human motion recognition initial model is tested according to the test sample set, and the model parameters of the human motion recognition initial model are adjusted based on the indicators obtained from the training and testing until the indicators meet the preset requirements, and a human motion recognition model is generated, so as to output the recognition action category result based on the human motion recognition model, thereby improving the efficiency and convenience of generating a human motion recognition model.

[0113] It should be understood that although Figure 2-Figure 4 as well as Figure 8 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Figure 2-Figure 4 as well as Figure 8 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0114] Second, as Fig. 9 As shown, a human motion recognition device is provided, which includes an information acquisition module 910, a set generation module 920 and a motion recognition module 930.

[0115] Among them, the information acquisition module 910 is used to obtain the human skeleton motion sequence and the corresponding acceleration information of the human body to be tested; the set generation module 920 is used to generate a multimodal information data set according to the human skeleton motion sequence and the corresponding acceleration information; the action recognition module 930 is used to input the multimodal information data set into a pre-trained human motion recognition model, and output the recognition action category result of the human body to be tested; wherein the human motion recognition model includes a spatiotemporal feature extraction module, a motion feature extraction module, a multi-source heterogeneous information fusion module and a motion recognition classification module; wherein the spatiotemporal feature extraction module is used to extract the spatiotemporal features of the skeleton spatiotemporal graph of the human skeleton motion sequence to obtain the spatiotemporal feature information; the motion feature extraction module is used to extract the motion features of the acceleration information to obtain the motion feature information; the multi-source heterogeneous information fusion module is used to perform feature fusion processing on the spatiotemporal feature information and the motion feature information to obtain the fused feature information; the action recognition classification module is used to perform action recognition classification on the fused feature information to obtain the recognition action category result.

[0116] In one embodiment, the motion recognition module 930 includes a motion recognition unit.

[0117] Among them, the action recognition unit is used to construct a spatiotemporal graph according to the human skeleton action sequence in the multimodal information data set to obtain a skeleton spatiotemporal graph; the action recognition unit is used to extract spatiotemporal features of the skeleton spatiotemporal graph based on the spatiotemporal feature extraction module to obtain spatiotemporal feature information; wherein the spatiotemporal feature extraction module adopts the ST-GCN spatiotemporal graph convolutional network; the action recognition unit is used to extract motion features of acceleration information in the multimodal information data set based on the motion feature extraction module to obtain motion feature information; wherein the motion feature extraction module adopts the CNN-Transformer network; the action recognition unit is used to perform feature fusion processing on the spatiotemporal feature information and the motion feature information based on the multi-source heterogeneous information fusion module to obtain fused feature information; wherein the multi-source heterogeneous information fusion module adopts the MSH IF multi-source heterogeneous information fuser; the action recognition unit is used to perform action recognition classification on the fused feature information based on the action recognition classification module to obtain the recognition action category result; wherein the action recognition classification module adopts a classifier.

[0118] In one embodiment, the information acquisition module 910 includes an information acquisition unit.

[0119] The information acquisition unit is used to acquire a target video signal and a corresponding target inertial signal of a human body to be tested; the target video signal includes a video signal of multiple movements of the human body to be tested continuously acquired by an ordinary camera; the target inertial signal includes acceleration signals of multiple movements of the human body to be tested acquired by each inertial sensor; the acceleration signal includes an X-axis acceleration signal, a Y-axis acceleration signal and a Z-axis acceleration signal; the inertial sensor is installed at a corresponding part to be tested of the human body to be tested; the information acquisition unit is used to perform time alignment processing on the target video signal and the corresponding target inertial signal to obtain a target video signal after time alignment processing and a corresponding target inertial signal after time alignment processing; the information acquisition unit is used to perform signal processing on the target video signal after time alignment processing and the corresponding target inertial signal after time alignment processing to obtain a human skeleton movement sequence and corresponding acceleration information.

[0120] In one of the embodiments, signal processing is performed on the target video signal after time alignment processing and the corresponding target inertial signal after time alignment processing to obtain a human skeletal motion sequence and corresponding acceleration information, including: inputting the target video signal after time alignment processing into a human posture estimation algorithm to obtain a human skeletal motion sequence; wherein the human posture estimation algorithm includes an Openpose algorithm and an Alphapose algorithm.

[0121] In one of the embodiments, signal processing is performed on the target video signal after time alignment processing and the corresponding target inertial signal after time alignment processing to obtain a human skeleton motion sequence and corresponding acceleration information, including: performing a two-norm transformation processing on the X-axis acceleration signal, the Y-axis acceleration signal and the Z-axis acceleration signal in each acceleration signal of the target inertial signal after time alignment processing to obtain acceleration information.

[0122] In one embodiment, the parts to be tested include the upper arms, waist and thighs; and the recognized action category results include sitting, walking, running, jumping and standing.

[0123] In one embodiment, the device includes a model training module.

[0124] Among them, the model training module is used to obtain a preset number of historical human skeletal motion sequences, corresponding historical acceleration information and corresponding historical motion type labels of the historical human bodies to be tested, and generate a corresponding historical multimodal information data sample set according to the historical human skeletal motion sequences, corresponding historical acceleration information and corresponding historical motion type labels of each historical human body to be tested; the model training module is used to randomly divide each historical multimodal information data sample set to generate a training sample set and a test sample set; the model training module is used to train a preset human motion recognition initial model according to the training sample set, and test the human motion recognition initial model according to the test sample set, adjust the model parameters of the human motion recognition initial model based on the indicators obtained from training and testing, until the indicators meet the preset requirements, generate a human motion recognition model, and output the recognition action category results based on the human motion recognition model.

[0125] For the specific definition of the human motion recognition device, please refer to the definition of the human motion recognition method above, which will not be repeated here. Each module in the above-mentioned human motion recognition device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0126] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.10 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the human skeletal motion sequence and corresponding acceleration information of the human body to be tested. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a human motion recognition method is implemented.

[0127] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0128] In a third aspect, a computer device is provided. The computer device includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the steps of any method in the above method embodiments are implemented.

[0129] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of any method in the above method embodiments are implemented.

[0130] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0131] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0132] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A human motion recognition method, the method comprising: Obtaining a human skeleton motion sequence and corresponding acceleration information of a human body to be tested; Generate a multimodal information data set according to the human skeleton motion sequence and the corresponding acceleration information; The multimodal information data set is input into a pre-trained human motion recognition model, and the recognition action category result of the human body to be tested is output; wherein the human motion recognition model includes a spatiotemporal feature extraction module, a motion feature extraction module, a multi-source heterogeneous information fusion module and a motion recognition classification module; wherein the spatiotemporal feature extraction module is used to extract spatiotemporal features from the skeletal spatiotemporal graph of the human skeletal motion sequence to obtain spatiotemporal feature information; the motion feature extraction module is used to extract motion features from the acceleration information to obtain motion feature information; the multi-source heterogeneous information fusion module is used to perform feature fusion processing on the spatiotemporal feature information and the motion feature information to obtain fused feature information; the motion recognition classification module is used to perform action recognition classification on the fused feature information to obtain the recognition action category result.

2. The method according to claim 1, characterized in that The step of inputting the multimodal information data set into a pre-trained human action recognition model and outputting the recognition action category result of the human body to be tested comprises: Constructing a spatiotemporal graph according to the human skeleton motion sequence in the multimodal information data set to obtain the skeleton spatiotemporal graph; Based on the spatiotemporal feature extraction module, the spatiotemporal features of the skeleton spatiotemporal graph are extracted to obtain the spatiotemporal feature information; wherein the spatiotemporal feature extraction module adopts the ST-GCN spatiotemporal graph convolutional network; Based on the motion feature extraction module, the motion features of the acceleration information in the multimodal information data set are extracted to obtain motion feature information; wherein the motion feature extraction module adopts a CNN-Transformer network; Based on the multi-source heterogeneous information fusion module, the spatiotemporal feature information and the motion feature information are subjected to feature fusion processing to obtain fused feature information; wherein the multi-source heterogeneous information fusion module adopts the MSHIF multi-source heterogeneous information fuser; Based on the action recognition classification module, the fused feature information is subjected to action recognition classification to obtain the recognized action category result; wherein the action recognition classification module adopts a classifier.

3. The method according to claim 1, characterized in that The step of obtaining a human skeleton motion sequence and corresponding acceleration information of a human body to be tested includes: Acquire a target video signal and a corresponding target inertial signal of the human body to be measured; the target video signal includes a video signal of multiple actions of the human body to be measured continuously collected by an ordinary camera; the target inertial signal includes acceleration signals of multiple actions of the human body to be measured collected by each inertial sensor; the acceleration signal includes an X-axis acceleration signal, a Y-axis acceleration signal and a Z-axis acceleration signal; the inertial sensor is installed at the corresponding part to be measured of the human body to be measured; Performing time alignment processing on the target video signal and the corresponding target inertial signal to obtain the target video signal after the time alignment processing and the corresponding target inertial signal after the time alignment processing; Signal processing is performed on the target video signal after the time alignment processing and the corresponding target inertial signal after the time alignment processing to obtain the human skeleton motion sequence and corresponding acceleration information.

4. The method according to claim 3, characterized in that: The step of performing signal processing on the target video signal after the time alignment processing and the corresponding target inertial signal after the time alignment processing to obtain the human skeleton motion sequence and corresponding acceleration information includes: The target video signal after the time alignment process is input into a human body posture estimation algorithm to obtain the human body skeleton motion sequence; wherein the human body posture estimation algorithm includes an Openpose algorithm and an Alphapose algorithm.

5. The method according to claim 3, characterized in that: The step of performing signal processing on the target video signal after the time alignment processing and the corresponding target inertial signal after the time alignment processing to obtain the human skeleton motion sequence and corresponding acceleration information includes: The X-axis acceleration signal, the Y-axis acceleration signal, and the Z-axis acceleration signal of each of the acceleration signals of the target inertial signal after the time alignment process are subjected to a two-norm transformation process to obtain the acceleration information.

6. The method according to claim 3, characterized in that The parts to be tested include upper arms, waist and thighs; the recognition action category results include sitting, walking, running, jumping and standing.

7. The method according to claim 1, characterized in that The method comprises: Obtain a preset number of historical human skeleton motion sequences, corresponding historical acceleration information, and corresponding historical motion type labels of historical human bodies to be tested, and generate a corresponding historical multimodal information data sample set according to the historical human skeleton motion sequences, corresponding historical acceleration information, and corresponding historical motion type labels of each of the historical human bodies to be tested; Performing random division processing on each of the historical multimodal information data sample sets to generate a training sample set and a test sample set; The preset human motion recognition initial model is trained according to the training sample set, and the human motion recognition initial model is tested according to the test sample set. The model parameters of the human motion recognition initial model are adjusted based on the indicators obtained from the training and testing until the indicators meet the preset requirements, and the human motion recognition model is generated to output the recognized action category result based on the human motion recognition model.

8. A human motion recognition device, characterized in that: The device comprises: An information acquisition module is used to obtain the human skeleton motion sequence and corresponding acceleration information of the human body to be tested; A set generation module, used for generating a multimodal information data set according to the human skeleton motion sequence and the corresponding acceleration information; An action recognition module is used to input the multimodal information data set into a pre-trained human action recognition model, and output the recognition action category result of the human body to be tested; wherein the human action recognition model includes a spatiotemporal feature extraction module, a motion feature extraction module, a multi-source heterogeneous information fusion module and an action recognition classification module; wherein the spatiotemporal feature extraction module is used to extract the spatiotemporal features of the skeletal spatiotemporal graph of the human skeletal action sequence to obtain spatiotemporal feature information; the motion feature extraction module is used to extract the motion features of the acceleration information to obtain the motion feature information; the multi-source heterogeneous information fusion module is used to perform feature fusion processing on the spatiotemporal feature information and the motion feature information to obtain fused feature information; the action recognition classification module is used to perform action recognition classification on the fused feature information to obtain the recognition action category result.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images