Human motion detection method based on high generalization

Abstract

The invention discloses a human motion detection method based on high generalization, and relates to the field of motion detection. Comprising the following steps: multi-modal data acquisition and preprocessing: deploying a sensor to perform multi-modal data acquisition, and preprocessing the multi-modal data; dynamic identity binding, wherein dynamic identity binding is achieved through gait cycle feature extraction and dynamic weight distribution according to the collected data; space-time feature extraction: constructing a double-flow space-time diagram convolutional network to perform feature extraction and fusion; and enhancing the confrontation visual angle, and constructing a visual angle invariant generative confrontation network. According to the method, through multi-modal data acquisition and analysis, other influence factors during data acquisition are increased, the subsequent detection precision is increased, the identity of a user can be bound by adopting a dynamic optimization method, the precision is further increased, in addition, a confrontation view angle enhancement form is adopted, the condition of single-view-angle shielding can be prevented, and the detection accuracy is improved. And combined training and real-time feedback correction are adopted, so that the precision is further improved.

Description

Technical Field

[0001] The present invention relates to the field of action detection, and in particular to a method for detecting highly generalized human actions. Background Art

[0002] The skeleton-based action detection algorithm is more in line with the similarity evaluation of fitness actions. First, it obtains the human skeleton through the human key point detection algorithm, and then conducts subsequent analysis on the extracted skeleton.

[0003] During exercise, different people perform the same action in different ways. Even the same person may perform the same action with slight differences at different times. At the same time, when exercising, it is also possible for other people to break in. Therefore, the accuracy of using single-dimensional data collection and analysis in the prior art is relatively low.

[0004] For this reason, the present invention proposes a method for detecting highly generalized human actions. Summary of the Invention

[0005] The purpose of the present invention is to solve the disadvantages existing in the prior art, and to propose a method for detecting highly generalized human actions.

[0006] In order to achieve the above purpose, the present invention adopts the following technical solutions:

[0007] A method for detecting highly generalized human actions includes the following steps:

[0008] S1: Multimodal data collection and preprocessing, deploying sensors to collect multimodal data, and preprocessing the multimodal data;

[0009] S2: Dynamic identity binding, realizing dynamic identity binding according to the collected data by gait cycle feature extraction and dynamic weight allocation;

[0010] S3: Spatiotemporal feature extraction, constructing a two-stream spatiotemporal graph convolutional network for feature extraction and fusion;

[0011] S4: Adversarial perspective enhancement, constructing a view-invariant generative adversarial network to generate multi-view training data and keeping the core action features unchanged;

[0012] S5: Multi-task joint training, designing a composite loss function to balance action recognition and feature preservation to prevent model overfitting;

[0013] S6: Real-time feedback correction, establishing a motion error quantization model for error comparison, and then correcting the model according to the error.

[0014] Preferably: In the step S1, the sensors include:

[0015] A camera array that captures RGB-D video streams, extracts 2D skeleton coordinates of key points using the AIphaPose framework, and generates a 3D skeleton sequence by fusing multi-view depth data using the ICP algorithm.

[0016] A heart rate sensor, which is a PPG sensor, is worn on the human body to collect human heart rate signals;

[0017] The myoelectric sensor is worn on the human body in the form of a MYO armband.

[0018] Preferably: in the step S2, in the gait cycle feature extraction:

[0019] The gait cycle feature extraction model is: GaitCode = LSTM (Δt rike ,θ knee ,υ stride ), where Δt rike is the time interval between two consecutive heel strikes, θ knee is the knee flexion angle, υ stride is the stride speed, LSTM is the long short-term memory network, its input dimension is 3, and the hidden layer dimension is 64;

[0020] The facial feature fusion model is: in is the facial ROI area image of the t-th frame, T is the number of sampling frames, ResNet50 is the pre-trained model with the last layer removed, and the output dimension is 512.

[0021] Preferably: in the step S2, the dynamic weight allocation model is: ID confidence =k1Sigmoid(W g GaitCode)+k2ReLU(W f FaceEmbed), where W g is the gait feature mapping matrix, W g ∈R 64×1 , W g is the face feature mapping matrix, W f ∈R 512×1 , Sigmoid is a mapping function that maps gait features to the interval [0, 1], and k1 and k2 are weights.

[0022] Preferably: in the step S2,

[0023] Preferably: in the step S3, the spatial flow model is in:

[0024] H (l) is the feature matrix of the l-th layer node, H (l) ∈RN×d , N = 17 joints, d = 256;

[0025] A ij Adjacency weight between joints i and j, with a value range of [0, 1];

[0026] W s (l) is a learnable spatial transformation matrix, W s (l) ∈R d×d ;

[0027] is a spatial bias term,

[0028] σ is the leakyReLU activation function.

[0029] Preferably: In the S3 step, the model of the dynamic flow is where:

[0030] δt is the temporal convolution kernel offset;

[0031] is a learnable temporal convolution kernel,

[0032] M attn is the temporal attention weight matrix, M attn ∈R 7×1 ;

[0033] is the Hadamard product.

[0034] Preferably: In the S5 step, the composite loss function is: ζ total = k′1ζ cls + k′2ζ recon + k′3ζ consist , where ζ cls is the classification loss, ζ recon is the reconstruction loss, ζ[[ID=б7]] consist is the consistency loss, and k′1, k′2, k′3 are weights.

[0035] Preferably: In the S5 step,

[0036] Preferably: In the S6 step, the motion error quantization model is: where P j user is the three-dimensional coordinate of the j-th joint of the user, P j ref is the three-dimensional coordinate of the corresponding joint in the standard action template, W j is the joint weight coefficient. It should be noted that there is a character "б" in line 67 which may be an incorrect character. If it is a misprint, please correct it for a more accurate translation.

[0037] The beneficial effects of the present invention are as follows:

[0038] 1. Through multi-modal data acquisition and analysis, the present invention adds other influencing factors during data acquisition, improves subsequent detection accuracy. At the same time, the method of dynamic optimization can bind the user's identity, further improving the accuracy. In addition, the form of adversarial perspective enhancement is adopted to prevent the situation of single-perspective occlusion, and joint training and real-time feedback correction are used to further improve the accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a flowchart of a method for detecting highly generalized human actions proposed by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0040] The technical solution of the present invention will be further described in detail below in conjunction with the specific embodiments.

[0041] In the description of the present invention, it should be noted that unless otherwise clearly specified and limited, the terms "installation", "connection", "connection", and "setting" should be understood in a broad sense. For example, it can be fixedly connected and set, or detachably connected and set, or integrally connected and set. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific situations.

[0042] Embodiment 1:

[0043] A method for detecting highly generalized human actions includes the following steps:

[0044] S1: Multi-modal data acquisition and preprocessing, deploying sensors to collect multi-modal data and preprocessing the multi-modal data;

[0045] S2: Dynamic identity binding, realizing dynamic identity binding according to the collected data by gait cycle feature extraction and dynamic weight allocation;

[0046] S3: Spatiotemporal feature extraction, constructing a two-stream spatiotemporal graph convolutional network for feature extraction and fusion;

[0047] S4: Adversarial perspective enhancement, constructing a view-invariant generative adversarial network to generate multi-view training data and keeping the core action features unchanged;

[0048] S5: Multi-task joint training, designing a composite loss function to balance action recognition and feature preservation to prevent model overfitting;

[0049] S6: Real-time feedback correction, establishing a motion error quantization model for error comparison, and then correcting the model according to the error.

[0050] In the step S1, the sensor includes:

[0051] A camera array that collects RGB-D video streams, extracts 2D skeleton coordinates of key points through the AIphaPose framework, and generates a 3D skeleton sequence by fusing multi-view depth data through the ICP algorithm;

[0052] A heart rate sensor, which is a PPG sensor worn on the human body to collect the human heart rate signal;

[0053] An electromyography sensor, which is worn on the human body in the form of a MYO armband.

[0054] In the step S2, in the gait cycle feature extraction:

[0055] The gait cycle feature extraction model is: GaitCode = LSTM(Δt rike , θ knee , υ stride ), where Δt rike is the time interval between two consecutive heel strikes, θ knee is the knee flexion angle, υ stride is the stride speed, LSTM is a long short-term memory network, and its input dimension is 3 and the hidden layer dimension is 64;

[0056] The face feature fusion model is: where is the facial ROI region image of the t-th frame, T is the number of sampled frames, ResNet50 is a pre-trained model without the last layer, and the output dimension is 512.

[0057] In the step S2, the dynamic weight allocation model is: ID confidence = k1Sigmoid(W g GaitCode)+k2ReLU(W f FaceEmbed), where W g is the gait feature mapping matrix, W g ∈R 64×1 , W g is the face feature mapping matrix, W f ∈R 512×1 , Sigmoid is a mapping function that maps the gait feature to the interval [0, 1], and k1 and k2 are weights.

[0058] In the step S3, the spatial flow model is where:

[0059] H (l) is the node feature matrix of the l-th layer, H (l) ∈R N×d, N = 17 joints, d = 256;

[0060] A ij The adjacency weight between joints i and j, with a value range of [0, 1];

[0061] W s (l) is a learnable spatial transformation matrix, W s (l) ∈R d×d ;

[0062] is the spatial bias term,

[0063] σ is the leakyReLU activation function.

[0064] In the S3 step, the model of the dynamic flow is where:

[0065] δt is the time series convolution kernel offset;

[0066] is a learnable time convolution kernel,

[0067] M attn is the time attention weight matrix, M attn ∈R 7×1 ;

[0068] is the Hadamard product.

[0069] In the S5 step, the composite loss function is: ζ total = k′1ζ cls [[ID=D57]]+ k′2ζ recon + k′3ζ consist , where ζ cls is the classification loss, ζ recon is the reconstruction loss, ζ consist is the consistency loss, and k′1, k′2, k′3 are weights.

[0070] In the S6 step, the motion error quantization model is: where P j user is the three-dimensional coordinate of the user's j-th joint, P j ref is the three-dimensional coordinate of the corresponding joint in the standard action template, W j is the joint weight coefficient.

[0071] Example 2:

[0072] A high-generalization human action detection method, which includes the following steps:

[0073] S1: Multimodal data acquisition and preprocessing, deploying sensors to acquire multimodal data, and preprocessing the multimodal data;

[0074] S2: Dynamic identity binding, achieving dynamic identity binding according to the collected data by using gait cycle feature extraction and dynamic weight assignment;

[0075] S3: Spatiotemporal feature extraction, constructing a two-stream spatiotemporal graph convolutional network for feature extraction and then fusion;

[0076] S4: Adversarial perspective enhancement, constructing a view-invariant generative adversarial network to generate multi-view training data while keeping the core action features unchanged;

[0077] S5: Multi-task joint training, designing a composite loss function to balance action recognition and feature preservation to prevent model overfitting;

[0078] S6: Real-time feedback correction, establishing a motion error quantization model for error comparison, and then correcting the model according to the error.

[0079] In the step S1, the sensors include:

[0080] A camera array, which acquires an RGB-D video stream, extracts the 2D skeleton coordinates of key points through the AIphaPose framework, and fuses multi-view depth data through the ICP algorithm to generate a 3D skeleton sequence;

[0081] A heart rate sensor, which is a PPG sensor and is worn on the human body to acquire the human heart rate signal;

[0082] An electromyography sensor, which is worn on the human body in the form of a MYO armband.

[0083] In the step S2, in the gait cycle feature extraction:

[0084] The gait cycle feature extraction model is: GaitCode = LSTM(Δt rike , θ knee , υ stride ), where Δt rike is the time interval between two consecutive heel strikes, θ knee is the knee flexion angle, υ stride is the stride speed, LSTM is a long short-term memory network, and its input dimension is 3 and the hidden layer dimension is 64;

[0085] The face feature fusion model is: where is the facial ROI region image of the t-th frame, T is the number of sampled frames, ResNet50 is a pre-trained model without the last layer, and the output dimension is 512.

[0086] In the S2 step, the dynamic weight allocation model is: ID confidence = k1Sigmoid(W g GaitCode)+k2ReLU(W f FaceEmbed), where W g is the gait feature mapping matrix, W g ∈R 64×1 and W g is the face feature mapping matrix, W f ∈R 512×1 , Sigmoid is a mapping function that maps the gait feature to the interval [0, 1], and k1, k2 are weights.

[0087] In the S2 step,

[0088] In the S3 step, the spatial flow model is where:

[0089] H (l) is the node feature matrix of the l-th layer, H (l) ∈R N×d , N = 17 joints, d = 256;

[0090] A ij is the adjacency weight between joints i and j, and the value range is [0, 1];

[0091] W s (l) is a learnable spatial transformation matrix, W s (l) ∈R d×d ;

[0092] is the spatial bias term,

[0093] σ is the leakyReLU activation function.

[0094] In the S3 step, the model of the dynamic flow is where:

[0095] δt is the offset of the temporal convolution kernel;

[0096] is a learnable temporal convolution kernel,

[0097] M attnis the time attention weight matrix, M attn ∈R 7×1 ;

[0098] is the Hadamard product.

[0099] In the S5 step, the composite loss function is: ζ total = k′1ζ cls + k′2ζ recon + k′3ζ consist , where ζ cls is the classification loss, ζ recon is the reconstruction loss, ζ consist is the consistency loss, and k′1, k′2, k′3 are weights.

[0100] In the S5 step,

[0101] In the S6 step, the motion error quantization model is: where P j user is the three-dimensional coordinate of the jth joint of the user, and P j ref is the three-dimensional coordinate of the corresponding joint in the standard action template, and W j is the joint weight coefficient.

[0102] In the present invention, through multi-modal data acquisition and analysis, other influencing factors during data acquisition are increased, the subsequent detection accuracy is improved. At the same time, the method of dynamic optimization can bind the user's identity, further increasing the accuracy. In addition, the form of adversarial perspective enhancement is adopted to prevent the situation of single-perspective occlusion, and the joint training and real-time feedback correction are adopted to further increase the accuracy.

[0103] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention, according to the technical solution and inventive concept of the present invention, makes equivalent substitutions or changes, and all should be covered within the protection scope of the present invention.

Claims

1. A high-generalization human action detection method, characterized in that, It includes the following steps: S1: Multimodal data acquisition and preprocessing. Deploy sensors to acquire multimodal data and preprocess the multimodal data; S2: Dynamic identity binding. Achieve dynamic identity binding according to the collected data by using gait cycle feature extraction and dynamic weight allocation; S3: Spatiotemporal feature extraction. Construct a two-stream spatiotemporal graph convolutional network for feature extraction and then fusion; S4: Adversarial perspective enhancement. Construct a view-invariant generative adversarial network to generate multi-view training data while keeping the core action features unchanged; S5: Multi-task joint training. Design a composite loss function to balance action recognition and feature preservation to prevent model overfitting; S6: Real-time feedback correction. Establish a motion error quantization model for error comparison and then correct the model according to the error.

2. The method for detecting highly generalized human actions according to claim 1, characterized in that In the step S1, the sensors include: A camera array that acquires an RGB-D video stream, extracts the 2D skeleton coordinates of key points through the AIphaPose framework, and generates a 3D skeleton sequence by fusing multi-view depth data through the ICP algorithm; A heart rate sensor, which is a PPG sensor, worn on the human body to collect the human heart rate signal; An electromyography sensor, which is worn on the human body in the form of a MYO armband.

3. A method for detecting human actions based on high generalization according to claim 1, characterized in that, In the step S2, in the gait cycle feature extraction: The gait cycle feature extraction model is: GaitCode = LSTM(Δt rike , θ knee , υ stride ), where Δt rike is the time interval between two consecutive heel strikes, θ knee is the knee flexion angle, υ stride is the stride speed, LSTM is the long short-term memory network, its input dimension is 3, and the hidden layer dimension is 64; The face feature fusion model is as follows: Wherein is the facial ROI region image of the t-th frame, T is the number of sampled frames, ResNet50 is a pre-trained model with the last layer removed, and the output dimension is 512.

4. The method for detecting highly generalized human actions according to claim 3, characterized in that, In the step S2, the dynamic weight allocation model is: ID confidence = k1Sigmoid(W g GaitCode)+k2ReLU(W f FaceEmbed), where W g is the gait feature mapping matrix, W g ∈R 64×1 , W g is the face feature mapping matrix, W f ∈R 512×1 , Sigmoid is a mapping function that maps the gait feature to the interval [0, 1], and k1 and k2 are weights.

5. A method for detecting highly generalized human actions according to claim 4, characterized in that, In the step S2, 6. The method for detecting a highly generalized human motion according to claim 1, wherein In the step S3, the spatial flow model is Where: H (l) is the node feature matrix of the l-th layer, H (l) ∈R N×d , N = 17 joints, d = 256; A ij The adjacency weight between joints i and j, with a value range of [0, 1]; W s (l) is a learnable spatial transformation matrix, is a spatial offset term, σ is the leakyReLU activation function.

7. A method for detecting highly generalized human actions according to claim 1, characterized in that, In the step S3, the model of the dynamic flow is Wherein: δt is the time series convolution kernel offset; is a learnable temporal convolution kernel, M attn is the temporal attention weight matrix, M attn ∈R 7×1 ; is the Hadamard product.

8. A method for detecting highly generalized human actions according to claim 1, characterized in that In the step S5, the composite loss function is: ζ total = k'1ζ cls + k'2ζ recon + k'3ζ consist , where ζ cls is the classification loss, ζ recon is the reconstruction loss, ζ consist is the consistency loss, and k'1, k'2, and k'3 are weights.

9. A method for detecting highly generalized human actions according to claim 8, characterized in that, In the step S5, 10. A method for detecting human actions based on high generalization according to claim 1, characterized in that, In the step S6, the motion error quantization model is as follows: where is the three-dimensional coordinate of the j-th joint of the user, is the three-dimensional coordinate of the corresponding joint in the standard action template, and W j is the joint weight coefficient.

Images