A motion detection method based on multi-motion features and two-stream network

Abstract

The present invention discloses a motion detection method based on multi-motion features and a dual-stream network, comprising the following steps: obtaining video data of a specified motion of a tested object; preprocessing the obtained video data; extracting feature sequences of the upper and lower limbs of the tester in different motions; constructing an improved dual-stream network model; and using a trained model to evaluate the obtained feature sequences and joint point data, so as to achieve the purpose of detecting the motion feature levels under different tasks. As an objective detection method, the present invention does not rely on professional sensors, has a low deployment cost, and can fuse the relevant features of different motion tasks, and finally output the stability level of the tester's motion features.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing and relates to a motion detection method based on multi-action features and a dual-stream network. Background Art

[0002] The detection of human motion features has always been the focus and difficulty of sports. Traditional machine vision methods are often used to evaluate the overall posture of the human body. Generally, the motion relationship in the video can be extracted by background segmentation, frame difference and optical flow, etc., which can achieve high accuracy. However, such methods are highly complex in design and are greatly affected by the environment. The algorithm needs to be redesigned for different motion features. Therefore, feature detection methods using convolutional neural networks have emerged, but a single convolutional neural network is difficult to handle the spatiotemporal relationship in the video. Due to the translation invariance and local perception of the convolution kernel, the network has a weak perception of global information. Moreover, the detection of motion features often only uses a single task, which is easy to miss the deeper information of the moving target. In addition, deep information can be obtained through professional sensors and deep learning methods, but professional sensors often rely on the help of professionals and professional equipment to a certain extent, and the cost and accuracy of the sensor are difficult to achieve. Therefore, it is difficult to balance portability and robustness in the detection of the continuity and stability features of motion. Summary of the invention

[0003] To solve the above problems, the present invention provides an objective detection method that takes into account the characteristic states of upper and lower limb movements. It uses an improved dual-stream network to analyze the spatiotemporal flow characteristics of the video, fuses information from different receptive fields, and introduces an attention mechanism to better model the dependency between features.

[0004] In order to solve the problems existing in the above existing problems, the present invention proposes a motion feature detection method using multi-motion features and a dual-stream network. By extracting multimodal feature sequences of upper and lower limbs in finger tapping and foot sensitivity tasks, and inputting them into an improved dual-stream network for evaluation, the effect of detecting motion stability features and vibration features is finally achieved.

[0005] To achieve the above object, the technical solution of the present invention is a method for identifying motion stability features based on multi-modal sequences under multi-action features and an improved dual-stream network, comprising the following steps:

[0006] The following steps are involved:

[0007] S1, obtaining video data of the object under test under specified action;

[0008] S2, preprocessing the acquired video data;

[0009] S3, extracting the characteristic sequence and joint point data of the tester's upper and lower limb movements;

[0010] S4, construct an improved two-stream network model;

[0011] S5, use the trained model to evaluate the obtained feature sequence and joint point data to obtain different motion stability grades.

[0012] Preferably, the video data for the specified action in S1 includes: a finger tapping task and a feet sensitivity task, video data captured by a camera at 30 frames per second, with the action repeated more than 10 times.

[0013] Preferably, the acquired video is preprocessed in S2, including: classifying different action videos; and using the TOFlow interpolation algorithm to increase the number of video frames.

[0014] Preferably, the S3 extracts the feature sequence and joint point data of the upper and lower limbs of the tester, including: using the Mediapipe posture estimation model to extract the joint key points of the hands and feet from the video processed by TOFlow interpolation, and calculating the amplitude, speed and acceleration sequence of the motion of the object under test through the key points.

[0015] Preferably, the extraction of the key points of the joints of the hands and feet includes:

[0016] The key points obtained are expressed as:

[0017]

[0018]

[0019] Where J is the key point of the upper limb hand, and P is the key point of the human body. Extract the coordinates of 21 key points of the hand and 33 key points of the human body, and record the coordinate value of the i-th key point of the upper limb in the t-th frame as The coordinate value of the i-th key point of the lower limb in the t-th frame is recorded as but represents the coordinate sequence of the i-th hand key point in the t-th frame; Represents the coordinate sequence of the i-th human key point in the t-th frame. and Both indicate whether the current key point is before 0. If the key point is before 0, the corresponding value is greater than 0, otherwise the corresponding value is less than 0.

[0020] Preferably, after extracting the key points of the joints of the hands and feet, calculating the amplitude, velocity and acceleration sequence of the object under test through the key points includes:

[0021] Calculate the Euclidean distance between key points, and the coordinates of key points 4 and 8 in the tth frame are expressed as and The distance formula for the tth frame of the finger tapping task is:

[0022]

[0023] The two-foot sensitivity task selects the Euclidean distance between key points 31 and 32. The distance formula for the two-foot sensitivity task at the tth frame is:

[0024]

[0025] Derivation under discrete conditions is equivalent to differential operation, and the characteristic sequence formula of finger tapping is:

[0026]

[0027]

[0028]

[0029] The characteristic sequence formula of the sensitivity of both feet in lower limb tasks is:

[0030]

[0031]

[0032]

[0033] Among them, FT represents the finger tapping task of the upper limbs, and LA represents the bilateral foot sensitivity task of the lower limbs; is the Euclidean distance between two points in the upper limb task, Amp FT is the upper limb amplitude sequence, is the Euclidean distance between two points of the lower limb task, Amp LA is the amplitude sequence of the lower limbs; is the upper limb velocity at frame t, Vel FT is the upper limb velocity sequence, is the lower limb velocity at frame t, Vel LA is the lower limb velocity sequence; is the upper limb acceleration at frame t, Acc FT is the upper limb acceleration sequence, is the acceleration of the lower limbs at the tth frame, Acc LA is the lower limb acceleration sequence.

[0034] Preferably, constructing the improved dual-stream network model in S4 includes:

[0035] The dual-stream network is divided into two parts. The upper part inputs the three feature sequences obtained, and the lower part inputs the obtained joint point coordinates. Among them, the upper network is constructed through convolutional layers and full-play average pooling layers, and the SKNET network is added to further extract motion features and expand the receptive field. The lower network uses a multi-head attention mechanism to construct a BT module, which is composed of multiple identical modules connected in series. Finally, the outputs of the two networks are weighted and fused to obtain the evaluation results.

[0036] Preferably, the upper half network includes:

[0037] It is composed of three convolutional layers and a global average pooling layer. The convolutional layer is connected to a batch normalization Norm layer and an activation function ReLU, and then connected to the SKNET network. The fully connected layer is replaced by a global average pooling layer, and the global average pooling layer is connected to Softmax to finally output the evaluation result.

[0038] Preferably, the lower half network comprises:

[0039] The lower half of the network is composed of four identical modules. Each module consists of two BT modules with the same model and different inputs, denoted as BT1 and BT2. The inputs are the coordinates of different joints in the same frame and the values ​​of the same joints in different frames. The weight matrices W q ,W k ,W v , calculate the query vector q t , key vector k t and weight vector v t , use the multi-head attention mechanism to calculate the attention score and determine the relevance of different joints. The corresponding formula is:

[0040]

[0041]

[0042] Among them, Output (l) is the output of the l-th layer of multi-head attention, V i Represents different joint points extracted, i,j are adjacent nodes, represents the query vector of the ith node in the lth layer of the current frame, represents the key vector and weight vector of the jth node under the lth layer of the current frame. The corresponding query vector and key vector are dot-producted to get the attention score and scaled. d is the dimension of the key vector, n is i is the output value of the i-th node, and C is the number of output nodes; after weighted summation through Softmax, it is repeated 8 times, and finally the results of multi-head attention are connected; after the output, batch normalization and a multi-layer perceptron are performed to obtain the final output of the module, the formula is as follows:

[0043] B l+1 =BT2(BT1(B l ))

[0044] BT1 is a spatial transformer module, BT2 is a temporal transformer module, B l ,B l+1 denotes the output of the lth and l+1th transformer layers.

[0045] Preferably, the loss function used by the model in S5 is:

[0046] The Focal Loss function is used to alleviate the problem of sample imbalance. Its formula is:

[0047] Loss = -α(1-P(lbl y |lbl x ) γ ln(P(lbl y |lbl x )))

[0048] Where α∈[0,1] is a balancing factor used to balance the number of samples; γ≥0 is a modulation coefficient used to adjust the sample loss so that the samples with a number less than the preset value have a larger weight value; P(lbl y |lbl x ) represents the probability that the label of data x is y.

[0049] The beneficial effects of the present invention are as follows:

[0050] By evaluating the amplitude, velocity and acceleration characteristics of the upper and lower limbs under different movements and the relative relationship between joints, the time flow, space flow and multiple motion characteristics are integrated, and the attention mechanism is used to strengthen the dependency between data, effectively increasing the detection accuracy.

[0051] It does not rely on professional sensors and equipment, effectively reduces the detection cost, and improves the portability and robustness of the detection solution. It can be used in fields such as medical auxiliary prevention and real-time video monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a flowchart of the steps of the motion feature detection method based on multi-motion features and dual-stream network of the present invention;

[0053] Figure 2 It is a flowchart of the steps of the motion feature detection method based on multi-motion features and dual-stream network according to a specific embodiment of the present invention;

[0054] Figure 3The SKNET module structure diagram used in the motion feature detection method based on multi-action features and dual-stream network of the present invention;

[0055] Figure 4 The improved dual-stream network structure diagram of the motion feature detection method based on multi-action features and dual-stream network of the present invention;

[0056] Figure 5 This is a structural diagram of the BT module used in the motion feature detection method based on multi-action features and dual-stream network of the present invention. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0058] On the contrary, the present invention covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present invention as defined by the claims. Further, in order to make the public have a better understanding of the present invention, some specific details are described in detail in the detailed description of the present invention below. Those skilled in the art can fully understand the present invention without the description of these details.

[0059] See also Figure 1 , the flow chart of the motion feature detection method based on multi-action features and dual-stream network, including the following steps:

[0060] S1, obtaining video data of the object under test under specified action;

[0061] S2, preprocessing the acquired video data;

[0062] S3, extracting the characteristic sequence and joint point data of the tester's upper and lower limb movements;

[0063] S4, construct an improved two-stream network model;

[0064] S5, use the trained model to evaluate the obtained feature sequence and joint point data to obtain different motion stability grades.

[0065] See also Figure 2 , which is a flowchart of the steps of implementing the motion feature detection method based on multi-motion features and a dual-stream network in the present invention, comprising the following steps:

[0066] S10: Obtaining video data of the object under test in a specified action.

[0067] Professionals completed the finger tapping task and foot sensitivity task, ensuring that each action was repeated more than ten times, and simultaneously used ordinary cameras to capture video at 30 frames per second. Professionals performed data annotation.

[0068] S20: Preprocess the acquired video data.

[0069] Since the finger tapping task and the foot sensitivity task in the video have both temporal and spatial characteristics, and pure video data is greatly affected by the frame rate, the TOFlow interpolation algorithm based on optical flow is used to interpolate the original video and then input it into the next link.

[0070] S30: Use the Mediapipe framework to extract the key point data of the hand and foot joints.

[0071] The Mediapipe framework is a cross-platform multimedia machine learning model application framework developed by Google Research, including data packets, data, computing units, graphs, and subgraphs. It also provides multiple available interfaces to facilitate the processing of audio and video data and the completion of different tasks. The present invention adopts the hand detection solution BlazeHands and the human body detection solution BlazePose.

[0072] For the upper limb assessment task, the hand detection solution BlazeHands is used. As a mature hand key point detection solution, it can obtain the distance relationship of the hand key points and obtain the corresponding 2.5D coordinate information. The present invention extracts the coordinates of 21 key points of the hand through this solution. Indicates whether the current key point is before 0. If the key point is before 0, its value is greater than 0, otherwise its value is less than 0. i ={x i ,y i ,z i} represents the coordinates of the i-th hand joint. The hand feature vector is recorded as:

[0073]

[0074] Since the corresponding features in the finger tapping task are mainly manifested in the relative spatial changes of the index finger and the thumb, the Euclidean distance between the two points is used to calculate the stability feature. The coordinates of key points 4 and 8 in the tth frame are and The corresponding distances are as follows:

[0075]

[0076] is the distance between key point 4 and key point 8. Since the derivative of discrete sequence is equivalent to difference, and the frame rate of the video is fixed, the speed is calculated by the difference of amplitude, and the acceleration is calculated by the difference of velocity. The final sequence is:

[0077]

[0078]

[0079]

[0080] For lower limb tasks, the human body detection solution BlazePose is used. As a lightweight human body detection architecture, 33 human body key points can be returned during the inference process, and the performance requirements of the device are relatively low. The present invention uses BlazePose to collect human body key points. Indicates whether the current key point is before 0. If the key point is before 0, its value is greater than 0, otherwise its value is less than 0. Note:

[0081]

[0082] Similarly, the two-foot sensitivity task uses key points 31 and 32 to calculate the input sequence, and the resulting Euclidean distance and correlation sequence are:

[0083]

[0084]

[0085]

[0086]

[0087] S40: Construct an improved two-stream network model.

[0088] The present invention collects three feature sequences of movement amplitude, speed and acceleration of upper and lower limbs respectively to better mine the main features of movement. At the same time, a dual-stream network is designed to fuse the feature sequence and joint position information, and SKNET module and BT module are added to introduce attention mechanism to enhance the connection between adjacent features, better extract deep-level features under different receptive fields, and more effectively capture the feature relationship under spatial and temporal dimensions.

[0089] The improved two-stream network is Figure 4As shown in the figure. The network consists of two parts. The first half inputs the three feature sequences of amplitude, speed and acceleration into the network respectively. The network as a whole consists of a convolution layer and a global average pooling layer (GAP). The convolution layer is followed by a batch normalization Norm layer and an activation function ReLU. Then the SKNET network is added to enhance the adaptability and feature extraction capabilities of the network, increase the depth appropriately, and finally output with a Softmax layer. The second half of the network consists of a hybrid transformer module BT (Blended Transformer). BT1 and BT2 have the same structure and input different sequences. Finally, after mixed weighting, the detection result of the motion feature level is output.

[0090] SKNET modules such as Figure 3 As shown in the Split operation, the original feature map is generated by 3×3 group convolution and 5×5 convolution (3*3Dilation=2) respectively. and Two feature maps. U through F gp The function (global average pooling) generates a 1×1×C feature map (s in the figure), and then passes F fc The function (composed of fully connected layers) generates a d×1 vector (z in the figure). The generated z is processed by the function of a and b added together, and multiplied by the original U1 and U2. This allows the weights to be set for the feature maps in the branches. Since the convolution kernels of different branches have different sizes, the network can autonomously select the appropriate convolution kernel. It fuses information from different receptive fields, suppresses secondary information, and enhances primary information.

[0091] The structure of BT module is as follows Figure 5 As shown, two identical BT modules BT1 and BT2 are used to input the coordinates of different joint points in the same frame and the values ​​of the same joint points in different frames respectively. The input sequence is transformed linearly and three weight matrices W q ,W k ,W v , calculate the query vector q t , key vector k t and weight vector v t , the weight matrix is ​​shared by all joints, and the multi-head attention mechanism is used to calculate the attention score and determine the different degrees of relevance. The corresponding formula is:

[0092]

[0093]

[0094] Among them, Output(l) is the output of the l-th layer of multi-head attention, V i Represents different joint points extracted, i,j are adjacent nodes, represents the query vector of the ith node in the lth layer of the current frame, represents the key vector and weight vector of the jth node under the lth layer of the current frame. d is the dimension of the key vector. The corresponding query vector and the key vector are dot-producted to get the attention score, and then scaled according to d. The Softmax formula is as above, n i is the output value of the i-th node, and C is the number of output nodes. After weighted summation through Softmax, it is repeated 8 times, and finally the results of multi-head attention are connected. After the output, batch normalization and a multi-layer perceptron (MLP) are performed to obtain the final output of the module. The output formula is as follows:

[0095] B l+1 =BT2(BT1(B l ))

[0096] BT1 is the spatial transformer module mentioned above, which inputs the spatial relationship of different key points at the same time. BT2 is the temporal transformer module mentioned above, which inputs the numerical relationship of different times at the same coordinate. l ,B l+1 denotes the output of the lth and l+1th transformer layers.

[0097] S50: Loss function improves sample imbalance problem.

[0098] Due to the imbalance of samples of different levels in sample collection, the discrimination weights in the network model training process will be biased towards the category with more data, resulting in an imbalance in classification accuracy. Therefore, the present invention adopts the Focal Loss loss function to increase the weight of the minority class that is more difficult to predict and reduce the weight of the majority class that is simpler by re-prediction weighting, so as to alleviate the problem of sample imbalance. The loss function formula is as follows:

[0099] Loss = -α(1-P(lbl y |lbl x ) γ ln(P(lbl y |lbl x )))

[0100] Where α∈[0,1] is a balancing factor used to balance the number of samples; γ≥0 is a modulation coefficient used to adjust the sample loss so that the samples with a number less than the preset value have a larger weight value; P(lbl y |lbl x ) represents the probability that the label of data x is y.

[0101] S60: Perform training based on the constructed model. The specific operations are as follows:

[0102] (1) Input the two-way feature sequences into the two-stream network;

[0103] (2) Calculate the loss function;

[0104] (3) Update the network weights and complete the training after the network converges.

[0105] Step S70: Use the trained model to evaluate the input feature sequence, and ultimately achieve the purpose of distinguishing different stability levels.

[0106] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A motion detection method based on multi-motion features and dual-stream network, characterized in that: The following steps are involved: S1, obtaining video data of the object under test under specified action; S2, preprocessing the acquired video data; S3, extracting the characteristic sequence and joint point data of the tester's upper and lower limb movements; S4, construct an improved two-stream network model; S5, using the trained model to evaluate the obtained feature sequence and joint point data to obtain different motion stability grades; The step S3 extracts the feature sequence and joint point data of the upper and lower limbs of the tester, including: extracting the joint key points of the hands and feet from the video processed by TOFlow interpolation using the Mediapipe posture estimation model, and calculating the amplitude, speed and acceleration sequence of the motion of the test object through the key points; The improved two-stream network model constructed in S4 includes: The dual-stream network is divided into two parts. The upper part inputs the three feature sequences obtained, and the lower part inputs the obtained joint point coordinates. The upper part of the network is constructed through convolutional layers and full-play average pooling layers, and the SKNET network is added to extract motion features and expand the receptive field. The lower part of the network uses a multi-head attention mechanism to construct a BT module, which is composed of multiple identical modules connected in series. Finally, the outputs of the two networks are weighted and fused to obtain the evaluation results. The upper network includes: It is composed of three convolutional layers and a global average pooling layer. A batch normalization Norm layer and an activation function ReLU are connected after the convolutional layer, and then connected to the SKNET network. The fully connected layer is replaced by a global average pooling layer. The global average pooling layer is connected to Softmax, and the evaluation result is finally output; The lower half of the network includes: The lower half of the network is composed of four identical modules. Each module consists of two BT modules with the same model and different inputs, denoted as BT1 and BT2. The inputs are the coordinates of different joints in the same frame and the values ​​of the same joints in different frames. The weight matrices W q ,W k ,W v , calculate the query vector q t , key vector k t and weight vector v t , use the multi-head attention mechanism to calculate the attention score and determine the relevance of different joints. The corresponding formula is: Among them, Output (l) is the output of the l-th layer of multi-head attention, V i Represents different joint points extracted, i,j are adjacent nodes, represents the query vector of the ith node in the lth layer of the current frame, represents the key vector and weight vector of the jth node under the lth layer of the current frame. The corresponding query vector and key vector are dot-producted to get the attention score and scaled. d is the dimension of the key vector, n is i is the output value of the i-th node, and C is the number of output nodes; after weighted summation through Softmax, it is repeated 8 times, and finally the results of multi-head attention are connected; after the output, batch normalization and a multi-layer perceptron are performed to obtain the final output of the module, the formula is as follows: B l+1 =BT2(BT1(B l )) BT1 is a spatial transformer module, BT2 is a temporal transformer module, B l ,B l+1 denotes the output of the lth and l+1th transformer layers.

2. The motion detection method based on multi-motion features and dual-stream network according to claim 1 is characterized in that: The video data of the specified action in S1 includes: a finger tapping task and a foot sensitivity task, and video data of the action being repeated more than 10 times at 30 frames per second shot by a camera.

3. The motion detection method based on multi-motion features and dual-stream network according to claim 1 is characterized in that: In the S2, the acquired video is preprocessed, including: classifying different action videos; using the TOFlow interpolation algorithm to increase the number of video frames.

4. The motion detection method based on multi-motion features and dual-stream network according to claim 1, characterized in that: The extraction of the key points of the joints of the hands and feet includes: The key points obtained are expressed as: Where J is the key point of the upper limb hand, P is the key point of the human body, extract the coordinates of 21 key points of the hand and 33 key points of the human body, and record the coordinate value of the i-th key point of the upper limb in the t-th frame as The coordinate value of the i-th key point of the lower limb in the t-th frame is recorded as but represents the coordinate sequence of the i-th hand key point in the t-th frame; Represents the coordinate sequence of the i-th human key point in the t-th frame; and Both indicate whether the current key point is before 0. If the key point is before 0, the corresponding value is greater than 0, otherwise the corresponding value is less than 0.

5. The motion detection method based on multi-motion features and dual-stream network according to claim 4 is characterized in that: After extracting the key points of the hand and foot joints, the amplitude, velocity and acceleration sequence of the object under test are calculated through the key points, including: Calculate the Euclidean distance between key points, and the coordinates of key points 4 and 8 in the tth frame are expressed as and The distance formula for the tth frame of the finger tapping task is: The two-foot sensitivity task selects the Euclidean distance between key points 31 and 32. The distance formula for the two-foot sensitivity task at the tth frame is: Derivation under discrete conditions is equivalent to differential operation, and the characteristic sequence formula of finger tapping is: The characteristic sequence formula of the sensitivity of both feet in lower limb tasks is: Among them, FT represents the finger tapping task of the upper limbs, and LA represents the bilateral foot sensitivity task of the lower limbs; is the Euclidean distance between two points in the upper limb task, Amp FT is the upper limb amplitude sequence, is the Euclidean distance between two points of the lower limb task, Amp LA is the amplitude sequence of the lower limbs; is the upper limb velocity at frame t, Vel FT is the upper limb velocity sequence, is the lower limb velocity at frame t, Vel LA is the lower limb velocity sequence; is the upper limb acceleration at frame t, Acc FT is the upper limb acceleration sequence, is the acceleration of the lower limbs at the tth frame, Acc LA is the lower limb acceleration sequence.

6. The motion detection method based on multi-motion features and dual-stream network according to claim 1, characterized in that: The loss function used by the model in S5 is: The Focal Loss function is used to alleviate the problem of sample imbalance. Its formula is: Loss=-α(1-P(lbl y |lbl x ) γ ln(P(lbl y |lbl x ))) Where α∈[0,1] is a balancing factor used to balance the number of samples; γ≥0 is a modulation coefficient used to adjust the sample loss so that the samples with a number less than the preset value have a larger weight value; P(lbl y |lbl x ) represents the probability that the label of data x is y.

Images