Human body motion posture tracking method and device

Abstract

The invention provides a human motion posture tracking method and related equipment, and the method comprises the steps: a plurality of thermal imaging infrared sensing devices sense a thermal imaging image sequence of an elbow joint, and a plurality of inertial measurement units sense an initial motion information vector sequence of a non-elbow joint; the control device converts the thermal image sequence into an infrared light vector sequence of an elbow joint and converts the initial motion information vector sequence into a motion information vector sequence of a non-elbow joint; based on a first neural network model, predicting a rotation angle vector of the elbow joint corresponding to the infrared light vector sequence of the elbow joint; splicing the rotation angle vector of the elbow joint and the motion information vector sequence of the non-elbow joint to obtain a multi-modal information vector sequence; based on a second neural network model combination, predicting a rotation angle vector of a non-elbow joint corresponding to the multi-modal information vector sequence; and determining the motion posture of the upper body of the human body according to the rotation angle vector of the elbow joint and the rotation angle vector of the non-elbow joint.

Description

Technical Field

[0001] The present application relates to the field of human motion capture and perception technology, and in particular to a method and device for tracking human motion posture. Background Art

[0002] With the rapid development of science and technology, human motion capture and perception technology has been widely used in many fields, including virtual reality, augmented reality, motion analysis and rehabilitation training, intelligent monitoring and human-computer interaction, etc. Human motion capture and perception technology needs to be applied to motion capture systems, which usually rely on expensive optical camera equipment and complex site layout, and have many limitations, such as high cost, high environmental requirements, and potential threats to user privacy.

[0003] In recent years, the application technology of smart sensor clothing has integrated micro sensors into daily wearable clothing to capture the movement information of the human body in real time. However, the application technology of most smart sensor clothing on the market currently has a lot of room for improvement in ensuring data accuracy, improving the robustness of smart sensor clothing, and optimizing user comfort.

[0004] In view of this, the present application urgently needs to find a method for tracking human motion posture that has low cost, low environmental requirements, low privacy threats, high posture determination accuracy, high device robustness, and high user comfort, as well as a human motion posture tracking device that uses this method. Summary of the invention

[0005] The present application provides a control method and device for a phase-shifted full-bridge converter to improve the accuracy of human body motion posture determination, device robustness and user comfort while reducing costs, environmental requirements and privacy threats.

[0006] A first aspect of the present application provides a human motion posture tracking method, which is applied to a human motion posture tracking device, wherein the human motion posture tracking device includes a short-sleeved garment and a control device; a plurality of thermal imaging infrared sensor devices and a plurality of inertial measurement units are arranged on the short-sleeved garment; the plurality of thermal imaging infrared sensor devices sense a thermal image sequence of an elbow joint, and the plurality of inertial measurement units sense an initial motion information vector sequence of a non-elbow joint; the control device converts the thermal image sequence into an infrared optical flow vector sequence of the elbow joint, and converts the initial motion information vector sequence into a motion information vector sequence of a non-elbow joint; based on a first neural network model, predicting a rotation angle vector of the elbow joint corresponding to the infrared optical flow vector sequence of the elbow joint; splicing the rotation angle vector of the elbow joint and the motion information vector sequence of the non-elbow joint to obtain a multimodal information vector sequence; based on a second neural network model combination, predicting a rotation angle vector of the non-elbow joint corresponding to the multimodal information vector sequence; and determining the motion posture of the upper body of the human body according to the rotation angle vector of the elbow joint and the rotation angle vector of the non-elbow joint.

[0007] In some embodiments of the first aspect, the short-sleeved garment includes two sleeves, and the plurality of thermal imaging infrared sensor devices are respectively disposed on the two sleeves.

[0008] In some embodiments of the first aspect, the short-sleeved garment further includes two short-sleeved shoulder lines and two armpit covering points, a first connecting line is formed between the two short-sleeved shoulder lines, a second connecting line is formed between the two armpit covering points, and a plurality of inertial measurement units are respectively arranged at the midpoint between the first connecting line and the second connecting line, and at the hem of the short-sleeved garment near the pelvis.

[0009] In some embodiments of the first aspect, the initial motion information vector sequence includes an angular velocity vector sequence and an acceleration vector sequence, the angular velocity vector sequence includes angular velocity vectors sorted by time, and the acceleration vector sequence includes acceleration vectors sorted by time. The control device also converts the angular velocity vector into a rotational acceleration vector based on the interval time between adjacent angular velocity vectors; adjusts the acceleration vector based on the proportion of the acceleration vector in the rotational acceleration vector to obtain a target acceleration vector; and obtains a motion information vector sequence of the non-elbow joint based on the spliced ​​target acceleration vector and angular velocity vector.

[0010] In some embodiments of the first aspect, the control device also obtains the weight of the acceleration vector based on the proportion of the acceleration vector in the rotational acceleration vector, preset linear layer weights and parameters, wherein the proportion of the acceleration vector in the rotational acceleration vector is the ratio of the quadratic norm of the vector of the acceleration vector to the norm of the vector of the rotational acceleration vector; and adjusts the acceleration vector according to the weight of the acceleration vector.

[0011] In some embodiments of the first aspect, the control device further converts the thermal image sequence into an infrared optical flow vector sequence according to an optical flow algorithm; and aligns the timestamps of the infrared optical flow vector sequence of the elbow joint and the motion information vector sequence of the non-elbow joint.

[0012] In some embodiments of the first aspect, the first neural network model is obtained by training a residual network model with an infrared optical flow vector sequence of an elbow joint as input data and an optical image sequence of the elbow joint as label data, and the optical image sequence of the elbow joint is used to indicate the true value of the rotation angle vector of the elbow joint; the second neural network model is obtained by combining a multimodal information vector sequence as input data and an optical image sequence of a non-elbow joint as label data to train a long short-term memory network model and a multi-layer perceptron model, and the optical image sequence of the non-elbow joint indicates the true value of the rotation angle vector of the non-elbow joint; When the sum of the first loss value of the residual network model and the second loss values ​​of the long short-term memory network model and the multi-layer perceptron model is within a preset range, the training of the residual network model, the long short-term memory network model and the multi-layer perceptron model is completed, wherein the first loss value is calculated based on the rotation angle vector of the elbow joint and the true value of the rotation angle vector of the elbow joint; the second loss value is calculated based on the rotation angle vector of the non-elbow joint and the true value of the rotation angle vector of the non-elbow joint.

[0013] According to a second aspect of the present application, there is provided a human body motion posture tracking device, which includes a short-sleeved garment and a control device; a plurality of thermal imaging infrared sensor devices and a plurality of inertial measurement units are arranged on the short-sleeved garment; the plurality of thermal imaging infrared sensor devices are used to sense the thermal image sequence of the elbow joint, and the plurality of inertial measurement units are used to sense the initial motion information vector sequence of the non-elbow joint; the control device is used to convert the thermal image sequence into the infrared optical flow vector sequence of the elbow joint, and convert the initial motion information vector sequence into the motion information vector sequence of the non-elbow joint; based on the first neural network model, the rotation angle vector of the elbow joint corresponding to the infrared optical flow vector sequence of the elbow joint is predicted; the rotation angle vector of the elbow joint and the motion information vector sequence of the non-elbow joint are spliced ​​to obtain a multimodal information vector sequence; based on the second neural network model combination, the rotation angle vector of the non-elbow joint corresponding to the multimodal information vector sequence is predicted; the motion posture of the upper body of the human body is determined according to the rotation angle vector of the elbow joint and the rotation angle vector of the non-elbow joint.

[0014] In some embodiments of the second aspect, the short-sleeved garment includes two sleeves, and the plurality of thermal imaging infrared sensor devices are respectively disposed on the two sleeves.

[0015] In some embodiments of the second aspect, the short-sleeved garment further includes two short-sleeved shoulder lines and two armpit covering points, a first connecting line is formed between the two short-sleeved shoulder lines, a second connecting line is formed between the two armpit covering points, and a plurality of inertial measurement units are respectively arranged at the midpoint between the first connecting line and the second connecting line, and at the hem of the short-sleeved garment near the pelvis.

[0016] The human motion posture tracking method and human motion posture tracking device of the present application are provided with a plurality of thermal imaging infrared sensing devices and a plurality of inertial measurement units on short-sleeved clothing to respectively sense the thermal image sequence of the elbow joint and the initial motion information vector sequence of the non-elbow joint. Then, the infrared light flow sequence of the elbow joint obtained according to the thermal image sequence and the motion information sequence of the non-elbow joint obtained according to the initial motion information vector sequence are integrated by the control device, and the human motion posture is determined based on the first neural network model and the second neural network model, thereby overcoming the limitations of the ambient light interference generated by using a single modal information and the data drift caused by long-term determination of the human motion posture, improving the stability and accuracy of the posture determination, and improving the robustness of the device. At the same time, the user can perform posture determination while wearing short-sleeved clothing, which improves the comfort. The capture of the joint movement of the upper body of the human body with high precision, low interference, low cost, low privacy threat (no optical camera equipment is used) and low environmental requirements (no need for complex site layout) is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0018] Figure 1 A schematic diagram of a structure of a human body motion posture tracking device provided in an embodiment of the present application; Figure 2 A schematic diagram of a process flow of a method for tracking human motion posture provided in an embodiment of the present application; Figure 3 An application scenario diagram of the human body motion posture tracking method provided in an embodiment of the present application; Figure 4 Another application scenario diagram of the human body motion posture tracking method provided in an embodiment of the present application; Figure 5 Another schematic diagram of the process of tracking the human body motion posture provided by the embodiment of the present application; Figure 6 A diagram showing another application scenario of the method for tracking human motion posture provided in an embodiment of the present application; Figure 7 and Figure 8 This is another application scenario diagram of the human body motion posture tracking method provided in an embodiment of the present application. Fig. 9 A schematic diagram of the structure of a control device provided in an embodiment of the present application.

[0019] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0020] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0021] The terms "first", "second", etc. involved in this application are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features.

[0022] The technical solution of the present application and how the technical solution of the present application solves the technical problem are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0023] The data vector involved in this application refers to data in the form of a vector. For example, an infrared light flow vector refers to an infrared light flow in the form of a vector.

[0024] Please combine Figures 1 to 3 , Figure 1 A schematic diagram of the structure of a human body motion posture tracking device according to an embodiment of the present application. Figure 2 A schematic diagram of a process of a human body motion posture tracking method according to an embodiment of the present application is shown in FIG. Figure 3 Schematic diagram of a scenario of a human body motion posture tracking method. The human body motion posture tracking method of an embodiment of the present application is applied to a human body motion posture tracking device.

[0025] like Figure 1As shown, the human motion posture tracking device 100 includes a control device 10 and a short-sleeved garment 20, on which a plurality of thermal imaging infrared sensor devices 30 and a plurality of inertial measurement units 40 are provided. The thermal imaging infrared sensor device 30 is used to sense the thermal image of the elbow joint and generate a thermal image sequence. The inertial measurement unit 40 is used to sense the initial motion information of the non-elbow joint and generate an initial motion information sequence. The control device 10 is provided with two kinds of modal information to determine the motion posture of the upper body of the human body. In some embodiments, the control device 10 can be a laptop computer, a mobile phone, a server, etc.

[0026] In some embodiments, the short-sleeved garment 20 includes two sleeves 21, and two thermal imaging infrared sensor devices 30 are respectively disposed on the two sleeves 21. The camera of the thermal imaging infrared sensor device 30 on one sleeve 21 faces the left elbow joint to sense the thermal image of the left elbow joint. The camera of the thermal imaging infrared sensor device 30 on the other sleeve 21 faces the right elbow joint to sense the thermal image of the right elbow joint.

[0027] It can be understood that the embodiment of the present application provides two thermal imaging infrared sensor devices 30 on the two sleeves 21 respectively, so that when the human arm performs various movements, the thermal imaging infrared sensor device 30 can capture the thermal image of the elbow joint, and the thermal image indicates the movement status of the elbow joint.

[0028] In some embodiments, the short-sleeved garment 20 further includes two short-sleeved shoulder lines 22 and two armpit covering points 23, a first connecting line 24 is formed between the two short-sleeved shoulder lines 22, and a second connecting line 25 is formed between the two armpit covering points 23. An inertial measurement unit 40 is disposed at a midpoint 26 between the first connecting line 24 and the second connecting line 25, and another inertial measurement unit 40 is disposed at the hem of the short-sleeved garment 20 near the pelvis.

[0029] It can be understood that the first connecting line 24 between the two short-sleeved shoulder lines 22, the midpoint 26 between the second connecting line 25 between the two armpit covering points 23, and the hem of the short-sleeved garment 20 have a stronger load-bearing capacity when they intersect at the sleeve 21, and do not conflict with the placement position of the thermal imaging infrared sensor device 30. Therefore, in the embodiment of the present application, by arranging the inertial measurement unit 40 at the midpoint 26 and the hem of the short-sleeved garment 20, when the human body performs various movements, the inertial measurement unit 40 can stably sense the movement information of the non-elbow joints (i.e., other upper body joints of the human body except the elbow joints) to reflect the movement status of the non-elbow joints of the human body.

[0030] It is understandable that compared with the related art, the user needs to wear tight clothing (sensors integrated on tight clothing) or wear tight sensing equipment to track posture. The present application sets the thermal imaging infrared sensor device 30 and the inertial measurement unit 40 on the short-sleeved clothing 20, and realizes the arrangement of multimodal sensors on the short-sleeved clothing for the first time and tracks the movement posture of the upper body of the human body, which can give the user the freedom to exercise in the form of short sleeves, improve the user's comfort, and do not set up optical camera equipment, reducing the threat of privacy leakage.

[0031] It should be noted that Figure 1 This is only a schematic diagram of a structure provided in the embodiment of the present application. Figure 1 The actual form of the various devices included in the Figure 1 The interaction mode between devices is limited, and in the specific application of the solution, it can be set according to actual needs.

[0032] like Figure 2 As shown, the human body motion posture tracking method may include the following steps: Step S110: A plurality of thermal imaging infrared sensor devices sense a thermal image sequence of the elbow joint.

[0033] The thermal image sequence of the elbow joints includes a thermal image sequence of the left elbow joint and a thermal image sequence of the right elbow joint. The thermal image sequence refers to thermal images sorted by time.

[0034] Step S120: multiple inertial measurement units sense initial motion information vector sequences of non-elbow joints.

[0035] The expression of the initial motion information vector sequence can be The initial motion information vector sequence refers to the motion information vectors sorted by time, through which the real-time motion state of the non-elbow joint can be understood.

[0036] Step S130: The control device converts the thermal image sequence into an infrared light flow vector sequence of the elbow joint, and converts the initial motion information vector sequence into a motion information vector sequence of the non-elbow joint.

[0037] Specifically, the control device 10 obtains the thermal image sequence of the elbow joint sent by the thermal imaging infrared sensor device 30, and converts the thermal image sequence into an infrared optical flow vector sequence according to the optical flow algorithm. The expression of the infrared optical flow vector sequence can be , represented by the thermal image sequence of the left elbow joint , and thermal image sequence of the right elbow joint Splicing to get infrared optical flow vector sequence .

[0038] The optical flow algorithm may be the Horn-Schunck algorithm. The infrared optical flow vector sequence refers to infrared optical flow vectors ordered by time, each infrared optical flow vector including the movement acceleration, movement angle and movement direction of each pixel point of the elbow joint in the thermal image, so that the real-time movement state of the elbow joint can be understood through the infrared optical flow vector sequence.

[0039] Next, the initial motion information vector sequence of the non-elbow joint sent by the inertial measurement unit 40 is obtained, and the initial motion information vector sequence of the non-elbow joint is optimized to obtain the motion information vector sequence of the non-elbow joint.

[0040] Finally, after aligning the timestamps of the infrared light flow vector sequence of the elbow joint and the motion information vector sequence of the non-elbow joint, step S140 is performed.

[0041] Step S140: The control device predicts the rotation angle vector of the elbow joint corresponding to the infrared light flow vector sequence of the elbow joint based on the first neural network model.

[0042] The expression of the rotation angle vector of the elbow joint can be .

[0043] Step S150: The control device splices the rotation angle vector of the elbow joint and the motion information vector sequence of the non-elbow joint to obtain a multimodal information vector sequence.

[0044] After obtaining the rotation angle vector of the elbow joint, the rotation angle vector of the elbow joint is data spliced ​​with the motion information vector sequence of the non-elbow joint to obtain a multimodal information vector sequence. That is, the multimodal information vector sequence includes two types of information: the rotation angle vector of the infrared elbow joint obtained based on the thermal imaging infrared sensor device 30, and the motion information vector sequence of the non-elbow joint obtained based on the inertial measurement unit 40.

[0045] Step S160: The control device predicts the rotation angle vector of the non-elbow joint corresponding to the multimodal information vector sequence based on the second neural network model combination.

[0046] The expression of the rotation angle vector of the non-elbow joint can be .

[0047] Step S170: The control device determines the movement posture of the upper body of the human body according to the rotation angle vector of the elbow joint and the rotation angle vector of the non-elbow joint.

[0048] Specifically, the rotation angle vector of the elbow joint and the rotation angle vector of the non-elbow joint are spliced ​​to obtain the movement posture vector of the upper body of the human body, thereby predicting the movement posture of the upper body of the human body in real time.

[0049] The expression of the upper body motion posture vector can be ,in Represents the motion posture vector of the upper body.

[0050] It is understandable that in the related art, there are motion capture systems based on single sensor applications such as inertial measurement units or visual sensors. Due to the single type of sensor and the single type of sensing data obtained, various limitations arise, resulting in low accuracy in determining the motion state and insufficient robustness of the motion capture system. Limitations include that the motion capture system based on the inertial measurement unit is susceptible to noise interference and data drift. The motion capture system based on the visual sensor is easily affected by the environment and obstructions when sensing data, resulting in inaccurate data acquisition. At the same time, when the visual sensor is deployed on clothes, there is also the problem of privacy leakage.

[0051] Compared with the application of a single sensor in the related art, in the above technical solution, multiple thermal imaging infrared sensing devices 30 and multiple inertial measurement units 40 are arranged on the short-sleeved clothing 20 to respectively sense the thermal image sequence of the elbow joint and the initial motion information vector sequence of the non-elbow joint. Then, the control device 10 integrates the infrared light flow sequence of the elbow joint obtained according to the thermal image sequence and the motion information sequence of the non-elbow joint obtained according to the initial motion information vector sequence. Based on the first neural network model and the second neural network model, the human body motion posture is determined, which overcomes the limitations of ambient light interference caused by the use of single modal information and data drift caused by long-term determination of human body motion posture, improves the stability and accuracy of posture determination, and improves the robustness of the device. At the same time, the user can perform posture determination while wearing the short-sleeved clothing 20, which improves comfort. The capture of the joint movement of the upper body of the human body with high precision, low interference, low cost, low privacy threat (no optical camera equipment) and low environmental requirements (no need for complex site layout) is achieved.

[0052] In some embodiments, the human body motion posture tracking method further includes controlling the device to construct a combination of the first neural network model and the second neural network model. Figure 4 First, the control device obtains training data as a training set, and the training data includes input data and label data. That is, the human body motion posture tracking 100 also includes an optical device 50, and the optical device 50 includes a camera or a digital camera.

[0053] The control device 10 obtains an optical image sequence of the elbow joint and an optical image sequence of the non-elbow joint through the optical device 50. An optical image sequence refers to an optical image sorted by time. After being processed, the optical image sequence of the elbow joint can indicate the true value of the rotation angle vector of the elbow joint, and after being processed, the optical image sequence of the non-elbow joint can indicate the true value of the rotation angle vector of the non-elbow joint. The optical image sequence of the elbow joint and the optical image sequence of the non-elbow joint can be used as label data. At the same time, the control device 10 retrieves the infrared light flow vector sequence of the elbow joint and the motion information vector sequence of the non-elbow joint.

[0054] Secondly, the control device 10 uses the infrared light flow vector sequence of the elbow joint as input data and the optical image sequence of the elbow joint as label data to train the residual network model (Residual Network, ResNet). The residual network model is a deep convolutional neural network model. By introducing jump connections (residual blocks), the problem of gradient disappearance in deep networks is solved, so that neural networks with many layers that need to learn complex features can be trained more effectively.

[0055] During training, the residual network model outputs the rotation angle vector of the elbow joint. Then, the control device 10 splices the rotation angle vector of the elbow joint with the motion information vector sequence of the non-elbow joint and generates a multimodal information vector sequence. The control device 10 uses the multimodal information vector sequence as input data and the optical image sequence of the non-elbow joint as label data to train the Long Short-Term Memory Network Model (LSTM) and the Multilayer Perceptron Model (MLP). The Long Short-Term Memory Network Model is a special recursive neural network model that can effectively capture long-term dependencies in sequence data using a gating mechanism, solving the problems of gradient vanishing and gradient exploding in traditional recurrent neural networks (RNNs).

[0056] It is understandable that the infrared optical flow vector sequence and the multimodal information vector sequence of the elbow joint in the training phase are used as training data. The infrared optical flow vector sequence of the elbow joint generated in step S130 and the multimodal information vector sequence generated in step S150 are used as prediction objects, which can be generated in the training phase or regenerated.

[0057] Until the total loss value of the total loss function of the residual network model, the long short-term memory network model and the multi-layer perceptron model is within a preset range, the first neural network model is constructed by the residual network model, and the second neural network model combination is constructed by the long short-term memory network model and the multi-layer perceptron model. That is, when the sum of the first loss value of the residual network model and the second loss value of the long short-term memory network model and the multi-layer perceptron model is within a preset range, the training of the first neural network model and the second neural network model combination is completed.

[0058] Exemplarily, the formula for determining the first loss value of the residual network model is as follows:

[0059] in, represents the first loss value, represents the rotation angle vector of the elbow joint, The expression can be , represents the rotation angle vector of the left elbow joint, represents the rotation angle vector of the right elbow joint, Indicates that the rotation angle vector has six dimensions. That is, the rotation angle vector of the elbow joint , the rotation angle vector of the left elbow joint and the rotation angle vector of the right elbow joint Get it by data splicing. Represents the true value of the rotation angle vector of the elbow joint. For specific expressions, refer to .

[0060] It can be understood that the rotation angle vector of the elbow joint in the training phase The true value of the rotation angle vector of the elbow joint The average of the squares of the differences is the first loss value .

[0061] Exemplarily, the second loss value determination formula of the long short-term memory network model and the multi-layer perceptron model is as follows:

[0062] in, represents the second loss value, Represents the rotation angle vector of the non-elbow joints during the training phase.

[0063] The expression of can be, , represents the rotation angle vector of the hip joint, represents the rotation angle vector of the spine, represents the rotation angle vector of the neck, Represents the joint vectors of the upper body except the elbow joint. It means that the rotation angle vector has fifty-four dimensions. It can be understood that the rotation angle vector of the non-elbow joint is , obtained by splicing the data of the joint vectors of the upper body except the elbow joint. Represents the true value of the rotation angle vector of the non-elbow joint.

[0064] It can be understood that the rotation angle vector of the non-elbow joint in the training phase The true value of the rotation angle vector of the non-elbow joint The average of the squares of the differences is the second loss value .

[0065] The formula for determining the sum of the first loss value of the residual network model and the second loss values ​​of the long short-term memory network model and the multilayer perceptron model is as follows:

[0066] in, represents the total loss value, The total loss value is the first loss value of the residual network model The second loss value of the long short-term memory network model and the multi-layer perceptron model sum.

[0067] After completing the construction of the second neural network model combination and the first neural network model, the infrared light flow vector sequence of the elbow joint retrieved in step S130 is input into the first neural network model, and the rotation angle vector of the elbow joint is output, thereby predicting the real-time motion state of the elbow joint. At the same time, the multimodal information vector sequence obtained in step S150 is input into the second neural network model combination, and the rotation angle vector of the non-elbow joint is output, thereby predicting the real-time motion state of the non-elbow joint.

[0068] It can be understood that compared with other traditional machine learning algorithms, the residual network model, as a convolutional neural network model, shows good performance in the infrared optical flow vector sequence processing task of the elbow joint.

[0069] It is understandable that in real scenes, clothes may wrinkle, deform, etc. as the human body moves, so it is very challenging to map part of the body image to the human body motion posture. Therefore, the embodiment of the present application uses deep learning technologies such as residual network model, long short-term memory network model and multi-layer perceptron model to achieve the mapping of the thermal image sequence of the elbow joint sensed by the thermal imaging infrared sensor device and the initial motion information vector sequence of the non-elbow joint sensed by the inertial measurement unit to the human body motion posture, so as to solve the problem of mapping the body image to the human body motion posture.

[0070] Since the data structures and feature dimensions sensed by the thermal imaging infrared sensor device and the inertial measurement unit are different, it is difficult to directly splice the two types of data and input them into the neural network for prediction. Therefore, in an embodiment of the present application, the rotation angle vector of the elbow joint corresponding to the infrared optical flow vector sequence of the elbow joint is first predicted through a residual network model. Then, through a combination of a long short-term memory network model and a multi-layer perceptron model, the multimodal information vector sequence obtained by splicing the rotation angle vector of the elbow joint and the motion information vector sequence of the non-elbow joint is predicted, corresponding to the rotation angle vector of the non-elbow joint. This solves the problem that the two types of data cannot be directly spliced ​​and simultaneously input into the neural network for prediction.

[0071] In addition, if the thermal imaging infrared sensor device and the inertial measurement unit are directly used to predict the joints they are placed at, they cannot fully utilize the correlation between the movements of different joints of the user to further improve the prediction effect. Therefore, the present invention designs a multimodal feature fusion framework based on a residual network model, a long short-term memory network model and a multi-layer perceptron model to fully utilize the correlation between the movements of different joints of the user and further improve the prediction effect.

[0072] In some embodiments, see Figure 5 and 6 The human body motion posture tracking method also includes the process that the control device optimizes the initial motion information vector sequence of the non-elbow joint using the attention mechanism model to obtain the motion information vector sequence of the non-elbow joint. Figure 5 As shown, the method also includes the following steps: Step S210: The control device converts the angular velocity vector into a rotational acceleration vector according to the interval time between adjacent angular velocity vectors.

[0073] The initial motion information vector sequence of the non-elbow joints obtained includes an angular velocity vector sequence and an acceleration vector sequence. Figure 6 As shown, the initial motion information vector sequence includes initial motion information vectors ordered in time. , the angular velocity vector sequence includes the angular velocity vectors sorted in time , the acceleration vector sequence includes the acceleration vectors ordered by time That is, the inertial measurement unit 40 includes an accelerometer and a gyroscope. The accelerometer is used to sense an acceleration vector sequence, and the gyroscope is used to sense an angular velocity vector sequence.

[0074] like Figure 6 As shown, according to the angular velocity vector Get the rotation acceleration vector , specifically, the formula for determining the rotational acceleration vector is as follows:

[0075] in, represents the rotation acceleration vector, represents the angular velocity vector, represents another adjacent angular velocity vector. That is, the rotation acceleration vector can be determined according to the interval time between adjacent angular velocity vectors (the interval time can be a unit time) and the adjacent angular velocity vectors. In one embodiment, t+1 represents the current moment, and t represents the previous moment.

[0076] Step S220: The control device adjusts the acceleration vector based on the proportion of the acceleration vector in the rotation acceleration vector to obtain a target acceleration vector.

[0077] like Figure 6 As shown, the rotation acceleration vector and the acceleration vector The input acceleration vector ratio determines the model and the output acceleration vector The rotational acceleration vector The proportion of .

[0078] The acceleration vector ratio model is as follows:

[0079] It can be understood that the quadratic norm of the acceleration vector is Norm of the vector with the rotation acceleration vector The ratio is the proportion of the acceleration vector to the rotation acceleration vector .

[0080] Next, we transform the acceleration vector In the acceleration vector The proportion of , input acceleration vector weight to determine the model, output acceleration vector weight , the weight of the acceleration vector is the attention matrix of the attention mechanism model. The acceleration vector weight determination model is as follows:

[0081] in, represents the linear layer weights, , Indicates the parameter, which is preset. That is, according to the proportion of the acceleration vector in the rotation acceleration vector , the preset linear layer weights and parameters Get the weight of the acceleration vector It can be understood that when the human body is assumed to be a rigid body, by increasing the attention matrix This linear layer (weight of the acceleration vector) allows the residual network model, the long short-term memory network model and the multi-layer perceptron model to have a certain degree of freedom to fine-tune the weights during the training process, making the constructed combination of the first neural network model and the second neural network model more adaptable to the real world.

[0082] Next, according to the weight of the acceleration vector Adjusting the acceleration vector , get the target acceleration vector .

[0083] It can be understood that the embodiment of the present application is based on the attention mechanism model, according to the acceleration vector The rotational acceleration vector The proportion of , determine whether the current motion is dominated by the rotational acceleration caused by rotation (that is, the larger the proportion, the less dominated by the rotational acceleration). If not, give the acceleration vector of the accelerometer of the current frame (that is, the current moment) a larger weight for regression. If so, give it a smaller weight.

[0084] Step S230: The control device obtains a motion information vector sequence of the non-elbow joint based on the spliced ​​target acceleration vector and angular velocity vector.

[0085] Specifically, Figure 6 As shown, the target acceleration vector and the angular velocity vector Data splicing , get the motion information vector of the non-elbow joint In this way, based on the motion information vectors of multiple non-elbow joints, a motion information vector sequence of the non-elbow joints is obtained.

[0086] It can be understood that the process of optimizing the initial motion information vector sequence of the non-elbow joints by using the attention mechanism model to obtain the motion information vector sequence of the non-elbow joints is suitable for both the training stage and the prediction stage.

[0087] It can be understood that due to the differences in body shape of different users and the movement of users during data collection, the inertial measurement unit produces different displacements and records noise, resulting in a decrease in the accuracy of human motion tracking. Under the assumption that the human body is a rigid body, the inertial measurement unit records translational acceleration, acceleration relative to the direction of gravity, and rotational acceleration. Among these three, the first two components are not affected by the wearing position, and only the rotational acceleration caused by rotation is sensitive to the displacement of the inertial measurement unit in the user's body. Therefore, if the acceleration vector of the accelerometer accounts for a larger proportion of the rotational acceleration vector of the gyroscope (that is, the larger the proportion of the acceleration vector of the accelerometer), a larger weight is given to the acceleration vector, making the target acceleration vector larger, so as to weaken the rotational acceleration vector of the gyroscope and reduce the influence of the displacement of the inertial measurement unit in the user's body. While retaining most of the relevant information, the combination of the first neural network model and the second neural network model can be made more robust to the offset position of the inertial measurement unit, thereby reducing the noise of the inertial measurement unit worn by different users. It is understandable that in the relevant technology, there are multimodal motion monitoring solutions that integrate multiple sensor data, that is, multiple sensing devices such as accelerometers, gyroscopes and magnetometers are usually used, combined with data fusion algorithms (such as Kalman filters) to estimate the posture and motion trajectory of the human body, and further integrate the data of visual sensors, pressure sensors or depth cameras, and use complex algorithms to achieve more comprehensive motion capture. At the same time, deep learning technology is used to optimize model performance to adapt to the body shape and sports style of different users.

[0088] However, the data fusion algorithm used in the multimodal motion monitoring solution still faces challenges such as long-term monitoring of data, resulting in reduced tracking accuracy and strong environmental dependence. In the embodiment of the present application, the attention mechanism model is used to calibrate the inertial measurement unit data noise in real time, and the streamer algorithm is used to calibrate the data of the thermal imaging infrared sensor device in real time, thereby enhancing the robustness of the combination of the first neural network model and the second neural network model for different user wearing scenarios.

[0089] See also Figure 7 and Figure 8 The above embodiments are described again through the following application scenario. The human body motion posture tracking device 100 performs the following steps: a1: The thermal imaging infrared sensor device 30 senses the thermal image sequence of the left and right elbow joints, the inertial measurement unit 40 senses the initial motion information vector sequence of the non-elbow joints, and the optical device 50 senses the optical image sequence of the elbow joints and non-elbow joints; a2: the control device 10 converts the thermal image sequence into an infrared optical flow vector sequence according to the optical flow algorithm, and converts the initial motion information vector sequence of the non-elbow joint into the motion information vector sequence of the non-elbow joint according to the attention mechanism model; a3: The control device 10 uses the infrared light flow vector sequence of the elbow joint as input data and the optical image sequence of the elbow joint as label data to train the residual network model, and outputs the rotation angle vector of the elbow joint and calculates the first loss value; a4: the control device 10 splices the rotation angle vector of the elbow joint and the motion information vector sequence of the non-elbow joint to obtain a multi-modal information vector sequence; a5: the control device 10 uses the multimodal information vector sequence as input data and the optical image sequence of the non-elbow joint as label data to train the long short-term memory network model and the multi-layer perceptron model, and outputs the rotation angle vector of the elbow joint and calculates the second loss value; a6: When the sum of the first loss value and the second loss value is within a preset range, the control device 10 constructs a combination of the first neural network model and the second neural network model; a7: The control device 10 obtains a real-time infrared optical flow vector sequence and an initial motion information sequence, inputs the real-time infrared optical flow vector sequence and the initial motion information sequence into a first neural network model and a second neural network model combination, and outputs the motion posture of the upper body of the human body.

[0090] The human motion posture tracking device 100 further includes an ESP32 circuit board 60 and a battery (not shown). The infrared thermal imaging sensor device 30 is produced by Heimann and its model is HTPA80×864dR2L3.9 / 0.8. Each infrared thermal imaging sensor device 30 has a height of 12.2 mm, a diameter of 20 mm, a field of view of 120×90 degrees, a resolution of 64×80, and a frame rate of 20 FPS. Each infrared thermal imaging sensor device 30 is connected to its respective ESP32 circuit board 60 (specification model ESP-WROOM-32) through a wire and is powered by a 5V, 900mAh battery (with a battery life of 2.45 hours). The infrared thermal imaging sensor device 30 sends and saves the measurement data to the control device 10 at the data collection site via WiFi. The inertial measurement unit 40 for obtaining the initial motion information vector sequence of the non-elbow joint is produced by Movella and its model is Xsens DOT. Each inertial measurement unit 40 has a length and width of 32 mm × 20 mm and a thickness of 10 mm. The data acquisition frame rate is 60 FPS, and is equipped with a 45 mAh LIR2032 rechargeable button battery (which can last for 6 hours) and a Bluetooth module (not shown) that can send measurement data to the control device 10.

[0091] Fig. 9 This is a schematic diagram of the structure of the control device provided in this application. Fig. 9 As shown, the control device 10 includes: Processor 11, memory 12 and bus 13; The memory 12 is used to store the computer program code of the processor 11; The processor 11 is configured to execute the technical solution of the human body motion posture tracking method in any of the aforementioned method embodiments by executing the computer program code.

[0092] Optionally, the memory 12 may be independent or integrated with the processor 11 .

[0093] The memory 12 is connected to the processor 11 via the bus 13 and completes communication between them.

[0094] Optionally, the memory 12 may include a random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0095] The bus 13 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0096] The above-mentioned processor can be a general-purpose processor, including a central processing unit CPU, a network processor (NP), etc.; it can also be a digital signal processor DSP, an application-specific integrated circuit ASIC, a field programmable gate array FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0097] The control device 10 is used to execute the technical solution provided in any of the aforementioned method embodiments, and its implementation principle and technical effect are similar and will not be repeated here.

[0098] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A human body motion posture tracking method, applied to a human body motion posture tracking device, characterized in that: The human motion posture tracking device comprises a short-sleeved garment and a control device; the short-sleeved garment is provided with a plurality of thermal imaging infrared sensor devices and a plurality of inertial measurement units; the plurality of thermal imaging infrared sensor devices sense a thermal image sequence of an elbow joint, and the plurality of inertial measurement units sense an initial motion information vector sequence of a non-elbow joint; The control device converts the thermal image sequence into an infrared light flow vector sequence of the elbow joint, and converts the initial motion information vector sequence into a motion information vector sequence of the non-elbow joint; Based on the first neural network model, predicting the rotation angle vector of the elbow joint corresponding to the infrared optical flow vector sequence of the elbow joint; splicing the rotation angle vector of the elbow joint and the motion information vector sequence of the non-elbow joint to obtain a multimodal information vector sequence; Based on the second neural network model combination, predicting the rotation angle vector of the non-elbow joint corresponding to the multimodal information vector sequence; The movement posture of the upper body of the human body is determined according to the rotation angle vector of the elbow joint and the rotation angle vector of the non-elbow joint.

2. The method according to claim 1, characterized in that The short-sleeved clothing comprises two sleeves, and the multiple thermal imaging infrared sensor devices are respectively arranged on the two sleeves.

3. The method according to claim 1 or 2, characterized in that: The short-sleeved garment also includes two short-sleeved shoulder lines and two armpit covering points, a first connecting line is formed between the two short-sleeved shoulder lines, a second connecting line is formed between the two armpit covering points, and the multiple inertial measurement units are respectively arranged at the midpoint between the first connecting line and the second connecting line, and at the hem of the short-sleeved garment close to the pelvis.

4. The method according to claim 1, characterized in that The initial motion information vector sequence includes an angular velocity vector sequence and an acceleration vector sequence, wherein the angular velocity vector sequence includes angular velocity vectors ordered by time, and the acceleration vector sequence includes acceleration vectors ordered by time. The control device further includes: Converting the angular velocity vector into a rotation acceleration vector according to the interval time between adjacent angular velocity vectors; Based on the proportion of the acceleration vector in the rotation acceleration vector, adjusting the acceleration vector to obtain a target acceleration vector; Based on the spliced ​​target acceleration vector and the angular velocity vector, a motion information vector sequence of the non-elbow joint is obtained.

5. The method according to claim 4, characterized in that The control device also The weight of the acceleration vector is obtained according to the proportion of the acceleration vector in the rotation acceleration vector, the preset linear layer weight and the parameter, wherein the proportion of the acceleration vector in the rotation acceleration vector is the ratio of the quadratic norm of the vector of the acceleration vector to the norm of the vector of the rotation acceleration vector; The acceleration vector is adjusted according to a weight of the acceleration vector.

6. The method according to claim 1, characterized in that The control device also Converting the thermal image sequence into the infrared optical flow vector sequence according to an optical flow algorithm; Align the timestamps of the infrared optical flow vector sequence of the elbow joint and the motion information vector sequence of the non-elbow joint.

7. The method according to claim 1, characterized in that The first neural network model is obtained by training a residual network model using the infrared optical flow vector sequence of the elbow joint as input data and the optical image sequence of the elbow joint as label data, wherein the optical image sequence of the elbow joint is used to indicate the true value of the rotation angle vector of the elbow joint; The second neural network model combination is obtained by training a long short-term memory network model and a multi-layer perceptron model using the multimodal information vector sequence as input data and an optical image sequence of a non-elbow joint as label data, wherein the optical image sequence of the non-elbow joint indicates a true value of a rotation angle vector of the non-elbow joint; When the sum of the first loss value of the residual network model and the second loss values ​​of the long short-term memory network model and the multi-layer perceptron model is within a preset range, the training of the residual network model, the long short-term memory network model and the multi-layer perceptron model is completed, wherein the first loss value is calculated based on the rotation angle vector of the elbow joint and the true value of the rotation angle vector of the elbow joint; the second loss value is calculated based on the rotation angle vector of the non-elbow joint and the true value of the rotation angle vector of the non-elbow joint.

8. A human body motion posture tracking device, characterized in that: The human motion posture tracking device comprises a short-sleeved garment and a control device; the short-sleeved garment is provided with a plurality of thermal imaging infrared sensor devices and a plurality of inertial measurement units; the plurality of thermal imaging infrared sensor devices are used to sense a thermal image sequence of an elbow joint, and the plurality of inertial measurement units are used to sense an initial motion information vector sequence of a non-elbow joint; The control device is used to convert the thermal image sequence into an infrared light flow vector sequence of the elbow joint, and convert the initial motion information vector sequence into a motion information vector sequence of the non-elbow joint; Based on the first neural network model, predicting the rotation angle vector of the elbow joint corresponding to the infrared optical flow vector sequence of the elbow joint; splicing the rotation angle vector of the elbow joint and the motion information vector sequence of the non-elbow joint to obtain a multimodal information vector sequence; Based on the second neural network model combination, predicting the rotation angle vector of the non-elbow joint corresponding to the multimodal information vector sequence; The movement posture of the upper body of the human body is determined according to the rotation angle vector of the elbow joint and the rotation angle vector of the non-elbow joint.

9. The human body motion posture tracking device according to claim 8, characterized in that: The short-sleeved clothing comprises two sleeves, and the multiple thermal imaging infrared sensor devices are respectively arranged on the two sleeves.

10. The human body motion posture tracking device according to claim 8 or 9, characterized in that: The short-sleeved garment also includes two short-sleeved shoulder lines and two armpit covering points, a first connecting line is formed between the two short-sleeved shoulder lines, a second connecting line is formed between the two armpit covering points, and the multiple inertial measurement units are respectively arranged at the midpoint between the first connecting line and the second connecting line, and at the hem of the short-sleeved garment close to the pelvis.

Images