Human body motion posture tracking equipment and short-sleeved clothes

Abstract

The utility model provides a human motion posture tracking device which comprises a short-sleeve garment and a control device, the short-sleeve garment comprises two sleeves and a garment body, the two sleeves are provided with a plurality of thermal imaging infrared sensing devices, and the garment body is provided with a plurality of inertial measurement units; the plurality of thermal imaging infrared sensing devices are used for sensing a thermal imaging sequence of an elbow joint, and the plurality of inertial measurement units are used for sensing an initial motion information vector sequence of a non-elbow joint; the control device comprises a wireless communication module, and the wireless communication module is used for inputting a thermal image sequence of an elbow joint and an initial motion information vector sequence of a non-elbow joint. The arrangement of the multi-mode sensor on the short-sleeved garment is realized for the first time, the freedom of movement of a user wearing the short-sleeved garment can be provided, the comfort of the user is improved, and meanwhile, the accuracy of data sensing is high, the cost is low, and the environmental requirement is low. Optical camera equipment is not arranged, so that the threat of privacy disclosure is reduced.

Description

Technical Field

[0001] This application relates to the field of human motion capture and sensing technology, and in particular to a human motion posture tracking device. Background Technology

[0002] With the rapid development of technology, human motion capture and sensing technology has been widely applied in many fields, including virtual reality, augmented reality, motion analysis and rehabilitation training, intelligent monitoring, and human-computer interaction. Human motion capture and sensing technology requires the use of motion capture systems, which typically rely on expensive optical camera equipment and complex site setups, resulting in several limitations such as high cost, stringent environmental requirements, and potential threats to user privacy.

[0003] In recent years, the application of smart sensing clothing technology has integrated miniature sensors into everyday clothing, enabling real-time capture of human motion information. Currently, users need to wear tight-fitting clothing (with sensors integrated into the clothing) or tight-fitting sensing equipment for posture tracking. There is still considerable room for improvement in ensuring data accuracy, enhancing the robustness of smart sensing clothing, and optimizing user comfort.

[0004] Therefore, this application urgently needs to find a human motion posture tracking device that is low in cost, has low environmental requirements and low privacy threats, while also having high posture determination accuracy and high user comfort. Utility Model Content

[0005] This application provides a human motion posture tracking device to improve the accuracy of human motion posture determination and user comfort, while reducing cost, environmental requirements and privacy threats.

[0006] The first aspect of this application provides a human motion posture tracking device, which includes a short-sleeved garment and a control device. The short-sleeved garment includes two sleeves and a body. Multiple thermal imaging infrared sensors are disposed on the two sleeves, and multiple inertial measurement units are disposed on the body. The multiple thermal imaging infrared sensors are used to sense thermal image sequences of the elbow joint, and the multiple inertial measurement units are used to sense initial motion information vector sequences of non-elbow joints. The control device includes a wireless communication module, which is used to input the thermal image sequences of the elbow joints and the initial motion information vector sequences of non-elbow joints.

[0007] In some embodiments of the first aspect, the two sleeves include a first sleeve and a second sleeve, and the plurality of thermal imaging infrared sensing devices include a first thermal imaging infrared sensing device and a second thermal imaging infrared sensing device.

[0008] The first thermal imaging infrared sensor is positioned near the cuff of the first sleeve, and the first camera of the first thermal imaging infrared sensor is facing the first elbow joint to sense the thermal image sequence of the first elbow joint.

[0009] The second thermal imaging infrared sensor is positioned near the cuff of the second sleeve, and the second camera of the second thermal imaging infrared sensor is directed toward the second elbow joint to sense a sequence of thermal images of the second elbow joint.

[0010] In some embodiments of the first aspect, the human motion posture tracking device further includes a first circuit board and a second circuit board, each having a first battery and a first wireless communication module. The first circuit board is connected to a first thermal imaging infrared sensor and is used to power the first thermal imaging infrared sensor via the first battery and to transmit a thermal image sequence of the first elbow joint to a control device via the first wireless communication module. The second circuit board is connected to a second thermal imaging infrared sensor and is used to power the second thermal imaging infrared sensor via the first battery and to transmit a thermal image sequence of the second elbow joint to the control device via the first wireless communication module.

[0011] In some embodiments of the first aspect, the garment body includes two short-sleeve shoulder lines and two armpit coverage points, a first connecting line is formed between the two short-sleeve shoulder lines, a second connecting line is formed between the two armpit coverage points, and a plurality of inertial measurement units include a first inertial measurement unit and a second inertial measurement unit, the first inertial measurement unit being disposed at the midpoint between the first connecting line and the second connecting line, and the second inertial measurement unit being disposed at the hem of the garment body near the pelvis.

[0012] In some embodiments of the first aspect, both the first inertial measurement unit and the second inertial measurement unit include a gyroscope and an accelerometer. The gyroscope is used to sense an angular velocity vector sequence, and the accelerometer is used to sense an acceleration vector sequence. The angular velocity vector sequence and the acceleration vector sequence form an initial motion information vector sequence.

[0013] In some embodiments of the first aspect, the first inertial measurement unit and the second inertial measurement unit further include a second battery and a second wireless communication module, the second battery being used to power the gyroscope and accelerometer, and the second wireless communication module being used to send an initial motion information vector sequence to the control device.

[0014] The second aspect of this application provides a short-sleeved garment, which includes two sleeves and a body. Multiple thermal imaging infrared sensors are provided on the two sleeves, and multiple inertial measurement units are provided on the body. The multiple thermal imaging infrared sensors are used to sense thermal image sequences of the elbow joint and upload them to a control device. The multiple inertial measurement units are used to sense initial motion information vector sequences of non-elbow joints and upload them to the control device.

[0015] In some embodiments of the second aspect, the two sleeves include a first sleeve and a second sleeve, and the plurality of thermal imaging infrared sensing devices include a first thermal imaging infrared sensing device and a second thermal imaging infrared sensing device.

[0016] A first thermal imaging infrared sensor is positioned near the cuff of the first sleeve, with its camera facing the first elbow joint, for sensing a thermal image sequence of the first elbow joint; a second thermal imaging infrared sensor is positioned near the cuff of the second sleeve, with its camera facing the second elbow joint, for sensing a thermal image sequence of the second elbow joint.

[0017] In some embodiments of the second aspect, the garment body includes two short-sleeve shoulder lines and two armpit coverage points, a first connecting line is formed between the two short-sleeve shoulder lines, a second connecting line is formed between the two armpit coverage points, and a plurality of inertial measurement units include a first inertial measurement unit and a second inertial measurement unit, the first inertial measurement unit being disposed at the midpoint between the first connecting line and the second connecting line, and the second inertial measurement unit being disposed at the hem of the garment body near the pelvis.

[0018] The human motion posture tracking device and short-sleeved clothing provided in this application integrate a thermal imaging infrared sensor on the sleeve of the short-sleeved clothing and an inertial measurement unit on the body of the clothing. This is the first time that a multimodal sensor arrangement has been implemented on short-sleeved clothing, allowing users greater freedom of movement while wearing the garment, improving user comfort, and providing high data sensing accuracy, low cost, and minimal environmental requirements. The absence of optical camera equipment reduces the threat of privacy breaches. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0020] Figure 1 A schematic diagram of the structure of the human motion posture tracking device provided in the embodiments of this application;

[0021] Figure 2 A structural schematic diagram of a short-sleeved garment provided in an embodiment of this application;

[0022] Figure 3 A schematic diagram of the structure of the thermal imaging infrared sensing device provided in the embodiments of this application;

[0023] Figure 4 A schematic diagram of the structure of an inertial measurement unit provided in an embodiment of this application;

[0024] Figure 5A flowchart illustrating the human motion posture tracking method provided in this application embodiment;

[0025] Figure 6 This is an application scenario diagram of the human motion posture tracking method provided in the embodiments of this application;

[0026] Figure 7 This is another application scenario diagram of the human motion posture tracking method provided in the embodiments of this application;

[0027] Figure 8 Another flowchart illustrating the human motion posture tracking method provided in this application embodiment;

[0028] Figure 9 This is another application scenario diagram of the human motion posture tracking method provided in the embodiments of this application;

[0029] Figure 10 and Figure 11 This is another application scenario diagram of the human motion posture tracking method provided in the embodiments of this application.

[0030] Figure 12 This is a schematic diagram of the control device provided in an embodiment of this application.

[0031] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments.

[0032] Detailed Implementation

[0033] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0034] The terms “first”, “second”, etc. used in this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated.

[0035] The technical solution of this application and how the technical solution of this application solves the technical problem will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0036] The data vectors involved in this application refer to data that exists in vector form. For example, an infrared optical flow vector refers to an infrared optical flow that exists in vector form.

[0037] Please see Figure 1 , Figure 1 This is a structural schematic diagram of a human motion posture tracking device according to an embodiment of this application.

[0038] like Figure 1 and Figure 2 As shown, the human motion posture tracking device 100 includes a control device 10 and a short-sleeved garment 20. (As shown...) Figure 2 (1) is a front view of short-sleeved garment 20. Figure 2 (2) is a rear view of the short-sleeved garment 20. The short-sleeved garment 20 includes two sleeves 21 and a body 22. Multiple thermal imaging infrared sensors 30 are provided on the two sleeves 21, and multiple inertial measurement units 40 are provided on the body 22.

[0039] A thermal imaging infrared sensor 30 is used to sense thermal images of the elbow joint and generate a sequence of thermal images. An inertial measurement unit 40 senses initial motion information of the non-elbow joint and generates an initial motion information sequence. The control device 10 includes a wireless communication module 11, which is used to input the thermal image sequence of the elbow joint and the initial motion information vector sequence of the non-elbow joint. In some embodiments, the control device 10 can be a laptop, mobile phone, server, etc.

[0040] Understandably, compared to related technologies, where users need to wear tight-fitting clothing (with sensors integrated on the clothing) or tight-fitting sensing equipment for posture tracking, this application places a thermal imaging infrared sensor 30 on the sleeve 21 of the short-sleeved garment 20 and an inertial measurement unit 40 on the body 22 of the short-sleeved garment 20. This is the first time that a multimodal sensor arrangement has been implemented on a short-sleeved garment 20, giving users freedom of movement while wearing the garment, improving user comfort, while also offering high data sensing accuracy, low cost, and low environmental requirements. The absence of optical camera equipment reduces the threat of privacy breaches.

[0041] In some embodiments, such as Figure 2 and Figure 3As shown, the two sleeves 21 include a first sleeve 211 and a second sleeve 212. Multiple thermal imaging infrared sensors 30 include a first thermal imaging infrared sensor 31 and a second thermal imaging infrared sensor 32. The first thermal imaging infrared sensor 31 is positioned near the cuff of the first sleeve 211, with its first camera 311 facing the first elbow joint 1. The first thermal imaging infrared sensor 31 senses the thermal image sequence of the first elbow joint 1 through its first camera 311. The second thermal imaging infrared sensor 32 is positioned near the cuff of the second sleeve 212, with its second camera 321 facing the second elbow joint 2. The second thermal imaging infrared sensor 32 senses the thermal image sequence of the second elbow joint 2 through its second camera 321. It can be understood that the first elbow joint 1 can be the left elbow joint, and the second elbow joint 2 can be the right elbow joint.

[0042] It is understood that, in this embodiment of the application, a first thermal imaging infrared sensor 31 and a second thermal imaging infrared sensor 32 are respectively provided on the first sleeve 211 and the second sleeve 212, so that when the human arm is performing various movements, the thermal imaging infrared sensor 30 can capture thermal images of the left elbow joint and the right elbow joint, and the thermal images indicate the movement status of the left elbow joint and the right elbow joint.

[0043] In some embodiments, such as Figure 2 As shown, the garment body 22 includes two short-sleeve shoulder lines 23 and two armpit coverage points 24. A first connecting line 231 is formed between the two short-sleeve shoulder lines 23, and a second connecting line 241 is formed between the two armpit coverage points 24. The multiple inertial measurement units 40 include a first inertial measurement unit 41 and a second inertial measurement unit 42. The first inertial measurement unit 41 is located at the midpoint 25 between the first connecting line 231 and the second connecting line 241, and the second inertial measurement unit 42 is located at the hem of the garment body 22 near the pelvis.

[0044] It is understandable that the midpoint 25 between the first line 231 connecting the two short-sleeve shoulder lines 23 and the second line 241 connecting the two armpit coverage points 24, as well as the hem of the garment 22, has a stronger load-bearing capacity than the sleeve 21 and does not conflict with the placement of the thermal imaging infrared sensor 30. Therefore, in this embodiment, by setting the first inertial measurement unit 41 at the midpoint 25 and the second inertial measurement unit 42 at the hem of the garment 22, when the human body performs various movements, the first inertial measurement unit 41 and the second inertial measurement unit 42 can stably sense the motion information of non-elbow joints (i.e., other upper body joints besides the elbow joint) to reflect the motion status of the human body's non-elbow joints.

[0045] In some embodiments, such as Figure 2 and Figure 3As shown, the human motion posture tracking device 100 also includes a first circuit board 51 and a second circuit board 52. Both the first circuit board 51 and the second circuit board 52 are equipped with a first battery 61 and a first wireless communication module 71. The first circuit board 51 is connected to a first thermal imaging infrared sensor 31. The first circuit board 51 is used to power the first thermal imaging infrared sensor 31 via the first battery 61 and to transmit a thermal image sequence of the first elbow joint 1 to the control device 10 via the first wireless communication module 71. The second circuit board 52 is connected to a second thermal imaging infrared sensor 32. The second circuit board 52 is used to power the second thermal imaging infrared sensor 32 via the first battery 61 and to transmit a thermal image sequence of the second elbow joint to the control device 10 via the first wireless communication module 71.

[0046] In one embodiment, the first thermal imaging sensor 31 and the second thermal imaging infrared sensor 32 are manufactured by Heimann, model HTPA80×864dR2L3.9 / 0.8, with a height of 12.2 mm, a diameter of 20 mm, a field of view of 120×90 degrees, a resolution of 64×80, and a frame rate of 20 FPS. The first circuit board 51 and the second circuit board 52 are both ESP32 circuit boards (model number ESP-WROOM-32). The first battery has specifications of 5V, 900mAh, and a battery life of 2.45 hours. The first wireless communication module 71 is a WiFi module.

[0047] In some embodiments, such as Figure 2 and 4 As shown, both the first inertial measurement unit 41 and the second inertial measurement unit 42 include a gyroscope 81 and an accelerometer 82. The gyroscope 81 is used to sense the angular velocity vector sequence, and the accelerometer 82 is used to sense the acceleration vector sequence. The angular velocity vector sequence and the acceleration vector sequence form the initial motion information vector sequence.

[0048] In some embodiments, the first inertial measurement unit 41 and the second inertial measurement unit 42 further include a second battery 83 and a second wireless communication module 84. The second battery 83 powers the gyroscope 81 and the accelerometer 82, and the second wireless communication module 84 transmits the initial motion information vector sequence to the control device 10. In one embodiment, the first inertial measurement unit 41 and the second inertial measurement unit 42 are manufactured by Movella, with the model number Xsens DOT. The first inertial measurement unit 41 or the second inertial measurement unit 42 has a length and width of 32mm × 20mm and a thickness of 10mm, and a data acquisition frame rate of 60FPS. The second battery 83 is a 45mAh LIR2032 rechargeable button battery (with a battery life of 6 hours). The second wireless communication module 84 is a Bluetooth module.

[0049] It should be noted that, Figures 1 to 4 This is merely a schematic diagram of one possible structure provided in this application. In the specific application of the solution, it can be set according to actual needs.

[0050] This application also provides a human motion posture tracking method, which is applied to a human motion posture tracking device. (See also...) Figure 5 The tracking method includes the following steps:

[0051] Step S110: Multiple thermal imaging infrared sensors sense a sequence of thermal images of the elbow joint.

[0052] like Figure 6 As shown, the thermal image sequence of the elbow joint includes a thermal image sequence of the left elbow joint and a thermal image sequence of the right elbow joint. The thermal image sequence refers to thermal images ordered by time.

[0053] Step S120: Multiple inertial measurement units sense the initial motion information vector sequence of the non-elbow joint.

[0054] The expression for the initial motion information vector sequence can be: The initial motion information vector sequence refers to motion information vectors ordered by time. Through these motion information vectors, the real-time motion state of non-elbow joints can be understood.

[0055] Step S130: The control device converts the thermal image sequence into an infrared optical flow vector sequence of the elbow joint, and converts the initial motion information vector sequence into a motion information vector sequence of the non-elbow joint.

[0056] Specifically, the control device 10 acquires a sequence of thermally rendered images of the elbow joint sent by the thermal imaging infrared sensor 30, and converts the thermally rendered image sequence into an infrared optical flow vector sequence according to an optical flow algorithm. The expression for the infrared optical flow vector sequence can be... This is represented as a sequence of thermal images of the left elbow joint. And thermal image sequence of the right elbow joint The infrared optical flow vector sequence is obtained by splicing. .

[0057] Optical flow algorithms can include the Horn-Schunck algorithm. An infrared optical flow vector sequence refers to infrared optical flow vectors ordered by time. Each infrared optical flow vector includes the acceleration, angle, and direction of motion of each pixel in the thermal image of the elbow joint. Therefore, the real-time motion state of the elbow joint can be understood through the infrared optical flow vector sequence.

[0058] Next, the initial motion information vector sequence of the non-elbow joints sent by the inertial measurement unit 40 is acquired, and the initial motion information vector sequence of the non-elbow joints is optimized to obtain the motion information vector sequence of the non-elbow joints.

[0059] Finally, after aligning the timestamps of the infrared optical flow vector sequence of the elbow joint and the motion information vector sequence of the non-elbow joint, step S140 is performed.

[0060] Step S140: The control device predicts the rotation angle vector of the elbow joint corresponding to the infrared optical flow vector sequence of the elbow joint based on the first neural network model.

[0061] The expression for the rotation angle vector of the elbow joint can be: .

[0062] Step S150: The control device splices the rotation angle vector of the elbow joint with the motion information vector sequence of the non-elbow joint to obtain a multimodal information vector sequence.

[0063] After obtaining the rotation angle vector of the elbow joint, the rotation angle vector of the elbow joint is concatenated with the motion information vector sequence of the non-elbow joint to obtain a multimodal information vector sequence. That is, the multimodal information vector sequence includes two types of information: the rotation angle vector of the infrared elbow joint obtained based on the thermal imaging infrared sensor 30, and the motion information vector sequence of the non-elbow joint obtained based on the inertial measurement unit 40.

[0064] Step S160: The control device predicts the rotation angle vector of the non-elbow joint corresponding to the multimodal information vector sequence based on the combination of the second neural network model.

[0065] The expression for the rotation angle vector of the non-elbow joint can be: .

[0066] Step S170: The control device determines the upper body movement posture based on the rotation angle vector of the elbow joint and the rotation angle vector of the non-elbow joint.

[0067] Specifically, the rotation angle vector of the elbow joint and the rotation angle vector of the non-elbow joint are concatenated to obtain the upper body motion posture vector, thereby predicting the upper body motion posture in real time.

[0068] The expression for the human upper body motion posture vector can be: ,in This represents the motion posture vector of the human upper body.

[0069] Understandably, related technologies include motion capture systems based on single sensors such as inertial measurement units (IMUs) or vision sensors. Due to the limited sensor types and the limited types of sensor data acquired, various limitations arise, leading to low accuracy in determining the motion state and insufficient robustness of the motion capture system. These limitations include: IMU-based motion capture systems are susceptible to noise interference and data drift; vision sensor-based motion capture systems are easily affected by environmental factors and obstructions, resulting in inaccurate data acquisition; and when vision sensors are deployed on clothing, privacy issues arise.

[0070] Compared to the use of a single sensor in related technologies, the above-mentioned technical solution utilizes multiple thermal imaging infrared sensors 30 and multiple inertial measurement units 40 mounted on the short-sleeved garment 20 to sense the thermal image sequence of the elbow joint and the initial motion information vector sequence of the non-elbow joints, respectively. Then, the control device 10 integrates the infrared optical flow sequence of the elbow joint obtained from the thermal image sequence and the motion information sequence of the non-elbow joints obtained from the initial motion information vector sequence—two modalities of information. Based on a first neural network model and a second neural network model, the human body's posture is determined. This overcomes the limitations of using single-modal information, such as ambient light interference and data drift caused by long-term posture determination, thus improving the stability and accuracy of posture determination and enhancing the robustness of the device. Furthermore, posture determination can be performed while the user is wearing the short-sleeved garment 20, improving comfort. This achieves high-precision, low-interference, low-cost, low-privacy-threat (no optical camera equipment used), and low-environmental-requirement (no complex site setup required) capture of upper body joint movements.

[0071] In some embodiments, the system further includes a control device for constructing a combination of the first neural network model and the second neural network model. See also... Figure 7 First, the control device acquires training data as the training set, which includes input data and label data. The control device 10 acquires optical image sequences of the elbow joint and non-elbow joint via optical device 50. Optical device 50 includes a camera or digital camera, etc. The optical image sequence refers to optical images ordered chronologically. After processing, the elbow joint optical image sequence can indicate the true value of the elbow joint rotation angle vector; similarly, the non-elbow joint optical image sequence can indicate the true value of the non-elbow joint rotation angle vector. The elbow joint and non-elbow joint optical image sequences can be used as label data. Simultaneously, the control device 10 retrieves the infrared optical flow vector sequence of the elbow joint and the motion information vector sequence of the non-elbow joint.

[0072] Secondly, the control device 10 uses the infrared optical flow vector sequence of the elbow joint as input data and the optical image sequence of the elbow joint as label data to train the residual network (ResNet) model. The residual network model is a deep convolutional neural network model. By introducing skip connections (residual blocks), it solves the gradient vanishing problem in deep networks, thus enabling more effective training of neural networks with many layers that need to learn complex features.

[0073] During training, the residual network model outputs the rotation angle vector of the elbow joint. Then, the control device 10 concatenates the elbow joint rotation angle vector with the motion information vector sequence of the non-elbow joints to generate a multimodal information vector sequence. The control device 10 uses the multimodal information vector sequence as input data and the optical image sequence of the non-elbow joints as labeled data to train a Long Short-Term Memory (LSTM) network model and a Multilayer Perceptron (MLP) model. The LSTM network model is a special type of recurrent neural network model that utilizes a gating mechanism to effectively capture long-term dependencies in sequential data, solving the gradient vanishing and gradient exploding problems in traditional recurrent neural networks (RNNs).

[0074] Understandably, the infrared optical flow vector sequence and multimodal information vector sequence of the elbow joint during the training phase are used as training data. The infrared optical flow vector sequence of the elbow joint generated in step S130 and the multimodal information vector sequence generated in step S150 are used as prediction objects, which can be generated during the training phase or regenerated.

[0075] The training continues until the total loss value of the combined loss function of the residual network model, the long short-term memory network model, and the multilayer perceptron model falls within a preset range. At this point, a first neural network model is constructed from the residual network model, and a second neural network model is constructed from the long short-term memory network model and the multilayer perceptron model. In other words, training of the first neural network model and the second neural network model combination is completed when the sum of the first loss value of the residual network model and the second loss value of the long short-term memory network model and the multilayer perceptron model falls within a preset range.

[0076] For example, the formula for determining the first loss value of the residual network model is as follows:

[0077]

[0078] in, This represents the first loss value. This represents the rotation angle vector of the elbow joint. The expression can be , This represents the rotation angle vector of the left elbow joint. This represents the rotation angle vector of the right elbow joint. This indicates that the rotation angle vector has six dimensions. That is, the rotation angle vector of the elbow joint. The vector of the rotation angle of the left elbow joint and the rotational angle vector of the right elbow joint The data was obtained by splicing the data together. This represents the true value of the rotation angle vector of the elbow joint; the specific expression can be found in [reference needed]. .

[0079] It is understandable that the rotation angle vector of the elbow joint during the training phase... The true value of the rotation angle vector of the elbow joint The average of the squared differences between them is the first loss value. .

[0080] For example, the formulas for determining the second loss value of the Long Short-Term Memory network model and the Multilayer Perceptron model are as follows:

[0081]

[0082] in, This represents the second loss value. This represents the rotation angle vector of the non-elbow joints during the training phase.

[0083] The expression can be, , The vector representing the rotation angle of the hip joint. The vector representing the rotation angle of the spine. This represents the rotation angle vector of the neck. This represents the vectors of the upper body joints, excluding the elbow joint. This indicates that the rotation angle vector has fifty-four dimensions. This can be understood as the rotation angle vector excluding the elbow joint. It is obtained by splicing together the data vectors of the upper body joints other than the elbow joint. This represents the true value of the rotation angle vector of the non-elbow joint.

[0084] It is understandable that the rotation angle vector of the non-elbow joints during the training phase... The true value of the rotation angle vector of the non-elbow joint The average of the squared differences between them is the second loss value. .

[0085] The formula for determining the sum of the first loss value of the residual network model and the second loss value of the long short-term memory network model and the multilayer perceptron model is as follows:

[0086]

[0087] in, This represents the total loss value. The total loss value is the first loss value of the residual network model. The second loss value compared with the Long Short-Term Memory network model and the Multilayer Perceptron model sum.

[0088] After completing the construction of the second neural network model combination and the first neural network model, the infrared optical flow vector sequence of the elbow joint retrieved in step S130 is input into the first neural network model, which outputs the rotation angle vector of the elbow joint, thereby predicting the real-time motion state of the elbow joint. Simultaneously, the multimodal information vector sequence obtained in step S150 is input into the second neural network model combination, which outputs the rotation angle vector of the non-elbow joint, thereby predicting the real-time motion state of the non-elbow joint.

[0089] Understandably, compared with other traditional machine learning algorithms, the residual network model, as a convolutional neural network model, has shown good performance in the task of processing infrared optical flow vector sequences of the elbow joint.

[0090] Understandably, in real-world scenarios, clothing may wrinkle or deform as the body moves, making it very challenging to map partial body images to human motion postures. Therefore, this application uses deep learning techniques such as residual network models, long short-term memory network models, and multilayer perceptron models to map the thermally generated image sequence of the elbow joint sensed by the thermal imaging infrared sensor and the initial motion information vector sequence of non-elbow joints sensed by the inertial measurement unit to human motion postures, thereby solving the problem of mapping body images to human motion postures.

[0091] Because the data structures and feature dimensions sensed by thermal imaging infrared sensors and inertial measurement units are different, it is difficult to directly concatenate these two types of data and input them into a neural network for prediction. Therefore, in this embodiment, a residual network model is first used to predict the rotation angle vector of the elbow joint corresponding to the infrared optical flow vector sequence of the elbow joint. Then, a combination of a long short-term memory network model and a multilayer perceptron model is used to predict the rotation angle vector of the non-elbow joint, which is obtained by concatenating the rotation angle vector of the elbow joint with the motion information vector sequence of the non-elbow joint. This solves the problem that it is impossible to directly concatenate these two types of data and simultaneously input them into a neural network for prediction.

[0092] Furthermore, if the thermal imaging infrared sensor and the inertial measurement unit directly predict the joints they are placing, they cannot fully utilize the correlation between the user's different joint movements to further improve the prediction effect. Therefore, this invention designs a multimodal feature fusion framework based on a combination of residual network model, long short-term memory network model and multilayer perceptron model, which fully utilizes the correlation between the user's different joint movements and further improves the prediction effect.

[0093] In some embodiments, please refer to Figure 8 and 9 This also includes the process by which the control device uses an attention mechanism model to optimize the initial motion information vector sequence of non-elbow joints to obtain the motion information vector sequence of non-elbow joints. That is, as... Figure 8 As shown, the method also includes the following steps:

[0094] Step S210: The control device converts the angular velocity vector into a rotational acceleration vector based on the time interval between adjacent angular velocity vectors.

[0095] The acquired initial motion information vector sequence for the non-elbow joints includes an angular velocity vector sequence and an acceleration vector sequence. For example... Figure 9 As shown, the initial motion information vector sequence includes initial motion information vectors sorted by time. The angular velocity vector sequence includes angular velocity vectors sorted by time. The acceleration vector sequence includes acceleration vectors ordered by time. That is, the inertial measurement unit 40 includes an accelerometer and a gyroscope. The accelerometer is used to sense the acceleration vector sequence, and the gyroscope is used to sense the angular velocity vector sequence.

[0096] like Figure 9 As shown, based on the angular velocity vector Obtain the rotational acceleration vector Specifically, the formula for determining the rotational acceleration vector is as follows:

[0097]

[0098] in, Represents the rotational acceleration vector. Represents the angular velocity vector. This represents the adjacent angular velocity vector. That is, the rotational acceleration vector can be determined based on the time interval between adjacent angular velocity vectors (the interval can be a unit of time) and the adjacent angular velocity vectors. In one implementation, t+1 represents the current time, and t represents the previous time.

[0099] Step S220: The control device adjusts the acceleration vector based on the proportion of the acceleration vector in the rotational acceleration vector to obtain the target acceleration vector.

[0100] like Figure 9 As shown, the rotational acceleration vector and acceleration vector The model is determined by the proportion of the input acceleration vector, and the output acceleration vector is determined by the proportion of the input acceleration vector. In the rotational acceleration vector percentage .

[0101] The acceleration vector proportion model is shown below:

[0102]

[0103] It is understandable that the second norm of the acceleration vector is... The norm of the vector with respect to the rotational acceleration vector The ratio is the proportion of the acceleration vector to the rotational acceleration vector. .

[0104] Next, the acceleration vector In acceleration vector percentage The input acceleration vector weights determine the model, and the output acceleration vector weights are... Weights of the acceleration vector The attention matrix is ​​for the attention mechanism model. The acceleration vector weights determine the model as shown below:

[0105]

[0106] in, Represents the linear layer weights. , This indicates a parameter, obtained by default. That is, it is based on the proportion of the acceleration vector in the rotational acceleration vector. Preset linear layer weights and parameters Obtain the weights of the acceleration vector It is understandable that when the human body is assumed to be a rigid body, an attention matrix can be added. This linear layer (weights of the acceleration vector) allows the residual network model, long short-term memory network model, and multilayer perceptron model to have a certain degree of freedom to fine-tune the weights during training, making the combination of the first and second neural network models more adaptable to the real world.

[0107] Next, based on the weights of the acceleration vector Adjusting the acceleration vector The target acceleration vector is obtained. .

[0108] It is understood that the embodiments of this application are based on an attention mechanism model, according to the acceleration vector. In the rotational acceleration vector percentage It determines whether the current motion is dominated by rotational acceleration caused by rotation (i.e., the larger the proportion, the less rotational acceleration is dominant). If not, it assigns a larger weight to the acceleration vector of the accelerometer in the current frame (i.e., the current moment) for regression; if so, it assigns a smaller weight.

[0109] Step S230: The control device obtains a sequence of motion information vectors for non-elbow joints based on the spliced ​​target acceleration vector and angular velocity vector.

[0110] Specifically, such as Figure 9 As shown, the target acceleration vector and angular velocity vector Data splicing Obtain the motion information vector of the non-elbow joint. Thus, based on the motion information vectors of multiple non-elbow joints, a sequence of motion information vectors for non-elbow joints is obtained.

[0111] It is understandable that the process of using an attention mechanism model to optimize the initial motion information vector sequence of non-elbow joints to obtain the motion information vector sequence of non-elbow joints is applicable to both the training and prediction stages.

[0112] It is understandable that differences in body shape among users and user movement during data acquisition cause the inertial measurement unit (IMU) to generate different displacements and record noise, leading to a decrease in the accuracy of human motion tracking. Assuming the human body is a rigid body, the IMU records translational acceleration, acceleration relative to the direction of gravity, and rotational acceleration. Of these three, the first two components are unaffected by the wearing position; only the rotational acceleration caused by rotation is sensitive to the IMU's displacement within the user's body part. Therefore, if the accelerometer's acceleration vector has a larger proportion of the gyroscope's rotational acceleration vector (i.e., a larger proportion of the accelerometer's acceleration vector), a greater weight is assigned to the acceleration vector, making the target acceleration vector larger. This weakens the gyroscope's rotational acceleration vector, reducing the impact of the IMU's displacement within the user's body part. This approach can retain most relevant information while making the combination of the first and second neural network models more robust to the IMU's offset position, thereby reducing noise from different users wearing the IMU.

[0113] Understandably, related technologies include multimodal motion monitoring schemes that integrate data from multiple sensors. These typically utilize various sensing devices such as accelerometers, gyroscopes, and magnetometers, combined with data fusion algorithms (such as Kalman filters) to estimate human posture and motion trajectory. Furthermore, they integrate data from visual sensors, pressure sensors, or depth cameras, employing complex algorithms to achieve more comprehensive motion capture. Simultaneously, deep learning techniques are used to optimize model performance to adapt to different user body types and movement styles.

[0114] However, the data fusion algorithms used in multimodal motion monitoring schemes still face challenges such as decreased tracking accuracy due to long-term monitoring data and strong environmental dependence. In this embodiment, an attention mechanism model is used to calibrate the noise of the inertial measurement unit data in real time, and a streaming light algorithm is used to calibrate the data of the thermal imaging infrared sensor in real time, thereby enhancing the robustness of the combination of the first neural network model and the second neural network model for different user wear scenarios.

[0115] Please see Figure 10 and Figure 11 The above-described embodiments will be further explained through the following application scenario. The human motion posture tracking device 100 performs the following steps:

[0116] a1: The thermal imaging infrared sensor 30 senses the thermal image sequence of the left and right elbow joints, the inertial measurement unit 40 senses the initial motion information vector sequence of the non-elbow joints, and the optical device 50 senses the optical image sequence of the elbow joints and non-elbow joints.

[0117] a2: The control device 10 converts the thermal image sequence into an infrared optical flow vector sequence according to the optical flow algorithm, and converts the initial motion information vector sequence of non-elbow joints into a motion information vector sequence of non-elbow joints according to the attention mechanism model.

[0118] a3: The control device 10 uses the infrared optical flow vector sequence of the elbow joint as input data, uses the optical image sequence of the elbow joint as label data to train the residual network model, and outputs the rotation angle vector of the elbow joint and calculates the first loss value.

[0119] a4: The control device 10 splices the rotation angle vector of the elbow joint and the motion information vector sequence of the non-elbow joint to obtain a multimodal information vector sequence;

[0120] a5: The control device 10 uses a multimodal information vector sequence as input data, uses an optical image sequence of non-elbow joints as label data to train a long short-term memory network model and a multilayer perceptron model, and outputs the rotation angle vector of the elbow joint and calculates the second loss value.

[0121] a6: When the sum of the first loss value and the second loss value is within a preset range, the control device 10 constructs a combination of a first neural network model and a second neural network model;

[0122] a7: The control device 10 acquires the real-time infrared optical flow vector sequence and the initial motion information sequence, inputs the real-time infrared optical flow vector sequence and the initial motion information sequence into the first neural network model and the second neural network model, and outputs the upper body motion posture of the human body.

[0123] Figure 12 A schematic diagram of the control device provided in this application. Figure 12 As shown, the control device 10 includes:

[0124] Wireless communication module 11, processor 12, memory 13 and bus 14;

[0125] The memory 13 is used to store the computer program code of the processor 12;

[0126] The processor 12 is configured to execute the technical solutions in any of the foregoing method embodiments by executing the computer program code.

[0127] Optionally, the memory 13 can be either standalone or integrated with the processor 12.

[0128] The memory 13 is connected to the processor 12 via the bus 14 and they communicate with each other.

[0129] Optionally, memory 13 may include random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device.

[0130] Bus 14 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus.

[0131] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0132] The control device 10 is used to execute the technical solutions provided in any of the aforementioned method embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0133] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0134] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A human motion posture tracking device, characterized in that, The human motion posture tracking device includes a short-sleeved garment and a control device. The short-sleeved garment includes two sleeves and a body. Multiple thermal imaging infrared sensors are installed on the two sleeves, and multiple inertial measurement units are installed on the body. The plurality of thermal imaging infrared sensors are used to sense the thermal image sequence of the elbow joint, and the plurality of inertial measurement units are used to sense the initial motion information vector sequence of the non-elbow joint. The control device includes a wireless communication module, which is used to input the thermal image sequence of the elbow joint and the initial motion information vector sequence of the non-elbow joints; The two sleeves include a first sleeve and a second sleeve, and the plurality of thermal imaging infrared sensing devices include a first thermal imaging infrared sensing device and a second thermal imaging infrared sensing device. The first thermal imaging infrared sensor is positioned close to the cuff of the first sleeve, and the first camera of the first thermal imaging infrared sensor is facing the first elbow joint to sense the thermal image sequence of the first elbow joint. The second thermal imaging infrared sensor is positioned near the cuff of the second sleeve, and the second camera of the second thermal imaging infrared sensor is directed toward the second elbow joint to sense a sequence of thermal images of the second elbow joint.

2. The human motion posture tracking device according to claim 1, characterized in that, The human motion posture tracking device also includes a first circuit board and a second circuit board, and a first battery and a first wireless communication module are provided on both the first circuit board and the second circuit board; The first circuit board is connected to the first thermal imaging infrared sensor and is used to power the first thermal imaging infrared sensor through the first battery and to send the thermal image sequence of the first elbow joint to the control device through the first wireless communication module. The second circuit board is connected to the second thermal imaging infrared sensor and is used to power the second thermal imaging infrared sensor through the first battery and to send the thermal image sequence of the second elbow joint to the control device through the first wireless communication module.

3. The human motion posture tracking device according to claim 1, characterized in that, The garment body includes two short-sleeve shoulder lines and two armpit coverage points. A first line is formed between the two short-sleeve shoulder lines, and a second line is formed between the two armpit coverage points. The plurality of inertial measurement units include a first inertial measurement unit and a second inertial measurement unit. The first inertial measurement unit is located at the midpoint between the first line and the second line, and the second inertial measurement unit is located at the hem of the garment body near the pelvis.

4. The human motion posture tracking device according to claim 3, characterized in that, Both the first inertial measurement unit and the second inertial measurement unit include a gyroscope and an accelerometer. The gyroscope is used to sense an angular velocity vector sequence, and the accelerometer is used to sense an acceleration vector sequence. The angular velocity vector sequence and the acceleration vector sequence form the initial motion information vector sequence.

5. The human motion posture tracking device according to claim 4, characterized in that, Both the first inertial measurement unit and the second inertial measurement unit further include a second battery and a second wireless communication module. The second battery is used to power the gyroscope and the accelerometer, and the second wireless communication module is used to send the initial motion information vector sequence to the control device.

6. A short-sleeved garment, characterized in that, The short-sleeved garment includes two sleeves and a body. Multiple thermal imaging infrared sensors are installed on the two sleeves, and multiple inertial measurement units are installed on the body. The plurality of thermal imaging infrared sensors are used to sense the thermal image sequence of the elbow joint and upload it to the control device; the plurality of inertial measurement units are used to sense the initial motion information vector sequence of non-elbow joints and upload it to the control device. The two sleeves include a first sleeve and a second sleeve, and the plurality of thermal imaging infrared sensing devices include a first thermal imaging infrared sensing device and a second thermal imaging infrared sensing device. The first thermal imaging infrared sensor is positioned close to the cuff of the first sleeve, and the camera of the first thermal imaging infrared sensor is facing the first elbow joint to sense the thermal image sequence of the first elbow joint. The second thermal imaging infrared sensor is positioned near the cuff of the second sleeve, and the camera of the second thermal imaging infrared sensor is directed toward the second elbow joint to sense a sequence of thermal images of the second elbow joint.

7. The short-sleeved garment according to claim 6, characterized in that, The garment body includes two short-sleeve shoulder lines and two armpit coverage points. A first line is formed between the two short-sleeve shoulder lines, and a second line is formed between the two armpit coverage points. The plurality of inertial measurement units include a first inertial measurement unit and a second inertial measurement unit. The first inertial measurement unit is located at the midpoint between the first line and the second line, and the second inertial measurement unit is located at the hem of the garment body near the pelvis.

Images