Human body posture prediction method and device and medium

Abstract

The invention discloses a human body posture prediction method and device and a medium. The method comprises the following steps: acquiring motion acceleration and angular velocity of each part of a human body; constructing a human body undirected space-time diagram structure: obtaining a human body undirected space-time diagram adjacent matrix based on the human body undirected space-time diagram structure; inputting the motion acceleration, the angular velocity and the human body undirected space-time diagram adjacency matrix of each part of the human body into a human body posture prediction algorithm model, and predicting a joint angle of each part of the human body; obtaining a human body posture based on the predicted joint angle of each part of the human body; by introducing a human body undirected time-space diagram structure, association is established between motion data of an inertial sensor worn by a human body and human body limb segments, the motion data and the human body limb segments are jointly used as input of a human body posture prediction algorithm model, key spatial feature information is provided for the human body posture prediction algorithm model, and the problem of complex irrelevant data interference is effectively solved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a method, device and medium for predicting human body posture. Background Art

[0002] On the one hand, the traditional human posture recognition based on inertial motion data uses the theory of human kinematics to recognize the real-time human posture, and cannot predict and estimate the human posture. On the other hand, the method of predicting human posture based on inertial motion data mainly uses convolutional neural networks, but the shortcomings of this type of network model are mainly manifested in the following aspects: 1. The design of the network model does not combine the characteristics of the human skeleton and inertial motion device; 2. The network design is not designed for the needs of time series prediction. Summary of the invention

[0003] The purpose of the present invention is to provide a human body posture prediction method, device and medium in view of the deficiencies in the prior art.

[0004] The objective of the present invention is achieved through the following technical scheme: A method for predicting human body posture, comprising:

[0005] Obtain the motion acceleration and angular velocity of various parts of the human body; the human body parts include the head and neck, spine, pelvis, left and right upper arms, left and right lower arms, left and right thighs, left and right calves, and left and right feet;

[0006] Construct an undirected spatiotemporal graph structure of the human body, and regard human body nodes as undirected spatiotemporal graph nodes. The edges of the undirected spatiotemporal graph include two types: (1) connections between graph nodes in each time frame; (2) connections between each graph node in multiple time frames. The human body nodes include cervical vertebrae, lumbar vertebrae, sacrum, left and right shoulder joints, left and right elbow joints, left and right hip joints, left and right knee joints, and left and right ankle joints. Based on the undirected spatiotemporal graph structure of the human body, obtain the adjacency matrix of the undirected spatiotemporal graph of the human body.

[0007] The motion acceleration, angular velocity and adjacency matrix of the human body's undirected space-time graph are input into the human body posture prediction algorithm model to predict the joint angles of various parts of the human body.

[0008] Obtaining human body posture based on the predicted joint angles of each part of the human body;

[0009] like Figure 1 As shown, the human posture prediction algorithm model includes a parsing module, an interactive prediction module, and an optimization module;

[0010] The analysis module is used to analyze key feature information from the motion acceleration and angular velocity of various parts of the human body;

[0011] The interaction prediction module predicts the motion state through feature interaction in time and space;

[0012] The optimization module is used to optimize the predicted motion state data sequence.

[0013] Furthermore, the acquisition of the motion acceleration and angular velocity of each part of the human body includes:

[0014] The positions where the human body wears inertial motion sensors include the head and neck, spine, pelvis, left and right upper arms, left and right lower arms, left and right thighs, left and right calves, and left and right feet; the inertial motion sensors are used to collect inertial motion data of various parts of the human body in the sensor coordinate system, and the inertial motion data includes inertial motion acceleration, angular velocity, and magnetic field strength;

[0015] By using the human body posture calibration method, the inertial motion data of each part of the human body in the sensor coordinate system is converted to the human body coordinate system to obtain the motion acceleration and angular velocity of each part of the human body.

[0016] Furthermore, the analysis module includes two integrated encoder modules; the input of the first integrated encoder module is the original input data , that is, the acceleration and angular velocity of each part of the human body; the input of the second integrated encoder module is the original input data and the output of the first integrated encoder module The residual combination of The original input data , original input data and the output of the first integrated encoder module The residuals of , the residuals of the three are combined;

[0017] The comprehensive encoder module includes a spatial position encoding module, a temporal convolutional network module, and an attention mechanism encoder module; the input of the spatial position encoding module is the original input data The input of the temporal convolutional network module is the combination of the output of the spatial position encoding module and the input of the comprehensive encoder module, and the input of the attention mechanism encoder module is the output of the temporal convolutional network module;

[0018] Among them, the temporal convolutional network module uses causal convolution and dilated convolution to process the original input data Time series tasks;

[0019] Among them, the spatial position encoding module uses absolute position encoding to generate an absolute position encoding matrix for the input data of the integrated encoder, and assigns a unique code to each position data in the input sequence;

[0020] Among them, the attention mechanism encoder module includes multiple layers of encoders, each layer of encoders includes multi-head self-attention, feedforward neural network, normalization layer and connected in sequence; among them, multi-head self-attention is composed of multiple self-attention modules, that is, the input vector is passed into n self-attention modules respectively, n outputs are obtained, and then the outputs of multiple self-attention modules are combined, passed into the feedforward neural network and normalization layer, and the final output Z is obtained.

[0021] Furthermore, the input of the interaction prediction module includes the original input data , the output of the parsing module and the adjacency matrix of the undirected space-time graph , and the output is obtained through the interactive prediction module ;

[0022] The interaction prediction module includes a spatial interaction module and a temporal interaction module;

[0023] Among them, based on the undirected space-time graph, the spatial interaction module uses the spatial segmentation strategy to spatially divide the limb segments of the human body graph structure. As input, the self-attention module is used to first solve the attention weights between the joints in each spatial domain, and then add and normalize them to obtain the inter-domain attention weights of the entire human body spatial domain. ;

[0024] At the same time, the spatial interaction module uses the original input data As input, a time window mechanism is introduced to slide on the time dimension to obtain joint information of multiple time windows. The query amount is the joint data of each time window as the comparison amount. The self-attention module is used to solve the attention weight between the original input data and the joint data of each time window, and then the sum and normalization are performed to obtain the attention weight of the entire time domain. ;

[0025] Will and The two attention weights are aggregated to extract the weight information of spatiotemporal information. The weight information is used to calculate the adjacency matrix of the undirected graph. Update to obtain the adjacency matrix that integrates spatiotemporal information ;

[0026] Finally, use the updated adjacency matrix Output from parsing module Use the Einstein summation convention to perform dimension fusion and obtain the output ;

[0027] The time interaction module As input, use dilated causal convolution and residual operations to interact in the time dimension to obtain output .

[0028] Furthermore, the spatial segmentation strategies include the following four:

[0029] Spatial segmentation strategy 1: All limb segments in the human body graph structure are considered as one, without spatial segmentation;

[0030] Spatial segmentation strategy 2: The human body structure is divided into two regions. The upper limb region includes the pelvis, L5 segment, L3 segment, T12 segment, T8 segment, head and neck, left and right hands, left and right forearms, left and right upper arms, and left and right shoulders; the lower limb region includes the pelvis, left and right thighs, left and right calves, and left and right feet.

[0031] Spatial segmentation strategy three: Divide the human body structure into three areas: upper limbs and trunk, upper limbs and arms, and lower limbs; the upper limbs and trunk include the pelvis, L5 segment, L3 segment, T12 segment, T8 segment, and head and neck; the upper limbs and arms include left and right hands, left and right forearms, left and right upper arms, left and right shoulders, and T8 segment; the lower limbs include the pelvis, left and right thighs, left and right calves, and left and right feet;

[0032] Spatial segmentation strategy 4: Divide all limb segments into five areas: upper limb trunk, upper limb left arm, upper limb right arm, lower limb left leg, lower limb right leg; upper limb trunk includes pelvis, L5 segment, L3 segment, T12 segment, T8 segment, head and neck; upper limb left arm includes left hand, left forearm, left upper arm, left shoulder, T8 segment; upper limb right arm includes right hand, right forearm, right upper arm, right shoulder, T8 segment; lower limb left leg includes pelvis, left thigh, left calf, left foot; lower limb right leg includes pelvis, right thigh, right calf, right foot.

[0033] Furthermore, the input of the optimization module is the output of the interaction prediction module ;The optimization module is composed of several single-stage optimization modules and a time series model based on the attention mechanism;

[0034] The single-stage optimization module includes a normal convolution layer, a dilated convolution layer, a batch normalization layer, and an activation function layer, which are connected in sequence; the convolution layer is used to adjust the input The feature dimension matches the number of feature maps in the network, using dilated convolutions and doubling the dilation rate in different single-stage optimization modules, and then using batch normalization and activation functions to obtain Finally, use the residual connection to input and Combine and get the output .

[0035] Furthermore, the time series model based on the attention mechanism includes an encoder module and a decoder module;

[0036] Use a recurrent neural network as an encoder, combined with the current time step input vector and the hidden state at the previous time step , transform outputs the hidden state of the current time step , the hidden layer state of the encoder Do weight distribution processing and use hidden state quantity Weight , get the context vector ; The decoder uses a recurrent neural network, using the output of the previous time step , encoder context vector and the hidden state at the previous time step , determine the hidden state of the current time step ;

[0037] Finally, based on the output of the previous time step , decoder hidden state vector , context vector , using a fully connected feedforward neural network to output the final prediction value , predict the joint angles of various parts of the human body.

[0038] The present invention also provides a human body posture prediction device, comprising:

[0039] The motion data acquisition module is used to obtain the motion acceleration and angular velocity of various parts of the human body; the human body parts include the head and neck, spine, pelvis, left and right upper arms, left and right forearms, left and right thighs, left and right calves, and left and right feet;

[0040] The joint angle prediction module constructs an undirected spatiotemporal graph structure of the human body, and regards the human body nodes as undirected spatiotemporal graph nodes. The edges of the undirected spatiotemporal graph include two types: (1) connections between graph nodes in each time frame; (2) connections between each graph node in multiple time frames; the human body nodes include cervical vertebrae, lumbar vertebrae, sacrum, left and right shoulder joints, left and right elbow joints, left and right hip joints, left and right knee joints, and left and right ankle joints; based on the undirected spatiotemporal graph structure of the human body, the adjacency matrix of the undirected spatiotemporal graph of the human body is obtained; the motion acceleration, angular velocity, and undirected spatiotemporal graph adjacency matrix of each part of the human body are input into the human posture prediction algorithm model to obtain the joint angles of each part of the human body;

[0041] The human posture prediction algorithm model includes an analysis module, an interactive prediction module, and an optimization module;

[0042] The analysis module is used to analyze key feature information from the motion acceleration and angular velocity of various parts of the human body;

[0043] The interaction prediction module predicts the motion state through feature interaction in time and space;

[0044] The optimization module is used to optimize the predicted motion state data sequence;

[0045] The posture conversion module is used to obtain the human posture based on the joint angles of various parts of the human body; it also includes obtaining ergonomic characteristic parameters based on the joint angles of various parts of the human body to evaluate and determine the potential risk of musculoskeletal injuries.

[0046] The present invention also provides a human body posture prediction device, comprising one or more processors, for implementing the above-mentioned human body posture prediction method.

[0047] The present invention also provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, it is used to implement the above-mentioned human body posture prediction method.

[0048] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0049] 1. This invention introduces the human body undirected spatiotemporal graph structure to associate the motion data of the inertial sensors worn by the human body with the human limb segments, and uses them together as the input of the human posture prediction algorithm model. Compared with traditional methods, this method provides key spatial feature information for the human posture prediction algorithm model and effectively solves the problem of interference from complex irrelevant data;

[0050] 2. The joint angle prediction module in the present invention is based on the undirected spatiotemporal graph structure of the human body. Through spatial interaction strategy and temporal interaction strategy, it simultaneously extracts and focuses on information of different spatial scales and different time spans, realizes multi-scale spatial information aggregation, and promotes the flow of complex spatiotemporal information;

[0051] 3. The posture conversion module in the present invention uses the predicted human joint angle data to predict the ergonomic characteristic parameters, realizing end-to-end ergonomic evaluation parameter prediction, and providing data support for evaluating and determining potential musculoskeletal injury risks. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0053] Figure 1 A schematic diagram of a human body posture prediction algorithm model provided by an embodiment of the present invention;

[0054] Figure 2 A schematic diagram of a parsing module structure provided by an embodiment of the present invention;

[0055] Figure 3 A schematic diagram of a cross-connection mutual detection module structure provided by an embodiment of the present invention;

[0056] Figure 4 A schematic diagram of an optimization module structure provided by an embodiment of the present invention;

[0057] Figure 5 A schematic diagram of a hardware structure provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The present invention is described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the features of the following embodiments and implementations can be combined with each other.

[0059] A human body posture prediction method of the present invention comprises the following steps:

[0060] (1) Obtaining the motion acceleration and angular velocity of various parts of the human body: The inertial motion sensors are worn on the head and neck, spine, pelvis, left and right upper arms, left and right forearms, left and right thighs, left and right calves, and left and right feet. The inertial motion data of various parts of the human body in the sensor coordinate system are collected and converted into motion data of various parts of the human body in the human coordinate system;

[0061] Specifically, the acquisition of the motion acceleration and angular velocity of each part of the human body includes:

[0062] The inertial motion sensor is used to collect inertial motion data of various parts of the human body in the sensor coordinate system, wherein the inertial motion data includes inertial motion acceleration, angular velocity and magnetic field strength;

[0063] The inertial motion data of each part of the human body in the sensor coordinate system is converted to the human body coordinate system to obtain the motion acceleration and angular velocity of each part of the human body.

[0064] Specifically, the inertial motion data collected in real time from various parts of the human body are converted into a human body coordinate system using a calibration posture method to obtain the motion acceleration and angular velocity of various parts of the human body;

[0065] In one embodiment, after the multi-node inertial motion sensor is worn by the human body, a posture calibration action needs to be completed, wherein the posture calibration action is that the wearer's arms hang naturally and are placed on both sides of the body, the legs are close together and stand upright, facing the geomagnetic north direction (determined by the magnetic field strength in the inertial motion data), and the inertial motion sensors placed on the head and neck, spine, pelvis, left and right upper arms, left and right forearms, left and right thighs, left and right calves, and left and right feet are obtained in the sensor coordinate system under the calibration posture. The acceleration and angular velocity of each part of the human body in the human body coordinate system under the calibration posture are combined to obtain the posture transformation matrix of each part of the human body in the real-time state; using the posture transformation matrix, the inertial motion data of each part of the human body in the sensor coordinate system are converted to obtain the motion acceleration and angular velocity of each part of the human body in the human body coordinate system.

[0066] (2) Construct an undirected space-time graph structure of the human body, and regard human body nodes (including cervical vertebrae, lumbar vertebrae, sacrum, left and right shoulder joints, left and right elbow joints, left and right hip joints, left and right knee joints, and left and right ankle joints) as undirected space-time graph nodes. The edges of the undirected space-time graph include two types: (1) connections between graph nodes in each time frame; (2) connections between each graph node in multiple time frames. Construct an undirected space-time graph structure of the human body to obtain an initial undirected space-time graph adjacency matrix; input the motion acceleration, angular velocity, and initial undirected space-time graph adjacency matrix of each part of the human body into the human posture prediction algorithm model to obtain the joint angles of each part of the human body; the human posture prediction algorithm model includes a parsing module, an interactive prediction module, and an optimization module;

[0067] The analysis module is used to analyze key feature information from the input inertial motion data;

[0068] The interaction prediction module predicts the motion state through feature interaction in time and space;

[0069] The optimization module is used to optimize the predicted motion state data sequence.

[0070] In one embodiment, if Figure 2 As shown, the parsing module includes two integrated encoder modules; the input of the first integrated encoder module is the original input data , that is, the acceleration and angular velocity of each part of the human body; the input of the second integrated encoder module is the original input data and the output of the first integrated encoder module The residual combination of The original input data , original input data and the output of the first integrated encoder module The residuals of The residual combination of

[0071] The comprehensive encoder module includes a spatial position encoding module, a temporal convolutional network module, and an attention mechanism encoder module; the input of the spatial position encoding module is the original input data The input of the temporal convolutional network module is the combination of the output of the spatial position encoding module and the input of the comprehensive encoder module, and the input of the attention mechanism encoder module is the output of the temporal convolutional network module;

[0072] Among them, the temporal convolutional network module uses causal convolution and dilated convolution to process the original input data time series tasks.

[0073] Among them, the spatial position encoding module uses absolute position encoding to generate an absolute position encoding matrix for the input data of the integrated encoder, and assigns a unique code to each position data in the input sequence.

[0074] Among them, the attention mechanism encoder module contains multiple layers of encoders connected in sequence, and each layer of encoder includes multi-head self-attention, feedforward neural network, and normalization layer. Multi-head self-attention is composed of multiple self-attention modules, that is, the input vector is passed into n self-attention modules respectively, and n outputs are obtained, namely Z1, Z2, Z3, ..., Zn; then the outputs of multiple self-attention modules are combined and passed into the feedforward neural network and normalization layer to obtain the final output Z of multi-head self-attention.

[0075] Two integrated encoder modules are used in the parsing module. The integrated encoder module uses residual connections as output. The residual connection in the first integrated encoder module uses the original input data. And the data processed by the spatial position encoding module, the temporal convolutional network module, and the attention mechanism encoder module , the residual connection of the second integrated encoder module uses the original input data , the output data of the first integrated encoder module , the output data of the second integrated encoder module , get the output .

[0076] In one embodiment, based on the positions of multi-node inertial motion sensors on the human body, an undirected spatiotemporal graph structure is mapped, and the cervical vertebra, lumbar vertebra, sacrum, left and right shoulder joints, left and right elbow joints, left and right hip joints, left and right knee joints, and left and right ankle joints of the human body are regarded as graph nodes. The edges of the graph include two types: (1) connections between graph nodes in each frame; and (2) connections between each graph node between multiple frames. It is the adjacency matrix of the graph structure. Each weight coefficient represents the spatial relationship between two nodes, and the diagonal node degree matrix D is used to normalize it.

[0077]

[0078]

[0079] The interactive prediction module input includes the original input data , output data of the parsing module and the initial undirected space-time graph adjacency matrix , and the output is obtained through the interactive prediction module .

[0080] like Figure 3 As shown, the interaction prediction module includes a spatial interaction module and a temporal interaction module.

[0081] Among them, based on the undirected space-time graph, the spatial interaction module uses the spatial segmentation strategy to spatially divide the limb segments of the human body graph structure. As input, the self-attention module is used to first solve the attention weights between the joints in each spatial domain separately. , and then add and normalize to obtain the inter-domain attention weights of the entire human body space domain .

[0082]

[0083] At the same time, the spatial interaction module uses the original input data As input, a time window mechanism is introduced to slide on the time dimension to obtain joint information of multiple time windows. The query amount is the joint data of each time window as the comparison amount. The self-attention module is used to solve the attention weight between the original input data and the joint data of each time window, and then the sum and normalization are performed to obtain the attention weight of the entire time domain. .

[0084] Aggregate the above two attention weights to extract the weight information of spatiotemporal information. Using this weight information, the adjacency matrix of the undirected graph is Update to obtain the adjacency matrix that integrates spatiotemporal information .

[0085]

[0086] Finally, use the updated adjacency matrix Output from parsing module Use the Einstein summation convention to perform dimension fusion and obtain the output .

[0087] The time interaction module As input, use dilated causal convolution and residual operations to interact in the time dimension to obtain output .

[0088] Among them, the spatial segmentation strategies include the following four:

[0089] Spatial segmentation strategy 1: All limb segments in the human body graph structure are considered as one without spatial segmentation.

[0090] Spatial segmentation strategy 2: The human body structure is divided into two areas. The upper limb area includes the pelvis, L5 segment, L3 segment, T12 segment, T8 segment, head and neck, left and right hands, left and right forearms, left and right upper arms, and left and right shoulders; the lower limb area includes the pelvis, left and right thighs, left and right calves, and left and right feet.

[0091] Spatial segmentation strategy three: Divide the human body structure into three areas: upper limbs and trunk, upper limbs and arms, and lower limbs. The upper limbs and trunk include the pelvis, L5 segment, L3 segment, T12 segment, T8 segment, and head and neck; the upper limbs and arms include left and right hands, left and right forearms, left and right upper arms, left and right shoulders, and T8 segment; the lower limbs include the pelvis, left and right thighs, left and right calves, and left and right feet.

[0092] Spatial segmentation strategy 4: Divide all limb segments into five regions: upper limb trunk, upper limb left arm, upper limb right arm, lower limb left leg, and lower limb right leg. The upper limb trunk includes the pelvis, L5 segment, L3 segment, T12 segment, T8 segment, and head and neck. The upper limb left arm includes the left hand, left forearm, left upper arm, left shoulder, and T8 segment. The upper limb right arm includes the right hand, right forearm, right upper arm, right shoulder, and T8 segment. The lower limb left leg includes the pelvis, left thigh, left calf, and left foot. The lower limb right leg includes the pelvis, right thigh, right calf, and right foot.

[0093] In one embodiment, the input of the optimization module is the output of the interaction prediction module .like Figure 4 As shown, the optimization module includes several single-stage optimization modules (this embodiment is illustrated with three single-stage optimization modules, but is not limited thereto) and a time series model based on the attention mechanism.

[0094]

[0095]

[0096] Each single-stage optimization module It includes convolutional layer, dilated convolutional layer, batch normalization layer and activation function layer, which are connected in sequence; the convolutional layer is used to adjust the input The feature dimension matches the number of feature maps in the network, using dilated convolutions and doubling the dilation rate in different single-stage optimization modules, and then using batch normalization and activation functions to obtain Finally, use the residual connection to input and Combine and get the output .

[0097] The previous stage input of the first single-stage optimization module for , get the output ;

[0098] The previous stage input of the second single-stage optimization module for , get the output ;

[0099] The previous stage input of the third single-stage optimization module for , get the output .

[0100] The continuous stacking uses three single-stage optimization modules, and finally uses the time series model based on the attention mechanism to complete sequence-to-sequence learning. In the time series model based on the attention mechanism, a recurrent neural network is used as an encoder, combined with the current time step input vector and the hidden state at the previous time step , transform outputs the hidden state of the current time step , the hidden layer state of the encoder Do weight distribution processing and use hidden state quantity Weight , get the context vector The decoder uses a recurrent neural network, using the output of the previous time step , the encoder context vector and the hidden state at the previous time step , determine the hidden state of the current time step ;

[0101] Finally, based on the output of the previous time step , decoder hidden state vector , context vector , using a fully connected feedforward neural network to output the final prediction value , and obtain the joint angles of the main limb segments of the human body.

[0102] (3) Obtain ergonomic characteristic parameters based on the joint angles of various parts of the human body;

[0103] Specifically, the joint angles of various parts of the human body are used to predict human posture based on ergonomic analysis to evaluate and determine the potential ergonomic state of the human body.

[0104] The present invention also provides a human body posture prediction device, comprising:

[0105] The motion data acquisition module is used to obtain the motion acceleration and angular velocity of various parts of the human body; the human body parts include the head and neck, spine, pelvis, left and right upper arms, left and right forearms, left and right thighs, left and right calves, and left and right feet;

[0106] The joint angle prediction module is used to construct the structure of the human body undirected space-time graph to obtain the adjacency matrix of the human body undirected space-time graph; the motion acceleration, angular velocity and adjacency matrix of the human body undirected space-time graph of each part of the human body are input into the human posture prediction algorithm model to predict the joint angles of each part of the human body; the structure of the human body undirected space-time graph is constructed: the human body nodes are regarded as undirected space-time graph nodes, and the edges of the undirected space-time graph include two types: (1) the connection between the graph nodes in each time frame; (2) the connection between each graph node in multiple time frames; the human body nodes include cervical vertebrae, lumbar vertebrae, sacrum, left and right shoulder joints, left and right elbow joints, left and right hip joints, left and right knee joints, and left and right ankle joints;

[0107] A posture conversion module is used to obtain ergonomic characteristic parameters based on the joint angles of various parts of the human body. An embodiment provided by the present invention is to convert the predicted joint angle data into musculoskeletal injury characteristic parameters of the wrist, elbow, shoulder, neck, back and leg, such as posture, duration and movement frequency, and evaluate and determine the potential musculoskeletal injury risk based on ergonomic methods.

[0108] It should be noted that the device embodiment shown in this embodiment matches the content of the above method embodiment, and the content of the above method embodiment can be referred to, which will not be repeated here.

[0109] Corresponding to the aforementioned embodiment of a human body posture prediction method, the present invention also provides an embodiment of a human body posture prediction device.

[0110] See also Figure 5 A human body posture prediction device provided by an embodiment of the present invention includes one or more processors for implementing a human body posture prediction method in the above embodiment.

[0111] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0112] An embodiment of a human posture prediction device of the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the internal memory and running them. From the hardware level, if Figure 5 As shown in FIG. 1 , a hardware structure diagram of a human posture prediction device of the present invention is provided in any device with data processing capability, except Figure 5 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0113] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0114] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. A person of ordinary skill in the art can understand and implement it without paying creative labor.

[0115] An embodiment of the present invention further provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, a human body posture prediction method in the above embodiment is implemented.

[0116] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of any device with data processing capability and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0117] The above embodiments are only used to illustrate the design ideas and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design ideas disclosed by the present invention are within the protection scope of the present invention.

Claims

1. A human body posture prediction method, characterized in that: include: Obtain the motion acceleration and angular velocity of each part of the human body; The human body parts include the head and neck, spine, pelvis, left and right upper arms, left and right lower arms, left and right thighs, left and right calves, and left and right feet; Constructing an undirected spatiotemporal graph structure of the human body: The human body nodes are regarded as undirected spatiotemporal graph nodes. The edges of the undirected spatiotemporal graph include two types: (1) the connection between graph nodes in each time frame; (2) the connection between each graph node in multiple time frames; the human body nodes include cervical vertebrae, lumbar vertebrae, sacrum, left and right shoulder joints, left and right elbow joints, left and right hip joints, left and right knee joints, and left and right ankle joints; Based on the structure of the human body undirected spatiotemporal graph, an adjacency matrix of the human body undirected spatiotemporal graph is obtained; The motion acceleration, angular velocity and adjacency matrix of the human body's undirected space-time graph are input into the human body posture prediction algorithm model to predict the joint angles of the human body. Obtaining human body posture based on the predicted joint angles of each part of the human body; The human posture prediction algorithm model includes an analysis module, an interactive prediction module, and an optimization module; The analysis module is used to analyze key feature information from the motion acceleration and angular velocity of various parts of the human body; The interaction prediction module predicts the motion state through feature interactions in time and space; The optimization module is used to optimize the predicted motion state data sequence.

2. A human body posture prediction method according to claim 1, characterized in that: The obtaining of the motion acceleration and angular velocity of each part of the human body includes: The positions where the human body wears inertial motion sensors include the head and neck, spine, pelvis, left and right upper arms, left and right lower arms, left and right thighs, left and right calves, and left and right feet; the inertial motion sensors are used to collect inertial motion data of various parts of the human body in the sensor coordinate system, and the inertial motion data includes inertial motion acceleration, angular velocity, and magnetic field strength; By using the human body posture calibration method, the inertial motion data of each part of the human body in the sensor coordinate system is converted to the human body coordinate system to obtain the motion acceleration and angular velocity of each part of the human body.

3. A human body posture prediction method according to claim 1, characterized in that: The analysis module includes two integrated encoder modules; the input of the first integrated encoder module is the original input data , that is, the acceleration and angular velocity of each part of the human body; the input of the second integrated encoder module is the original input data and the output of the first integrated encoder module The residual combination of The original input data , original input data and the output of the first integrated encoder module The residuals of , the residuals of the three are combined; The comprehensive encoder module includes a spatial position encoding module, a temporal convolutional network module, and an attention mechanism encoder module; the input of the spatial position encoding module is the original input data The input of the temporal convolutional network module is the combination of the output of the spatial position encoding module and the input of the comprehensive encoder module, and the input of the attention mechanism encoder module is the output of the temporal convolutional network module; Among them, the temporal convolutional network module uses causal convolution and dilated convolution to process the original input data Time series tasks; Among them, the spatial position encoding module uses absolute position encoding to generate an absolute position encoding matrix for the input data of the integrated encoder, and assigns a unique code to each position data in the input sequence; Among them, the attention mechanism encoder module includes multiple layers of encoders, each layer of encoders includes multi-head self-attention, feedforward neural network, normalization layer and connected in sequence; among them, multi-head self-attention is composed of multiple self-attention modules, that is, the input vector is passed into n self-attention modules respectively, n outputs are obtained, and then the outputs of multiple self-attention modules are combined, passed into the feedforward neural network and normalization layer, and the final output Z is obtained.

4. A human body posture prediction method according to claim 1, characterized in that: The input of the interactive prediction module includes the original input data , the output of the parsing module and the adjacency matrix of an undirected space-time graph , and the output is obtained through the interactive prediction module ; The interaction prediction module includes a spatial interaction module and a temporal interaction module; Among them, based on the undirected space-time graph, the spatial interaction module uses the spatial segmentation strategy to spatially divide the limb segments of the human body graph structure. As input, the self-attention module is used to first solve the attention weights between the joints in each spatial domain, and then add and normalize them to obtain the inter-domain attention weights of the entire human body spatial domain. ; At the same time, the spatial interaction module uses the original input data As input, a time window mechanism is introduced to slide on the time dimension to obtain joint information of multiple time windows. The query amount is the joint data of each time window as the comparison amount. The self-attention module is used to solve the attention weight between the original input data and the joint data of each time window, and then the sum and normalization are performed to obtain the attention weight of the entire time domain. ; Will and The two attention weights are aggregated to extract the weight information of spatiotemporal information. The weight information is used to calculate the adjacency matrix of the undirected graph. Update to obtain the adjacency matrix that integrates spatiotemporal information ; Finally, use the updated adjacency matrix Output from parsing module Use the Einstein summation convention to perform dimension fusion and obtain the output ; The time interaction module As input, use dilated causal convolution and residual operations to interact in the time dimension to obtain output .

5. A human body posture prediction method according to claim 4, characterized in that: There are four types of space partitioning strategies: Spatial segmentation strategy 1: All limb segments in the human body graph structure are considered as one, without spatial segmentation; Spatial segmentation strategy 2: The human body structure is divided into two regions. The upper limb region includes the pelvis, L5 segment, L3 segment, T12 segment, T8 segment, head and neck, left and right hands, left and right forearms, left and right upper arms, and left and right shoulders; the lower limb region includes the pelvis, left and right thighs, left and right calves, and left and right feet. Spatial segmentation strategy three: Divide the human body structure into three areas: upper limbs and trunk, upper limbs and arms, and lower limbs; the upper limbs and trunk include the pelvis, L5 segment, L3 segment, T12 segment, T8 segment, and head and neck; the upper limbs and arms include left and right hands, left and right forearms, left and right upper arms, left and right shoulders, and T8 segment; the lower limbs include the pelvis, left and right thighs, left and right calves, and left and right feet; Spatial segmentation strategy 4: Divide all limb segments into five areas: upper limb trunk, upper limb left arm, upper limb right arm, lower limb left leg, lower limb right leg; upper limb trunk includes pelvis, L5 segment, L3 segment, T12 segment, T8 segment, head and neck; upper limb left arm includes left hand, left forearm, left upper arm, left shoulder, T8 segment; upper limb right arm includes right hand, right forearm, right upper arm, right shoulder, T8 segment; lower limb left leg includes pelvis, left thigh, left calf, left foot; lower limb right leg includes pelvis, right thigh, right calf, right foot.

6. A human body posture prediction method according to claim 1, characterized in that: Optimizing module input is the output of the interaction prediction module ; The optimization module is composed of several single-stage optimization modules connected with a time series model based on the attention mechanism; The single-stage optimization module includes a normal convolution layer, a dilated convolution layer, a batch normalization layer, and an activation function layer, which are connected in sequence; the convolution layer is used to adjust the input The feature dimension matches the number of feature maps in the network, using dilated convolutions and doubling the dilation rate in different single-stage optimization modules, and then using batch normalization and activation functions to obtain Finally, use the residual connection to input and Combine and get the output .

7. A human body posture prediction method according to claim 6, characterized in that: The time series model based on the attention mechanism includes an encoder module and a decoder module; Use a recurrent neural network as an encoder, combined with the current time step input vector and the hidden state at the previous time step , transform outputs the hidden state of the current time step , the hidden layer state of the encoder Do weight distribution processing and use hidden state quantity Weight , get the context vector ; The decoder uses a recurrent neural network, using the output of the previous time step , encoder context vector and the hidden state at the previous time step , determine the hidden state of the current time step ; Finally, based on the output of the previous time step , decoder hidden state vector , context vector , using a fully connected feedforward neural network to output the final prediction value , predict the joint angles of various parts of the human body.

8. A human body posture prediction device, characterized in that: include: The motion data acquisition module is used to obtain the motion acceleration and angular velocity of various parts of the human body; The human body parts include the head and neck, spine, pelvis, left and right upper arms, left and right lower arms, left and right thighs, left and right calves, and left and right feet; The joint angle prediction module constructs an undirected spatiotemporal graph structure of the human body, and regards the human body nodes as undirected spatiotemporal graph nodes. The edges of the undirected spatiotemporal graph include two types: (1) connections between graph nodes in each time frame; (2) connections between each graph node in multiple time frames; the human body nodes include cervical vertebrae, lumbar vertebrae, sacrum, left and right shoulder joints, left and right elbow joints, left and right hip joints, left and right knee joints, and left and right ankle joints; based on the undirected spatiotemporal graph structure of the human body, the adjacency matrix of the undirected spatiotemporal graph of the human body is obtained; the motion acceleration, angular velocity, and undirected spatiotemporal graph adjacency matrix of each part of the human body are input into the human posture prediction algorithm model to obtain the joint angles of each part of the human body; The human posture prediction algorithm model includes an analysis module, an interactive prediction module, and an optimization module; The analysis module is used to analyze key feature information from the motion acceleration and angular velocity of various parts of the human body; The interaction prediction module predicts the motion state through feature interactions in time and space; The optimization module is used to optimize the predicted motion state data sequence; The posture conversion module is used to obtain the human posture based on the joint angles of various parts of the human body; it also includes obtaining ergonomic characteristic parameters based on the joint angles of various parts of the human body to evaluate and determine the potential risk of musculoskeletal injuries.

9. A human body posture prediction device, characterized in that: The method comprises one or more processors for implementing a human body posture prediction method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, it is used to implement a human body posture prediction method according to any one of claims 1 to 7.

Images