A human body posture estimation method and device integrating human-centered intelligence

Abstract

The present application provides a method and device for human posture estimation that integrates human-centered intelligence. The method provided in the present application includes: obtaining target visual input data and target inertial input data; constructing a human posture estimation network model; inputting the target visual input data and the target inertial input data into the human posture estimation network model to obtain visual modal spatial features and inertial modal spatial features; based on the key spatial correlation of the human body, the corresponding visual modal spatial features and inertial modal spatial features are fused to obtain spatial fusion features; the spatial fusion features are subjected to multi-scale temporal feature transformation to obtain the original global time domain features, and the original global time domain features are subjected to temporal feature extraction to obtain multimodal spatiotemporal features; the target human posture is calculated based on time information and spatial information. The method and device for human posture estimation that integrates human-centered intelligence provided in the present application can achieve accurate estimation of human posture with a relatively low amount of calculation.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to a method and device for estimating human posture by integrating human-centered intelligence. Background Art

[0002] As a key manufacturing paradigm under the framework of Industry 5.0, human-machine collaboration can effectively combine the advantages of human-machine team members to achieve efficient and reconfigurable manufacturing units. In order to cooperate with human partners to complete operational tasks, human-centered intelligent manufacturing with relevant skills such as human perception and cognition is essential for collaborative robots. Human posture estimation, as a key technology for human perception in human-centered intelligent manufacturing, can solve the parametric mathematical model of the human body from sensor data, thereby providing collaborative robots with recognizable human information, enabling them to make correct responses during the collaboration process.

[0003] At present, methods for estimating human posture include estimation methods based on visual input of optical characteristics and human body part measurement methods based on inertial sensor input. The estimation method based on visual input of optical characteristics has poor performance under extreme lighting or occlusion conditions and requires a large amount of calculation. The human body part measurement method based on inertial sensor input is greatly disturbed by the measurement environment and restricts human operations, which does not meet the application requirements of complex scenarios. Summary of the invention

[0004] In view of this, the present application provides a method for estimating human posture by integrating human-centered intelligence, the method comprising:

[0005] Acquire target visual input data and target inertial input data; the target inertial input data includes target position and speed;

[0006] Constructing a human posture estimation network model; wherein the human posture estimation network model includes a visual modality for processing image data and an inertial modality for processing inertial data;

[0007] Inputting the target visual input data and the target inertial input data into the human body posture estimation network model to obtain visual modal space features and inertial modal space features;

[0008] Based on the key spatial correlation of the human body, the corresponding visual modality space features and the inertial modality space features are fused to obtain a spatial fusion feature;

[0009] The spatial fusion feature is subjected to a multi-scale time series feature transformation to obtain an original global time domain feature, and the original global time domain feature is subjected to a time series feature extraction to obtain a multi-modal spatiotemporal feature; wherein the multi-modal spatiotemporal feature includes time information and space information;

[0010] The target human body posture is calculated based on the time information and the spatial information.

[0011] In some embodiments, the acquiring target inertial input data includes:

[0012] The quaternion of the inertial sensor data is subjected to coordinate transformation to obtain the target inertial input data aligned with the target visual input data space, wherein the coordinate transformation comprises:

[0013] ;

[0014] Among them, the is the rotation quaternion between limb segments corresponding to the inertial measurement unit, is the rotation quaternion of the inertial measurement unit measured in the inertial sensor coordinate system, is the rotation quaternion between the inertial sensor coordinate system and the global coordinate system, is the rotation quaternion between the visual sensor coordinate system and the global coordinate system.

[0015] In some embodiments, the method for acquiring the target inertial input data includes:

[0016] The speed of the inertial sensor data is subjected to coordinate transformation to obtain the target inertial input data that is spatially aligned with the target visual input data, wherein the coordinate transformation comprises:

[0017] ;

[0018] Among them, the is the speed in the visual sensor coordinate system, v is the speed of the inertial measurement unit measured by the inertial measurement unit, is the rotation matrix between the inertial sensor coordinate system and the visual sensor coordinate system.

[0019] The acceleration of the inertial sensor data is subjected to coordinate transformation to obtain the target inertial input data that is spatially aligned with the target visual input data, wherein the coordinate transformation comprises:

[0020] ;

[0021] Among them, the is the speed in the visual sensor coordinate system, a is the speed of the inertial measurement unit measured by the inertial measurement unit, is the rotation matrix between the inertial sensor coordinate system and the visual sensor coordinate system.

[0022] The above three parts of target inertial input data are spliced ​​to obtain the final target inertial input data:

[0023] ;

[0024] Among them, the is the rotation quaternion between the visual sensor coordinate system and the global coordinate system, is the speed in the visual sensor coordinate system, is the speed in the visual sensor coordinate system.

[0025] In some embodiments, performing multi-scale time series feature transformation on the spatial fusion feature to obtain the original global time domain feature, and performing time series feature extraction on the original global time domain feature to obtain the multimodal spatiotemporal feature includes:

[0026] The spatial fusion features are spliced ​​and linearly transformed to generate original time domain features;

[0027] Extracting time series features from the original time domain features based on the human body key space to obtain several groups of local time domain features;

[0028] Splicing and linearly changing a plurality of the local time domain features to obtain the original global time domain feature;

[0029] The original global time domain features are subjected to time series feature extraction to obtain multimodal spatiotemporal features; wherein the multimodal spatiotemporal features include time information and space information.

[0030] In some embodiments, based on the key spatial correlation of the human body, the corresponding visual modality space features and the inertial modality space features are fused to obtain the spatial fusion features, including:

[0031] Determine the visual modal space features and the inertial modal space features corresponding to the same human body key space, and use cross Transformer fusion to obtain initial spatial fusion features;

[0032] Using a spatial Transformer to extract semantic information from the initial spatial fusion features to obtain the spatial fusion features;

[0033] Among them, the multimodal feature fusion attention mechanism of the cross Transformer is:

[0034] ;

[0035] in is the query matrix obtained by linearly changing the visual modality space features of the g-th partition, and are the key matrix and value matrix obtained by linearly changing the inertial modal space features of the g-th partition, and crossattention is the obtained cross-modal attention matrix.

[0036] In some embodiments, the inputting the target visual input data and the target inertial input data into the human body posture estimation network model to obtain visual modal space features and inertial modal space features includes:

[0037] Processing the target visual input data through the visual modality based on a semantic graph convolutional neural network to obtain the visual modality spatial features;

[0038] The inertial modality spatial feature is obtained by grouping the target inertial input data based on the human body key space through the inertial modality.

[0039] In some embodiments, the local time domain features are concatenated and linearly changed to obtain the original global time domain features, including:

[0040] According to the different human body key spaces in each frame, the corresponding local time domain features are spliced ​​and processed through a fully connected layer to obtain the original global time domain features;

[0041] And / or, the spatial fusion features are spliced ​​and linearly transformed to generate original time domain features, including:

[0042] According to different human body key spaces, the feature vectors of the joint points of each frame in the human body key space are spliced ​​and processed through a fully connected layer to obtain the original time domain features.

[0043] In some embodiments, acquiring the target human body posture based on the multimodal spatiotemporal features includes:

[0044] Performing batch normalization and convolution operations on the multimodal spatiotemporal features to obtain relative position estimates of joints throughout the human body relative to the root joints;

[0045] Based on the relative position estimation value, the target human body posture is obtained.

[0046] In some embodiments, the step of constructing a human posture estimation network model includes:

[0047] Inputting the training sample set into an initial human posture estimation network model, and iteratively training the initial human posture estimation network model;

[0048] Inputting the verification sample set into the initial human posture estimation network model to obtain a training loss value L;

[0049] ;

[0050] Wherein, T is the length of the input time series, J is the number of joint points in each frame, and P is i，j represents the estimated relative position of the jth joint point in the i-th frame, Represents the true value of the relative position of the jth joint point in the i-th frame;

[0051] The weight corresponding to the human posture estimation network model is determined based on the training loss value.

[0052] The second aspect of the present invention provides a human posture estimation device integrating human-centered intelligence manufacturing, the device comprising an acquisition module, a construction module, a processing module and a fusion module; wherein;

[0053] The acquisition module acquires target visual input data and target inertial input data; the target inertial input data includes target position, target speed and target acceleration;

[0054] The construction module is used to construct a human posture estimation network model; wherein the human posture estimation network model includes a visual modality for processing image data and an inertial modality for processing inertial data;

[0055] The processing module is used to input the target visual input data and the target inertial input data into the human body posture estimation network model to obtain visual modal space features and inertial modal space features;

[0056] The fusion module is used to fuse the corresponding visual modality space features and the inertial modality space features based on the key spatial correlation of the human body to obtain a spatial fusion feature;

[0057] The processing module is used to perform multi-scale time series feature transformation on the spatial fusion feature to obtain the original global time domain feature, and perform time series feature extraction on the original global time domain feature to obtain multi-modal spatiotemporal feature; wherein the multi-modal spatiotemporal feature includes time information and space information;

[0058] The processing module is used to calculate the target human body posture based on the time information and the spatial information.

[0059] The human body posture estimation method and device provided by the present application, which integrates human-centered intelligence, considers the image data and spatial data of the human body at the same time through the human body posture estimation network, so as to fully combine the information coupling mechanism between multi-modal sensors, retain the advantages of visual data without cumulative errors and without affecting human body movement, while inertial data is not affected by illumination and occlusion. In addition, the present invention fully considers the influence of the spatial correlation of human joint kinematics on the temporal correlation of human posture by fusing the visual modal space features and the inertial modal space features, and then performs splicing processing and linear change processing, thereby ensuring the reliability of the fusion of multiple modal data. The present invention completes the estimation of human body posture based on visual information and inertial information, while avoiding increasing excessive calculation amount and memory consumption, ensuring the reliability and accuracy of the human body posture finally obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 A flowchart of the first embodiment of the human body posture estimation method integrating human-centered intelligence provided in this application;

[0061] Figure 2 A schematic diagram of a first embodiment of a human body key space provided in this application;

[0062] Figure 3 A flowchart of Embodiment 2 of the method for estimating human posture by integrating human-centered intelligence provided in this application;

[0063] Figure 4 A flowchart of Embodiment 3 of the method for estimating human posture by integrating human-centered intelligence provided in this application;

[0064] Figure 5 A hardware structure diagram of a human body posture estimation device integrated with human-based intelligence manufacturing in this application;

[0065] Figure 6 This is a structural diagram of the first embodiment of the human body posture estimation device integrating human-centered intelligence provided in this application. DETAILED DESCRIPTION

[0066] Here, exemplary embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application.

[0067] The terms used in this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in this article refers to and includes any or all possible combinations of one or more associated listed items.

[0068] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0069] The present application provides a method and device for human posture estimation that integrates human-centered intelligence manufacturing, so as to reliably and accurately estimate human posture.

[0070] The human body posture estimation method and device integrated with human-centered intelligent manufacturing provided in the present application first process multiple frames of target image data to obtain target visual input data, and process inertial sensor data to obtain target inertial input data, and then construct a human body posture estimation network model; wherein the human body posture estimation network model includes a visual modality for processing image data and an inertial modality for processing inertial data, and then the target visual input data and the target inertial input data are input into the human body posture estimation network model to obtain visual modality space features and inertial modality space features, and then based on different human body key spaces, the corresponding visual modality space features and inertial modality space features are fused to obtain spatial fusion features, and then the spatial fusion features are spliced ​​and linearly changed to obtain original global time domain features, and the original global time domain features are extracted for time series features to obtain multimodal spatiotemporal features; wherein the multimodal spatiotemporal features contain time information and space information, and finally the target human body posture is obtained based on the multimodal spatiotemporal features. In this way, the image data and spatial data of the human body are considered simultaneously through the human body posture estimation network, so as to fully combine the information coupling mechanism between multi-modal sensors, retain the advantages of visual data without cumulative error and without affecting human body movement, and combine the advantages of inertial data without being affected by illumination and occlusion. Furthermore, by fusing the visual modal space features and the inertial modal space features, and then performing splicing and linear change processing, the influence of the spatial correlation of human joint kinematics on the temporal correlation of human posture is fully considered, and the reliability of the fusion of multiple modal data is guaranteed. In this way, the estimation of human posture can be completed based on visual information and inertial information, ensuring the reliability and accuracy of the human posture finally obtained.

[0071] Specific embodiments are given below to introduce the technical solution of the present application in detail.

[0072] Figure 1 This is a flowchart of the first embodiment of the human posture estimation method integrating human-centered intelligence provided by this application. Please refer to Figure 1 , the method provided in this embodiment may include:

[0073] S101, obtaining target visual input data and target inertial input data; the target inertial input data includes target position and speed.

[0074] Specifically, the target image data refers to a series of image frames continuously captured from the visual sensor, which contain visual information of the person being measured at different time points. In specific implementation, the specific device for capturing the target image data is determined according to actual needs, and is not limited in this embodiment. For example, in one embodiment, multiple frames of target image data can be acquired through a camera. For another example, multiple frames of target image data can be acquired through a depth capture device.

[0075] Furthermore, the target visual input data is visual feature data extracted from the target image data and used for posture estimation. In specific implementation, the target image data may be preprocessed to obtain the target visual input data.

[0076] Furthermore, the inertial sensor data is the movement information of the limbs of the person to be tested captured by the inertial sensor, and the inertial sensor data reflects the dynamic changes of the target object. In specific implementation, the specific device for obtaining the inertial sensor data is determined according to actual needs, and this is not limited in this embodiment. For example, in one embodiment, the inertial sensor data can be obtained by an inertial sensor. For another example, the inertial sensor data can be obtained by an accelerometer and a gyroscope.

[0077] Furthermore, the target inertial input data is inertial feature data extracted from the inertial sensor data and used for posture estimation. In specific implementation, the inertial sensor data can be preprocessed to obtain the target inertial input data.

[0078] In specific implementation, the target inertia input data includes the target position of the person to be tested, which reflects the spatial coordinates of different limbs of the person to be tested. At the same time, the target inertia input data also includes the speed data of different limbs of the person to be tested in motion.

[0079] S102, constructing a human posture estimation network model; wherein the human posture estimation network model includes a visual modality for processing image data and an inertial modality for processing inertial data.

[0080] Specifically, the human body posture estimation network model is used to estimate the joint position and posture of the human body through input data. It should be noted that the human body posture estimation network model can process multiple types of input data and output a parameterized representation of the human body posture.

[0081] Furthermore, image data is data related to images, and inertial data is data related to inertia. In specific implementation, for example, in one embodiment, the target visual input data can be determined as image data, and the target inertial input data can be determined as inertial data.

[0082] Furthermore, the visual modality in the human posture estimation network model is used to specifically process image data, and the inertial modality is used to specifically process inertial data. In specific implementation, the visual modality and inertial data can be constructed for image data and inertial data respectively when constructing the human posture estimation network model.

[0083] S103, inputting the target visual input data and the target inertial input data into the human body posture estimation network model to obtain visual modal space features and inertial modal space features.

[0084] In specific implementation, in this step, the target visual input data can be processed based on the visual modality in the human posture estimation network model to obtain the corresponding visual modality space features; and the target inertial input data can be processed based on the inertial modality in the human posture estimation network model to obtain the inertial modality space features.

[0085] For example, a semantic graph convolutional neural network is used to extract spatial feature information of visual modality input data to form visual modality spatial features.

[0086] For another example, a spatial Transformer is used to extract spatial features of inertial modal input data located in different human limb segments in groups to form inertial modal spatial features.

[0087] S104: Based on the key spatial correlation of the human body, the corresponding visual modal space features and the inertial modal space features are fused to obtain spatial fusion features.

[0088] Specifically, the human body key space is a joint area divided according to the human body anatomical structure. In specific implementation, the specific joints and specific parts included in the human body key space are determined according to actual needs, and this embodiment is not limited to this.

[0089] Figure 2 This is a schematic diagram of the first embodiment of the human body key space provided by this application. Please refer to Figure 2The key space of the human body consists of five parts, namely the trunk (the trunk includes the hip, spine, neck, and head), the right upper limb (the right upper limb includes the hip, spine, right upper arm, right forearm, and right wrist), the left upper limb (the left upper limb includes the hip, spine, left upper arm, left forearm, and left wrist), the right lower limb (the right lower limb includes the hip, right thigh, right calf, and right ankle) and the left lower limb (the left lower limb includes the hip, left thigh, left calf, and left ankle); it should be noted that the limb segment is represented by the line connecting two adjacent joints.

[0090] It should be noted that the inertial sensor data acquired in the same human body key space may be determined as the same group of data.

[0091] Furthermore, the visual modal space features and the inertial modal space features may be processed based on a traditional feature fusion method, or a feature fusion method based on a neural network may be used to process the visual modal space features and the inertial modal space features. This is not limited in the present embodiment.

[0092] When this step is implemented specifically, the spatial fusion features obtained by fusing the visual modality spatial features and the inertial modality spatial features include information from the target visual input data and the target inertial input data.

[0093] S105. Perform a multi-scale time series feature transformation on the spatial fusion feature to obtain an original global time domain feature, perform time series feature extraction on the original global time domain feature to obtain a multimodal spatiotemporal feature; wherein the multimodal spatiotemporal feature includes time information and space information.

[0094] Specifically, the spatial fusion features can be integrated through multi-scale temporal feature transformation. At the same time, the temporal features and spatial features of the spatial fusion features are made more obvious through multi-scale temporal feature transformation processing, which facilitates the extraction of temporal features. In specific implementation, the multi-scale temporal feature transformation can be realized based on splicing and linear transformation.

[0095] Furthermore, the spatial fusion features corresponding to different human body key spaces can be connected into a vector data through splicing. Through linear change processing, the spliced ​​data can be projected and transformed to extract high-level features.

[0096] Furthermore, the multimodal spatiotemporal features include time information and space information, and the multimodal spatiotemporal features can be used to describe the changes of the person to be tested in the spatiotemporal dimensions.

[0097] It should be noted that the original global time domain features can be extracted by traditional time series feature extraction methods, or by neural network time series feature extraction methods, which are not limited in this embodiment. In specific implementation, for example, in one embodiment, the original global time domain time series feature extraction can be completed by methods such as recursive neural networks or long short-term memory networks.

[0098] S106: Calculate the target human body posture based on the time information and the spatial information.

[0099] Specifically, the target human body posture is the posture corresponding to the person to be measured that is finally determined. The positions and angles of the joints of the person to be measured can be understood based on the target human body posture to complete the human body posture estimation.

[0100] In specific implementation, since the target inertial input data contains the target position and speed information of the limbs and joints of the person to be tested, the human body posture information that can reflect the target position of the limbs and joints of the person to be tested is finally calculated. In other words, the multimodal spatiotemporal features describe the changes in human body posture in time and space. Therefore, the corresponding target human body posture can be directly determined based on the time information and space information.

[0101] A specific embodiment is given below to introduce in detail the process of calculating the target human body posture based on time information and space information:

[0102] (1) Performing batch normalization and convolution operations on the multimodal spatiotemporal features to obtain output features.

[0103] Specifically, batch normalization can be used to standardize data such as the mean and variance in multimodal data, and then convolution operations are used to extract features of the batch normalized data to obtain output features.

[0104] (2) Acquire the coordinates of the human body joint points based on the output features to obtain the target human body posture.

[0105] In specific implementation, in this step, the output features can be mapped to the coordinates of each joint point of the person to be tested through a regression model or a fully connected layer, and then the converted coordinates of the joint points of the human body are determined as the target human body posture.

[0106] The human body posture estimation method integrated with human-centered intelligent manufacturing provided in this embodiment first processes multiple frames of target image data to obtain target visual input data, and processes inertial sensor data to obtain target inertial input data, and then constructs a human body posture estimation network model; wherein the human body posture estimation network model includes a visual modality for processing image data and an inertial modality for processing inertial data, and then the target visual input data and the target inertial input data are input into the human body posture estimation network model to obtain visual modality space features and inertial modality space features, and then based on different human body key spaces, the corresponding visual modality space features and inertial modality space features are fused to obtain spatial fusion features, and then the spatial fusion features are spliced ​​and linearly changed to obtain original global time domain features, and the original global time domain features are extracted for time series features to obtain multimodal spatiotemporal features; wherein the multimodal spatiotemporal features include time information and space information, and finally the target human body posture is obtained based on the multimodal spatiotemporal features. In this way, the image data and spatial data of the human body are considered simultaneously through the human body posture estimation network, so as to fully combine the information coupling mechanism between multi-modal sensors, retain the advantages of visual data without cumulative error and without affecting human body movement, and combine the advantages of inertial data without being affected by illumination and occlusion. Furthermore, by fusing the visual modal space features and the inertial modal space features, and then performing splicing and linear change processing, the influence of the spatial correlation of human joint kinematics on the temporal correlation of human posture is fully considered, and the reliability of the fusion of multiple modal data is guaranteed. In this way, the estimation of human posture can be completed based on visual information and inertial information, ensuring the reliability and accuracy of the human posture finally obtained.

[0107] Figure 3 This is a flow chart of the second embodiment of the human posture estimation method integrating human-centered intelligence provided by this application. Please refer to Figure 3 , the method provided in this embodiment may include:

[0108] S301, process multiple frames of target image data to obtain target visual input data.

[0109] In specific implementation, the target data images of all frames can be processed by a preset image processing tool to obtain a two-dimensional sequence of human joint points, and pixel coordinates can be normalized to obtain target visual input data.

[0110] S302: Perform coordinate transformation on the inertial sensor data to obtain the target inertial input data that is spatially aligned with the target visual input data.

[0111] Specifically, by performing coordinate transformation on the inertial sensor data, the target inertial input data can be aligned with the target visual input data, so that the multimodal data can be jointly analyzed and processed in the same coordinate system, ensuring the reliability of the data.

[0112] In some embodiments, the quaternion of the inertial sensor data is subjected to coordinate transformation to obtain the target inertial input data aligned with the target visual input data space, the coordinate transformation comprising:

[0113] ;

[0114] Among them, the is the rotation quaternion between limb segments corresponding to the inertial measurement unit, is the rotation quaternion of the inertial measurement unit measured in the inertial sensor coordinate system, is the rotation quaternion between the inertial sensor coordinate system and the global coordinate system, is the rotation quaternion between the visual sensor coordinate system and the global coordinate system.

[0115] In specific implementation, the rotation quaternion includes a real part and three imaginary parts, and the rotation axis and angle information are represented by the rotation quaternion.

[0116] It should be noted that the four components of the rotation quaternion together define the axis and angle of rotation. The rotation quaternion can smoothly represent the posture changes of limb information in continuous time.

[0117] In some embodiments, the velocity of the inertial sensor data is subjected to coordinate transformation to obtain the target inertial input data spatially aligned with the target visual input data, the coordinate transformation comprising:

[0118] ;

[0119] Among them, the is the speed in the visual sensor coordinate system, v is the speed of the inertial measurement unit measured by the inertial measurement unit, is the rotation matrix between the inertial sensor coordinate system and the visual sensor coordinate system.

[0120] The acceleration of the inertial sensor data is subjected to coordinate transformation to obtain the target inertial input data that is spatially aligned with the target visual input data, wherein the coordinate transformation comprises:

[0121] ;

[0122] Among them, the is the speed in the visual sensor coordinate system, a is the speed of the inertial measurement unit measured by the inertial measurement unit, is the rotation matrix between the inertial sensor coordinate system and the visual sensor coordinate system;

[0123] The above three parts of target inertial input data are spliced ​​to obtain the final target inertial input data:

[0124] ;

[0125] Among them, the is the rotation quaternion between the visual sensor coordinate system and the global coordinate system, is the speed in the visual sensor coordinate system, is the speed in the visual sensor coordinate system.

[0126] In the specific implementation, the speed information of the inertial sensor is measured, and then the rotation matrix between the inertial sensor and the visual sensor coordinate system is calculated based on the speed data of the inertial sensor. The rotation matrix describes the direction and angle difference between the two coordinate systems.

[0127] The human body posture estimation method integrating human-centered intelligence provided in this embodiment compares and analyzes the data of the visual sensor and the inertial sensor in the same coordinate system, eliminating the data error caused by the different coordinate systems and improving the overall performance and accuracy of the visual sensor and the inertial sensor.

[0128] S303, constructing a human posture estimation network model; wherein the human posture estimation network model includes a visual modality for processing image data and an inertial modality for processing inertial data.

[0129] For the specific implementation process of this step, please refer to the relevant description in the above embodiment, which will not be repeated here.

[0130] S304: Process the target visual input data through the visual modality-based semantic graph convolutional neural network to obtain the visual modality spatial features.

[0131] Specifically, after processing the semantic information in the target visual input data, the semantic convolutional neural network can extract and understand the relationship and semantic structure between the joints of the person to be tested in the image.

[0132] In specific implementation, the target visual input data is extracted and processed by a semantic graph convolutional neural network, which can effectively capture the semantic information of the image and the visual features related to the human body posture from the target visual input data.

[0133] In another embodiment of the present invention, the spatial information of human body posture contained in different modalities is first extracted through a dual-stream network structure, and then the visual and inertial modal features are fused in the spatial dimension. Through the cross-modal attention mechanism, the visual features of human joints can focus on the relevant inertial modal features, and the introduction of the limb segment partitioning strategy can effectively prevent the excessive adjustment of the inertial modal features.

[0134] S305 , extracting spatial features of the target inertial input data by grouping based on the human body key space using the inertial modality to obtain the inertial modal spatial features.

[0135] In specific implementation, in this step, the target inertial input data is grouped according to the key space of the human body, and then the spatial features of the data of each group are extracted to obtain the inertial modal space features of each partition.

[0136] Furthermore, the target inertial input data contains the target position and speed information of the person to be tested. After grouping and extracting spatial features through the key space of the human body, the obtained inertial modal space features reflect the target position of the limbs and joints of the person to be tested. In other words, the position information of the limbs and joints of the person to be tested can be determined through the inertial space features.

[0137] S306: Determine the visual modal space features and the inertial modal space features corresponding to the same human body key space, and fuse them to obtain an initial spatial fusion feature.

[0138] Specifically, the visual modal space features and the inertial modal space features are divided into zones according to the human body key space, and the features belonging to the same zone are fused to obtain the initial spatial fusion features. For example, in one embodiment, in combination with the above embodiment, the visual modal space features and the inertial modal space features belonging to the left upper limb are fused to obtain the initial spatial fusion features corresponding to the left upper limb.

[0139] In specific implementation, according to the spatial correlation of human joint kinematics, a cross-mutual sensor can be used to fuse the visual modal space features and the inertial modal space features in different partitions to output the initial spatial fusion features.

[0140] For example, a spatial Transformer is used to extract spatial features of the inertial modal input data located in different human limb segments in groups to form inertial modal spatial features.

[0141] For example, based on the spatial correlation of human joint kinematics, a cross Transformer is used to perform multimodal feature fusion of visual modal space features and inertial modal space features in different limb segment partitions to output the initial spatial fusion features.

[0142] The attention mechanism can be determined using the following formula:

[0143] ;

[0144] Among them, Q v g is the query matrix obtained by linearly changing the visual modality space features of the g-th partition, K g imu With V g imu are the key matrix and value matrix obtained by linearly changing the inertial modal space features of the g-th partition, and crossattention is the obtained cross-modal attention matrix.

[0145] S307: extract high-level semantic information from the initial spatial fusion features to obtain the spatial fusion features.

[0146] Specifically, the high-level semantic information in the initial spatial fusion features can be extracted based on a traditional high-level semantic information extraction method, or the high-level semantic information in the initial spatial fusion features can be extracted by a high-level semantic information extraction method of a neural network. For example, the high-level semantic information of the initial spatial fusion features is extracted again based on a spatial transformer, and the spatial fusion features are output. This is not limited in this embodiment.

[0147] In the specific implementation, higher-level and more abstract semantic information is extracted from the initial spatial fusion features to obtain spatial fusion features. In this way, the essential characteristics of the data can be better represented based on the spatial fusion features.

[0148] S308: performing concatenation and linear transformation processing on the spatial fusion features to generate original time domain features.

[0149] Specifically, due to the kinematic correlation of human joints (i.e., differences in the temporal motion characteristics of different limb segment partitions), it is necessary to first perform temporal feature extraction in different human key spaces, and then perform temporal feature extraction on a full-body scale.

[0150] In the specific implementation, a possible implementation method is given below to introduce in detail the process of obtaining the original time domain features:

[0151] (1) According to different human body key spaces, the feature vectors of the joint points of each frame in the human body key space are spliced ​​and processed through a fully connected layer to obtain the original time domain features;

[0152] In a specific implementation, for example, in one embodiment, in combination with the above embodiment, the feature vectors of the joint points of all frames in the left upper limb are concatenated and then processed through a fully connected layer to obtain the original time domain features.

[0153] In this way, the spatial information of human posture contained in different modalities is extracted, and then the visual and inertial modal features are fused in the spatial dimension. Through the cross-modal attention mechanism, the visual features of human joints can focus on the relevant inertial modal features. Due to the introduction of the limb segment partitioning strategy, the over-adjustment of the inertial modal features can be effectively prevented.

[0154] S309: extracting time series features from the original time domain features based on the human body key space to obtain a first preset number of groups of local time domain features.

[0155] In specific implementation, the original time series features may be extracted based on a traditional time series feature extraction method, or may be extracted based on a neural network time series feature extraction method, which is not limited in this embodiment.

[0156] Furthermore, the specific number of the first preset number is set according to actual needs, and is not limited in this embodiment. For example, the first preset number can be set to be the same as the number of key spaces of the human body.

[0157] S310. Perform time series feature extraction on the original time domain features based on the human body key space to obtain a first preset number of groups of local time domain features.

[0158] In the specific implementation, a possible line of sight method is given below to introduce in detail the process of obtaining the original time domain features:

[0159] (1) According to the different human body key spaces in each frame, the corresponding local time domain features are spliced ​​and processed through a fully connected layer to obtain the original global time domain features;

[0160] In a specific implementation, for example, in one embodiment, in combination with the above embodiment, the feature vectors of the joint points corresponding to the left upper limb in a frame are first spliced, then processed through a fully connected layer, and then the joint points of other areas such as the right upper limb are processed in turn, and finally the processing in the frame data is completed. After all the frame data are processed in turn, the original time domain features are obtained.

[0161] S311. Perform time series feature extraction on the original global time domain features to obtain multimodal spatiotemporal features; wherein the multimodal spatiotemporal features include time information and space information.

[0162] For the specific implementation process of this step, please refer to the relevant description in the above embodiment, which will not be repeated here.

[0163] S312: Acquire the target human body posture based on the multimodal spatiotemporal features.

[0164] For the specific implementation process of this step, please refer to the relevant description in the above embodiment, which will not be repeated here.

[0165] The human body posture estimation method integrated with human-centered intelligent manufacturing provided in this embodiment first processes multiple frames of target image data to obtain target visual input data, and processes inertial sensor data to obtain target inertial input data, and then constructs a human body posture estimation network model; wherein the human body posture estimation network model includes a visual modality for processing image data and an inertial modality for processing inertial data, and then the target visual input data and the target inertial input data are input into the human body posture estimation network model to obtain visual modality space features and inertial modality space features, and then based on different human body key spaces, the corresponding visual modality space features and inertial modality space features are fused to obtain spatial fusion features, and then the spatial fusion features are spliced ​​and linearly changed to obtain original global time domain features, and the original global time domain features are extracted for time series features to obtain multimodal spatiotemporal features; wherein the multimodal spatiotemporal features include time information and space information, and finally the target human body posture is obtained based on the multimodal spatiotemporal features. In this way, by effectively utilizing the human posture information in the human-machine collaboration process without increasing the amount of calculation and memory consumption, the spatial information and temporal information of the human posture are better modeled by considering the characteristics and coupling of multimodal data describing human motion and considering the spatial correlation of human joint kinematics to build a network model, and the position information of the whole body joints is output in real time. Combined with the preset feedback control strategy, the robot is given the ability to recognize human intentions. Furthermore, by introducing inertial information, when occlusion occurs, the average joint position error after rigid alignment is reduced, which can ensure the accuracy of human posture estimation and the effect of human posture estimation.

[0166] Figure 4 This is a flowchart of the third embodiment of the human posture estimation method integrating human-centered intelligence provided by this application. Please refer to Figure 4 Based on the above embodiment, the step of constructing a human posture estimation network model includes:

[0167] S401: Input a training sample set into an initial human posture estimation network model, and perform a second preset number of iterative training on the initial human posture estimation network model.

[0168] Specifically, the initial human body posture estimation network model can be constructed based on a traditional human body posture estimation network model, which is not limited in this embodiment.

[0169] Furthermore, the specific value of the second preset number is set according to actual needs, and is not set in this embodiment.

[0170] It should be noted that the parameters such as the ratio of the training set and the validation set of the network model, the number of iterations, and the learning rate can be set according to the movement characteristics of the human body and the characteristics of the human-machine collaboration scene. For example, in one embodiment, the ratio can be between 4:1 and 5:1; the number of iterations can be in the range of [50,150].

[0171] In this step, the training set is input into the initial human posture estimation network model for iterative training, the joint estimation value position error obtained by iterative training of the initial human posture estimation network model is obtained, and the parameters of the initial human posture estimation network model are continuously updated.

[0172] S402: Input the verification sample set into the initial human posture estimation network model to obtain a training loss value.

[0173] When this step is implemented, the verification sample set is input into the initial human posture estimation network model to obtain the training loss value.

[0174] A specific embodiment is given below to introduce the process of obtaining the training loss value in detail:

[0175] (1) Calculating the training loss value based on a first formula; wherein the first formula is:

[0176] ;

[0177] Where T is the length of the input time series, J is the number of joint points in each frame, and P i，j represents the estimated relative position of the jth joint point in the i-th frame, Represents the true relative position value of the j-th joint point in the i-th frame.

[0178] S403: Determine a weight corresponding to the human posture estimation network model based on the training loss value.

[0179] In specific implementation, the training loss values ​​can be sorted in order from large to small, and then the minimum training loss value is determined as the weight corresponding to the human posture estimation network model.

[0180] It should be noted that after completing the human body posture estimation, the target visual input data and the target inertial input data can be divided into new training sample sets and verification sample sets according to the ratio of the training sample sets and the verification sample sets, so as to train the human body posture estimation network model again according to the new training sample sets and verification sample sets to complete the update of the human body posture estimation network model.

[0181] It is understandable that the human posture estimation network model can be used to monitor the movement status of workers in assembly scenarios. In the human-machine collaborative assembly scenario, synchronized multimodal data is input into the trained human posture estimation network in real time to obtain the relative spatial position of the joints of the whole body of the human body in real time, detect the movement status of the human body, and identify the human operation intention. Considering the complexity and diversity of the assembled product structure, a series of robot feedback control strategies are pre-set according to the general characteristics of the product assembly scenario to meet the interaction needs of the operator.

[0182] The human body posture estimation method integrated with human-centered intelligent manufacturing provided in this embodiment first inputs the training sample set into the initial human body posture estimation network model, performs a second preset number of iterative training on the initial human body posture estimation network model, and then inputs the verification sample set into the initial human body posture estimation network model to obtain the training loss value, and finally determines the weight corresponding to the human body posture estimation network model based on the training loss value. In this way, the effect of the human body posture estimation network model training can be guaranteed, the robustness of the human body posture estimation network model can be improved, and the accuracy of human body posture estimation can be guaranteed.

[0183] The present invention also provides a specific embodiment of human posture estimation integrating human-centered intelligence.

[0184] When extracting single-modal spatial features, the visual modality and the inertial modality are both mapped to 64 dimensions; the time domain Transformer modules described in step S222 and step S224 both include 2 layers of Transformer encoders; the network iteration learning rate is 0.0005, the learning rate decay is 0.99, and the learning rate decays to 0.5 every 10 epochs, the batch size is 256, and the number of iterations is 130.

[0185] In order to prove the advantages of the human posture estimation method and system based on the fusion of visual and inertial information described in the present invention over other human posture estimation models, the common models are used to solve the posture of the same batch of data. Model 1 fuses visual and inertial information through recurrent neural network, dual coordinate strategy and hidden state feedback mechanism; Model 2 uses residual model attention network to adaptively fuse visual and inertial modal features; Model 3 extracts the spatial features of visual data through semantic graph convolution network, and designs sensor information deformation layer to combine inertial sensor information with visual information; Model 4 uses cross Transformeer to fuse the inertial sensor feature sequence with the visual feature map of the center frame.

[0186] This application measures the human body posture estimation accuracy of each model through the Mean Per Joint Position Error (MPJPE). It measures the average Euclidean distance between the predicted joint position and the actual joint position. The smaller the Mean Per Joint Position Error, the closer the model's estimated value of the human body posture is to the actual value, that is, the system accuracy is high. The Mean Per Joint Position Error is an important indicator to measure the quality of the model. Table 1 shows the MPJPE of existing commonly used models and the model of the present invention, where the MPJPE value of the model represents the Mean Per Joint Position Error.

[0187] Table 1 MPJPE of existing commonly used models and the model of the present invention

[0188]

[0189] It can be seen from Table 1 that, compared with the existing human posture estimation algorithm based on the fusion of a small number of visual and inertial sensors, the average joint position error MPJPE of the present invention is reduced by 6.3-15.4 mm.

[0190] In order to prove that the method and system for human posture estimation based on the fusion of vision and inertial information described in the present invention have occlusion robustness, a model variant of the present invention that does not contain inertial information is used to perform posture solution on the same batch of data.

[0191] This application measures the occlusion robustness of the model by the Procrustes-aligned Mean PerJoint Position Error (P-MPJPE) and its growth after rigid alignment. Table 2 is the occlusion robustness analysis of the model of the present invention.

[0192] Table 2 Occlusion robustness analysis of the model of the present invention

[0193] method Unobstructed case P-MPJPE P-MPJPE with occlusion Error increment This embodiment model 26.87 31.57 4.70 This embodiment model variant 39.23 46.97 7.74

[0194] Compared with the variant of the present model without the introduction of inertial information, in the same scene, compared with the variant of the neural network of the present invention without inertial information input, when occlusion occurs, the increment of the average joint position error P-MPJPE after rigid alignment decreases from 7.74 mm to 4.70 mm, a decrease of 39.3%.

[0195] The human body posture estimation method and system based on the fusion of visual and inertial information provided by the present invention have technical effects that are significantly better than existing commonly used models and can effectively cope with scenes with occlusion.

[0196] In summary, the present invention effectively utilizes human posture information in the process of human-machine collaboration without increasing too much computing power and memory consumption. It constructs a network model by considering and analyzing the characteristics of multimodal data describing human motion and its coupling and considering the spatial correlation of human joint kinematics to better model the spatial information and timing information of human posture. It outputs the solved whole-body joint position information in real time, and combines the preset feedback control strategy to give the robot the ability to recognize human intentions.

[0197] Corresponding to the aforementioned embodiment of a human body posture estimation method integrating human-based intelligence, the present application also provides an embodiment of a human body posture estimation device integrating human-based intelligence.

[0198] The present application provides an embodiment of a human posture estimation device that integrates human-based intelligence and can be applied to a human posture estimation device that integrates human-based intelligence and can be implemented in a manner ... Figure 5 As shown, it is a hardware structure diagram of the human body posture estimation device integrated with human-based intelligence manufacturing in this application, except Figure 5 In addition to the processor, memory, network interface, and non-volatile memory shown, the human body posture estimation device integrating human-based intelligence in which the device in the embodiment is located may also include other hardware according to the actual function of the human body posture estimation device integrating human-based intelligence, which will not be described in detail.

[0199] Figure 6 This is a schematic diagram of the structure of the first embodiment of the human posture estimation device integrating human-oriented intelligence provided by this application. Figure 6 , the device provided in this embodiment includes an acquisition module 610, a construction module 620, a processing module 630 and a fusion module 640; wherein;

[0200] The acquisition module 610 is used to process multiple frames of target image data to acquire target visual input data; process inertial sensor data to acquire target inertial input data;

[0201] The construction module 620 is used to construct a human posture estimation network model; wherein the human posture estimation network model includes a visual modality for processing image data and an inertial modality for processing inertial data;

[0202] The processing module 630 is used to input the target visual input data and the target inertial input data into the human posture estimation network model to obtain visual modal space features and inertial modal space features;

[0203] The fusion module 640 is used to fuse the corresponding visual modality space features and the inertial modality space features based on different human body key spaces to obtain spatial fusion features;

[0204] The processing module 630 is further used to perform splicing and linear change processing on the spatial fusion features to obtain original global time domain features, and perform time series feature extraction on the original global time domain features to obtain multimodal spatiotemporal features; wherein the multimodal spatiotemporal features include time information and space information;

[0205] The processing module 630 is further configured to acquire a target human body posture based on the multimodal spatiotemporal features.

[0206] The device of this embodiment can be used to perform Figure 1 The steps, specific implementation principles and implementation processes of the method embodiment shown are similar and will not be repeated here.

[0207] Optionally, the processing module 630 is specifically used to perform splicing and linear change processing on the spatial fusion features to generate original time domain features;

[0208] The processing module 630 is further specifically configured to extract time series features from the original time domain features based on the human body key space to obtain a first preset number of groups of local time domain features; wherein the first preset number is the number of partitions of the human body key space;

[0209] The processing module 630 is further specifically configured to perform concatenation and linear change processing on the local time domain features to obtain the original global time domain features.

[0210] Optionally, the fusion module 640 is specifically used to determine the visual modality space feature and the inertial modality space feature corresponding to the same human body key space, and fuse them to obtain an initial spatial fusion feature;

[0211] The fusion module 640 is further specifically configured to extract high-level semantic information from the initial spatial fusion features to obtain the spatial fusion features.

[0212] Optionally, the processing module 630 is further specifically configured to process the target visual input data through the visual modality based semantic graph convolutional neural network to obtain the visual modality spatial features;

[0213] The processing module 630 is further specifically configured to extract the target inertial input data by grouping the inertial modality based on the human body key space to obtain the inertial modality spatial feature.

[0214] Optionally, the acquisition module 610 is specifically configured to perform coordinate transformation on the inertial sensor data to obtain the target inertial input data that is spatially aligned with the target visual input data.

[0215] Optionally, the processing module 630 is further specifically configured to splice the corresponding local time domain features according to the different human body key spaces in each frame and process them through a fully connected layer to obtain the original global time domain features;

[0216] The processing module 630 is further specifically used to splice the feature vectors of the joint points of each frame in the human body key space according to different human body key spaces, and then process them through a fully connected layer to obtain the original time domain features;

[0217] Optionally, the processing module 630 is further specifically configured to perform batch normalization processing and convolution operation on the multimodal spatiotemporal features to obtain output features;

[0218] The processing module 630 is further specifically configured to obtain the coordinates of the human body joint points based on the output features to obtain the target human body posture.

[0219] Optionally, the construction module 620 is specifically used to input the training sample set into the initial human posture estimation network model, and perform a second preset number of iterative training on the initial human posture estimation network model;

[0220] The construction module 620 is further specifically used to input the verification sample set into the initial human posture estimation network model to obtain a training loss value;

[0221] The construction module 620 is further specifically used to determine the weight corresponding to the human posture estimation network model based on the training loss value.

[0222] Optionally, the training loss value is calculated based on a first formula; wherein the first formula is:

[0223] ;

[0224] Where T is the length of the input time series, J is the number of joint points in each frame, and P i，j represents the estimated relative position of the jth joint point in the i-th frame, Represents the true relative position value of the j-th joint point in the i-th frame.

[0225] Please continue to refer to Figure 5 The present application also provides a human posture estimation device that integrates human-centered intelligence, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of any one of the methods provided in the first aspect of the present application are implemented.

[0226] The present application also provides a computer-readable storage medium having a computer program stored thereon, and when the program is executed by a processor, the steps of any one of the methods provided in the present application are implemented.

[0227] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0228] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0229] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A human posture estimation method integrating human-oriented intelligence manufacturing, characterized in that: The method comprises: Acquire target visual input data and target inertial input data; the target inertial input data includes target position and speed; Acquiring the target inertial input data, including: The quaternion of the inertial sensor data is subjected to coordinate transformation to obtain the target inertial input data aligned with the target visual input data space, wherein the coordinate transformation comprises: ； Among them, the is the rotation quaternion between limb segments corresponding to the inertial measurement unit, is the rotation quaternion of the inertial measurement unit measured in the inertial sensor coordinate system, is the rotation quaternion between the inertial sensor coordinate system and the global coordinate system, is the rotation quaternion between the visual sensor coordinate system and the global coordinate system; The method for acquiring the target inertial input data comprises: The speed of the inertial sensor data is subjected to coordinate transformation to obtain the target inertial input data that is spatially aligned with the target visual input data, wherein the coordinate transformation comprises: ； Among them, the is the speed in the visual sensor coordinate system, v is the speed of the inertial measurement unit measured by the inertial measurement unit, is the rotation matrix between the inertial sensor coordinate system and the visual sensor coordinate system; The acceleration of the inertial sensor data is subjected to coordinate transformation to obtain the target inertial input data that is spatially aligned with the target visual input data, wherein the coordinate transformation comprises: ； Among them, the is the acceleration in the visual sensor coordinate system, a is the acceleration of the inertial measurement unit measured by the inertial measurement unit, is the rotation matrix between the inertial sensor coordinate system and the visual sensor coordinate system; The three parts of target inertial input data obtained , and After splicing, the final target inertial input data is obtained: ； Among them, the is the rotation quaternion between the visual sensor coordinate system and the global coordinate system, is the speed in the visual sensor coordinate system, is the acceleration in the visual sensor coordinate system; Constructing a human posture estimation network model; wherein the human posture estimation network model includes a visual modality for processing image data and an inertial modality for processing inertial data; Inputting the target visual input data and the target inertial input data into the human body posture estimation network model to obtain visual modal space features and inertial modal space features; Based on the key spatial correlation of the human body, the corresponding visual modality space features and the inertial modality space features are fused to obtain a spatial fusion feature; The method of fusing the corresponding visual modality space features and the inertial modality space features based on the key spatial correlation of the human body to obtain a spatial fusion feature includes: Determine the visual modal space features and the inertial modal space features corresponding to the same human body key space, and use cross Transformer fusion to obtain initial spatial fusion features; Using a spatial Transformer to extract semantic information from the initial spatial fusion features to obtain the spatial fusion features; Among them, the multimodal feature fusion attention mechanism of the cross Transformer is: ； in is the query matrix obtained by linearly changing the visual modality space features of the g-th partition, and are the key matrix and value matrix obtained by linearly changing the inertial modal space features of the g-th partition, and crossattention is the obtained cross-modal attention matrix; The spatial fusion feature is subjected to a multi-scale time series feature transformation to obtain an original global time domain feature, and the original global time domain feature is subjected to a time series feature extraction to obtain a multi-modal spatiotemporal feature; wherein the multi-modal spatiotemporal feature includes time information and space information; The target human body posture is calculated based on the time information and the spatial information.

2. The method according to claim 1, characterized in that The step of performing multi-scale time series feature transformation on the spatial fusion feature to obtain the original global time domain feature, and performing time series feature extraction on the original global time domain feature to obtain the multi-modal spatiotemporal feature includes: The spatial fusion features are spliced ​​and linearly transformed to generate original time domain features; Extracting time series features from the original time domain features based on the human body key space to obtain several groups of local time domain features; Splicing and linearly changing a plurality of the local time domain features to obtain the original global time domain feature; The original global time domain features are subjected to time series feature extraction to obtain multimodal spatiotemporal features; wherein the multimodal spatiotemporal features include time information and space information.

3. The method according to claim 1, characterized in that The step of inputting the target visual input data and the target inertial input data into the human body posture estimation network model to obtain visual modal space features and inertial modal space features includes: Processing the target visual input data through the visual modality based on a semantic graph convolutional neural network to obtain the visual modality spatial features; The inertial modality spatial feature is obtained by grouping the target inertial input data based on the human body key space through the inertial modality.

4. The method according to claim 2, characterized in that: The step of concatenating and linearly changing the plurality of local time domain features to obtain the original global time domain feature includes: According to the different human body key spaces in each frame, the corresponding local time domain features are spliced ​​and processed through a fully connected layer to obtain the original global time domain features; And / or, the spatial fusion features are spliced ​​and linearly transformed to generate original time domain features, including: According to different human body key spaces, the feature vectors of the joint points of each frame in the human body key space are spliced ​​and processed through a fully connected layer to obtain the original time domain features.

5. The method according to claim 1, characterized in that The calculating the target human body posture based on the time information and the spatial information includes: Performing batch normalization and convolution operations on the multimodal spatiotemporal features to obtain relative position estimates of joints throughout the human body relative to the root joints; Based on the relative position estimation value, the target human body posture is obtained.

6. The method according to claim 1, characterized in that The method of constructing a human body posture estimation network model comprises: Inputting the training sample set into an initial human posture estimation network model, and iteratively training the initial human posture estimation network model; Inputting the verification sample set into the initial human posture estimation network model to obtain a training loss value L; ； Wherein, t is the length of the input time series, J is the number of joint points in each frame, and P i，j represents the estimated relative position of the jth joint point in the i-th frame, Represents the true value of the relative position of the jth joint point in the i-th frame; The weight corresponding to the human posture estimation network model is determined based on the training loss value.

7. A human posture estimation device integrating human-oriented intelligence, characterized in that: The device comprises an acquisition module, a construction module, a processing module and a fusion module; wherein, The acquisition module acquires target visual input data and target inertial input data; the target inertial input data includes target position and speed; Acquiring the target inertial input data, including: The quaternion of the inertial sensor data is subjected to coordinate transformation to obtain the target inertial input data aligned with the target visual input data space, wherein the coordinate transformation comprises: ； Among them, the is the rotation quaternion between limb segments corresponding to the inertial measurement unit, is the rotation quaternion of the inertial measurement unit measured in the inertial sensor coordinate system, is the rotation quaternion between the inertial sensor coordinate system and the global coordinate system, is the rotation quaternion between the visual sensor coordinate system and the global coordinate system; The method for acquiring the target inertial input data comprises: The speed of the inertial sensor data is subjected to coordinate transformation to obtain the target inertial input data that is spatially aligned with the target visual input data, wherein the coordinate transformation comprises: ； Among them, the is the speed in the visual sensor coordinate system, v is the speed of the inertial measurement unit measured by the inertial measurement unit, is the rotation matrix between the inertial sensor coordinate system and the visual sensor coordinate system; The acceleration of the inertial sensor data is subjected to coordinate transformation to obtain the target inertial input data that is spatially aligned with the target visual input data, wherein the coordinate transformation comprises: ； Among them, the is the acceleration in the visual sensor coordinate system, a is the acceleration of the inertial measurement unit measured by the inertial measurement unit, is the rotation matrix between the inertial sensor coordinate system and the visual sensor coordinate system; The three parts of target inertial input data obtained , and After splicing, the final target inertial input data is obtained: ； Among them, the is the rotation quaternion between the visual sensor coordinate system and the global coordinate system, is the speed in the visual sensor coordinate system, is the acceleration in the visual sensor coordinate system; The construction module is used to construct a human posture estimation network model; wherein the human posture estimation network model includes a visual modality for processing image data and an inertial modality for processing inertial data; The processing module is used to input the target visual input data and the target inertial input data into the human body posture estimation network model to obtain visual modal space features and inertial modal space features; The fusion module is used to fuse the corresponding visual modality space features and the inertial modality space features based on the key spatial correlation of the human body to obtain a spatial fusion feature; The method of fusing the corresponding visual modality space features and the inertial modality space features based on the key spatial correlation of the human body to obtain a spatial fusion feature includes: Determine the visual modal space features and the inertial modal space features corresponding to the same human body key space, and use cross Transformer fusion to obtain initial spatial fusion features; Using a spatial Transformer to extract semantic information from the initial spatial fusion features to obtain the spatial fusion features; Among them, the multimodal feature fusion attention mechanism of the cross Transformer is: ； in is the query matrix obtained by linearly changing the visual modality space features of the g-th partition, and are the key matrix and value matrix obtained by linearly changing the inertial modal space features of the g-th partition, and crossattention is the obtained cross-modal attention matrix; The processing module is used to perform multi-scale time series feature transformation on the spatial fusion feature to obtain the original global time domain feature, and perform time series feature extraction on the original global time domain feature to obtain multi-modal spatiotemporal feature; wherein the multi-modal spatiotemporal feature includes time information and space information; The processing module is used to calculate the target human body posture based on the time information and the spatial information.

Images