Method and apparatus for determining three-dimensional human pose

Abstract

The application provides a three-dimensional human posture determination method and device, and relates to the technical field of computers. The method comprises the following steps: obtaining original measurement data collected by an inertial measurement unit sensor, an original point cloud data set collected by a millimeter wave radar, and original image data collected by a camera; using a kinematics regression decoder to decode a feature vector obtained by performing feature extraction on the original measurement data, and combining a human kinematics model to generate a first point cloud data set; performing feature extraction and processing on the original point cloud data set and the original image data to determine a second point cloud data set; generating a third point cloud data set according to the first point cloud data set and the second point cloud data set; performing voxelization processing on the third point cloud data set to generate a three-dimensional feature map based on a voxel tensor; and processing feature parameters in a heat map corresponding to the three-dimensional feature map to determine a three-dimensional human posture, thereby improving the accuracy and reliability of three-dimensional human posture determination.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a method and device for determining three-dimensional human posture. BACKGROUND

[0002] Physical fitness test, as an important means to assess the physical fitness and health status of the people, can greatly promote individual health management and improve the overall health level of the society. At present, the data recording in large-scale physical fitness test of colleges and athletes still generally adopts manual counting method, which is not only inefficient, but also difficult to accurately capture the action details. Therefore, it is of great significance to analyze the human key node action in physical fitness test through intelligent technology, so as to evaluate the action standardization of the tester and improve the intelligent level of the test. Thanks to the rapid development of artificial intelligence technology, the technology based on human key node analysis has been widely applied in the fields related to sports science, such as standard sports guidance and unmanned test.

[0003] In related technologies, intelligent image recognition algorithm is usually used for posture analysis, but this kind of method highly depends on the quality of image and video, and is easily affected by factors such as light change, tester clothing and action occlusion, which often leads to poor imaging quality and weak color features, and finally makes the human posture analysis result not accurate enough. Therefore, how to improve the accuracy of human posture estimation is very important. SUMMARY

[0004] The present application provides a method and device for determining three-dimensional human posture.

[0005] According to a first aspect of the present application, a method for determining three-dimensional human posture is provided, which comprises:

[0006] Obtaining original measurement data collected by an inertial measurement unit sensor, original point cloud data set collected by a millimeter wave radar, and original image data collected by a camera;

[0007] Decoding the feature vector obtained by feature extraction on the original measurement data by using a kinematics regression decoder, and generating a first point cloud data set in combination with a human kinematics model;

[0008] Performing feature extraction and processing on the original point cloud data set and the original image data to determine a second point cloud data set;

[0009] Generating a third point cloud data set according to the first point cloud data set and the second point cloud data set;

[0010] Performing voxelization processing on the third point cloud data set to generate a three-dimensional feature map based on voxel tensor;

[0011] The feature parameters in the heat map corresponding to the three-dimensional feature map are processed to determine a three-dimensional human body posture.

[0012] Optionally, the feature vectors obtained after the feature extraction on the original measurement data are decoded by using a kinematics regression decoder, and a first point cloud data set is generated in combination with a human body kinematics model, including:

[0013] The original measurement data are subjected to feature extraction to determine a first feature vector;

[0014] A difference vector between two consecutive frames in the first feature vector is determined;

[0015] The first feature vector and the difference vector are fused to generate a second feature vector;

[0016] The second feature vector is decoded by using a kinematics regression decoder to determine a rotation parameter;

[0017] An initial three-dimensional human body posture model is generated according to an initial body shape parameter, a joint tree length of a human body kinematics model and the rotation parameter;

[0018] Three-dimensional coordinates of a vertex in the initial three-dimensional human body posture model are mapped to generate a first point cloud data set.

[0019] Optionally, the feature extraction and processing on the original point cloud data set and the original image data are performed to determine a second point cloud data set, including:

[0020] The original point cloud data set is processed to determine an initial depth map;

[0021] The initial depth map is subjected to feature extraction to generate a depth feature vector;

[0022] The original image data are subjected to feature extraction to determine a corresponding semantic feature vector;

[0023] The depth feature vector and the semantic feature vector are fused to generate a second feature vector;

[0024] The second feature vector is processed to determine a target depth map;

[0025] The target depth map is processed to generate a second point cloud data set.

[0026] Optionally, the feature extraction on the initial depth map to generate a depth feature vector includes:

[0027] An initial depth value in the initial depth map is subjected to normalization processing to determine a first depth value;

[0028] filling a missing region in the initial depth map according to the first depth value to determine a second depth value corresponding to the missing region;

[0029] processing the first depth value and the second depth value to generate a depth feature vector.

[0030] Optionally, the generating a third point cloud dataset according to the first point cloud dataset and the second point cloud dataset comprises:

[0031] converting the first point cloud dataset and the second point cloud dataset to generate a fourth point cloud dataset and a fifth point cloud dataset corresponding thereto respectively;

[0032] determining, from the fourth point cloud dataset, a reference point corresponding to each fifth point cloud data in the fifth point cloud dataset respectively;

[0033] updating a confidence weight of each fifth point cloud data based on a coordinate of the reference point;

[0034] generating a third point cloud dataset according to the confidence weight of each fifth point cloud data and the fourth point cloud dataset.

[0035] Optionally, the updating a confidence weight of each fifth point cloud data based on a coordinate of the reference point comprises:

[0036] determining a distance value between each fifth point cloud data and the corresponding reference point according to a coordinate of each fifth point cloud data and a coordinate of the reference point, and updating the confidence weight of the fifth point cloud data whose distance value is greater than a first threshold value;

[0037] and / or,

[0038] determining a cosine value of an included angle between a first normal vector of a region to which each fifth point cloud data belongs and a second normal vector of a region to which the corresponding reference point belongs, and updating the confidence weight of the fifth point cloud data whose cosine value is less than a second threshold value.

[0039] Optionally, the voxelizing the third point cloud dataset to generate a three-dimensional feature map based on a voxel tensor comprises:

[0040] determining a three-dimensional cubic bounding box according to a coordinate of each third point cloud data in the third point cloud dataset;

[0041] dividing the three-dimensional cubic bounding box into a plurality of voxel grids;

[0042] generating a voxel tensor according to a positional relationship between each voxel grid and each third point cloud data.

[0043] The voxel tensor is used to extract features to generate a three-dimensional feature map.

[0044] Optionally, processing the feature parameters in the heatmap corresponding to the three-dimensional feature map to determine the three-dimensional human pose includes:

[0045] The three-dimensional feature map is input into the target network model, and after processing by the target network model, a heat map of each target joint is generated.

[0046] The location of the maximum thermal value in the thermal map of each target joint is determined as the three-dimensional coordinate of the corresponding target joint;

[0047] The three-dimensional coordinates of each target joint are fused to determine the three-dimensional human posture.

[0048] According to a second aspect of this application, a three-dimensional human posture determination device is provided, comprising:

[0049] The acquisition module is used to acquire raw measurement data collected by the inertial measurement unit sensor, raw point cloud dataset collected by the millimeter-wave radar, and raw image data collected by the camera.

[0050] The first generation module is used to decode the feature vector obtained after feature extraction from the original measurement data using a kinematic regression decoder, and combine it with a human kinematic model to generate a first point cloud dataset.

[0051] The first determining module is used to extract and process features from the original point cloud dataset and the original image data to determine the second point cloud dataset.

[0052] The second generation module is used to generate a third point cloud dataset based on the first point cloud dataset and the second point cloud dataset;

[0053] The third generation module is used to perform voxelization processing on the third point cloud dataset in order to generate a three-dimensional feature map based on the voxel tensor.

[0054] The second determining module is used to process the feature parameters in the heat map corresponding to the three-dimensional feature map in order to determine the three-dimensional human posture.

[0055] Optionally, the first generation module includes:

[0056] The first determining unit is used to extract features from the original measurement data to determine a first feature vector.

[0057] The second determining unit is used to determine the difference vector between two consecutive frames in the first feature vector;

[0058] a fusion unit, configured to fuse the first feature vector and the difference vector to generate a second feature vector;

[0059] a decoding processing unit, configured to decode the second feature vector by using a kinematic regression decoder to determine a rotation parameter;

[0060] a third determining unit, configured to generate an initial three-dimensional human pose model according to an initial body shape parameter of a human kinematic model, a joint tree length and the rotation parameter;

[0061] The third determining unit is further configured to perform mapping processing on three-dimensional coordinates of vertices in the initial three-dimensional human pose model to generate a first point cloud data set.

[0062] Optionally, the first determining module comprises:

[0063] a fourth determining unit, configured to process the original point cloud data set to determine an initial depth map;

[0064] a first generating unit, configured to perform feature extraction on the initial depth map to generate a depth feature vector;

[0065] a fifth determining unit, configured to perform feature extraction on the original image data to determine a corresponding semantic feature vector;

[0066] a second generating unit, configured to perform fusion processing on the depth feature vector and the semantic feature vector to generate a second feature vector;

[0067] a sixth determining unit, configured to process the second feature vector to determine a target depth map;

[0068] a third generating unit, configured to process the target depth map to generate a second point cloud data set.

[0069] Optionally, the first generating unit is specifically configured to:

[0070] perform normalization processing on an initial depth value in the initial depth map to determine a first depth value;

[0071] fill a missing region in the initial depth map according to the first depth value to determine a second depth value corresponding to the missing region;

[0072] perform processing on the first depth value and the second depth value to generate a depth feature vector.

[0073] Optionally, the second generating module comprises:

[0074] a fourth generating unit, configured to perform conversion processing on the first point cloud dataset and the second point cloud dataset to generate a fourth point cloud dataset and a fifth point cloud dataset respectively;

[0075] a seventh determining unit, configured to determine, from the fourth point cloud dataset, a reference point corresponding to each fifth point cloud data in the fifth point cloud dataset respectively;

[0076] an updating unit, configured to update a confidence weight of each fifth point cloud data based on a coordinate of the reference point;

[0077] a fifth generating unit, configured to generate a third point cloud dataset according to the confidence weight of each fifth point cloud data and the fourth point cloud dataset.

[0078] Optionally, the updating unit is specifically configured to:

[0079] determine a distance value between each fifth point cloud data and the corresponding reference point according to a coordinate of each fifth point cloud data and a coordinate of the reference point, and update the confidence weight of the fifth point cloud data whose distance value is greater than a first threshold value;

[0080] and / or,

[0081] determine a cosine value of an included angle between a first normal vector of a region to which each fifth point cloud data belongs and a second normal vector of a region to which the corresponding reference point belongs, and update the confidence weight of the fifth point cloud data whose cosine value is less than a second threshold value.

[0082] Optionally, the third generating module is configured to:

[0083] determine a three-dimensional cubic bounding box according to a coordinate of each third point cloud data in the third point cloud dataset;

[0084] divide the three-dimensional cubic bounding box into a plurality of voxel grids;

[0085] generate a voxel tensor according to a positional relationship between each voxel grid and each third point cloud data;

[0086] perform feature extraction on the voxel tensor to generate a three-dimensional feature map.

[0087] Optionally, the second determining module is configured to:

[0088] input the three-dimensional feature map into a target network model to generate a heat map of each target joint through processing of the target network model;

[0089] determine a position where a maximum heat value in the heat map of each target joint is located as a three-dimensional coordinate of the corresponding target joint.

[0090] The three-dimensional coordinates of each target joint are fused to determine the three-dimensional human posture.

[0091] According to a third aspect of this application, an electronic device is provided, comprising: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, implements any of the above-described methods for determining a three-dimensional human posture.

[0092] According to a fourth aspect of this application, a computer-readable storage medium is provided, on which computer program instructions are stored, which, when executed by a processor, implement any of the above-described methods for determining a three-dimensional human posture.

[0093] In summary, the method and apparatus for determining three-dimensional human posture provided in this application have at least the following beneficial effects: First, the original measurement data collected by the inertial measurement unit sensor, the original point cloud dataset collected by the millimeter-wave radar, and the original image data collected by the camera are acquired. Then, the feature vectors obtained after feature extraction from the original measurement data are decoded using a kinematic regression decoder, and a first point cloud dataset is generated by combining it with a human kinematic model. Next, the original point cloud dataset and the original image data are subjected to feature extraction and processing to determine a second point cloud dataset. Based on the first and second point cloud datasets, a third point cloud dataset is generated. Then, the third point cloud dataset is voxelized to generate a three-dimensional feature map based on voxel tensors. Finally, the feature parameters in the heatmap corresponding to the three-dimensional feature map are processed to determine the three-dimensional human posture. Therefore, by processing the original measurement data, the original point cloud dataset, and the original image data, a corresponding point cloud dataset is generated. Then, by processing each point cloud dataset, a three-dimensional feature map is generated, thereby determining the three-dimensional human pose. In other words, through multimodal data fusion and processing, the determined three-dimensional human pose can be made more accurate and reliable, thus improving the accuracy and reliability of the three-dimensional human pose determination. Attached Figure Description

[0094] To more clearly illustrate the specific embodiments of this application or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0095] Figure 1 A flowchart illustrating a method for determining a three-dimensional human pose, provided for an embodiment of this application;

[0096] Figure 2Another flowchart of a method for determining a three-dimensional human pose provided by an embodiment of the present application;

[0097] Figure 3 Another flowchart of a method for determining a three-dimensional human pose provided by an embodiment of the present application;

[0098] Figure 4 Another flowchart of a method for determining a three-dimensional human pose provided by an embodiment of the present application;

[0099] Figure 5 A schematic diagram of generating a second point cloud data set provided by an embodiment of the present application;

[0100] Figure 6 A schematic diagram of generating a third point cloud data set provided by an embodiment of the present application;

[0101] Figure 7 A structural diagram of a device for determining a three-dimensional human pose provided by an embodiment of the present application;

[0102] Figure 8 A structural diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0103] In order to make the above and other features and advantages of the present application clearer, the present application will be further described below with reference to the drawings. It should be understood that the specific embodiments given herein are for illustrative purposes only and are not limiting.

[0104] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. It will be apparent, however, to one skilled in the art that the present application can be practiced without the specific details. In other instances, well-known steps or operations are not described in detail in order to avoid obscuring the present application.

[0105] The method for determining a three-dimensional human pose provided by an embodiment of the present application can be executed by a device for determining a three-dimensional human pose provided by an embodiment of the present application, which can be configured in an electronic device.

[0106] It should be noted that the method for determining a three-dimensional human pose provided by the present application can be applied to any scene requiring human pose estimation, such as physical fitness testing, personalized fitness, rehabilitation training guidance, etc., which is not limited by the present application.

[0107] Reference Figure 1 The present application provides a method for determining a three-dimensional human pose, which comprises:

[0108] In step 101, original measurement data collected by an inertial measurement unit sensor, original point cloud data set collected by a millimeter wave radar, and original image data collected by a camera are obtained.

[0109] The inertial measurement unit (IMU) sensor can be multiple, one of which can be fixed to the center of the human body as a reference unit, and the remaining inertial measurement unit sensors can be fixed to other joint parts of the human body, such as the head, hands, legs, etc., for collecting motion information of multiple joints of the human body to obtain original measurement data. The original measurement data can include acceleration, angular velocity, and direction, etc. In addition, the inertial measurement unit sensor can be any type or any specification of IMU, which is not limited in the present application.

[0110] In addition, the millimeter wave radar can be a single-view millimeter wave radar, which can collect spatial information of the human body to form the original point cloud data set.

[0111] In addition, the original image data can be human body image information collected by a camera, which can be a single-view camera, or a double-view stereo camera, etc., which is not limited in the present application.

[0112] In step 102, the feature vector obtained by feature extraction of the original measurement data is decoded by using a kinematics regression decoder, and a first point cloud data set is generated in combination with a human body kinematics model.

[0113] After obtaining the original measurement data collected by the inertial measurement unit sensor, the feature extraction can be performed to obtain the feature vector, and then the decoding processing is performed by using the kinematics regression decoder, and the first point cloud data set is generated in combination with the human body kinematics model.

[0114] Optionally, the original measurement data can be first feature extracted to determine a first feature vector, and then a difference vector between two consecutive frames in the first feature vector is further determined, and the first feature vector and the difference vector are fused to generate a second feature vector, and then the decoding processing is performed on the second feature vector by using the kinematics regression decoder to determine a rotation parameter, and then an initial three-dimensional human body posture model can be generated according to the initial body shape parameter, the joint tree length, and the rotation parameter of the human body kinematics model, and then the three-dimensional coordinates of the vertices in the initial three-dimensional human body posture model are mapped to generate the first point cloud data set.

[0115] The original measurement data can be extracted by any desirable manner to determine the first feature vector. For example, a long short term memory (LSTM) recurrent neural network can be used as a feature extractor, and the original measurement data is input into the LSTM to be processed by the LSTM to output the corresponding first feature vector. Alternatively, a CNN or an RNN can be used as a feature extractor to determine the corresponding first feature vector from the original measurement data, and the like. The application does not limit this.

[0116] Optionally, to further improve the accuracy and reliability of data processing, the original measurement data can be filtered by a Kalman filter, a complementary filter, or the like, and then the processed original measurement data is used as input data of the LSTM. The input data is in time sequence and includes normalized direction and acceleration of the leaf joint relative to the root. After processing by the LSTM, the first feature vector is output, and the like. The application does not limit this.

[0117] It can be understood that the first feature vector can sufficiently retain absolute information of the current pose, and the difference vector can effectively highlight the change trend of the instantaneous speed, acceleration, and angular velocity. After the two are fused, the second feature vector can represent a more accurate motion trend on the basis of retaining the current pose.

[0118] Optionally, to further capture the motion trend, not only the absolute measurement value of the current frame in the original measurement data can be encoded, but also a difference vector between two consecutive frames and an absolute position feature representing the difference vector can be determined. Then, the first feature vector, the difference vector, and the absolute position feature can be combined to generate the second feature vector, thereby further enhancing the representation ability of the feature to the motion speed and direction and providing more abundant information for subsequent feature extraction.

[0119] Optionally, the second feature vector can be further fused with a position embedding vector. The position embedding vector can be understood as order information of the second feature vector, and can effectively represent the order and relative position relationship of the second feature vector in the time dimension. Therefore, the second feature vector can be fused with the position embedding vector, so that the second feature vector effectively fuses the time sequence context and order information perception. Therefore, the accuracy and reliability are effectively guaranteed when the second feature vector is processed subsequently.

[0120] Optionally, the first feature vector, the difference vector and the position embedding vector can also be collectively taken as input data of a time encoder, so as to generate a second feature vector through processing of the time encoder. The time encoder can include a two-layer bidirectional gated recurrent unit (GRU), and the position embedding vector can be used to refine the time feature to generate a second feature vector of T*2048, and the like, which are not limited in the present application.

[0121] It can be understood that after the second feature vector is determined, the second feature vector can be further decoded by using a kinematics regression decoder to determine the rotation parameter.

[0122] The kinematics regression decoder can be any decoder that can determine the rotation parameter, such as a multi-layer perception regressor or a graph convolution network, and the like, which are not limited in the present application.

[0123] In addition, the rotation parameter can be various, such as an axis-angle type or a quaternion, and the like, which are not limited in the present application.

[0124] It can be understood that the kinematics regression decoder provided in the embodiments of the present application can be based on the human motion tree structure, and can adopt a bottom-up regression strategy, such as from a child node to a parent node, to process the second feature vector, and then determine the corresponding rotation parameter joint by joint, thereby further improving the accuracy and reliability of the rotation parameter.

[0125] In addition, the human kinematics model can be any predefined model, such as a skinned multi-person linear model (SMPL) or a generative 3D human shape and articulated pose model (GHUM), and the like, which are not limited in the present application.

[0126] The determined rotation parameter can be taken as input data and input into the human kinematics model, and the human kinematics model combines the initial body shape parameter and the joint tree length to process the rotation parameter, so as to generate a corresponding initial 3D human pose model. The 3D coordinates of the surface grid vertices of the initial 3D human pose model are projected to a three-dimensional space to generate a first point cloud data set.

[0127] Therefore, in the embodiments of the present application, the original measurement data collected by the inertial measurement unit sensor can be first feature extracted to obtain a feature vector, then decoded by a kinematics regression decoder to obtain a rotation parameter, and then processed by a human body kinematics model to obtain a corresponding initial three-dimensional human body posture model. The three-dimensional coordinates of the vertices in the initial three-dimensional human body posture model are then mapped to generate a first point cloud data set. Since the motion pattern in the time sequence is fully mined in this process, the first point cloud data set determined can more accurately reflect the dynamic characteristics of the human body posture in the time dimension, providing a basis for the reliability of subsequent processing.

[0128] Step 103, feature extraction and processing of the original point cloud data set and the original image data are performed to determine a second point cloud data set.

[0129] After obtaining the original point cloud data set collected by the millimeter wave radar and the original image data collected by the camera, the two can be first fused, and then the second point cloud data set is further determined. For example, the original point cloud data set and the original image data can be respectively feature extracted to obtain a feature vector, and then the feature vector is fused and the second point cloud data set is generated according to the fused feature vector, or the original point cloud data set and the original image data can be processed in other ways to generate the second point cloud data set, etc. The present application does not limit this.

[0130] It can be understood that since the original point cloud data set is collected by the millimeter wave radar and the original image data is collected by the camera, the second point cloud data set determined according to the original point cloud data set and the original image data combines the characteristics of accurate millimeter wave radar data and rich image data semantic information, that is, the second point cloud data set determined by multi-modal data fusion has accurate three-dimensional coordinates and rich semantic information, providing a more reliable and dense data basis for subsequent processing.

[0131] Step 104, a third point cloud data set is generated according to the first point cloud data set and the second point cloud data set.

[0132] It can be understood that the first point cloud data set can be directly fused with the second point cloud data to obtain a third point cloud data set; or the first point cloud data set can be used to update and fill the second point cloud data set, and the updated second point cloud data set is determined as the third point cloud data set; or the first point cloud data set and the second point cloud data set can be processed respectively, and then the third point cloud data set is generated, and the like. Since the first point cloud data can fully reflect the characteristics of the human body posture in the time dimension, the second point cloud data set can fully reflect the characteristics and image information in the spatial dimension, and after processing the first point cloud data set and the second point cloud data set, the third point cloud data set generated further contains time characteristics and spatial characteristics through multi-modal data fusion, has more comprehensive information, and can better perform subsequent processing.

[0133] In step 105, the third point cloud data set is voxelized to generate a three-dimensional feature map based on a voxel tensor.

[0134] The third point cloud data set can be voxelized by a voxel grid to obtain a voxel tensor, and then the voxel tensor is processed to generate a three-dimensional feature map; or other methods can be used to voxelize the third point cloud data set to generate a three-dimensional feature map, and the like, which are not limited in the present application.

[0135] It can be understood that since the third point cloud data set is obtained by multi-modal data fusion and contains both time characteristics and spatial characteristics, the three-dimensional feature map generated by processing the third point cloud data set also has more comprehensive and rich information, which provides a solid foundation for ensuring the accuracy of subsequent processing.

[0136] In step 106, the feature parameters in the heat map corresponding to the three-dimensional feature map are processed to determine a three-dimensional human body posture.

[0137] The feature parameters can be understood as heat values. After generating the three-dimensional feature map, the three-dimensional feature map can be further processed to generate a corresponding heat map, and then the three-dimensional human body posture is determined according to the heat values contained in the heat map. Since the three-dimensional feature map contains comprehensive and rich time and space characteristics, the three-dimensional human body posture determined based on the heat map generated from the three-dimensional feature map is also more accurate and reliable. Therefore, in the embodiments of the present application, the original measurement data collected by the inertial measurement unit sensor, the original point cloud data set collected by the millimeter wave radar, and the original image data collected by the camera are processed in series, multi-modal data processing and fusion are realized, the determined three-dimensional human body posture is more accurate and reliable, and the accuracy and reliability of the three-dimensional human body posture determination are improved.

[0138] The embodiment of the application can first acquire the original measurement data collected by the inertial measurement unit sensor, the original point cloud data set collected by the millimeter wave radar, and the original image data collected by the camera, then decode the feature vectors obtained after the original measurement data is extracted by a kinematics regression decoder, generate a first point cloud data set in combination with a human body kinematics model, extract and process the original point cloud data set and the original image data to determine a second point cloud data set, generate a third point cloud data set according to the first point cloud data set and the second point cloud data set, then perform voxelization processing on the third point cloud data set to generate a three-dimensional feature map based on voxel tensors, and then process the feature parameters in the heat map corresponding to the three-dimensional feature map to determine a three-dimensional human body posture. Therefore, by processing the original measurement data, the original point cloud data set, and the original image data to generate corresponding point cloud data sets, and then processing each point cloud data set to generate a three-dimensional feature map and determine a three-dimensional human body posture, the three-dimensional human body posture determined through multi-modal data fusion and processing is more accurate and reliable, thereby improving the accuracy and reliability of the determination of the three-dimensional human body posture.

[0139] As shown in Figure 2 The method for determining a three-dimensional human body posture can include the following steps:

[0140] Step 201: acquiring original measurement data collected by an inertial measurement unit sensor, original point cloud data set collected by a millimeter wave radar, and original image data collected by a camera.

[0141] Step 202: decoding the feature vectors obtained after the original measurement data is extracted by a kinematics regression decoder, and generating a first point cloud data set in combination with a human body kinematics model.

[0142] Step 203: extracting and processing the original point cloud data set and the original image data to determine a second point cloud data set.

[0143] Optionally, the original point cloud data set can be processed to determine an initial depth map, then the initial depth map is extracted to obtain a corresponding depth feature vector, the original image data is extracted to determine a corresponding semantic feature vector, the depth feature vector and the semantic feature vector are fused to generate a second feature vector, the second feature vector is processed to determine a target depth map, and the target depth map is processed to generate a second point cloud data set.

[0144] The original point cloud data set can be converted into a camera coordinate system by combining the internal and external parameters of the camera to obtain an initial depth map. Since there may be no initial depth value at some positions in the initial depth map, a binary mask identifying the missing area can be created to identify the missing area. The initial depth values in the initial depth map can then be normalized to determine a first depth value, and the missing area can be filled according to the first depth value to generate a second depth value corresponding to the target area. The first depth value and the second depth value are then processed to generate a depth feature vector.

[0145] The known initial depth value d in the initial depth map can be first randomly scaled and then linearly transformed to a predetermined value range. In order to prevent decoding overflow, a random compression factor can also be introduced to enhance the robustness of the model to unknown depth ranges, and then the first depth value can be obtained . For the missing area in the initial depth map where the initial depth value is missing, the second depth value corresponding to the missing area can be densified by linear interpolation using the known first depth values around the missing area.

[0146] It can be understood that the initial depth values d max , d min can be determined first, and then normalized in combination with d max , d min , and linearly transformed to a predetermined value range using linear transformation coefficients such as 0.5, 2, etc., and in combination with the introduced random compression factor , the first depth value can be determined.

[0147] In addition, the first depth value and the second depth value can be visualized as a grayscale or color image, which is the preprocessed depth map. The preprocessed depth map can then be processed to generate a depth feature vector. For example, a pre-trained variational autoencoder VAE can be used as an encoder to encode the preprocessed depth map into a low-dimensional latent space to generate a depth feature vector. Alternatively, a generative adversarial network GAN or other model can be used to extract features from the preprocessed depth map to generate a depth feature vector, etc. The present application does not limit this.

[0148] Optionally, for a single-channel depth map, it can also be converted into three-channel data by channel replication to meet the data processing requirements, etc. The present application does not limit this.

[0149] After that, the original image data can be continuously processed, such as being encoded into a low-dimensional latent space by using a variational autoencoder (VAE) encoding to generate multi-level and fine-grained semantic feature vectors, or being feature-extracted by using a generative adversarial network (GAN) or other models to generate semantic feature vectors, and the like, which are not limited in the present application.

[0150] It can be understood that there are various ways to fuse the depth feature vector and the semantic feature vector to generate a second feature vector. For example, the depth feature vector and the semantic feature vector can be directly spliced, and the result is the second feature vector; or the two can be weighted and fused, and the result is the second feature vector. Or the semantic feature vector can be gradually and hierarchically fused into the depth feature vector through a cross-attention mechanism and hierarchical feature splicing, to further repair the second depth value of the missing area, ensuring that the repaired second depth value is semantically reasonable and strictly consistent with the first depth value in geometric scale.

[0151] Optionally, since the known depth area containing the first depth value and the missing area containing the second depth value in the preprocessed depth map are position-encoded during the process of generating the depth feature vector from the preprocessed depth map, the position feature vector corresponding to the position encoding can be used as an input together with the depth feature vector, and then fused with the semantic feature vector to generate the second feature vector. The VAE encoder encoded mask can be processed to generate a position feature vector, or a position feature vector can be generated by other means, and the like, which are not limited in the present application.

[0152] It can be understood that the second feature vector can be processed, such as being decoded by using a VAE decoder to obtain a complete target depth map, or being processed by using a diffusion model, a Transformer generator, or the like to generate a complete target depth map, and the like, which are not limited in the present application.

[0153] In addition, after the target depth map is determined, the target depth map can be inversely transformed by random scale normalization to restore its absolute metric scale, to obtain a dense depth map aligned with the original image pixels, and further using the intrinsic matrix of the camera to back-project each pixel point in the dense depth map to a three-dimensional space to calculate its corresponding three-dimensional coordinates, thereby generating a high-quality and dense three-dimensional point cloud, i.e., a second point cloud dataset, which is consistent with the input image perspective.

[0154] Therefore, in the embodiments of the present application, in the process of generating the second point cloud data set, the original sparse point cloud data and the original image data are fully considered, and the depth feature vector and the semantic feature vector are further fused to generate a more accurate and reliable second feature vector, which is then processed to generate a second point cloud data set containing more information, thereby providing a solid foundation for the accuracy and reliability of subsequent processing using the second point cloud data set.

[0155] Step 204: converting and processing the first point cloud data set and the second point cloud data set to generate a fourth point cloud data set and a fifth point cloud data set corresponding respectively.

[0156] The first point cloud data set and the second point cloud data set can be subjected to coordinate system conversion and timestamp alignment processing to generate a fourth point cloud data set and a fifth point cloud data set corresponding respectively. For example, based on the system calibration parameters completed in advance, the first point cloud data set and the second point cloud data set can be converted to the same world coordinate system, and the two point cloud data sets can be subjected to timestamp alignment to obtain a fourth point cloud data set and a fifth point cloud data set corresponding respectively, so that it can be ensured that the current processing is spatial data at the same time, thereby effectively improving the accuracy and reliability.

[0157] Step 205: determining a reference point corresponding to each fifth point cloud data in the fifth point cloud data set from the fourth point cloud data set.

[0158] Since the fourth point cloud data set and the fifth point cloud data set each contain a plurality of point cloud data, for each fifth point cloud data in the fifth point cloud data set, a number of points in the fourth point cloud data set closest to the fifth point cloud data in Euclidean distance can be found to form a near-neighbor point set, and then the centroid position or average position of the near-neighbor point set can be further determined, and this position can be taken as the reference point corresponding to the fifth point cloud data in the fourth point cloud data set.

[0159] Step 206: updating the confidence weight of each fifth point cloud data based on the coordinates of the reference point.

[0160] Optionally, the distance value between each fifth point cloud data and the corresponding reference point can be determined according to the coordinates of each fifth point cloud data and the coordinates of the reference point, and then the confidence weight of the fifth point cloud data with a distance value greater than a first threshold value can be updated.

[0161] The distance value between each fifth point cloud data and the corresponding reference point can be determined by the Euclidean distance formula, the Manhattan distance formula, etc., which is not limited in the present application.

[0162] In addition, the first threshold value can be a preset value, or a dynamic distance threshold value can be set according to prior knowledge of human body part size, or the first threshold value can be adjusted as needed, and the application does not limit this.

[0163] It can be understood that the smaller the distance value, the closer the spatial position between the fifth point cloud data and the corresponding reference point, and the higher the possibility that they belong to the same part of the human body. The larger the distance value, the farther the spatial position between the fifth point cloud data and the corresponding reference point, and the greater the possibility that they come from the environment background or are noise interference. Therefore, if the distance exceeds the first threshold value, it can be considered that the fifth point cloud data can be a noise point or an interference point from the environment background, and the confidence weight thereof can be reduced, thereby updating the confidence weight thereof.

[0164] Optionally, an included angle cosine value between the first normal vector of the region to which each fifth point cloud data belongs and the second normal vector of the region to which the corresponding reference point belongs can be determined, and then the confidence weight of the fifth point cloud data whose included angle cosine value is less than a second threshold value is updated.

[0165] The second threshold value can be a preset value, or it can be adjusted as needed, and the application does not limit this.

[0166] The region to which the fifth point cloud data belongs can be a region containing a certain set area value of the fifth point cloud data, and correspondingly, the region to which the reference point belongs can be a region containing the same set area value of the reference point. Then, the included angle cosine value between the first normal vector of the region to which each fifth point cloud data belongs and the second normal vector of the region to which the corresponding reference point belongs can be determined. It can be understood that the larger the included angle cosine value, the smaller the angle between them, that is, the more consistent the geometric structure of the region corresponding to the first normal vector and the region corresponding to the second normal vector, and the closer the surface orientation. The smaller the included angle cosine value, the larger the angle between them, that is, the more conflicting the geometric structure of the region corresponding to the first normal vector and the region corresponding to the second normal vector, and the more deviated the surface orientation. Therefore, if the included angle cosine value of the first normal vector and the second normal vector is less than the second threshold value, it can be considered that the geometric structure of the surface where the fifth point cloud data is located is in conflict, and the confidence weight thereof can be reduced, thereby updating the confidence weight thereof.

[0167] Step 207, generating a third point cloud data set according to the confidence weight of each fifth point cloud data and the fourth point cloud data set.

[0168] It can be understood that the same initial confidence weight can be set for each fifth point cloud data, so that after part of the fifth point cloud data is updated according to the distance value or the included angle cosine value, the confidence weight of the updated part of the fifth point cloud data is reduced. If the updated confidence weight of the fifth point cloud data is less than the third threshold, the fourth point cloud data at the corresponding position in the fourth point cloud data set can be directly filled with the fourth point cloud data; or the fourth point cloud data determined to be at the corresponding position with the fifth point cloud data can be used to fill the fifth point cloud data, so that the occluded area can be further optimized, that is, the fourth point cloud data provides a reliable geometric prior of the occluded part of the human kinematic model, thereby generating the final optimized complete third point cloud data fused with the three-dimensional human surface.

[0169] In step 208, the third point cloud data set is voxelized to generate a three-dimensional feature map based on voxel tensors.

[0170] In step 209, the feature parameters in the heat map corresponding to the three-dimensional feature map are processed to determine the three-dimensional human pose.

[0171] In the embodiments of the present application, the original measurement data collected by the inertial measurement unit sensor, the original point cloud data set collected by the millimeter wave radar, and the original image data collected by the camera can be obtained first. Then, the feature vectors obtained by feature extraction of the original measurement data are decoded and processed by the kinematic regression decoder, and the first point cloud data set is generated in combination with the human kinematic model. The original point cloud data set and the original image data are feature extracted and processed to determine the second point cloud data set. Then, the first point cloud data set and the second point cloud data set are converted and processed to generate the corresponding fourth point cloud data set and fifth point cloud data set, respectively. The reference points corresponding to each fifth point cloud data in the fifth point cloud data set are determined from the fourth point cloud data set. The confidence weight of each fifth point cloud data is updated based on the coordinates of the reference points. The third point cloud data set is generated based on the confidence weight of each fifth point cloud data and the fourth point cloud data set. Then, the third point cloud data set is voxelized to generate a three-dimensional feature map based on voxel tensors. The feature parameters in the heat map corresponding to the three-dimensional feature map are processed to determine the three-dimensional human pose. Thus, by processing the original measurement data, the original point cloud data set, and the original image data to generate the corresponding point cloud data set, and then processing each point cloud data set to generate a three-dimensional feature map and determine a three-dimensional human pose, the determined three-dimensional human pose can be more accurate and reliable through multi-modal data fusion and processing, thereby improving the accuracy and reliability of the determination of the three-dimensional human pose.

[0172] As Figure 3 shown, the three-dimensional human body posture determination method can include the following steps:

[0173] Step 301, obtaining the original measurement data collected by the inertial measurement unit sensor, the original point cloud data set collected by the millimeter wave radar, and the original image data collected by the camera.

[0174] Step 302, using a kinematic regression decoder to decode the feature vectors obtained after feature extraction of the original measurement data, and combining a human body kinematic model to generate a first point cloud data set.

[0175] Step 303, performing feature extraction and processing on the original point cloud data set and the original image data to determine a second point cloud data set.

[0176] Step 304, generating a third point cloud data set according to the first point cloud data set and the second point cloud data set.

[0177] Step 305, voxelizing the third point cloud data set to generate a three-dimensional feature map based on voxel tensors.

[0178] Optionally, a three-dimensional cubic bounding box can be determined according to the coordinates of each third point cloud data in the third point cloud data set, and then the three-dimensional cubic bounding box can be divided into a plurality of voxel grids, and a voxel tensor can be generated according to the positional relationship between each voxel grid and each third point cloud data, and then the voxel tensor can be feature extracted to generate a three-dimensional feature map.

[0179] Among them, a three-dimensional spatial distribution of the third point cloud data can be determined according to the coordinates of each third point cloud data in the third point cloud data set, and then a three-dimensional cubic bounding box that can completely wrap the three-dimensional spatial distribution can be determined accordingly. Then the bounding box can be uniformly divided in X, Y, and Z dimensions to obtain a plurality of voxel grids with a size of Each voxel is a tiny cubic unit, which can represent a discrete position in three-dimensional space.

[0180] For any voxel grid, if it contains at least one third point cloud data inside, the value of the voxel grid can be assigned to 1, indicating that the voxel grid is currently occupied; if it does not contain any third point cloud data inside, the value of the voxel grid can be assigned to 0, indicating that the voxel grid is currently in an idle state, and then each voxel grid can be traversed to generate a voxel tensor V with a size of

[0181] ​The generated voxel tensor V can be input into a trained three-dimensional convolutional neural network model. The network model is stacked by multiple layers of 3D convolution, 3D pooling and activation function, and the 3D convolution kernel can slide in the depth, height and width dimensions, so as to effectively capture the human structure semantic information and the spatial context relationship between joints in the point cloud data. Therefore, after layer-by-layer feature extraction by the network model, a high-dimensional and semantic-rich three-dimensional feature map F can be output, which can represent the geometric structure and potential joint position of the whole human body in the voxel space.

[0182] In step 306, the three-dimensional feature map is input into a target network model to generate a heat map of each target joint through processing of the target network model.

[0183] The target network model can be a trained three-dimensional convolutional neural network model, or can be other models capable of generating heat maps, and the present application does not limit the target network model.

[0184] In addition, the target joint can be a joint corresponding to the inertial measurement sensor, or can be other joints, and the present application does not limit the target joint.

[0185] Therefore, in the embodiment of the present application, for each target joint j of the human body such as the left shoulder and the right knee, the three-dimensional feature map is input into the target network model, and a dedicated output branch of the target network model can output a corresponding three-dimensional heat map through processing of the target network model.

[0186] In step 307, the position of the maximum heat value in the heat map of each target joint is determined as the three-dimensional coordinate of the corresponding target joint.

[0187] It can be understood that since the size of the three-dimensional heat map is the same as the size of the voxel grid, the heat value of each position in the three-dimensional heat map represents the probability that the voxel grid is the position of the target joint j. Generally, the three-dimensional heat map will form a peak value at the real three-dimensional position of the target joint j, and the Gaussian distribution will decay as the distance from the position increases. Therefore, the position of the maximum heat value in the heat map of each target joint can be determined as the coordinate of the corresponding target joint.

[0188] In step 308, the three-dimensional coordinates of each target joint are fused to determine the three-dimensional human body pose.

[0189] It can be understood that after the three-dimensional coordinates of each target joint are determined, the three-dimensional coordinates of each target joint can be sequentially connected according to the human skeleton structure, and a complete three-dimensional human skeleton can be reconstructed, so as to determine the three-dimensional human posture, and then realize the estimation and determination of the final three-dimensional human posture from the multi-modal point cloud data fusion, and improve the accuracy and reliability of the three-dimensional human posture determination.

[0190] It should be noted that the method for determining a three-dimensional human posture provided in the present application can be applied to any scene of three-dimensional human posture determination. The following will take the physical test scene as an example to briefly describe the determination process of the three-dimensional human posture provided in the present application.

[0191] Firstly, as shown in Figure 4 , the original measurement data collected by the inertial measurement unit sensor, the original point cloud data set collected by the millimeter wave radar, and the original image data collected by the camera are acquired first. Then, the original measurement data is processed by the 3D human kinematics model estimation based on the IMU to generate the first point cloud data set. The original point cloud data set and the original image data are first fused, and then the depth is completed to determine the second point cloud data set. Then, the first point cloud data set and the second point cloud data set are fused to generate the third point cloud data set. Then, the third point cloud data set is processed by the point cloud voxelization using the three-dimensional cubic bounding box to determine the voxel tensor, and the voxel tensor is input into the 3D CNN encoding and decoding structure neural network model. After processing by the network model, the heat map of each target joint can be output, and the three-dimensional human posture can be determined by further processing the heat map of each target joint.

[0192] Further, in the process of processing the original measurement data by the 3D human kinematics model estimation based on the IMU to generate the first point cloud data set, the t1, t2, …, T N-1 , T N corresponding to each time respectively can be first processed by the inertial measurement unit (IMU) to extract the features of the original measurement data to generate the first feature vector. Then, the difference vector between the two consecutive frames in the first feature vector is determined, and the first feature vector and the difference vector are fused and enhanced in features and then input into the LSTM. After processing by the LSTM, the second feature vector is generated. Then, the second feature vector is decoded by the kinematics regression decoder to determine the rotation parameter. Then, the initial three-dimensional human posture model is generated according to the initial body shape parameter, the joint tree length and the rotation parameter of the human kinematics model. Then, the three-dimensional coordinates of the vertices in the initial three-dimensional human posture model are mapped to generate the first point cloud data set.

[0193] In addition, in the process of fusing the original point cloud dataset and the original image data and performing depth completion to determine the second point cloud dataset, a double-branch diffusion model can be used for processing, as shown in Figure 5 The original image data can be encoded by a reference branch in the double-branch diffusion model to generate a semantic feature vector. Then, the original point cloud dataset can be converted into the camera coordinate system by combining the intrinsic and extrinsic parameters of the camera to obtain an initial depth map, i.e., a sparse depth map. The binary mask of the region can be further identified, and the missing region can be filled according to the first depth value in the sparse depth map to generate a second depth value corresponding to the target region. The first depth value and the second depth value are normalized to generate a depth feature vector. Then, the depth feature vector is fused by an estimation branch in the double-branch diffusion model to generate a second feature vector in combination with the semantic feature vector, so as to determine the target depth map completed by the semantic feature vector, and further generate the second point cloud dataset.

[0194] Then, the first point cloud dataset generated based on the IMU human pose estimation and the second point cloud dataset completed based on the radar and the image information can be processed in the coordinate system and time synchronization to generate a fourth point cloud dataset and a fifth point cloud dataset, respectively, as shown in Figure 6 Then, the reference points corresponding to each fifth point cloud dataset are determined according to the nearest neighbor matching search. Then, the low-confidence or occluded regions are discriminated according to the distance constraint filtering and the normal consistency constraint. Then, the final third point cloud dataset is generated based on the confidence weighted fusion or directly filled by the fourth point cloud data.

[0195] It should be noted that the above process is only illustrative and cannot be regarded as a limitation on the process of determining the three-dimensional human pose in the embodiments of the present application.

[0196] The embodiment of the application can first acquire original measurement data collected by an inertial measurement unit sensor, an original point cloud data set collected by a millimeter wave radar, and original image data collected by a camera, then decode a feature vector obtained after feature extraction of the original measurement data by using a kinematics regression decoder, generate a first point cloud data set in combination with a human body kinematics model, perform feature extraction and processing on the original point cloud data set and the original image data to determine a second point cloud data set, then generate a third point cloud data set according to the first point cloud data set and the second point cloud data set, perform voxelization processing on the third point cloud data set to generate a three-dimensional feature map based on a voxel tensor, then input the three-dimensional feature map into a target network model to generate a heat map of each target joint through processing of the target network model, and determine a position of a maximum heat value in the heat map of each target joint as a three-dimensional coordinate of the corresponding target joint, then perform fusion processing on the three-dimensional coordinates of each target joint to determine a three-dimensional human body posture. Thus, the original measurement data, the original point cloud data set, and the original image data are processed to generate corresponding point cloud data sets, then each point cloud data set is processed to generate a three-dimensional feature map, and then the three-dimensional human body posture is determined, that is, through multi-modal data fusion and processing, the determined three-dimensional human body posture is more accurate and reliable, and the accuracy and reliability of the determination of the three-dimensional human body posture are improved.

[0197] According to the application, a three-dimensional human body posture determination device 800 is provided, as shown in the accompanying drawings, which comprises an acquisition module 810, a first generation module 820, a first determination module 830, a second generation module 840, a third generation module 850, and a second determination module 860. Figure 7

[0198] The acquisition module 810 is configured to acquire original measurement data collected by an inertial measurement unit sensor, an original point cloud data set collected by a millimeter wave radar, and original image data collected by a camera.

[0199] The first generation module 820 is configured to decode a feature vector obtained after feature extraction of the original measurement data by using a kinematics regression decoder, and generate a first point cloud data set in combination with a human body kinematics model.

[0200] The first determination module 830 is configured to perform feature extraction and processing on the original point cloud data set and the original image data to determine a second point cloud data set.

[0201] The second generation module 840 is configured to generate a third point cloud data set according to the first point cloud data set and the second point cloud data set.

[0202] ​The third generation module 850 is configured to voxelize the third point cloud data set to generate a three-dimensional feature map based on voxel tensors.

[0203] The second determination module 860 is configured to process a feature parameter in a heat map corresponding to the three-dimensional feature map to determine a three-dimensional human body pose.

[0204] Optionally, the first generation module 820 comprises:

[0205] The first determination unit is configured to perform feature extraction on the original measurement data to determine a first feature vector.

[0206] The second determination unit is configured to determine a difference vector between two consecutive frames in the first feature vector.

[0207] The fusion unit is configured to fuse the first feature vector and the difference vector to generate a second feature vector.

[0208] The decoding processing unit is configured to perform decoding processing on the second feature vector by using a kinematic regression decoder to determine a rotation parameter.

[0209] The third determination unit is configured to generate an initial three-dimensional human body pose model according to an initial body shape parameter, a joint tree length of a human body kinematic model, and the rotation parameter.

[0210] The third determination unit is further configured to perform mapping processing on three-dimensional coordinates of a vertex in the initial three-dimensional human body pose model to generate a first point cloud data set.

[0211] Optionally, the first determination module 830 comprises:

[0212] The fourth determination unit is configured to process the original point cloud data set to determine an initial depth map.

[0213] The first generation unit is configured to perform feature extraction on the initial depth map to generate a depth feature vector.

[0214] The fifth determination unit is configured to perform feature extraction on the original image data to determine a corresponding semantic feature vector.

[0215] The second generation unit is configured to fuse the depth feature vector and the semantic feature vector to generate a second feature vector.

[0216] The sixth determination unit is configured to process the second feature vector to determine a target depth map.

[0217] The third generation unit is configured to process the target depth map to generate a second point cloud data set.

[0218] Optionally, the first generating unit is specifically used for:

[0219] normalizing the initial depth value in the initial depth map to determine a first depth value;

[0220] filling a missing region in the initial depth map according to the first depth value to determine a second depth value corresponding to the missing region;

[0221] processing the first depth value and the second depth value to generate a depth feature vector.

[0222] Optionally, the second generating module 840 comprises:

[0223] a fourth generating unit, configured to perform conversion processing on the first point cloud data set and the second point cloud data set to generate a fourth point cloud data set and a fifth point cloud data set corresponding thereto, respectively;

[0224] a seventh determining unit, configured to determine, from the fourth point cloud data set, a reference point corresponding to each fifth point cloud data in the fifth point cloud data set;

[0225] an updating unit, configured to update a confidence weight of each fifth point cloud data based on a coordinate of the reference point;

[0226] a fifth generating unit, configured to generate a third point cloud data set according to the confidence weight of each fifth point cloud data and the fourth point cloud data set.

[0227] Optionally, the updating unit is specifically used for:

[0228] determining a distance value between each fifth point cloud data and the corresponding reference point according to a coordinate of each fifth point cloud data and a coordinate of the reference point, and updating the confidence weight of the fifth point cloud data whose distance value is greater than a first threshold value;

[0229] and / or,

[0230] determining a cosine value of an included angle between a first normal vector of a region to which each fifth point cloud data belongs and a second normal vector of a region to which the corresponding reference point belongs, and updating the confidence weight of the fifth point cloud data whose cosine value is less than a second threshold value.

[0231] Optionally, the third generating module 850 is used for:

[0232] determining a three-dimensional cubic bounding box according to a coordinate of each third point cloud data in the third point cloud data set;

[0233] dividing the three-dimensional cubic bounding box into a plurality of voxel grids;

[0234] generate a voxel tensor according to a positional relationship between each of the voxel grids and each of the third point cloud data;

[0235] perform feature extraction on the voxel tensor to generate a three-dimensional feature map.

[0236] Optionally, the second determining module 860 is configured to:

[0237] input the three-dimensional feature map into a target network model to generate a heat map of each target joint through processing of the target network model;

[0238] determine a position where a maximum heat value in the heat map of each target joint is located as a three-dimensional coordinate of the corresponding target joint;

[0239] perform fusion processing on the three-dimensional coordinates of each target joint to determine a three-dimensional human body pose.

[0240] The three-dimensional human body pose determination apparatus provided in the present application can first acquire original measurement data collected by an inertial measurement unit sensor, an original point cloud data set collected by a millimeter wave radar, and original image data collected by a camera. Then, a kinematics regression decoder is used to decode a feature vector obtained through feature extraction of the original measurement data, and a first point cloud data set is generated in combination with a human body kinematics model. The original point cloud data set and the original image data are then subjected to feature extraction and processing to determine a second point cloud data set. According to the first point cloud data set and the second point cloud data set, a third point cloud data set is generated. Then, voxelization processing is performed on the third point cloud data set to generate a three-dimensional feature map based on a voxel tensor. Then, a feature parameter in a heat map corresponding to the three-dimensional feature map is processed to determine a three-dimensional human body pose. Thus, through processing of the original measurement data, the original point cloud data set, and the original image data, corresponding point cloud data sets are generated. Then, through processing of the point cloud data sets, a three-dimensional feature map is generated, and a three-dimensional human body pose is determined. That is, through multi-modal data fusion and processing, the determined three-dimensional human body pose is more accurate and reliable, and the accuracy and reliability of three-dimensional human body pose determination are improved.

[0241] It should be understood that the specific features, operations and details described above with respect to the method of the present application can be similarly applied to the apparatus and system of the present application, or vice versa. In addition, each step of the method of the present application described above can be performed by the corresponding components or units of the apparatus or system of the present application.

[0242] It should be understood that each module / unit of the apparatus of the present application can be implemented in whole or in part by software, hardware, firmware, or a combination thereof. Each module / unit can be embedded in a processor of an electronic device in hardware or firmware form, or independent of the processor, or stored in a memory of the electronic device in software form to be invoked by the processor to perform operations of each module / unit. Each module / unit can be implemented as an independent component or module, or two or more modules / units can be implemented as a single component or module.

[0243] As shown in Figure 8 The present application provides an electronic device 900, which includes a processor 901 and a memory 902 having computer program instructions stored therein. When the processor 901 executes the computer program instructions, each step of the method for determining a three-dimensional human pose is implemented. The electronic device 900 can be a server, a terminal, or any other electronic device with necessary computing and / or processing capabilities.

[0244] In one embodiment, the electronic device 900 can include a processor, a memory, a network interface, a communication interface, etc. connected by a system bus. The processor of the electronic device 900 can be configured to provide necessary computing, processing, and / or control capabilities. The memory of the electronic device 900 can include a non-volatile storage medium and an internal memory. The non-volatile storage medium can store an operating system, a computer program, etc. The internal memory can provide an environment for running the operating system and the computer program in the non-volatile storage medium. The network interface and the communication interface of the electronic device 900 can be configured to connect and communicate with external devices through a network. The computer program, when executed by the processor, performs the steps of the method of the present application.

[0245] The present application provides a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implement the method for determining a three-dimensional human pose.

[0246] Those skilled in the art can understand that the steps of the method of the present application can be instructed by a computer program to relevant hardware such as the electronic device 900 or the processor, and the computer program can be stored in a non-transitory computer-readable storage medium, which, when executed, causes the steps of the present application to be performed. According to the circumstances, any reference to a memory, storage, or other medium herein can include a non-volatile or volatile memory. Examples of non-volatile memory include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk, etc. Examples of volatile memory include random access memory (RAM), external cache memory, etc.

[0247] The technical features described above can be combined arbitrarily. Although all possible combinations of the technical features are not described, any combination of the technical features should be considered to be covered by the present specification, as long as there is no contradiction in such a combination.

[0248] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit it; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method of determining a three-dimensional human pose, characterized by, The method comprises: acquiring original measurement data collected by an inertial measurement unit sensor, an original point cloud data set collected by a millimeter wave radar, and original image data collected by a camera; decoding a feature vector obtained after feature extraction on the original measurement data by using a kinematics regression decoder, and generating a first point cloud data set in combination with a human body kinematics model; performing feature extraction and processing on the original point cloud data set and the original image data to determine a second point cloud data set; generating a third point cloud data set according to the first point cloud data set and the second point cloud data set; performing voxelization processing on the third point cloud data set to generate a three-dimensional feature map based on a voxel tensor; processing a feature parameter in a heat map corresponding to the three-dimensional feature map to determine a three-dimensional human body pose; wherein the decoding of the feature vector obtained after the feature extraction on the original measurement data by using the kinematics regression decoder and the generation of the first point cloud data set in combination with the human body kinematics model comprise: performing feature extraction on the original measurement data to determine a first feature vector; determining a difference vector between two consecutive frames in the first feature vector; fusing the first feature vector and the difference vector to generate a second feature vector; decoding the second feature vector by using the kinematics regression decoder to determine a rotation parameter; generating an initial three-dimensional human body pose model according to an initial body shape parameter, a joint tree length of the human body kinematics model, and the rotation parameter; mapping and processing three-dimensional coordinates of vertices in the initial three-dimensional human body pose model to generate the first point cloud data set.

2. The method of claim 1, wherein, The feature extraction and processing on the original point cloud data set and the original image data to determine the second point cloud data set comprise: processing the original point cloud data set to determine an initial depth map; performing feature extraction on the initial depth map to generate a depth feature vector; performing feature extraction on the original image data to determine a corresponding semantic feature vector; fusing the depth feature vector and the semantic feature vector to generate a second feature vector; processing the second feature vector to determine a target depth map; processing the target depth map to generate the second point cloud data set.

3. The method of claim 2, wherein, The feature extraction on the initial depth map to generate the depth feature vector comprises: performing normalization processing on an initial depth value in the initial depth map to determine a first depth value; filling a missing area in the initial depth map according to the first depth value to determine a second depth value corresponding to the missing area; processing the first depth value and the second depth value to generate a depth feature vector.

4. The method of claim 1, wherein, The generation of the third point cloud data set according to the first point cloud data set and the second point cloud data set comprises: performing conversion processing on the first point cloud data set and the second point cloud data set to generate a fourth point cloud data set and a fifth point cloud data set corresponding thereto, respectively; determining a reference point corresponding to each fifth point cloud data in the fifth point cloud data set from the fourth point cloud data set; updating a confidence weight of each of the fifth point cloud data based on the coordinates of the reference points; generating a third point cloud data set according to the confidence weight of each of the fifth point cloud data and the fourth point cloud data set.

5. The method of claim 4, wherein, The updating of the confidence weight of each of the fifth point cloud data based on the coordinates of the reference points comprises: determining a distance value between each of the fifth point cloud data and the corresponding reference point according to the coordinates of each of the fifth point cloud data and the coordinates of the reference points, and updating the confidence weight of the fifth point cloud data whose distance value is greater than a first threshold value; and / or determining a cosine value of an included angle between a first normal vector of a region to which each of the fifth point cloud data belongs and a second normal vector of a region to which the corresponding reference point belongs, and updating the confidence weight of the fifth point cloud data whose cosine value is less than a second threshold value.

6. The method of claim 1, wherein, The voxelization processing of the third point cloud data set to generate a three-dimensional feature map based on a voxel tensor comprises: determining a three-dimensional cubic bounding box according to the coordinates of each of the third point cloud data in the third point cloud data set; dividing the three-dimensional cubic bounding box into a plurality of voxel grids; generating a voxel tensor according to the positional relationship between each of the voxel grids and each of the third point cloud data; performing feature extraction on the voxel tensor to generate a three-dimensional feature map.

7. The method of claim 1, wherein, The processing of a feature parameter in a heat map corresponding to the three-dimensional feature map to determine a three-dimensional human body posture comprises: inputting the three-dimensional feature map into a target network model to generate a heat map of each target joint through processing of the target network model; determining a position where a maximum heat value in the heat map of each target joint as a three-dimensional coordinate of the corresponding target joint; fusing the three-dimensional coordinates of each target joint to determine a three-dimensional human body posture.

8. An apparatus for determining a three-dimensional human pose, the apparatus comprising: The method comprises: acquiring original measurement data collected by an inertial measurement unit sensor, an original point cloud data set collected by a millimeter wave radar, and original image data collected by a camera; decoding and processing a feature vector obtained by performing feature extraction on the original measurement data by using a kinematics regression decoder, and generating a first point cloud data set in combination with a human body kinematics model; determining a second point cloud data set by performing feature extraction and processing on the original point cloud data set and the original image data; generating a third point cloud data set according to the first point cloud data set and the second point cloud data set; performing voxelization processing on the third point cloud data set to generate a three-dimensional feature map based on a voxel tensor; processing a feature parameter in a heat map corresponding to the three-dimensional feature map to determine a three-dimensional human body posture; The first generation module comprises: a first determination unit configured to perform feature extraction on the original measurement data to determine a first feature vector; a second determination unit configured to determine a difference vector between two consecutive frames in the first feature vector. a fusion unit configured to fuse the first feature vector and the difference vector to generate a second feature vector; a decoding processing unit configured to decode the second feature vector by using a kinematic regression decoder to determine a rotation parameter; a third determining unit configured to generate an initial three-dimensional human pose model according to an initial body shape parameter of a human kinematic model, a joint tree length, and the rotation parameter; the third determining unit is further configured to perform mapping processing on three-dimensional coordinates of vertices in the initial three-dimensional human pose model to generate a first point cloud data set.

9. An electronic device, comprising: The electronic device comprises a processor and a memory storing computer program instructions; The processor executes the computer program instructions to implement the method for determining a three-dimensional human pose according to any one of claims 1-7.

Images