A training and recognition method for point cloud human pose estimation models based on knowledge transfer

Abstract

This application provides a method for training and recognizing a knowledge transfer-based pose estimation model. The training method includes: acquiring model training data, which includes corresponding and preprocessed camera point cloud data and radar point cloud data; training a pre-constructed first training module based on the camera point cloud data to obtain a teacher model; training a pre-constructed second training module based on the radar point cloud data to obtain a student model; and training the student model based on the radar point cloud data to obtain a target pose estimation model. The technical solution of this application can improve the accuracy of human pose recognition results based on sparse point clouds acquired by existing millimeter-wave radar.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a method for training and recognizing a point cloud human pose estimation model based on knowledge transfer. Background Technology

[0002] Human pose estimation is fundamental to many research areas. In practical applications, it often transforms into the prediction of key human points, such as in behavior analysis, gait recognition, and person tracking. These all require precise key point coordinates for further analysis and judgment. Current technologies commonly use millimeter-wave radar to acquire point cloud data for pose estimation. However, most single-chip radars suffer from limited antennas and sparse point cloud density, resulting in low accuracy in pose estimation based on point cloud data acquired by single-chip radar. Therefore, improving the accuracy of human pose recognition based on the sparse point cloud data acquired by existing millimeter-wave radar has become a pressing technical challenge. Summary of the Invention

[0003] The embodiments of this application provide a training and recognition method for a knowledge transfer point cloud human pose estimation model, which can at least to some extent improve the accuracy of human pose recognition results based on sparse point clouds acquired by existing millimeter-wave radar.

[0004] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0005] According to one aspect of the embodiments of this application, a method for training a knowledge transfer point cloud human pose estimation model is provided, comprising:

[0006] Acquire model training data, which includes corresponding and preprocessed camera point cloud data and radar point cloud data;

[0007] The first training module is trained based on the camera point cloud data. The Euclidean distance between the predicted and true values ​​of human joints output by the first training module and the velocity weights corresponding to the true values ​​are weighted and averaged to obtain the first loss term corresponding to the first training module. The first training module is then trained based on the first loss term to obtain the teacher model.

[0008] The pre-constructed second training module is trained based on the radar point cloud data. The mean square error between the output of the second training module and the output of the teacher model is used as the second loss term corresponding to the second training module. The second training module is then trained based on the second loss term to obtain the student model.

[0009] The student model is further optimized based on the radar point cloud data. The Euclidean distance between the predicted and actual values ​​of the human joints output by the student model and the velocity weights corresponding to the actual values ​​are weighted and averaged to obtain the third loss term of the student model. The student model is then optimized a second time based on the third loss term to obtain the target pose estimation model after point cloud optimization.

[0010] According to one aspect of the embodiments of this application, a method for recognizing human pose from point clouds is provided, the method comprising:

[0011] Acquire radar point cloud data to be identified;

[0012] The radar point cloud data to be identified is input into a pre-trained attitude estimation model so that the attitude estimation model outputs the corresponding attitude recognition result. The attitude estimation model is trained by the training method described in the above embodiments.

[0013] According to one aspect of this application, a network model for human pose estimation is provided, comprising: acquiring a training dataset; adding one-dimensional temporal information to the original spatial information to construct a temporally continuous spatiotemporal sequence of point cloud data to determine the time of the millimeter-wave point cloud; increasing the number of point cloud channels using DenseBlock, extracting point cloud frame features, and combining them with a feature connection method as input to the next layer; encoding the spatiotemporal local structure in the point cloud video using point 4D convolution to embed the spatiotemporal local structure presented in the point cloud video, combining positional information into the point cloud features, and then using a Transformer to capture the appearance and motion information in the entire video through a self-attention mechanism on the embedded local features, thereby better extracting the dynamic features of the point cloud; segmenting the point cloud data processed by P4Transformer according to a previously calculated and saved list of point cloud counts per frame, determining the maximum, minimum, and average values ​​of each frame after determining the point cloud in each frame, emphasizing the point cloud features and fixing the number of points in each frame through this hybrid pooling connection method, and finally completing the extraction of point cloud features of T frames with a length of C, providing point cloud features for subsequent pose prediction networks. The TTransformer network is trained using camera point cloud information. Then, the first half of the STransformer network, i.e. the feature extraction part, is trained using the TTransformer network. First, the TTransformer network extracts feature information from the camera point cloud. Then, the STransformer network extracts feature information using millimeter-wave point cloud, which is from the same source as the camera point cloud, and optimizes the difference between the two features. Finally, the STransformer network is fine-tuned using millimeter-wave point cloud information, and the 3D human pose based on skeletal point representation is output through dual-stream data fusion.

[0014] According to one aspect of the embodiments of this application, a training apparatus for a knowledge transfer point cloud human pose estimation model is provided, comprising:

[0015] The first acquisition module is used to acquire model training data, which includes corresponding and preprocessed camera point cloud data and radar point cloud data.

[0016] The model training module is used to train a pre-constructed first training module based on the camera point cloud data. It calculates a weighted average of the Euclidean distance between the predicted and actual values ​​of human joints output by the first training module and the corresponding velocity weights to obtain a first loss term for the first training module. The first training module is then trained based on this first loss term to obtain a teacher model. Next, a pre-constructed second training module is trained based on the radar point cloud data. The mean square error between the output of the second training module and the output of the teacher model is used as a second loss term for the second training module. The second training module is then trained based on this second loss term to obtain a student model. Finally, the student model is further optimized based on the radar point cloud data. The Euclidean distance between the predicted and actual values ​​of human joints output by the student model and the corresponding velocity weights is calculated as a weighted average to obtain a third loss term for the student model. The student model is then further optimized based on this third loss term to obtain a target pose estimation model optimized by the point cloud.

[0017] According to one aspect of the embodiments of this application, a point cloud human pose recognition device is provided, comprising:

[0018] The second acquisition module is used to acquire radar point cloud data to be identified;

[0019] The data processing module is used to input the radar point cloud data to be identified into a pre-trained attitude estimation model, so that the attitude estimation model outputs the corresponding attitude recognition result. The attitude estimation model is trained by the training method described in the above embodiments.

[0020] According to one aspect of the embodiments of this application, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the training method for a point cloud human pose estimation model based on knowledge transfer and the point cloud human pose recognition method as described in the above embodiments.

[0021] According to one aspect of the embodiments of this application, an electronic device is provided, including: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the training method for a knowledge transfer-based point cloud human pose estimation model and the point cloud human pose recognition method as described in the above embodiments.

[0022] According to one aspect of the embodiments of this application, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the training method for the knowledge transfer point cloud human pose estimation model and the point cloud human pose recognition method provided in the above embodiments.

[0023] In some embodiments of this application, the technical solutions are as follows: Model training data is acquired, including corresponding and preprocessed camera point cloud data and radar point cloud data. A pre-constructed first training module is trained based on the camera point cloud data. The Euclidean distance between the predicted and true values ​​of the human joints output by the first training module, along with the velocity weights corresponding to the true values, is weighted and averaged to obtain a first loss term for the first training module. The first training module is then trained based on the first loss term to obtain a teacher model. Next, a pre-constructed second training module is trained based on the radar point cloud data. The mean square error between the output of the second training module and the output of the teacher model is used as a second loss term for the second neural network. The second training module is then trained based on the second loss term to obtain a more accurate pose estimation model, i.e., a student model. The student model is then further optimized based on the radar point cloud data. The Euclidean distance between the predicted and true values ​​of the human joints output by the student model, along with the velocity weights corresponding to the true values, is weighted and averaged to obtain a third loss term for the student model. The student model is then further optimized based on the third loss term to obtain a target pose estimation model. Therefore, the target attitude estimation model obtained by fusion training using dual-channel data (i.e., camera point cloud data and radar point cloud data) can improve the accuracy of attitude recognition results in practical applications, based on the point cloud data acquired by millimeter-wave radar alone.

[0024] In addition, by using the speed-optimized MPJPE (mean (per) joint position error) as the loss term for model training, if only some skeletal points move during human movement, the model can pay more attention to the skeletal points of the moving parts with speed, thereby improving the accuracy of pose recognition results.

[0025] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0026] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0027] Figure 1 A flowchart illustrating a training method for a point cloud human pose estimation model based on knowledge transfer according to an embodiment of this application is shown.

[0028] Figure 2 This paper illustrates a network architecture diagram of a point cloud human pose estimation model for knowledge transfer according to an embodiment of this application.

[0029] Figure 3 A block diagram of a training apparatus for a point cloud human pose estimation model based on knowledge transfer according to an embodiment of this application is shown.

[0030] Figure 4 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown. Detailed Implementation

[0031] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art.

[0032] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0033] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0034] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0035] Figure 1 The diagram illustrates a training method for a point cloud human pose estimation model based on knowledge transfer according to an embodiment of this application. This method can be applied to a terminal device or a server. The terminal device may include, but is not limited to, one or more of smartphones, tablets, laptops, and desktop computers. The server may be a physical server or a cloud server, etc. This application does not impose any special limitations on these.

[0036] It should be noted that the number of terminal devices or servers can be arbitrary, and this application does not impose any special restrictions on this. For example, the server can be a server cluster composed of multiple servers.

[0037] Please refer to Figure 1 The training method includes at least steps S110 to S140, which are described in detail below (the following explanation uses the application of this method to a terminal device as an example, hereinafter referred to as the terminal):

[0038] In step S110, model training data is obtained, which includes corresponding and preprocessed camera point cloud data and radar point cloud data.

[0039] In this embodiment, the terminal can access model training data stored in its own storage space. This model training data includes corresponding and preprocessed camera point cloud data and radar point cloud data. The camera point cloud data can be acquired by a depth camera, and based on this camera point cloud data, corresponding real skeleton point information can be pre-determined for subsequent training. The radar point cloud data can be obtained by converting Doppler data acquired by millimeter-wave radar. It should be noted that "corresponding" in this application means that the camera point cloud data and radar point cloud data are acquired in the same scene.

[0040] In one embodiment of this application, preprocessing of camera point cloud data and radar point cloud data includes:

[0041] Background segmentation, frame extraction, and point cloud downsampling are performed on the pre-acquired camera point cloud data to obtain camera point cloud data that contains only human bodies and a predetermined number of key points in each frame.

[0042] The pre-acquired millimeter-wave radar data is converted from Doppler data into radar point cloud data, and noise is filtered out from the radar point cloud data.

[0043] The processed camera point cloud data and radar point cloud data are subjected to coordinate matching and stored to obtain model training data.

[0044] In this embodiment, the point cloud data acquired by the depth camera and the millimeter-wave radar are not completely consistent in spatial position under the acquisition environment, and both point cloud data contain noise. Therefore, they need to be preprocessed to facilitate subsequent model training.

[0045] Specifically, the depth camera can provide the approximate outline of the environment within its monitoring range, presented as a dense point cloud. First, the coordinates of human skeleton points in the camera point cloud data are determined. Then, based on these coordinates, the human body point cloud information is derived from the camera point cloud data. This human body point cloud information is then segmented from the background point cloud to obtain camera point cloud data containing only the human body. Furthermore, due to the high density of the camera point cloud data, frame extraction and point cloud downsampling are performed to ensure that each frame contains only a predetermined number (e.g., 512) of key points.

[0046] Next, the acquired millimeter-wave radar data is converted from Doppler data to point cloud data at the board level to obtain radar point cloud data. In one example, the number of points in the acquired radar point cloud data can be adjusted by calculating a threshold. Static clutter filtering and phasor mean cancellation algorithms are then applied to the radar point cloud data to remove noise, obtaining the desired radar point cloud data. Then, the processed camera point cloud data and radar point cloud data are subjected to coordinate matching to align them. The aligned camera point cloud data and radar point cloud data are stored to obtain model training data.

[0047] In one embodiment of this application, the processed camera point cloud data and radar point cloud data are subjected to coordinate matching processing and stored to obtain model training data, including:

[0048] Based on the processed camera point cloud data and radar point cloud data, each point cloud is divided into head point cloud and other part point clouds.

[0049] The centroids of the head point cloud and the point clouds of other parts are calculated respectively, and the sliding window smoothing algorithm is applied to the centroids to suppress noise.

[0050] The camera point cloud data and the radar point cloud data are loaded into the same coordinate system, and the Kabsch point cloud registration algorithm is used to make the overlap of their centroid trajectories meet the predetermined requirements in order to achieve the target state.

[0051] Obtain the rotation matrix corresponding to the target state, and adjust the camera point cloud data and radar point cloud data according to the rotation matrix. Store the adjusted camera point cloud data and radar point cloud data as model training data.

[0052] In this embodiment, the X, Y, and Z axes defined by the hardware-level depth camera and the millimeter-wave radar are not consistent, and their relative positions are difficult to maintain during each data acquisition. Therefore, the idea of ​​center alignment of moving objects is adopted to replace absolute coordinate alignment and match, thus completing the alignment of point cloud data. Specifically, the terminal can acquire pre-processed camera point cloud data and millimeter-wave radar point cloud data, divide each point cloud into head point cloud and other part point clouds, and calculate the center motion trajectory of the centroid of the head point cloud and other part point clouds over time. A sliding window smoothing algorithm is then applied to the center motion trajectory to further suppress noise.

[0053] Then, the camera point cloud data and radar point cloud data are loaded into the same coordinate system, and the Kabsch point cloud registration algorithm is used to maximize the overlap of their center motion trajectories to achieve the target state, i.e., the best fit. Next, the rotation matrix corresponding to the target state is obtained, and the original point cloud data is rotated according to this rotation matrix for matching, thereby achieving coordinate matching between the camera point cloud data and the radar point cloud data. The coordinate-matched camera point cloud data and radar point cloud data are stored as subsequent model training data.

[0054] Please continue to refer to this. Figure 1 In step S120, the pre-constructed first training module is trained based on the camera point cloud data. The Euclidean distance between the predicted and true values ​​of the human joints output by the first training module and the velocity weights corresponding to the true values ​​are weighted and averaged to obtain the first loss term corresponding to the first training module. The first training module is then trained based on the first loss term to obtain the teacher model.

[0055] In this embodiment, those skilled in the art can pre-build a first training module to train the teacher model. Specifically, camera point cloud data from the model training data is input into the pre-built first training module so that the first training module outputs predicted values ​​of human joint points.

[0056] It should be noted that during movement (such as waving a hand), only some skeletal points move, resulting in a relatively small MPJPE value when averaged across all skeletal points. Therefore, to increase attention to skeletal points in the dynamic parts with velocity, this application proposes a velocity-weighted hybrid MPJPE. Specifically, the Euclidean distance between the predicted and actual values ​​of human joint points output by the first training module, along with the velocity weights corresponding to the actual values, is weighted and averaged to obtain the first loss term for the first training module. The parameters of the first training module are then adjusted and optimized based on this first loss term, and the optimized first training module is used as the teacher model. The use of a velocity-weighted optimized MPJPE index as the first loss term to train the first training module results in better performance for attitude estimation based on millimeter-wave radar.

[0057] First, the input data is standardized by subtracting the mean of the coordinates of each joint. Then, the Euclidean distance between adjacent frames is calculated, and the distance tensor is expanded. Finally, the distances between adjacent frames are standardized again, the average distances between all frames are calculated, and the velocity weights are obtained by dividing the average distance. This scales the distance values ​​to a relatively consistent range. Here, each frame represents human pose data at a specific point in time, containing key information about the human skeleton.

[0058] In step S130, the pre-constructed second training module is trained based on the radar point cloud data. The mean square error between the output of the second training module and the output of the teacher model is used as the second loss term corresponding to the second training module. The second training module is then trained based on the second loss term to obtain the student model.

[0059] In this embodiment, a pre-constructed second training module is trained based on radar point cloud data. The mean square error between the output of the second training module and the output of the teacher model is calculated and used as the second loss term corresponding to the second training module. The second training module is then trained based on the second loss term, so that the trained second training module is as close as possible to the performance of the teacher model, and thus serves as the student model.

[0060] Please continue to refer to this. Figure 1 In step S140, the student model is optimized again based on the radar point cloud data. The Euclidean distance between the predicted and actual values ​​of the human joints output by the student model and the velocity weights corresponding to the actual values ​​are weighted and averaged to obtain the third loss term corresponding to the student model. The student model is then optimized a second time based on the third loss term to obtain the target attitude estimation model after point cloud optimization.

[0061] In this embodiment, after obtaining the student model, the terminal continues to train the student model based on radar point cloud data. At this time, the Euclidean distance between the predicted value and the true value of the human joint point output by the student model and the velocity weight corresponding to the true value are weighted and averaged to obtain the third loss term corresponding to the student model. Then, the student model is optimized a second time based on the third loss term to obtain the target pose estimation model after point cloud optimization.

[0062] Therefore, by using the concept of knowledge transfer and employing dual-path data input fusion training, the target pose estimation model obtained through fusion training can be greatly improved in judging human pose when only radar point cloud data is input, thus improving the accuracy of pose recognition results.

[0063] In one embodiment of this application, the first training module and the second training module have the same architecture, both including a connected data format processing module, a point cloud feature extraction network, and a pose estimation network.

[0064] The data format processing module (PreDeal module) converts the point cloud data format and adds one-dimensional temporal information to the original spatial information of the point cloud data to construct a temporally continuous spatiotemporal sequence of point cloud data, thereby determining the time information corresponding to the point cloud data. It should be noted that because the number of point clouds acquired in each frame of the millimeter-wave radar is inconsistent, when data is extracted from the radar point cloud data and stitched together into the same point cloud list, it will be impossible to distinguish the frame corresponding to each point; for example, it will be impossible to determine which points are in the first frame and which are in the second frame.

[0065] Please refer to Figure 2 While loading radar point cloud data, one-dimensional time data t is added. For example, a batch of point cloud data (3, n) read from the dataset, where 3 represents the XYZ 3D information and n represents the number of uncertain point clouds input at a certain moment, is concatenated using a matrix concatenation method. An increasing number multiplied by an identity matrix is ​​appended after XYZ to represent the time information t, forming a (4, n) matrix. Physically, this concatenates the coordinates of each point with an increasing time index t, transforming each segmented point cloud data into a 4D tensor form. This ensures that the sequential information of the radar point cloud data is preserved while the data is read, facilitating subsequent frame-by-frame feature extraction. In one example, the input point cloud data can be segmented and saved into multiple frames according to a specified frame length to complete the data format preprocessing.

[0066] The point cloud feature extraction network (FramePointNet) consists of three parts: connected dense blocks (DenseBlock), a point 4D Transformer model (P4Transformer), and a frame pooling system (FramePool).

[0067] Specifically, DenseBlock is derived from the dense blocks in DenseNet. DenseNet is a feedforward convolutional neural network that connects each layer in a dense block to all preceding layers along the channel dimension, improving the information flow between layers. DenseBlock is a dense block containing many layers, each with the same feature map size, and the layers are densely connected.

[0068] The densely connected layers within each dense block generate a series of dense feature maps. These feature maps are concatenated dimensionally to form the output of the dense block. These feature maps contain information from all preceding layers and are considered local features of the dense block. In one example, a Bottleneck block can be used inside the DenseBlock to reduce the number of features and thus improve computational efficiency; specifically, a 1x1 convolution is added to the original network structure. After the dense block, an intermediate layer can be optionally inserted to adjust the dimension and size of the feature maps. Following this, a fully connected layer (MLP) is added to further process the features. The fully connected layer is used for non-linear transformations, feature extraction, and combination, and can include one or more linear layers with activation functions, as well as optional regularization layers (such as batch normalization layers or Dropout layers). After the fully connected layer, for classification tasks, a softmax activation function is used as the final classifier, generating a probability distribution for each class.

[0069] Therefore, DenseNet improves the efficiency of information and gradient transmission in the network. Each layer can directly obtain the gradient from the loss function and the input signal, so as to train deeper networks.

[0070] Furthermore, when inconsistencies occur between different point cloud frames, point cloud videos exhibit an irregular and disordered spatial state. Point tracking techniques, often used to capture dynamic information in point cloud videos, rely on point color for tracking and may therefore be unable to handle colorless point clouds. Since points may flow in or out across frames, calculating accurate point trajectories is extremely difficult. Therefore, a Point 4D Transformer (P4DTransformer) network is used to model the original point cloud video. The P4Transformer network consists of two main parts: a point 4D convolution and a Transformer model.

[0071] Specifically, unlike grid-based convolution, point-based convolution should avoid empty regions. Therefore, local spatiotemporal regions need to be generated before performing point-based 4D convolution. Specifically, this is first done based on the time step s. t Select a subset of frames. Next, use farthest-point sampling (FPS) for each frame, N′ = N / s. s The points are sampled a second time, where s sIt is the spatial secondary sampling rate. Then these secondary sampling points are transferred to a distance r. t The most recent frame. The original and transferred subsampled points form the central axis of a spatiotemporal local region. Finally, based on the spatial radius r of each selected subsampled point. s or distance r t The most recent frame is used for spatial neighborhood search.

[0072] Point 4D convolution encodes the spatiotemporal local structure in point cloud videos, which is then used to embed the spatiotemporal local structure presented in the point cloud video. Let... Let N represent the coordinates and features of a point in the t-th frame of a point cloud video, where N and C represent the number of points and the number of feature channels, respectively. The key to convolution is learning a kernel (including direction and magnitude) of all displacements from the central grid to adjacent grids, and then applying it to the grid features to capture local structure.

[0073] Point 4D convolution is described as follows:

[0074]

[0075] Where (x,y,z)∈P t ,(δ x ,δ y ,δ z ,δ t ) represents spatiotemporal displacement. Σ represents the features of a point at location (x, y, z, t) in a point cloud video. Σ can be implemented using different pooling methods, namely sum pooling, max pooling, and average pooling. G is the spatiotemporal local region surrounding the point (x, y, z, t). It's important to note that since space and time are orthogonal and independent, G can be decomposed into a series of spatial regions defined by a spatial radius r. s and time radius r t definition.

[0076] Because displacements in grid data are both discrete and regular, traditional convolution algorithms can directly learn a kernel ζ for all displacements within a region. However, point coordinates are continuous and irregular, and the number of possible displacements is infinite. Therefore, a kernel can be generated indirectly through a function, rather than being learned directly. Specifically, Yes (δ) x ,δ y ,δ z ,δ t The parameterized function is used to generate the kernel based on the displacement.

[0077] The Transformer model captures appearance and motion information from the entire point cloud video by applying a self-attention mechanism to the embedded local features, thus better extracting the dynamic features of the point cloud. After point 4D convolution, the spatiotemporal local region of frame t is encoded as F′. t This approach allows for the merging of related regions based on similarity, rather than explicit tracking, since similar local regions have similar representations. Furthermore, point locations also reflect relationships between local regions, which can be leveraged to enhance the representation of self-attention. Therefore, the anchor coordinates (x, y, z, t) and local region features are combined as input to the Transformer model:

[0078]

[0079] in It is the weight for transforming 4D coordinates. This is the input for self-attention. Self-attention can be described as mapping a query and a set of key-value pairs to an output, where the query, key, and value are generated from the input itself, and the output is calculated as a weighted sum of these values:

[0080] Q = W q ·I,K=W k ·I,V=W v ·I,

[0081]

[0082] O = V·attention(Q,K),

[0083] in C k C v These are the dimensions of the key and the dimension of the value, respectively. First, a query is generated based on the input I. key value Then, calculate the dot product of queries with all keys and apply the softmax function to obtain the weights. Finally, output The calculation is a weighted sum of values ​​V. Specifically, for a point (x, y, z, t), its new feature is calculated as:

[0084] O (x,y,z,t) =∑attention(Q,K) (x,y,z,t),(x′,y′,z′,t′) ×V (x′,y′,z′,t′)

[0085] Where (x′,y′,z′,t′) is the set of 4D coordinates belonging to I. The Transformer updates each local feature by adding more information from similar or related regions, max pooling merges the local features into global features, and then a multilayer perceptron (MLP) maps them to action prediction.

[0086] Compared to traditional point tracking methods, the P4Transformer model avoids potential difficulties during tracking, such as points appearing or disappearing between different frames. Furthermore, the P4Transformer model can handle point clouds without color information and exhibits excellent performance in tasks such as 3D action recognition and 4D semantic segmentation. Therefore, the P4Transformer model holds great potential for point cloud video modeling.

[0087] Next, the FramePool function performs frame-by-frame pooling to extract global point cloud features. For each frame, which may have a higher or lower number of points, a hybrid pooling method is used to unify the number of points per frame to a fixed value. First, the point cloud data processed by the P4Transformer model is segmented according to a previously calculated and saved list of point cloud counts for each frame. After determining the point cloud for each frame, the maximum, minimum, and average number of points in each frame are calculated. This hybrid pooling connection method emphasizes point cloud features and fixes the number of points per frame, ultimately extracting point cloud features for T frames of length C, providing point cloud features for the subsequent pose estimation network.

[0088] The PoseNet network includes at least one two-stream spatiotemporal attention mechanism module. Each module comprises two spatiotemporal attention branches: one for time followed by space, and the other for space followed by time. The spatial attention mechanism aims to build connections between multiple keypoint clouds over a period of time; the temporal attention mechanism aims to focus on the spatial changes of the same keypoint cloud at different times.

[0089] In practical applications, a two-stream spatiotemporal Transformer (DSTformer) is used as a motion encoder to capture long-range spatiotemporal relationships between skeletal keypoints. The DSTformer is a two-stream spatiotemporal Transformer that serves as a general backbone for human motion modeling and possesses an adaptive fusion mechanism. It consists of N two-stream fusion modules, each containing two branches: spatial or temporal multi-head self-attention (MHSA) and multilayer perceptron (MLP). Spatial MHSA (S-MHSA) describes the relationships between different joints within a time step, while temporal MHSA (T-MHSA) describes the motion of a single joint.

[0090] The DSTformer takes a 2D skeleton as input and outputs 3D motion. Building upon intra-frame and inter-frame body joint interactions obtained through spatial and temporal multi-head attention networks (MHSA), it incorporates basic building blocks to fuse spatial and temporal information from the stream. These spatial and temporal MHSA blocks are stacked in different orders, forming two parallel computation branches. The output features of the two branches are fused using adaptive weights predicted by an attention regressor. Then, the dual-stream fusion module repeats this process N times, fusing the two streams together. The fusion weights are dynamically balanced based on the spatiotemporal characteristics of the input, adapting to different inputs.

[0091] PoseNet and DSTformer share a similar structure. First, a TTransformer network (a temporal Transformer) is trained using camera point cloud data. Then, the first half of a STransformer network (a spatial Transformer), specifically the feature extraction part, is trained using the TTransformer network. The TTransformer network first extracts features from the camera point cloud data, and then the STransformer network extracts features from radar point cloud data (which shares the same source as the camera point cloud data), optimizing the difference between the two sets of features. Finally, the STransformer network is fine-tuned using radar point cloud data. Specifically, millimeter-wave radar point cloud data can be used as input to the model. Labels are then used to train the model, and backpropagation and optimization algorithms are used to update the model's parameters, thereby fine-tuning the spatial attention mechanism. Next, the output features of the two branches are fused using adaptive weights predicted by the attention regressor, and finally, a linear layer outputs a skeletal point image of the human body.

[0092] In one embodiment of this application, a method for recognizing human pose from point clouds is also provided, the method comprising:

[0093] Acquire radar point cloud data to be identified;

[0094] The radar point cloud data to be identified is input into a pre-trained attitude estimation model so that the attitude estimation model outputs the corresponding attitude recognition result. The attitude estimation model is trained by the training method described in the above embodiments.

[0095] Therefore, the pose estimation model obtained through the above training method can improve the accuracy of human pose recognition results based on the sparse point cloud acquired by existing millimeter-wave radar. It should be noted that for details not disclosed in the embodiments of the human pose recognition method of this application, please refer to the embodiments of the training method for the knowledge transfer-based pose estimation model described above.

[0096] The following describes an apparatus embodiment of this application, which can be used to perform the methods described in the above embodiments of this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the methods described in the above embodiments of this application.

[0097] Figure 3 A block diagram of a training apparatus for a pose estimation model of knowledge transfer according to an embodiment of this application is shown.

[0098] Reference Figure 3 As shown, a training apparatus for a knowledge transfer-based pose estimation model according to an embodiment of this application includes:

[0099] The first acquisition module is used to acquire model training data, which includes corresponding and preprocessed camera point cloud data and radar point cloud data.

[0100] The model training module is used to train a pre-constructed first training module based on the camera point cloud data. It calculates a weighted average of the Euclidean distance between the predicted and actual values ​​of human joints output by the first training module and the corresponding velocity weights to obtain a first loss term for the first training module. The first training module is then trained based on this first loss term to obtain a teacher model. Next, a pre-constructed second training module is trained based on the radar point cloud data. The mean square error between the output of the second training module and the output of the teacher model is used as a second loss term for the second training module. The second training module is then trained based on this second loss term to obtain a student model. Finally, the student model is further optimized based on the radar point cloud data. The Euclidean distance between the predicted and actual values ​​of human joints output by the student model and the corresponding velocity weights is calculated as a weighted average to obtain a third loss term for the student model. The student model is then further optimized based on this third loss term to obtain a target pose estimation model optimized by the point cloud.

[0101] In one embodiment of this application, the first training module and the second training module include a data format processing module, a point cloud feature extraction network, and a pose estimation network connected to each other;

[0102] The data format processing module converts the point cloud data into a format and adds one-dimensional time information to the original spatial information of the point cloud data to construct a time-continuous point cloud spatiotemporal sequence data.

[0103] The point cloud feature extraction network includes interconnected dense blocks, a point 4D Transformer model, and frame pooling;

[0104] The pose estimation network includes at least one dual-stream spatiotemporal attention mechanism module, and each dual-stream spatiotemporal attention mechanism module includes two spatiotemporal attention branches.

[0105] In one embodiment of this application, the preprocessing includes:

[0106] Background segmentation, frame extraction, and point cloud downsampling are performed on the pre-acquired camera point cloud data to obtain camera point cloud data that contains only human bodies and a predetermined number of key points in each frame.

[0107] The pre-acquired millimeter-wave radar data is converted from Doppler data into radar point cloud data, and noise is filtered out from the radar point cloud data.

[0108] The processed camera point cloud data and radar point cloud data are subjected to coordinate matching and stored to obtain model training data.

[0109] In one embodiment of this application, the processed camera point cloud data and radar point cloud data are subjected to coordinate matching processing and stored to obtain model training data, including:

[0110] Based on the processed camera point cloud data and radar point cloud data, each point cloud is divided into head point cloud and other part point clouds.

[0111] The centroids of the head point cloud and the point clouds of other parts are calculated respectively, and the sliding window smoothing algorithm is applied to the centroids to suppress noise.

[0112] The camera point cloud data and the radar point cloud data are loaded into the same coordinate system, and the Kabsch point cloud registration algorithm is used to maximize the overlap of their center motion trajectories in order to achieve the target state.

[0113] Obtain the rotation matrix corresponding to the target state, and adjust the camera point cloud data and radar point cloud data according to the rotation matrix. Store the adjusted camera point cloud data and radar point cloud data as model training data.

[0114] In one embodiment of this application, a point cloud human pose recognition device is also provided, comprising:

[0115] The second acquisition module is used to acquire radar point cloud data to be identified;

[0116] The processing module is used to input the radar point cloud data to be identified into a pre-trained attitude estimation model so that the attitude estimation model outputs the corresponding attitude recognition result. The attitude estimation model is trained by the training method as described in any one of claims 1 to 4.

[0117] Figure 4 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown.

[0118] It should be noted that, Figure 4 The computer system of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0119] like Figure 4 As shown, the computer system includes a Central Processing Unit (CPU) 301, which can perform various appropriate actions and processes based on programs stored in Read-Only Memory (ROM) 302 or programs loaded from storage portion 308 into Random Access Memory (RAM) 303, such as performing the methods described in the above embodiments. The RAM 303 also stores various programs and data required for system operation. The CPU 301, ROM 302, and RAM 303 are interconnected via a bus 304. An Input / Output (I / O) interface 305 is also connected to the bus 304.

[0120] The following components are connected to I / O interface 305: an input section 306 including a keyboard, mouse, etc.; an output section 307 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to I / O interface 305 as needed. Removable media 311, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 310 as needed so that computer programs read from them can be installed into storage section 308 as needed.

[0121] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 309, and / or installed from removable medium 311. When the computer program is executed by central processing unit (CPU) 301, it performs various functions defined in the system of this application.

[0122] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. The transmitted data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0123] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0124] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0125] In another aspect, this application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods described in the above embodiments.

[0126] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0127] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the method according to the embodiments of this application.

[0128] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0129] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A training method for a point cloud human pose estimation model based on knowledge transfer, characterized in that, include: Acquire model training data, which includes corresponding and preprocessed camera point cloud data and radar point cloud data; The first training module is trained based on the camera point cloud data. The Euclidean distance between the predicted and true values ​​of human joints output by the first training module and the velocity weights corresponding to the true values ​​are weighted and averaged to obtain the first loss term corresponding to the first training module. The first training module is then trained based on the first loss term to obtain the teacher model. The pre-constructed second training module is trained based on the radar point cloud data. The mean square error between the output of the second training module and the output of the teacher model is used as the second loss term corresponding to the second training module. The second training module is then trained based on the second loss term to obtain the student model. The student model is further optimized based on the radar point cloud data. The Euclidean distance between the predicted and actual values ​​of the human joints output by the student model and the velocity weights corresponding to the actual values ​​are weighted and averaged to obtain the third loss term of the student model. The student model is then optimized a second time based on the third loss term to obtain the target pose estimation model after point cloud optimization.

2. The method according to claim 1, characterized in that, The first training module and the second training module include a data format processing module, a point cloud feature extraction network, and a pose estimation network that are connected to each other; The data format processing module converts the point cloud data into a format and adds one-dimensional time information to the original spatial information of the point cloud data to construct a time-continuous point cloud spatiotemporal sequence data. The point cloud feature extraction network includes interconnected dense blocks, a point 4D Transformer model, and frame pooling; The pose estimation network includes at least one dual-stream spatiotemporal attention mechanism module, and each dual-stream spatiotemporal attention mechanism module includes two spatiotemporal attention branches.

3. The method according to claim 1 or 2, characterized in that, The preprocessing includes: Background segmentation, frame extraction, and point cloud downsampling are performed on the pre-acquired camera point cloud data to obtain camera point cloud data that contains only human bodies and a predetermined number of key points in each frame. The pre-acquired millimeter-wave radar data is converted from Doppler data into radar point cloud data, and noise is filtered out from the radar point cloud data. The processed camera point cloud data and radar point cloud data are subjected to coordinate matching and stored to obtain model training data.

4. The method according to claim 3, characterized in that, The processed camera point cloud data and radar point cloud data are subjected to coordinate matching and stored to obtain model training data, including: Based on the processed camera point cloud data and radar point cloud data, each point cloud is divided into head point cloud and other part point clouds. The centroids of the head point cloud and the point clouds of other parts are calculated respectively, and the sliding window smoothing algorithm is applied to the centroids to suppress noise. The camera point cloud data and the radar point cloud data are loaded into the same coordinate system, and the Kabsch point cloud registration algorithm is used to maximize the overlap of their center motion trajectories in order to achieve the target state. Obtain the rotation matrix corresponding to the target state, and adjust the camera point cloud data and radar point cloud data according to the rotation matrix. Store the adjusted camera point cloud data and radar point cloud data as model training data.

5. A method for recognizing human pose in point clouds, characterized in that, include: Acquire radar point cloud data to be identified; The radar point cloud data to be identified is input into a pre-trained attitude estimation model so that the attitude estimation model outputs the corresponding attitude recognition result. The attitude estimation model is trained by the training method as described in any one of claims 1 to 4.

6. A training device for a knowledge transfer-based point cloud human pose estimation model, characterized in that, include: The first acquisition module is used to acquire model training data, which includes corresponding and preprocessed camera point cloud data and radar point cloud data. The model training module is used to train a pre-constructed first training module based on the camera point cloud data. It calculates a weighted average of the Euclidean distance between the predicted and actual values ​​of human joints output by the first training module and the corresponding velocity weights to obtain a first loss term for the first training module. The first training module is then trained based on this first loss term to obtain a teacher model. Next, a pre-constructed second training module is trained based on the radar point cloud data. The mean square error between the output of the second training module and the output of the teacher model is used as a second loss term for the second training module. The second training module is then trained based on this second loss term to obtain a student model. Finally, the student model is further optimized based on the radar point cloud data. The Euclidean distance between the predicted and actual values ​​of human joints output by the student model and the corresponding velocity weights is calculated as a weighted average to obtain a third loss term for the student model. The student model is then further optimized based on this third loss term to obtain a target pose estimation model optimized by the point cloud.

7. The apparatus according to claim 6, characterized in that, The first training module and the second training module include a data format processing module, a point cloud feature extraction network, and a pose estimation network that are connected to each other; The data format processing module converts the point cloud data into a format and adds one-dimensional time information to the original spatial information of the point cloud data to construct a time-continuous point cloud spatiotemporal sequence data. The point cloud feature extraction network includes interconnected dense blocks, a point 4D Transformer model, and frame pooling; The pose estimation network includes at least one dual-stream spatiotemporal attention mechanism module, and each dual-stream spatiotemporal attention mechanism module includes two spatiotemporal attention branches.

8. A point cloud human posture recognition device, characterized in that, include: The second acquisition module is used to acquire radar point cloud data to be identified; The processing module is used to input the radar point cloud data to be identified into a pre-trained attitude estimation model so that the attitude estimation model outputs the corresponding attitude recognition result. The attitude estimation model is trained by the training method as described in any one of claims 1 to 4.

9. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 4 and 5.

10. An electronic device, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the method as described in any one of claims 1 to 4 and 5.

Images