Multi-person point cloud human body posture reconstruction method and system based on millimeter wave radar

Abstract

The invention discloses a multi-person point cloud human body posture reconstruction method and system based on a millimeter wave radar, and the method comprises the steps: firstly generating a three-dimensional point cloud through radar echo preprocessing, and achieving the noise filtering and target separation through the combination of detection region division and a multi-stage DBSCAN clustering algorithm; secondly, a multi-scale feature fusion neural network is constructed, projection plane interference is isolated through grouping convolution, space details are reserved based on a deep high-resolution module, key feature extraction is enhanced through an improved ResNet18 embedded coordinate attention and channel attention mechanism, and end-to-end human body 23 joint three-dimensional coordinate regression is achieved. The system supports synchronous reconstruction of postures of multiple persons through real-time point cloud classification and parallel reasoning, and the frame rate reaches 30-40 FPS in a complex scene. The method gives consideration to privacy protection and high robustness, and can be widely applied to the fields of smart home, smart health care and security monitoring.

Description

Technical Field

[0001] The present invention relates to the fields of millimeter-wave radar smart home and smart health care technology, and in particular to a method and system for reconstructing multi-person point cloud human postures based on millimeter-wave radar. Background Art

[0002] With the continuous development of intelligent driving, intelligent security, and intelligent healthcare, gesture recognition technology has become a key tool for human-environment interaction. Current gesture recognition methods primarily rely on vision-based technologies, such as RGB images and depth cameras. While these methods offer excellent recognition accuracy, they suffer from significant deficiencies in privacy protection and adaptability to complex environments. For example, low light, strong interference, and fog obstruction can severely impact the effectiveness of vision-based technologies. In contrast, millimeter-wave radar, as an emerging contactless sensing technology, offers new possibilities for gesture recognition due to its excellent penetration, privacy protection, and adaptability to complex environments. However, the sparsity and high noise levels of millimeter-wave radar point cloud data present significant technical challenges when applied to gesture recognition, especially in multi-person scenarios.

[0003] Compared to single-person gesture recognition, gesture reconstruction in multi-person scenarios requires dealing with more complex issues, such as target separation and matching, as well as interference caused by occlusion, overlap, and dynamic interaction. These factors significantly increase the technical difficulty, but also provide new directions for academic research and practical applications. By optimizing the processing algorithm of millimeter-wave radar point cloud data, the accuracy and robustness of non-visual perception technology can be further improved, enabling it to operate stably in dynamic and complex environments. At the same time, this research will also help expand the application boundaries of millimeter-wave radar, extending it from basic target detection to the more complex field of human behavior perception, providing reliable technical support for scenarios such as smart security, elderly fall detection, and hospital patient monitoring.

[0004] Millimeter-wave radar can measure the distance and velocity of targets, exhibiting high sensitivity and anti-interference capabilities, and holds broad application prospects in the field of human posture reconstruction. Furthermore, radar-based sensing technology is less susceptible to environmental conditions such as lighting, temperature, and humidity, eliminating the risk of privacy breaches. Compared to traditional microwave radar, millimeter-wave radars possess the inherent properties of small wavelengths and large bandwidth, making them more amenable to system miniaturization and high-precision target measurement. They also offer the common radar advantages of all-weather and all-day environmental adaptability, non-contact detection, and privacy protection.

[0005] Therefore, there is a need for a method and system that can utilize the excellent detection performance of millimeter-wave radar to achieve non-contact multi-person human posture reconstruction. Summary of the Invention

[0006] In view of this, the purpose of the present invention is to provide a method and system for multi-person point cloud human posture reconstruction based on millimeter-wave radar. This method utilizes the excellent detection performance of millimeter-wave radar to apply non-contact multi-person human posture reconstruction to the fields of smart home and smart health care.

[0007] In order to achieve the above object, the present invention provides the following technical solutions:

[0008] The method for reconstructing human body posture from multi-person point cloud based on millimeter wave radar provided by the present invention comprises the following steps:

[0009] S1: Collects radar echo data of multiple people in the scene;

[0010] S2: Process the radar echo data and construct human body point cloud data;

[0011] S3: Project the point cloud data to the xy plane and perform the DBSCAN clustering algorithm on the two-dimensional plane;

[0012] S4: Design a multi-scale feature extraction and fusion network based on point cloud data, build a backbone network, and obtain a pre-trained pose estimation model through training and optimization;

[0013] S5: Project the corresponding point clouds of each human body after clustering to obtain their respective orthogonal plane images;

[0014] S6: The projected orthogonal plane images are sent to the posture reconstruction network in turn, and the posture reconstruction data corresponding to the human target is obtained after model inference.

[0015] Furthermore, the data processing method in S2 is performed according to the following steps:

[0016] S21: Perform mean cancellation and DC component suppression on the acquired radar raw echo data to remove environmental noise and clutter interference, and perform a two-dimensional Fourier transform to obtain a range-velocity spectrum;

[0017] S22: Based on the range-velocity spectrogram obtained in S21, multiple two-dimensional constant false alarm detections are performed to detect strong target reflection points in the scene and extract the corresponding distance and velocity information. Based on this, MUSIC super-resolution angle measurement is further performed in the elevation and azimuth transmit and receive antenna channel dimensions to obtain the corresponding angle information of the target point, and finally the range and elevation angle position of the target relative to the radar in space are obtained.

[0018] S23: Based on the distance and elevation angle information of the target point cloud relative to the radar, the point cloud data is converted from the polar coordinate system to the three-dimensional Cartesian coordinate system based on the coordinate transformation principle;

[0019] S24: Setting the scene area range to preliminarily filter out the noise point cloud, and further applying the DBSCAN clustering algorithm to remove outliers and noise points.

[0020] Furthermore, the projection and two-dimensional clustering of the human body point cloud data in S3 are performed according to the following steps:

[0021] Based on the point cloud data obtained above, the z-axis coordinate in the three-dimensional point cloud coordinates is removed, and the x-axis and y-axis coordinates are retained as classification data; the projected two-dimensional data is subjected to DBSCAN clustering processing again. In order to suppress point cloud diffusion and point cloud interference caused by multipath reflection, reasonable clustering parameters are set to obtain the number of target categories in the clustered scene and the point cloud data labels corresponding to each category.

[0022] Furthermore, the construction of the feature extraction network and the backbone network in S4 is carried out according to the following steps:

[0023] S41: Grouped convolution processing, which processes three single-channel images and evenly divides the input data into multiple groups in the channel dimension. Each group performs a convolution operation independently to avoid information interference between different channels, reduce model complexity, and retain the characteristic information of each channel;

[0024] S42: Multi-scale feature extraction, introducing a deep high-resolution representation module. By connecting multi-resolution sub-networks in parallel, it maintains high-resolution feature representation and enhances the ability to capture human posture details.

[0025] S43: Backbone network construction, based on the ResNet18 architecture, using the residual learning framework to enhance feature transfer efficiency through shortcut connections, and introducing the channel attention mechanism and coordinate attention mechanism to achieve feature extraction and deep mining, ultimately obtaining the feature vector;

[0026] S44: output of key point estimation, flattening the feature vector obtained in S43, and then sending it to a multi-layer perceptron to process the feature vector to obtain the human body key points corresponding to the current frame.

[0027] Furthermore, the projection of the point cloud data of each human body in S5 is performed according to the following steps:

[0028] According to the corresponding category labels of each human target point cloud in the detection area obtained in S3, the coordinate information of each target's corresponding three-dimensional point cloud in the Cartesian coordinate system is projected onto three sets of two-dimensional xy, xz and yz planes respectively, to achieve the dimensional splitting of the three-dimensional point cloud data. In order to facilitate the calculation and processing of the convolutional neural network, a three-dimensional scene grid with the same proportion as the detection area is created, and the projected two-dimensional point cloud data is further classified into a two-dimensional grid image, and the three-dimensional point cloud is discretized with three sets of images. At the same time, the human joint coordinate information obtained by the motion capture device is used as the control label of the network output, and the point cloud data and the corresponding labels are divided into training set, verification set and test set according to the number of frames;

[0029] Furthermore, the two-dimensional constant false alarm detection is specifically performed according to the following steps:

[0030] In echo data processing, the OS-CFAR algorithm was selected after actual testing and analysis, taking into account the application scenario and the characteristics of the ambient background noise. The algorithm sorts the sampled values ​​of reference cells surrounding the target cell, selects the kth ordered value as the background noise estimate, and multiplies this value by a preset scaling factor to generate a dynamic detection threshold. If the power of a cell exceeds this threshold, the point is identified as a target signal candidate. This effectively suppresses contamination from interfering targets in non-uniform environments such as those with multiple targets or at the edge of clutter. Compared to the traditional averaging method, CA-CFAR is more robust.

[0031] Furthermore, the training of the neural network model includes the following steps:

[0032] After setting the model hyperparameters, the point cloud data contained in the training set is fed into the model and then inferred to obtain the estimated values ​​of the target human key points. The model estimated values ​​and the true value labels corresponding to the radar frames in the training set are then further passed into the loss function to calculate the error loss between the estimated values ​​and the true values. Gradient backpropagation is performed according to the loss results, and the model parameters are updated based on the calculated values ​​during the propagation process. The training process is iterated according to the learning conditions set by the hyperparameters to obtain the neural network model after learning optimization.

[0033] Furthermore, the loss function is designed in the following steps:

[0034] In order to measure the position error and posture error between the estimated output and the true value, it is assumed that P is the true three-dimensional coordinate of the human joint in the Cartesian coordinate system, The three-dimensional coordinates of the human joints output by the network are specifically expressed as follows:

[0035]

[0036] Among them, N represents different joints of the human body;

[0037] Assume Proot 、 The real center point of the human body and the network-inferred center point of the human body:

[0038]

[0039] Describes the offset error of the center point of the human body, position loss L loc It can be expressed as:

[0040]

[0041] Describe the human body posture, that is, the offset error of each joint point, the posture loss L pos It can be expressed as:

[0042]

[0043] Combining position loss and posture loss, the final loss function is as follows:

[0044] L=L loc +L pos

[0045] Among them, L represents the final loss function.

[0046] The multi-person point cloud human posture reconstruction system based on millimeter-wave radar provided by the present invention includes a data acquisition module, a point cloud data generation module, a data preprocessing module, a point cloud extraction module, a point cloud data enhancement module, a multi-person point cloud classification module, a neural network model building module, and a multi-person posture real-time reconstruction module;

[0047] The data acquisition module is used to collect radar original echo signals in a multi-target scene;

[0048] The point cloud data generation module is used to pre-process the original echo data and organize it to obtain a three-dimensional point cloud of the scene;

[0049] The data preprocessing module is used to perform DC clutter filtering and inter-frame cancellation processing on the received data; obtain a two-dimensional range-Doppler joint spectrum through fast Fourier transform;

[0050] The point cloud extraction module is used to perform two-dimensional constant false alarm detection on the two-dimensional joint spectrum, extract possible target points in the range-velocity spectrum and store them in a target point sequence; further obtain the index of the target point corresponding to the distance and velocity dimensions, and extract the corresponding antenna dimension vector in the three-dimensional data cube based on the index; perform the MUSIC super-resolution algorithm in the two antenna channel dimensions of pitch and azimuth to obtain the corresponding pitch angle and azimuth angle, and finally obtain the corresponding point cloud position information and convert it into a Cartesian coordinate system;

[0051] The point cloud data enhancement module is used to filter out interfering point clouds outside the scene and outlier point clouds within the scene to optimize the quality of the human target point cloud;

[0052] The multi-person point cloud classification module is used to cluster point cloud data and match point clouds with human targets in the scene; it achieves multi-person target clustering in complex scenes through three-level progressive processing and outputs optimized point cloud classification results;

[0053] The neural network model building module is used to build a multi-scale feature fusion convolutional neural network and obtain a posture reconstruction network model after training and optimization;

[0054] The multi-person posture real-time reconstruction module is used to input the processed human target point clouds in the scene into the neural network model and reconstruct the human key point data after inference and estimation in real time, and draw the three-dimensional human skeleton in real time based on the key point data to achieve visual display.

[0055] Furthermore, the neural network model building module includes a group convolution module, a high-resolution extraction module, and a backbone network module;

[0056] The grouped convolution module evenly groups the three orthogonally projected single-channel images input in the channel dimension, and performs convolution operations independently on each group;

[0057] The high-resolution extraction module builds a multi-resolution parallel processing architecture based on deep high-resolution representation technology. Through dense interactive connections across resolution sub-networks, it integrates high-order semantic information extracted by low-resolution sub-networks while maintaining the spatial accuracy of the original feature map, forming a hierarchical feature pyramid structure.

[0058] The backbone network module adopts the ResNet18 residual framework, with a coordinate attention mechanism embedded in each residual block to generate an adaptive weight mask to enhance the response of key areas; the deep network embeds a channel attention module to output a high-density feature vector; and through a cascaded fully connected network, the three-dimensional feature tensor is mapped to the three-dimensional coordinate space of the human joint, realizing end-to-end nonlinear regression from multimodal features to posture parameters.

[0059] The beneficial effects of the present invention are:

[0060] The present invention provides a multi-person point cloud human pose reconstruction method and system based on millimeter-wave radar, belonging to the field of millimeter-wave radar intelligent perception and gesture recognition technology. To address the challenges of existing visual technologies, such as poor privacy and adaptability to complex environments, as well as the difficulty of multi-person pose reconstruction due to the sparsity and noise interference of radar point clouds, a multi-stage point cloud enhancement and hierarchical clustering method is proposed. First, a three-dimensional point cloud is generated through radar echo preprocessing. This is then combined with detection region partitioning and a multi-level DBSCAN clustering algorithm to achieve noise filtering and target separation. Next, a multi-scale feature fusion neural network is constructed, using grouped convolution to isolate interference from projection planes. A deep, high-resolution module is used to preserve spatial details. Key feature extraction is enhanced by embedding coordinate attention and channel attention mechanisms in an improved ResNet-18, achieving end-to-end three-dimensional coordinate regression of 23 human joints. Through real-time point cloud classification and parallel inference, the system supports simultaneous multi-person pose reconstruction, achieving a frame rate of 30-40 FPS in complex scenarios. This method, which balances privacy protection with high robustness, is widely applicable in smart homes, smart healthcare, and security monitoring.

[0061] The present invention provides a multi-person point cloud human posture reconstruction method and system based on millimeter-wave radar. Based on the original radar echo signal, radar signal preprocessing and three-dimensional point cloud generation are achieved through DC clutter suppression, incoherent accumulation, signal filtering, MUSIC angle measurement algorithm, and constant false alarm detection. Based on the obtained three-dimensional point cloud of the scene, the target appearance area is further divided to achieve preliminary filtering of interfering point clouds outside the area, and the DBSCAN algorithm is used to further filter out outliers within the area, resulting in an optimized high-quality dense target point cloud. When multiple human targets are present in the scene, the DBSCAN algorithm is again used to perform secondary clustering on the xy plane, and the human targets are matched one-to-one with the point cloud data. After the point clouds corresponding to the human targets are calibrated, the point clouds are projected onto three sets of planes: xy, xz, and yz. The corresponding point cloud projection results are fed into the network in the form of three images for each target. The estimated key point information corresponding to each human target is obtained through real-time inference. This method can accurately and real-timely estimate the posture of multiple targets in a scene, expanding the application of posture estimation in smart homes, smart cities, virtual reality, and other fields. The method also effectively improves the quality of the reflection point cloud of human targets in the scene, providing reliable data input for the subsequent neural network model; the multi-level clustering algorithm design can achieve correct and stable classification of multi-target point clouds in multiple scenes in a short time, optimizing the stability and generalization of multi-person posture information estimation.

[0062] This method can be used for multi-person posture estimation. Without infringing on user privacy, it uses electromagnetic signals to perceive multiple human targets in the scene, and uses signal processing and deep learning methods to achieve real-time human posture reconstruction of multiple targets. It can effectively monitor the dynamics of indoor personnel and can be effectively deployed in various environmental scenarios. It has a wide range of application scenarios in smart monitoring, somatosensory games and other fields.

[0063] This method leverages the excellent detection performance of millimeter-wave radar and focuses on the needs of contactless multi-person human posture reconstruction in the fields of smart home and smart health care to carry out research on multi-person human posture reconstruction technology based on millimeter-wave radar. It not only has high theoretical research value, but also provides new technologies for the actual implementation of millimeter-wave radar technology and human-computer interaction in complex environments.

[0064] Other advantages, objects, and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art upon examination of the following description or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to make the purpose, technical solutions and beneficial effects of the present invention more clear, the present invention provides the following drawings for illustration.

[0066] Figure 1 Flowchart of the multi-person point cloud pose reconstruction method.

[0067] Figure 2 Schematic diagram of 3D point cloud in multi-target scene.

[0068] Figure 3 Flowchart of point cloud enhancement and clustering.

[0069] Figure 4 A schematic diagram of the network structure.

[0070] Figure 5 Schematic diagram of motion capture equipment.

[0071] Figure 6 Schematic diagram of the real-time posture reconstruction effect for multiple people. DETAILED DESCRIPTION

[0072] The present invention will be further described below with reference to the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments are not intended to limit the present invention.

[0073] Example 1

[0074] like Figure 1As shown, the method for reconstructing multi-person point cloud human posture based on millimeter wave radar provided in this embodiment realizes real-time reconstruction of multi-person human posture in multiple scenes based on millimeter wave radar, and specifically includes the following steps:

[0075] S1: Collects radar echo data of multiple people in the scene;

[0076] S2: Process the radar echo data and construct human body point cloud data;

[0077] S3: Project the point cloud data to the xy plane and perform the DBSCAN clustering algorithm on the two-dimensional plane;

[0078] S4: Design a multi-scale feature extraction and fusion network based on point cloud data, build a backbone network, and obtain a pre-trained pose estimation model through training and optimization;

[0079] S5: Project the corresponding point clouds of each human body after clustering to obtain their respective orthogonal plane images;

[0080] S6: The projected orthogonal plane images are sent to the posture reconstruction network in turn, and the posture reconstruction data corresponding to the human target is obtained after model inference.

[0081] The data processing method in S2 of this embodiment is performed according to the following steps:

[0082] S21: Perform mean cancellation and DC component suppression on the acquired radar raw echo data to remove environmental noise and clutter interference, and perform a two-dimensional Fourier transform to obtain a range-velocity spectrum;

[0083] S22: Based on the range-velocity spectrogram obtained in S21, multiple two-dimensional constant false alarm detections are performed to detect strong target reflection points in the scene and extract the corresponding distance and velocity information. Based on this, MUSIC super-resolution angle measurement is further performed in the elevation and azimuth transmit and receive antenna channel dimensions to obtain the corresponding angle information of the target point, and finally the range and elevation angle position of the target relative to the radar in space are obtained.

[0084] S23: Based on the distance and elevation angle information of the target point cloud relative to the radar, the point cloud data is converted from the polar coordinate system to the three-dimensional Cartesian coordinate system based on the coordinate conversion principle;

[0085] S24: Setting the scene area range to preliminarily filter out the noise point cloud, and further applying the DBSCAN clustering algorithm to remove outliers and noise points.

[0086] The projection and two-dimensional clustering of the human body point cloud data in S3 in this embodiment are performed according to the following steps:

[0087] Based on the point cloud data obtained above, the z-axis coordinates in the three-dimensional point cloud coordinates are removed, and only the x-axis and y-axis coordinates are retained as classification data. The projected two-dimensional data is once again subjected to DBSCAN clustering processing. In order to suppress point cloud diffusion and point cloud interference caused by multipath reflection, reasonable clustering parameters are set to obtain the number of target categories in the clustered scene and the point cloud data labels corresponding to each category.

[0088] The construction of the feature extraction network and the backbone network in S4 in this embodiment is carried out according to the following steps:

[0089] S41: Grouped convolution processing, which processes three single-channel images and evenly divides the input data into multiple groups in the channel dimension. Each group performs a convolution operation independently to avoid information interference between different channels, reduce model complexity, and retain the characteristic information of each channel;

[0090] S42: Multi-scale feature extraction. Based on S41, a deep high-resolution representation module is introduced. By connecting multi-resolution sub-networks in parallel, the high-resolution representation of features is maintained, enhancing the ability to capture human posture details.

[0091] S43: Backbone network construction, based on the ResNet18 architecture, using the residual learning framework to enhance feature transfer efficiency through shortcut connections, and introducing the channel attention mechanism and coordinate attention mechanism to achieve feature extraction and deep mining, ultimately obtaining the feature vector;

[0092] S44: output of key point estimation, flattening the feature vector obtained in S43, and then sending it to a multi-layer perceptron to process the feature vector to obtain the human body key points corresponding to the current frame.

[0093] In this embodiment, the projection of the point cloud data of the human body in S5 is performed according to the following steps:

[0094] According to the corresponding category labels of each human target point cloud in the detection area obtained in S3, the coordinate information of each target's corresponding three-dimensional point cloud in the Cartesian coordinate system is projected onto three sets of two-dimensional xy, xz and yz planes respectively, to achieve the dimensional splitting of the three-dimensional point cloud data. In order to facilitate the calculation and processing of the convolutional neural network, a three-dimensional scene grid with the same proportion as the detection area is created, and the projected two-dimensional point cloud data is further classified into a two-dimensional grid image, and the three-dimensional point cloud is discretized with three sets of images. At the same time, the human joint coordinate information obtained by the motion capture device is used as the control label of the network output, and the point cloud data and the corresponding labels are divided into training set, verification set and test set according to the number of frames;

[0095] The two-dimensional constant false alarm detection in this embodiment is specifically performed according to the following steps:

[0096] In echo data processing, the OS-CFAR algorithm was selected after actual testing and analysis, taking into account the application scenario and the characteristics of the ambient background noise. The algorithm sorts the sampled values ​​of reference cells surrounding the target cell, selects the kth ordered value as the background noise estimate, and multiplies this value by a preset scaling factor to generate a dynamic detection threshold. If the power of a cell exceeds this threshold, the point is identified as a target signal candidate. This effectively suppresses contamination from interfering targets in non-uniform environments such as those with multiple targets or at the edge of clutter. Compared to the traditional averaging method, CA-CFAR is more robust.

[0097] The point cloud noise filtering in this embodiment is specifically performed according to the following steps:

[0098] First, the three-dimensional range of the target area in space is pre-set, and the point cloud data outside this range is filtered out to suppress noise interference;

[0099] Next, to further enhance the concentration of the point clouds corresponding to the respective targets, the DBSCAN algorithm based on density clustering is performed on the point clouds after spatial filtering. By setting an appropriate neighborhood radius and minimum number of neighborhood points, the data points connected to the high-density area are divided into clusters. This algorithm can automatically identify clusters of any shape and effectively handle noise outside the cluster.

[0100] Finally, the above processing improves the concentration of the point cloud and effectively suppresses the noise inside and outside the area.

[0101] The training of the neural network model described in this embodiment includes the following steps:

[0102] After setting the model hyperparameters, the point cloud data contained in the training set is fed into the model and then inferred to obtain the estimated values ​​of the target human key points. The model estimated values ​​and the true value labels corresponding to the radar frames in the training set are then further passed into the loss function to calculate the error loss between the estimated values ​​and the true values. Gradient backpropagation is performed according to the loss results, and the model parameters are updated based on the calculated values ​​during the propagation process. The training process is iterated according to the learning conditions set by the hyperparameters to obtain the neural network model after learning optimization.

[0103] The design of the loss function in this embodiment is specifically performed according to the following steps:

[0104] In order to measure the position error and posture error between the estimated output and the true value, it is assumed that P is the true three-dimensional coordinate of the human joint in the Cartesian coordinate system, The three-dimensional coordinates of the human joints output by the network are specifically expressed as follows:

[0105]

[0106] Wherein, N represents different joints of the human body; in this embodiment, N=23;

[0107] P N Indicates the real three-dimensional coordinates corresponding to the 23rd human key point; Represents the estimated three-dimensional coordinates of the 23rd human key point output by the network;

[0108] Assume P root 、 The real center point of the human body and the network-inferred center point of the human body:

[0109]

[0110] Among them, P i Represents the real three-dimensional coordinates corresponding to the i-th human key point; Indicates that the network outputs the estimated three-dimensional coordinates of the i-th human key point;

[0111] Describes the offset error of the center point of the human body, position loss L loc It can be expressed as:

[0112]

[0113] Describe the human body posture, that is, the offset error of each joint point, the posture loss L pos It can be expressed as:

[0114]

[0115] Among them, J i Represents the weighted weight corresponding to the loss of the i-th joint point. The weight is generated according to the point cloud density at the i-th joint point and is used to adjust the importance of different joint points in network regression.

[0116] Combining position loss and posture loss, the final loss function is as follows:

[0117] L=L loc +L pos

[0118] Among them, L represents the final loss function.

[0119] Example 2

[0120] In this embodiment, a method for reconstructing human body postures from multi-person point clouds based on millimeter wave radar will be described in detail. The specific implementation framework is as follows: Figure 1 As shown, the following steps are included:

[0121] Step 1: Data reception and preprocessing:

[0122] The antenna receives raw millimeter-wave radar echo data in real time and mixes the transmit and receive signals using an onboard hardware mixer to generate an intermediate frequency (IF) signal. This IF signal is then processed into a data cube consisting of the sampling point dimension, the chirp dimension, and the virtual antenna dimension. DC clutter filtering is performed, and the data at each sampling point is averaged in the chirp dimension and subtracted from the original data to remove static clutter from the echo and suppress interfering signals. Inter-frame cancellation is further performed by subtracting the IF signal data between adjacent frames to suppress interference from static targets in the environment and highlight signals corresponding to moving targets. After preprocessing, the ADC raw complex signal in the range dimension (sampling point dimension) of the data cube is fast Fourier transformed to obtain a one-dimensional range spectrum. A second fast Fourier transform is then performed on the range spectrum along the slow-time dimension (chirp dimension). Overall, a two-dimensional FFT operation is performed to obtain a two-dimensional range-Doppler joint spectrum.

[0123] Step 2: Generate 3D point cloud data:

[0124] The two-dimensional range-Doppler joint spectrum obtained in step 1 is then subjected to two-dimensional constant false alarm detection (CFAR) to detect strong reflection points corresponding to human bodies in the target scene. This involves sorting the sampled values ​​of reference cells surrounding the target cell, selecting the kth ordered value as the background noise estimate, and multiplying it by a preset scaling factor to generate a dynamic detection threshold. This effectively suppresses contamination from interfering targets in non-uniform environments such as those with multiple targets or clutter edges. If the power of a cell exceeds the threshold, the point is identified as a candidate target signal point and stored in the target point sequence for subsequent processing.

[0125] After obtaining the target point's index in the distance and velocity dimensions, the antenna channel dimension vector corresponding to that index in the 3D data cube is extracted. To obtain 3D coordinate information, the antenna channel is divided into two dimensions, elevation and azimuth, based on the onboard antenna distribution. The MUSIC super-resolution algorithm is then applied to the extracted target point vectors in both elevation and azimuth dimensions to determine the corresponding elevation and azimuth angles, respectively. This results in the polar coordinate position information corresponding to the point in the 3D point cloud data. Furthermore, based on the principle of coordinate transformation, the polar coordinates are converted to a Cartesian coordinate system to facilitate subsequent point cloud projection and clustering processing.

[0126] Step 3: point cloud enhancement and cluster label processing:

[0127] After step 2, we get Figure 2 The three-dimensional point cloud in space is shown. Figure 2 The scene shown has two human targets in the detection area. From the point cloud distribution, we can see that there are roughly two clusters of human point clouds.

[0128] like Figure 3As shown, Figure 3 This is a flow chart of the point cloud enhanced clustering processing algorithm. The point cloud enhanced clustering processing algorithm includes filtering out-of-domain point clouds, filtering out outliers, filtering out in-domain point clouds, and point cloud cluster matching. The specific process is as follows:

[0129] First, set the detection range of the millimeter-wave radar according to the scene, that is, determine the length, width, and height of the detection area and the location of the radar. Figure 2 The yellow area in the middle represents the detection area, and the grid line area represents the non-detection interference area. By dividing the detection area, we can initially filter out Figure 2 The interfering point cloud outside the shown area is enhanced once.

[0130] Next, the point cloud in the detection area is enhanced twice. The DBSCAN density peak clustering algorithm is used, the core point radius is set to 1m, and the minimum number of neighborhood points is set to 5. Through the preliminary clustering of the point cloud in the domain, it is possible to retain the overall point cloud and filter out the outlier interference points in the domain when multiple targets are close to each other. When multiple targets are far apart, it can be used as a pre-processing for target point cloud enhancement, and initially realize target classification and point cloud condensation.

[0131] Finally, the DBSCAN clustering parameters were modified to achieve three-stage point cloud enhancement. To accurately match objects in the scene with the point cloud, the DBSCAN clustering parameters were modified to favor the differentiation of high-density point cloud clusters. The core point radius was set to 0.4m, and the minimum number of neighborhood points was set to 8. This smaller clustering radius and stricter neighborhood criteria optimize the matching of individual human point clouds while further suppressing interfering points. Finally, after multiple rounds of point cloud enhancement and clustering, the module outputs the point cloud clustering results for each object.

[0132] Step 4: Model building and training:

[0133] The neural network architecture built is as follows Figure 4 As shown in the figure, during the model building phase, the specific process is as follows:

[0134] First, the three orthogonally projected single-channel input images are evenly grouped in the channel dimension, and a separate convolution operation is performed on each group of data. By introducing a grouped convolution structure, feature interference between different projection planes is effectively isolated, reducing the model parameter size and computational complexity, while retaining the independent spatial representation of poses in each channel, laying the foundation for subsequent multi-scale feature fusion.

[0135] Next, a multi-resolution parallel processing architecture is constructed based on the deep high-resolution representation module. By densely connecting sub-networks across resolutions, the semantic information extracted by low-resolution sub-networks is integrated while maintaining the high resolution of the original feature maps, forming a hierarchical feature pyramid structure. This design effectively avoids the spatial information loss caused by pooling operations in traditional convolutional networks, significantly improving the ability to capture subtle joints and dynamic posture details.

[0136] Then, using the ResNet18 residual network as the core framework, combined with an improved residual block structure, a coordinate attention (CA) mechanism is integrated within each residual block. This mechanism extracts position-sensitive features along both the height and width dimensions through bidirectional spatial encoding, generating adaptive weight masks to enhance the response of key regions. Furthermore, a channel attention (SE) module is embedded deep within the network to dynamically calibrate the importance weights of feature channels. Cross-layer feature reuse is achieved through shortcut connections, ensuring efficient gradient propagation and ultimately outputting a high-dimensional dense feature vector.

[0137] Finally, the three-dimensional feature tensor output by the backbone network is flattened into a one-dimensional vector and then fed into a multilayer perceptron for nonlinear mapping. This perceptron uses a cascaded fully connected layer structure to achieve end-to-end regression from abstract features to the three-dimensional coordinates of the 23 human joints.

[0138] After the model is built, in the network training phase, a multi-task joint optimization strategy is adopted. The three-dimensional image projection obtained by point cloud analysis is used as the network input, and the key point coordinates corresponding to the motion capture device are used as labels. The motion capture action is as follows: Figure 5 As shown in the figure, the training set includes actions such as falling, climbing, sitting, standing up, walking, and waving. After constructing the dataset, the training and validation sets were divided into a ratio of 7:3. End-to-end training was performed using a combination of position offset loss and posture reconstruction loss. During training, the Adam optimizer was used, with an initial learning rate set to 0.001 and an exponential decay strategy (with a decay coefficient of 0.7) to dynamically adjust the learning rate. Each training batch contained 64 frames of data, and the total number of training rounds was set to 400. Following the aforementioned hyperparameter settings, the posture reconstruction neural network model was obtained after training and optimization.

[0139] Step 5: Point cloud input and model inference output:

[0140] In a multi-target scenario, multiple point cloud clusters matching the target are generated through processing in step 3. Each point cloud cluster is fed into the single-person pose reconstruction network one by one, sorted by category ID size, from smallest to largest. Real-time model inference is performed and the 3D coordinates corresponding to the 24x3 human skeleton key points are output, resulting in a corresponding 3D plot of the target. This plot is then displayed in real time in the 3D interface. The above process is repeated for each point cloud cluster in the scene, and the inference results are added to the 3D drawing interface one by one.

[0141] exist Figure 6 middle, Figure 6 This diagram shows the real-time reconstruction and 3D display module for multiple people. The left side shows the real-time reconstructed 3D skeleton display interface; the upper right corner shows the real-time camera interface for comparison; and the lower right corner shows the real-time point cloud data of the scene. Due to the short inference time for each single target, this method enables real-time reconstruction and display of multiple targets in a scene. Leveraging the parallel inference performance of the GPU, the deployed multi-target real-time reconstruction system runs at 30-40 frames per second on an NVIDIA GeForce GTX 4080 graphics card, demonstrating real-time and stability.

[0142] Example 3

[0143] In this embodiment, a millimeter-wave radar and a motion capture device constitute an acquisition system. In practical applications, the radar and network configuration described in the method of the present invention can be as follows. This preferred embodiment uses a 60GHz millimeter-wave radar sensor from Skerry Technology Co., Ltd. to transmit and receive signals. This radar features an integrated RF transceiver chip, the RC6011-AIP. This industrial-grade millimeter-wave radar integrates a four-transmitter, four-receiver multi-channel antenna and an on-chip programmable FMCW frequency modulation controller, capable of generating a variety of complex frequency-modulated continuous wave (FMCW) signal outputs. The radar also provides analog baseband differential signal output, with an intermediate frequency (IF) baseband signal bandwidth of 10 MHz.

[0144] The following describes in detail a multi-person point cloud pose reconstruction system based on millimeter-wave radar. The multi-person point cloud pose reconstruction system provided in this embodiment includes a data acquisition module, a point cloud data generation module, a data preprocessing module, a point cloud extraction module, a point cloud data enhancement module, a multi-person point cloud classification module, a neural network model building module, and a multi-person pose real-time reconstruction module; the neural network model building module includes a group convolution module, a high-resolution extraction module, and a backbone network module.

[0145] The data acquisition module is used to collect the original radar echo signal in the multi-target scene. From the start of the millimeter wave radar, the radar will process the received data through hardware mixing, and upload the sampled intermediate frequency signal to the host computer through the serial port. The computer receives the data in real time and decodes the corresponding intermediate frequency according to the encoding format.

[0146] The point cloud data generation module is used to pre-process the original echo data and organize it to obtain a three-dimensional point cloud of the scene; after obtaining the original radar echo data, it is first processed into a data cube consisting of sampling point dimension, chirp dimension and virtual antenna dimension to facilitate feeding into the data pre-processing module. After completing the pre-processing process, a fast Fourier transform is performed on the ADC original complex signal of the distance dimension in the data cube, and a one-dimensional distance spectrum is obtained through this operation. Then, a second fast Fourier transform is performed along the slow time dimension in the distance spectrum to obtain a two-dimensional range-Doppler joint spectrum. The two-dimensional spectrum is then sent to the point cloud extraction module to extract potential target information in the spectrum;

[0147] The data preprocessing module performs DC clutter filtering on the received data. This involves averaging the data at each sampling point in the chirp dimension and performing a subtraction to remove static clutter from the echo, thereby suppressing interfering signals. Furthermore, inter-frame cancellation is performed by performing a differential operation on the intermediate frequency signal data between adjacent frames, thereby suppressing interference from static targets in the environment and highlighting the signals corresponding to moving targets.

[0148] In the point cloud extraction module, a two-dimensional constant false alarm detection (CFAR) is first performed on the two-dimensional joint spectrum to extract possible target points in the range-velocity spectrum and store them in a target point sequence. The index of the target point in the range and velocity dimensions is further obtained, and the corresponding antenna dimension vector in the three-dimensional data cube is extracted based on the index. Then, the MUSIC super-resolution algorithm is performed on the two antenna channel dimensions of pitch and azimuth to obtain the corresponding pitch angle and azimuth angle, ultimately obtaining the corresponding point cloud position information and converting it into a Cartesian coordinate system for processing.

[0149] The point cloud data enhancement module is used to filter out interfering point clouds outside the scene and outlier point clouds within the scene to optimize the quality of the human target point cloud. First, the interfering point clouds in the non-detection area are filtered out according to the preset detection area range to complete the initial point cloud enhancement. Then, the DBSCAN algorithm is used for secondary enhancement. The core radius is set to 1m and the neighborhood threshold is set to 5 points to implement preliminary clustering. While retaining the overall point cloud of multiple targets, outliers in the domain are filtered out to optimize the overall point cloud quality in the domain.

[0150] The multi-person point cloud classification module is used to cluster point cloud data and associate point clouds with human targets in the scene. After completing point cloud enhancement, it performs three enhancements by optimizing DBSCAN parameters to a core radius of 0.4m and a neighborhood threshold of 8 points to enhance the differentiation of human target point cloud clusters and suppress interference signals. Combined with the point cloud data enhancement module, a three-level progressive processing method is used to achieve accurate clustering of multiple human targets in complex scenes, and output the optimized point cloud classification results.

[0151] The neural network model building module is used to build a multi-scale feature fusion convolutional neural network and obtain a posture reconstruction network model after training and optimization; the neural network model building module in this embodiment includes a group convolution module, a high-resolution extraction module and a backbone network module:

[0152] The grouped convolution module evenly groups the three orthogonally projected single-channel input images in the channel dimension, and performs convolution operations independently on each group. This effectively suppresses interference between different projection planes through a feature flow isolation mechanism, reducing the number of model parameters and computational complexity while retaining the ability of each channel to independently represent the spatial distribution of posture.

[0153] The high-resolution extraction module builds a multi-resolution parallel processing architecture based on deep high-resolution representation technology. Through dense cross-resolution sub-network interconnections, it integrates high-order semantic information extracted by low-resolution sub-networks while maintaining the spatial accuracy of the original feature map, forming a hierarchical feature pyramid structure.

[0154] The backbone network module, with the ResNet18 residual framework at its core, adopts an improved residual block structure. Each residual block embeds a coordinate attention mechanism to generate an adaptive weight mask to enhance the response of key regions. The deep network embeds a channel attention module, ultimately outputting a high-density feature vector. Finally, a cascaded fully connected network maps the 3D feature tensor to the 23-joint 3D coordinate space of the human body, achieving end-to-end nonlinear regression from multimodal features to posture parameters.

[0155] The multi-person pose real-time reconstruction module is used to input processed point clouds of human targets in the scene into a neural network model and reconstruct them in real time to obtain inferred and estimated human key point data. Based on this key point data, a three-dimensional human skeleton is drawn in real time for visual display. In this embodiment, the multi-person pose real-time reconstruction module utilizes a pre-trained network model to estimate and output in real time. The real-time reconstruction module obtains point cloud clusters corresponding to scene targets from the point cloud classification module in real time and serially inputs the corresponding point clouds into the pre-trained model. The model relies on the computer's GPU for high-speed real-time inference and adds the output three-dimensional key point information of the human targets one by one to the interface sequence of the three-dimensional drawing, traversing the point cloud clusters on a frame-by-frame basis. This approach effectively displays the dynamic changes of multiple targets in the environment in real time. Given sufficient computing power, it is possible to achieve real-time reconstruction of 4-8 multiple targets in a scene, offering high flexibility.

[0156] The above embodiments are merely preferred embodiments for the purpose of fully illustrating the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are within the scope of protection of the present invention. The scope of protection of the present invention shall be subject to the claims.

Claims

1. A multi-person point cloud human posture reconstruction method based on millimeter-wave radar, characterized by: The following steps are involved: S1: Collects radar echo data of multiple people in the scene; S2: Process the radar echo data and construct human body point cloud data; S3: Project the point cloud data to the xy plane and perform the DBSCAN clustering algorithm on the two-dimensional plane; S4: Design a multi-scale feature extraction and fusion network based on point cloud data, build a backbone network, and obtain a pre-trained pose estimation model through training and optimization; S5: Project the corresponding point clouds of each human body after clustering to obtain their respective orthogonal plane images; S6: The projected orthogonal plane images are sent to the posture reconstruction network in turn, and the posture reconstruction data corresponding to the human target is obtained after model inference.

2. The method for reconstructing multi-person human body posture from millimeter-wave radar point clouds according to claim 1, wherein: The method for data processing in S2 is performed according to the following steps: S21: Perform mean cancellation and DC component suppression on the acquired radar raw echo data to remove environmental noise and clutter interference, and perform a two-dimensional Fourier transform to obtain a range-velocity spectrum; S22: Based on the range-velocity spectrogram obtained in S21, multiple two-dimensional constant false alarm detections are performed to detect strong target reflection points in the scene and extract the corresponding distance and velocity information. Based on this, MUSIC super-resolution angle measurement is further performed in the elevation and azimuth transmit and receive antenna channel dimensions to obtain the corresponding angle information of the target point, and finally the range and elevation angle position of the target relative to the radar in space are obtained. S23: Based on the distance and elevation angle information of the target point cloud relative to the radar, the point cloud data is converted from the polar coordinate system to the three-dimensional Cartesian coordinate system based on the coordinate conversion principle; S24: Setting the scene area range to preliminarily filter out the noise point cloud, and further applying the DBSCAN clustering algorithm to remove outliers and noise points.

3. The method for reconstructing multi-person human body posture from millimeter-wave radar point clouds according to claim 1, wherein: The projection and two-dimensional clustering of the human body point cloud data in S3 are performed according to the following steps: Based on the point cloud data obtained above, the z-axis coordinate in the three-dimensional point cloud coordinates is removed, and the x-axis and y-axis coordinates are retained as classification data; the projected two-dimensional data is subjected to DBSCAN clustering processing again. In order to suppress point cloud diffusion and point cloud interference caused by multipath reflection, reasonable clustering parameters are set to obtain the number of target categories in the clustered scene and the point cloud data labels corresponding to each category.

4. The method for reconstructing multi-person human body posture from millimeter-wave radar point clouds according to claim 1, wherein: The construction of the feature extraction network and the backbone network in S4 is carried out according to the following steps: S41: Grouped convolution processing, which processes three single-channel images and evenly divides the input data into multiple groups in the channel dimension. Each group performs a convolution operation independently to avoid information interference between different channels, reduce model complexity, and retain the characteristic information of each channel; S42: Multi-scale feature extraction, introducing a deep high-resolution representation module. By connecting multi-resolution sub-networks in parallel, it maintains high-resolution feature representation and enhances the ability to capture human posture details. S43: Backbone network construction, based on the ResNet18 architecture, using the residual learning framework to enhance feature transfer efficiency through shortcut connections, and introducing the channel attention mechanism and coordinate attention mechanism to achieve feature extraction and deep mining, ultimately obtaining the feature vector; S44: output of key point estimation, flattening the feature vector obtained in S43, and then sending it to a multi-layer perceptron to process the feature vector to obtain the human body key points corresponding to the current frame.

5. The method for reconstructing multi-person human body posture from millimeter-wave radar point clouds according to claim 1, wherein: The projection of the point cloud data of the human body in S5 is performed according to the following steps: According to the corresponding category labels of each human target point cloud in the detection area obtained in S3, the coordinate information of the three-dimensional point cloud corresponding to each target in the Cartesian coordinate system is projected onto three sets of two-dimensional xy, xz and yz planes respectively to realize the dimensional splitting of the three-dimensional point cloud data. In order to facilitate the calculation and processing of the convolutional neural network, a three-dimensional scene grid with the same proportion as the detection area is created, and the projected two-dimensional point cloud data is further classified into a two-dimensional grid image. The three-dimensional point cloud is discretized with three sets of images. At the same time, the human joint coordinate information obtained by the motion capture device is used as the control label of the network output, and the point cloud data and the corresponding labels are divided into training set, validation set and test set according to the number of frames.

6. The method for reconstructing multi-person human body posture from millimeter-wave radar point clouds according to claim 1, wherein: The two-dimensional constant false alarm detection is specifically performed according to the following steps: In echo data processing, the application scenario and the characteristics of environmental background noise are taken into consideration. After actual test analysis, the OS-CFAR algorithm is selected. The algorithm sorts the sampling values ​​of the reference units around the unit to be detected, selects the kth ordered value as the background noise estimate, and then multiplies it by a preset scaling factor to generate a dynamic detection threshold. If the power of a unit exceeds the threshold, the point is judged as a candidate target signal point, thereby effectively suppressing the contamination of interfering targets in non-uniform environments such as multiple targets or clutter edges. Compared with the traditional averaging method CA-CFAR, the algorithm is more robust.

7. The method for reconstructing multi-person human body posture from millimeter-wave radar point clouds according to claim 1, wherein: The training of the neural network model includes the following steps: After setting the model hyperparameters, the point cloud data contained in the training set is fed into the model and then inferred to obtain the estimated values ​​of the target human key points. The model estimated values ​​and the true value labels corresponding to the radar frames in the training set are then further passed into the loss function to calculate the error loss between the estimated values ​​and the true values. Gradient backpropagation is performed according to the loss results, and the model parameters are updated based on the calculated values ​​during the propagation process. The training process is iterated according to the learning conditions set by the hyperparameters to obtain the neural network model after learning optimization.

8. The method for reconstructing multi-person human body posture from millimeter-wave radar point clouds according to claim 1, wherein: The design of the loss function is specifically carried out according to the following steps: In order to measure the position error and posture error between the estimated output and the true value, it is assumed that P is the true three-dimensional coordinate of the human joint in the Cartesian coordinate system, The three-dimensional coordinates of the human joints output by the network are specifically expressed as follows: Among them, N represents different joints of the human body; Assume P root 、 The real center point of the human body and the network-inferred center point of the human body: Describes the offset error of the center point of the human body, position loss L loc It can be expressed as: Describe the human body posture, that is, the offset error of each joint point, the posture loss L pos It can be expressed as: Combining position loss and posture loss, the final loss function is as follows: L=L loc +L pos Among them, L represents the final loss function.

9. A multi-person point cloud human posture reconstruction system based on millimeter-wave radar, characterized by: It includes data acquisition module, point cloud data generation module, data preprocessing module, point cloud extraction module, point cloud data enhancement module, multi-person point cloud classification module, neural network model building module, and multi-person posture real-time reconstruction module; The data acquisition module is used to collect radar original echo signals in a multi-target scene; The point cloud data generation module is used to pre-process the original echo data and organize it to obtain a three-dimensional point cloud of the scene; The data pre-processing module is used to perform DC clutter filtering operations and inter-frame cancellation processing on the received data; The two-dimensional range-Doppler joint spectrum is obtained by fast Fourier transform; The point cloud extraction module is used to perform two-dimensional constant false alarm detection on the two-dimensional joint spectrum, extract possible target points in the range-velocity spectrum and store them in a target point sequence; further obtain the index of the target point corresponding to the distance and velocity dimensions, and extract the corresponding antenna dimension vector in the three-dimensional data cube based on the index; perform the MUSIC super-resolution algorithm in the two antenna channel dimensions of pitch and azimuth to obtain the corresponding pitch angle and azimuth angle, and finally obtain the corresponding point cloud position information and convert it into a Cartesian coordinate system; The point cloud data enhancement module is used to filter out interfering point clouds outside the scene and outlier point clouds within the scene to optimize the quality of the human target point cloud; The multi-person point cloud classification module is used to cluster point cloud data and match point clouds with human targets in the scene; it achieves multi-person target clustering in complex scenes through three-level progressive processing and outputs optimized point cloud classification results; The neural network model building module is used to build a multi-scale feature fusion convolutional neural network and obtain a posture reconstruction network model after training and optimization; The multi-person posture real-time reconstruction module is used to input the processed human target point clouds in the scene into the neural network model and reconstruct the human key point data after inference and estimation in real time, and draw the three-dimensional human skeleton in real time based on the key point data to achieve visual display.

10. The millimeter-wave radar-based multi-person point cloud human posture reconstruction system according to claim 9, characterized in that: The neural network model building module includes a group convolution module, a high-resolution extraction module, and a backbone network module; The grouped convolution module evenly groups the three orthogonally projected single-channel images input in the channel dimension, and performs convolution operations independently on each group; The high-resolution extraction module builds a multi-resolution parallel processing architecture based on deep high-resolution characterization technology; Through dense interactive connections across resolution sub-networks, while maintaining the spatial accuracy of the original feature map, the high-order semantic information extracted by the low-resolution sub-network is integrated to form a hierarchical feature pyramid structure; The backbone network module adopts the ResNet18 residual framework, with a coordinate attention mechanism embedded in each residual block to generate an adaptive weight mask to enhance the response of key areas; the deep network embeds a channel attention module to output a high-density feature vector; and through a cascaded fully connected network, the three-dimensional feature tensor is mapped to the three-dimensional coordinate space of the human joint, realizing end-to-end nonlinear regression from multimodal features to posture parameters.

Images