Millimeter wave and infrared modal native fusion human body semantic segmentation method and system

Abstract

The invention discloses a human body semantic segmentation method and system based on millimeter wave and infrared modal native fusion, and the method comprises the steps: firstly obtaining millimeter wave point cloud data generated by human body motion, and outputting millimeter wave features through a millimeter wave point cloud feature extraction module; a human body heat map is obtained, and infrared features are output through a low-resolution infrared feature extraction module; then the two modal features serve as input of a multi-modal feature fusion module, and self-adaptive intelligent complementary fusion is carried out to break through intrinsic modal differences of the two modal features in the aspects of spatial resolution, sampling frequency and the like; the fusion features are used as input of a human body semantic output module based on multi-task learning, and accurate semantic segmentation of human body parts is achieved. According to the method, the inherent modal difference challenge between the millimeter wave radar and the low-resolution infrared is overcome, efficient human body semantic segmentation on the premise of low cost and privacy protection is realized, and a new path is provided for fine-grained human body semantic segmentation.

Description

Technical Field

[0001] This invention belongs to the field of wireless sensing technology, and mainly relates to a human semantic segmentation method and system that natively fuses millimeter wave and infrared modes. Background Technology

[0002] Accurate segmentation and understanding of the human head, torso, and limbs to obtain semantic information about the human body are fundamental requirements for building the Internet of Things (IoT) and the Internet of Everything. For example, in the field of sports and health, precise limb detail detection can provide accurate posture estimation and training guidance; in the field of smart elderly care, the recognition of specific semantic parts makes fall detection more reliable; and in the field of virtual reality, accurate limb semantic segmentation is a core prerequisite for natural interaction.

[0003] Existing human semantic segmentation methods can be broadly categorized into three types based on perceptual modality: visual, depth, and radio frequency (RF). Visual solutions rely on high-definition cameras, which, while providing rich texture information, are highly susceptible to variations in lighting, occlusion, and privacy concerns, making them difficult to deploy in sensitive locations such as bedrooms and bathrooms. Depth solutions, such as depth cameras and LiDAR, acquire spatial structure through dense point clouds, but these methods are costly, computationally demanding, and significantly affected by multipath interference and strong light, preventing widespread adoption. In recent years, millimeter-wave radar has garnered significant attention in the field of human perception due to its all-weather operation, low cost, and inherent privacy protection. Based on the principle of frequency-modulated continuous wave (FMCV), it transmits and receives millimeter-wave signals reflected from the human body. Through digital signal processing procedures such as three-dimensional Fourier transform, signal detection, and angle estimation, it can output a five-dimensional millimeter-wave point cloud in real time, containing three-dimensional coordinates, radial velocity, and reflection intensity. However, due to limitations in antenna size and digital signal processing capabilities, a single frame of millimeter-wave point cloud contains only a few dozen target reflection points, which is two orders of magnitude lower than that of lidar. At the same time, the radio frequency characteristics of millimeter-wave signals are prone to missing point clouds on curved surfaces such as the head and arms due to specular reflection, making it difficult to effectively apply traditional shape- or topology-based semantic segmentation algorithms.

[0004] On the other hand, low-resolution infrared sensors, such as the Panasonic AMG8833, can capture the thermal radiation contours of the human body through 8×8 pixel thermal images, providing continuous cues of human shape without exposing any visual information, and supplementing millimeter-wave modes with key semantic information. However, existing multimodal fusion solutions all adopt non-intelligent native simple cascading, direct feature stitching, or post-processing result overlay modes. Such solutions lack an adaptive fusion mechanism designed for the heterogeneity of the two modalities, and cannot actively resolve the inherent and significant differences between them in spatial resolution, sampling frequency, deployment perspective, and physical dimensions. At the spatial resolution level, this manifests as the difference between millimeter-wave sparse point clouds and infrared low-resolution thermal images; at the sampling frequency level, it manifests as the difference between millimeter-wave high sampling rates and infrared low sampling rates; at the deployment perspective level, it manifests as the difference between radar ranging perspective and infrared thermometry perspective; and at the physical dimension level, it manifests as the difference between geometric coordinate parameters and thermal radiation grayscale values. This deficiency inevitably leads to the problems of "spatiotemporal mismatch" and "semantic gap" when directly fusing. The former manifests as asynchronous data acquisition, and the latter manifests as the inability to establish a correlation between point cloud geometric features and infrared thermal features, ultimately failing to achieve true deep complementary fusion.

[0005] In summary, neither single-modal nor simple non-intelligent native fusion methods can meet the requirements for high-precision, robust, and privacy-friendly human semantic segmentation in complex environments. Summary of the Invention

[0006] This invention addresses the challenge of balancing low cost, privacy protection, and fine-grained human semantic segmentation in existing technologies. It provides a method and system for human semantic segmentation that natively fuses millimeter-wave and infrared modal data. First, millimeter-wave point cloud data generated by human motion is acquired, and millimeter-wave features are extracted by a millimeter-wave point cloud feature extraction module. Then, a human thermal image is acquired, and infrared features are extracted by a low-resolution infrared feature extraction module. The two modal features are then used as input to a multimodal feature fusion module, adaptively and intelligently complementing each other to overcome the inherent modal differences in spatial resolution, sampling frequency, etc. The fused features are then used as input to a human semantic output module based on multi-task learning, achieving accurate semantic segmentation of human body parts. This method overcomes the inherent modal differences between millimeter-wave radar and low-resolution infrared, innovatively proposing an intelligent native fusion architecture. By combining millimeter-wave radar and low-resolution infrared sensors—two low-cost, privacy-preserving devices—a deep fusion scheme that adaptively resolves modal differences is constructed, achieving efficient human semantic segmentation while maintaining low cost and privacy, providing a new path for fine-grained human semantic segmentation.

[0007] To achieve the above objectives, the technical solution adopted by this invention is: a human semantic segmentation method based on the native fusion of millimeter wave and infrared modes, comprising the following steps:

[0008] S1, Millimeter-wave point cloud feature extraction: Millimeter-wave point cloud data generated by human motion is obtained from commercial millimeter-wave radar. A spatiotemporal feature encoder for point cloud construction is built through an improved point cloud network PointNet, a self-attention network Attention, and a bidirectional long short-term memory network Bi-LSTM to extract point cloud features.

[0009] S2, Infrared Feature Extraction: Infrared thermal images generated by human motion are acquired from a low-resolution infrared sensor, and an upsampling algorithm based on Gaussian pyramids is used to obtain an upsampled infrared thermal image; the upsampled infrared thermal image is then input into an infrared feature extractor for infrared feature extraction.

[0010] S3, Feature Fusion: The point cloud features output from step S1 and the infrared features output from step S3 are fused. Based on a nonlocal network (NLN), cross-modal semantic associations are dynamically established through learnable parameters, utilizing a learnable projection matrix. , Millimeter-wave spatial location information and infrared thermal distribution semantic information are mapped to a unified query-key feature space, and the cosine similarity matrix is ​​calculated. The semantic correlation between different modal features is dynamically quantified; a K-nearest neighbor strategy is used to form a neighborhood set, the contribution of the neighborhood set features is dynamically adjusted and weighted and aggregated, and the aggregation result is added to the original millimeter-wave features to output the fused millimeter-wave features.

[0011] S4, Output: Construct a human semantic segmentation model, which includes at least a semantic segmentation branch module and a pose estimation branch module. Input the features fused in step S3 into the human semantic segmentation model, and jointly train the semantic segmentation branch module and the pose estimation branch module. The pose estimation network uses human pose structure information to guide the semantic segmentation network to learn and achieve human semantic parsing output.

[0012] As an improvement of the present invention, in step S1, the PointNet point cloud network is designed based on a two-layer shared-weight multilayer perceptron (MLP). The first layer of the MLP maps the point cloud to 32 dimensions, and the output is: The second-layer MLP maps the output of the first layer to 64 dimensions, obtaining pointwise features. : ;in, Let i be the i-th point cloud in a single frame of millimeter-wave point cloud acquired at time t, where i ∈ (1, N), and N is the number of point clouds in each frame. The learnable parameters of the first layer MLP, These are the learnable parameters for the second-layer MLP.

[0013] The self-attention network calculates a weight for each point and sums them up using a weighted average. The relevant attention weight formula is as follows: The following weighted summation aggregation method is used to generate millimeter-wave frame feature representations. :

[0014]

[0015] The output features of the self-attention network Attention Inputting a bidirectional long short-term memory network (Bi-LSTM) captures temporal dependencies and outputs 128-dimensional temporally enhanced millimeter-wave features. The formula is as follows:

[0016]

[0017] in For the parameters of Bi-LSTM;

[0018] The output features of the Bi-LSTM bidirectional long short-term memory network The 32-dimensional point cloud features from the PointNet feature upscaling stage are stitched together to form 160-dimensional millimeter-wave high-dimensional features. This serves as the input for the subsequent multimodal feature fusion module.

[0019] As another improvement of the present invention, in step S2, the 8×8 temperature pixel thermal image is upsampled using the Gaussian pyramid algorithm; the Gaussian pyramid algorithm consists of 4 layers of Gaussian pyramids, each layer is smoothed with a 5×5 Gaussian kernel of σ=1.5 and then upsampled by 2 times, and the sampling is repeated until 128×128 pixels are obtained to obtain the enhanced infrared thermal image.

[0020] As another improvement of the present invention, the method for calculating the cosine similarity matrix S in step S3 is as follows:

[0021]

[0022] in Indicates L2 normal form; This is the mapped millimeter-wave Query feature vector; This is the mapped infrared key feature vector;

[0023] The K-nearest neighbor strategy specifically involves adaptively filtering based on semantic relevance and millimeter-wave features. The K most relevant infrared features form a neighborhood set. Through a learnable weight matrix The contribution of neighborhood set features is dynamically adjusted and weighted based on the real-time information value of millimeter-wave and infrared features. The aggregation result is then added to the original millimeter-wave features to achieve on-demand supplementation of cross-modal feature information.

[0024] As another improvement of the present invention, the goal of the semantic segmentation branch module in step S4 is to minimize the error between the predicted category and the true category at each point, using cross-entropy loss. Predict the semantics of H distinct body parts:

[0025]

[0026] Where N is the number of points and H is the number of categories; It is a 0 and 1 sign function, when the class of sample n is h, =1, otherwise 0; It is the predicted probability that sample n belongs to category h;

[0027] The goal of the pose estimation branch module is to guide the semantic segmentation network module to understand human structure, using mean squared error loss to minimize the error between the predicted skeletal keypoint locations and their actual locations.

[0028]

[0029] in Indicates L2 normal form, and These are the predicted and actual 3D position coordinates of the m-th skeletal joint, respectively, where M is 24, corresponding to the 24 major joints of the human skeleton; the joint optimization objective is to minimize the weighted sum of the two loss functions:

[0030] ;

[0031] in and These are the hyperparameters of the model.

[0032] To achieve the above objectives, the present invention also adopts the following technical solution: a human semantic segmentation system that natively fuses millimeter wave and infrared modalities, comprising at least a millimeter wave point cloud feature extraction module, a low-resolution infrared feature extraction module, a multimodal feature fusion module, and a human semantic output module based on multi-task learning.

[0033] The millimeter-wave point cloud feature extraction module extracts point cloud features from the millimeter-wave point cloud frame sequence through a spatiotemporal feature encoder, which is constructed from an improved point cloud network PointNet, a self-attention network Attention, and a bidirectional long short-term memory network Bi-LSTM.

[0034] The low-resolution infrared feature extraction module includes at least an infrared feature extractor. It takes an infrared thermal image generated by human motion as input, enriches the semantic information of the infrared thermal image through an upsampling algorithm based on Gaussian pyramids, and then inputs it into the infrared feature extractor for infrared feature extraction.

[0035] The multimodal feature fusion module is based on an intelligent native fusion mechanism constructed from a nonlocal neural network (NLN). It fuses the millimeter-wave point cloud features extracted by the millimeter-wave point cloud feature extraction module and the infrared features extracted by the low-resolution infrared feature extraction module. The mechanism dynamically captures cross-modal semantic associations, fills the semantic gaps between different features, establishes cross-modal feature mapping relationships, and outputs a fused high-dimensional feature representation.

[0036] The human semantic output module based on multi-task learning includes at least a semantic segmentation branch module and a pose estimation branch module; it inputs the high-order feature representation output by the multimodal feature fusion module, and jointly trains the semantic segmentation and pose estimation branch modules to achieve human semantic parsing output.

[0037] As an improvement of the present invention, in the millimeter-wave point cloud feature extraction module, the point cloud network PointNet is based on a two-layer shared weight multilayer perceptron (MLP). The first layer MLP maps the 5-dimensional point cloud to 32 dimensions, and the second layer MLP maps the output of the first layer to 64 dimensions to obtain point-by-point features.

[0038] The self-attention network Attention is constructed based on Softmax and linear layers. It calculates weights for each point and sums them up to generate millimeter-wave frame feature representations.

[0039] As another improvement of the present invention, the infrared feature extractor is based on a pre-trained residual network to extract the depth features of the infrared thermal image. After the output feature map is flattened, it is dimensionality reduced by a multilayer perceptron (MLP) to obtain 160-dimensional infrared features.

[0040] Compared with existing technologies, the present invention has the following advantages: The present invention discloses a human semantic segmentation method and system that natively fuses millimeter-wave and infrared modes. Through innovative design of an intelligent native fusion architecture of millimeter-wave radar and low-resolution infrared and a multi-task learning mechanism, it achieves multi-dimensional technological breakthroughs in the field of human semantic segmentation. The specific technical contributions can be summarized as follows:

[0041] (1) This invention proposes a novel multimodal fusion system, filling the gap in low-cost, privacy-friendly segmentation solutions. This invention proposes an intelligent native fusion human semantic segmentation system combining millimeter-wave radar and a low-resolution infrared sensor. It specifically utilizes the continuous human contour features of infrared thermal images to supplement the semantic loss caused by the sparsity and specular reflection of millimeter-wave point clouds, forming a cross-modal complementary mode. The system can use low-cost commercial sensors, eliminating the need for visual image acquisition, and inherently possesses privacy protection characteristics. It can be widely adapted to human-centered perception scenarios such as sports and health, smart elderly care, and smart homes, providing a new technical path for the field of human semantic segmentation, which currently lacks low-cost privacy solutions.

[0042] (2) This invention breaks through the bottleneck of multimodal heterogeneity differences and solves the problems of spatiotemporal mismatch and semantic gap in traditional fusion. In view of the inherent huge differences between millimeter wave and low resolution infrared in spatial resolution, sampling frequency, deployment view and physical dimensions, this invention constructs an intelligent native fusion mechanism based on nonlocal neural network (NLN) through multimodal feature fusion in step S3: it eliminates the difference in physical dimensions by unifying feature space mapping, solves the problem of asynchronous sampling frequency by dynamic semantic association matching, and adapts the deployment view deviation by cross-view feature calibration, and finally realizes the deep complementary fusion of the two modes, providing a reusable technical paradigm for the efficient fusion of heterogeneous modes.

[0043] (3) Step S4 proposed in this invention is based on human semantic output of multi-task learning. It combines human semantic segmentation (main task) and human pose (auxiliary task) for joint learning. It innovatively uses human pose reconstruction as an auxiliary task and uses the prior human skeletal structure extracted by the pose estimation task to guide the semantic segmentation network to accurately identify key parts such as limbs and head missing in millimeter-wave point cloud. It effectively makes up for the defect of single segmentation task's dependence on sparse point cloud and achieves accurate human semantic segmentation. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the overall structure of the system of the present invention;

[0045] Figure 2 This is a schematic diagram of the millimeter-wave point cloud feature extraction module in the system of the present invention;

[0046] Figure 3 This is a schematic diagram of the low-resolution infrared feature extraction module in the system of the present invention;

[0047] Figure 4 This is a schematic diagram of the structure of the multimodal feature fusion module and the human semantic output module based on multi-task learning in the system of the present invention. Detailed Implementation

[0048] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0049] Example 1

[0050] The human semantic segmentation system based on native fusion of millimeter-wave and infrared modalities described in this invention includes at least a millimeter-wave point cloud feature extraction module, a low-resolution infrared thermal image feature extraction module, a multimodal feature fusion module, and a human semantic output module based on multi-task learning. The overall system structure is as follows: Figure 1 As shown.

[0051] Among them, the millimeter-wave point cloud feature extraction module, such as Figure 2 As shown, using millimeter-wave point clouds of human motion collected by a millimeter-wave radar device deployed from a third-person perspective as input, a spatiotemporal feature encoder is constructed through an improved point cloud network (PointNet), a self-attention network (Attention), and a bidirectional long short-term memory network (Bi-LSTM). This encoder can map sparse 5-dimensional millimeter-wave point clouds into high-dimensional spatiotemporal features, effectively alleviating the problem of missing body part point clouds caused by specular reflection in millimeter-wave point clouds, and providing feature support with both spatial location and motion characteristics for subsequent fusion.

[0052] Low-resolution infrared thermal image feature extraction module, such as Figure 3 As shown, an 8×8 pixel infrared thermal image of human motion collected by a low-resolution infrared sensor deployed from a third-person perspective is used as input. First, the image is upsampled using a Gaussian pyramid algorithm to increase the resolution to 128×128 pixels, enriching the semantic information of human contour in the infrared thermal image. Then, it is input into a pre-trained residual network (ResNet) and mapped to high-dimensional spatiotemporal features to extract continuous human contour and thermal distribution semantic information that can supplement the millimeter-wave point cloud.

[0053] Multimodal feature fusion module such as Figure 4 As shown on the left, the core module of the system’s intelligent native fusion adopts a parallel nonlocal network (NLN) architecture. This architecture can adaptively capture the K-nearest neighbor semantic association between millimeter-wave features and infrared features, resolve the inherent differences between the two modes in terms of spatial resolution, physical dimensions, sampling frequency, etc., and finally output the fused high-dimensional feature representation, realizing intelligent native complementary aggregation of cross-modal information.

[0054] Human semantic output module based on multi-task learning, such as Figure 4As shown on the right, the high-dimensional fusion features output by the multimodal feature fusion module are used as input; by jointly training the semantic segmentation and pose estimation dual tasks, the semantic segmentation network is guided to accurately locate human body parts by leveraging the prior human skeletal structure extracted by pose estimation, and finally achieves point-by-point accurate segmentation of human body parts, outputting a human point cloud with semantic labels.

[0055] Example 2

[0056] This invention presents a human semantic segmentation method that natively fuses millimeter-wave and infrared modalities. It integrates low-resolution infrared sensors with millimeter-wave radar, utilizing human semantic information provided by infrared thermal images and adaptively supplementing millimeter-wave features with intelligent native deep fusion technology. This constructs a low-cost, privacy-preserving, fine-grained human semantic segmentation system, providing technical support for various human-centered fusion perception systems. The method addresses three key technical challenges:

[0057] Firstly, there is the issue of sparse point clouds and semantic gaps in millimeter-wave radar. Low-cost millimeter-wave radars are limited by antenna size and affected by the specular reflection effect of millimeter-wave signals, resulting in extremely low output point cloud density. Simultaneously, curved areas such as the human head and arms are prone to point cloud gaps due to specular reflection, leading to an inability to fully represent the human body structure. When traditional point cloud networks (such as PointNet) directly process such sparse point clouds, they struggle to extract complete human structural features, easily leading to localized misidentifications such as mistaking the torso for the head, ultimately resulting in incorrect human semantic segmentation results.

[0058] Secondly, there is the challenge of cross-modal spatiotemporal alignment. Millimeter-wave point clouds represent spatial locations using three-dimensional Cartesian coordinates, while low-resolution infrared thermal images represent temperature distribution using two-dimensional pixel arrays. These two modalities differ significantly in spatial resolution and physical dimensions, making direct fusion prone to spatiotemporal mismatch and semantic gaps. Furthermore, their sampling frequencies differ and their deployment perspectives are biased. Existing fusion mechanisms based on direct feature stitching or traditional attention do not have adaptive logic designed for these differences, making it impossible to establish accurate correspondences between modalities in dynamic scenes. Therefore, a mechanism with adaptive adjustment capabilities needs to be designed to achieve cross-modal spatiotemporal alignment.

[0059] Third, there is the challenge of efficient complementary fusion of cross-modal information. While infrared thermal images can provide continuous human contour information, compensating for the structural deficiencies of millimeter-wave point clouds, their low 8×8 pixel resolution limits the detailed information available. Conversely, while millimeter-wave point clouds contain high-dimensional spatial motion information, they suffer from structural sparsity. Currently, there is a lack of mature technical mechanisms for the efficient complementary fusion of semantic information between these two technologies, and no existing models or universal methods can achieve deep complementary fusion.

[0060] Therefore, this invention innovatively designs an efficient cross-modal fusion method that adapts to the characteristics of two modalities. It achieves high-precision human semantic segmentation through multimodal feature adaptive intelligent native fusion and multi-task joint learning, specifically including the following steps:

[0061] Step S1, millimeter-wave point cloud feature extraction: The millimeter-wave point cloud generated by human motion is obtained by a millimeter-wave radar device deployed from a third-view perspective as input. A point cloud spatiotemporal feature encoder is constructed by an improved point cloud network (PointNet), a self-attention network (Attention), and a bidirectional long short-term memory network (Bi-LSTM) to extract point cloud features.

[0062] S11: Utilizing a third-person perspective millimeter-wave radar device, millimeter-wave point clouds reflected from human motion are collected. The number of point clouds in each frame is then uniformly determined through random sampling. , No. Real-time human body millimeter-wave radar point cloud frame record ; For the first The i-th point in a single frame of millimeter-wave point cloud acquired at any time, i∈(1,N), where N is the number of point clouds in each frame of data;

[0063] S12: Millimeter-wave radar point cloud frame As input to PointNet, a point cloud network designed based on a two-layer shared-weight multilayer perceptron (MLP), the first layer of the MLP maps the 5-dimensional point cloud to 32 dimensions. The output of the first layer of the MLP can be represented as: The second-layer MLP maps the output of the first layer to 64 dimensions, obtaining pointwise features. The output can be represented as: ;in The learnable parameters of the first layer MLP, These are the learnable parameters for the second-layer MLP;

[0064] S13: The self-attention network is constructed based on Softmax + linear layers. It calculates a weight for each point and sums the weights. The relevant attention weight formula is: The following weighted summation aggregation method is used to generate millimeter-wave frame feature representations. :

[0065]

[0066] in For linear layer parameters, These are the parameters of the self-attention network (Attention);

[0067] S14: The output features of the self-attention network (Attention) Inputting a bidirectional long short-term memory (Bi-LSTM) network captures temporal dependencies and outputs 128-dimensional temporally enhanced millimeter-wave features. The formula is as follows:

[0068]

[0069] in For the parameters of Bi-LSTM;

[0070] S15: Output features of the Bidirectional Long Short-Term Memory Network (Bi-LSTM) High-dimensional point cloud features in the feature upscaling stage of PointNet By splicing the data, high-dimensional features of millimeter waves can be formed. This serves as the input for the subsequent multimodal feature fusion module.

[0071] Step S2, Low-resolution infrared feature extraction: The infrared thermal image generated by human motion is obtained by a low-resolution infrared sensor deployed from a third-view perspective as input. The semantic information of the infrared thermal image is enriched by the Gaussian pyramid upsampling algorithm. Then, it is used as the input of the pre-trained residual network (ResNet) and mapped into a high-dimensional spatiotemporal feature representation for infrared feature extraction.

[0072] S21: For the 8×8 temperature pixel thermal image acquired by the low-resolution infrared sensor, upsampling is performed using a Gaussian pyramid algorithm. This algorithm consists of 4 layers of Gaussian pyramids. Each layer is smoothed with a 5×5 Gaussian kernel of σ=1.5 and then gradually upsampled by a factor of 2. This process is repeated until the image reaches 128×128 pixels, resulting in an infrared thermal image with enhanced details. ;

[0073] S22: Output a high-quality 128×128 pixel infrared thermal image from the upsampling algorithm. As input to the infrared feature extractor, which is based on a pre-trained residual network (ResNet18), the feature extractor extracts the depth feature map of the infrared thermal image. The formula is as follows:

[0074]

[0075] in These are the network parameters for ResNet18;

[0076] S23: Transfer feature map After being flattened into a one-dimensional vector, the infrared features are input into a multilayer perceptron (MLP) for dimensionality reduction, resulting in 160-dimensional infrared features. The formula is as follows:

[0077]

[0078] in For feature flattening operation, These are the network parameters for the MLP.

[0079] Step S3, Multimodal Feature Fusion: The point cloud features output in Step S1 are fused together. and the infrared features output in step S3 Feature fusion is performed based on a nonlocal network (NLN). Cross-modal semantic associations are dynamically established through learnable parameters to avoid spatiotemporal mismatch and fusion bias caused by fixed rules. The final output is a unified feature representation that takes into account both millimeter-wave spatial location information and infrared thermal distribution semantic information. First, a learnable projection matrix is ​​utilized... , The two are mapped to a unified Query-Key feature space; then, the cosine similarity matrix is ​​calculated. The semantic correlation between different modal features is dynamically quantified; a K-nearest neighbor strategy is used to form a neighborhood set, the contribution of the neighborhood set features is dynamically adjusted and weighted and aggregated, and the aggregation result is added to the original millimeter-wave features to output the fused and enhanced millimeter-wave features.

[0080] S31: Millimeter wave characteristics With infrared features As input, targeting millimeter-wave features With infrared features Due to the significant heterogeneity arising from different physical sensing principles, directly calculating the correlation degree can lead to semantic misalignment. Therefore, a learnable query projection matrix is ​​introduced. With Key projection matrix The two features are mapped to a Query-Key feature space of the same dimension to resolve modal heterogeneity. The calculation method is as follows:

[0081]

[0082]

[0083] in This is the mapped millimeter-wave Query feature vector; This is the mapped infrared key feature vector;

[0084] S32: Perform cross-modal semantic association quantization on the two modal vectors in the Query-Key feature space, using cosine similarity calculation. and This method focuses solely on the "directional consistency" (semantic association) of feature vectors, unaffected by the "magnitude" (feature strength). It adaptively matches feature correspondences between two modalities under asynchronous sampling frequencies and deployment perspective biases, avoiding the spatiotemporal mismatch issues that easily arise from direct concatenation or fixed-weight fusion. The cosine similarity matrix... The calculation method is as follows:

[0085]

[0086] Where ||·|| represents the L2 norm and the cosine similarity matrix. The value range is [-1, 1];

[0087] S33: Based on semantic similarity matrix The calculation results employ a K-Nearest Neighbor (K-NN) strategy for adaptive selection and millimeter-wave features. The K infrared feature components with the highest semantic relevance in each dimension form a neighborhood set matrix. The non-fixed neighborhood matrix The element values ​​represent the index numbers of the infrared feature components, which change dynamically with real-time features, and the neighborhood set matrix. The calculation method is as follows:

[0088]

[0089] in This is a 160×K neighborhood set matrix, where each row represents the index of the K infrared feature components that have the highest correlation with the corresponding millimeter-wave feature component. This indicates that the indices of the K most similar values ​​are taken, where K is the number of nearest neighbors;

[0090] S34: Simultaneously filter the similarity matrices corresponding to the K infrared feature components with the highest semantic relevance in each dimension. The K weight values ​​in the matrix form the neighborhood set weight matrix. The element values ​​of this weight matrix represent the semantic association strength of the corresponding infrared feature components, and the neighborhood set weight matrix. The calculation method is as follows:

[0091]

[0092] in This is a 160×K neighborhood set weight matrix, where each row represents the similarity value of the K infrared feature components that have the highest correlation with the corresponding millimeter-wave feature component. This indicates selecting the top K similarity values;

[0093] S35: Based on Neighborhood Set Matrix and neighborhood set weight matrix Infrared features are weighted and aggregated to obtain supplementary infrared features. These supplementary features are then added to the original millimeter-wave features to achieve on-demand supplementation of cross-modal features, thus solving the semantic loss problem caused by the sparsity of millimeter-wave point clouds. The final output is the fused enhanced feature, with a dimension of 160, calculated as follows:

[0094]

[0095] in Features of millimeter waves The i-th dimension component; Neighborhood set matrix The element value in the i-th row and k-th column represents the index of the k-th infrared feature component that has the highest correlation with the i-th millimeter-wave feature component; Neighborhood set weight matrix The element value in the i-th row and k-th column represents the semantic association strength between the i-th millimeter wave feature component and the k-th nearest neighbor infrared feature component; Infrared characteristics Corresponding index eigencomponents.

[0096] Step S4, Human semantic output based on multi-task learning: includes at least a semantic segmentation branch module and a pose estimation branch module. This module takes the high-dimensional feature representation output by the multimodal feature fusion module as input and jointly trains the semantic segmentation and pose estimation branch modules. The pose estimation network uses human pose structure information to guide the semantic segmentation network to learn and realize human semantic parsing output.

[0097] S41: Human semantic output module based on multi-task learning with fused features As input, perform semantic segmentation and pose estimation tasks simultaneously;

[0098] S42: The semantic segmentation branch module, with fused features As input, a multilayer perceptron (MLP) is used for human semantic mapping, with the following formula:

[0099]

[0100] in It is an N×H matrix representing the category prediction result for each point, where N is the number of points in the current frame; H=6 is the number of categories, corresponding to the six parts of the human body: head, torso, left and right arms, and left and right legs. Network parameters for MLP;

[0101] S43: The semantic segmentation branch module aims to minimize the error between the predicted class and the true class at each point, using cross-entropy loss. Predict the semantics of H distinct body parts:

[0102]

[0103] Where N is the number of points and H is the number of categories. It is a 0 and 1 sign function, when the class of sample n is h, =1, otherwise 0; It is the predicted probability that sample n belongs to category h;

[0104] S44: The pose estimation branch module, with fused features As input, its goal is to guide the semantic segmentation network module to better understand human structure. A multilayer perceptron (MLP) is used for human pose estimation and 3D keypoint mapping, with the following formula:

[0105]

[0106] in This is an M×3 matrix representing the three-dimensional position coordinates of M skeletal points in the human body, where M is 24. Network parameters for MLP;

[0107] S44: The pose estimation branch module uses mean squared error loss to minimize the error between the predicted skeletal keypoint positions and the actual positions.

[0108]

[0109] in Indicates L2 normal form, and These are the predicted and corresponding true 3D position coordinates of the m-th skeletal joint, respectively.

[0110] S45: A human semantic output module based on multi-task learning, whose joint optimization objective is to minimize the weighted sum of two loss functions:

[0111]

[0112] in and These are the model's hyperparameters, used to balance the importance of the two tasks. The entire model is trained in an end-to-end manner.

[0113] This invention systematically addresses two core challenges in human semantic segmentation by constructing a millimeter-wave-infrared cross-modal fusion network: the lack of semantic information in sparse millimeter-wave point clouds and the problem of multimodal spatiotemporal dynamic changes. On one hand, by improving the PointNet-BiLSTM joint encoder, the 5-dimensional millimeter-wave point cloud is upscaled to 160-dimensional spatiotemporal features, fully mining motion and structural cues in the sparse point cloud. On the other hand, Gaussian upsampling and ResNet18 are used to map 8×8 infrared heatmaps into 160-dimensional continuous contour features, significantly compensating for the insufficient resolution of the heatmap. The key innovation of this invention lies in the introduction of a parallel nonlocal network (NLN) to achieve K-nearest neighbor interaction of dual-modal features, establishing fine-grained semantic associations between millimeter-wave and infrared heatmaps for the first time, successfully resolving the inherent differences between the two in spatial resolution, physical dimensions, and sampling frequency. Ultimately, this invention relies solely on low-cost commercial millimeter-wave radar and a 64-pixel (8×8) infrared sensor to achieve high-precision semantic segmentation of six parts of the human body: head, torso, left and right arms, and left and right legs. It establishes a new paradigm for non-contact, privacy-preserving, and highly robust three-dimensional human semantic segmentation, laying a practical and accurate perception technology foundation for human-centered perception scenarios such as smart homes, medical rehabilitation, and human-computer interaction.

[0114] Test case

[0115] To verify the effectiveness of the proposed method, human semantic segmentation experiments were conducted in three different environments to demonstrate the performance advantages and impact of the proposed method. The experiments were performed using the following configurations:

[0116] Hardware settings:

[0117] 1. Millimeter-wave radar IWR6843ISK-ODS, used to acquire millimeter-wave point cloud data;

[0118] 2. AMG8833 Low-Resolution Infrared Sensor: Used for acquiring low-resolution infrared thermal images.

[0119] 3. Azure Kinect v2, used to capture the real pose of the human arm and train the network;

[0120] Experimental Design:

[0121] 1. Select three different environments as follows:

[0122] (1) Corridor (4.0 m × 2.0 m): Place tables, chairs, and metal cabinets to simulate multipath interference;

[0123] (2) Meeting room (3.0 m × 1.8 m): Metal partition screen and conference table, producing strong reflections;

[0124] (3) Outdoor lawn (4.5 m × 3.0 m): open and unobstructed, verifying environmental robustness.

[0125] 2. Thirty-four subjects were recruited to walk freely in an area 1.1 m in front of the radar and perform 12 types of daily actions (such as raising hands, rotating the torso, waving hands, etc.), and a total of 400,000 frames of multimodal data were collected.

[0126] 3. The millimeter-wave radar is fixed on an aluminum tripod, 0.87 m above the ground, facing the center of the subject's chest and abdomen, at a horizontal distance of 1.1 m, and continuously transmits and receives 60 GHz FMCW signals at 10 fps.

[0127] 4. The low-resolution infrared sensor is mounted on an independent miniature gimbal, 0.96 m high, located 15 cm to the right of the radar. The optical axis is parallel to the radar's main beam, and the center of the sensor lens is aligned with the subject's chest to ensure that the 8×8 thermal image covers the complete outline of the human body.

[0128] 5. Azure Kinect v2 captures 3D mesh data at 15 fps as ground truth.

[0129] 6. PointNet, PointCNN, P4Transformer, and DGCNN were selected as comparison methods, with DGCNN being the best existing method.

[0130] 7. Select three comparison criteria, namely:

[0131] Average accuracy (mAcc): Defines the average proportion of correctly classified point clouds across all parts;

[0132] Mean Intersection over Union (mIoU): Calculates the overlap between the predicted and actual segmented regions;

[0133] The average Dice coefficient (mDice) measures the similarity between the prediction and the true segment.

[0134] The experimental results are shown in Table 1-3 below:

[0135] Table 1 Performance of each model in human semantic segmentation

[0136]

[0137] As shown in the table above, our system exhibits the best performance, specifically with an average accuracy (mAcc) of 83.21%, an average intersection-over-union (mIoU) of 80.27%, and an average Dice coefficient of 82.03%. Among existing single-modal models, even though DGCNN achieves the best mAcc for human semantic segmentation at 82.37%, it still falls short of the multimodal fusion model proposed in this paper. These results demonstrate that, compared to existing single-modal methods, the low-resolution infrared image features in our system provide more effective information for human semantic segmentation from millimeter-wave point clouds, effectively improving the prediction accuracy of human semantic segmentation.

[0138] Table 2. Human semantic segmentation performance of different body parts in this system

[0139]

[0140] Table 2 further analyzes the semantic segmentation performance of different body parts in this system, and the results are shown in Table 2. It shows that the segmentation accuracy of the torso and head is the highest, approaching 90%, which is attributed to the large reflective area and high millimeter-wave signal intensity of these two parts. However, the limbs (arms and legs) have higher joint degrees of freedom and significant specular reflection during movement. Simultaneously, they are more susceptible to occlusion, leading to increased sparsity of point cloud data and relatively lower segmentation accuracy. Nevertheless, this system compensates for the deficiencies of millimeter-wave point clouds in limb areas by fusing the thermal distribution characteristics of low-resolution infrared thermal images, maintaining a limb segmentation accuracy of around 80%, which is superior to single-modal millimeter-wave based systems.

[0141] To verify the effectiveness of each module in this system, the following ablation experiments will be conducted: (1) Full version: All components of this system. (2) No infrared modality (w / o IR): The infrared thermal image feature extraction block is removed, while other modules are retained. That is, the system no longer accepts infrared thermal image input, but only performs human semantic segmentation based on millimeter-wave point cloud input. (3) Direct feature concatenation (N2C): The method of fusing image features using parallel nonlocal networks (NLN) is replaced by directly concatenating and processing the extracted features. (4) No pose estimation branch module (w / o Pose): The pose estimation branch module in the human semantic output module based on multi-task learning is removed, while other modules are retained. That is, the system does not perform task supervision, and only the human semantic segmentation result is output.

[0142] Table 3 Ablation Experiment Results

[0143]

[0144] Table 3 above presents the results of the ablation experiments. The experimental data shows that removing the infrared mode from the system significantly degrades its performance. Specifically, the system's mAcc, mIoU, and mDice decreased by 7.38%, 7.68%, and 7.25%, respectively. This indicates that the infrared mode plays a crucial role in the system. In particular, the main function of the infrared mode is to enhance the semantic information in millimeter waves when processing sparse point clouds, thereby improving the semantic segmentation accuracy of the system. The experimental results demonstrate that this mode fulfills its intended function.

[0145] Regarding multimodal feature fusion, using NLN to fuse image features demonstrates superior performance compared to directly stitching together multimodal features. Analysis of experimental results shows that NLN effectively correlates the semantic information of 3D point clouds and 2D heatmaps, using infrared heatmap features to supplement the semantic cues missing in millimeter-wave point clouds. In contrast, simply stitching together the two features directly can easily lead to the loss of crucial information.

[0146] Removing the pose estimation branch module from the model also significantly degrades the system's performance on multimodal datasets. Specifically, the system's mAcc, mIoU, and mDice decreased by 6.74%, 6.53%, and 6.76%, respectively. Analysis of the experimental results shows that the pose estimation branch module provides global human pose information to the entire network architecture. This information can serve as prior knowledge for semantic segmentation, guiding the semantic segmentation network's learning and improving the performance of human semantic parsing.

[0147] In summary, this proposal presents a human semantic segmentation system and method that integrates millimeter-wave radar and low-resolution infrared intelligent native fusion. First, a millimeter-wave radar deployed from a third-view perspective acquires millimeter-wave point clouds generated by human motion. These points are then input into a millimeter-wave point cloud feature extraction module. After processing the spatial structure and motion information of the sparse point cloud, the module outputs millimeter-wave modal features that characterize the three-dimensional position and velocity of the human body. Second, a low-resolution infrared sensor deployed from the same third-view perspective acquires human thermal images. These images are then input into a low-resolution infrared feature extraction module. After enhancing the contour information of the thermal images, the module outputs infrared modal features that reflect the continuous shape of the human body. Finally, the millimeter-wave and infrared modal features are input into a multimodal feature fusion module. This module, relying on an intelligent native fusion mechanism, dynamically resolves the inherent differences in spatial resolution and physical dimensions between the two modalities, achieving adaptive complementary fusion where "millimeter-wave compensates for insufficient infrared detail, and infrared compensates for the sparse defects of millimeter-wave." The final output is a fused feature that combines spatial motion information and continuous contour information. Finally, the fused features are input into a human semantic output module based on multi-task learning. This module, through joint training of semantic segmentation and pose estimation tasks, utilizes prior knowledge of human pose structure to guide the semantic segmentation network to accurately locate human body parts, ultimately achieving point-by-point accurate segmentation of six types of human body parts: head, torso, left and right arms, and left and right legs. This method effectively overcomes the modal difference challenge between millimeter-wave radar and low-resolution infrared, systematically solving the two core challenges of human semantic segmentation: the lack of semantic information in sparse millimeter-wave point clouds and the problem of multimodal spatiotemporal dynamic changes. Under the premise of using low-cost sensors and ensuring the protection of perceptual privacy, it successfully achieves fine-grained human semantic segmentation, realizing accurate human semantic segmentation in privacy scenarios.

[0148] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. A human semantic segmentation method based on native fusion of millimeter wave and infrared modes, characterized in that... It includes the following steps: S1, Millimeter-wave point cloud feature extraction: Obtain millimeter-wave point cloud data generated by human motion, and construct a point cloud spatiotemporal feature encoder through an improved point cloud network PointNet, a self-attention network Attention, and a bidirectional long short-term memory network Bi-LSTM to extract point cloud features. S2, Infrared Feature Extraction: Acquire infrared thermal images generated by human movement, and obtain upsampled infrared thermal images through an upsampling algorithm based on Gaussian pyramids; input the fused infrared thermal images into the infrared feature extractor for infrared feature extraction; S3, Feature Fusion: The point cloud features output from step S1 and the infrared features output from step S3 are fused. Based on a nonlocal network (NLN), cross-modal semantic associations are dynamically established through learnable parameters, utilizing a learnable projection matrix. , Millimeter-wave spatial location information and infrared thermal distribution semantic information are mapped to a unified query-key feature space, and the cosine similarity matrix is ​​calculated. The semantic correlation between different modal features is dynamically quantified; a K-nearest neighbor strategy is used to form a neighborhood set, the contribution of the neighborhood set features is dynamically adjusted and weighted and aggregated, and the aggregation result is added to the original millimeter-wave features to output the fused millimeter-wave features. S4, Output: Construct a human semantic segmentation model, which includes at least a semantic segmentation branch module and a pose estimation branch module. Input the millimeter-wave features fused in step S3 into the human semantic segmentation model, and jointly train the semantic segmentation branch module and the pose estimation branch module network. The pose estimation network uses human pose structure information to guide the semantic segmentation network to learn and achieve human semantic parsing output.

2. The human semantic segmentation method based on native fusion of millimeter-wave and infrared modes as described in claim 1, characterized in that: In step S1, the PointNet point cloud network is designed based on a two-layer shared-weight multilayer perceptron (MLP) architecture. The first layer of the MLP maps the point cloud to 32 dimensions, and the output is: The second-layer MLP maps the output of the first layer to 64 dimensions, obtaining pointwise features. : ;in, Let i be the i-th point in a single frame of millimeter-wave point cloud acquired at time t, where i ∈ (1, N), and N is the number of point clouds in each frame. The learnable parameters of the first layer MLP, These are the learnable parameters for the second-layer MLP; The attention network calculates a weight for each point and sums them up. The relevant attention weight formula is as follows: The millimeter-wave frame feature representation is generated using the following weighted summation aggregation method. : ； in, For linear layer parameters, These are the parameters of a self-attention network; Representing millimeter-wave frame features Inputting a bidirectional long short-term memory network (Bi-LSTM) outputs 128-dimensional millimeter-wave features. : ； in For the parameters of Bi-LSTM; Output features By concatenating the 32-dimensional high-dimensional point cloud features obtained from the PointNet feature upscaling stage, a 160-dimensional millimeter-wave high-dimensional feature is obtained. .

3. The human semantic segmentation method based on native fusion of millimeter-wave and infrared modes as described in claim 1, characterized in that: In step S2, for the 8×8 temperature pixel infrared thermal image, upsampling is performed using the Gaussian pyramid algorithm. The algorithm consists of 4 layers of Gaussian pyramids. Each layer is smoothed with a 5×5 Gaussian kernel of σ=1.5 and then upsampled by 2 times, and the sampling is repeated until 128×128 pixels are reached.

4. The human semantic segmentation method based on native fusion of millimeter wave and infrared modes as described in claim 1, characterized in that: In step S3, the cosine similarity matrix S is calculated as follows: ； in Indicates L2 normal form; This is the mapped millimeter-wave Query feature vector; This is the mapped infrared key feature vector; The K-nearest neighbor strategy specifically involves adaptively filtering based on semantic relevance and millimeter-wave features. The K most relevant infrared features form a neighborhood set. Through a learnable weight matrix The contribution of neighborhood set features is dynamically adjusted and weighted based on the real-time information value of millimeter-wave and infrared features. The aggregation result is then added to the original millimeter-wave features to achieve on-demand supplementation of cross-modal feature information.

5. The human semantic segmentation method based on native fusion of millimeter-wave and infrared modes as described in claim 1, characterized in that: The goal of the semantic segmentation branch module in step S4 is to minimize the error between the predicted class and the true class at each point, using cross-entropy loss. Predict the semantics of H distinct body parts: ； Where N is the number of points and H is the number of categories. It is a 0 and 1 sign function, when the class of sample n is h, =1, otherwise 0; It is the predicted probability that sample n belongs to category h; The goal of the pose estimation branch module is to guide the semantic segmentation network module to understand human structure, using mean squared error loss to minimize the error between the predicted skeletal keypoint locations and their actual locations. ； in Indicates L2 normal form, and These are the predicted and corresponding true 3D position coordinates of the m-th skeletal joint, respectively, with M taking the value 24; The overall objective of the joint optimization is to minimize the weighted sum of the two loss functions: ； in and These are the hyperparameters of the model.

6. A human semantic segmentation system that natively fuses millimeter-wave and infrared modalities, characterized in that: It includes at least a millimeter-wave point cloud feature extraction module, a low-resolution infrared feature extraction module, a multimodal feature fusion module, and a human semantic output module based on multi-task learning. The millimeter-wave point cloud feature extraction module extracts point cloud features from the millimeter-wave point cloud frame sequence through a spatiotemporal feature encoder, which is constructed from an improved point cloud network PointNet, a self-attention network Attention, and a bidirectional long short-term memory network Bi-LSTM. The low-resolution infrared feature extraction module includes at least an infrared feature extractor. It takes an infrared thermal image generated by human motion as input, enriches the semantic information of the infrared thermal image through an upsampling algorithm based on Gaussian pyramids, and then inputs it into the infrared feature extractor for infrared feature extraction. The multimodal feature fusion module: based on an intelligent native fusion mechanism constructed from a nonlocal neural network (NLN), it performs feature fusion on the millimeter-wave point cloud features extracted by the millimeter-wave point cloud feature extraction module and the infrared features extracted by the low-resolution infrared feature extraction module. The mechanism dynamically captures cross-modal semantic associations, fills the semantic gaps between different features, establishes cross-modal feature mapping relationships, and outputs a fused high-dimensional feature representation. The human semantic output module based on multi-task learning includes at least a semantic segmentation branch module and a pose estimation branch module; The high-order feature representation output by the input multimodal feature fusion module is used to jointly train the semantic segmentation and pose estimation branch modules to achieve human semantic parsing output.

7. The human semantic segmentation system with native fusion of millimeter wave and infrared modes as described in claim 6, characterized in that: In the millimeter-wave point cloud feature extraction module, the PointNet point cloud network is based on a two-layer shared weight multilayer perceptron (MLP). The first layer MLP maps the 5-dimensional point cloud to 32 dimensions, and the second layer MLP maps the output of the first layer to 64 dimensions, thus obtaining point-by-point features. The self-attention network Attention is constructed based on Softmax and linear layers. It calculates weights for each point and sums them up to generate millimeter-wave frame feature representations.

8. The human semantic segmentation system with native fusion of millimeter wave and infrared modes as described in claim 6, characterized in that: The infrared feature extractor is based on a pre-trained residual network to extract depth features from infrared thermal images. After the output feature map is flattened, it is dimensionality-reduced by a multilayer perceptron (MLP) to obtain 160-dimensional infrared features.

Images