Cross-modal person retrieval and identification method and system based on multi-dimensional biological characteristics

Abstract

The application discloses a kind of multi-dimensional biological characteristics-based cross-modal personnel retrieval and identification method and system, extract initial millimeter wave feature through radio frequency feature extraction module, then output visual feature after visual feature extraction module, two kinds of modal characteristics are used as the input of cross-modal coordination representation module, relieve inherent modal difference between two kinds of characteristics by self-attention mechanism and cross-modal attention mechanism, output fused feature representation;Fusion feature is used as the input of multi-dimensional biological characteristics projection module, to extract height, BMI and the biological recognition feature of posture, finally use cross-modal personnel retrieval module, under the premise of protecting privacy, realize the accurate matching of radar detection personnel in privacy protection area and personnel collected by open area camera.This application method creatively fuses multi-biological characteristics and cross-modal representation learning, effectively solves the technical challenges such as significant modal difference and unstable gait, and provides a new path for cross-modal personnel re-identification.

Description

Technical Field

[0001] This invention relates to the field of wireless sensing technology, and in particular to a method and system for cross-modal personnel retrieval and identification based on multidimensional biometrics. Background Technology

[0002] Personnel retrieval and identification is a core technology in human-centered applications such as smart city construction, public safety, and intelligent surveillance. It aims to match and identify the same target individual across different modalities, scenarios, or time periods. For example, in security monitoring, this technology can seamlessly track the object across non-overlapping camera networks, assisting in quickly locating the target's trajectory. In smart home scenarios, the system can identify family members through the user's unique biometrics and automatically adjust environmental settings such as lighting and temperature to match personal preferences. In public transportation hubs such as airports and train stations, this technology promotes efficient passenger flow management and improves travel safety and convenience by tracking individuals at different checkpoints.

[0003] Existing methods primarily rely on widely deployed cameras and various visual models to retrieve and identify people across different camera views. However, the coverage of vision-based systems is limited by serious privacy concerns, making them unsuitable for operation in privacy-sensitive locations such as residential areas, hospitals, and private offices. To address privacy issues and provide a user-friendly sensing approach, recent research has explored wireless sensing technologies such as Wi-Fi and millimeter-wave radar. Among these, millimeter-wave radar stands out due to its low cost and all-weather sensing capabilities, becoming a reliable sensing infrastructure for human perception applications. By placing a millimeter-wave radar system in front of a user and transmitting and receiving reflected radio frequency spectrum signals, converting them into millimeter-wave point cloud data, continuous capture of the user's motion status can be achieved. Thanks to the characteristics of radio frequency sensing, this non-visual sensing method means that the system does not record any facial information or other sensitive personal data, thus protecting user privacy.

[0004] While some progress has been made in millimeter-wave-based people retrieval and identification, these methods typically require data collection and training on a large number of prior instances of the target individual in the same environment. They rely on closed datasets and primarily focus on dynamic motion patterns such as gait and posture. They cannot handle unknown individuals or new environments, and their practicality is limited in dynamic scenarios with constantly changing populations in the real world. These practical limitations prompted us to propose a novel cross-RF-vision people retrieval and identification solution. When millimeter-wave radar detects a person in a camera-restricted area, the system can match and identify the same person from cameras widely deployed in open areas.

[0005] However, achieving such cross-modal retrieval and recognition faces two key challenges. First, there are inherent modal differences between the two modalities. RGB cameras provide fine-grained visual details of individuals but lack depth information, resulting in the loss of motion-related information such as stride length and speed. In contrast, millimeter-wave radar provides 3D point clouds with distance and speed information, but due to the limited resolution of low-cost radar, the point clouds are extremely sparse, with only a few dozen human-related points per frame, lacking fine-grained body features and making direct matching difficult. Second, because millimeter-wave point clouds and RGB videos are typically captured at different times and locations, an individual's gait may vary due to environmental factors, unexpected events, or emotional states, introducing significant instability into the retrieval and recognition process. Traditional methods struggle to address this gait instability caused by cross-temporal and spatiotemporal data acquisition. Therefore, a novel cross-modal person retrieval and recognition method is needed to mitigate modal differences and overcome gait instability. Summary of the Invention

[0006] This invention addresses the challenges of mitigating modal differences and gait instability in existing technologies by providing a cross-modal person retrieval and identification method and system based on multi-dimensional biometrics. First, an initial millimeter-wave feature extraction module extracts initial features, followed by a visual feature extraction module that outputs visual features. These two modal features are then used as input to a cross-modal coordinated representation module. Self-attention and cross-modal attention mechanisms mitigate the inherent modal differences between the two features, resulting in a fused feature representation. This fused feature is then used as input to a multi-dimensional biometric projection module to extract biometric features such as height, BMI, and posture. Finally, a cross-modal person retrieval module is used to accurately match personnel detected by radar in privacy-preserving areas with personnel captured by cameras in open areas, while protecting privacy. This invention creatively integrates multi-biological features and cross-modal representation learning, effectively solving technical challenges such as significant modal differences and gait instability, and providing a new path for cross-modal person re-identification.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is: a cross-modal person retrieval and identification system based on multidimensional biometrics, comprising at least a radio frequency feature extraction module, a visual feature extraction module, a cross-modal coordinated representation module, a multidimensional biometric projection module, and a cross-modal person retrieval module.

[0008] The radio frequency feature extraction module: constructs a point cloud spatiotemporal feature encoder by using an improved point cloud network PointNet, a Transformer encoder, and a bidirectional long short-term memory network Bi-LSTM to extract point cloud features from millimeter-wave point cloud data generated by human motion, thereby obtaining the radio frequency features of human motion.

[0009] The visual feature extraction module acquires a video frame sequence of human motion, processes the video frame sequence using a pre-trained ResNet-50 model and a Bi-LSTM network, and obtains visual features.

[0010] The cross-modal coordinated representation module: inputs the radio frequency features output by the radio frequency feature extraction module and the visual features output by the visual feature extraction module, performs self-attention processing, and generates fused and enhanced radio frequency features and visual features through a bidirectional cross-modal attention mechanism;

[0011] The multidimensional biometric projection module uses a multilayer perceptron (MLP) with shared weights to map the fused features output by the cross-modal coordinated representation module onto three dedicated prediction heads for posture, height, and BMI, thereby extracting biometric features of posture, height, and BMI.

[0012] The cross-modal personnel retrieval module performs L2 normalization on the biometric features extracted by the multi-dimensional biometric projection module, maps them to a unified metric space, and performs sorting and matching by calculating the Euclidean distance between the radio frequency query samples and the visual image library samples to complete personnel retrieval and identification.

[0013] To achieve the above objectives, the present invention also adopts the following technical solution: a cross-modal person retrieval and identification method based on multidimensional biometrics, comprising at least the following steps:

[0014] S1. Point cloud feature extraction: Obtain raw millimeter-wave point cloud data generated by human motion, map the raw millimeter-wave point cloud data into high-dimensional point features through a multilayer perceptron (MLP) with shared weights, model the spatial relationship between points in each frame using a Transformer encoder, aggregate point features through an attention mechanism to generate frame-level representations, and finally model temporal dependencies through a Bi-LSTM network and aggregate output features through a SUM function to generate radio frequency feature representations.

[0015] S2. Visual Feature Acquisition: Acquire video frame sequences of hot-moving motion, extract spatial features of each frame through a pre-trained ResNet-50 model, and then capture the temporal evolution characteristics of motion through a Bi-LSTM network to generate visual feature representations.

[0016] S3. Cross-modal coordination: Through the self-attention mechanism, the radio frequency features output in step S1 and the visual features output in step S2 are internally enhanced respectively. The query, key and value projection are calculated to generate enhanced features. A bidirectional cross-modal attention mechanism is adopted to realize bidirectional feature exchange from radio frequency to vision and from vision to radio frequency. Complementary information is aggregated by attention weight weighting to output a high-dimensional feature representation of the fused radio frequency features and visual features.

[0017] S4. Biometric feature extraction: The high-dimensional features obtained in step S3 are mapped to three dedicated prediction heads, namely pose, height and BMI, using a multi-task joint supervised training through the L2 loss function to extract biometric features.

[0018] S5. Cross-modal personnel retrieval: The features extracted in step S4 are processed by L2 normalization and mapped to a unified metric space. The Euclidean distance between the radio frequency query samples and the visual image library samples is calculated for sorting and matching to complete personnel retrieval and identification.

[0019] Compared with the prior art, the present invention has the following technical advantages and effects:

[0020] (1) The cross-millimeter-wave-visual modality personnel retrieval and identification method and system based on multidimensional biometrics proposed in this invention can extend personnel retrieval and identification capabilities from open areas where cameras are widely deployed to privacy-sensitive areas where cameras are prohibited. By deploying millimeter-wave radar in privacy areas without recording any facial or visual identity information, and utilizing existing cameras in open areas, cross-modal personnel retrieval and identification are realized, fundamentally avoiding the privacy leakage risk caused by installing RGB cameras in sensitive places.

[0021] (2) In step S3 of the method of the present invention, the self-attention mechanism is used to enhance the consistency of features within the same modality. The complementary feature exchange between millimeter-wave radar and visual modality is realized through the cross-modal attention mechanism, which effectively alleviates the modal differences between 3D sparse radar point cloud and 2D RGB image.

[0022] (3) In the multidimensional biometric projection step S4 proposed in the method of the present invention, by extracting long-term stable biometric features such as height, BMI and posture, the gait instability caused by time, environment and emotional factors is effectively overcome, and the robustness and recognition accuracy of the system are significantly improved.

[0023] (4) Under the premise of protecting privacy, the method of the present invention creatively integrates multiple biometric features and cross-modal representation learning, effectively solving technical challenges such as significant modal differences and unstable gait, and realizing accurate matching of personnel detected by radar in privacy-protected areas and personnel collected by cameras in open areas, providing a new path for cross-modal personnel re-identification. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the system structure of the present invention;

[0025] Figure 2 This is a schematic diagram of the radio frequency feature extraction module in the system of the present invention;

[0026] Figure 3This is a schematic diagram of the visual feature extraction module in the system of the present invention;

[0027] Figure 4 This is a schematic diagram illustrating the working principle of the cross-modal attention mechanism in the system of this invention;

[0028] Figure 5 This is a schematic diagram of the ablation experiment results in the test examples of this invention;

[0029] Figure 6 This is a graph showing the experimental results of the impact of the input sequence length on system performance in the test examples of this invention. Detailed Implementation

[0030] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0031] Example 1

[0032] A multi-dimensional biometric transmillimeter-wave visual modality person retrieval and identification system, such as Figure 1 As shown, it includes at least a radio frequency feature extraction module, a visual feature extraction module, a cross-modal coordinated representation module, a multi-dimensional biometric projection module, and a cross-modal personnel retrieval module.

[0033] The radio frequency feature extraction module is used to process point cloud sequences captured by millimeter-wave radar. First, millimeter-wave point cloud data generated by human motion is acquired using millimeter-wave radar equipment. Then, a spatiotemporal feature encoder for point clouds is constructed using an improved point cloud network PointNet, a Transformer encoder, and a bidirectional long short-term memory network Bi-LSTM to extract point cloud features.

[0034] like Figure 2 As shown, the radio frequency feature extraction module first extracts the original 5D millimeter-wave point cloud. A four-layer shared-weight multilayer perceptron (MLP) maps N×5 features to N×29 high-dimensional point features. Each MLP layer includes a Conv1D convolutional layer, a ReLU activation function, and a BN layer to enhance feature representation. A Transformer encoder is then used to model the spatial relationships between points within each frame. Next, a multi-head self-attention mechanism is used to process the frame sequence, calculate the spatial dependencies within the point cloud, and use positional encoding to preserve spatial location information. Finally, point features are aggregated to generate an enhanced frame-level representation. Finally, the frame-level feature sequences are received via Bi-LSTM and processed sequentially from the first frame to the last frame and from the last frame to the first frame via forward and backward LSTM networks, respectively, to obtain temporal dependencies. The final radar feature representation is then aggregated using the SUM function. It effectively captures the spatiotemporal dynamic characteristics of human movement.

[0035] The visual feature extraction module is used to process video frame sequences captured by RGB cameras. It captures human gait video sequences using RGB cameras, extracts spatial features from each frame using a pre-trained ResNet-50 model, and then captures the temporal evolution characteristics of the gait using a Bi-LSTM network to generate discriminative visual feature representations.

[0036] Specifically, such as Figure 3 As shown, firstly, using RGB video sequences... As input, a ResNet-50 model pre-trained on a large-scale image dataset is used to extract spatial semantic features for each frame. The ResNet-50 model contains multiple residual blocks, each consisting of convolutional layers, ReLU activation functions, and pooling layers. Through multiple convolutional operations, visual semantic features are extracted progressively from low to high levels. Subsequently, the frame-level feature sequence is received via Bi-LSTM. By using forward and backward LSTM networks to process the data sequentially from the first frame to the last frame and from the last frame to the first frame, respectively, the temporal evolution features of gait are captured. Finally, the temporal features output by the Bi-LSTM are aggregated using the SUM function to generate a discriminative visual feature representation. This fully preserves information about an individual's appearance and movement patterns during the walking process.

[0037] In the cross-modal coordinated representation module, the radio frequency (RF) features output from the RF feature extraction module and the visual features output from the visual feature extraction module are input. Internal feature enhancement is performed on both RF and visual features using a self-attention mechanism, calculating query, key, and value projections to generate enhanced features. Then, a bidirectional cross-modal attention mechanism is employed to achieve bidirectional feature exchange between RF and visual modes, and complementary information is aggregated through attention weights. This mechanism effectively mitigates the modal differences between 3D sparse millimeter-wave point clouds and 2D RGB images, fully utilizing the complementary characteristics between the two modalities to output a fused high-dimensional feature representation.

[0038] The core cross-modal coordination representation module's cross-modal attention mechanism is as follows: Figure 4 As shown: First, radio frequency characteristics and visual features Self-attention processing is performed separately, where radio frequency features are processed through a projection matrix. , and Generate query ,key Sum Visual features are projected through a matrix. , and Generate query ,key Sum ,calculate and and application The function obtains the attention weights within the modality, and the self-attention-enhanced radio frequency features are obtained by weighted summation. and visual features To enhance feature consistency within a single mode; subsequently, a bidirectional cross-modal attention mechanism is used to achieve feature complementarity and exchange between radar and visual modalities, first from the radio frequency mode to the visual modal, where the radio frequency features enhanced by the self-attention mechanism are transferred. and visual features As input, radio frequency characteristics pass Generate query Visual features pass and Generate key Sum ,calculate The attention weights are obtained by applying the softmax function, and the enhanced radio frequency features are generated by weighted summation of the visual value projections using these attention weights. Then comes the conversion from visual modality to radio frequency modality, which enhances the radio frequency features through a self-attention mechanism. and visual features As input, radio frequency characteristics pass and Generate key Sum Visual features pass Generate query ,calculate The attention weights are obtained by applying the softmax function, and the enhanced radio frequency features are generated by weighted summation of the radio frequency values ​​projected by the attention weights. This effectively alleviates modal differences.

[0039] In the multidimensional biometric projection module, a multilayer perceptron (MLP) with shared weights is used to coordinate the cross-modal features. and The predictions are mapped to three dedicated prediction heads to predict an individual's pose, height, and BMI stable biometrics. The prediction results are supervised by an L2 loss function. The auxiliary losses of the radio frequency and vision branches are combined to construct a multi-task joint training objective, enabling the network to learn dynamic gait patterns and stable biometrics simultaneously, thereby solving the gait instability problem and enhancing the model's robustness to gait changes.

[0040] In the cross-modal personnel retrieval module, the features supervised by the multi-biometric projector are L2 normalized and mapped to a unified feature space. The matching result is determined by calculating the Euclidean distance between the radio frequency query sample and the visual image library sample, thereby achieving accurate matching of personnel between the radio frequency privacy region and the visual open region, and completing personnel retrieval and identification.

[0041] Example 2:

[0042] A method for cross-millimeter-wave-visual modality person retrieval and identification based on multidimensional biometrics, such as Figure 1 As shown, cross-modal millimeter-wave vision-based person retrieval and recognition is achieved by creatively integrating multi-biometric features with cross-modal representation learning. The specific steps include:

[0043] Step S1, Radio Frequency Feature Extraction: Millimeter-wave point cloud data generated by human motion is acquired using millimeter-wave radar equipment. A point cloud spatiotemporal feature encoder is constructed using an improved point cloud network PointNet, a Transformer encoder, and a bidirectional long short-term memory network Bi-LSTM to extract point cloud features.

[0044] S11: Millimeter-wave radar point cloud sequence As input, where This represents the total number of frames in the sequence, where t represents the index of the current frame, and 1 ≤ t ≤ Each frame Includes N 5D radar points Each point ,in () represents the 3D spatial coordinates of that point. This indicates the signal strength at that point. This indicates the radial velocity at that point;

[0045] S12: Using a multilayer perceptron (MLP) with shared weights, each original point pi,t is mapped to a high-dimensional feature representation:

[0046] ;

[0047] in It is the encoded feature vector of point i in frame t. Indicates MLP parameters;

[0048] S13: Model the spatial relationships between points within each frame using a Transformer encoder:

[0049] ;

[0050] in Represents the original point features. Point features representing spatial enhancement, Includes multi-head self-attention and feedforward network parameters;

[0051] S14: Calculate a normalized importance score for each point using an attention mechanism, and aggregate them into a frame-level representation:

[0052] ;

[0053] in The frame features generated for the t-th frame. Let A() be the normalized exponential function, and let A() represent the attention scoring function. It is an attention parameter;

[0054] S15: Using a bidirectional long short-term memory (Bi-LSTM) network to model temporal dependencies in frame feature sequences:

[0055] ;

[0056] in These are Bi-LSTM parameters;

[0057] S16: Aggregate Bi-LSTM output features using the SUM function This generates the final radio frequency feature representation. .

[0058] Step S2, Visual Feature Extraction: Human gait video sequences are captured using an RGB camera. Spatial features of each frame are extracted using a pre-trained ResNet-50 model, and then the temporal evolution characteristics of the gait are captured by a Bi-LSTM network to generate discriminative visual feature representations.

[0059] S21: Convert the RGB video sequence As input, where t represents the current frame index, 1 ≤ t ≤ , Represents RGB frames. It is the length of the sequence that records human gait behavior;

[0060] S22: Spatial features are extracted for each frame using a pre-trained ResNet-50 model:

[0061] ;

[0062] Where flatten means flattening the multidimensional feature map output by ResNet into a one-dimensional vector. These are ResNet parameters. This represents the initial visual features of frame t;

[0063] S23: Frame feature sequence The data is fed into a Bi-LSTM network to capture the temporal evolution characteristics of gait:

[0064] ;

[0065] in These are Bi-LSTM parameters. This represents the output characteristics of Bi-LSTM in frame t.

[0066] S24: Aggregating Bi-LSTM output features using the SUM function This generates the final visual feature representation. It effectively captures spatiotemporal information in video sequences.

[0067] Step S3, Cross-modal Coordination: Taking the initial radio frequency (RF) and visual features output from steps S1 and S2 as input, the RF and visual features are first enhanced internally using a self-attention mechanism, calculating query, key, and value projections to generate enhanced features. Then, a bidirectional cross-modal attention mechanism is employed to achieve bidirectional feature exchange between RF and visual modes, aggregating complementary information through attention weights. This mechanism effectively mitigates the modal differences between 3D sparse millimeter-wave point clouds and 2D RGB images, fully utilizing the complementary characteristics between the two modalities to output a fused high-dimensional feature representation.

[0068] S31: Use a self-attention mechanism to enhance features of the same modality. First, enhance the radio frequency features. As input, the query projection of the radio frequency characteristics is calculated. Key projection Sum projection :

[0069] ;

[0070] in , , It is a learnable projection matrix for radio frequency modes;

[0071] Similarly, for visual features Query projection of computational visual features Key projection Sum projection :

[0072] ;

[0073] in , , It is a learnable projection matrix for visual modalities;

[0074] S32: To prevent gradient instability, the attention weights of the radio frequency modes are calculated by introducing a scaling factor. Attention weights with visual modalities :

[0075] , ;

[0076] in It is the dimension of the projected features;

[0077] S33: The optimized output features are obtained through weighted summation.

[0078] , ;

[0079] in and These are radio frequency features and visual features enhanced by a self-attention mechanism, respectively.

[0080] S34: Cross-modal attention mechanism, firstly, from radio frequency modality to visual modality, the radio frequency features enhanced by the self-attention mechanism are transferred. and visual features As input, the query projection of the radio frequency features and the key-value projection of the visual features are calculated:

[0081] ;

[0082] in It is a learnable projection matrix of radio frequency modes. and It is a learnable projection matrix of the visual modality;

[0083] S35: Calculate attention weights using the scaled dot product between the radio frequency query and the visual key projection.

[0084] ;

[0085] in It is the dimension of the projected features;

[0086] S36: Obtain the visual features enhanced by the cross-modal attention mechanism by weighted summation of the visual value projections using attention weights.

[0087] ;

[0088] in These are visual features enhanced through cross-modal attention mechanisms;

[0089] S37: To achieve reverse interaction from the visual modality to the radio frequency modality, the radio frequency features enhanced by the self-attention mechanism are... and visual features As input, the query projection of visual features is calculated, along with the key-value projection of radio frequency features:

[0090] ;

[0091] in It is a learnable projection matrix of the visual modality. and It is a learnable projection matrix of radio frequency modes;

[0092] S38: Calculate attention weights using the scaled dot product between the visual query and the RF key projection.

[0093] ;

[0094] in It is the dimension of the projected features;

[0095] S39: The RF features enhanced by the cross-modal attention mechanism are obtained by weighted summation of the RF values ​​projected by the attention weights.

[0096] ;

[0097] in It is a radio frequency feature enhanced by a cross-modal attention mechanism.

[0098] Step S4, Biometric feature acquisition: The cross-modal coordinated representation features are mapped to three dedicated prediction heads for pose, height and BMI using a multilayer perceptron (MLP) with shared weights. Multi-task joint supervised training is performed through the L2 loss function to constrain the network to capture dynamic gait patterns and stable biometric features at the same time, thereby enhancing the model's robustness to gait changes.

[0099] S41: For the RF branch, with enhanced RF characteristics. As input, the data is mapped to three dedicated prediction heads via a weighted MLP:

[0100] ;

[0101] in , , These represent the predicted posture, height, and BMI, respectively. , , These are the parameters of the corresponding MLP;

[0102] S42: For the RF branch, a multi-task supervision strategy is adopted, using the L2 loss function to supervise the difference between the predicted results and the actual measured values ​​p, h, and b:

[0103] ;

[0104] in This represents the L2 paradigm, used to calculate the Euclidean distance between the predicted and actual values.

[0105] S43: Overall loss of the RF branch It can be represented as:

[0106] ;

[0107] in =0.25 and =0.25 are the loss weighting coefficients for height and BMI, respectively, to balance the importance of different biological characteristics;

[0108] S44: For visual branches, with enhanced visual features As input, the data is mapped to three dedicated prediction heads via a weighted MLP:

[0109] ;

[0110] in , , These represent the predicted posture, height, and BMI, respectively. , , These are the parameters of the corresponding MLP;

[0111] S45: For the vision branch, a multi-task supervision strategy is adopted, using the L2 loss function to supervise the difference between the prediction results and the true measurements p, h, and b:

[0112] ;

[0113] in This represents the L2 paradigm, used to calculate the Euclidean distance between the predicted and actual values.

[0114] S46: Overall loss of the visual branch It can be represented as:

[0115] ;

[0116] in =0.25 and =0.25 are the loss weight coefficients for height and BMI, respectively, to ensure consistency between the two modalities in biometric prediction;

[0117] S47: The final overall training objective is represented as a weighted sum of the auxiliary losses on the two modal branches:

[0118] ;

[0119] in =0.5 and =0.5 is a coefficient that balances the contributions of the two modes to achieve joint optimization.

[0120] Step S5, Cross-modal Person Retrieval Module: The features extracted by the multi-dimensional biometric projection module are subjected to L2 normalization and mapped to a unified metric space. The Euclidean distance between the radio frequency query samples and the visual image library samples is calculated for sorting and matching.

[0121] S51: Features enhanced by cross-modal coordinated representation supervised by a multidimensional biometric projection module. and Each part is subjected to L2 normalization and mapped to a unified metric space:

[0122] ;

[0123] in Indicates L2 normal form, As a radio frequency query sample As a sample for a visual image library;

[0124] S52: Calculate the Euclidean distance between the RF query sample and all visual library samples:

[0125] , i = 1, 2, ..., n;

[0126] Where n is the total number of samples in the visual image library. This represents the distance between the radio frequency query sample and the i-th visual sample;

[0127] S53: Sort the visual image library samples in ascending order based on the calculated Euclidean distance to obtain the sorted index sequence:

[0128] ;

[0129] Obtain the sorted sample index sequence ,in ;

[0130] S54: Select the top k samples from the sorting results as... Matching result set: ;

[0131] S55: From Select the top-ranked sample from the matching results set. This will be output as the final matching result.

[0132] In summary, this invention systematically addresses two core challenges in cross-modal person retrieval and recognition: significant differences between radio frequency and visual modalities, and gait instability, by constructing a cross-modal coordinated representation module and a multi-dimensional biometric projection module. On one hand, it enhances and complements features through self-attention and cross-modal attention mechanisms, effectively mitigating modal differences between 3D radar point clouds and 2D RGB images, achieving cross-modal feature complementarity fusion. On the other hand, it innovatively introduces a multi-dimensional biometric projection module to accurately extract long-term stable biometric features such as individual height, BMI, and posture. Through multi-task joint training, the network is constrained to simultaneously capture dynamic gait patterns and stable biometric features, overcoming gait instability caused by time, environment, and emotional factors. Ultimately, this invention provides non-contact, privacy-preserving, and highly robust cross-modal person re-identification using only low-cost millimeter-wave radar and ordinary RGB cameras, laying a practical and accurate perception foundation for privacy-sensitive scenarios such as smart city monitoring, smart home recognition, and public safety control.

[0133] Test case

[0134] To verify the effectiveness of the proposed method, personnel retrieval and identification experiments were conducted in three different environments: classroom, corridor, and lobby, demonstrating the performance advantages and impact of the proposed method.

[0135] The experiment was performed using the following configuration:

[0136] Hardware settings:

[0137] 1. Millimeter-wave radar IWR6843ISK-ODS, used to acquire millimeter-wave point cloud data;

[0138] 2. Azure Kinect v2, used for acquiring RGB images and 3D skeleton joints;

[0139] Experimental Design:

[0140] 1. Select three different environments as test sites: classroom, corridor and hall, to simulate diverse real-world application scenarios.

[0141] 2. The experiment involved a natural walking behavior. Subjects walked naturally from a distance of 3 meters to the millimeter-wave radar and camera, and the system continuously captured their gait characteristics. Each subject performed the walking task multiple times in three different environments, fully covering different lighting conditions, background interference, and changes in walking speed.

[0142] 3. The IWR6843ISK-ods millimeter-wave radar and Kinect camera were placed on a specific bracket to collect millimeter-wave point cloud data and visual modal data of the subjects, respectively.

[0143] 4. A total of 680 gait samples were collected from 34 subjects (aged 19-30 years, including 20 males and 14 females, height range 152-187 cm) in three environments and segmented into 20-frame sequences. Height and weight information for each subject were recorded simultaneously, and BMI values ​​were calculated for supervised training of the multidimensional biometric projection module.

[0144] 5. PTNet, PGNet, CrossRF, and Mission were selected as comparison methods, with Mission being the best existing method.

[0145] The experimental results are shown in Tables 1 and 2 below:

[0146] Table 1. Performance of each model in cross-modal person retrieval and identification.

[0147]

[0148] Table 1 above presents a comparison of the performance of each model in cross-modal person retrieval and recognition. As shown in Table 1, the mmVID method presented in this study exhibits the best performance, specifically with a Top-1 accuracy of 91.3%, a Top-3 accuracy of 95.4%, and a Top-5 accuracy of 97.5%, representing an improvement of approximately 7% compared to the existing best method, Mission. These results demonstrate that the cross-modal coordinated representation module and multi-dimensional biometric projection module proposed in this system effectively address the modal differences between different sensors and the gait instability caused by cross-temporal and spatiotemporal data acquisition, thereby improving the performance of cross-modal person retrieval and recognition.

[0149] Further analysis of the performance of the multidimensional biometric projection module of this system is shown in Table 2 below:

[0150] Table 2 Performance of the Multidimensional Biometric Projection Module in This System

[0151]

[0152] As shown in Table 2 above, this system can effectively predict stable biometric features such as height, BMI, and body posture in both radio frequency (RF) and visual modes. Specifically, the mean absolute errors for height, BMI, and body posture in the RF mode are 4.39 cm, 1.63 kg / m, and 1.63 kg / m, respectively. 2 The mean absolute errors for the two measurements, 3.90 cm and 3.90 cm, respectively, are 5.67 cm and 2.78 kg / m in the visual modality. 2 The dimensions of 4.62 cm provide a strong discriminative feature for cross-modal person retrieval and identification.

[0153] To verify the effectiveness of each module in this system, the following ablation experiments will be conducted: (1) Full version: All components of this system. (2) Without multi-dimensional biometric projection module (w / o B): The multi-dimensional biometric projection module is removed, while other modules are retained. That is, the system only uses cross-modal coordinated features for personnel re-identification and does not introduce biometric supervision such as height, BMI and posture. (3) Without cross-modal coordinated representation module (w / o C): The cross-modal attention mechanism is disabled, and the output features of the multi-modal feature extraction module are directly used for personnel retrieval and identification without inter-modal feature complementarity exchange. (4) Both modules are disabled at the same time (w / o C+B): The cross-modal coordinated representation module and the multi-dimensional biometric projection module are removed at the same time, and the system degenerates into simple multi-modal feature splicing and direct retrieval.

[0154] Experimental results are as follows Figure 5 As shown, removing the multidimensional biometric projection module reduced Top-1 and Top-5 accuracies by 13.8% and 11.3%, respectively; removing the cross-modal coordinated representation module resulted in a more significant performance decline, with Top-1 and Top-5 accuracies dropping to 66.5% and 72.8%, respectively; and disabling both modules simultaneously reduced Top-1 and Top-5 accuracies to 51.4% and 59.8%, respectively. This fully demonstrates the necessity and synergistic effect of the two modules, especially the multidimensional biometric projection module, which effectively overcomes gait perturbation problems caused by time, environment, and emotional factors, significantly improving system robustness.

[0155] Experimental results on the impact of different input sequence lengths on system performance are as follows: Figure 6 As shown, by Figure 6 As can be seen, the system's recognition accuracy steadily increases with the increase of sequence length. When the sequence length reaches 20 frames, the system reaches its peak performance, with a Top-1 accuracy of 91.3% and a Top-5 accuracy of 97.5%. It is worth noting that even under the extreme condition of using only 5 frames, approximately 0.5 seconds of data, the system still maintains a Top-1 accuracy of 79.5%, fully demonstrating the system's strong robustness to short sequences or fragmented input data.

[0156] In summary, this solution proposes a method and system for cross-millimeter-wave-visual modality person retrieval and identification based on multidimensional biometrics. By constructing a cross-modal coordinated representation module and a multidimensional biometric projection module, it systematically addresses the two core challenges of cross-modal person retrieval and identification: significant differences between radio frequency and visual modalities and gait instability. This solution enables contactless and privacy-friendly person retrieval and identification in privacy-sensitive scenarios.

[0157] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. A cross-modal personnel retrieval and identification system based on multidimensional biometrics, characterized in that: It includes at least a radio frequency feature extraction module, a visual feature extraction module, a cross-modal coordinated representation module, a multi-dimensional biometric projection module, and a cross-modal personnel retrieval module. The radio frequency feature extraction module: constructs a point cloud spatiotemporal feature encoder by using an improved point cloud network PointNet, a Transformer encoder, and a bidirectional long short-term memory network Bi-LSTM to extract point cloud features from millimeter-wave point cloud data generated by human motion, thereby obtaining the radio frequency features of human motion. The visual feature extraction module acquires a video frame sequence of human motion, processes the video frame sequence using a pre-trained ResNet-50 model and a Bi-LSTM network, and obtains visual features. The cross-modal coordinated representation module: inputs the radio frequency features output by the radio frequency feature extraction module and the visual features output by the visual feature extraction module, performs self-attention processing, and generates fused and enhanced radio frequency features and visual features through a bidirectional cross-modal attention mechanism; The multidimensional biometric projection module uses a multilayer perceptron (MLP) with shared weights to map the fused features output by the cross-modal coordinated representation module onto three dedicated prediction heads for posture, height, and BMI, thereby extracting biometric features of posture, height, and BMI. The cross-modal personnel retrieval module performs L2 normalization on the biometric features extracted by the multi-dimensional biometric projection module, maps them to a unified metric space, and performs sorting and matching by calculating the Euclidean distance between the radio frequency query samples and the visual image library samples to complete personnel retrieval and identification.

2. A cross-modal person retrieval and identification method based on multidimensional biometrics, characterized in that, It should include at least the following steps: S1. Point cloud feature extraction: Obtain raw millimeter-wave point cloud data generated by human motion, map the raw millimeter-wave point cloud data into high-dimensional point features through a multilayer perceptron (MLP) with shared weights, model the spatial relationship between points in each frame using a Transformer encoder, aggregate point features through an attention mechanism to generate frame-level representations, and finally model temporal dependencies through a Bi-LSTM network and aggregate output features through a SUM function to generate radio frequency feature representations. S2. Visual Feature Acquisition: Acquire video frame sequences of hot-moving motion, extract spatial features of each frame through a pre-trained ResNet-50 model, and then capture the temporal evolution characteristics of motion through a Bi-LSTM network to generate visual feature representations. S3. Cross-modal coordination: Through the self-attention mechanism, the radio frequency features output in step S1 and the visual features output in step S2 are internally enhanced respectively. The query, key and value projection are calculated to generate enhanced features. A bidirectional cross-modal attention mechanism is adopted to realize bidirectional feature exchange from radio frequency to vision and from vision to radio frequency. Complementary information is aggregated by attention weight weighting to output a high-dimensional feature representation of the fused radio frequency features and visual features. S4. Biometric feature extraction: The high-dimensional features obtained in step S3 are mapped to three dedicated prediction heads, namely pose, height and BMI, using a multi-task joint supervised training through the L2 loss function to extract biometric features. S5. Cross-modal personnel retrieval: The features extracted in step S4 are processed by L2 normalization and mapped to a unified metric space. The Euclidean distance between the radio frequency query samples and the visual image library samples is calculated for sorting and matching to complete personnel retrieval and identification.

3. The cross-modal person retrieval and identification method based on multidimensional biometrics as described in claim 2, characterized in that: Step S1 specifically includes the following steps: S11: Acquire millimeter-wave radar point cloud sequence ,in This represents the total number of frames in the sequence, where t represents the index of the current frame, and 1 ≤ t ≤ Each frame Includes N 5D radar points Each point ,in () represents the 3D spatial coordinates of that point. This indicates the signal strength at that point. This indicates the radial velocity at that point; S12: Utilize a multilayer perceptron (MLP) with shared weights to process each original point. Mapped to a high-dimensional feature representation: ； in It is the encoded feature vector of point i in frame t. Indicates MLP parameters; S13: Model the spatial relationships between points within each frame using a Transformer encoder: ； in Represents the original point features. Point features representing spatial enhancement, Includes multi-head self-attention and feedforward network parameters; S14: Calculate a normalized importance score for each point using an attention mechanism, and aggregate them into a frame-level representation: ； in The frame features generated for the t-th frame. Let A() be the normalized exponential function, and let A() represent the attention scoring function. It is an attention parameter; S15: Using a bidirectional long short-term memory network (Bi-LSTM) to model temporal dependencies in frame feature sequences: ； in These are Bi-LSTM parameters; S16: Aggregate Bi-LSTM output features using the SUM function This generates the final radio frequency feature representation. .

4. The cross-modal person retrieval and identification method based on multidimensional biometrics as described in claim 2, characterized in that: In step S2, visual feature acquisition, a pre-trained ResNet-50 model is used to extract spatial features for each frame: ； Where flatten means flattening the multidimensional feature map output by ResNet into a one-dimensional vector. These are ResNet parameters. This represents the initial visual features of frame t; Represents an RGB frame; frame feature sequence The data is fed into a Bi-LSTM network to capture the temporal evolution characteristics of gait: ； in These are Bi-LSTM parameters. This represents the output characteristics of Bi-LSTM in frame t.

5. The cross-modal person retrieval and identification method based on multidimensional biometrics as described in claim 3 or 4, characterized in that: Step S3 specifically includes the following steps: S31: Calculate radio frequency characteristics query projection Key projection Sum projection : ； in , , It is a learnable projection matrix for radio frequency modes; Computational visual features query projection Key projection Sum projection : ； in , , It is a learnable projection matrix for visual modalities; S32: Calculate the attention weights for the radio frequency modes. Attention weights with visual modalities : ， ； in It is the dimension of the projected features; S33: The optimized output features are obtained through weighted summation. ， ； in and These are radio frequency features and visual features enhanced by a self-attention mechanism, respectively. S3 4: Calculate the query projection of radio frequency features and the key-value projection of visual features: ； in It is a learnable projection matrix of radio frequency modes. and It is a learnable projection matrix of the visual modality; S35: Calculate attention weights using the scaled dot product between the radio frequency query and the visual key projection. ； in It is the dimension of the projected features; S36: The visual features enhanced by the cross-modal attention mechanism are obtained by weighting the attention weights obtained in step S35 onto the visual value projection. ； in These are visual features enhanced through cross-modal attention mechanisms; S37: Query projection of computational visual features and key-value projection of radio frequency features: ； in It is a learnable projection matrix of the visual modality. and It is a learnable projection matrix of radio frequency modes; S38: Calculate attention weights using the scaled dot product between the visual query and the RF key projection. ； in It is the dimension of the projected features; S39: The RF characteristics enhanced by the cross-modal attention mechanism are obtained by weighted summation of the RF values ​​projected by the attention weights in step S38. ； in It is a radio frequency feature enhanced by a cross-modal attention mechanism.

6. The cross-modal person retrieval and identification method based on multidimensional biometrics as described in claim 2, characterized in that: In step S4, for radio frequency features, the L2 loss function is used to supervise the difference between the predicted results and the actual measured values ​​p, h, and b: ； in , , These represent the predicted posture, height, and BMI, respectively. This represents the L2 paradigm, used to calculate the Euclidean distance between the predicted and actual values. Overall loss of radio frequency characteristics for: ； in and These are the loss weighting coefficients for height and BMI, respectively. For visual features, the L2 loss function is used to supervise the difference between the predicted results and the true measurements p, h, and b: ； in , , These represent the predicted posture, height, and BMI, respectively. This represents the L2 paradigm, used to calculate the Euclidean distance between the predicted and actual values. Overall loss of visual features for: 。 7. The cross-modal person retrieval and identification method based on multidimensional biometrics as described in claim 5, characterized in that: Step S5 specifically includes the following steps: S51: Features that enhance cross-modal coordinated representation and Each part is subjected to L2 normalization and mapped to a unified metric space: ； in Indicates L2 normal form, As a radio frequency query sample As a sample for a visual image library; S52: Calculate the Euclidean distance between the RF query sample and all visual library samples: , i = 1, 2, ..., n; Where n is the total number of samples in the visual image library. This represents the distance between the radio frequency query sample and the i-th visual sample; S53: Sort the visual image library samples in ascending order based on the calculated Euclidean distance to obtain the sorted index sequence: ； Obtain the sorted sample index sequence ,in ; S54: Select the top k samples from the sorting results as... Matching result set: ； S55: From Select the top-ranked sample from the matching results set. This will be output as the final matching result.

Images