A Wi-Fi identity and posture recognition method based on multi-modal fusion and continuous similarity comparison learning

Abstract

The application discloses a Wi-Fi identity and posture recognition method based on multi-modal fusion and continuous similarity contrast learning. The method utilizes a Wi-Fi transceiving device to collect channel state information in the human body activity process, carries out inter-antenna ratio operation, phase correction, sliding window segmentation and data enhancement on the channel state information, and obtains amplitude data and Doppler spectrum data; antenna channel attention is adopted to adaptively learn feature weights of different antenna pairs; the amplitude data and the Doppler spectrum data are respectively input into double encoders for feature extraction, and the complementary features of the two modalities are adaptively fused through soft attention fusion; a continuous similarity supervision signal is constructed according to the matching relationship between sample identity labels and posture labels, the feature distribution is constrained by using continuous similarity contrast learning in a unified projection space, and multi-task joint optimization is carried out in combination with identity classification loss and posture classification loss, so as to obtain identity recognition results and posture recognition results.

Description

Technical Field

[0001] This invention relates to the fields of wireless sensing, human activity recognition, and artificial intelligence, and in particular to a joint Wi-Fi identity and posture recognition method based on multimodal channel state information fusion, antenna channel attention, and continuous similarity supervised contrastive learning. Background Technology

[0002] Human identification and posture recognition are fundamental to applications such as smart homes, health monitoring, personnel management, abnormal behavior detection, and human-computer interaction. Existing identification solutions typically employ cameras, wearable devices, or dedicated radar. Cameras are susceptible to lighting, occlusion, and viewing angle, and pose privacy risks; wearable devices require continuous wear by the user; and dedicated radar increases deployment costs. Wi-Fi devices are widely distributed, and their signals are reflected, scattered, and diffracted by human movement during propagation. Human identification characteristics and posture changes can be characterized through amplitude, phase, and Doppler variations in Channel State Information (CSI). Therefore, Wi-Fi offers advantages such as low cost, contactless operation, and privacy protection, as it is device-aware.

[0003] However, raw CSI is susceptible to carrier frequency offset, sampling time offset, hardware random phase, environmental multipath, and anomalous subcarriers. Different transmit and receive antennas have varying sensitivities to human motion, and directly stitching together all antenna links introduces redundant information and low-quality link noise. While amplitude features alone provide relatively stable signals, they are insufficient for characterizing speed and fine-grained motion changes; while phase or Doppler features alone are easily affected by hardware errors and environmental variations. Traditional direct feature stitching struggles to adaptively utilize different modes based on the samples.

[0004] Furthermore, existing Wi-Fi identification methods often train identity recognition and pose recognition separately, or rely solely on a shared backbone network for multi-task learning. Identity and pose information are coupled within CSI, and limited multi-label data can easily lead to overfitting. Conventional supervised contrastive learning typically only distinguishes between positive and negative samples, failing to describe partial similarity relationships such as "same identity but different pose" or "different identity but same pose." Using two separate projectors may also fragment the shared structure of the two types of labels. Therefore, a joint identification method that can simultaneously utilize multiple antennas, multiple modalities, and continuous identity-pose similarity relationships is needed. Summary of the Invention

[0005] The technical problem to be solved by this invention is: to address the issues of random phase shift, inconsistent link quality among different antennas, significant differences in amplitude and Doppler modes in Wi-Fi CSI, semantic coupling between identity and attitude, and easy overfitting under small sample conditions, and to provide a Wi-Fi identity and attitude recognition method based on multimodal fusion and continuous similarity comparison learning.

[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution.

[0007] A Wi-Fi identity and pose recognition method based on multimodal fusion and continuous similarity comparison learning includes the following steps: Step 1: Obtain amplitude data and Doppler spectrum data constructed from Wi-Fi channel state information; Step 2: Perform antenna channel attention weighting: Take each antenna pair or channel state information ratio link in the amplitude data and Doppler spectrum data as the input channel, obtain the channel description vector through global compression, generate the attention weight of each input channel according to the channel description vector, and use the attention weight to weight the corresponding input channel. Step 3: Perform dual encoder feature extraction and soft attention fusion: Input the weighted amplitude data and Doppler spectrum data into the first encoder and the second encoder, which have different parameters, respectively, to obtain amplitude features and Doppler features; generate normalized soft attention weights based on the amplitude features and Doppler features, and use the normalized soft attention weights to fuse the two features to obtain fused features; Step 4: Perform continuous similarity supervised contrastive learning: Input the fused features into a single shared projection head and normalize the projection results to map each sample to a unified embedding space. Set hierarchical continuous similarity weights according to the matching relationship between the identity labels and pose labels of any two samples in the training batch, constrain the feature distribution of samples in the unified embedding space according to the hierarchical continuous similarity weights, and calculate the continuous similarity supervised contrastive loss. The supervised contrastive loss is calculated based on continuous similarity weights, the similarity between sample embeddings, and a temperature parameter. For anchor samples without valid similar samples, the contrastive loss calculation is skipped or the corresponding loss is set to zero. Using a single shared projection head enables different label relationships to form a continuous structure within the same embedding space, avoiding semantic fragmentation caused by independent projection spaces.

[0008] Step 5: Perform multi-task joint optimization and prediction; input the fused features into the identity classification head and pose classification head respectively to obtain the identity prediction result and pose prediction result, and calculate the identity classification loss and pose classification loss; jointly optimize the antenna channel attention, the first encoder, the second encoder, soft attention fusion, the single shared projection head, the identity classification head and the pose classification head based on the identity classification loss, pose classification loss and continuous similarity supervised contrast loss.

[0009] Furthermore, in step one, the data collection process is as follows: using a Wi-Fi transceiver device with at least two transmitting antennas and at least two receiving antennas, the Wi-Fi complex channel state information corresponding to each transmitting and receiving antenna pair, each subcarrier, and each time sampling point is continuously collected during the process of the human body performing the posture or action to be identified; each piece of information is labeled with an identity tag and a posture tag; The process involves preprocessing Wi-Fi complex channel state information and constructing amplitude and Doppler spectrum data: Amplitude data is extracted from the Wi-Fi complex channel state information; a complex ratio is calculated using a CSI link with a shared transmit or receive antenna, and a preset positive number is added to the denominator to avoid numerical instability. Due to hardware errors at the shared transmitter or receiver of the link involved in the ratio calculation, the ratio can at least partially offset the common random phase shift; the phase of the ratio is unwrapped and time-differentiated along the time dimension, and a Doppler frequency shift sequence is obtained based on the phase change of the sampling frequency. The Doppler spectrum is then obtained through short-time Fourier transform; outlier suppression, smoothing, interpolation, normalization, and sliding window segmentation are performed on the amplitude and Doppler data.

[0010] Furthermore, in step two, the antenna channel attention weighting is specifically as follows: For each antenna pair or channel state information ratio link in the amplitude data and Doppler spectrum data as input channels, an antenna channel attention module is set up respectively. The antenna channel attention module performs global average pooling along the non-channel dimension for each input channel to obtain a channel description vector characterizing the overall response of each link. The channel description vector is sequentially input into a dimension-reduced fully connected layer, a nonlinear activation layer, a dimension-upgraded fully connected layer and a Sigmoid activation layer to obtain the attention weight of each input channel. The attention weight is then multiplied and weighted by the corresponding input channel channel by channel.

[0011] Furthermore, in step two, the input of the antenna channel attention module includes four antenna pairs or channel state information ratio links, and the channel compression ratio of the dimensionality reduction fully connected layer is four; the parameters of the antenna channel attention module corresponding to the amplitude data and the antenna channel attention module corresponding to the Doppler spectrum data are independent of each other.

[0012] Furthermore, in step three, the dual encoder feature extraction and soft attention fusion are specifically as follows: The first encoder and the second encoder employ convolutional neural networks with identical structures but without shared parameters to extract spatial-temporal variation features of amplitude data and temporal-frequency texture features of Doppler spectral data, respectively. The extracted amplitude features and Doppler features are transformed to the same dimension, and attention scores for the two features are calculated. The attention scores are normalized using the Softmax function so that the sum of the soft attention weights of the two modalities is equal to one. The two features are then weighted and summed or weighted and concatenated using the soft attention weights to obtain the fused features.

[0013] Furthermore, in step three, both the first encoder and the second encoder use DenseNet121, which includes densely connected blocks, transition layers, and global average pooling layers. When a weighted concatenation method is used, the weighted amplitude features and Doppler features are concatenated, and the fused features are generated through linear layers or convolutional layers.

[0014] Furthermore, in step four, the hierarchical continuous similarity weights are set to four levels according to the label relationship between samples: the highest weight is set when the identity labels and pose labels of two samples are the same; the first intermediate weight is set when the identity labels of two samples are different but the pose labels are the same; the second intermediate weight is set when the identity labels of two samples are the same but the pose labels are different; and the lowest weight is set when the identity labels and pose labels of two samples are different. The highest weight is greater than the first intermediate weight and the second intermediate weight, and both the first intermediate weight and the second intermediate weight are greater than the lowest weight.

[0015] Further, in step four, the highest weight, the first intermediate weight, the second intermediate weight, and the lowest weight are 1.0, 0.7, 0.3, and 0, respectively; the single shared projection head maps the fused features into 128-dimensional projection features and normalizes the projection features; the continuous similarity supervised contrast loss is calculated based on the hierarchical continuous similarity weight, the similarity between projection features, and the temperature parameter, where the temperature parameter is 0.1.

[0016] Furthermore, in step five, the multi-task joint optimization and prediction are specifically as follows: The identity classification head and pose classification head output the identity prediction probability and pose prediction probability, respectively, and calculate the identity cross-entropy loss and pose cross-entropy loss, respectively. The identity cross-entropy loss, pose cross-entropy loss and continuous similarity supervision comparison loss are weighted and summed to form the total loss and end-to-end training is performed. After training, the fusion features corresponding to the test sample are input into the identity classification head and pose classification head, respectively, and the identity category and pose category are output.

[0017] The beneficial effects of this invention are: (1) By reducing hardware random phase shift through CSI ratio, phase unwrapping and phase difference, and converting phase change into Doppler spectrum, the stability and interpretability of motion features are improved.

[0018] (2) By adaptively selecting different transmit and receive links through antenna channel attention, the redundant information and low-quality link noise introduced by directly splicing multiple antenna data can be reduced.

[0019] (3) Heterogeneous modal features are extracted by independent amplitude encoders and Doppler encoders, and then soft attention is used for sample adaptive fusion to make full use of the complementary advantages of amplitude stability and Doppler motion sensitivity.

[0020] (4) By using the identity-pose four-level continuous similarity target, completely similar, partially similar and dissimilar samples are uniformly constrained to the same embedding space, which alleviates the problem of improper handling of partially similar samples by traditional binary supervised contrastive learning.

[0021] (5) Improve the recognition accuracy, robustness and generalization ability in limited training data and complex multipath environments by jointly optimizing identity classification, pose classification and contrastive learning. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the Wi-Fi identity and posture recognition model based on multimodal fusion and continuous similarity comparison learning according to the present invention.

[0023] Figure 2 This is a visualization diagram of the amplitude, phase, and Doppler mode data corresponding to the four antenna pairs of this invention.

[0024] Figure 3 This is a schematic diagram of the experimental environment for collecting Wi-Fi channel status information according to the present invention. Detailed Implementation

[0025] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that the following embodiments are only used to explain the present invention and are not intended to limit the scope of protection of the present invention; equivalent substitutions made to the network structure, parameters, number of antennas, sampling method and classification category without departing from the concept of the present invention may fall within the scope of the technical solution of the present invention.

[0026] Step 1: Wi-Fi channel status information collection and sample construction.

[0027] like Figure 3As shown, in an indoor experimental environment, a Wi-Fi transmitter and receiver are positioned relative to each other. A human body performs a preset posture or action within the wireless sensing area formed by the two. This embodiment uses a 2×2 MIMO structure: the transmitter is a development board equipped with an Intel AX210 wireless network card, and the receiver is a computing device equipped with an Intel AX201 wireless network card. Both the transmitter and receiver are connected to two dual-band antennas, each approximately 85 cm above the ground. The transmitter continuously sends Wi-Fi data packets, and the receiver records the complex CSI corresponding to each transmit / receive antenna pair at a sampling rate of approximately 50 Hz. The device model and sampling rate are only examples; other commercial or dedicated devices capable of outputting CSI can also be used.

[0028] Participants perform preset postures or actions within the perception area formed by the transmitter and receiver. Each record is simultaneously labeled with an identity tag and a posture tag. Posture categories may include, but are not limited to, normal walking, jumping, walking with a bent-over posture, walking with both hands holding an object, limping, walking with one hand in a limited position, and walking with one hand carrying an object. To increase spatial variation, participants can perform actions along multiple directions or routes.

[0029] The continuous CSI is divided into segments using a sliding window with a length of 300 sampling frames and a step size of 100 sampling frames. Complex data for each antenna link, time point, and subcarrier are retained within each window; outlier suppression and smoothing are applied to the amplitude, and valid subcarriers are interpolated or resampled to a uniform number. In one embodiment, the amplitude samples form a tensor of size 4×300×484.

[0030] Step two: phase correction and Doppler spectrum generation.

[0031] The complex CSI data for four transmit and receive antennas within the same window is processed to obtain amplitude, phase, and Doppler mode data. Outlier suppression, smoothing, and normalization are performed on the amplitude data. Complex ratio calculations are performed on CSI links with a common transmit or receive antenna, and the phase ratio is extracted and unwrapped and subjected to first-order time difference along the time axis. The phase change is then converted into a Doppler frequency shift based on the sampling rate, and a short-time Fourier transform is performed on each link and subcarrier. The window function used can be a Hann window, with the window length, overlap rate, and frequency resolution determined based on the sampling rate.

[0032] like Figure 2As shown, four antenna pairs form interconnected but differently responding modal images under the same human activity. The amplitude mode reflects the signal strength changes of each link as the human activity occurs; the phase mode reflects the time change after link ratio and phase correction; and the Doppler mode reflects the time-frequency energy distribution caused by human movement. Different antenna pairs have different propagation paths and spatial sensitivities, resulting in differences in texture, variation range, and noise level among the three modes. The absolute value of the short-time Fourier transform result is taken and logarithmically compressed, truncated, or normalized, then interpolated along the time and frequency axes to a fixed size. In one embodiment, each link forms a 64×64 Doppler image, and the four links together form a 4×64×64 tensor. The model primarily uses the amplitude and Doppler modes as complementary inputs, while the phase mode is used for phase correction, Doppler construction, and signal characteristic analysis.

[0033] Step 3: Antenna channel attention, dual encoder feature extraction and soft attention fusion.

[0034] like Figure 1 As shown, the preprocessed amplitude and Doppler data are input into the corresponding antenna channel attention modules. Each antenna channel attention module performs global average pooling along the non-channel dimensions, outputting four link weights through a two-layer fully connected network with a compression ratio of 4, and multiplying them channel-by-channel with the original input. The parameters of the antenna channel attention modules for the two modes are independent of each other, so as to adjust them according to... Figure 2 The response differences of different antenna pairs are shown, and sensitive links in amplitude mode and Doppler mode are selected respectively.

[0035] The weighted inputs are fed into two separate DenseNet121 encoders. For cases where the number of input channels is not equal to three, the number of input channels in the first convolutional layer can be modified to the number of links, or a 1×1 convolution can be mapped to the required number of channels for the encoder. The classification layer of the encoder is removed, retaining only the convolutional backbone and the global average pooling output. The two encoders can have the same structure, but their parameters are not shared.

[0036] Amplitude feature F m After aligning the dimensions of the Doppler features F_d using a linear transformation, scalar or vector attention scores are calculated respectively, and then a is obtained by Softmax. m And a_d. The fusion feature F is derived from a m F m +a_dF_d is obtained. During training, the attention weights are updated together with the samples and network parameters, enabling the system to improve a when the amplitude is more reliable. m When the motion spectrum is more discriminative, a_d is improved.

[0037] Step 4: Continuous similarity supervised contrastive learning and multi-task joint training.

[0038] like Figure 1 As shown, the fused features obtained from soft attention fusion are simultaneously input into the identity classification head, pose classification head, and shared projection head. The identity classification head and pose classification head output the identity recognition result and pose recognition result, respectively; the shared projection head consists of one or more fully connected layers, outputting a 128-dimensional vector and performing L2 normalization. Within a training batch, a continuous similarity weight matrix is ​​constructed based on the identity label and pose label, and its diagonal is not included in the loss calculation.

[0039] In this embodiment, the weights for the four relationships—"same identity and same posture," "different identity but same posture," "same identity but different posture," and "different identity and posture"—are set to 1.0, 0.7, 0.3, and 0, respectively. These asymmetric intermediate weights reflect the different contributions of posture consistency and identity consistency to the task structure. In other applications, intermediate weights can also be determined through a validation set, ensuring that identical samples have the highest weight and completely different samples have the lowest weight.

[0040] A temperature parameter of 0.1 is used to calculate the continuous similarity supervised contrast loss; both the identity classification head and the pose classification head use cross-entropy loss. The three losses are summed with weights of 1:1:1, and backpropagation is used to update the parameters of AntennaSE, dual encoder, DualSoftFuse, shared projection head, and the two classification heads. During training, data augmentation can be performed using random jitter, random noise, amplitude scaling, temporal occlusion, or subcarrier occlusion.

[0041] During the inference phase, there is no need to calculate the intra-batch contrast loss. The test CSI is simply processed sequentially through preprocessing, AntennaSE, dual encoders, and DualSoftFuse. Then, the two classification heads output the identity and pose probabilities respectively, and the category with the highest probability is selected as the recognition result. The shared projection head can be removed or retained for feature retrieval and visualization.

[0042] Step 5: System Deployment.

[0043] The CSI acquisition module is deployed at the receiving end or on an edge device communicating with the receiving end; the preprocessing module can perform real-time packet parsing, window caching, CSI ratio generation, and Doppler spectrum generation; the identification network can be deployed on a server with a graphics processor, or it can be deployed on an edge computing platform after model pruning, quantization, or knowledge distillation. The identification results can be sent to health monitoring, personnel management, or abnormal behavior alarm applications via local or network interfaces.

[0044] When the number of transmit and receive antennas, the number of subcarriers, or the input size changes, the number of input channels of the channel attention module and the first layer structure of the encoder can be adjusted accordingly. When the task is expanded from posture recognition to action, gait, or abnormal behavior recognition, the posture label and the output dimension of the classification head can be replaced without changing the construction of continuous similarity and the joint training method.

Claims

1. A Wi-Fi identity and pose recognition method based on multimodal fusion and continuous similarity comparison learning, characterized in that, The steps include the following: Step 1: Obtain amplitude data and Doppler spectrum data constructed from Wi-Fi channel state information; Step 2: Perform antenna channel attention weighting: Take each antenna pair or channel state information ratio link in the amplitude data and Doppler spectrum data as the input channel, obtain the channel description vector through global compression, generate the attention weight of each input channel according to the channel description vector, and use the attention weight to weight the corresponding input channel. Step 3: Perform dual encoder feature extraction and soft attention fusion: Input the weighted amplitude data and Doppler spectrum data into the first encoder and the second encoder, which have different parameters, respectively, to obtain amplitude features and Doppler features; Normalized soft attention weights are generated based on the amplitude features and Doppler features, and the two features are fused using the normalized soft attention weights to obtain fused features. Step 4: Perform continuous similarity supervised contrastive learning: Input the fused features into a single shared projection head and normalize the projection results to map each sample to a unified embedding space. Set hierarchical continuous similarity weights based on the matching relationship between the identity label and pose label of any two samples in the training batch, and constrain the feature distribution of samples in the unified embedding space according to the hierarchical continuous similarity weights. Calculate the continuous similarity supervised contrastive loss. Step 5: Perform multi-task joint optimization and prediction: Input the fused features into the identity classification head and pose classification head respectively to obtain identity prediction results and pose prediction results, and calculate the identity classification loss and pose classification loss. Optimize the antenna channel attention, first encoder, second encoder, soft attention fusion, single shared projection head, identity classification head, and pose classification head jointly based on the identity classification loss, pose classification loss, and continuous similarity supervised contrastive loss.

2. The Wi-Fi identity and pose recognition method based on multimodal fusion and continuous similarity comparison learning according to claim 1, characterized in that, In step one, the data collection process is as follows: using a Wi-Fi transceiver device with at least two transmitting antennas and at least two receiving antennas, the Wi-Fi complex channel state information corresponding to each transmitting and receiving antenna pair, each subcarrier, and each time sampling point is continuously collected during the process of the human body performing the posture or action to be identified; each piece of information is labeled with an identity tag and a posture tag; Wi-Fi complex channel state information is preprocessed and amplitude data and Doppler spectrum data are constructed: amplitude data is extracted from Wi-Fi complex channel state information; complex ratio calculation is performed through CSI links with a common transmit antenna or a common receive antenna, and a preset positive number is added to the denominator to avoid numerical instability. Due to hardware errors of the shared transmitter or receiver of the link participating in the ratio calculation, the ratio can at least partially offset the common random phase offset. The phase of the comparison value is unwrapped and time-differenced along the time dimension. The Doppler frequency shift sequence is obtained based on the phase change of the sampling frequency, and the Doppler spectrum is obtained through short-time Fourier transform. Outlier suppression, smoothing, interpolation, normalization and sliding window segmentation are performed on the amplitude data and Doppler data.

3. The Wi-Fi identity and pose recognition method based on multimodal fusion and continuous similarity comparison learning according to claim 1, characterized in that, In step two, the antenna channel attention weighting is performed as follows: For each antenna pair or channel state information ratio link in the amplitude data and Doppler spectrum data as input channels, an antenna channel attention module is set up respectively. The antenna channel attention module performs global average pooling along the non-channel dimension for each input channel to obtain a channel description vector characterizing the overall response of each link. The channel description vector is sequentially input into a dimension-reduced fully connected layer, a nonlinear activation layer, a dimension-upgraded fully connected layer and a Sigmoid activation layer to obtain the attention weight of each input channel. The attention weight is then multiplied and weighted by the corresponding input channel channel by channel.

4. The Wi-Fi identity and pose recognition method based on multimodal fusion and continuous similarity comparison learning according to claim 3, characterized in that, In step two, the input to the antenna channel attention module includes four antenna pairs or channel state information ratio links, and the channel compression ratio of the dimensionality reduction fully connected layer is four; the parameters of the antenna channel attention module corresponding to the amplitude data and the antenna channel attention module corresponding to the Doppler spectrum data are independent of each other.

5. The Wi-Fi identity and pose recognition method based on multimodal fusion and continuous similarity comparison learning according to claim 1, characterized in that, In step three, the dual encoder feature extraction and soft attention fusion are specifically as follows: The first encoder and the second encoder employ convolutional neural networks with identical structures but without shared parameters to extract spatial-temporal variation features of amplitude data and temporal-frequency texture features of Doppler spectral data, respectively. The extracted amplitude features and Doppler features are transformed to the same dimension, and attention scores for the two features are calculated. The attention scores are normalized using the Softmax function so that the sum of the soft attention weights of the two modalities is equal to one. The two features are then weighted and summed or weighted and concatenated using the soft attention weights to obtain the fused features.

6. The Wi-Fi identity and pose recognition method based on multimodal fusion and continuous similarity comparison learning according to claim 5, characterized in that, In step three, both the first encoder and the second encoder use DenseNet121, which includes densely connected blocks, transition layers, and global average pooling layers. When a weighted concatenation method is used, the weighted amplitude features and Doppler features are concatenated, and the fused features are generated through linear layers or convolutional layers.

7. The Wi-Fi identity and pose recognition method based on multimodal fusion and continuous similarity comparison learning according to claim 1, characterized in that, In step four, The hierarchical continuous similarity weights are set to four levels according to the label relationship between samples: the highest weight is set when the identity label and posture label of two samples are the same; the first intermediate weight is set when the identity label of two samples is different but the posture label is the same; the second intermediate weight is set when the identity label of two samples is the same but the posture label is different; and the lowest weight is set when the identity label and posture label of two samples are different. The highest weight is greater than the first intermediate weight and the second intermediate weight, and both the first intermediate weight and the second intermediate weight are greater than the lowest weight.

8. The Wi-Fi identity and pose recognition method based on multimodal fusion and continuous similarity comparison learning according to claim 7, characterized in that, In step four, the highest weight, the first intermediate weight, the second intermediate weight, and the lowest weight are 1.0, 0.7, 0.3, and 0, respectively. The single shared projection head maps the fused features into 128-dimensional projected features and normalizes the projected features; it calculates the continuous similarity supervised contrast loss based on the hierarchical continuous similarity weight, the similarity between projected features, and the temperature parameter, where the temperature parameter is 0.

1.

9. The Wi-Fi identity and pose recognition method based on multimodal fusion and continuous similarity comparison learning according to any one of claims 1-8, characterized in that, In step five, the multi-task joint optimization and prediction are specifically as follows: The identity classification head and pose classification head output the identity prediction probability and pose prediction probability, respectively, and calculate the identity cross-entropy loss and pose cross-entropy loss, respectively. The identity cross-entropy loss, pose cross-entropy loss and continuous similarity supervision comparison loss are weighted and summed to form the total loss and end-to-end training is performed. After training, the fusion features corresponding to the test sample are input into the identity classification head and pose classification head, respectively, and the identity category and pose category are output.

Images