Cross-modal recognition matching method among multiple moving targets

Abstract

This invention relates to a cross-modal identification and matching method for multiple moving targets, belonging to the fields of information security and chaotic communication. In the radio frequency (RF) branch, the invention performs temporal phase unwrapping and linear clock drift stripping on the original CSI complex matrix, and combines dynamic subcarrier filtering and wavelet scattering networks to extract translation-invariant micro-Doppler dynamic features. In the visual branch, a spatiotemporal graph convolutional network is used to accurately extract and encode the human skeleton topology in the visual modality. An asymmetric dual-branch feature encoder is designed in the matching stage, and a gated dynamic sparse cross-attention mechanism is introduced to adaptively evaluate modal confidence and perform deep feature interaction and complementary enhancement. Experimental tests show that the system maintains extremely high cross-modal identity association accuracy even under extreme conditions such as severe visual occlusion or high dynamic radio frequency interference, breaking down the semantic barriers between RF and visual signals at the feature level, providing a definite prior guarantee for accurate coordinate fusion in multi-target environments.

Description

Technical Field

[0001] This invention relates to a cross-modal identification and matching method among multiple moving targets, belonging to the fields of information security and chaotic communication. Background Technology

[0002] Indoor positioning is a key supporting technology for smart cities, smart homes, and other fields. Robust positioning and identity matching in multi-target concurrent scenarios have always been industry challenges. Existing solutions mainly achieve cross-modal positioning by fusing WiFi channel state information (CSI) and visual signals. CSI can capture micro-Doppler features caused by human body micro-movements, while the visual skeleton can extract the topological structure of human motion. The two are naturally complementary, which theoretically can improve positioning accuracy and fault tolerance, and is suitable for complex scenarios such as multi-person offices and conference rooms.

[0003] Existing cross-modal matching technologies are mainly divided into two categories: one is the traditional two-stream network method, such as XModalID and AutoID, which extracts CSI and visual features respectively and completes the matching through shallow linear mapping. This type of scheme relies too much on gait features, only supports single-target scenes, cannot effectively decouple signal interference when multiple targets are running concurrently, and is prone to misidentification. The other type is the standard cross-attention method, which uses a global dense attention matrix to realize feature interaction. However, this type of method is susceptible to CSI multipath clutter interference and uses static fusion weights. When visual occlusion occurs or radio frequency signals are interfered with, the matching accuracy will drop sharply, and the robustness is insufficient.

[0004] In summary, existing technologies generally suffer from the following drawbacks: heterogeneous modal feature distributions vary greatly; purely data-driven methods without physical guidance are prone to learning environmental noise and have poor generalization; global dense attention is prone to generating false bindings and has a high false alarm rate; static fusion weights cannot adaptively shield failed modes and have weak anti-interference capabilities; when multiple targets are concurrent, radio frequency micro-Doppler aliasing and visual overlap problems are prominent, matching accuracy drops rapidly, and it is difficult to meet the application requirements of complex indoor scenarios. Summary of the Invention

[0005] To address the technical challenges in multi-target indoor positioning scenarios, such as significant differences in the heterogeneous feature distributions of CSI and visual modalities, susceptibility to environmental interference, and poor robustness of static fusion weights, leading to misidentification, low accuracy, and weak generalization in cross-modal target matching, this invention provides a cross-modal recognition and matching method for multiple moving targets. The technical solution is as follows: In a first aspect, the present invention provides a cross-modal localization target matching method, comprising: Step 1: WiFi mode processing, the original CSI samples are sequentially subjected to temporal phase unwrapping, sliding window linear phase error removal, and dynamic subcarrier filtering, and the filtered CSI feature vector is output; Step 2: Visual modality processing, using a pose estimation algorithm to extract the coordinate sequence of key points of the human skeleton from video frames and construct visual features; Step 3: The selected CSI feature vectors and visual features are fed into independent deep neural network branches. The CSI encoding branch uses wavelet transform to convert the selected CSI feature vectors into a time-frequency graph, extracts translation-invariant features through a wavelet scattering network, and concatenates them with physical statistical features to form a multi-dimensional fused feature vector. The multi-dimensional fused feature vector is then input into a dynamic sparse mask to obtain a high-dimensional CSI semantic vector. The visual encoding branch constructs a spatiotemporal graph from the visual features, extracts spatiotemporal features through a spatiotemporal graph convolutional network, and then inputs it into a graph convolutional sparse encoder to output a visual semantic vector with the same dimension as the high-dimensional CSI semantic vector. Step 4: Cross-modal feature fusion and enhancement. The high-dimensional CSI semantic vector and the visual semantic vector are input into a gated dynamic sparse cross-attention module. The processing of this module includes: Step 41: Calculate the correlation score matrix of the bimodal features using a two-layer multilayer perceptron; Step 42: Select features based on the correlation score matrix to generate a binary sparse mask, and perform masking to obtain sparse features; Step 43: Calculate the temporal variance of the sparsified CSI features and the visual features respectively, input the temporal variance into a single-layer perceptron and obtain the modal confidence score through Sigmoid activation, and normalize the modal confidence score to obtain the gating weights. Step 44: Use bidirectional cross-attention to perform intermodal feature interaction, connect the cross-attention output with the original sparse features through residual connection and nonlinear activation to obtain the final fused feature vector used for measurement; Step 5: Project the fused feature vector onto a shared unit hypersphere, calculate the cosine similarity between the enhanced CSI features and the visual features, and output the matching result of the target's cross-modal identity through bipartite graph matching.

[0006] Thus, by preprocessing the CSI signal and extracting wavelet scattering features, static multipath and clock drift interference are effectively suppressed. Simultaneously, combined with topological modeling of visual skeleton features, high signal-to-noise ratio and low redundancy basic features are provided for subsequent matching, reducing the impact of environmental noise at its source. Secondly, this invention introduces a gated dynamic sparse cross-attention module. This module, on the one hand, filters redundant noise in the feature dimension through a sparse masking mechanism, avoiding false binding problems caused by global dense attention and significantly reducing the false alarm rate. On the other hand, this module achieves adaptive evaluation of modality reliability through dynamic gating weights. When a modality fails due to occlusion or multipath interference, the system automatically reduces its weight and enhances the contribution of another modality, completely solving the problem of a precipitous drop in accuracy of static fusion weights under harsh conditions.

[0007] Furthermore, experimental tests show that the system maintains extremely high cross-modal identity association accuracy even under extreme conditions such as severe visual occlusion or high dynamic range radio frequency interference. It breaks down the semantic barriers between radio frequency and visual signals at the feature level, providing a definite prior guarantee for accurate coordinate fusion in multi-target environments. This invention achieves robust cross-modal matching in multi-target scenarios, significantly improving positioning accuracy and possessing stronger environmental generalization capabilities. It effectively solves the problem of identity misbinding in concurrent multi-target scenarios, providing reliable technical support for multi-target localization in complex indoor scenes.

[0008] Optionally, the dynamic subcarrier selection process in step 1 includes: Step 11: Based on the standardized amplitude and normalized phase, calculate the mean amplitude and mean phase of each subcarrier within the current time window, and construct a two-dimensional statistical feature vector characterizing the global characteristics of the subcarrier; Step 12: Concatenate the statistical feature vectors of all subcarriers to form a global feature vector; Step 13: Input the global feature vector into a lightweight gating network consisting of a fully connected layer and a Sigmoid activation function to learn the attention weights for each subcarrier; Step 14: Multiply the original input time series data element by element using the attention weights to achieve dynamic reweighting and generate the filtered feature representation.

[0009] In this way, by extracting the amplitude and phase mean values ​​of subcarriers within the time window to construct global statistical features, attention weights are generated through a lightweight fully connected network. This automatically enhances high signal-to-noise ratio subcarriers and suppresses frequency bands severely affected by static multipath interference. Without the need for manually setting fixed thresholds, this avoids the accidental deletion of effective information due to environmental changes under traditional fixed threshold strategies. It also continuously obtains high-quality subcarrier features with stable distribution, high information density, and physical consistency, providing a low-noise, highly robust CSI input foundation for subsequent cross-modal matching. This can effectively improve the feature stability and positioning reliability of the system in dynamic indoor environments.

[0010] Optionally, step 3 uses the Complex Morlet wavelet as the mother wavelet to perform a continuous wavelet transform (CWT) on the selected CSI feature vectors. The calculation formula is as follows:

[0011] in, The filtered feature vectors, , For the mother wavelet function, As a scale factor, is the translation factor, and * denotes complex conjugation.

[0012] Thus, by converting the one-dimensional time-domain CSI signal into a time-frequency map through continuous wavelet transform, and utilizing the high similarity between the Complex Morlet wavelet and the reflected pulse signal of a moving object, amplitude and phase information are effectively separated and preserved, achieving accurate characterization of micro-Doppler features. The direct technical effects are: it solves the problem that traditional time-frequency analysis methods cannot take into account time-frequency resolution, and provides a reliable time-frequency analysis foundation for subsequent wavelet scattering networks to extract translation-invariant features and construct physically guided multi-dimensional feature vectors. It significantly improves the ability of CSI features to represent human motion patterns and enhances the robustness of cross-modal matching to different action speeds and time deviations.

[0013] Optionally, the processing procedure of the wavelet scattering network includes: Using a low-pass filter For the filtered CSI feature vectors Perform direct smoothing to extract the zeroth-order scattering average feature:

[0014] Using multi-scale wavelet filters For the filtered CSI feature vectors Perform wavelet convolution and modulus operation, then pass through a low-pass filter. Extracting first-order scattering features:

[0015] Output of the magnitude of the first-order scattering feature Further implementation standards are Wavelet transform and modulus operation yield second-order scattering characteristics:

[0016] The zero-order, first-order, and second-order scattering features are combined into multi-scale translation-invariant scattering features.

[0017] In this way, by using low-pass filtering, multi-scale wavelet convolution and modulus operation, zero-order, first-order and second-order scattering features are extracted layer by layer to form multi-scale translation invariant features that are non-expansive to local deformation of the input signal. This effectively eliminates high-frequency phase oscillations, suppresses environmental noise, and makes the features naturally robust to changes in action speed and start time deviation. This provides stable and highly discriminative CSI basic features for subsequent dynamic sparse masks and enhances the generalization ability of cross-modal matching in complex scenarios.

[0018] Optionally, the physical statistical characteristics include: Doppler peak frequency, spectral centroid frequency, mean, standard deviation, skewness, and kurtosis of the signal envelope intensity.

[0019] Thus, by defining physical statistical features including Doppler peak frequency, spectral centroid frequency, and the mean, standard deviation, skewness, and kurtosis of the signal envelope intensity, clear physical meaning is provided for CSI features. These features directly correspond to core physical quantities in human kinematics, accurately characterizing limb movement speed, energy perturbation, and temporal changes in motion, effectively compensating for the lack of interpretability and overfitting to environmental noise inherent in purely data-driven features. Fusing these features with deep wavelet scattering features balances the abstract expressive power and physical fidelity of the features, improving the stability and generalization of cross-modal matching. This allows the system to reliably extract motion features even in complex multipath and multi-target interference scenarios, further enhancing the accuracy and anti-interference capability of identity matching.

[0020] Optionally, the forward propagation process of the dynamic sparse mask is represented as follows:

[0021] in, and They represent the first The weight matrix and bias terms of a fully connected layer. Represents the internal activation mapping, The term represents the activation function. The Dropout layer is used to achieve feature sparsity and noise suppression. This represents the multidimensional fusion feature vector.

[0022] Thus, feature depth mapping is achieved through multi-layer linear transformation and non-linear activation, and adaptive sparse noise reduction is completed by combining the Dropout layer. This effectively filters out environmental clutter and redundant features, completes feature dimension alignment, and outputs a high-purity CSI semantic vector, providing reliable feature support for cross-modal feature fusion matching.

[0023] Optionally, the forward propagation process of the graph convolutional sparse encoder is represented as:

[0024] In the formula, This represents the spatiotemporal features output by the spatiotemporal graph convolutional network. The adjacency matrix representing the topology of the human skeleton. , It consists of two graph convolutional layers.

[0025] Thus, by relying on two-layer graph convolution to mine the spatial topology of skeletal joints and the temporal motion correlation, and combining normalization and random deactivation for sparse noise reduction, we can effectively purify and align visual features, output stable and standardized visual semantic vectors, and ensure the consistency and reliability of bimodal feature matching.

[0026] Secondly, the present invention provides a cross-modal identification and matching system for multiple moving targets, comprising: The WiFi mode processing module is configured to sequentially perform temporal phase unwrapping, sliding window linear phase error removal, and dynamic subcarrier filtering on the original CSI samples, and output the filtered CSI feature vector. The visual modality processing module is configured to extract the coordinate sequence of human skeleton key points from video frames using a pose estimation algorithm to construct visual features; The heterogeneous feature extraction module is configured to feed the filtered CSI feature vector and visual features into independent deep neural network branches. The CSI encoding branch uses wavelet transform to convert the filtered CSI feature vector into a time-frequency graph, extracts translation-invariant features through a wavelet scattering network, and concatenates them with physical statistical features to form a multi-dimensional fused feature vector. The multi-dimensional fused feature vector is then input into a dynamic sparse mask to obtain a high-dimensional CSI semantic vector. The visual encoding branch constructs a spatiotemporal graph from the visual features, extracts spatiotemporal features through a spatiotemporal graph convolutional network, and then inputs it into a graph convolutional sparse encoder to output a visual semantic vector with the same dimension as the high-dimensional CSI semantic vector. The cross-modal feature fusion and enhancement module is configured to input the high-dimensional CSI semantic vector and the visual semantic vector into a gated dynamic sparse cross-attention module. The processing of this module includes: calculating the correlation score matrix of the dual-modal features through a two-layer multilayer perceptron; selecting features based on the correlation score matrix to generate a binary sparse mask, and performing masking to obtain sparse features; calculating the temporal variance of the sparsed CSI features and the visual features respectively, inputting the temporal variance into a single-layer perceptron and activating it with Sigmoid to obtain a modal confidence score, normalizing the modal confidence score to obtain a gate weight; performing inter-modal feature interaction with bidirectional cross-attention, and performing a residual connection between the cross-attention output and the original sparse features and nonlinear activation to obtain the final fused feature vector used for measurement. A cross-modal localization target matching system was thus constructed, fully adapted to the method. Through the division of labor and cooperation among various functional modules, the multimodal feature processing, encoding, and adaptive fusion process were completely reproduced. The system has a high degree of modularity and tight logical coupling, and can stably achieve accurate cross-modal matching of multiple targets indoors, significantly improving overall operational reliability and scene adaptability.

[0027] Thirdly, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, it implements the steps of the cross-modal identification and matching method for multiple moving targets as described in any of the preceding claims.

[0028] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, characterized in that: when the computer program is executed by a processor, it implements the steps of the cross-modal identification and matching method for multiple moving targets as described in any of the preceding claims. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is an algorithm architecture diagram of the cross-modal identification and matching method among multiple moving targets according to the present invention.

[0031] Figure 2 This is the algorithm architecture diagram of the gated dynamic sparse cross-attention module of this invention.

[0032] Figure 3 This is an experimental scenario diagram of the present invention.

[0033] Figure 4 This is a comparative analysis chart of the matching accuracy of various algorithms in a typical scenario.

[0034] Figure 5 This is a robustness analysis diagram of each algorithm under occlusion conditions.

[0035] Figure 6 This is a system reliability analysis diagram under different target densities.

[0036] Figure 7 This is an experimental analysis diagram of the incremental ablation, a key module of this invention.

[0037] Figure 8 This is a performance comparison chart of four cross-modal matching algorithms. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0039] Example 1: This embodiment provides a cross-modal identification and matching method for multiple moving targets, including: Step 1: WiFi mode processing, the original CSI samples are sequentially subjected to temporal phase unwrapping, sliding window linear phase error removal, and dynamic subcarrier filtering, and the filtered CSI feature vector is output; Step 2: Visual modality processing, using a pose estimation algorithm to extract the coordinate sequence of key points of the human skeleton from video frames and construct visual features; Step 3: The selected CSI feature vectors and visual features are fed into independent deep neural network branches. The CSI encoding branch uses wavelet transform to convert the selected CSI feature vectors into a time-frequency graph, and extracts translation-invariant features through a wavelet scattering network. After channel concatenation with the physical statistical features, a multi-dimensional fused feature vector is formed. The multi-dimensional fused feature vector is input into a dynamic sparse mask to obtain a high-dimensional CSI semantic vector. The visual encoding branch constructs a spatiotemporal graph from the visual features, extracts spatiotemporal features through a spatiotemporal graph convolutional network, and then inputs it into a graph convolutional sparse encoder to output a visual semantic vector with the same dimension as the high-dimensional CSI semantic vector. Step 4: Cross-modal feature fusion and enhancement. The high-dimensional CSI semantic vector and the visual semantic vector are input into a gated dynamic sparse cross-attention module. The processing steps of this module include: Step 41: Calculate the correlation score matrix of the bimodal features using a two-layer multilayer perceptron; Step 42: Select features based on the correlation score matrix to generate a binary sparse mask, and perform masking to obtain sparse features; Step 43: Calculate the temporal variance of the sparsified CSI features and the visual features respectively. Input the temporal variance into a single-layer perceptron and activate it with Sigmoid to obtain the modal confidence score. Normalize the modal confidence score to obtain the gating weights. Step 44: Use bidirectional cross-attention to perform intermodal feature interaction, connect the cross-attention output with the original sparse features through residual connection and nonlinear activation to obtain the final fused feature vector used for measurement; Step 5: Project the fused feature vector onto the shared unit hypersphere, calculate the cosine similarity between the enhanced CSI features and the visual features, and output the matching result of the target's cross-modal identity through bipartite graph matching.

[0040] Thus, by preprocessing the CSI signal and extracting wavelet scattering features, static multipath and clock drift interference are effectively suppressed. Simultaneously, combined with topological modeling of visual skeleton features, high signal-to-noise ratio and low redundancy basic features are provided for subsequent matching, reducing the impact of environmental noise at its source. Secondly, this embodiment introduces a gated dynamic sparse cross-attention module. This module, on the one hand, filters redundant noise in the feature dimension through a sparse masking mechanism, avoiding false binding problems caused by global dense attention and significantly reducing the false alarm rate. On the other hand, this module achieves adaptive evaluation of modality reliability through dynamic gating weights. When a modality fails due to occlusion or multipath interference, the system automatically reduces its weight and enhances the contribution of another modality, completely solving the problem of a precipitous drop in accuracy of static fusion weights under harsh conditions.

[0041] Furthermore, this embodiment achieves robust cross-modal matching in multi-target scenarios, which not only significantly improves positioning accuracy but also has stronger environmental generalization capabilities. It effectively solves the problem of mis-binding identities when multiple targets are concurrent, providing reliable technical support for multi-target positioning in complex indoor scenarios.

[0042] Example 2: This embodiment provides a cross-modal identification and matching method among multiple moving targets. Based on Embodiment 1, the process of dynamic subcarrier screening is limited.

[0043] The specific steps of this embodiment are as follows: Step 1: For the WiFi mode, the original CSI sample First, phase jumps are eliminated by time-domain phase unwrapping. Then, a sliding window is used to remove linear phase errors caused by clock drift. Finally, a dynamic subcarrier filtering module is used to remove frequency bands severely affected by static multipath interference, and the filtered feature vector is output. .

[0044] Traditional subcarrier selection methods mostly employ fixed threshold strategies based on variance or amplitude, which cannot adapt to dynamically changing environments and are prone to the accidental deletion of valid information. Therefore, this embodiment introduces a lightweight, learnable subcarrier gating module. This module utilizes a neural network to automatically perceive the quality of subcarriers and achieves dynamic weighting through an attention mechanism, thereby obtaining a batch of high-quality subcarriers with stable distribution, high information density, and physical consistency.

[0045] To evaluate each subcarrier in the current time window The overall quality within a given timeframe cannot be determined solely by the instantaneous value at a single moment. This embodiment extracts its global statistical characteristics over time.

[0046] Based on standardized amplitude With normalized phase First, calculate their mean over the entire time slice. For the , Subcarriers ( Construct a two-dimensional statistical feature vector representing its global characteristics. :

[0047] in, and These represent the mean amplitude and mean phase of the subcarrier within the current window, respectively. This feature vector effectively compresses temporal redundancy, retaining only the core statistics reflecting channel quality.

[0048] To capture the correlation and relative importance between different subcarriers, this embodiment uses the statistical feature vectors of all subcarriers. The features are concatenated to form a global feature vector with a dimension of 2K. :

[0049] Then, the vector The input is fed into a lightweight gating network consisting of fully connected layers and a sigmoid activation function. This network learns the contribution of different subcarriers to the final localization task and outputs a... dimensional attention weight vector .

[0050] No. Attention weights for each subcarrier The generation process and its mathematical expression can be simplified as follows:

[0051] in, and These are the learnable weights and bias parameters in the network. This is the Sigmoid activation function.

[0052] After obtaining the weight vector, the original input time-series data is dynamically reweighted by element-wise multiplication to generate the filtered feature representation. :

[0053] Through this mechanism, the network can automatically enhance the characteristic amplitude of high signal-to-noise ratio subcarriers while suppressing subcarriers that are severely affected by noise.

[0054] Step 2: For visual modalities, the system does not directly process RGB pixels, but instead uses a pose estimation algorithm to extract the coordinate sequence of human skeleton key points from video frames to construct visual samples. This process effectively removes interference from background, lighting, and clothing color, retaining only the geometric information describing human movement.

[0055] For the input visual video frame sequence, this embodiment uses the OpenPose pose estimation algorithm to extract human joints in each frame. Define the first... The set of human skeletons detected in a visual sample is For each target, extract The first key point, The original pixel coordinates of the key points are as follows: ,in This is the confidence score for that point.

[0056] The raw pixel coordinates depend on the camera's mounting position and the distance between the target and the lens, which differs from the distribution of position-sensitive CSI signals. To construct view-invariant motion features, the skeletal coordinates need to be normalized with the torso as the center.

[0057] First, select the hip as the reference center point in the human body. Normalized relative coordinates The calculation is as follows:

[0058] in, and These are the width and height of the human bounding box detected in the current frame, respectively.

[0059] This step eliminates the absolute position and scale differences of the target on the image plane, ensuring that the features describe only the relative motion posture of the limbs.

[0060] Step 3: The selected CSI feature vectors and visual features are fed into independent deep neural network branches. The CSI encoding branch uses wavelet transform to convert the selected CSI feature vectors into a time-frequency graph, extracts translation-invariant features through a wavelet scattering network, and concatenates them with physical statistical features to form a multi-dimensional fused feature vector. The multi-dimensional fused feature vector is then input into a dynamic sparse mask to obtain a high-dimensional CSI semantic vector. The visual encoding branch constructs a spatiotemporal graph from the visual features, extracts spatiotemporal features through a spatiotemporal graph convolutional network, and then inputs it into a graph convolutional sparse encoder to output a visual semantic vector with the same dimension as the high-dimensional CSI semantic vector. Step 4: Cross-modal feature fusion and enhancement, inputting the high-dimensional CSI semantic vector and visual semantic vector into the gated dynamic sparse cross-attention module.

[0061] Step 5: The enhanced feature vectors from Step 4 are projected onto a shared unit hypersphere. The system calculates the CSI features. With visual features cosine similarity By using bipartite graph matching, accurate binding between radio frequency (RF) targets and visual targets can be achieved in multi-target concurrent scenarios, outputting a list of RF and visual multi-target identity bindings. .

[0062] This embodiment constructs global statistical features by extracting the amplitude and phase mean values ​​of subcarriers within a time window. Attention weights are then generated via a lightweight fully connected network to automatically enhance high signal-to-noise ratio subcarriers and suppress frequency bands severely affected by static multipath interference. This eliminates the need for manually setting fixed thresholds, avoiding the accidental deletion of effective information due to environmental changes under traditional fixed threshold strategies. Furthermore, it continuously obtains high-quality subcarrier features with stable distribution, high information density, and physical consistency, providing a low-noise, highly robust CSI input foundation for subsequent cross-modal matching. This effectively improves the feature stability and positioning reliability of the system in dynamic indoor environments.

[0063] Example 3: This embodiment provides a cross-modal identification and matching method for multiple moving targets. Based on embodiment 1 or 2, the wavelet transform in step 3 is limited.

[0064] This embodiment employs Continuous Wavelet Transform (CWT) for multi-scale time-frequency analysis. Complex Morlet wavelets are chosen as the mother wavelet because their shape is highly similar to the pulse signal reflected from a moving object, and they can effectively separate amplitude and phase information.

[0065] Define continuous wavelet transform For the filtered features With the mother wavelet after scaling and translation Inner product:

[0066] in, , For the mother wavelet function, As a scale factor, is the translation factor, and * denotes complex conjugation.

[0067] This embodiment converts a one-dimensional time-domain CSI signal into a time-frequency map using continuous wavelet transform. By leveraging the high similarity between ComplexMorlet wavelets and the reflected pulse signals of moving objects, amplitude and phase information are effectively separated and preserved, achieving accurate characterization of micro-Doppler features. The direct technical benefits are: it solves the problem that traditional time-frequency analysis methods cannot simultaneously achieve time-frequency resolution, and provides a reliable time-frequency analysis foundation for subsequent wavelet scattering networks to extract translation-invariant features and construct physically guided multi-dimensional feature vectors. It significantly improves the ability of CSI features to represent human motion patterns and enhances the robustness of cross-modal matching to different action speeds and time deviations.

[0068] Example 4: This embodiment provides a cross-modal identification and matching method for multiple moving targets. Based on any one of embodiments 1 to 3, the processing procedure of the wavelet scattering network in step 3 is limited.

[0069] To address the limitations of CNNs in directly processing time-frequency graphs, this embodiment introduces a wavelet scattering network. This network constructs a feature representation that preserves both the physical texture of the signal, such as micro-Doppler details, and mathematical stability against time shifts and deformations through cascaded nonlinear wavelet transforms and low-pass filtering operations.

[0070] To extract stable physical features from the high-frequency oscillations of micro-Doppler, wavelet scattering networks rely on an iterative "convolution-modulus operation-pooling" computation process. This embodiment specifically designs a three-layer scattering architecture to deconstruct the signal layer by layer. First, the network utilizes a low-pass filter... Features after filtering the input Direct smoothing is performed to extract the zero-order scattering average feature reflecting the DC component of the signal, i.e.:

[0071] Simple low-pass filtering inevitably results in the loss of high-frequency information; therefore, a set of multi-scale wavelet filters is introduced into the network. Further processing is then performed. By performing wavelet convolution and modulo operation on the signal, the network effectively eliminates high-frequency phase oscillations. After converting the complex signal into a real signal, a low-pass filter is used again. The first-order scattering feature is extracted, and its expression is:

[0072] This first-order feature effectively captures the main energy frequency band distribution of the signal, but it is still insufficient for describing the non-stationary changes brought about by complex human movements. Therefore, the network outputs the first-order modulus value. A deeper level of exploration was conducted by reintroducing a scale of [missing information]. Wavelet transform and modulus operation were used to obtain second-order scattering features.

[0073] It accurately extracts transient fluctuation details such as amplitude modulation and frequency modulation contained in the signal envelope. Through the above cascaded operations, the original CSI signal is successfully mapped into a set of multi-scale scattering coefficient vectors. More importantly, this feature vector is non-extensive to local deformations of the input signal, meaning the variation in feature distance is limited to the degree of deformation of the input signal itself. This implies that even if the CDV system encounters situations where different users have different action speeds or deviations in the start time of their actions during actual operation, the network can still robustly extract highly consistent physical motion patterns. This provides solid robustness and strong generalization ability for subsequent cross-modal matching algorithms in complex and variable scenarios.

[0074] Example 5: This embodiment provides a cross-modal identification and matching method for multiple moving targets, which, based on any one of the above four embodiments, limits the physical statistical features.

[0075] This embodiment focuses on three core physical quantities describing human kinematics: the Doppler peak frequency, which reflects the maximum radial velocity of limb movement. The frequency of the centroid of the spectrum reflecting the center of the velocity-weighted average of motion. And the signal envelope strength, which reflects the amplitude of the disturbance of signal energy by motion. To accurately capture the dynamic distribution characteristics of these physical quantities over the duration of the action, this embodiment calculates the mean values ​​describing the average speed and range of variation of the action. Standard deviation, a measure of the asymmetry in velocity distribution , skewness in measuring the suddenness of movement and kurtosis, a measure of the steepness of the data distribution. .

[0076] By concatenating the above statistics, a 12-dimensional physical statistical feature vector is constructed. Its mathematical expression is defined as:

[0077] Although this feature vector has low dimensionality, it contains the velocity profile of the action, energy fluctuations, and temporal evolution patterns, exhibiting strong physical interpretability. To balance the deep abstraction capability and physical fidelity of the features, we fused the manually statistical features with the deep features extracted in the previous steps in parallel.

[0078] definition To obtain the translation-invariant features output by the wavelet scattering network, an early fusion strategy is adopted, combining the two through channel-dimensional concatenation to form the final multidimensional fused feature vector:

[0079] in, This represents the eigenvector output by the wavelet scattering network.

[0080] This embodiment provides clear physical meaning support for CSI features by defining physical statistical features including Doppler peak frequency, spectral centroid frequency, and the mean, standard deviation, skewness, and kurtosis of the signal envelope intensity. These features directly correspond to core physical quantities in human kinematics, accurately characterizing limb movement speed, energy perturbation, and temporal changes in motion, effectively compensating for the lack of interpretability and overfitting to environmental noise inherent in purely data-driven features. Fusing these features with deep wavelet scattering features balances the abstract expressive power and physical fidelity of the features, improving the stability and generalization of cross-modal matching. This allows the system to reliably extract motion features even in complex multipath and multi-target interference scenarios, further enhancing the accuracy and anti-interference capability of identity matching.

[0081] Example 6: This embodiment provides a cross-modal identification and matching method for multiple moving targets, such as... Figure 1 As shown, the specific steps of this embodiment are as follows: Step 1: For the WiFi mode, the original CSI sample First, phase jumps are eliminated by time-domain phase unwrapping. Then, a sliding window is used to remove linear phase errors caused by clock drift. Finally, a dynamic subcarrier filtering module is used to remove frequency bands severely affected by static multipath interference, and the filtered feature vector is output. The detailed processing procedure is as follows: The CSI sample set containing human micro-motion features is denoted as ,in This represents the total number of valid radio frequency samples within this time window. For the... CSI samples Its essence is a complex matrix containing the time dimension and the subcarrier dimension, mathematically represented as:

[0082] In the formula, The number of effective subcarriers, For amplitude, For phase.

[0083] The original CSI phase value was truncated at Within the principal value interval, when the target motion causes the true phase change to exceed this range, the observed data will exhibit discontinuous periodic jumps, disrupting the time-domain continuity of the signal. Let... For the first Each subcarrier at time The original measured phase. To recover the true time-domain evolution of the phase, a time-domain phase unwrapping operation is first performed. Let The mathematical definition of the unwrapped phase sequence is as follows:

[0084] For discrete time series The algorithm detects the phase difference between adjacent sampling points. When the absolute value of the difference exceeds... At that time, it was assumed that phase winding had occurred. The defolded phase... for:

[0085] in, It is up to the number The cumulative count of the number of phase transitions detected up to each sampling point eliminates the artificial discontinuities introduced by the range limitation of the arctangent function, laying the foundation for subsequent linear fitting.

[0086] Phase after untangling The data still contains a significant linear trend term, mainly caused by clock asynchrony at the transmitting and receiving ends and static components such as direct sunlight paths in the environment. This large linear drift is invalid environmental noise, drowning out weak human body movement signals. To address this issue, this embodiment uses least-squares fitting within a sliding window to estimate and eliminate this linear trend. Within a length of... Within the sliding window, the calibrated phase Represented as:

[0087] in, For the fitted linear noise component, the parameters The linear rate of change of the phase corresponds to the residual carrier frequency offset; This represents the initial fixed phase offset, corresponding to the timing error caused by packet detection delay. By subtracting this linear component from the original unwrapped signal, the calibration process can effectively isolate environmental static noise and hardware clock errors, thereby recovering the originally suppressed subtle dynamic micro-Doppler changes induced by human dynamic behavior.

[0088] To avoid difficulties in neural network training convergence caused by uneven data distribution, the signals need to be standardized. First, the calibrated CSI data stream is reconstructed. For the... Each subcarrier at time Its eigenvector is composed of the original amplitude and the calibrated phase. The calibrated CSI stream is represented as:

[0089] For the amplitude time series of each subcarrier Z-score standardization is performed to eliminate the influence of dimensions and unify the data distribution. Its mathematical expression is:

[0090] in The mean amplitude, The standard deviation is denoted as .

[0091] Subsequently, the calibrated phase time series To align its numerical range with the sensitivity range of the neural network activation function, a minimum-maximum normalization strategy is employed. This linear transformation strictly maps the phase value to the [0,1] interval, eliminating the potential impact of a large phase value on the gradient.

[0092] in, and These represent the maximum and minimum phase values ​​within the current time window, respectively. After the above processing, the standardized amplitude... With normalized phase Together, they form a CSI feature sequence with a high signal-to-noise ratio and a regular distribution, providing high-quality data input for the subsequent feature extraction module.

[0093] In complex indoor multipath propagation environments, WiFi signals exhibit significant frequency-selective fading characteristics. Not all subcarriers have the same information content regarding target behavior in a specific environment. Some subcarriers may be extremely sensitive to human dynamics and contain rich Doppler features, while other subcarriers may be dominated by static object reflections and environmental noise, resulting in extremely low signal-to-noise ratios.

[0094] This embodiment employs a lightweight learnable subcarrier gating module. This module utilizes a neural network to automatically perceive the quality of the subcarriers and achieves dynamic weighting through an attention mechanism, thereby obtaining a batch of high-quality subcarriers with stable distribution, high information density, and physical consistency.

[0095] To evaluate each subcarrier in the current time window The overall quality within a given timeframe cannot be determined solely by the instantaneous value at a single moment. This embodiment extracts its global statistical characteristics over time.

[0096] Based on standardized amplitude With normalized phase First, calculate their mean over the entire time slice. For the , Subcarriers ( Construct a two-dimensional statistical feature vector representing its global characteristics. :

[0097] in, and These represent the mean amplitude and mean phase of the subcarrier within the current window, respectively. This feature vector effectively compresses temporal redundancy, retaining only the core statistics reflecting channel quality.

[0098] To capture the correlation and relative importance between different subcarriers, this embodiment uses the statistical feature vectors of all subcarriers. The features are concatenated to form a global feature vector with a dimension of 2K. :

[0099] Then, the vector The input is fed into a lightweight gating network consisting of fully connected layers and a sigmoid activation function. This network learns the contribution of different subcarriers to the final localization task and outputs a... dimensional attention weight vector .

[0100] No. Attention weights for each subcarrier The generation process and its mathematical expression can be simplified as follows:

[0101] in, and These are the learnable weights and bias parameters in the network. This is the Sigmoid activation function.

[0102] After obtaining the weight vector, the original input time-series data is dynamically reweighted by element-wise multiplication to generate the filtered feature representation. :

[0103] Through this mechanism, the network can automatically enhance the characteristic amplitude of high signal-to-noise ratio subcarriers while suppressing subcarriers that are severely affected by noise.

[0104] Step 2: For visual modalities, the system does not directly process RGB pixels, but instead uses a pose estimation algorithm to extract the coordinate sequence of human skeleton key points from video frames to construct visual samples. This process effectively removes interference from background, lighting, and clothing color, retaining only the geometric information describing human movement.

[0105] For the input visual video frame sequence, this embodiment uses the OpenPose pose estimation algorithm to extract human joints in each frame. Define the first... The set of human skeletons detected in a visual sample is For each target, extract The first key point, The original pixel coordinates of the key points are as follows: ,in This is the confidence score for that point.

[0106] The raw pixel coordinates depend on the camera's mounting position and the distance between the target and the lens, which differs from the distribution of position-sensitive CSI signals. To construct view-invariant motion features, the skeletal coordinates need to be normalized with the torso as the center.

[0107] First, select the hip as the reference center point in the human body. Normalized relative coordinates The calculation is as follows:

[0108] in, and These are the width and height of the human bounding box detected in the current frame, respectively.

[0109] This step eliminates the absolute position and scale differences of the target on the image plane, ensuring that the features describe only the relative motion posture of the limbs.

[0110] Step 3: The features filtered in Step 1 and the skeletal data extracted in Step 2 are fed into two independent deep neural network branches to extract CSI semantic vectors and visual semantic vectors, respectively.

[0111] CSI coding branch A Complex Morlet wavelet transform is employed to convert the one-dimensional time-domain signal into a time-frequency plot, and translation-invariant features are extracted using a wavelet scattering network. Subsequently, the feature vector is input to a Dynamic Sparse Masker (DSE), which suppresses environmental noise features through a learnable masking mechanism, outputting a high-dimensional CSI semantic vector. The detailed calculation process is as follows: This embodiment employs Continuous Wavelet Transform (CWT) for multi-scale time-frequency analysis. Complex Morlet wavelets are chosen as the mother wavelet because their shape is highly similar to the pulse signal reflected from a moving object, and they can effectively separate amplitude and phase information.

[0112] Define continuous wavelet transform For the filtered features With the mother wavelet after scaling and translation Inner product:

[0113] in, , For the mother wavelet function, As a scale factor, is the translation factor, and * denotes complex conjugation.

[0114] While CWT (Constant Wavelength Spectrum Map) effectively displays micro-Doppler features, directly inputting it into a CNN (Constant Wave Neural Network) can lead to overfitting in scenarios with limited data, as CNNs lack prior knowledge of the signal's physical structure and require extensive training data to learn basic feature representations. Furthermore, CNNs are sensitive to translations and minute temporal distortions of the input signal, limiting their generalization ability across different speeds or rhythms. Moreover, purely data-driven CNN features struggle to establish a clear correlation with physical motion patterns.

[0115] To address the limitations of CNNs in directly processing time-frequency graphs, this embodiment introduces a wavelet scattering network. This network constructs a feature representation that preserves both the physical texture of the signal, such as micro-Doppler details, and mathematical stability against time shifts and deformations through cascaded nonlinear wavelet transforms and low-pass filtering operations.

[0116] To extract stable physical features from the high-frequency oscillations of micro-Doppler, wavelet scattering networks rely on an iterative "convolution-modulus operation-pooling" computation process. This embodiment specifically designs a three-layer scattering architecture to deconstruct the signal layer by layer. First, the network utilizes a low-pass filter... Features after filtering the input Direct smoothing is performed to extract the zero-order scattering average feature reflecting the DC component of the signal, i.e.:

[0117] Simple low-pass filtering inevitably results in the loss of high-frequency information; therefore, a set of multi-scale wavelet filters is introduced into the network. Further processing is then performed. By performing wavelet convolution and modulo operation on the signal, the network effectively eliminates high-frequency phase oscillations. After converting the complex signal into a real signal, a low-pass filter is used again. The first-order scattering feature is extracted, and its expression is:

[0118] This first-order feature effectively captures the main energy frequency band distribution of the signal, but it is still insufficient for describing the non-stationary changes brought about by complex human movements. Therefore, the network outputs the first-order modulus value. A deeper level of exploration was conducted by reintroducing a scale of [missing information]. Wavelet transform and modulus operation were used to obtain second-order scattering features.

[0119] It accurately extracts transient fluctuation details such as amplitude modulation and frequency modulation contained in the signal envelope. Through the above cascaded operations, the original CSI signal is successfully mapped into a set of multi-scale scattering coefficient vectors. More importantly, this feature vector is non-extensive to local deformations of the input signal, meaning the variation in feature distance is limited to the degree of deformation of the input signal itself. This implies that even if the CDV system encounters situations where different users have different action speeds or deviations in the start time of their actions during actual operation, the network can still robustly extract highly consistent physical motion patterns. This provides solid robustness and strong generalization ability for subsequent cross-modal matching algorithms in complex and variable scenarios.

[0120] While wavelet scattering networks can automatically learn complex nonlinear feature representations from data, their inherent black-box nature makes the decision-making process difficult to interpret. In the field of wireless sensing, purely data-driven methods are prone to overfitting to spurious correlations specific to the environment; the model may only remember the multipath background noise of a particular room, failing to learn the true patterns of human movement. To improve the interpretability and generalization robustness of the model, this embodiment proposes a physics-guided multi-dimensional feature fusion strategy. This strategy extracts deep scattering features while simultaneously extracting a set of handcrafted statistical features with clear physical meaning from the time-spectrum graph, aiming to deeply combine the expressive power of data-driven approaches with the prior knowledge of physical models.

[0121] Specifically, the system focuses on three core physical quantities describing human kinematics: the Doppler peak frequency, which reflects the maximum radial velocity of limb movement. The frequency of the centroid of the spectrum reflecting the center of the velocity-weighted average of motion. And the signal envelope strength, which reflects the amplitude of the disturbance of signal energy by motion. To accurately capture the dynamic distribution characteristics of these physical quantities over the duration of the action, this embodiment calculates the mean values ​​describing the average speed and range of variation of the action. Standard deviation, a measure of the asymmetry in velocity distribution , skewness in measuring the suddenness of movement and kurtosis, a measure of the steepness of the data distribution. .

[0122] By concatenating the above statistics, a 12-dimensional physical statistical feature vector is constructed. Its mathematical expression is defined as:

[0123] Although this feature vector has low dimensionality, it contains the velocity profile of the action, energy fluctuations, and temporal evolution patterns, exhibiting strong physical interpretability. To balance the deep abstraction capability and physical fidelity of the features, we fused the manually statistical features with the deep features extracted in the previous steps in parallel.

[0124] definition To obtain the translation-invariant features output by the wavelet scattering network, an early fusion strategy is adopted, combining the two through channel-dimensional concatenation to form the final multidimensional fused feature vector:

[0125] To address the vulnerability of CSI signals to environmental multipath interference and high-frequency transient noise, the CSI branch employs a Dynamic Sparse Encoder (DSE) for feature extraction. The DSE filters environmental background noise through sparsification operations, retaining only Doppler frequency shift features highly correlated with subtle human movements. The network structure consists of three fully connected layers, a GELU activation function, and a Dropout layer cascaded together. Let the input CSI feature vector, after wavelet scattering and multi-dimensional feature fusion, be... The forward propagation process of DSE can be represented as:

[0126] in, and They represent the first The weight matrix and bias terms of a fully connected layer. This represents the internal activation mapping. Through this deep nonlinear mapping, the CSI signal is projected into a metric space of uniform dimension.

[0127] Visual coding branch A spatiotemporal graph is constructed from the sequence of skeletal keypoints, and the spatiotemporal graph convolutional network ST-GCN is used to capture the spatial dependencies and temporal motion patterns between joints. After processing by the graph convolutional sparse encoder GCSE, a visual semantic vector of the same dimension is output. The specific steps are as follows: The core computation of ST-GCN consists of two parts: spatial graph convolution and temporal convolution. For the first... Input features of layer networks The update rule for spatial graph convolution can be expressed as:

[0128] in: It is defined according to the spatial partitioning strategy. The adjacency matrix of each subgraph; yes The degree matrix is ​​used for normalization; It is the first Layer-learnable weight matrix; It is an activation function.

[0129] After spatial convolution, the network passes through Temporal convolutional kernels aggregate neighborhood features over time, capturing the dynamic evolution of actions. After stacking multiple layers of ST-GCN, the feature maps output by the network undergo global average pooling to generate fixed-dimensional visual feature vectors. This vector highly compresses biological features such as gait, arm swing amplitude, and movement rhythm of the human body, and has a high degree of consistency with the micro-Doppler features extracted by CSI modality in terms of physical semantics.

[0130] Unlike CSI signals, which exhibit a time-series distribution, visual human skeleton data is essentially a spatiotemporal graph topology in non-Euclidean space. Therefore, the visual branch employs a Graph Convolutional Sparse Encoder (GCSE). This encoder's network structure consists of two stacked graph convolutional networks, combined with layer normalization and Dropout mechanisms to prevent overfitting. For the initial input visual spatiotemporal graph features... Its encoding process captures spatial morphology and motion rhythm by aggregating information from adjacent joint nodes:

[0131] In the formula, An adjacency matrix representing the topology of the human skeleton. GCSE can adaptively sparsify static torso features that contribute little to identity recognition, while enhancing highly dynamic extremity features.

[0132] Step 4: After obtaining the preliminary feature representation, convert the CSI semantic vector... With visual semantic vectors Input-gated Dynamic Sparse Cross-Attention Module (GD-SCA). This module dynamically calculates the gating weights based on the confidence level of the features. and By using a cross-attention mechanism, the feature representation of one modality is enhanced by another, ultimately resulting in a fused feature vector used for measurement. and The specific implementation process is as follows: First, the GD-SCA module needs to evaluate the semantic consistency of features across different modalities across various feature dimensions. The system constructs a two-layer multilayer perceptron to process the feature vectors output by the dual-branch feature encoder. and Generate a correlation score matrix between features .

[0133]

[0134] To reduce computational complexity and eliminate redundant noise dimensions, the system selects the top-ranked relevance metrics based on the score matrix. Feature generation of binary sparse masks The features output by the dual-branch feature encoder are masked to obtain sparsed features, which serve as a candidate set for subsequent attention calculations.

[0135] In real-world sensing scenarios, modal reliability is time-varying. Through a gating weighting mechanism, the system can adaptively sense the quality of the current channel or field of view. First, it calculates... and The time-domain variance is denoted as and The variance values ​​are input into a single-layer perceptron and combined with a sigmoid activation function to obtain the modality confidence score:

[0136]

[0137] The confidence scores are normalized to obtain the final gating weights. and When the CSI signal suffers from severe multipath noise interference, the system automatically reduces the modal weights at the signal end, thereby reducing the contamination and interference of poor CSI on high-purity visual features. When there is severe occlusion or sudden change in illumination in the visual monitoring field of view, the system will reduce the weights at the visual end, enhancing the contribution of CSI radio frequency features in matching decisions.

[0138] After determining the dynamic gating weights, the module uses a cross-attention mechanism to guide information interaction between modalities. This mechanism uses one modality as the query and the other as the key and value to obtain the enhancement relationship from the signal end to the visual end and the complementary relationship from the visual end to the signal end, respectively, denoted as... and Subsequently, the system performs a residual connection between the output of the cross-attention module and the original input features to prevent gradient vanishing and feature degradation problems in deep networks, and obtains the final fused feature vector for measurement through a non-linear activation function. and Specific gating dynamic sparse cross-attention multimodal matching, such as Figure 2 As shown.

[0139] Step 5: Finally, the enhanced feature vectors are projected onto a shared unit hypersphere. The system calculates CSI features. With visual features cosine similarity By using bipartite graph matching, accurate binding between radio frequency (RF) targets and visual targets can be achieved in multi-target concurrent scenarios, outputting a list of RF and visual multi-target identity bindings. .

[0140] In this embodiment, dynamic subcarrier selection overcomes the shortcomings of traditional fixed thresholds, adaptively selecting high-quality subcarriers to suppress CSI multipath noise at the source; Complex Morlet wavelet transform accurately analyzes micro-Doppler time-frequency information; wavelet scattering network extracts multi-scale invariant features with deformation robustness, and combines them with physical statistical feature fusion to balance data expressiveness and physical interpretability; dynamic sparse mask and graph convolutional sparse encoder respectively complete dual-modal feature purification, noise reduction and dimension alignment; the core gated dynamic sparse cross-attention module avoids false bindings through sparse masking, achieves modal dynamic trade-offs by relying on adaptive gating weights, and then performs bidirectional attention interaction and residual enhancement feature representation. Each module advances layer by layer and works together to significantly improve the matching accuracy and generalization ability in complex occlusion, radio frequency interference and multi-target aliasing scenarios, and the overall anti-interference, positioning stability and identity matching accuracy are significantly improved.

[0141] Example 7: This embodiment provides a cross-modal localization target matching system, including: The WiFi mode processing module is configured to sequentially perform temporal phase unwrapping, sliding window linear phase error removal, and dynamic subcarrier filtering on the original CSI samples, and output the filtered CSI feature vector. The visual modality processing module is configured to extract the coordinate sequence of human skeleton key points from video frames using a pose estimation algorithm to construct visual features; The heterogeneous feature extraction module is configured to feed the filtered CSI feature vector and visual features into independent deep neural network branches. The CSI encoding branch uses wavelet transform to convert the filtered CSI feature vector into a time-frequency graph, extracts translation-invariant features through a wavelet scattering network, and concatenates them with physical statistical features to form a multi-dimensional fused feature vector. The multi-dimensional fused feature vector is then input into a dynamic sparse mask to obtain a high-dimensional CSI semantic vector. The visual encoding branch constructs a spatiotemporal graph from the visual features, extracts spatiotemporal features through a spatiotemporal graph convolutional network, and then inputs it into a graph convolutional sparse encoder to output a visual semantic vector with the same dimension as the high-dimensional CSI semantic vector. The cross-modal feature fusion and enhancement module is configured to input the high-dimensional CSI semantic vector and the visual semantic vector into a gated dynamic sparse cross-attention module. The processing of this module includes: calculating the correlation score matrix of the dual-modal features through a two-layer multilayer perceptron; selecting features based on the correlation score matrix to generate a binary sparse mask, and performing masking to obtain sparse features; calculating the temporal variance of the sparsed CSI features and the visual features respectively, inputting the temporal variance into a single-layer perceptron and activating it with Sigmoid to obtain a modal confidence score, normalizing the modal confidence score to obtain a gate weight; performing inter-modal feature interaction with bidirectional cross-attention, and performing a residual connection between the cross-attention output and the original sparse features and nonlinear activation to obtain the final fused feature vector used for measurement. This embodiment constructs a cross-modal localization target matching system that is fully adapted to the method. Through the division of labor and cooperation among various functional modules, the multimodal feature processing, encoding, and adaptive fusion process are fully reproduced. The system has a high degree of modularity and tight logical coupling, and can stably achieve accurate cross-modal matching of multiple targets indoors, significantly improving the overall operational reliability and scene adaptability.

[0142] To verify the technical effectiveness of the gated dynamic sparse cross-attention (GD-SCA) mechanism proposed in this invention, tests were conducted, and the test process is as follows.

[0143] Experimental scenarios such as Figure 3 As shown, a radio frequency-vision heterogeneous data acquisition platform was set up in the lobby of a college at a university. The test site area is approximately [area missing]. To comprehensively cover the test area and construct a multi-dimensional spatial perception field, this experiment employed a specific topology layout for the heterogeneous sensing devices. At the radio frequency (RF) sensing end, a star transceiver topology was adopted: four iBeacon wireless beacons were fixedly deployed at the four physical corners of the conference room as RF transmitters, continuously broadcasting 2.4GHz probe packets; simultaneously, an ESP32-S3 edge computing probe was deployed at the geometric center of the hall as the RF receiver, responsible for omnidirectional silent monitoring of the converging physical layer signals and extracting the CSI complex matrix of 64 subcarriers. At the visual sensing end, the RealVision USB4KCAM30H high-definition monocular camera was used, mounted on a fixed tripod at the edge of the venue. By adjusting the camera's pitch and yaw angles, it was ensured that its effective field of view could completely cover the central receiving node and the surrounding core activity area, and that it maintained strict timestamp alignment with the RF probe.

[0144] In this configuration, the system employs a device-free passive sensing mode for data acquisition. The test target does not need to carry any terminal; as it moves freely indoors, its torso and limbs act as dynamic scatterers, frequently cutting the cross-RF links from the four corners to the center, generating rich human micro-Doppler frequency shifts at the central ESP32-S3 node. Simultaneously, a monocular camera records the visual skeletal deformation of its movements, thus achieving homogeneous alignment of heterogeneous features from a physical perspective.

[0145] (1) Matching accuracy analysis under normal working conditions In a typical indoor environment without extreme interference, the system demonstrates superior absolute accuracy and global search ranking compared to traditional cross-modal retrieval algorithms. During testing, the system simultaneously acquired RF micro-Doppler complex sequences and monocular visual skeleton sequences in a conference room setting, excluding long-term line-of-sight occlusion. The experiment extracted the global cross-modal correlation matrix output by the system. The rank-1 accuracy and mean average precision (mAP) of the canonical correlation analysis (CCA), two-stream spatiotemporal network (Two-Stream), standard cross-attention (SCA), and the GD-SCA mechanism of this invention were calculated and recorded.

[0146] As shown in Table 1, CCA, as a traditional shallow machine learning algorithm, can only perform linear mappings and cannot effectively represent the nonlinear manifold mapping relationships between high-dimensional heterogeneous data, resulting in a Rank-1 accuracy of only 45.2%. Although the Two-Stream baseline model introduces wavelet networks and ST-GCN for deep feature extraction, improving the accuracy to 68.5%, its mAP performance suffers from a severe bottleneck due to the lack of feature interactions between modalities. The standard SCA model achieves tight coupling at the feature level through the Transformer mechanism, further increasing its accuracy to 82.1%.

[0147] Table 1 Comparison and Analysis of Matching Accuracy under Normal Operating Conditions

[0148] from Figure 4 As can be seen, the GD-SCA mechanism of this invention can utilize dynamic gating mechanisms to deeply mine the homology association between physical action rhythms and skeleton topology in a unified semantic metric space, thereby achieving a comprehensive improvement in matching accuracy. It achieves the best performance in both Rank-1 and mAP, reaching 94.3% and 91.8% respectively.

[0149] (2) Robustness analysis of the system under shading conditions To verify the system's fault tolerance and robustness in target identity binding under extreme conditions, such as visual modal failure caused by pedestrian occlusion and radio frequency modal distortion caused by furniture arrangement, an experiment was conducted by artificially introducing visual target occlusion segments from 0s to 4s and superimposing dynamic multipath interference, continuously recording the attenuation curves of the Rank-1 accuracy of the four comparison algorithms mentioned above.

[0150] Figure 5 Experimental decay curves show that when large areas of the visual feature sequence are missing due to occlusion, the matching accuracy of the conventional Two-Stream and standard SCA models drops precipitously. As shown in Table 2, the SCA accuracy drops from 82.1% to 25.5%. In contrast, the GD-SCA algorithm exhibits extremely superior robustness against interference, maintaining a usable matching accuracy of 78.6% even under extreme occlusion of 4 seconds.

[0151] Table 2 System robustness analysis under shading conditions

[0152] This is mainly due to the synergy of two physical mechanisms: one is the gating weights. It can adaptively evaluate modal confidence in real time based on the variance of the physical characteristics of the input stream. When vision encounters occlusion, the valve mechanism instantly reduces the fusion weight of the visual stream, instead relying heavily on pure radio frequency micro-Doppler features; secondly, sparse masking. The low-correlation clutter connections caused by multipath interference are precisely forced to zero at the underlying matrix operation level. This allows the GD-SCA mechanism to maintain matching accuracy in a stable high-confidence range even when the input features are severely disturbed.

[0153] (3) System reliability analysis under different target densities In real-world office or meeting scenarios, the concurrent density of indoor targets is a crucial indicator for evaluating the performance of localization matching algorithms. As the number of pedestrians increases, the multipath effect of the radio frequency channel intensifies exponentially, leading to severe aliasing of micro-Doppler features. Simultaneously, frequent overlap between visual targets significantly reduces the confidence level of skeleton extraction. This study evaluates the impact of different target densities on the Rank-1 accuracy of various matching algorithms to verify the reliability of the GD-SCA algorithm in densely populated environments.

[0154] The experimental data are shown in Table 3. The matching accuracy of all four algorithms decreased to varying degrees as the number of targets increased. When the number of participants increased from 1 to 5, the Rank-1 accuracy of both the CCA and Two-Stream algorithms fluctuated drastically. In a 5-person concurrent scenario, due to the inability to effectively isolate the features of a single target from complex mixed signals, the matching accuracy dropped below 40%, essentially losing its ability to bind multiple targets. The standard SCA algorithm performs well with fewer targets, but as target density increases, its attention score matrix is ​​affected by false alarm features, and its accuracy declines significantly faster than the algorithm of this invention. Figure 6 As shown. The GD-SCA algorithm in Even under extreme stress testing with concurrent users, it maintained a Rank-1 matching accuracy of 81.2%. This is mainly due to the dynamic sparse mask. The design forces the filtering out of low-confidence similarity scores generated by multipath aliasing, enabling the model to accurately locate the radio frequency and visual mappings belonging to the same target from the "messy" feature distribution.

[0155] Table 3 System reliability analysis under different target densities

[0156] (4) Analysis of incremental ablation experiments of key modules To quantitatively evaluate the incremental contribution of each core component in the positioning system proposed in this invention to the final matching performance, the following five progressive comparative models were constructed: Comparison Model 1: It uses basic FFT radio frequency features and 2D-CNN visual features, and does not include any cross-modal attention interactions; Comparison Model 2: Wavelet scattering network and spatiotemporal graph convolutional network are added as the backbone of feature extraction.

[0157] Comparison Model 3: Based on Comparison Model 2, a standard cross-attention fusion mechanism is added.

[0158] Comparison Model 4: Introducing the innovative gating weight allocation module of this paper into the attention mechanism.

[0159] Comparison with Model 5: Adding a dynamic sparse mask to Model 4 This constitutes a complete gated dynamic sparse cross-attention network.

[0160] As shown in Table 4, the introduction of each module significantly improved the system's positioning accuracy and matching stability. Quantitatively, Comparison Model 2 provided the most substantial leap in original accuracy compared to the basic Comparison Model 1, with the Rank-1 index jumping from 54.2% to 72.5%, a year-on-year increase of 18.3%. This validates the correctness of the spatiotemporal feature modeling path chosen in this paper. With the successive addition of the SCA fusion module in Comparison Model 3 and the gating mechanism module in Comparison Model 4, the system maintained a high hit rate while the mAP index, representing retrieval quality, steadily climbed to 85.1%, narrowing the semantic gap between modalities. Ultimately, the GD-SCA system of this invention achieved peak values ​​in both metrics, demonstrating the strong complementarity of each sub-module in terms of functional implementation.

[0161] Table 4. Experimental Analysis of Incremental Ablation of Key Modules

[0162] Figure 7 This paper presents the Rank-1 and mAP comparison results of different feature fusion and matching models in cross-modal target matching tasks. The evolution trend of the experimental data clearly shows that with the gradual introduction of mechanisms such as dynamic sparse screening, gating weight allocation, and cross-attention interaction, the model performance continuously improves. Specifically, Model 1, which directly matches basic FFT radio frequency features with CNN visual features, exhibits the worst performance due to the significant semantic gap caused by heterogeneous features. Model 2, after introducing ST-GCN to encode visual skeleton features, improves the metrics to 72.5% and 65.2%, respectively, verifying that the spatiotemporal topological features of the visual skeleton can effectively capture the biological characteristics of human motion, thereby enhancing the robustness of cross-modal matching. Building on this, Model 3 adds a standard cross-attention mechanism to achieve cross-modal feature interaction, further improving Rank-1 and mAP, demonstrating the positive effect of deep cross-modal feature interaction on eliminating the semantic gap. Model 4, after adding a gating weight allocation module, achieves metrics of 89.4% and 85.1%, proving that adaptive modal quality assessment and weight allocation can effectively suppress interference from invalid modalities.

[0163] The GD-SCA model proposed in this invention combines dynamic sparse dimension filtering and gated dynamic sparse cross-attention mechanism, ultimately achieving a Rank-1 matching accuracy of 94.3% and an mAP of 91.8%, which are 40.1 and 45 percentage points higher than the basic model, respectively, and 4.9 and 6.7 percentage points higher than the comparison model 4, respectively. This fully verifies the effectiveness and superiority of the mechanism proposed in this invention in enhancing cross-modal feature discriminability and improving matching accuracy.

[0164] (5) Comparative experiment of cross-modal matching algorithms To objectively evaluate the overall performance of the proposed GD-SCA algorithm, experiments were conducted to compare it with the CA-Trans algorithm based on cascaded attention cross-modal feature alignment, the IR-ViT algorithm based on dynamic uncertainty estimation, and the Swin-Cross cross-modal fusion algorithm. To ensure fairness in the comparison, the inputs of all algorithms were uniformly replaced with the wavelet scattering network and ST-GCN constructed in this paper to extract CSI and visual features, thus ensuring that all algorithms extract high-dimensional semantic features of the same quality. To conduct cross-modal interactions.

[0165] CA-Trans is derived from the paper “ZHAO Z, HU HM, ZHANG H, et al. Improving color constancy using chromaticity-line prior[J]. IEEE Transactions on Multimedia, 2022, 25: 3642-3656.”.

[0166] IR-ViT originates from the paper "ŽUKAIT D. Teobromino ir vitamino D poveikis ūmiaivirus sukeltai žarnyno pažaidai[D]. Lithuanian University of Health Sciences(Lithuania), 2025,".

[0167] Swin-Cross comes from the paper "HU

[0168] As can be seen from the overall statistics in Table 5, the proposed GD-SCA algorithm demonstrates significant performance advantages in cross-modal matching tasks. In the two core metrics of Rank-1 and mAP, GD-SCA achieved 94.3% and 91.8% respectively, leading the Swin-Cross model by 4.1% and 5.4% accuracy. The CA-Trans and IR-ViT algorithms, with Rank-1 accuracy of 85.4% and 88.6% respectively, are inferior to GD-SCA because they employ a globally dense attention matrix, attempting to find cross-modal correlations across all feature dimensions. However, in indoor environments, CSI signals are accompanied by extremely severe wall reflections and multipath clutter. The globally dense attention mechanism is prone to falsely binding meaningless pseudo-Doppler clutter with visual features, leading to mismatches. The core limitation of the Swin-Cross model lies in its moving window mechanism being limited by the local receptive field, making it better suited for handling instantaneous local feature intersections. In the physical scenario of this experiment, human gait and limb movements manifest as a long-range, periodic micro-Doppler frequency shift rhythm at the radio frequency end.

[0169] Table 5 Comparison of Cross-Modal Matching Algorithms

[0170] The comparative experiments above confirm that, simultaneously by Figure 8 It can be seen that the GD-SCA algorithm proposed in this invention deeply integrates the physical characteristics and temporal patterns of indoor CSI, and has advantages in the specific and highly challenging cross-modal positioning task of radio frequency-vision, improving accuracy and robustness.

[0171] Some steps in the embodiments of the present invention can be implemented using software, and the corresponding software program can be stored in a readable storage medium, such as an optical disc or a hard disk.

[0172] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for cross-modal recognition matching between multiple moving targets, characterized in that, The method comprises: Step 1: WiFi modal processing, sequentially performing time domain phase unwrapping, sliding window linear phase error removal and dynamic subcarrier screening on original CSI samples, and outputting screened CSI feature vectors; Step 2: visual modal processing, using a pose estimation algorithm to extract a human body skeleton key point coordinate sequence from a video frame, and constructing a visual feature; Step 3: the screened CSI feature vectors and the visual feature are respectively sent into independent deep neural network branches, wherein a CSI coding branch adopts wavelet transform to convert the screened CSI feature vectors into time-frequency graphs, and extracts shift-invariant features through a wavelet scattering network, and after channel splicing with physical statistical features, a multi-dimensional fusion feature vector is formed, the multi-dimensional fusion feature vector is input into a dynamic sparse masker to obtain a high-dimensional CSI semantic vector; a visual coding branch constructs the visual feature into a space-time graph, extracts space-time features through a space-time graph convolution network, and then inputs the space-time features into a graph convolution sparse encoder to output a visual semantic vector with the same dimension as the high-dimensional CSI semantic vector; Step 4: cross-modal feature fusion and enhancement, the high-dimensional CSI semantic vector and the visual semantic vector are input into a gated dynamic sparse cross-attention module, and the processing process of the module comprises: Step 41: calculating a correlation score matrix of the double-modal features through two layers of multilayer perceptrons; Step 42: selecting a feature to generate a binary sparse mask according to the correlation score matrix, and obtaining a sparse feature through mask processing; Step 43: respectively calculating time domain variances of the sparse CSI feature and the visual feature, inputting the time domain variances into a single-layer perceptron and obtaining modal confidence scores through Sigmoid activation, and normalizing the modal confidence scores to obtain gating weights; Step 44: performing inter-modal feature interaction through bidirectional cross-attention, connecting the cross-attention output and the original sparse feature through a residual connection and performing nonlinear activation to obtain a final fusion feature vector used for measurement; Step 5: projecting the fusion feature vector onto a shared unit hypersphere, calculating the cosine similarity of the enhanced CSI feature and the visual feature, and outputting a matching result of a target cross-modal identity through bipartite graph matching.

2. The method of claim 1, wherein, The process of dynamic subcarrier screening in step 1 comprises: Step 11: based on the normalized amplitude and the normalized phase, calculating the amplitude mean value and the phase mean value of each subcarrier in the current time window, and constructing a two-dimensional statistical feature vector representing the global characteristics of the subcarrier; Step 12: splicing the statistical feature vectors of all subcarriers to form a global feature vector; Step 13: inputting the global feature vector into a lightweight gating network composed of a fully connected layer and a Sigmoid activation function to learn the attention weights of each subcarrier; Step 14: multiplying the original input time series data element by element using the attention weights to realize dynamic reweighting and generate a screened feature representation.

3. The method of claim 1, wherein, In step 3, a Complex Morlet wavelet is used as a mother wavelet to perform continuous wavelet transform CWT on the screened CSI feature vectors, and the calculation formula is: wherein, is the feature vector after screening, , is the mother wavelet function, is the scale factor, is the translation factor, and * denotes complex conjugate.

4. The method of claim 1, wherein, The processing process of the wavelet scattering network comprises: Utilizing low pass filters The filtered CSI feature vector Direct smoothing is performed to extract zeroth order scattering average features: By multi-scale wavelet filter On the filtered CSI feature vector Perform wavelet convolution and modulus operation, and then pass through a low-pass filter Extract the first-order scattering features: Magnitude output of first order scattering features Further performing wavelet transform with scale and modulus operation, second order scattering features are obtained: The zero-order, first-order, and second-order scattering features are combined into multi-scale shift-invariant scattering features.

5. The method of claim 1, wherein, The physical statistical features include a Doppler peak frequency, a spectral centroid frequency, a mean, a standard deviation, a skewness, and a kurtosis of a signal envelope intensity.

6. The method of claim 1, wherein, A forward propagation process of the dynamic sparse masker is represented as: wherein, and respectively represent the weight matrix and bias term of the fully connected layer, represent the internal activation mapping, represent the activation function, and the Dropout layer is used to realize feature sparsification and noise suppression, represent the multi-dimensional fusion feature vector.​ 7. The method of claim 1, wherein, A forward propagation process of the graph convolution sparse encoder is represented as: In the formula, represents the spatio-temporal features output by the spatio-temporal graph convolution network, represents an adjacency matrix of the human body skeleton topology, , is a two-layer graph convolution layer.

8. A multi-mobile target cross-modal recognition matching system, characterized in that, The system comprises: The WiFi modal processing module is configured to sequentially perform time-domain phase unwrapping, sliding window linear phase error removal, and dynamic subcarrier screening on the original CSI sample, and output a screened CSI feature vector; The visual modal processing module is configured to extract a human body skeleton key point coordinate sequence from a video frame by using a pose estimation algorithm, and construct a visual feature; The heterogeneous feature extraction module is configured to input the screened CSI feature vector and the visual feature into independent deep neural network branches respectively, wherein a CSI encoding branch converts the screened CSI feature vector into a time-frequency graph by using wavelet transform, extracts shift-invariant features by using a wavelet scattering network, and forms a multi-dimensional fusion feature vector after channel splicing with physical statistical features, inputs the multi-dimensional fusion feature vector into a dynamic sparse masker, and obtains a high-dimensional CSI semantic vector; a visual encoding branch constructs the visual feature into a space-time graph, extracts space-time features by using a space-time graph convolution network, and inputs the space-time features into a graph convolution sparse encoder, and outputs a visual semantic vector with the same dimension as the high-dimensional CSI semantic vector; The cross-modal feature fusion and enhancement module is configured to input the high-dimensional CSI semantic vector and the visual semantic vector into a gated dynamic sparse cross-attention module, and a processing process of the module includes: calculating a correlation score matrix of the double-modal features by using two layers of multilayer perceptrons; selecting a feature to generate a binary sparse mask according to the correlation score matrix, performing mask processing to obtain a sparse feature; calculating time domain variances of the CSI feature and the visual feature after being sparse respectively, inputting the time domain variances into a single-layer perceptron and obtaining a modal confidence score by using a Sigmoid activation, normalizing the modal confidence score to obtain a gating weight; performing inter-modal feature interaction by using bidirectional cross-attention, connecting a cross-attention output and an original sparse feature by using a residual connection, and obtaining a final fusion feature vector used for measurement by using a nonlinear activation.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the multi-mobile target cross-modal recognition matching of any one of claims 1 to 7.

10. A computer readable storage medium having stored thereon a computer program, characterized in that: The computer program is executed by the processor to implement the steps of the multi-mobile target cross-modal recognition matching of any one of claims 1 to 7.

Images