A high-robust visual wireless dynamic fusion method based on uncertainty modeling

Abstract

This invention discloses a robust visual-wireless dynamic fusion method based on uncertainty modeling, comprising: acquiring a video frame sequence of target human motion, acquiring skeleton joint point data and radar data; processing the skeleton joint point data to generate skeleton sequence data; processing the wireless data to generate a time-velocity micro-Doppler image; extracting features from the skeleton sequence to obtain visual modal features; extracting features from the micro-Doppler image to obtain radar modal features; performing intermodal feature interaction between the visual modal features and the radar modal features, using Dirichlet distribution to model class probabilities, thereby utilizing uncertainty to achieve intramodal feature selection and intermodal feature fusion. This invention employs channel feature interaction and dynamic fusion strategies, effectively utilizing complementary information between modalities to improve fusion effect and system robustness, and solves the problems of effective complementarity of visual and wireless modal features and overall performance degradation caused by modal absence.

Description

Technical Field

[0001] This invention relates to the field of action recognition and multimodal fusion technology, and more specifically, to a highly robust visual wireless dynamic fusion method based on uncertainty modeling. Background Technology

[0002] Currently, visual-wireless modal fusion technology mainly utilizes the complementary characteristics of two modalities to improve recognition performance. Visual modalities have high spatial resolution, while radar modalities are more robust to the environment; therefore, the advantages of each modality can be leveraged to enhance overall discrimination capability. In the field of action recognition using visual and wireless multimodal fusion, most methods rely on the complementarity between the two modalities to achieve more efficient discrimination. This complementarity refers to the fact that vision can effectively recognize planar actions, such as waving or drawing an "X," but is insensitive to radial actions, such as pushing forward or knocking. Radar, on the other hand, is more sensitive to radial actions but less sensitive to horizontal actions. Utilizing this complementarity can effectively improve the accuracy of overall action recognition, but it overlooks the fact that when a single modality fails, the modal fusion process transmits noise instead of complementary information. Existing methods cannot solve the interference problem caused by noise, thus significantly reducing the effectiveness of modal fusion.

[0003] In summary, during the fusion of two modalities, if a single modality fails (such as when vision is affected by lighting), the modal fusion effect will be affected by noise interference from the missing modality, resulting in a significant decrease in overall performance. Summary of the Invention

[0004] To address the performance degradation issues in multimodal fusion caused by effective complementarity and modality loss, this invention provides a robust visual wireless dynamic fusion method based on uncertainty modeling. This invention primarily utilizes the complementarity between modalities and the uncertainty of modal features to achieve effective modal complementarity and dynamic modal fusion. This not only improves the accuracy of action recognition but also effectively enhances the accuracy of recognition when a single modality is missing.

[0005] The technical means employed in this invention are as follows:

[0006] A robust visual wireless dynamic fusion method based on uncertainty modeling includes the following steps:

[0007] S1. Obtain the target human body motion video frame sequence, and use the Openpose algorithm to process the target human body motion video sequence into skeleton joint point data; acquire radar data of the target human body motion through radar acquisition, wherein the radar data is in the form of a cubic matrix, and the matrix includes a fast time dimension, a slow time dimension and the number of antennas.

[0008] S2. Process the skeleton joint data to generate skeleton sequence data; perform signal processing on the radar data to extract micro-Doppler information of human movement and generate time-velocity micro-Doppler images.

[0009] S3. Perform feature extraction on the skeleton sequence to obtain visual modal features; perform feature extraction on the micro-Doppler image to obtain radar modal features;

[0010] S4. Perform inter-modal feature interaction on the visual modal features and the radar modal features, thereby utilizing uncertainty to achieve intra-modal feature selection and inter-modal feature fusion. The inter-modal feature interaction on the visual modal features and the radar modal features includes:

[0011] Visual and radar modal features are extracted and passed through intermediate layer features of the network layers. The intermediate layer features are then subjected to global and average pooling along the channel dimension. The intermediate layer features of the visual modal features are also subjected to global and average pooling to obtain two channel feature vectors. The four channel feature vectors are concatenated and passed through an MLP layer to obtain two sets of interaction weight vectors. These weight vectors are multiplied by the intermediate layer features of the visual and radar modal features, respectively, and then added to the modal features of the original branch before being fed into the next layer of the network. The modal features of the original branch refer to the intermediate layer features that have not undergone global and average pooling. Finally, the interaction features of the two modal branches are obtained.

[0012] Furthermore, uncertainty is utilized to achieve intra-modal feature selection and inter-modal feature fusion, including:

[0013] The class probability is modeled using the Dirichlet distribution. From the original branch modal features and interaction features, effective modal features are selected for single-modal action recognition based on the uncertainty of the features, and participate in the final fusion decision. The fusion decision is set to include: sequence-to-sequence modal fusion and dynamic inter-modal fusion of the decision layer.

[0014] The sequence-to-sequence modal fusion is set as follows: based on the effective features obtained by intra-modal feature selection, the information of the horizontal and vertical dimensions is unfolded to obtain the sequence, and then the two modalities perform sequence-to-sequence cross-attention calculation to obtain the fused features;

[0015] The intermodal dynamic fusion is set as follows: effective features obtained based on intramodal feature selection and fusion features obtained from sequence-to-sequence modal fusion. Dirichlet distribution is also used to model class probabilities. By utilizing uncertainty, the most effective modality is selected from the three views to obtain the final dynamic fusion result. The three views include visual modal features, radar modal features, and fusion features obtained after fusing visual features and radar features.

[0016] Further, the skeleton joint data is processed to generate skeleton sequence data, including: for each frame, extracting the X and Y coordinates and confidence scores of each joint of the human body, and superimposing the data of 300 frames to obtain skeleton sequence data.

[0017] Furthermore, signal processing is performed on the wireless data to extract micro-Doppler information of human movement and generate time-velocity micro-Doppler maps, including:

[0018] Based on the aforementioned wireless data acquisition of the time difference and frequency information of the transmitted signal and the echo signal, the distance information and velocity information of each part of the target human body are calculated according to the time difference and frequency information, thereby obtaining the time-velocity micro-Doppler image.

[0019] Furthermore, visual modal features are extracted from the skeleton sequence, including: processing the human skeleton sequence based on a spatiotemporal graph convolutional network to obtain visual modal features, wherein the spatiotemporal graph convolutional network includes eleven stacked spatiotemporal convolutional layers.

[0020] Further, feature extraction is performed on the micro-Doppler image to obtain radar modal features, including: processing the micro-Doppler image based on a residual network to obtain radar modal features, wherein the residual network includes eight stacked residual blocks.

[0021] Compared with the prior art, the present invention has the following advantages:

[0022] This invention dynamically fuses features from two modalities. When a single modality provides noise, uncertainty is utilized to reduce the weight of that branch, thereby increasing the feature weight of the effective modality. Furthermore, channel interaction is used to achieve modal complementarity, fully leveraging the complementarity of the two modalities. When both modalities are functioning correctly, it provides higher accuracy estimation. And when a single modality fails, an intra-modal feature selection algorithm selects more effective features from the interactive features and the original features. This invention not only achieves the complementarity of previous methods but also enables effective complementarity. While realizing modal fusion, it further improves the fusion effect.

[0023] This invention addresses the problem of overall model performance degradation when a single mode fails by dynamically selecting effective modal features by utilizing the uncertainty of modality; furthermore, it achieves effective modal complementarity by performing feature interaction in the channel dimension and utilizing intramodal feature selection. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 The network architecture diagram for implementing the highly robust visual wireless dynamic fusion method based on uncertainty modeling of the present invention is shown in the figure.

[0026] Figure 2 This is a block diagram illustrating intramodal feature selection and intermodal dynamic fusion using uncertainty in an embodiment of the present invention.

[0027] Figure 3 This is a flowchart of sequence-to-sequence feature fusion in an embodiment of the present invention. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0029] like Figure 1 As shown, this invention provides a highly robust visual wireless dynamic fusion method based on uncertainty modeling, comprising:

[0030] S1. Obtain the target human body motion video frame sequence, and use the Openpose algorithm to process the target human body motion video sequence into skeleton joint point data; acquire radar data of the target human body motion through radar acquisition, wherein the radar data is in the form of a cubic matrix, and the matrix includes a fast time dimension, a slow time dimension and the number of antennas.

[0031] In this invention, the Openpose algorithm is used to directly estimate the coordinates of skeleton joints and the confidence level of each joint from a video frame sequence. The echo signal acquired by radar is combined with the transmitted signal and processed into an intermediate frequency (IF) signal. Then, time-frequency analysis is performed on the IF signal to obtain a radar data cube matrix representing the target human motion. This radar data cube matrix represents the fast time dimension, the slow time dimension, and the number of antennas.

[0032] S2. Process the skeletal joint data, extract 18 joints from each frame of the human body, and assemble them into skeletal sequence data. Perform signal processing on the wireless data, using MATLAB to process the wireless signal data, extract the micro-Doppler information of human movement, and create a time-velocity micro-Doppler map.

[0033] Specifically, for each frame, the X and Y coordinates and confidence levels of each joint point on the human body are extracted, and the data from 300 frames are superimposed to obtain skeleton sequence data. For the aforementioned wireless data, the distance information of each part of the target human body is calculated using the time difference and frequency information of the transmitted and echo signals, and the velocity information of each part is further calculated, resulting in a time-velocity micro-Doppler image.

[0034] S3. Perform feature extraction on the skeleton sequence to obtain visual modal features; perform feature extraction on the micro-Doppler image to obtain radar modal features.

[0035] In this invention, the overall network utilizes the idea of ​​deep learning and is mainly divided into two parts: one is to extract visual features from the skeleton sequence of the visual modality, and the other is to extract radar features from the micro-Doppler image of the radar modality.

[0036] Specifically, the input to the visual branch features is a skeleton sequence (N, 3, 300, 18, 2), where N represents the batch size, 3 refers to the x and y coordinates and confidence score of each skeleton point, 300 is the number of frames, 18 represents the number of skeleton points, and 2 represents the two people with the highest confidence scores detected. This sequence passes through eleven layers of spatiotemporal graph convolutional networks to obtain the final visual modality output, called the original visual features. The first four layers are called the visual model Block 1. The spatiotemporal convolutional layers have 64 channels, the spatial graph convolution kernel size is 3, used to extract the spatial features of the human skeleton points, and the temporal graph convolution kernel size is 9, used to extract temporal information between sequences. The activation function is ReLU. The input first passes through the spatial graph convolution, then through the temporal graph convolution, and finally uses a residual structure for output. In the first layer, the number of channels increases from 3 to 64. The number of channels remains unchanged in the following three layers. The output is fed into the next layer and interacts with the radar modality features at the channel level. This interaction is then multiplied by weights and added to the original branch features before being fed into the next layer. This interaction feature is called the interactive feature. Layers five, six, and seven are called the visual model Block 2. The spatiotemporal convolutional layer has 128 channels, the spatial graph convolution kernel size is 3, and the temporal graph convolution kernel size is 9. The activation function is ReLU. The input passes through the fifth layer, where the number of channels increases from 64 to 128. The number of channels remains unchanged in the following two layers. The output is fed into the next layer and interacts with the radar modality features at the channel level. This interaction is then multiplied by weights and added to the original branch features before being fed into the next layer. This interaction feature is called the interactive feature. Layers 8, 9, and 10 are called the visual model Block 3. The spatiotemporal convolutional layers have 256 channels, the spatial graph convolutions have a kernel size of 3, and the temporal graph convolutions have a kernel size of 9. The activation function is ReLU. The input passes through layer 8, where the number of channels increases from 128 to 256. The number of channels remains unchanged in the following two layers. The output is fed into the next layer and interacts with the radar modality features in a channel dimension. This interaction feature is then multiplied by weights and added to the original branch features before being fed into the next layer. The eleventh layer is called the visual model Block 4. The spatial graph convolutions have a kernel size of 3, the temporal graph convolutions have a kernel size of 9, and the activation function is ReLU. The number of channels increases from 256 to 512, resulting in two outputs: the original features and the interaction features. These two types of features are then processed through average pooling, flattening, and a multilayer perceptron to obtain ten-class features (without using softmax).

[0037] The input to the radar branch is (N, 3, 128, 128), where N is the batch size, 3 represents the RGB three channels, and 128*128 represents the image's width and height. It sequentially passes through 8 residual blocks to obtain the final radar modality output, called the original radar feature. The input micro-Doppler image passes through a convolutional layer with a kernel size of 3 and a ReLU activation function, increasing the number of channels from 3 to 64. It is then input into the residual blocks. The first and second residual blocks are called the radar model Block1, which contains two convolutional layers with a kernel size of 3, undergoes batch normalization, uses the ReLU activation function, and passes through a residual structure to obtain the feature output, increasing the number of channels from 64 to 64. The output is fed into the next network layer and also interacts with the visual modality features in the channel dimension. This interaction feature is then multiplied by weights and added to the original branch features before being input into the next network layer. The third and fourth residual blocks are called Radar Model Block 2. They contain two convolutional layers with a kernel size of 3, batch normalized, and using ReLU activation. The residual structure then produces feature outputs, increasing the number of channels from 64 to 128. The output is fed into the next network layer and also interacts with the visual modality features at the channel level. This interaction is multiplied by weights and added to the original branch features before being fed into the next network layer. These interactive features are called the interaction features. The fifth and sixth residual blocks are called Radar Model Block 3. They also contain two convolutional layers with a kernel size of 3, batch normalized, and using ReLU activation. The residual structure then produces feature outputs, increasing the number of channels from 128 to 256. The output is fed into the next network layer and also interacts with the visual modality features at the channel level. This interaction is multiplied by weights and added to the original branch features before being fed into the next network layer. These interactive features are called the interaction features. The seventh and eighth residual blocks are called the radar model Block 4. They contain two convolutional layers with a kernel size of 3, batch normalized, and using ReLU as the activation function. The residual structure then produces feature outputs, increasing the number of channels from 256 to 512. Two outputs are obtained: the original features and the interactive features. These two features are then processed through average pooling, flattening, and a multilayer perceptron to obtain ten-class features (without using softmax).

[0038] S4. Perform inter-modal feature interaction, intra-modal feature selection, and inter-modal feature fusion on the visual modal features and the radar modal features. This mainly includes: modeling class probabilities using the Dirichlet distribution, from the original features ( Figure 1 (dashed feature flow in the middle) and interactive features ( Figure 2In the solid line feature flow, based on the uncertainty of the features, effective modal features are selected for single-modal action recognition and participate in the final fusion decision. The fusion decision is set to include: sequence-to-sequence modal fusion and dynamic inter-modal fusion of the decision layer.

[0039] The sequence-to-sequence modal fusion is set as follows: based on the effective features obtained by intramodal feature selection, the information of the horizontal dimension is unfolded to obtain the sequence, and then the two modalities perform sequence-to-sequence cross-attention calculation to obtain the fused features.

[0040] The intermodal dynamic fusion is set as follows: effective features obtained by intramodal feature selection and fusion features obtained by sequence-to-sequence modal fusion are used. Dirichlet distribution is also used to model class probabilities. By utilizing uncertainty, the most effective modality is selected from three views to obtain the final dynamic fusion result. The three views include visual modal features, radar modal features, and fusion features obtained after fusing visual features and radar features.

[0041] Specifically, intermodal feature interaction refers to the interaction of channel-dimensional features between the output feature maps of two modalities in Block1, Block2, Block3, and Block4, respectively, to obtain the interaction features of each branch. m∈{V,R}, where V represents the visual modality and R represents the radar modality. The feature maps of the two modalities are averaged and globally pooled along the channel dimension. The intermediate layer features of the visual modality are obtained by global and average pooling to get two channel feature vectors. The intermediate layer features of the visual modality are also obtained by global and average pooling to get two channel feature vectors. The four channel feature vectors are concatenated and passed through an MLP layer to get two sets of interaction weight vectors. These weight vectors are multiplied by the intermediate layer features of the visual modality and the radar modality, respectively, and then added to the modal features of the original branch before being fed into the next layer of the network. The modal features of the original branch refer to the intermediate layer features that have not undergone global and average pooling. Finally, the interaction features of the two modal branches are obtained.

[0042]

[0043] Where i = (2, 3, 4), F is the result obtained from the modal's raw input through the first block. block-i This refers to the function expression of the i-th block, the final... After passing through the MLP, ten categories of output are obtained.

[0044] The intra-modal feature selection described above primarily utilizes the uncertainty inherent in evidence-based deep learning theory. Theoretically, the ten-class results {μ1,μ2,...,μ} output after a linear layer are used to determine the optimal class. KUsing the ReLU activation function to set negative values ​​to zero, the resulting feature vector is {e1, e2, ..., e}. K (K represents the number of classification categories, which is 10 in this embodiment) can represent the reliability of each category and the uncertainty of this set of features, as shown in the formula:

[0045]

[0046] in This is called the Dirichlet strength, b k This is called credibility, and it's also the normalized feature. u represents the uncertain feature, which effectively represents the uncertainty of this set of features. If a single modality fails, its uncertainty is often high, thus helping the model select features from another modality, thereby achieving dynamic fusion. The value of the uncertain feature is between [0,1]. The closer it is to 1, the less credible the modality is, i.e., the more failed it is; the closer it is to 0, the more credible the modality is. The formula for calculating the normalized feature (credibility) after dynamic fusion is:

[0047]

[0048] in, It is a measure of the amount of conflict between two feature quality sets, a scaling factor. Used for normalization. Dirichlet distribution parameters {α1, α2, ..., α} K} from {e1,e2,...,e K} + 1 equals . For example Figure 1 As shown, This refers to the result obtained by calculation from the dashed lines in the system block diagram, which refers to the original feature result obtained after feature extraction from the input. m This refers to the uncertainty of the original feature before the dashed line to the IFS module. This refers to the result obtained through calculation from the solid lines in the system block diagram, and the interactive feature result obtained after feature extraction from the input. m This refers to the uncertainty of the interaction feature between the solid line and the IFS module.

[0049] The optimized EDL loss function is:

[0050]

[0051] Where, q θ m(μm|x m ) refers to the Dirichlet distribution that μm follows, x m {m=1,2} refers to the mode.

[0052] This application uses the Dirichlet distribution to model class probabilities and leverages uncertainty for dynamic fusion, aiming to quantify the uncertainty of the output results. According to DS evidence theory, uncertainty and confidence parameters actually model "probabilities of probabilities," and the Dirichlet distribution can model these "probabilities of probabilities." Therefore, this application applies the formula framework of the Dirichlet distribution to quantify the required uncertainty. By using the Dirichlet distribution to model the probabilities of probabilities, the original probability values ​​are transformed into a distribution, thus requiring optimization of the standard cross-entropy loss.

[0053] The standard cross-entropy loss is:

[0054] logp(y|μ m )

[0055] Since μ has a specific value in the standard cross-entropy loss calculation, while μ is a distribution in this application, constrained mode fusion can be performed by minimizing this expectation, i.e., the method in this application:

[0056]

[0057] At this time μ m The distribution of is the Dirichlet distribution.

[0058] The intermodal feature fusion includes sequence-to-sequence modal fusion and dynamic intermodal fusion at the decision layer. Sequence-to-sequence modal fusion involves flattening the spatial-dimensional information based on effective features obtained from intramodal feature selection to obtain a sequence, where the feature dimension is X∈R. N×C Then, sequence-to-sequence cross-attention is performed between the two modalities to obtain the fused features. The specific formula is pseudo:

[0059]

[0060]

[0061] In this formula, subscripts v and r represent visual and radar respectively, K and V are the weight vectors obtained by linearly processing features X, and G = KV represents the global context vector. The superscript inter indicates intra-modal, and the final result U represents the attention result. This is a reconstruction of the original attention formula.

[0062] The two obtained features are subjected to residual connection, concatenation, average pooling, and linear layer to obtain fused features.

[0063] The aforementioned intermodal dynamic fusion is based on effective features obtained from intramodal feature selection and fusion features obtained from sequence-to-sequence modal fusion. Through the above dynamic fusion algorithm, the most effective modal features are selected from the three views to obtain the final dynamic fusion result. The three views include visual modal features, radar modal features, and fusion features obtained after fusing visual and radar features.

[0064] S5. Perform end-to-end training on the overall framework, using cross-entropy loss and EDL evidence loss for optimized training.

[0065] Cross-entropy loss is used to constrain the output classification results of each branch modality, as well as the fusion result. EDL evidence loss further optimizes the model by utilizing uncertainty, resulting in a highly robust network that maintains overall recognition accuracy even with missing modalities. It also effectively complements modalities, preventing noise from being transmitted due to missing modalities from interfering with the final fusion result.

[0066] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A robust visual wireless dynamic fusion method based on uncertainty modeling, characterized in that, Includes the following steps: S1. Obtain the target human body motion video frame sequence, and use the Openpose algorithm to process the target human body motion video sequence into skeleton joint point data; acquire radar data of the target human body motion through radar acquisition, wherein the radar data is in the form of a cubic matrix, and the matrix includes a fast time dimension, a slow time dimension and the number of antennas. S2. Process the skeleton joint data to generate skeleton sequence data; perform signal processing on the radar data to extract micro-Doppler information of human movement and generate time-velocity micro-Doppler images. S3. Perform feature extraction on the skeleton sequence to obtain visual modal features; Radar mode features are obtained by performing feature extraction on the micro-Doppler image; S4. Perform inter-modal feature interaction on the visual modal features and the radar modal features, thereby utilizing uncertainty to achieve intra-modal feature selection and inter-modal feature fusion. The inter-modal feature interaction on the visual modal features and the radar modal features includes: Visual modal features and radar modal features are extracted and passed through intermediate layer features of the network layers. The intermediate layer features are then subjected to global and average pooling along the channel dimension. The intermediate layer features of the visual modal features are obtained by global and average pooling to obtain two channel feature vectors. The intermediate layer features of the radar modal features are also obtained by global and average pooling to obtain two channel feature vectors. The four channel feature vectors are concatenated and passed through an MLP layer to obtain two sets of interaction weight vectors. These weight vectors are multiplied by the intermediate layer features of the visual modal and radar modal, respectively, and then added to the modal features of the original branch before being fed into the next layer of the network. The modal features of the original branch refer to the intermediate layer features that have not undergone global and average pooling. Finally, the interaction features of the two modal branches are obtained. Leveraging uncertainty to achieve intra-modal feature selection and inter-modal feature fusion includes: The class probability is modeled using the Dirichlet distribution. From the original branch modal features and interaction features, effective modal features are selected for single-modal action recognition based on the uncertainty of the features, and participate in the final fusion decision. The fusion decision is set to include: sequence-to-sequence modal fusion and dynamic inter-modal fusion of the decision layer. The sequence-to-sequence modal fusion is set as follows: based on the effective features obtained by intra-modal feature selection, the information of the horizontal and vertical dimensions is unfolded to obtain the sequence, and then the two modalities perform sequence-to-sequence cross-attention calculation to obtain the fused features; The intermodal dynamic fusion is set as follows: effective features obtained based on intramodal feature selection and fusion features obtained from sequence-to-sequence modal fusion. Dirichlet distribution is also used to model class probabilities. By utilizing uncertainty, the most effective modality is selected from the three views to obtain the final dynamic fusion result. The three views include visual modal features, radar modal features, and fusion features obtained after fusing visual features and radar features.

2. The robust visual wireless dynamic fusion method based on uncertainty modeling according to claim 1, characterized in that, The skeleton joint data is processed to generate skeleton sequence data, including: for each frame, extracting the X and Y coordinates and confidence scores of each human joint, and superimposing the data of 300 frames to obtain skeleton sequence data.

3. The highly robust visual wireless dynamic fusion method based on uncertainty modeling according to claim 1, characterized in that, Signal processing is performed on radar data to extract micro-Doppler information of human movement and generate time-velocity micro-Doppler maps, including: Based on the radar data, the time difference and frequency information of the transmitted signal and the echo signal are obtained. The distance information and velocity information of each part of the target human body are calculated according to the time difference and frequency information, thereby obtaining the time-velocity micro-Doppler image.

4. The highly robust visual wireless dynamic fusion method based on uncertainty modeling according to claim 1, characterized in that, Visual modal features are obtained by extracting features from the skeleton sequence, including: processing the human skeleton sequence based on a spatiotemporal graph convolutional network to obtain visual modal features, wherein the spatiotemporal graph convolutional network includes eleven stacked spatiotemporal convolutional layers.

5. The robust visual wireless dynamic fusion method based on uncertainty modeling according to claim 1, characterized in that, The radar modal features are obtained by extracting features from the micro-Doppler image, including: processing the micro-Doppler image based on a residual network to obtain radar modal features, wherein the residual network includes eight stacked residual blocks.

Images