High-robustness visual wireless dynamic fusion method based on uncertainty modeling
Through the visual wireless dynamic fusion method based on uncertainty modeling, the complementarity and feature uncertainty between modes are used to achieve high robust fusion of visual and wireless modes, and the performance degradation in single mode failure is solved The problem improves the accuracy of action recognition and the robustness of modal fusion.
Patent Information
- Application Number
- CN202411144506.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-20
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-08-20
AI Technical Summary
The existing visual wireless modal fusion technology is susceptible to noise interference when a single mode fails, resulting in overall performance degradation and cannot effectively solve the performance degradation caused by modal loss.
Using a highly robust visual wireless dynamic fusion method based on uncertainty modeling, effective modal complementarity and dynamic fusion can be achieved through the complementarity between modals and the uncertainty of modal features, and using the Dirichlet distribution pairs Probability modeling, select effective modal features for single-modal recognition and fusion decisions, and perform sequence-to-sequence modal fusion and dynamic fusion between modals.
It improves the accuracy of action recognition, especially when a single mode fails, can effectively improve the recognition accuracy, enhances the robustness of modal fusion, avoids noise interference, and achieves higher recognition accuracy.
Smart Images

Figure CN119068392A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of motion recognition and multimodal fusion, and in particular to a highly robust visual wireless dynamic fusion method based on uncertainty modeling. Background Art
[0002] At present, the visual wireless modal fusion technology mainly uses the complementary characteristics of the two modalities to improve the recognition performance. The visual modality has a high spatial resolution, and the radar modality has a high robustness to the environment. Therefore, the advantages of the two modalities can be used to improve the overall discrimination ability. At present, in the field of action recognition with visual and wireless multimodal fusion, most of them rely on the complementarity between the two modalities to achieve more efficient discrimination. The complementarity means that vision can effectively recognize planar actions, such as waving, drawing x, etc., but is not sensitive to radial actions, such as pushing forward, knocking on the door, etc. Radar is just the opposite, more sensitive to radial actions, but not sensitive to horizontal actions. Using this complementarity, the accuracy of overall action recognition can be effectively improved, but it is ignored that when a single modality fails, what is transmitted in modal fusion is not complementary information, but noise. Existing methods cannot solve the interference problem caused by noise, which greatly reduces the effect of modal fusion.
[0003] In summary, in the process of two-modality fusion, if a single modality fails (such as vision is affected by light), the modality fusion effect will be disturbed by the noise of the missing modality, resulting in a significant decrease in the overall performance. Summary of the invention
[0004] In view of the problems of effective complementarity and performance degradation caused by modality loss in multimodal fusion, the present invention provides a highly robust visual wireless dynamic fusion method based on uncertainty modeling. The present invention mainly utilizes the complementarity between modalities and the uncertainty of modal features to achieve effective modal complementarity and dynamic modal fusion, which can not only improve the accuracy of action recognition, but also effectively improve the accuracy of recognition when a single modality is missing.
[0005] The technical means adopted by the present invention are as follows:
[0006] A highly robust visual wireless dynamic fusion method based on uncertainty modeling includes the following steps:
[0007] S1, obtaining a target human body motion video frame sequence, and using the Openpose algorithm to process the target human body motion video sequence into skeleton joint point data; obtaining radar data of the target human body motion through radar acquisition, wherein the radar data is in the form of a cubic matrix, and the matrix includes a fast time dimension, a slow time dimension and the number of antennas;
[0008] S2, processing the skeleton joint point data to generate skeleton sequence data; performing signal processing on the radar data, extracting micro-Doppler information of human body movements, and generating a time-speed micro-Doppler map;
[0009] S3, extracting features from the skeleton sequence to obtain visual modal features; extracting features from the micro-Doppler image to obtain radar modal features;
[0010] S4. Performing inter-modal feature interaction on the visual modal feature and the radar modal feature, thereby realizing intra-modal feature selection and inter-modal feature fusion by using uncertainty, wherein performing inter-modal feature interaction on the visual modal feature and the radar modal feature includes:
[0011] The visual modal features and radar modal features are extracted through the middle layer features of the network layer, and the middle layer features are globally and averagely pooled along the channel dimension. The middle layer features of the visual modal features are globally and averagely pooled to obtain two channel feature vectors. The middle layer features of the visual modal features are also globally and averagely pooled to obtain two channel feature vectors. The four channel feature vectors are concatenated and passed through the MLP layer to obtain two sets of interaction weight vectors, which are multiplied with the middle layer features of the visual modality and radar modality respectively and added to the modal features of the original branch and sent to the next layer of the network. The modal features of the original branch refer to the middle layer features that have not been globally and averagely pooled, and finally the interaction features of the two modal branches are obtained.
[0012] Furthermore, uncertainty is used to achieve intra-modal feature selection and inter-modal feature fusion, including:
[0013] The class probability is modeled using Dirichlet distribution. From the original branch modal features and the interaction features, effective modal features are selected according to the uncertainty of the features for single-modal action recognition and participate in the final fusion decision. The fusion decision is set to include: sequence-to-sequence modal fusion and dynamic inter-modal fusion at the decision layer.
[0014] The sequence-to-sequence modality fusion is set as follows: based on the effective features obtained by feature selection within the modality, the information of the spatial and spatial dimensions is flattened to obtain a sequence, and then the two modalities perform sequence-to-sequence cross-attention calculation to obtain a fused feature;
[0015] The inter-modal dynamic fusion is set as follows: based on the effective features obtained by intra-modal feature selection and the fused features obtained by sequence-to-sequence modal fusion, the class probability is modeled using Dirichlet distribution, and uncertainty is utilized to select the most effective modality from the three views, thereby obtaining the final dynamic fusion result. The three views include visual modal features, radar modal features, and fused features obtained after the visual features and radar features are fused.
[0016] Furthermore, the skeleton joint point data is processed to generate skeleton sequence data, including: for each frame, extracting the X and Y coordinates and confidence of each joint point of the human body, and superimposing the data of 300 frames to obtain the skeleton sequence data.
[0017] Furthermore, the wireless data is signal processed to extract the micro-Doppler information of human body movements and generate a time-speed micro-Doppler map, including:
[0018] The time difference and frequency information of the transmission signal and the echo signal are obtained based on the wireless data, and the distance information of each part of the target human body and the speed information of each part are calculated according to the time difference and frequency information, so as to obtain a time-speed micro-Doppler map.
[0019] Furthermore, feature extraction is performed on the skeleton sequence to obtain visual modal features, including: processing the human skeleton sequence based on a spatiotemporal graph convolutional network to obtain visual modal features, wherein the spatiotemporal graph convolutional network includes eleven stacked spatiotemporal convolutional layers.
[0020] Furthermore, feature extraction is performed on the micro-Doppler image to obtain radar modal features, including: processing the micro-Doppler image based on a residual network to obtain the radar modal features, wherein the residual network includes eight stacked residual blocks.
[0021] Compared with the prior art, the present invention has the following advantages:
[0022] The present invention dynamically fuses the features of two modes. When a single mode provides noise, it uses uncertainty to reduce the weight of this branch, and increases the feature weight of the effective mode. It also uses channel interaction to achieve complementarity between modes, giving full play to the complementarity of the two modes. When both modes work normally, it can provide a higher accuracy estimate, and when a single mode fails, it can select more effective features from the interactive features and the original features through the intra-modal feature selection algorithm. The present invention can not only achieve the complementarity in previous methods, but also perform effective complementarity. While achieving modal fusion, it further improves the fusion effect.
[0023] The present invention utilizes the uncertainty of the modal to dynamically select effective modal features, thereby solving the problem of overall model performance degradation when a single modal fails. It also performs feature interaction in the channel dimension and utilizes intra-modal feature selection to achieve effective modal complementarity. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0025] Figure 1 A network architecture diagram for realizing a highly robust visual wireless dynamic fusion method based on uncertainty modeling of the present invention.
[0026] Figure 2 This is a block diagram of utilizing uncertainty to perform intra-modal feature selection and inter-modal dynamic fusion in an embodiment of the present invention.
[0027] Figure 3 This is a flowchart of sequence-to-sequence feature fusion in an embodiment of the present invention. DETAILED DESCRIPTION
[0028] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0029] like Figure 1 As shown, the present invention provides a highly robust visual wireless dynamic fusion method based on uncertainty modeling, comprising:
[0030] S1, obtaining a target human body motion video frame sequence, and using the Openpose algorithm to process the target human body motion video sequence into skeleton joint point data; obtaining radar data of the target human body motion through radar acquisition, wherein the radar data is in the form of a cubic matrix, and the matrix includes a fast time dimension, a slow time dimension and the number of antennas;
[0031] In the present invention, the coordinates of the skeleton joint points and the confidence of each joint point are estimated directly from the video frame sequence using the Openpose algorithm. The echo signal collected by the radar is combined with the transmission signal to process into an intermediate frequency signal, and then the intermediate frequency signal is subjected to time-frequency analysis to obtain the radar data cube matrix of the target human body motion, where the radar data cube matrix represents the fast time dimension, the slow time dimension and the number of antennas.
[0032] S2. Process the skeleton joint data, extract 18 joints of each frame of the human body, and form skeleton sequence data. Perform signal processing on the wireless data, use MATLAB to process the wireless signal data, extract the micro-Doppler information of the human body movement, and make a time-speed micro-Doppler map.
[0033] Specifically, for each frame, the X and Y coordinates and confidence of each joint point of the human body are extracted, and the data of 300 frames are superimposed to obtain the skeleton sequence data. For the wireless data, the time difference and frequency information of the transmitted signal and the echo signal are used to calculate the distance information of each part of the target human body, and further calculate the speed information of each part, and process it to obtain the time-speed micro-Doppler map.
[0034] S3. Extract features from the skeleton sequence to obtain visual modal features; extract features from the micro-Doppler image to obtain radar modal features.
[0035] In the present invention, the overall network utilizes the idea of deep learning and is mainly divided into two parts: one is to extract visual features from the skeleton sequence of the visual modality, and the other is to extract radar features from the micro-Doppler map of the radar modality.
[0036] Specifically, the input of the visual branch feature is a skeleton sequence (N, 3, 300, 18, 2), where N represents the BatchSize number, 3 refers to the x, y coordinates and confidence values of each skeleton point, 300 is the number of frames, 18 represents the number of skeleton points, and 2 represents the two people detected with the highest confidence. It will pass through the eleven-layer spatiotemporal graph convolution network in sequence to obtain the final visual modality output, which is called the original visual feature. The first four layers are called the visual model Block1. The number of channels of the spatiotemporal convolution layer is 64, the convolution kernel size of the spatial graph convolution is 3, which is used to extract the spatial features of the human skeleton points, and the convolution kernel size of the temporal graph convolution is 9, which is used to extract the temporal information between sequences. The activation function is Relu. The input will first pass through the spatial graph convolution, then the temporal graph convolution, and finally use the residual structure for output. The number of channels in the first layer changes from 3 to 64, and the number of channels in the following three layers remains unchanged. The output result will be input into the next layer of the network on the one hand, and will interact with the radar modal features in the channel dimension on the other hand, and then multiply by the weight, and add the original branch features to continue to input into the next layer of the network. This feature is an interactive feature. The fifth, sixth, and seventh layers are called the visual model Block2. The number of channels in the spatiotemporal convolution layer is 128, the convolution kernel size of the spatial graph convolution is 3, the convolution kernel of the temporal graph convolution is 9, and the activation function is Relu. After the input passes through the fifth layer, the number of channels changes from 64 to 128, and the number of channels in the following two layers remains unchanged. The output result will be input into the next layer of the network on the one hand, and will interact with the radar modal features in the channel dimension on the other hand, and then multiply by the weight, and add the original branch features to continue to input into the next layer of the network. This feature is an interactive feature. The eighth, ninth, and tenth layers are called visual model Block3. The number of channels in the spatiotemporal convolution layer is 256, the convolution kernel size of the spatial graph convolution is 3, the convolution kernel size of the temporal graph convolution is 9, and the activation function is Relu. After the input passes through the eighth layer, the number of channels changes from 128 to 256, and the number of channels in the following two layers remains unchanged. The output result will be input into the next layer of the network on the one hand, and will interact with the radar modal features in the channel dimension on the other hand, and then multiplied by the weight, and added to the original branch feature to continue to be input into the next layer of the network. This feature is the interactive feature. The eleventh layer is called visual model Block4. The convolution kernel size of the spatial graph convolution is 3, the convolution kernel size of the temporal graph convolution is 9, and the activation function is Relu. The number of channels changes from 256 to 512, and two output results are obtained, one is the original feature, and the other is the interactive feature. The two features are average pooled, flattened, and multi-layer perceptron to obtain ten-class features (without softmax).
[0037] The input of the radar branch is (N, 3, 128, 128), where N is the BatchSize number, 3 is the RGB three channels, and 128*128 represents the length and width of the image. It will pass through 8 layers of residual blocks in sequence to obtain the final radar modal output, which is called the original radar feature. The input micro-Doppler image passes through a convolution layer with a convolution kernel of 3 and a Relu activation function, and the number of channels changes from 3 to 64. It is then input into the residual block. The first and second layers of the residual block are called radar model Block1, which has two convolution layers with a convolution kernel size of 3, and is batch normalized. The activation function is Relu, and the feature output is obtained through the residual structure, and the number of channels changes from 64 to 64. The output result will be input into the next layer of the network on the one hand, and on the other hand, it will interact with the visual modality feature in the channel dimension, then multiply the weight, and add it to the original branch feature to continue to input into the next layer of the network. This feature is an interactive feature. The third and fourth residual blocks are called radar model Block2, which has two convolutional layers, the convolution kernel size is 3, and after batch normalization, the activation function is Relu, and the feature output is obtained through the residual structure, and the number of channels changes from 64 to 128. The output result will be input to the next layer of the network on the one hand, and on the other hand, it will interact with the visual modality features in the channel dimension, then multiply by the weight, and add the original branch features to continue to input to the next layer of the network. This feature is the interactive feature. The fifth and sixth residual blocks are called radar model Block3, which has two convolutional layers, the convolution kernel size is 3, and after batch normalization, the activation function is Relu, and the feature output is obtained through the residual structure, and the number of channels changes from 128 to 256. The output result will be input to the next layer of the network on the one hand, and on the other hand, it will interact with the visual modality features in the channel dimension, then multiply by the weight, and add the original branch features to continue to input to the next layer of the network. This feature is the interactive feature. The seventh and eighth residual blocks are called radar model Block4, which has two convolutional layers, with a convolution kernel size of 3, batch normalization, Relu activation function, and feature output through the residual structure. The number of channels changes from 256 to 512. Two output results are obtained, one is the original feature and the other is the interaction feature. The two features are average pooled, flattened, and multi-layer perceptron to obtain ten-class features (without using softmax).
[0038] S4, performing inter-modal feature interaction, intra-modal feature selection and inter-modal feature fusion on the visual modal features and the radar modal features. Mainly including: using Dirichlet distribution to model the class probability, from the original features ( Figure 1 The dashed feature flow in Figure 2In the solid feature flow in ), according to the uncertainty of the features, effective modal features are selected for single-modal action recognition and participate in the final fusion decision, which is set to include: sequence-to-sequence modal fusion and dynamic fusion between modalities at the decision layer.
[0039] Among them, the sequence-to-sequence modal fusion is set as follows: based on the effective features obtained by feature selection within the modality, the information of the spatial and spatial dimensions is flattened to obtain the sequence, and then the two modalities perform sequence-to-sequence cross-attention calculation to obtain the fused features.
[0040] The dynamic fusion between modalities is set as follows: based on the effective features obtained by intra-modal feature selection and the fused features obtained by sequence-to-sequence modal fusion, the class probability is modeled using Dirichlet distribution. By utilizing uncertainty, the most effective modality is selected from the three views to obtain the final dynamic fusion result. The three views include visual modal features, radar modal features, and fused features obtained by fusing visual features and radar features.
[0041] Specifically, inter-modal feature interaction refers to the output feature maps of the two modalities in Block1, Block2, Block3, and Block4, respectively, to perform feature interaction in the channel dimension between the two modalities to obtain the interaction features of the two branches. m∈{V,R}, V represents the visual modality, and R represents the radar modality. The feature maps of the two modal features are averaged and globally pooled along the channel dimension. The intermediate layer features of the visual modality features are globally and averagely pooled to obtain two channel feature vectors. The intermediate layer features of the visual modality features are also globally and averagely pooled to obtain two channel feature vectors. The four channel feature vectors are concatenated and passed through the MLP layer to obtain two sets of interaction weight vectors, which are multiplied with the intermediate layer features of the visual modality and radar modality respectively and added to the modal features of the original branch and sent to the next layer of the network. The modal features of the original branch refer to the intermediate layer features that have not been globally and averagely pooled, and finally the interaction features of the two modal branches are obtained.
[0042]
[0043] where i = (2, 3, 4), is the result obtained by passing the original input of the modality through the first Block, F block-i Refers to the function expression of the i-th block, the final After MLP, we get ten types of output.
[0044] The intra-modal feature selection mentioned above mainly utilizes the uncertainty in the theory of evidence deep learning. The theory proves that the ten-class result {μ1,μ2,...,μ K}, use the Relu activation function to set negative values to zero, and the resulting feature vector {e1,e2,...,e K} (K represents the number of classification categories, which is 10 in the embodiment of the present application) can represent the reliability of each class and the uncertainty of this set of features, and the formula is:
[0045]
[0046] in is called the Dirichlet intensity, b k It is called credibility, which is also a normalized feature. u is an uncertain feature, which can well represent the uncertainty of this set of features. If a single mode fails, its uncertainty is often very high, which helps the model select the features of another mode, thereby achieving dynamic fusion. The value of the uncertain feature is between [0,1]. The closer it is to 1, the more unreliable the mode is, that is, the mode fails, and the closer it is to 0, the more reliable the mode is. The calculation formula for the normalized feature (credibility) after dynamic fusion is:
[0047]
[0048] in, is a measure of the amount of conflict between two feature quality sets, and the scaling factor Used for normalization. Dirichlet distribution parameters {α1,α2,...,α K} is composed of {e1,e2,...,e K}+1. Figure 1 As shown, It refers to the result obtained by calculation by the dotted line in the system block diagram, and it refers to the original feature result obtained after the input is extracted. m Refers to the uncertainty of the original feature before the dotted line to the IFS module. It refers to the result obtained by calculation after the solid line in the system block diagram, and it refers to the interactive feature result obtained after the input is extracted. m refers to the uncertainty of this interaction feature before the solid line to the IFS module.
[0049] The optimized EDL loss function is:
[0050]
[0051] Among them, q θ m(μm|x m ) refers to the Dirichlet distribution of μm, x m {m=1,2} refers to the mode.
[0052] In this application, the Dirichlet distribution is used to model the class probability and use uncertainty to do dynamic fusion, which is to quantify the uncertainty of the output results. According to the DS evidence theory, the uncertainty and credibility parameters actually model the "probability of probability", and the Dirichlet distribution can just model the "probability of probability", that is, this application applies the formula framework of the Dirichlet distribution to quantify the required uncertainty. The Dirichlet distribution is used to model the probability of probability, so that the original probability value becomes a distribution, so it is necessary to optimize the standard cross entropy loss.
[0053] The standard cross entropy loss is:
[0054] logp(y|μ m )
[0055] Since μ has a specific value when the standard cross entropy loss is calculated, and μ is a distribution in this application, constrained modal fusion can be performed by minimizing this expectation, that is, in this application:
[0056]
[0057] At this time μ m The distribution of is the Dirichlet distribution.
[0058] The inter-modal feature fusion includes: sequence-to-sequence modal fusion and dynamic inter-modal fusion at the decision layer. The sequence-to-sequence modal fusion is to flatten the information of the spatial and spatial dimensions to obtain a sequence based on the effective features obtained by intra-modal feature selection. The feature dimension is X∈R N×C Then the two modalities perform sequence-to-sequence cross-attention calculation to obtain fusion features. The specific formula is:
[0059]
[0060]
[0061] The subscripts v and r represent vision and radar respectively, K and V are weight vectors obtained by linear processing of feature X, and G = KV represents the global context vector. The superscript inter represents intra-modality, and the final result U represents the attention result. The above is obtained by reconstructing the original attention formula.
[0062] The two features obtained are subjected to residual connection, concatenation, average pooling and linear layer to obtain fused features.
[0063] The inter-modal dynamic fusion is based on the effective features obtained by selecting intra-modal features and the fusion features obtained by sequence-to-sequence modal fusion. Through the above-mentioned dynamic fusion algorithm, the most effective modal features are selected from the three views to obtain the final dynamic fusion result. The three views include visual modal features, radar modal features, and fusion features obtained after the fusion of visual features and radar features.
[0064] S5. The overall framework is trained end-to-end, and cross entropy loss and EDL evidence loss are used for optimization training.
[0065] Cross entropy loss is used to constrain the output classification results of each branch modality and the fusion results. EDL evidence loss uses uncertainty to further optimize the model. The trained network can have strong robustness and maintain the overall recognition accuracy of the model for the problem of modality missing. At the same time, it effectively complements the modalities and does not cause transmission noise due to modality missing, which will interfere with the final fusion result.
[0066] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A highly robust visual wireless dynamic fusion method based on uncertainty modeling, It is characterized in that The following steps are involved: S1, obtaining a target human body motion video frame sequence, and using the Openpose algorithm to process the target human body motion video sequence into skeleton joint point data; obtaining radar data of the target human body motion through radar acquisition, wherein the radar data is in the form of a cubic matrix, and the matrix includes a fast time dimension, a slow time dimension and the number of antennas; S2, processing the skeleton joint point data to generate skeleton sequence data; performing signal processing on the radar data, extracting micro-Doppler information of human body movements, and generating a time-speed micro-Doppler map; S3, extracting features from the skeleton sequence to obtain visual modality features; Extracting features from the micro-Doppler image to obtain radar modal features; S4. Performing inter-modal feature interaction on the visual modal feature and the radar modal feature, thereby realizing intra-modal feature selection and inter-modal feature fusion by using uncertainty, wherein performing inter-modal feature interaction on the visual modal feature and the radar modal feature includes: The visual modal features and radar modal features are extracted through the middle layer features of the network layer, and the middle layer features are globally and averagely pooled along the channel dimension. The middle layer features of the visual modal features are globally and averagely pooled to obtain two channel feature vectors. The middle layer features of the visual modal features are also globally and averagely pooled to obtain two channel feature vectors. The four channel feature vectors are concatenated and passed through the MLP layer to obtain two sets of interaction weight vectors, which are multiplied with the middle layer features of the visual modality and radar modality respectively and added to the modal features of the original branch and sent to the next layer of the network. The modal features of the original branch refer to the middle layer features that have not been globally and averagely pooled, and finally the interaction features of the two modal branches are obtained.
2. According to the highly robust visual wireless dynamic fusion method based on uncertainty modeling according to claim 1, It is characterized in that Using uncertainty to achieve intra-modal feature selection and inter-modal feature fusion, including: The class probability is modeled using Dirichlet distribution. From the original branch modal features and the interaction features, effective modal features are selected according to the uncertainty of the features for single-modal action recognition and participate in the final fusion decision. The fusion decision is set to include: sequence-to-sequence modal fusion and dynamic inter-modal fusion at the decision layer. The sequence-to-sequence modality fusion is set as follows: based on the effective features obtained by feature selection within the modality, the information of the spatial and spatial dimensions is flattened to obtain a sequence, and then the two modalities perform sequence-to-sequence cross-attention calculation to obtain a fused feature; The inter-modal dynamic fusion is set as follows: based on the effective features obtained by intra-modal feature selection and the fused features obtained by sequence-to-sequence modal fusion, the class probability is modeled using Dirichlet distribution, and uncertainty is utilized to select the most effective modality from the three views, thereby obtaining the final dynamic fusion result. The three views include visual modal features, radar modal features, and fused features obtained after the visual features and radar features are fused.
3. According to the highly robust visual wireless dynamic fusion method based on uncertainty modeling described in claim 1, It is characterized in that The skeleton joint point data are processed to generate skeleton sequence data, including: for each frame, extracting X and Y coordinates and confidence of each joint point of the human body, and superimposing 300 frames of data to obtain skeleton sequence data.
4. According to the highly robust visual wireless dynamic fusion method based on uncertainty modeling according to claim 1, It is characterized in that Perform signal processing on wireless data, extract micro-Doppler information of human motion, and generate time-speed micro-Doppler maps, including: The time difference and frequency information of the transmission signal and the echo signal are obtained based on the wireless data, and the distance information of each part of the target human body and the speed information of each part are calculated according to the time difference and frequency information, so as to obtain a time-speed micro-Doppler map.
5. According to the highly robust visual wireless dynamic fusion method based on uncertainty modeling according to claim 1, It is characterized in that Extracting features from the skeleton sequence to obtain visual modal features includes: processing the human skeleton sequence based on a spatiotemporal graph convolutional network to obtain visual modal features, wherein the spatiotemporal graph convolutional network includes eleven stacked spatiotemporal convolutional layers.
6. According to the highly robust visual wireless dynamic fusion method based on uncertainty modeling according to claim 1, It is characterized in that Extracting features from the micro-Doppler image to obtain radar modal features includes: processing the micro-Doppler image based on a residual network to obtain the radar modal features, wherein the residual network includes eight stacked residual blocks.
Citation Information
Patent Citations
Action representation and identification method based on visual and wireless bimodal joint perception
CN115116124A
Violence detection system based on time-space information credible fusion
CN117876746A
Multi-target vehicle detection and re-identification method based on radar and video fusion
GB202313217D0
