A multi-modal hierarchical cross-fusion recognition method based on vision and WiFi

Abstract

The application belongs to the technical field of non-contact human behavior, and discloses a multi-modal layered cross fusion recognition method based on vision and WiFi. The method uses common commercial cameras and WiFi signal acquisition equipment to sample human behavior, cooperates with a special deep learning network to extract discriminative features of videos and WiFi signals, uses a multi-modal layered cross fusion method to mine complementary information relationship of the two modalities, realizes complementation of heterogeneous information, eliminates redundancy between information, makes up for the problem of inaccurate recognition of single modal in adverse conditions (obstacles, dim light, improper angle, etc.), and realizes high-precision and high-robustness recognition of human actions. The application fills the vacancy of multi-modal fusion and complementation in the field of non-contact sensing, and creates an application example for passive sensing and recognition based on deep learning. The method can be widely applied to fields such as precise care, diet monitoring and intrusion detection.

Description

Technical Field

[0001] This invention relates to the field of non-contact human behavior technology, and in particular to a multimodal hierarchical cross-fusion recognition method based on vision and WiFi. Background Technology

[0002] Human movements are a manifestation of daily life, and their automatic recognition is fundamental to the study of human life patterns and habits. It supports research in precision care, healthcare, and security monitoring, and has broad application prospects in urban public safety management, nursing homes, and digital entertainment, making it popular in both industry and academia. As one of the most common recognition technologies, video-based human behavior recognition has experienced rapid development. Video RGB data, with its intuitive appearance features including color, texture, and shape, along with information about the human's surrounding environment, can better assist computers in recognizing behaviors that interact with the environment. Furthermore, with the exponential growth in the number of videos, increased computing power, and the development of machine learning algorithms, video-based human behavior recognition has gained even more favor among researchers. The emergence of wireless sensing technology, especially WiFi, has provided new ideas for human movement recognition, presenting a continuous and non-invasive sensing method. Researchers can utilize existing WiFi communication facilities to collect, analyze, and identify ubiquitous WiFi human behavior signals, enabling accurate monitoring of targets without requiring any specific equipment. In addition, the advantages of WiFi signals—being unaffected by light and having the ability to penetrate walls—have accelerated its research status in the field of human movement recognition.

[0003] However, human behavior recognition based on the single modality of video and WiFi still faces many challenges. In most cases, visual information, as a direct signal, can accurately identify actions. However, it is unfavorable for video monitoring in scenarios such as low light, object obstruction, and improper viewing angles. Furthermore, WiFi signals are susceptible to hardware noise, environmental deployment, and movement speed, resulting in signal instability and hindering effective target action recognition. These problems may reduce the user experience in human-computer interaction games and are less effective for monitoring abnormal behavior in elderly people living alone or children. However, the low light requirements and wall-penetrating capabilities of WiFi signals can serve as powerful supplementary information to video, compensating for these shortcomings. Conversely, the rich and relatively stable feature information of video can assist WiFi signals. Therefore, multimodal fusion of video and WiFi signals can fully complement each other, overcoming the limitations of single-modal prediction and becoming an effective technical approach in the field of human action recognition. Currently, human behavior recognition methods based on the fusion of video and WiFi signals either rely on simple fusion based on the decision layer, such as "Lang Deng, Jianfei Yang, Shenghai Yuan, Han Zou, Chris Xiaoxuan Lu and LihuaXie. GaitFi: Robust Device-Free Human Identification via WiFi and Vision Multimodal Learning[J].IEEE Internet of Things Journal.2023,10(1):625-636."; or map the deep features of the two modalities to a common space, such as "Jun Guo, Mei Shi, Xingwu Zhu, Wei Huang, Yi He, Weiwei Zhang, and Zhanyong Tang. Improving human action recognition by jointly exploiting video and WiFi clues[J].Neurocomputing.2021,458(C):14-23." However, there is still no information sharing between modalities, and the complementarity between multimodal features cannot be effectively utilized. Summary of the Invention

[0004] This invention addresses the problem of effectively utilizing and fusing complementary information from video and WiFi signals to achieve accurate human motion recognition even in adverse environments such as low light, object obstruction, and unfavorable viewing angles. By fully leveraging the complementary auxiliary recognition capabilities of WiFi signals and video, and employing an effective hierarchical cross-fusion scheme to integrate features from different levels of the two modalities, the final human motion recognition result is obtained, thus solving the problem of unsatisfactory recognition performance under adverse conditions. This invention, by combining computer vision, wireless sensing, and attention mechanisms, provides theoretical basis and practical experience for the field of human behavior recognition, contributing new solutions for applications such as health monitoring, intrusion detection, and human-computer interaction.

[0005] The technical solution of this invention: a multimodal hierarchical cross-fusion recognition method based on vision and WiFi, comprising the following steps:

[0006] Step 1: Multimodal sequence data acquisition;

[0007] Multimodal sequence data acquisition includes video behavioral sequence image data acquisition and WiFi signal data acquisition;

[0008] Step 2: Preprocessing of multimodal sequence data;

[0009] Multimodal sequence data preprocessing includes video behavior sequence image processing and WiFi signal data processing; video behavior sequence image processing is used to remove redundant video behavior sequence images, and extract representative video behavior sequence images from the intermediate segments of the removed redundant video behavior sequence images; WiFi signal data processing is used to denoise and reduce the dimensionality of WiFi signal data, generate a channel state information correlation coefficient matrix, and reconstruct unique WiFi signal data.

[0010] Step 3: Feature extraction and fusion of multimodal sequence data;

[0011] Multimodal sequence data feature extraction and fusion includes a low-level implicit feature extraction module and a high-level hierarchical cross-fusion module. In the low-level implicit feature extraction module, a deep learning network is used to obtain low-level multimodal implicit features from representative video behavior sequence images and reconstructed WiFi signal data, including video behavior sequence image features and WiFi signal data features. Then, the high-level hierarchical cross-fusion module realizes the complementary fusion of the two modal features of video behavior sequence images and WiFi signal data to obtain fused features.

[0012] Step 4: Multimodal loss optimization and classification;

[0013] Multimodal loss optimization and classification is used to integrate the loss of video behavior sequence image features, WiFi data signal features, and fusion feature loss to determine the multimodal loss. The multimodal loss is then used to optimize the deep learning network in multimodal sequence data feature extraction and fusion, and the behavior classification result is output.

[0014] The preprocessing of the multimodal sequence data is as follows:

[0015] The WiFi signal data is processed by extracting the original amplitude and phase information from the WiFi channel state information. An antenna ratio method is used to eliminate additional time-varying random phase shifts between the WiFi signal transmitter and receiver. A Hampel filter is used to remove noise and outliers caused by equipment defects and the surrounding environment. After obtaining the purified WiFi signal data, PCA is used for dimensionality reduction to filter redundant information caused by high similarity between subcarriers of the same antenna pair. The WiFi signal data is further reconstructed by dividing it into M WiFi channel state information subsequences along the time dimension. For each WiFi channel state information subsequence, the correlation coefficient matrix between different antenna pairs is calculated and concatenated to obtain the channel state information correlation coefficient matrix. The obtained channel state information correlation coefficient matrix is ​​multiplied by its transpose to obtain the relationship between the M WiFi channel state information subsequences, resulting in a uniquely reconstructed WiFi signal data C.

[0016] The video behavior sequence image is formed by dividing the original video behavior sequence image corresponding to WiFi signal data into N video behavior sub-sequences of the same length according to the number of frames, and randomly extracting a sub-image from each video behavior sub-sequence to form a behavior sequence image sample V.

[0017] The feature extraction and fusion of the multimodal sequence data are as follows:

[0018] In low-level multimodal implicit feature extraction, the neural learning network includes a video feature extraction network 3DCNN and a spatiotemporal network DTNet; the video feature extraction network 3DCNN is used to extract the RGB features of the behavioral sequence image samples V, thereby obtaining the low-level features f of the video behavioral sequence images. v =3DCNN(V), using a spatiotemporal network DTNet composed of dilated convolutions and temporal convolutions to extract the correlation features of the reconstructed WiFi signal data C, thus obtaining the low-level features f of the WiFi signal data. c =DTNet(C), thus obtaining low-level multimodal implicit features;

[0019] In high-level multimodal cross-fusion, a Transformer-based hierarchical cross-fusion approach is used to extract high-level features from low-level multimodal implicit features while fusing high-level features from both video behavior sequence images and WiFi signal data to obtain fused features; v,i =CP(f v ),f c,i =CP(f c Z i ←Tf(C(f v,i ,f c,i )), f v,i+1 =CP(C(Z) i ,f v,i )),f c,i+1 =CP(C(Z) i ,f c,i Tf(·) represents the Transformer operation, C(·) represents the contact operation in the modal dimension, CP(·) represents the convolution and pooling operations, and i represents the number of times the CP and Transformer operations are performed.

[0020] The multimodal loss optimization and classification are detailed below:

[0021] Loss of video action sequence image features L v Loss of WiFi signal data features L c and multimodal fusion feature loss L vc To integrate the results, determine the multimodal loss optimization and classification function, L = αL v +βL c +γL vc A deep learning network for feature extraction and fusion of multimodal sequence data is optimized by multimodal loss and classification function L, and the output behavior classification results are generated.

[0022] The WiFi signal data acquisition device includes a router as the transmitter and a computer as the receiver, which are placed separately in the room. The transmitter is used to send WiFi signals carrying channel state information, and the receiver is used to receive superimposed WiFi signal data caused by human actions in the environment. The video data acquisition device is a camera installed at the WiFi signal receiver to capture images of the action sequence that is being performed and corresponds to the WiFi signal data.

[0023] The beneficial effects of this invention are as follows: The multimodal hierarchical cross-fusion recognition method based on vision and WiFi proposed in this invention fully utilizes the rich visual information of video and the advantages of WiFi signal data, such as its ability to penetrate walls and its immunity to light. Based on the concept of multimodal fusion, it cross-fused visual features based on video behavior sequence images and signal features based on WiFi signal data to achieve complementarity of heterogeneous information, eliminate redundancy between information, and compensate for the inaccuracy of single-modal recognition under adverse conditions (obstacles, dim lighting, improper viewing angles, etc.), thus achieving high-precision and robust recognition of human actions. This fills the gap in multimodal fusion and complementarity in the field of non-contact sensing, providing an application example for passive sensing and recognition based on deep learning. This method can be widely applied to fields such as precision care, diet monitoring, and intrusion detection. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of a multimodal hierarchical cross-fusion recognition method based on vision and WiFi according to the present invention.

[0025] Figure 2 This is a schematic diagram of the structure for multimodal sequence data feature extraction and fusion provided by the present invention. Detailed Implementation

[0026] The specific embodiments of the present invention are described in detail below with reference to the technical solution (and accompanying drawings).

[0027] This invention provides a multimodal hierarchical cross-fusion recognition method based on vision and WiFi. It uses common commercial cameras and WiFi signal acquisition devices to sample human behavior, and combines them with a unique deep learning algorithm to extract discriminative features from video and WiFi signals. The multimodal hierarchical cross-fusion method is used to mine the complementary information relationship between the two modalities, which makes up for the problem of inaccurate recognition of single modality under adverse conditions (obstacles, dim lighting, improper viewing angle, etc.), and achieves high-precision and high-robust recognition of human behavior.

[0028] A multimodal hierarchical cross-fusion recognition method based on vision and WiFi, comprising the following steps:

[0029] Step 1: Multimodal Sequence Data Acquisition. When a user enters the room, the WiFi signal transmitter sends out a WiFi signal. The user's movement or actions significantly disrupt the WiFi signal. The Logitech camera, a video data acquisition device, simultaneously captures the user's behavior, thereby obtaining synchronized WiFi signal data and video behavior sequence image data.

[0030] Step 2, Multimodal Sequence Data Preprocessing. The synchronized WiFi signal data and video behavior sequence images obtained in Step 1 are preprocessed as follows:

[0031] For WiFi signal data, the original amplitude and phase information are extracted from the WiFi channel state information. The antenna ratio method is used to eliminate the additional time-varying random phase offset generated between the WiFi transmitter and receiver. The Hampel filter is used to remove noise and outliers caused by equipment defects and the surrounding environment. After obtaining the purified WiFi signal data, PCA is used for dimensionality reduction to filter out redundant information caused by high similarity between subcarriers of the same antenna pair. The WiFi signal data is further reconstructed by dividing it into M WiFi channel state information subsequences along the time dimension. For each WiFi channel state information subsequence, the correlation coefficient matrix between different antenna pairs is obtained and concatenated to obtain the channel state information correlation coefficient matrix. The obtained channel state information correlation coefficient matrix is ​​multiplied by its transpose to obtain the relationship between the M WiFi channel state information subsequences, resulting in the reconstructed WiFi signal data C.

[0032] For video behavior sequence images, the original video behavior sequence images corresponding to WiFi signal data are divided into N video behavior subsequences of the same length according to the number of frames. A sub-image is randomly extracted from each video behavior subsequence to form a behavior sequence image sample V.

[0033] Step 3: Multimodal sequence data feature extraction and fusion. In low-level multimodal implicit feature extraction, a 3D CNN video feature extraction network is used to extract the RGB features of the behavioral sequence image samples V, obtaining the low-level features f of the video behavioral sequence images. v =3DCNN(V), using a spatiotemporal network DTNet composed of dilated convolutions and temporal convolutions to extract the correlation features of the reconstructed WiFi signal data C, thus obtaining the low-level features f of the WiFi signal data. c =DTNet(C), thus obtaining low-level multimodal implicit features;

[0034] In high-level multimodal cross-fusion, a Transformer-based hierarchical cross-fusion approach is used to extract high-level features while fusing high-level features from both video behavior sequence images and WiFi signal data to obtain fused features; v,i =CP(f v ),f c,i =CP(f c Z i ←Tf(C(f v,i ,f c,i )), f v,i+1 =CP(C(Z) i ,f v,i )),f c,i+1 =CP(C(z) i ,fc,i Tf(·) represents the Transformer operation, C(·) represents the contact operation in the modal dimension, CP(·) represents the convolution and pooling operations, and i represents the number of times the operation is performed by CP and Transformer.

[0035] Step 4, Multimodal Loss Optimization and Classification. The loss function L is applied to the image features of the video action sequence. v Loss of WiFi signal data features L c and multimodal fusion feature loss L vc To integrate the results, determine the multimodal loss optimization and classification function, L = αL v +βL c +γL vc A deep learning network for feature extraction and fusion of multimodal sequence data is optimized by multimodal loss and classification function L, and the output behavior classification results are generated.

[0036] like Figure 1 As shown, a multimodal hierarchical cross-fusion recognition method based on vision and WiFi includes multimodal sequence data acquisition (including camera, WiFi signal transmitter, WiFi signal receiver, and user), multimodal sequence data preprocessing (including WiFi signal data preprocessing and video behavior sequence image preprocessing), multimodal sequence data feature extraction and fusion (including low-level multimodal implicit feature extraction and high-level multimodal cross-fusion), and multimodal loss optimization and classification.

[0037] For WiFi data preprocessing, as a specific implementation, the original amplitude information A and the original phase information Ψ are extracted from the WiFi channel state information; the antenna ratio method is used to eliminate the additional time-varying random phase offset generated between the WiFi signal transmitter and the WiFi signal receiver, generating... The sequence data (including amplitude and phase) is given, where m is the number of transceiver channels (i.e., antenna pairs) and n is the number of antennas selected for quotient calculation. In this example, m = 3 and n = 2. A Hampel filter is used to remove noise and outliers caused by equipment defects and the surrounding environment. After obtaining the purified WiFi signal data, PCA is used for dimensionality reduction to filter out redundant information caused by high similarity between subcarriers of the same antenna pair. The WiFi signal data is further reconstructed by dividing it into M non-overlapping WiFi channel state information subsequences along the time dimension. For each WiFi channel state information subsequence, the correlation coefficient matrix between different antenna pairs is obtained and then concatenated to obtain the channel state information correlation coefficient matrix. The obtained channel state information correlation coefficient matrix is ​​multiplied by its own transpose to obtain the relationship between the M WiFi channel state information subsequences, resulting in the reconstructed WiFi signal data C with uniqueness.

[0038] like Figure 2 As shown, in low-level multimodal implicit feature extraction, the following is used: Figure 2 The 3DCNN video feature extraction network extracts the RGB features of the processed action sequence image V, obtaining the low-level features f of the video action sequence image. v =3DCNN(V), and use the spatiotemporal network DTNet, which is composed of dilated convolution and temporal convolution, to extract the correlation features of the reconstructed WiFi signal data C, and obtain the low-level features f of the WiFi signal data. c =DTNet(C), thus obtaining low-level multimodal implicit features; in high-level multimodal cross-fusion, using Figure 2 The high-level multimodal cross-fusion part uses a Transformer-based hierarchical cross-fusion method to extract high-level features while fusing high-level features from both video behavior sequence images and WiFi signal data to obtain fused features; v,i =CP(f v ),f c,i =CP(f c Z i ←Tf(C(f v,i ,f c,i )), f v,i+1 =CP(C(Z) i ,f v,i )),f c,i+1 =CP(C(z) i ,f c,i Tf(·) represents the Transformer operation, C(·) represents the contact operation in the modal dimension, CP(·) represents the convolution and pooling operations, and i represents the number of times the operation is performed by CP and Transformer.

Claims

1. A multimodal hierarchical cross-fusion recognition method based on vision and WiFi, characterized in that, The steps include the following: Step 1: Multimodal sequence data acquisition; Multimodal sequence data acquisition includes video behavioral sequence image data acquisition and WiFi signal data acquisition; Step 2: Preprocessing of multimodal sequence data; Multimodal sequence data preprocessing includes video action sequence image processing and WiFi signal data processing; Video behavior sequence image processing is used to remove redundant video behavior sequence images, and extracts representative video behavior sequence images from the discontinuous segments of the removed video behavior sequence images; WiFi signal data processing is used to denoise and reduce the dimensionality of WiFi signal data, generate a channel state information correlation coefficient matrix, and reconstruct unique WiFi signal data; specifically as follows: The WiFi signal data is processed by extracting the original amplitude and phase information from the WiFi channel state information. An antenna ratio method is used to eliminate additional time-varying random phase shifts between the WiFi signal transmitter and receiver. A Hampel filter is used to remove noise and outliers caused by equipment defects and the surrounding environment. After obtaining the purified WiFi signal data, PCA is used for dimensionality reduction to filter redundant information caused by high similarity between subcarriers of the same antenna pair. The WiFi signal data is further reconstructed by dividing it into M WiFi channel state information subsequences along the time dimension. For each WiFi channel state information subsequence, the correlation coefficient matrix between different antenna pairs is calculated and concatenated to obtain the channel state information correlation coefficient matrix. The obtained channel state information correlation coefficient matrix is ​​multiplied by its transpose to obtain the relationship between the M WiFi channel state information subsequences, resulting in a uniquely reconstructed WiFi signal data C. The video behavior sequence image is formed by dividing the original video behavior sequence image corresponding to WiFi signal data into N video behavior sub-sequences of the same length according to the number of frames, and randomly extracting a sub-image from each video behavior sub-sequence to form a behavior sequence image sample V. Step 3: Feature extraction and fusion of multimodal sequence data; Multimodal sequence data feature extraction and fusion includes a low-level implicit feature extraction module and a high-level hierarchical cross-fusion module. The low-level implicit feature extraction module uses a deep learning network to obtain low-level multimodal implicit features from representative video behavior sequence images and reconstructed WiFi signal data, including video behavior sequence image features and WiFi signal data features. Then, the high-level hierarchical cross-fusion module achieves complementary fusion of the two modal features from the video behavior sequence images and WiFi signal data to obtain the fused features. Specifically: In low-level multimodal implicit feature extraction, neural learning networks include a video feature extraction network (3D CNN) and a spatiotemporal network (DTNet). The 3D CNN is used to extract the RGB features of the action sequence image samples V, thus obtaining the low-level features of the video action sequence images. By extracting the correlation features of the reconstructed WiFi signal data C using the spatiotemporal network DTNet, which is composed of dilated convolution and temporal convolution, the low-level features of the WiFi signal data are obtained. This allows us to obtain low-level multimodal implicit features; In high-level multimodal cross-fusion, a Transformer-based hierarchical cross-fusion approach is used to extract high-level features from low-level multimodal implicit features while fusing high-level features from both video behavior sequence images and WiFi signal data to obtain fused features. , , , This represents a Transformer operation. The contact operation represents the modal dimension. Represents convolution and pooling operations. Indicates the process and the number of operations of the Transformer; Step 4: Multimodal loss optimization and classification; Multimodal loss optimization and classification is used to integrate the loss of video action sequence image features, the loss of WiFi data signal features, and the fusion feature loss to determine the multimodal loss. The multimodal loss is then used to optimize the deep learning network in multimodal sequence data feature extraction and fusion, and the action classification result is output. Specifically as follows: Loss on image features of video action sequences Loss of WiFi signal data characteristics and multimodal fusion feature loss The data is integrated to determine the multimodal loss optimization and classification function. Through multimodal loss optimization and classification function A deep learning network for optimizing feature extraction and fusion of multimodal sequence data is developed to output behavior classification results.

2. The multimodal hierarchical cross-fusion recognition method based on vision and WiFi according to claim 1, characterized in that, The WiFi signal data acquisition device includes a router as the transmitter and a computer as the receiver, which are placed separately in the room. The transmitter is used to send WiFi signals carrying channel state information, and the receiver is used to receive superimposed WiFi signal data caused by human actions in the environment. The video data acquisition device is a camera installed at the WiFi signal receiver to capture images of the action sequence that is being performed and corresponds to the WiFi signal data.

Images