Multi-modal sensing method based on wifi and vision

Abstract

The invention relates to an environment perception technology, and discloses a multi-mode perception method based on wifi and vision, which realizes accurate perception of an environment under different illumination / visual conditions. According to the method, when a multi-mode sensing model is trained, image data and CSI data of a sensed space are obtained at the same time, and preprocessing and time alignment are carried out on the data; then, image features of the image data are extracted, action recognition is carried out, and a visual perception result is obtained; taking the visual perception result as priori knowledge of the CSI data, inputting the visual perception result and the CSI data into a CSI data processing module, and extracting CSI features; then, carrying out feature fusion on the image features and the CSI features to obtain fusion features; and according to the fusion feature line classification, obtaining a predicted environment perception result. And finally, calculating loss through the predicted environment perception result and a pre-labeled environment perception result, and training a multi-modal perception model by adopting a back propagation method. The method is suitable for sensing the user action state in the environment.

Description

Technical Field

[0001] The present invention relates to environment perception technology, and in particular to a multimodal perception method based on Wi-Fi and vision. Background Art

[0002] Environmental perception refers to the collection of data in the environment through a series of sensors (such as temperature sensors, humidity sensors, visual sensors, etc.), and the analysis and effective information extraction to understand the situation in the environment. At present, visual perception is still the most mainstream in environmental perception. With the continuous development of technologies such as deep learning and computer graphics, the processing power, recognition accuracy and speed of visual perception have been greatly improved. However, visual sensors are highly dependent on lighting conditions and visual conditions. In addition, there are visual blind spots. Therefore, using only visual sensors for environmental perception has certain limitations.

[0003] On the other hand, wireless perception can use the channel characteristic information of RF signals when propagating in space combined with some algorithms (such as time of flight TOF, angle of arrival / angle of departure AOA / AOD, etc.) to perceive and identify objects. It has a large perception range and is insensitive to lighting conditions and visual conditions. It can be used as a supplement when visual perception conditions are limited. However, the various perception models based on wireless perception are currently not accurate enough in actual perception results due to the lack of high-quality data sets and non-standard standards. Summary of the invention

[0004] The technical problem to be solved by the present invention is to propose a multimodal perception method based on wifi and vision to achieve accurate perception of the environment under different lighting / visual conditions.

[0005] The technical solution adopted by the present invention to solve the above technical problems is: A multimodal perception method based on wifi and vision is applied to a perceived space, wherein a visual sensor and multiple AP devices are deployed in the perceived space; the multiple AP devices are networked; the multimodal perception method comprises: A. Training multimodal perception model: The multimodal perception model includes a visual perception module, a CSI (Channel State Information) data processing module, a feature fusion module and a classifier module; the training process includes: A1. Use the visual sensor and AP device to simultaneously obtain the image data and CSI data of the perceived space; A2. Preprocess the acquired image data and CSI data; A3, aligning the preprocessed image data and CSI data; A4. Use the visual perception module to extract image features of image data and perform action recognition to obtain visual perception results; A5. Using the visual perception result as the prior knowledge of CSI data, and inputting it into the CSI data processing module together with the CSI data to extract CSI features; A6. Perform feature fusion using a feature fusion module according to the image features and the CSI features to obtain fusion features; A7. Classify using a classifier module according to the fusion features to obtain a predicted environmental perception result; A8. Calculate the loss based on the predicted environment perception results and the pre-labeled environment perception results, and use the back propagation method to train the multimodal perception model; B. Perform multimodal perception tasks: The image data and CSI data of the perceived space are collected at the same time, and after preprocessing and alignment, they are used as the input of the trained multimodal perception model to obtain the environmental perception results.

[0006] Further, in step A1, the CSI data is CSI data collected by the AP device when it is set in different ways in the sensed space; the different ways of setting include: Multiple APs are set in different frequency bands, or multiple APs are set in different spatial positions of the sensed space in different arrangements, or different numbers of transceiver devices are set in multiple APs to form different combinations.

[0007] Furthermore, in step A2, preprocessing the image data includes: after unifying the format and resolution of the image data, performing noise reduction, filtering, smoothing and normalization processing.

[0008] Furthermore, in step A2, preprocessing the CSI data includes: The CSI data obtained by AP devices set up in different ways are filtered, intensity aligned, resampled, time aligned and length-consistent.

[0009] Furthermore, in step A3, aligning the preprocessed image data and CSI data includes: The time point when the target object appears in the perceived space is taken as the alignment origin, and the preprocessed image data and CSI data are aligned in time series along the alignment origin.

[0010] Furthermore, in step A4, the visual perception module adopts a video action recognition model, including but not limited to: C3D, TSN, TSM or TSCNN.

[0011] Furthermore, in step A5, the CSI data processing module adopts a neural network, including but not limited to CNN, RNN, LSTM or Transformer.

[0012] Furthermore, in step A6, the feature fusion module adopts a cross attention mechanism to fuse the image features and the CSI features by calculating weights.

[0013] The beneficial effects of the present invention are: (1) After synchronously collecting image data and CSI data in the perceived space, a highly mature visual perception model is used to pre-classify the image data in the perceived space, thereby providing prior information for CSI data perception to guide the generation of CSI features and obtain more accurate feature expressions of the information, thereby improving the accuracy of the trained multimodal perception model.

[0014] (2) In terms of acquiring CSI sample data, multiple APs under networking conditions are set up in different frequency bands, in different spatial positions, and in different transceiver combinations to acquire CSI data collected by AP devices, thereby diversifying the sources of CSI data and obtaining richer and more effective CSI data. This provides data support for training accurate multimodal perception models and enables the model to achieve better prediction results in scenarios with different AP settings, thereby improving the generalization ability of the model.

[0015] (3) Based on the multimodal perception model trained by the present invention, the image features and CSI features extracted from the perceived space are weighted and fused through the attention mechanism, and the environmental perception results are predicted based on the fused features, thereby realizing the mutual complementation of the two modal perception networks. When the lighting / visual conditions are good, the visual perception network will be used as the main perception means, and when the lighting / visual conditions are poor, the CSI perception network will be used as the main perception means, thereby ensuring the performance of the entire model under various lighting / visual conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 Schematic diagram of the structure of a multimodal perception model in an embodiment of the present invention; Figure 2 This is a diagram of the multimodal perception model training process in an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The present invention aims to provide a multimodal perception method based on wifi and vision to achieve accurate perception of the environment under different lighting / visual conditions. The core idea is: visual perception can collect images in the perceived space through a camera, and extract and identify the action features of the images based on a visual perception network, so as to obtain the user action state in the space. However, visual perception is limited by lighting conditions and visual conditions, and the perception range is also limited by visual blind spots; wifi wireless perception is based on the channel state information (CSI) of the wifi signal. When the user moves on the propagation path of the wifi signal, the CSI information of the wifi signal, such as signal amplitude and phase, will produce slight changes. These subtle changes are captured by the AP device and submitted to the CSI perception network for analysis and processing, so as to predict the user action state in the space. In contrast, wireless perception does not rely on lighting and visual conditions, and has a larger perception range. Therefore, the present invention combines the perception of the two modes to achieve the perception of the user action state in the environment of the perceived space.

[0018] In specific implementation, the present invention synchronously collects image data and CSI data in the perceived space, and after preprocessing and time alignment processing, uses a highly mature visual perception model to extract features and pre-classify the image data in the perceived space, thereby obtaining visual perception results, and provides prior information for CSI data perception based on the visual perception results to guide the generation of CSI features, so as to obtain more accurate CSI feature expressions of information, and inputs image features and CSI features into the classifier for classification after calculating fusion features through the attention mechanism, and trains on the training data set in this way to obtain a trained multimodal perception model. In addition, in order to train a more generalized and more accurate model, in terms of the acquisition of CSI sample data, multiple APs under networking conditions are used to obtain CSI data collected by AP devices in multiple ways such as different frequency band settings, different spatial position arrangement settings, and different transceiver combination settings, so as to diversify the sources of CSI data, so as to obtain richer and more effective CSI data, and provide data support for training accurate multimodal perception models.

[0019] Based on the multimodal perception model trained by the present invention, when the lighting / visual conditions are good, the visual perception network will be used as the main perception means, and when the lighting / visual conditions are poor, the CSI perception network will be used as the main perception means, thereby ensuring the performance of the entire model under various lighting / visual conditions.

[0020] The scheme of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0021] Example Multi-link operation (MLO) is a key function in wifi 7 technology. Specifically, it allows devices to connect and use multiple frequency bands (such as 2.4GHz, 5GHz and 5.8GHz) for parallel transmission at the same time, and supports frequency band switching and load balancing according to network conditions, thereby improving the speed, efficiency and stability of the network. MLO meets the multi-AP conditions, which makes it possible to obtain diverse CSI data. Accordingly, the AP device in this embodiment can use a wifi 7 device, and the visual sensor uses a camera.

[0022] Deploy the camera and multiple AP devices in the same perception space and adjust the camera's internal parameters: Use calibration tools such as checkerboard to calibrate the camera's internal parameters to determine the camera's internal parameter matrix (focal length, optical center, distortion coefficient, etc.). In addition, multiple AP devices need to be meshed.

[0023] Based on the deployment of the above-mentioned hardware devices, in the multimodal perception method based on wifi and vision provided in this embodiment, the multimodal perception model is first trained. After obtaining the trained multimodal perception model, it can be deployed in actual application scenarios to perform perception tasks.

[0024] Among them, the architecture of the multimodal perception model is as follows Figure 1 As shown, it includes a visual perception module, a CSI data processing module, a feature fusion module and a classifier module; the visual perception module is used to perform feature extraction and action recognition on image data to obtain visual perception results, and specifically, a currently more mature video action recognition model can be used, including but not limited to: C3D, TSN, TSM or TSCNN, etc.; the CSI data processing module is used to use the visual perception results as prior knowledge, and extract CSI features by analyzing and processing CSI data. Specifically, a neural network can be used, including but not limited to CNN, RNN, LSTM or Transformer, etc.; the feature fusion module is used to fuse visual features and CSI features to generate fused features. Specifically, a cross-attention mechanism can be used to fuse image features and CSI features by calculating weights; the classifier module is used to classify according to the fused features to obtain predicted environmental perception results, and specifically, more mainstream multi-layer perceptron MLP, support vector machine SVM, etc. can be used.

[0025] The process of training the multimodal perception model in this embodiment can be found in Figure 2 , which mainly includes the following steps: S1, Image and CSI data acquisition In this step, the camera and AP device are used to simultaneously obtain the image data and CSI data of the sensed space. In order to obtain richer and more effective CSI data, this embodiment can use different settings to set the AP device, where the different settings include different frequency band settings, different spatial position settings, different combination settings, etc. The following are detailed descriptions: Different frequency band settings: multiple AP devices are set in different frequency bands, such as 2.4GHz / 5GHz / 5.8GHz, to collect CSI data based on different frequency bands and different subcarriers.

[0026] Different spatial position settings: that is, multiple AP devices are set in different physical positions in the sensed space in different arrangements, such as left-right arrangement, up-down arrangement, triangle arrangement, square arrangement, ring arrangement, etc., and CSI data is collected under these different spatial position conditions.

[0027] Different combination settings: that is, by changing the relative number of transceivers in multiple AP devices, different combinations such as one transmitter and multiple receivers, multiple transmitters and one receiver, and multiple transmitters and multiple receivers are formed, and CSI data is collected under these different combination conditions.

[0028] By using the above means, CSI data of multiple APs in different settings can be collected, thereby providing better data support for subsequent model training.

[0029] S2. Image and CSI data preprocessing In this step, the acquired image data and CSI data are preprocessed to improve data quality. The preprocessing methods for image data include: first unifying the data format to ensure the uniformity of resolution and format, then performing noise reduction, filtering, and smoothing to reduce noise interference and highlight key areas in the image; finally, the image needs to be normalized, scaling the pixel values ​​to between 0 and 1, or using mean-variance normalization, which helps to speed up the convergence of the algorithm and improve the accuracy and stability of the model.

[0030] The methods for preprocessing CSI data include: Filtering: CSI data is collected and calculated directly by the chip and then assembled in the QOS frame or beacon frame. Since there may be unstable voltage or current inside the chip, or the chip is subject to electromagnetic interference, the collected CSI data often has glitches or noise, affecting the data quality. Therefore, filtering is used to reduce glitches.

[0031] Intensity alignment: Since the signal strength of different subcarriers is affected by external environments such as the length of the propagation path, refraction, reflection, and obstruction, the intensity values ​​of different subcarriers vary greatly and are uncontrollable, affecting data quality. Therefore, it is necessary to align the intensity of CSI data. By aligning the signal strengths of different subcarriers to the same reference intensity in equal proportions, the data fluctuations caused by different subcarriers can be reduced while retaining the additional information brought by multiple subcarriers as much as possible.

[0032] Sampling processing: Since the frequency of CSI data collected by different devices may be inconsistent, the CSI data needs to be sampled. Linear interpolation or resampling can be used to change the frequency of CSI data to facilitate subsequent time alignment.

[0033] Time alignment: Since commands between different devices may be delayed or have performance differences, the time of CSI data collected by different devices may be inconsistent. It is necessary to calibrate the time of different data and align them to ensure the temporal and spatial consistency of different data. Usually, the time sensitivity of CSI data perception is not high, and the time span of a perception unit is about 5s. Therefore, sub-second time alignment can be performed based on the system's own timestamp.

[0034] Length consistency: The CSI data collected by multiple APs is processed consistently with the training sample data volume as the minimum unit to ensure that the number of samples collected by all APs is the same.

[0035] S3, image and CSI data alignment In this step, the pre-processed image data and CSI data are aligned. Due to the inherent differences between AP devices and cameras, as well as command delays or performance differences, the time of CSI data and image data will be inconsistent. The time of the two types of data needs to be calibrated and aligned to ensure the temporal and spatial consistency of different data. Based on the alignment of different CSI data in the aforementioned step S2, the image data can be calibrated, edited, and cropped through high-precision timestamps and various video software to achieve alignment with the CSI data and consistent length.

[0036] S4. Extract image features of image data and perform action recognition In this step, the visual perception module is used to extract the image features of the image data and perform action recognition to obtain visual perception results. The recognized visual perception results will be used as prior knowledge of the CSI network in the future, and participate in the screening of CSI data, the adjustment of network parameters, etc., to guide the generation of CSI features and optimize the training of the CSI network. The image features will be used for subsequent fusion with the CSI features.

[0037] S5. Combine prior knowledge to extract CSI features In this step, the visual perception result is used as the prior knowledge of the CSI data and is input into the CSI data processing module together with the CSI data to extract CSI features. Prior knowledge is used for CSI data screening. For example, the CSI network can select CSI data at the time point corresponding to the image frame with good visual classification effect or high confidence according to the visual perception result; in addition, the weight matrix of the CSI network can also be set according to the visual perception result to speed up the convergence efficiency of training.

[0038] S6. Image feature and CSI feature fusion In this step, according to the image features and CSI features, a feature fusion module is used to perform feature fusion to obtain fused features. For example, a cross attention mechanism is used to dynamically balance the weights of the two modal feature values ​​to calculate the fused feature value.

[0039] S7. Classification based on fusion features In this step, classification is performed using a classifier module based on the fusion features to obtain predicted environmental perception results.

[0040] S8. Calculate prediction loss and use back propagation to train the model In this step, the loss is calculated based on the predicted environmental perception results and the pre-labeled environmental perception results (i.e., the true value), and the multimodal perception model is trained using the back-propagation method until the training termination conditions are met, such as model convergence, or a certain number of iterative training rounds are reached, to obtain a trained multimodal perception model.

[0041] In practical applications, the multimodal perception model trained as above is deployed, and the image data and CSI data of the perceived space are collected simultaneously. After preprocessing and alignment, they are used as the input of the multimodal perception model to obtain the environmental perception results.

[0042] Finally, it should be noted that the above embodiments are only preferred implementations and are not intended to limit the present invention. It should be pointed out that for those skilled in the art, several modifications, equivalent replacements, improvements, etc. can be made without departing from the scope of the present invention and the scope of protection of the claims, and all of these should be included in the protection scope of the present invention.

Claims

1. A multimodal perception method based on wifi and vision, applied to a perceived space, wherein a visual sensor and a plurality of AP devices are deployed in the perceived space; the plurality of AP devices are networked; characterized in that: The multimodal sensing method includes: A. Training multimodal perception model: The multimodal perception model includes a visual perception module, a CSI data processing module, a feature fusion module and a classifier module; the training process includes: A1. Use the visual sensor and AP device to simultaneously obtain the image data and CSI data of the perceived space; A2. Preprocess the acquired image data and CSI data; A3, aligning the preprocessed image data and CSI data; A4. Use the visual perception module to extract image features of image data and perform action recognition to obtain visual perception results; A5. Using the visual perception result as the prior knowledge of CSI data, and inputting it into the CSI data processing module together with the CSI data to extract CSI features; A6. According to the image features and CSI features, a feature fusion module is used to perform feature fusion to obtain fusion features; A7. Classify using a classifier module according to the fusion features to obtain a predicted environmental perception result; A8. Calculate the loss based on the predicted environment perception results and the pre-labeled environment perception results, and use the back propagation method to train the multimodal perception model; B. Perform multimodal perception tasks: The image data and CSI data of the perceived space are collected at the same time, and after preprocessing and alignment, they are used as the input of the trained multimodal perception model to obtain the environmental perception results.

2. A multimodal perception method based on wifi and vision as claimed in claim 1, characterized in that: In step A1, the CSI data is CSI data collected when the AP device is set in different ways in the sensed space; The different configurations include: Multiple APs are set in different frequency bands, or multiple APs are set in different spatial positions of the sensed space in different arrangements, or different numbers of transceiver devices are set in multiple APs to form different combinations.

3. The multimodal perception method based on wifi and vision as claimed in claim 1, characterized in that: In step A2, preprocessing the image data includes: after unifying the format and resolution of the image data, performing noise reduction, filtering, smoothing and normalization processing.

4. The multimodal perception method based on wifi and vision as claimed in claim 2, characterized in that: In step A2, preprocessing the CSI data includes: The CSI data obtained by AP devices set up in different ways are filtered, intensity aligned, resampled, time aligned and length-consistent.

5. A multimodal perception method based on wifi and vision as claimed in claim 4, characterized in that: In step A3, aligning the preprocessed image data and CSI data includes: The time point when the target object appears in the perceived space is taken as the alignment origin, and the preprocessed image data and CSI data are aligned in time series along the alignment origin.

6. The multimodal perception method based on wifi and vision as claimed in claim 1, characterized in that: In step A4, the visual perception module adopts a video action recognition model, including: C3D, TSN, TSM or TSCNN.

7. The multimodal perception method based on wifi and vision as claimed in claim 1, characterized in that: In step A5, the CSI data processing module adopts a neural network, including: CNN, RNN, LSTM or Transformer.

8. A multimodal perception method based on wifi and vision as described in any one of claims 1 to 7, characterized in that: In step A6, the feature fusion module adopts a cross attention mechanism to fuse the image features and CSI features by calculating weights.

Images