Multi-modal human body activity identification method and device based on WIFI and video

Abstract

The invention relates to a multi-mode human body activity identification method based on WIFI and videos, and the method comprises the steps: S1A, processing WIFI data, and obtaining WIFI space features; s1B, extracting human skeleton information in the video image to obtain visual features; s2A, performing spatial processing on the visual spatial features and the WIFI spatial features to obtain fused spatial features; s2B, performing time processing on the visual spatial features and the WIFI spatial features to obtain fusion time features; and S3, analyzing the fusion space feature and the fusion time feature to obtain a human body activity type. The multi-mode human body activity identification method based on the WIFI and the video has the advantages of being real-time and low in power consumption.

Description

Technical Field

[0001] The present invention relates to the field of human body recognition, and in particular to a multimodal human activity recognition method and device based on WIFI and video; Background Art

[0002] Human behavior recognition is primarily categorized into two technical approaches: vision-based and wireless signal-based. Vision-based solutions use a camera to record the area to be detected and analyze the spatial features of the captured images for human action recognition, as well as the temporal features of the images at different times. Combining these spatial and temporal features, they can identify the type of human behavior within the area. However, this vision-based approach is overly sensitive to light and performs poorly in low-light environments. Wireless signal-based solutions primarily utilize Wi-Fi signals, detecting changes in the amplitude and phase of Wi-Fi signals caused by human obstruction in the environment. Human behavior is identified by analyzing these changes in amplitude and phase at different times. While effective in low light, they struggle to distinguish multiple targets.

[0003] In order to combine the advantages of visual recognition and wireless signal recognition, decision fusion is usually used to perform multimodal fusion of visual and wireless signal information. The decision fusion recognition method is used to recognize the acquired WIFI CSI data and video data respectively. Figure 1 , Figure 1 The following is a flowchart of an existing multimodal fusion recognition method. The specific steps of the existing recognition method include:

[0004] S10A: performing data preprocessing on the collected channel state information to obtain preprocessed Wi-Fi signal state information;

[0005] S10B: preprocessing the collected data to obtain preprocessed video data;

[0006] S20A: Extract features from the pre-processed WIFI channel state information to obtain WIFI features;

[0007] S20B: performing feature extraction on the preprocessed video data to obtain visual features;

[0008] S30A: Identify the WIFI feature and obtain a WIFI identification output result;

[0009] S30B: Identify the video features and obtain a video recognition output result;

[0010] S40: performing decision fusion on the WIFI recognition output result and the video recognition output result to obtain a behavior recognition result.

[0011] Although decision fusion can combine the advantages of Wi-Fi recognition and visual recognition, it requires two independent processing units to perform complete behavioral recognition on visual signals and wireless signals respectively. This results in the consumption of a large amount of computing resources when performing large-scale multimodal fusion recognition, which increases the power consumption of the processing device; it also leads to longer processing time, a long interval between detection and obtaining results, and a lack of real-time performance. Summary of the Invention

[0012] Based on this, the purpose of the present invention is to achieve real-time and low power consumption of multimodal fusion of vision and WIFI, and to provide a multimodal human activity recognition method based on WIFI and video.

[0013] A multimodal human activity recognition method based on Wi-Fi and video includes S1A: processing Wi-Fi data to obtain Wi-Fi spatial features; S1B: extracting human skeleton information in video images to obtain visual features; S2A: spatially processing the visual spatial features and Wi-Fi spatial features to obtain fused spatial features; S2B: temporally processing the visual spatial features and Wi-Fi spatial features to obtain fused temporal features; and S3: analyzing the fused spatial features and fused temporal features to obtain human activity types.

[0014] Compared with the existing technology, the multimodal human activity recognition method based on Wi-Fi and video described in the present invention identifies the human activity type by fusing visual spatial features and Wi-Fi spatial features. This eliminates the need to perform separate calculations on visual and Wi-Fi features to obtain two recognition results, effectively reducing the amount of data required for calculation, thereby reducing computing time and energy consumption. At the same time, it can fully utilize the complementarity between modalities to improve recognition accuracy, achieving the high accuracy of multimodal fusion while also having the advantages of high real-time performance and low energy consumption.

[0015] Furthermore, step S2A specifically includes:

[0016] S2A1a: Perform average pooling on the Wi-Fi spatial features to obtain Wi-Fi global features:

[0017] S2A1b: Perform average pooling on the visual spatial features to obtain the visual global features;

[0018] S2A2: Process the visual global features and the Wi-Fi global features using a softmax function to obtain attention weights;

[0019] S2A3: Perform Conv1×1 convolution on the Wi-Fi spatial features, and use the attention weight to weight the convolved Wi-Fi spatial features. Then, add them to the spatial features through residual connection to obtain fused spatial features. The formula is:

[0020] C WV =Conv1x1(Att×C W )+C W

[0021] Among them, C WV is the fusion spatial feature, Att is the attention weight, C W It is the WIFI space feature.

[0022] Furthermore, step S2B specifically includes:

[0023] S2B1a: Performing time convolution on the Wi-Fi feature to obtain a Wi-Fi time feature;

[0024] S2B1b: performing temporal convolution on the visual features to obtain visual temporal features;

[0025] S2B2: Concatenate the Wi-Fi time feature and the visual time feature to obtain a fusion feature;

[0026] S2B3: Perform 1D convolution on the fused features to obtain a query matrix, a key matrix, and a value matrix;

[0027] S2B4: Use the softmax function to process the query matrix, key matrix and value matrix and fuse them with the above.

[0028] Furthermore, the temporal convolution in step S2B1a and step S2B2b is to perform dilated convolution with a convolution kernel of 3×3 and a dilation rate (Dilation) of 1 and dilated convolution with a convolution kernel of 3×3 and a dilation rate of 2 on the input data in sequence.

[0029] Furthermore, in step S1A, WIFI features are extracted from the WIFI information, and moving inverted bottleneck convolution and universal inverted bottleneck convolution are sequentially performed on the WIFI features to obtain the WIFI spatial features; in step S1B, video features are extracted from the video image, and moving inverted bottleneck convolution and universal inverted bottleneck convolution are sequentially performed on the video features to obtain the video spatial features.

[0030] Furthermore, the moving inverted bottleneck convolution performs Conv1×1 convolution, depthwise separable convolution with a dilation factor of 3 and a convolution kernel size of 5, and Conv1×1 convolution on the input data in sequence, and then concatenates the results of two consecutive Conv1×1 convolutions with the WIFI features or visual features to obtain the convolution result of the moving inverted bottleneck convolution.

[0031] Furthermore, the universal inverted bottleneck convolution first performs depthwise separable convolution with a convolution kernel size of 5, Conv1×1 convolution, depthwise separable convolution with a convolution kernel size of 3, and Conv1×1 convolution on the input data in sequence, and performs a first splicing on the processing result with the input WIFI features and visual features; then, a depthwise separable convolution with a convolution kernel size of 3 and two Conv1×1 convolutions are performed on the result obtained after the first splicing, and the result is spliced ​​for a second time with the result obtained after the first splicing to obtain the convolution result of the universal inverted bottleneck convolution.

[0032] Based on the same inventive concept, the present invention also provides a multimodal human activity recognition system based on WIFI and video, which includes a WIFI transmitter that transmits WIFI signals to the surrounding environment; a WIFI receiver that receives the WIFI signals in the surrounding environment; a camera that captures video images in the environment; and a human activity recognition device that receives the WIFI signals received by the WIFI receiver and the video images captured by the camera, and identifies the type of human activity in the environment based on information from the WIFI signals and the video images.

[0033] Based on the same inventive concept, the present invention also provides a multimodal human activity recognition device based on WIFI and video, which includes a WIFI data processing unit for processing the WIFI data to obtain WIFI spatial features; a video image processing unit for extracting human skeleton information in the video image to obtain visual features; a spatial feature fusion unit for performing spatial processing on the visual spatial features and WIFI spatial features to obtain fused spatial features; a temporal feature fusion unit for performing temporal processing on the visual spatial features and WIFI spatial features to obtain fused temporal features; and an analysis unit for analyzing the fused spatial features and fused temporal features to obtain human activity types.

[0034] Based on the same inventive concept, the present invention further provides an electronic device comprising a processor; a memory for storing a computer program executed by the processor; wherein the processor implements any of the above-mentioned control methods for improving the life of a projector liquid crystal screen when executing the computer program;

[0035] In order to better understand and implement the present invention, the following is a detailed description of the present invention with reference to the accompanying drawings; BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 Flowchart of the existing multimodal fusion recognition method;

[0037] Figure 2 is a schematic diagram of an identification system of the present invention;

[0038] Figure 3 is a module diagram of the human activity recognition device of the present invention;

[0039] Figure 4 Flowchart of the method executed by the human activity recognition device

[0040] Figure 5 Waveform diagrams of the WIFI amplitude data before and after filtering;

[0041] Figure 6 This is a structural diagram of the first moving inverted bottleneck convolution model;

[0042] Figure 7 is a structural diagram of the first universal inverted bottleneck unit;

[0043] Figure 8 A schematic diagram of extracting and superimposing human skeleton information on frame images;

[0044] Figure 9 Schematic diagram of the lightweight cross-attention model;

[0045] Figure 10 This is the structural diagram of the first time convolution model. DETAILED DESCRIPTION

[0046] Current multimodal fusion methods for human activity recognition perform complete temporal and spatial processing on visual features and Wi-Fi features, respectively, to obtain visual and Wi-Fi recognition results. This consumes enormous computing resources, resulting in long processing times, a lack of real-time performance, and high power consumption. To address the issues of long processing times and high power consumption in the prior art, the present invention first fuses visual spatial features and Wi-Fi spatial features, and then uses the fused features to identify the type of human activity. Furthermore, to improve the utilization of visual and Wi-Fi features during the fusion process, the present invention also optimizes the steps in the multimodal fusion process.

[0047] See also Figure 2 , Figure 2The figure is a schematic diagram of a multimodal human activity recognition system based on Wi-Fi and video according to the present invention. The human activity recognition system according to the present invention includes a Wi-Fi transmitter, a Wi-Fi receiver, a camera, and a human activity recognition device. The Wi-Fi transmitter transmits a Wi-Fi signal into the surrounding environment, which is received by the Wi-Fi receiver. The Wi-Fi signal information consists of amplitude and phase information. In a wireless communication system based on orthogonal frequency division multiplexing and multiple-input multiple-output technology, the channel bandwidth of the Wi-Fi signal is divided into multiple subcarriers in different orthogonal narrowband bands based on orthogonal frequency division multiplexing technology, each of which independently transmits data. Each subcarrier has independent amplitude and phase information. The channel state information (CSI) of the Wi-Fi signal includes the amplitude and phase information of each subcarrier. The Wi-Fi signal will contact the human body in the environment, causing reflection, scattering, or fading, causing its phase and amplitude information to change, thereby reflecting the human body's posture. The Wi-Fi receiver receives the Wi-Fi signal containing the superposition of each subcarrier in the surrounding environment and transmits the relevant data of the Wi-Fi information to the human activity recognition device. At the same time, the camera shoots the area to be detected in the surrounding environment and transmits the captured video image to the human activity recognition device.

[0048] See also Figure 3 and Figure 4 , Figure 3 is a module diagram of the human activity recognition device, Figure 4 Flowchart of the method executed by the human activity recognition device. The human activity recognition device includes a WIFI information processing unit, a video image processing unit, a spatial feature fusion unit, a temporal feature fusion unit and an analysis unit.

[0049] The WIFI data processing unit executes step S1A: processing the WIFI data to obtain WIFI spatial features. Specifically, the WIFI data processing unit includes a WIFI amplitude extraction module M1, a WIFI filtering module M2, a first moving inverted bottleneck convolution model M3, and a first general inverted bottleneck convolution model M4.

[0050] The WIFI amplitude extraction module M1 executes step S1A1: extracting WIFI amplitude data from the WIFI information. The amplitude information in the WIFI signal is more stable than the phase information and is less affected by the environment. The recognition result obtained using the WIFI amplitude data is more accurate and stable.

[0051] The WIFI filtering module M2 executes step S1A2: performing discrete wavelet transform on the WIFI amplitude data to obtain WIFI features. Although the amplitude information in the WIFI information is more stable than the phase information, it is still affected by environmental noise and device interference. Therefore, the present invention uses discrete wavelet transform (DWT) to reduce noise on the WIFI data to improve data quality. Figure 5 , Figure 5 The waveform diagrams of the WIFI amplitude data before and after filtering are shown. In the waveform diagram, the horizontal axis is the number of packets, the vertical axis is the amplitude, and different curves represent different subcarriers; Figure 5 The left side shows the waveform of the Wi-Fi amplitude data before filtering, and the right side shows the waveform of the Wi-Fi feature after filtering. It can be clearly seen that the Wi-Fi amplitude data after discrete wavelet transform is less cluttered and of higher quality. By decomposing the signal into different frequency components and convolving the signal with a wavelet function, the discrete wavelet transform can not only analyze the signal details at multiple scales, but also capture the signal characteristics in the time and frequency domains. The calculation formula of the discrete wavelet transform is:

[0052]

[0053] Wherein, j is the scale factor, k is the shift factor, f(t) is the signal to be wavelet transformed, i.e., the WIFI signal, and ψ is the Daubechies wavelet basis function. The formula of the wavelet basis function is:

[0054]

[0055] Among them, j is the scale factor, k is the translation parameter, and t is the time variable.

[0056] The first moving inverted bottleneck convolution model M3 executes step S1A3: performing moving inverted bottleneck convolution on the WIFI feature to obtain the WIFI convolution feature. Figure 6 , Figure 6This is a structural diagram of the first moving inverted bottleneck convolution model. The first moving inverted bottleneck convolution model M3 is divided into a main branch and a shortcut branch. The convolution results of the main branch and the shortcut branch are spliced ​​to obtain the convolution result of the moving inverted bottleneck convolution model. Among them, the main branch includes 5 convolution layers, the shortcut branch includes 2 convolution layers, and the main branch and the shortcut branch share the same initial convolution layer. The initial convolution layer performs Conv1×1 convolution, batch normalization (BN) processing, and rectified linear unit (Relu) processing on the input WIFI features in sequence to obtain the processing result of the initial convolution layer. The second convolution layer of the main branch performs depthwise separable convolution (DWConv) with a dilation factor of 3 and a convolution kernel size of 5, BN processing, and Relu processing on the processing result of the initial convolution layer to obtain the processing result of the second convolution layer. The structure of the third convolution layer is the same as that of the second convolution layer. The fourth convolutional layer sequentially performs Conv1×1 convolution and BN processing on the convolution results of the third convolutional layer to obtain the processing results of the third convolutional layer. The structure of the fifth convolutional layer is the same as that of the fourth convolutional layer. The second convolutional layer of the shortcut branch uses the processing results of the initial convolutional layer as input. Its structure is the same as that of the fifth convolutional layer of the main branch. Its processing results are concatenated with the processing results of the fifth convolutional layer of the main branch as the convolution results of the moving inverted bottleneck convolution model. In this embodiment, the WiFi convolution features are obtained by performing five consecutive moving inverted bottleneck convolutions on the WiFi features.

[0057] The first universal inverted bottleneck convolution model M4 executes step S1A4: performing universal inverted bottleneck convolution on the WIFI convolution feature to obtain the WIFI spatial feature. Figure 7 , Figure 7 This is a structural diagram of the first universal inverted bottleneck model. The first universal inverted bottleneck convolution model M4 includes an Extra DW part and an FFN part, wherein the Extra DW part includes four convolutional layers, the FFN part includes three convolutional layers, and the FFN part processes the results of the Extra DW part as input.

[0058] The first convolutional layer of the Extra DW portion sequentially performs a depthwise separable convolution with a convolution kernel size of 5 and a batch normalization (BN) process on the input Wi-Fi convolution features to obtain the processing result. The second convolutional layer sequentially performs a Conv1×1 convolution, a BN process, and a Relu process on the convolution result of the first convolutional layer. The third convolutional layer sequentially performs a depthwise separable convolution with a convolution kernel size of 3 and a BN process on the convolution result of the second convolutional layer. The fourth convolutional layer sequentially performs a Conv1×1 convolution, a BN process, and a Relu process on the convolution result of the third convolutional layer. The processing result of the fourth convolutional layer is concatenated with the input data of the first convolutional layer to obtain the convolution result of the Extra DW portion.

[0059] The first convolutional layer of the FFN part sequentially performs a depthwise separable convolution with a convolution kernel size of 3 and a batch normalization (BN) process on the processing results of the Extra DW part to obtain the processing results. The second convolutional layer of the FFN part sequentially performs a Conv1×1 convolution, a BN process, and a ReLU process on the convolution results of the first convolutional layer of the FFN part. The third convolutional layer of the FFN part sequentially performs a Conv1×1 convolution and a BN process on the processing results of the second convolutional layer of the FFN part. The processing results are then concatenated with the convolution results of the Extra DW part to obtain the convolution results of the FFN part, which are the Wi-Fi spatial features.

[0060] The Conv1×1 convolution uses a kernel of size 1 to perform convolution, compressing the number of channels while preserving the size. Batch normalization calculates the mean and variance of each feature map and adjusts the input data distribution to a standard normal distribution with a mean of 0 and a variance of 1. The rectified linear unit is an activation function used to introduce nonlinearity to avoid overfitting.

[0061] The video image processing unit executes step S1B: extracting human skeleton information from the video image to obtain visual features. Specifically, the video image processing unit includes a frame image extraction module M5, a visual feature extraction module M6, a second moving inverted bottleneck convolution model M7, and a second universal inverted bottleneck convolution model M8.

[0062] The frame image extraction module M5 executes step S1B1: extracting frame images from the video. The camera continuously captures multiple frame images at a constant rate every second. For example, for a 15-frame video, the recording device captures 15 frames every second. In this case, frame images are extracted from the video at regular intervals, reducing the data volume while ensuring detection accuracy. In this embodiment, the last frame is extracted every one-third of a second, meaning three frames are extracted every second, ensuring detection accuracy while reducing data volume.

[0063] The visual feature extraction module M6 executes step S1B2: extracting the human skeleton information from each frame image and superimposing it on an image to obtain visual features. The human skeleton key point detection technology is used in the extraction of the skeleton information, which is divided into two methods: bottom-up and top-down. Among them, the bottom-up method detects the joints of the human body in the frame image and connects the joints with lines to obtain the human skeleton information; the top-down method locates the human body boundary box and then identifies the joints of the individuals in the boundary box. Please refer to Figure 8 , Figure 8 The figure is a schematic diagram of extracting and superimposing human skeleton information on frame images. Skeletal key point detection technology is used to extract skeleton information of the human body in each frame image, and the human body posture is represented by lines to obtain human skeleton information composed of lines. The human skeleton information composed of lines in each frame image is then superimposed on an image with a black background to obtain a skeleton image containing multiple human skeleton information as visual features. In this embodiment, the bottom-up OpenPose model is used to perform skeleton key point detection to obtain the human skeleton information in the frame image to improve processing speed and parallel processing capabilities.

[0064] The second moving inverted bottleneck convolution model M7 executes step S1B3: performing a moving inverted bottleneck convolution on the visual features to obtain visual convolution features. The second moving inverted bottleneck convolution model M7 has the same structure as the first moving inverted bottleneck convolution model M3. The moving inverted bottleneck convolution process in step S1B3 is the same as that in step S1A3.

[0065] The second universal inverted bottleneck convolution model M8 executes step S1B4: performing universal inverted bottleneck convolution on the visual features to obtain visual spatial features. The second universal inverted bottleneck convolution model M8 has the same structure as the first universal inverted bottleneck convolution model M4. The universal inverted bottleneck convolution process in step S1B4 is the same as that in step S1A4.

[0066] The spatial feature fusion unit performs step S2A: spatially processing the visual spatial features and the Wi-Fi spatial features to obtain fused spatial features. The spatial feature fusion unit is a lightweight cross-attention model, which includes a pooling layer M9, a softmax layer M10, and a spatial feature fusion layer M11.

[0067] See also Figure 9 , Figure 9 The pooling layer M9 simultaneously performs step S2A1a: performing average pooling on the Wi-Fi spatial features to obtain Wi-Fi global features; and step S2A1b: performing average pooling on the visual spatial features to obtain visual global features.

[0068] The average pooling method traverses the image of each channel in the image, adds the values ​​of each pixel in the image of the channel and divides it by the total number of pixels to obtain the average value of the channel. At this time, each channel generates a single value. The formula for average pooling the visual spatial features is:

[0069]

[0070] Among them, Q V is the visual global feature, H is the height of the image, W is the width of the image, C V (i, j) is the coordinate of each pixel in the channel image.

[0071] The WIFI spatial feature C W The formula for average pooling is:

[0072]

[0073] Among them, K W is the WIFI global feature, H is the height of the image, W is the width of the image, C w (i, j) are the coordinates of each pixel in the channel image. By performing average pooling on the Wi-Fi spatial features, the computational complexity is significantly reduced while ensuring that global context information is effectively captured.

[0074] The softmax layer M10 performs step S2A2: using the softmax function to process the visual global features and the WIFI global features to obtain the attention weight. The specific formula of the softmax function is:

[0075] Att=Softmax(Q V ×K W )

[0076] Among them, Att is the attention weight, Q V is the visual global feature, K W It is a global feature of Wi-Fi.

[0077] The softmax function converts a real number vector into a probability distribution. It converts each element in the input vector into a positive number through an exponential operation, and then divides it by the sum of the exponential operations of all elements to obtain a new vector in which each element has a value range between (0, 1) and the sum is 1.

[0078] The spatial feature fusion layer M11 executes step S2A3: Conv1×1 convolution is performed on the WIFI spatial feature, and the convolutional WIFI spatial feature is weighted using the attention weight, and then added to the spatial feature through residual connection to obtain a fused spatial feature. The weighted feature is processed using Conv1×1 convolution and added to the spatial feature C through residual connection. W The addition formula is:

[0079] C WV =Conv1x1(Att×C W )+C W

[0080] Among them, C WV is the fusion spatial feature, Att is the attention weight, C W It is the WIFI space feature.

[0081] The temporal feature fusion module executes step S2B: performing temporal processing on the visual spatial features and the WiFi spatial features to obtain a fused temporal feature. Furthermore, the temporal feature fusion unit includes a temporal convolution model M12, a temporal feature splicing module M13, and a self-attention model.

[0082] The temporal convolution model M12 simultaneously performs step S2B1a: performing temporal convolution on the WIFI spatial features to obtain WIFI temporal features; and step S2B1b: performing temporal convolution on the visual features to obtain visual temporal features. Figure 10 , Figure 10The following is a structural diagram of the temporal convolution model. The temporal convolution model used includes two dilated convolution layers (Dilated Conv) and two intermediate processing layers with the same structure. The two dilated convolution layers are the first dilated convolution layer and the second dilated convolution layer. The first dilated convolution layer performs a dilated convolution with a convolution kernel of 3×3 and a dilation rate of 1 on the input WiFi spatial features. One intermediate processing layer sequentially performs BN, ReLU, and Dropout on the convolution results of the first dilated convolution layer. The second dilated convolution layer performs a dilated convolution with a convolution kernel of 3×3 and a dilation rate of 2. The other intermediate processing layer processes the convolution results of the second dilated convolution layer, and the resulting processing results are the WiFi temporal features. In this embodiment, the WiFi spatial features are first flattened and formatted to a B×C×S format that is more suitable for time series analysis.

[0083] Dropout is a regularization technique that reduces the risk of overfitting by randomly discarding the outputs of some neurons during neural network training. In this embodiment, the dropout probability of Dropout is set to 0.2.

[0084] The time feature splicing module M13 executes step S2B2: splicing the WiFi time feature and the visual time feature to obtain a fusion feature.

[0085] The self-attention model includes a 1D convolution layer M14 and a temporal feature fusion layer M15.

[0086] The 1D convolution layer M14 executes step S2B3: performing a 1D convolution on the fused features to obtain a query matrix, a key matrix, and a value matrix. This 1D convolution extracts local features by sliding a fixed-size convolution kernel over the input data, capturing spatial and temporal dependencies within the sequence. In this embodiment, the dimensions of the query and key matrices are reduced to one-eighth of the input dimension, reducing computational complexity.

[0087] The temporal feature fusion layer M15 executes step S2B4: using the softmax function to process the query matrix, key matrix, and value matrix, and performing a residual connection with the fused feature to obtain a fused temporal feature. The specific formula for obtaining the fused temporal feature using the softmax function combined with the residual connection is:

[0088]

[0089] Among them, T is the fusion feature, Q T is the query matrix, K T is the bond matrix, V T is the value matrix, dk is a scaling factor, and γ is a learnable factor used to adjust the output of the self-attention model. In this embodiment, the value of the learnable factor is 0.01.

[0090] The analysis unit performs step S3: analyzing the fused spatial features and the fused temporal features to obtain the human activity type. Specifically, the analysis unit includes a multimodal splicing module M16 and a fully connected layer M17.

[0091] The multimodal splicing module M16 executes step S31: splicing the fused spatial features and the fused temporal features to obtain multimodal fusion features.

[0092] The fully connected layer M17 executes step S32: analyzing the multimodal fusion features to identify the human activity type. This fully connected layer first performs batch normalization and activation function processing on the multimodal fusion features. It then uses adaptive average pooling to convert the processed multimodal fusion features to a 1×1 size and flattens them in length and width. Finally, the human activity type is identified.

[0093] The technical effects of the present invention are described below by experimental data:

[0094] In the experiment, a router with a frequency band of 2.4GHz was used as the WIFI transmission module M0, and the transmission WIFI signal propagating in space was collected at a data acquisition frequency of 100Hz. A network camera with a resolution of 640×480 was used to respectively obtain human movements in the detection area. The multimodal human activity recognition method based on WIFI and video of the present invention was used to process the video and WIFI signals. A normal lighting environment was simulated by turning on the lights, and a low-light environment was simulated by turning off the lights. The recognition accuracy was obtained by dividing the number of correctly predicted samples by the total number of predicted samples and multiplying by 100. The specific formula is as follows:

[0095]

[0096] Among them, TP is the number of samples correctly predicted as positive, TN is the number of samples correctly predicted as negative, FP is the number of samples incorrectly predicted as positive, and FN is the number of samples incorrectly predicted as negative.

[0097] In addition, the F1 value (F1-score) is used to measure the recognition effect. The F1 value is an indicator for comprehensively evaluating the performance of the classification model. Its formula is:

[0098]

[0099] Among them, Precision is the precision rate and Recall is the recall rate.

[0100] A recognition system using the method of the present invention achieved a recognition accuracy of 99.5% under good lighting conditions, with an F1 score of 99.48%. Furthermore, the method generated an average of 0.09M parameters during processing, an average number of floating-point operations (indicating computational complexity) of 20.55M, and an average inference time of 14.39ms.

[0101] Comparative Example

[0102] For the same experimental scenario, the recognition accuracy using WIFI unimodality under good lighting conditions was 96.27%, and the F1 value was only 96.12%, both lower than the recognition accuracy and F1 value of the multimodal human activity recognition method based on WIFI and video under the same conditions of the present invention. The recognition accuracy using visual unimodality under good lighting conditions was 99%, and the recognition accuracy under poor lighting conditions was only 98.88%, also lower than the recognition accuracy of the multimodal human activity recognition method based on WIFI and video under the same conditions of the present invention. Therefore, whether under good or poor lighting conditions, the recognition accuracy of human movements using this method is excellent.

[0103] In addition, the Gaitfi model and WiVi model, which use existing multimodal fusion, were used to process video data and Wi-Fi signals respectively to identify actions. The WiVi model uses decision fusion. The processing efficiency of each model is compared, and the specific efficiency is shown in the following table:

[0104] Model Average parameter quantity Average number of floating-point operations Average inference time Model of the present invention 0.09M 20.55M 14.39ms Gaitfi Model 0.22M 25.23M 14.85ms WiVi Model 82.81M 66806.42M 688.14ms

[0105] As can be seen from the table, the average parameter amount of the Gaitfi model is 0.22M, the average number of floating-point operations is 25.23M, and the average inference time is 14.85ms. The average parameter amount processed by the WiVi model is 82.81M, the average number of floating-point operations is 66806.42M, and the average inference time is 688.14ms. Therefore, when using this method for action recognition, the average parameter amount of 0.09M generated is much smaller than the existing Gaitfi model's 0.22M and the WiVi model's 82.81M. The required average number of floating-point operations of 20.55M is also smaller than the existing Gaitfi model's 25.23M and much smaller than the WiVi model's 66806.42M; the required average inference time of 14.39ms is less than the existing Gaitfi model's 14.85ms and much smaller than the WiVi model's 688.14ms. Therefore, the method of the present invention is superior to the existing WiVi model and Gaitfi model in terms of parameter quantity, number of floating-point operations and inference time, and has excellent real-time performance and low power consumption.

[0106] Compared with the existing technology, the multimodal human activity recognition method based on Wi-Fi and video described in the present invention fuses visual spatial features and Wi-Fi spatial features before recognition, eliminating the need to perform full calculations on both features separately. This can effectively reduce the amount of data that needs to be calculated, thereby reducing computing time and energy consumption. It has the advantages of high accuracy of multimodal fusion while also being highly real-time and low energy consumption.

[0107] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to limit the embodiments of the present application. The singular forms of "a", "said" and "the" used in the embodiments of the present application and the claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that, unless otherwise specified, "multiple" refers to two or more; the terms "first", "second", "third", etc. are only used to distinguish, and are not used to describe a specific order or sequence, nor can they be understood to indicate or imply relative importance. The term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items. When the above description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. In the description of the present application, for those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances.

[0108] The above-described embodiments merely represent several implementations of the present invention. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make numerous modifications and improvements without departing from the spirit of the present invention, and the present invention is intended to encompass such modifications and variations.

Claims

1. A multimodal human activity recognition method based on WIFI and video, characterized in that: include: S1A: Process the Wi-Fi data to obtain Wi-Fi spatial features; S1B: Extract human skeleton information from video images to obtain visual features; S2A: performing spatial processing on the visual spatial features and the WIFI spatial features to obtain fused spatial features; S2B: performing temporal processing on the visual spatial features and the WiFi spatial features to obtain a fused temporal feature; S3: Analyze the fused spatial features and fused temporal features to obtain the human activity type.

2. The multimodal human activity recognition method based on WIFI and video according to claim 1, characterized in that: Step S2A specifically includes: S2A1a: Perform average pooling on the Wi-Fi spatial features to obtain Wi-Fi global features: S2A1b: Perform average pooling on the visual spatial features to obtain the visual global features; S2A2: Process the visual global features and the Wi-Fi global features using a softmax function to obtain attention weights; S2A3: Perform Conv1×1 convolution on the Wi-Fi spatial features, and use the attention weight to weight the convolved Wi-Fi spatial features. Then, add them to the spatial features through residual connection to obtain fused spatial features. The formula is: C VV =Conv1x1(Att×C W )+C W Among them, C WV is the fusion spatial feature, Att is the attention weight, C W It is the WIFI space feature.

3. The multimodal human activity recognition method based on WIFI and video according to claim 1, characterized in that: Step S2B specifically includes: S2B1a: Performing time convolution on the Wi-Fi feature to obtain a Wi-Fi time feature; S2B1b: performing temporal convolution on the visual features to obtain visual temporal features; S2B2: Concatenate the Wi-Fi time feature and the visual time feature to obtain a fusion feature; S2B3: Perform 1D convolution on the fused features to obtain a query matrix, a key matrix, and a value matrix; S2B4: Use the softmax function to process the query matrix, key matrix and value matrix, and perform residual connection with the fusion feature to obtain the fusion time feature. The specific formula is: Among them, T is the fusion feature, Q T is the query matrix, K T is the bond matrix, V T is the value matrix, d k is a scaling factor and γ is a learnable factor used to adjust the output of the self-attention model.

4. The multimodal human activity recognition method based on WIFI and video according to claim 3 is characterized in that: The temporal convolution in step S2B1a and step S2B2b is to perform dilated convolution with a convolution kernel of 3×3 and a dilation rate of 1 and dilated convolution with a convolution kernel of 3×3 and a dilation rate of 2 on the input data in sequence.

5. The multimodal human activity recognition method based on WIFI and video according to claim 1, characterized in that: In step S1A, extracting WIFI features from the WIFI information, and sequentially performing moving inverted bottleneck convolution and universal inverted bottleneck convolution on the WIFI features to obtain the WIFI spatial features; In step S1B, video features are extracted from the video image, and moving inverted bottleneck convolution and universal inverted bottleneck convolution are sequentially performed on the video features to obtain the video spatial features.

6. The multimodal human activity recognition method based on WIFI and video according to claim 5, characterized in that: The moving inverted bottleneck convolution performs Conv1×1 convolution, depthwise separable convolution with a dilation factor of 3 and a convolution kernel size of 5, and Conv1×1 convolution on the input data in sequence, and then concatenates the results of two consecutive Conv1×1 convolutions with the WiFi feature or visual feature to obtain the convolution result of the moving inverted bottleneck convolution.

7. The multimodal human activity recognition method based on WIFI and video according to claim 5, characterized in that: The universal inverted bottleneck convolution first performs depth-wise separable convolution with a convolution kernel size of 5, Conv1×1 convolution, depth-wise separable convolution with a convolution kernel size of 3, and Conv1×1 convolution on the input data in sequence, and performs a first splicing on the processing results with the input WIFI features and visual features; then, a depth-wise separable convolution with a convolution kernel size of 3 and two Conv1×1 convolutions are performed on the result obtained after the first splicing, and the result obtained after the first splicing is spliced ​​for a second time to obtain the convolution result of the universal inverted bottleneck convolution.

8. A multimodal human activity recognition system based on WIFI and video, characterized in that: include: A WIFI transmitter transmits WIFI signals to the surrounding environment; A WIFI receiver, which receives the WIFI signal in the surrounding environment; a camera that captures video images of the surrounding environment; A human activity recognition device receives the WIFI signal received by the WIFI receiver and the video image captured by the camera, and recognizes the type of human activity in the environment based on the information of the WIFI signal and the video image.

9. A multimodal human activity recognition device based on WIFI and video, characterized in that: include A WIFI data processing unit, configured to process the WIFI data to obtain WIFI spatial features; A video image processing unit, used to extract human skeleton information from video images to obtain visual features; A spatial feature fusion unit, configured to perform spatial processing on the visual spatial features and the WIFI spatial features to obtain a fused spatial feature; A temporal feature fusion unit, configured to perform temporal processing on the visual spatial features and the WiFi spatial features to obtain a fused temporal feature; The analysis unit is used to analyze the fused spatial features and the fused temporal features to obtain the human activity type.

10. An electronic device, characterized in that: include: processor; a memory for storing a computer program executed by the processor; When the processor executes the computer program, the multimodal human activity recognition method based on WIFI and video is implemented as described in any one of claims 1 to 8.

Images