A real-time human behavior recognition method based on cross-domain dual-modal feature fusion

Abstract

The application discloses a real-time human behavior recognition method based on cross-domain dual-mode feature fusion, and steps of the method comprise the following steps: 1, collecting CSI action sample data and video sample data; 2, pre-processing the collected data; 3, constructing a dual-thread action feature extraction network to extract action features in the CSI sample and the video sample; 4, constructing a feature fusion and feature alignment network to fuse the action features of the two modes; and 5, inputting the fused features into a classifier network to recognize human actions. The application realizes behavior recognition based on CSI and video dual-mode information, and has strong recognition accuracy and anti-interference capability.

Description

Technical Field

[0001] This invention relates to the fields of wireless communication technology and computer vision, specifically to a real-time human behavior recognition method that integrates cross-domain dual-modal feature fusion. Background Technology

[0002] With the development of science and technology, human behavior recognition has been widely applied in fields such as health management, video surveillance, and smart homes. Existing human behavior recognition technologies mainly include methods based on wearable sensor devices and methods based on computer vision, both of which have achieved good recognition results in their respective fields. However, wearable device-based human behavior recognition methods can, to some extent, hinder normal human behavior, and a complete set of motion sensing equipment typically requires high costs, making it unsuitable for large-scale human behavior recognition. While computer vision-based human behavior detection methods are relatively easy to deploy, their accuracy is affected by factors such as line-of-sight propagation and lighting intensity.

[0003] In recent years, action recognition based on Wi-Fi has attracted widespread attention. Wi-Fi signals are unaffected by line-of-sight propagation, and collecting human action samples using widely available commercial Wi-Fi devices is cost-effective. Human activity recognition based on Channel State Information (CSI) mainly includes Received Signal Strength Indication (RSSI) and Channel State Information (CSI). CSI is physical layer information of the wireless channel extracted from Wi-Fi devices, providing fine-grained action information and has been widely used in Wi-Fi-based action recognition research. However, due to the time-varying characteristics of wireless channels and the influence of multipath effects, CSI-based human activity recognition technology still faces technical challenges in areas such as multi-person detection and cross-domain recognition. Summary of the Invention

[0004] The present invention addresses the shortcomings of existing methods by proposing a real-time human behavior recognition method based on cross-domain dual-modal feature fusion. This method aims to adapt to complex recognition scenarios and enable single-person or multi-person behavior recognition in environments with interference, thereby improving recognition accuracy and anti-interference capabilities.

[0005] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0006] The real-time human behavior recognition method based on cross-domain dual-modal feature fusion of the present invention is characterized by the following steps:

[0007] Step 1: Collect CSI motion sample data and video sample data;

[0008] Step 1.1: Select p1 location regions and p2 random location regions with a spacing of d within the rectangular area. The p1 location regions are the training locations, and the p2 random location regions are the test locations, and p1 + p2 = m.

[0009] Step 1.2: Place a router as a WIFI signal transmitting device on the periphery of the rectangular area, denoted as AP; place a PC with a wireless network card as a receiving device on the periphery of the rectangular area, denoted as RP1; and place a camera as a video data receiving device on the periphery of the rectangular area, denoted as RP2.

[0010] Step 1.3: Have q experimental personnel perform n actions at m positions, where q1 personnel are the trainees and q2 personnel are the testers, and q1 + q2 = q. Receive CSI data of the actions of the q experimental personnel through RP1; at the same time, use RP2 to collect video data of the actions of the q experimental personnel.

[0011] Step 2: Preprocess the data;

[0012] Step 2.1: Denoise the amplitude values ​​of the CSI data using wavelet transform, and then perform equal-interval sampling to obtain the CSI action sample set T = {(x j i ,y j i )|i∈[1,q],j∈[1,m]},where, x j i Let y represent the CSI action sample of the i-th person at the j-th position. j i For x j i The corresponding real tags;

[0013] Video data is sampled at equal intervals to construct a video motion sample dataset V = {(v j i ,y j i )|i∈[1,q],j∈[1,m]},where, v j i y represents the video motion sample of the i-th person at the j-th position. j i For v j i The corresponding real tags, and in, Indicates video action sample v j i The k-th frame image; f represents the total number of frames of the video motion samples;

[0014] Step 3: Construct a dual-modal human behavior recognition network, including: a feature extraction module, a feature fusion module, a feature alignment module, and a classification module;

[0015] Step 3.1: Construct the feature extraction module, including: a CSI motion feature extraction module and a video motion feature extraction module;

[0016] Step 3.1.1: The CSI action feature extraction module is used to extract CSI action samples x. j i CSI action sample features

[0017] Step 3.1.2: The video motion feature extraction module is used to extract video motion samples v. j i Video motion features

[0018] Step 3.2: The feature fusion module consists of three fully connected layers (FC).

[0019] CSI action sample features and video motion features The input is fed into the feature fusion module, and the fused features are obtained using equation (1).

[0020]

[0021] In equation (1), FC1 represents the first fully connected layer, FC2 represents the second fully connected layer, and FC3 represents the third fully connected layer;

[0022] Step 3.3: The feature alignment module consists of a gradient inversion layer (GRL) and a discriminator network (Disc).

[0023] The gradient inversion layer (GRL) preserves the fused features of the input during forward propagation of the network. Its own gradient; during backpropagation of the network, the fused features of the input. The gradient is inverted and multiplied by the parameter λ, where λ∈[0,1].

[0024] The discriminator network Disc consists of two one-dimensional convolutional layers, one Leaky ReLU layer, and one fully connected layer.

[0025] The fusion feature The input is processed through the discriminator network Disc, and then sequentially through a first 1D convolutional layer, a Leak ReLU layer, a second 1D convolutional layer, and a fully connected layer to obtain the domain features.

[0026] The domain features The classification probability sequence is obtained after applying the Softmax function, and the category corresponding to the highest probability in the classification probability sequence is taken as x. j i and v j i The domain tag;

[0027] Step 3.4, from x j i and v j i The cross-entropy loss function Loss1 is constructed using the domain labels and the people and locations in their real labels;

[0028] Step 3.5: The classification module is constructed from fully connected layers (FC).

[0029] The fusion feature After being processed in the classification module, a feature vector sequence is obtained. After processing by the Softmax function, a feature probability vector is obtained, and the category corresponding to the highest probability in the feature probability vector is taken as the predicted action type.

[0030] Step 3.6, from x j i and v j i The cross-entropy loss function Loss2 is constructed by comparing the predicted action types with the action types in the actual labels.

[0031] Step 3.7: Construct the total loss Loss = Loss1 + αLoss2, where α is the weighting factor;

[0032] Step 4: Based on the CSI action sample set T and the video action sample dataset V, the bimodal human behavior recognition network is trained using the gradient descent method, and the loss is backpropagated to update the network parameters until the loss converges, thereby obtaining a trained human behavior recognition model for recognizing actions at any location.

[0033] The real-time human behavior recognition method based on cross-domain dual-modal feature fusion described in this invention is characterized in that the CSI action feature extraction module in step 3.1 consists of a GAT network, an InT network, an average pooling layer (Avgpool), and a Flatten network; wherein the GAT network consists of a graph attention layers; and the InT network consists of b Inception networks.

[0034] The CSI action sample xj i The input is fed into a bimodal human behavior recognition network, and after being processed by the GAT network of the CSI action feature extraction module in the feature extraction module, graph attention features are obtained.

[0035] The graph attention features The input is fed into an InT network for feature extraction to obtain action features.

[0036] The action features After being processed sequentially by an Avgpool layer and a Flatten network, CSI action sample features are obtained.

[0037] The video motion feature extraction module in step 3.2 includes: a spatial feature extraction module and a temporal feature extraction module;

[0038] The spatial feature extraction module consists of f parallel spatial feature extraction units, each of which is composed of a ResNet-50 residual network and a Flatten layer.

[0039] The time feature extraction module consists of an s-layer LSTM long short-term memory network and a flatten layer.

[0040] The video action sample The input is processed by f parallel spatial feature extraction units of the spatial feature extraction module in the video action feature extraction module within the bimodal human behavior recognition network, resulting in the corresponding spatial action features. in, Represents the k-th spatial feature extraction unit pair Extracted spatial motion features;

[0041] The spatial motion characteristics After being processed by the time feature extraction module, video action features are obtained.

[0042] The present invention provides an electronic device, comprising a memory and a processor, wherein the memory is used to store a program that supports the processor in executing the real-time human behavior recognition method, and the processor is configured to execute the program stored in the memory.

[0043] The present invention discloses a computer-readable storage medium on which a computer program is stored, wherein the computer program is executed by a processor to perform the steps of the real-time human behavior recognition method.

[0044] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0045] 1. This invention combines two human behavior recognition technologies: CSI-based human activity recognition and computer vision-based human activity recognition. It fully utilizes the complementary characteristics of the two modal sample information, effectively solving the technical problem that single-modal methods are susceptible to environmental interference, and achieving anti-interference and high-precision human behavior recognition.

[0046] 2. This invention uses a discriminator network to perform feature-level alignment training on the domain change information of personnel and location in the sample, which enhances the feature extractor's ability to extract domain-independent features. Through one training, it can realize the recognition of human behavior of any person and location, thereby improving the model's generalization ability to personnel and location changes. Attached Figure Description

[0047] Figure 1 This is a flowchart illustrating the implementation of the present invention;

[0048] Figure 2 This is a diagram of the CSI action feature extraction module of the present invention;

[0049] Figure 3 This is a diagram of the video motion feature extraction module of the present invention;

[0050] Figure 4 This is a diagram of the feature fusion module of the present invention;

[0051] Figure 5 This is a network diagram of the discriminator of the present invention. Detailed Implementation

[0052] In this embodiment, as Figure 1 As shown, a real-time human behavior recognition method based on cross-domain dual-modal feature fusion is performed according to the following steps;

[0053] Step 1: Collect CSI motion sample data and video sample data;

[0054] Step 1.1: Select p1 location regions and p2 random location regions with a spacing of d within the rectangular area. The p1 locations are the training locations, and the p2 random location regions are the test locations, and p1 + p2 = m. In this embodiment, 9 locations with a spacing of 2 meters are selected, of which 6 locations are randomly selected as training locations and the other 3 locations are used as test locations.

[0055] Step 1.2: Place a router as a WIFI signal transmitting device (AP) on the periphery of the rectangular area, place a PC with a wireless network card as a receiving device (RP1) on the periphery of the rectangular area, and place a camera as a video data receiving device (RP2) on the periphery of the rectangular area. In this embodiment, the WIFI transmitting device (AP) is a TL-WDR6500 router, the WIFI receiving device is a PC with an Intel 5300 network card, and the video acquisition device is a TL-IPC55A surveillance camera.

[0056] Step 1.3: Have q experimental personnel perform n actions at m locations, where q1 personnel are trainees and q2 personnel are testers, and q1 + q2 = q. RP1 receives CSI data of the actions of the q experimental personnel; simultaneously, RP2 collects video data of the actions of the q experimental personnel. In this embodiment, there are 3 trainees and 2 testers, performing 10 action types. The CSI data sampling rate is 1000 data packets per second. One transmit antenna and three receive antennas are used, with 30 subcarriers per link, collecting a total of 90 subcarrier signals. The video data acquisition frame rate is 15 frames per second, and the sampling format is single-channel grayscale image.

[0057] Step 2: Preprocess the data;

[0058] Step 2.1: Denoise the amplitude values ​​of the CSI data using wavelet transform, and then perform equal-interval sampling to obtain the CSI action sample set T = {(x j i ,y j i )|i∈[1,q],j∈[1,m]},where, x j i Let y represent the CSI action sample of the i-th person at the j-th position. j i For x j i The corresponding real labels; in this embodiment, 400 data packets are retained in the time dimension for each CSI data, resulting in CSI sample x. j i The format is 90×400, where 90 represents the number of subcarriers, 400 represents the timestamp length of the CSI for each subcarrier, and the label is y. j i This includes three types of information: personnel, location, and action type.

[0059] Video data is sampled at equal intervals to construct a video motion sample dataset V = {(v j i ,yj i )|i∈[1,q],j∈[1,m]},where, v j i y represents the video motion sample of the i-th person at the j-th position. j i For v j i The corresponding real tags, and in, Indicates video action sample v j i The k-th frame image; f represents the total number of frames of the video action samples; in this embodiment, 20 frames are retained after sampling each video data at equal intervals;

[0060] Step 3: Construct a dual-modal human behavior recognition network, including: a feature extraction module, a feature fusion module, a feature alignment module, and a classification module;

[0061] Step 3.1: Construct a feature extraction module, including: a CSI motion feature extraction module and a video motion feature extraction module;

[0062] Step 3.1.1: The CSI action feature extraction module consists of a single GAT network, a single InT network, a single Avgpooling layer, and a single Flatten network. Its network structure is as follows: Figure 2 As shown; the GAT network consists of a graph attention layers; the InT network consists of b Inception networks; in this embodiment, the GAT has 3 graph attention layers, each with 90 graph nodes, corresponding to 90 subcarrier data; the Int network has 3 Inception networks, and the kernel sizes of the three parallel one-dimensional convolutional networks Conv1d in each Inception network are 15, 31, and 63 respectively;

[0063] CSI Action Samples x j i The input is fed into a bimodal human behavior recognition network, and after being processed by the GAT network of the CSI action feature extraction module in the feature extraction module, graph attention features are obtained.

[0064] Attention features The input is fed into an InT network for feature extraction to obtain action features.

[0065] Action characteristics The inputs are sequentially fed into an Avgpool layer and a Flatten network for processing to obtain CSI action sample features.

[0066] Step 3.1.2: The video action feature extraction module includes a spatial feature extraction module and a temporal feature extraction module, and its network structure is as follows: Figure 3 As shown;

[0067] The spatial feature extraction module consists of f parallel spatial feature extraction units, each of which consists of a ResNet-50 residual network layer and a Flatten layer. In this embodiment, the number of parallel spatial feature extraction modules, f, is 20, corresponding to 20 frames of image data for each video action sample.

[0068] The temporal feature extraction module consists of an s-layer LSTM long short-term memory network and a flatten layer; in this embodiment, the LSTM long short-term memory network has 2 layers.

[0069] Video motion samples The input is placed into a bimodal human behavior recognition network and processed by f parallel spatial feature extraction units in the spatial feature extraction module of the video action feature extraction module to obtain the corresponding spatial action features. in, Represents the k-th spatial feature extraction unit pair Extracted spatial motion features;

[0070] Spatial motion characteristics After processing in the input time feature extraction module, video action features are obtained.

[0071] Step 3.2: The feature fusion module consists of three fully connected layers (FC), and its network structure is as follows. Figure 4 As shown;

[0072] CSI action sample features and video motion features The input is fed into the feature fusion and feature alignment module, and the fused features are obtained using equation (1).

[0073]

[0074] In equation (1), FC1 represents the first fully connected layer, FC2 represents the second fully connected layer, and FC3 represents the third fully connected layer;

[0075] Step 3.3: The feature alignment module consists of a gradient inversion layer (GRL) and a discriminator network (Disc).

[0076] Gradient Reversal Layer (GRL) preserves the fused features of the input during forward propagation of the network. Self-gradient; during network backpropagation, the fused features of the input are... The gradient is inverted and multiplied by the parameter λ, where λ∈[0,1]; in this embodiment, the value of λ is defined as: β is the ratio of the current iteration number to the total number of iterations.

[0077] The discriminator network Disc consists of two one-dimensional convolutional layers, one Leaky ReLU layer, and one fully connected layer. Its network structure is as follows: Figure 5 As shown;

[0078] Fusion features The input is processed through the discriminator network Disc, and then sequentially through a first 1D convolutional layer, a LeakReLU layer, a second 1D convolutional layer, and a fully connected layer to obtain the domain features.

[0079] Domain features The classification probability sequence is obtained after applying the Softmax function, and the category corresponding to the highest probability in the classification probability sequence is taken as x. j i and v j i The domain tag;

[0080] Step 3.4, from x j i and v j i The cross-entropy loss function Loss1 is constructed using the domain labels and the people and locations in their real labels;

[0081] Step 3.5: The classification module is constructed from fully connected layers (FC).

[0082] Fusion features After processing in the input classification module, a sequence of feature vectors is obtained. After processing by the Softmax function, a feature probability vector is obtained, and the category corresponding to the highest probability in the feature probability vector is taken as the predicted action type.

[0083] Step 3.6, from x j i and v j i The cross-entropy loss function Loss2 is constructed by comparing the predicted action types with the action types in the actual labels.

[0084] Step 3.7: Construct the total loss Loss = Loss1 + αLoss2, where α is the weighting factor; in this embodiment, α is 0.5.

[0085] Step 4: Based on the CSI action sample set T and the video action sample dataset V, the bimodal human behavior recognition network is trained using the gradient descent method, and the loss is backpropagated to update the network parameters until the loss converges, thus obtaining a trained human behavior recognition model for recognizing actions at any location.

[0086] In this embodiment, an electronic device includes a memory and a processor. The memory stores a program that supports the processor in executing the above-described method, and the processor is configured to execute the program stored in the memory.

[0087] In this embodiment, a computer-readable storage medium stores a computer program, which is executed by a processor to perform the steps of the above method.

Claims

1. A real-time human behavior recognition method based on cross-domain dual-modal feature fusion, characterized in that, The procedure is as follows: Step 1: Collect CSI motion sample data and video sample data; Step 1.1: Select areas within the rectangular region where all spacing is d. Location area and A number of random location regions, among which... Each location region is a training location. A random location region was used as the test location, and + = m; Step 1.2: Place a router as a Wi-Fi signal transmitting device (AP) on the periphery of the rectangular area, and place a PC with a wireless network card as a receiving device on the periphery of the rectangular area (Dedicated as AP). A camera is placed around the perimeter of the rectangular area as a video data receiving device, denoted as . ; Step 1.3: Have q experimenters perform n actions at m positions, where... [Number] people were selected as trainees. One person was selected as a tester, and and through Receive CSI data on the actions of q experimenters; simultaneously, use Collect video data of the actions of q experimental participants; Step 2: Preprocess the data; Step 2.1: Denoise the amplitude values ​​of the CSI data using wavelet transform, and then perform equal-interval sampling to obtain the CSI action sample set. ={( , )| , },in, This represents the CSI action sample of the i-th person at the j-th position. for The corresponding real tags; Video data is sampled at equal intervals to construct a video motion sample dataset. ={( , )| , },in, This represents the video action sample of the i-th person at the j-th position. for The corresponding real tags, and = ,in, Represents video motion samples The k-th frame image; f represents the total number of frames of the video motion samples; Step 3: Construct a dual-modal human behavior recognition network, including: a feature extraction module, a feature fusion module, a feature alignment module, and a classification module; Step 3.1: Construct the feature extraction module, including: a CSI action feature extraction module and a video action feature extraction module; Step 3.1.1: The CSI action feature extraction module consists of a GAT network, an InT network, an average pooling layer (Avgpool), and a Flatten network; wherein the GAT network consists of a graph attention layers; and the InT network consists of b Inception networks. The CSI action sample The input is fed into a bimodal human behavior recognition network, and after being processed by the GAT network of the CSI action feature extraction module in the feature extraction module, graph attention features are obtained. ; The graph attention features The input is fed into an InT network for feature extraction to obtain action features. ; The action features After being processed sequentially by an Avgpool layer and a Flatten network, CSI action sample features are obtained. ; Step 3.1.2: The video action feature extraction module includes: a spatial feature extraction module and a temporal feature extraction module; The spatial feature extraction module consists of f parallel spatial feature extraction units, each of which is composed of a ResNet-50 residual network and a Flatten layer. The time feature extraction module consists of an s-layer LSTM long short-term memory network and a flatten layer. The video action sample = The input is processed by f parallel spatial feature extraction units of the spatial feature extraction module in the video action feature extraction module within the bimodal human behavior recognition network, resulting in the corresponding spatial action features. ;in, Represents the k-th spatial feature extraction unit pair Extracted spatial motion features; The spatial motion characteristics After being processed by the time feature extraction module, video action features are obtained. ; Step 3.2: The feature fusion module consists of three fully connected layers. constitute; CSI action sample features and video motion features The input is fed into the feature fusion module, and the fused features are obtained using equation (1). ; = ( ( ) + ( )) (1) In equation (1), This indicates the first fully connected layer. This indicates the second fully connected layer. This indicates the third fully connected layer; Step 3.3: The feature alignment module consists of a gradient inversion layer (GRL) and a discriminator network (Disc). The gradient inversion layer (GRL) preserves the fused features of the input during forward propagation of the network. Its own gradient; during backpropagation of the network, the fused features of the input. Invert the gradient and multiply by the parameter ,and ∈ ; The discriminator network Disc consists of two one-dimensional convolutional layers, one Leaky ReLU layer, and one fully connected layer. The fusion feature The input is processed through the discriminator network Disc, and then sequentially through a first 1D convolutional layer, a LeakReLU layer, a second 1D convolutional layer, and a fully connected layer to obtain the domain features. ; The domain features The classification probability sequence is obtained after applying the Softmax function, and the category corresponding to the highest probability in the classification probability sequence is taken as... and The domain tag; Step 3.4, from and The cross-entropy loss function Loss1 is constructed using the domain labels and the people and locations in their real labels; Step 3.5: The classification module is constructed from fully connected layers (FC). The fusion feature After being processed in the classification module, a feature vector sequence is obtained. After processing by the Softmax function, a feature probability vector is obtained, and the category corresponding to the highest probability in the feature probability vector is taken as the predicted action type. Step 3.6, by and The cross-entropy loss function Loss2 is constructed by comparing the predicted action types with the action types in the actual labels. Step 3.7: Construct the total loss Loss = Loss1 + Loss2, where As a weighting factor; Step 4: Based on the CSI action sample set and video motion sample dataset The bimodal human behavior recognition network is trained using gradient descent, and the loss is backpropagated to update the network parameters until the loss converges, thereby obtaining a trained human behavior recognition model for recognizing actions at any location.

2. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store a program that supports the processor in executing the real-time human behavior recognition method of claim 1, and the processor is configured to execute the program stored in the memory.

3. A computer-readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to perform the steps of the real-time human behavior recognition method according to claim 1.

Images