Human behavior recognition method based on fusion attention mechanism prototype neural network

Abstract

The invention discloses a human behavior recognition method based on a fusion attention mechanism prototype neural network, and relates to the technical field of deep learning, and the method comprises the steps: preprocessing original data to obtain target channel state information, dividing the target channel state information to obtain a support set and a query set, using an encoder to obtain sample feature vectors of each support sample and each query sample, calculating a category prototype vector of each category according to the sample feature vectors of all the support samples belonging to each category, and calculating the category prototype vector of each category according to the sample feature vector of the query sample and the category prototype vector of each category; and calculating the probability distribution that the query samples are classified into various categories, updating the model parameters of the human behavior recognition model according to the probability distribution that all the query samples are classified into various categories, and predicting the test samples in the test set by using the trained human behavior recognition model. The human body behavior recognition method and device can still quickly and accurately perform human body behavior recognition under the condition of limited data samples.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to a method for human behavior recognition based on a prototype neural network with a fusion attention mechanism. Background Art

[0002] Human activity recognition (HAR) is a multidisciplinary research field, and its core goal is to identify and classify various human activity patterns by analyzing the data generated by human activities, such as images, sensor signals, etc. HAR technology has shown broad application potential in many fields such as smart home, health monitoring, security monitoring, and virtual reality. In particular, the method of human activity detection based on channel state information (CSI) data in WiFi wireless signals has attracted much attention due to its non-invasiveness and easy deployment. By analyzing the fluctuations and characteristics of CSI data, this method can effectively identify human activity states including walking, running, jumping, sitting, etc. In addition, CSI data is easy to collect and process, and does not require complex algorithms or a large amount of computing resources. Human activity recognition technology based on CSI has significant advantages in practicality and promotion.

[0003] CSI signal human activity detection based on small sample learning is a technology that identifies human activity status by analyzing the changes in channel state information in WiFi wireless signals. It belongs to the research scope of human activity recognition. Its main goal is to solve the problems of insufficient generalization ability and cost of traditional deep learning methods in data-scarce environments. By applying the idea of ​​small sample learning to human activity detection of CSI signals, the model's dependence on large-scale data sets can be significantly reduced, thereby coping with the problems of difficult data collection and high demand for large-scale data. Currently, small sample learning technology is widely used in the field of human activity detection, but it still encounters several challenges: current research mostly relies on complex network models to process CSI data in order to build a high-precision detection model. In the face of time-sensitive tasks, research faces the dual challenges of limited sample number and strict timeliness requirements for problem solving. In addition, the accuracy and efficiency of the model in actual deployment also need to be improved. Therefore, there is an urgent need for a method that can provide fast and accurate detection in the case of limited data samples to effectively address these challenges. Summary of the invention

[0004] The purpose of this application is to provide a human behavior recognition method based on a prototype neural network with a fusion attention mechanism, which can quickly and accurately recognize human behavior when data samples are limited.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] This application provides a method for human behavior recognition based on a prototype neural network with a fusion attention mechanism, including:

[0007] Obtaining original channel state information;

[0008] Preprocessing the original channel state information to obtain target channel state information corresponding to the original channel state information;

[0009] The target channel state information is divided into a training set, a test set and a validation set, and a support set and a query set are determined according to the training set; the support set includes a number of support samples; the query set includes a number of query samples;

[0010] Input the support set and the query set into the encoder to obtain an encoded feature tensor; the encoded feature tensor includes a sample feature vector of each support sample and a sample feature vector of each query sample; the encoder includes a first convolution module, a second convolution module, a self-attention mechanism module, a third convolution module, a fusion module and a flattening layer; the first convolution module includes a plurality of first convolution blocks connected in sequence; the second convolution module includes a plurality of second convolution blocks connected in sequence; the third convolution module includes a plurality of depth-separable convolution blocks;

[0011] For each category, a category prototype vector of the category is calculated according to the sample feature vectors of all support samples belonging to the category;

[0012] A classifier is used to calculate the probability distribution of the query sample being classified into each category according to the sample feature vector of the query sample and the category prototype vector of each category; the encoder and the classifier constitute a human behavior recognition model;

[0013] According to the probability distribution of all the query samples being classified into various categories, the model parameters of the human behavior recognition model are updated to obtain a trained human behavior recognition model;

[0014] The test samples of the test set are input into the trained human behavior recognition model to obtain the predicted category of the test samples.

[0015] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0016] The present application provides a method for human behavior recognition based on a prototype neural network with a fusion attention mechanism, preprocesses original channel state information to obtain target channel state information corresponding to the original channel state information, divides the target channel state information into a training set and a test set, determines a support set and a query set based on the training set, inputs the support set and the query set into an encoder, obtains a sample feature vector of each support sample and a sample feature vector of each query sample, for each category, calculates a category prototype vector of the category based on the sample feature vectors of all support samples belonging to the category, calculates the probability distribution of the query sample being classified into each category based on the sample feature vector of the query sample and the category prototype vectors of each category, updates the model parameters of a human behavior recognition model based on the probability distribution of all query samples being classified into each category, inputs the test sample of the test set into the trained human behavior recognition model, and obtains the predicted category of the test sample, the present application adds a self-attention mechanism to the prototype neural network, and uses the fusion attention mechanism prototype neural network (human behavior recognition model) to perform human behavior recognition, thereby realizing fast and accurate human behavior recognition even when data samples are limited. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 A flowchart of a method for human behavior recognition based on a prototype neural network with an integrated attention mechanism is provided in one embodiment of the present application;

[0019] Figure 2 A block diagram of a model provided in accordance with an embodiment of the present application;

[0020] Figure 3 A schematic diagram of an algorithm flow provided for an embodiment of the present application. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0022] The basic process of CSI human activity detection with few-shot learning involves using WiFi devices to collect CSI data covering a variety of activities, followed by denoising and normalizing the data to extract key features. Next, a model for few-shot learning is constructed using techniques such as transfer learning or meta-learning. The model is trained using limited data and its performance is evaluated through cross-validation. Finally, the model is optimized to ensure stability and accuracy in practical applications and then deployed in a human activity recognition system for real-time detection. Current research has mainly explored a variety of techniques such as twin networks and prototype networks to construct models for few-shot learning. Among these methods, prototype networks are favored for their excellent performance and generalization ability.

[0023] The core steps of the prototype network method mainly include the following aspects: first, divide the data set into a support set and a query set, and then use a convolutional neural network as a feature extractor to extract features of sample data from the support set and the query set. Then use the features in the support set to construct the prototype of each category, that is, the center point of the category, which represents the feature distribution of the category. In the model training phase, the goal is to enable the model to learn how to classify new query samples based on these prototypes. Specifically, for each new query sample, the model calculates its similarity with all category prototypes. Finally, the model will classify the query sample into the category with the highest similarity based on these similarity metrics.

[0024] In the field of prototype network technology research, the current focus is mainly on the following key aspects: first, the optimization of feature extraction, which aims to improve the expressiveness and discrimination of features through advanced algorithms; second, the improvement of prototype calculation to more accurately capture the central features of the category; and finally, the refinement of similarity calculation to ensure that the model can more accurately evaluate the similarity between samples and prototypes. However, although the above methods have achieved remarkable results in improving model accuracy, they have also led to an increase in model complexity, which not only reduces the model's reasoning efficiency, but also may affect the model's generalization performance when dealing with complex scenarios. Therefore, future research needs to take into account both the efficiency and generalization ability of the model while improving performance.

[0025] Existing small sample learning techniques are not accurate enough in complex action recognition and generalization across datasets, and their real-time performance is limited due to the high computational complexity of the model. In addition, the accuracy and efficiency of the model in actual deployment also need to be improved.

[0026] In response to the above problems, this application aims to propose a human activity recognition technology that integrates convolutional neural networks and attention mechanisms, so that it can still provide a fast and accurate detection method when data samples are limited and task time is limited.

[0027] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0028] In an exemplary embodiment, Figure 1 As shown, a method for human behavior recognition based on a prototype neural network with a fusion attention mechanism is provided, and the method includes the following steps 201 to 206. Among them:

[0029] Step 201: Acquire original channel state information.

[0030] Step 202: pre-process the original channel state information to obtain target channel state information corresponding to the original channel state information.

[0031] Step 203: Divide the target channel state information into a training set, a test set and a validation set, and determine a support set and a query set according to the training set; the support set includes a number of support samples; the query set includes a number of query samples.

[0032] Step 204: Input the support set and the query set into the encoder to obtain an encoded feature tensor; the encoded feature tensor includes a sample feature vector of each support sample and a sample feature vector of each query sample; the encoder includes a first convolution module, a second convolution module, a self-attention mechanism module, a third convolution module, a fusion module and a flattening layer; the first convolution module includes a plurality of first convolution blocks connected in sequence; the second convolution module includes a plurality of second convolution blocks connected in sequence; the third convolution module includes a plurality of depth-separable convolution blocks; for each category, the category prototype vector of the category is calculated according to the sample feature vectors of all support samples belonging to the category; the classifier is used to: calculate the probability distribution of the query sample being classified into each category according to the sample feature vector of the query sample and the category prototype vector of each category; the encoder and the classifier constitute a human behavior recognition model.

[0033] Step 205: According to the probability distribution of all the query samples being classified into various categories, the model parameters of the human behavior recognition model are updated to obtain a trained human behavior recognition model.

[0034] Step 206: Input the test samples of the test set into the trained human behavior recognition model to obtain the predicted category of the test samples.

[0035] By implementing the above steps 201 to 206 , human behavior recognition can be performed quickly and accurately even when data samples are limited.

[0036] The principle block diagram of the model is as follows Figure 2As shown, it is mainly divided into three stages, namely data preprocessing, data partitioning, and model construction. In the data preprocessing stage, the present application first performs segmentation and completion on the CSI data to unify the data size. Then the data is subjected to time-frequency transformation and filtering, the data is converted from the time domain to the frequency domain, and the outliers and noise in the sample are removed to improve the accuracy of recognition between similar actions. In the data partitioning stage, in order to enhance the generalization ability of the model and prevent overfitting during the training process, the present application uses a random sampling method to divide the data into a training set and a test set. In the training set, for the recognition task of human activity posture, the present application randomly extracts samples and divides them into a support set and a query set. The support set is used for the model to learn the features of each category; the query set is used to evaluate the model's classification ability for new situations. In the model construction stage, the present application first designs a custom module that combines convolution operations and attention mechanisms. The module uses a self-attention mechanism to capture long-distance dependencies in the input data, and the convolution operation enhances the model's perception of local features. Subsequently, the present application integrates the module into the model to construct a complete network structure including an encoder and a classifier. The encoder consists of a series of convolutional layers and custom modules, which are responsible for extracting multi-scale and multi-dimensional features of the input data; the classifier is based on these features and classifies by calculating the Euclidean distance between the query sample and the category prototype vector, achieving rapid adaptation and accurate classification of new categories.

[0037] In another exemplary embodiment of the present application, the algorithm flow chart of the model of the human behavior recognition method based on the fusion attention mechanism prototype neural network is as follows: Figure 3 In order to optimize the data quality and improve the accuracy of data analysis, the raw data is also preprocessed, and the above step 202 is replaced by the following steps 301 to 306:

[0038] Step 301: segmenting and completing the original channel state information to obtain completed data;

[0039] Step 302: Calculate the amplitude of the supplemented data to obtain amplitude data;

[0040] Step 303: Perform fast Fourier transform on the amplitude data to convert the signal from the time domain to the frequency domain to obtain first transformed data;

[0041] Step 304: performing wavelet transform on the first transformed data to obtain second transformed data;

[0042] Step 305: performing an inverse wavelet transform on the second transformed data to obtain third transformed data;

[0043] Step 306: Convert the third transformed data into a torch tensor; the torch tensor is the target channel state information corresponding to the original channel state information.

[0044] First, the CSI data is segmented and completed to unify the data size. Segmentation mainly involves cutting out the collected CSI data that is not related to the subject's human activities, removing redundant information and reducing the amount of data. Then, the linear interpolation method is used to complete the data. The principle is shown in formula (1).

[0045]

[0046] in, To complete the data, that is, to supplement the data value at the location where packet loss occurs; x0 and y0 are the position and data value of the previous point adjacent to the data location where packet loss occurs, respectively; x1 and y1 are the position and data value of the next point adjacent to the supplementary packet loss location, respectively; x is the location where missing data is waiting to be supplemented when packet loss occurs.

[0047] The amplitude is calculated below to highlight the key change points of the signal. The CSI matrix received per unit time is expressed as shown in formula (1).

[0048]

[0049] Among them, H i,j is the CSI information per unit time on the jth subcarrier for the i-th transmitting antenna and receiving antenna pair. ||H i,j || is CSI information H i,j The amplitude value is calculated as shown in formula (3).

[0050] ||H i,j ||=abs(H i,j ) (3);

[0051] Where abs(H) is the modulo operation on complex numbers.

[0052] Then, the signal is converted from the time domain to the frequency domain through the fast Fourier transform (FFT) to prepare for frequency feature analysis. The signal is then decomposed at multiple scales through the wavelet transform (DWT) to extract local features and capture subtle changes in human activity. The inverse wavelet transform (IDWT) ensures the integrity of signal reconstruction. Finally, the data is converted into a torch tensor and the shape is adjusted to meet the model input requirements. The above steps ensure the cleanliness, standardization, and applicability of the data before inputting the model.

[0053] In step 203, the train_test_split() function is used to divide the training set and the test set, and then the support set and the query set are divided by random sampling. The support samples in the support set and the query samples in the query set are both target signal state information data.

[0054] The human behavior recognition model is named Proto-CSNet network, which includes an encoder and a classifier.

[0055] In the process of constructing the feature extraction module (ie, encoder), the present application first adopts a network architecture consisting of a series of convolution blocks conv_block to construct a first convolution module. In this embodiment, the first convolution module includes 4 first convolution blocks, and the initialization stage of each first convolution block in the first convolution module is composed of a first convolution layer, which is equipped with a convolution kernel of size 3×3, and the step size and padding are set to 1 to accurately capture the local features of the sample. Subsequently, a batch normalization layer is introduced to normalize the output of the first convolution layer, aiming to reduce internal covariate shift and optimize the training process by accelerating model convergence. Then, a nonlinear transformation is introduced through the ReLU activation function to enhance the model's learning ability for complex function mapping. Finally, each first convolution block ends with a maximum pooling layer, which uses a 2×2 pooling window and a step size of 2 to reduce the spatial dimension of the feature vector while expanding the receptive field of the model. Each first convolutional block gradually extracts the feature features of the input data through the convolutional layer. From the shallow layer to the deep layer, the feature representation gradually evolves from the intuitive attributes of the original data to more abstract and advanced features. The network structure is deepened by stacking multiple first convolutional blocks, which not only enhances the hierarchical and refined features, but also maintains parameter efficiency and spatial resolution. The feature extraction of multiple first convolutional blocks in the encoder can be expressed in the form shown in formula (4).

[0056] feature (n) =conv_block (n) (x) (4);

[0057] Among them, conv_block (n) Indicates that the first convolution block operation is applied n times, gradually extracting features from the input until the deepest feature of the network is reached (n) Preferably, the number of the first convolution blocks in the first convolution module may be four.

[0058] The second convolution module includes four second convolution blocks, each of which includes a second convolution layer, wherein the first three second convolution blocks in the second convolution module are parallel structures, which are used to calculate the query, key and value respectively; the fourth second convolution block is used for position encoding.

[0059] After the convolution operation of the above series of convolutional layers, the model of the present application transmits the convolution output result of the first convolutional module as input data to three parallel second convolutional layers (conv1, conv2, conv3), which are used to generate the query, key and value required by the self-attention mechanism layer. These second convolutional layers all use 1×1 convolution kernels to maintain the spatial dimension of the input feature vector to ensure that suitable input conditions are provided for the subsequent attention mechanism. The calculation formulas for query q, key k and value v are as follows:

[0060] q = conv1(x) = Conv 1×1 (x) (5);

[0061] k = conv2(x) = Conv 1×1 (x) (6);

[0062] v = conv3(x) = Conv 1×1 (x) (7);

[0063] Among them, Conv 1×1 represents a convolution operation with a 1×1 convolution kernel, x is the input feature vector, and is the convolution output result of the first convolution module. In addition, there is another second convolution layer (conv_p) dedicated to position encoding, with 2 input channels and head_dim output channels. The purpose is to capture and encode the spatial position information of the input data and provide the model with contextual awareness of the spatial dimension. The calculation process of the fourth second convolution layer in the second convolution module is as follows:

[0064] pe = position(H,W) (8);

[0065] Among them, H and W are the height and width of the convolution output result of the first convolution module, respectively, and position(H,W) is the position code generated by the position function.

[0066] The self-attention weight is calculated based on q, k, v and position encoding pe calculated in the above steps, so that the model can fully perceive the global context and enable local features to integrate the information of the entire input sequence.

[0067] att=softmax(∑head (q att ⊙(k att +q pe -unfold rpe )))(9);

[0068] Where q_att, k_att, and v_att represent query, key, and value tensors, respectively. ⊙ represents the Hadamard product (element-wise multiplication); q pe Represents the positional encoding of the query; unfold rpe represents the expanded position encoding; head is the number of heads in the attention mechanism, head dim Indicates the dimensions of each head.

[0069] The value features are weighted and aggregated according to the self-attention weights to generate a comprehensive feature representation to improve the model's ability to capture complex dependencies and contextual information.

[0070]

[0071] in, is the size of the area covered by the convolution kernel, kernel att Represents the size of the convolution kernel, att i is the expansion of the i-th element in the attention weight tensor; unfold pad (v att ) i is the input feature v att The ith region after filling and expanding, v att Represents the value tensor, which is used for weighted summation when calculating attention weights.

[0072] While the self-attention mechanism layer is being calculated, the human behavior recognition model uses a third convolution module consisting of two depthwise separable convolution blocks to improve computational efficiency and optimize feature extraction. The first depthwise separable convolution block dep_conv consists of a depthwise separable convolution layer, which mainly performs preliminary convolution extraction on the fused features; while the second depthwise separable convolution block dep_conv1 further refines the feature extraction process on this basis. The second depthwise separable convolution block consists of multiple depthwise separable convolution layers, each of which contains a depthwise separable convolution kernel, and padding and stride are used to ensure the consistency of the output feature vector with the input size. After each depthwise separable convolution layer, the model introduces a batch normalization BN layer and a ReLU activation function to enhance the model's ability to express nonlinear features, thereby effectively improving the quality of feature extraction and the overall performance of the model.

[0073] out conv1= dep_conv(f_conv) (11);

[0074] out conv2 = dep_conv1(f_conv) (12);

[0075] In the formula, out conv1 represents the output of the first depth-wise separable convolutional block, out conv2 represents the output of the second depth-wise separable convolutional block, f_conv represents the feature vector obtained after permutation and reshaping, dep_conv and dep_conv1 are both deep convolutional layers used to extract features. Finally, the model fuses the outputs of the two depth-wise separable convolutional blocks.

[0076]

[0077] In the formula, out conv is the output of the third convolution module, that is, the fused feature obtained by fusing the outputs of the two depth-separable convolution blocks.

[0078] In the output stage of the encoder part of the human behavior recognition module, the fusion module in the human behavior recognition model in this application uses two learnable weight parameters rate1 and rate2 to adjust the mixing ratio of the output of the attention module and the convolution module to achieve dynamic adjustment of the model behavior. The output of the fusion module is shown in formula (14):

[0079] output=rate1×out att +rate2×out conv (14);

[0080] Among them, out att is the output of the self-attention mechanism module. Through this weighted combined feature representation, the model can accurately capture local features and effectively model long-distance dependencies while maintaining computational efficiency, significantly improving the ability to understand and represent complex data structures. After the fusion module, the output of the fusion module is flattened through the flattening layer.

[0081] This application combines the high performance of convolutional neural networks in local feature extraction with the advantages of self-attention mechanisms in capturing global dependencies. By adopting a parallel processing strategy, the model can analyze local features and global context at the same time. The above-mentioned custom module is effectively connected in series with the traditional convolutional layer. The design of this series structure significantly improves the overall performance of the model by expanding the depth and breadth of feature extraction.

[0082] The prototype construction phase firstly uses the predefined number of categories n way , the number of support set samples for each category nsupport And the number of query set samples n query Divide the input data to form a support set x support and query set x query . Subsequently, the two sets are concatenated into a single tensor x, and the previously constructed feature extraction module (i.e., encoder) is used to perform forward propagation on it to obtain the encoded feature tensor z, in which each dimension represents the feature representation of a sample. Then, the feature part corresponding to the support set is separated from the encoded feature tensor z. The category prototype vector of the category is calculated based on the sample feature vectors of all support samples belonging to the category, specifically including: calculating the average value of the sample feature vectors of all support samples belonging to the category to obtain the average value of the category; and determining the average value as the category prototype vector of the category. Categories may include actions such as lying down, running, squatting, and jumping.

[0083] For each category c, the category prototype vector of the category is generated by calculating the average value of all supporting sample features belonging to the category Right now:

[0084]

[0085] in, represents the prototype feature vector of the cth class, represents the feature vector of the i-th support sample of the c-th class, n support is the number of support samples for each category, n way is the total number of categories.

[0086] In step 204, the probability distribution of the query sample being classified into each category is calculated according to the sample feature vector of the query sample and the category prototype vector of each category, which may include steps 401 to 403:

[0087] Step 401: Calculate the Euclidean distance between the sample feature vector of the query sample and the category prototype vector of each category;

[0088] Step 402: Negate the Euclidean distance between the sample feature vector of the query sample and the category prototype vector of each category to obtain a negative distance;

[0089] Step 403: Utilize the softmax function to convert the negative distance into the probability distribution of the query sample being classified into various categories.

[0090] The Euclidean distance is used to quantify the gap between each sample feature vector in the query set and the prototype vector of each category. This choice is mainly based on the convenience and effectiveness of the Euclidean distance in intuitively reflecting the proximity of two points in high-dimensional space. For each query sample z in the query set qWith each category prototype vector (c=1, 2, ..., C where C is the total number of categories in the query set), calculate the Euclidean distance between the query sample and the prototype vector of each category, and the distance calculation formula is as follows:

[0091]

[0092] Among them, d qc represents the distance from the query sample to the prototype vector of category c; z q is the sample feature vector of the query sample, with a length of z; is the prototype feature vector of category c, also with length z.

[0093] The distance between the sample feature vector of each query sample in the query set and the prototype vector of each category is negative, and the softmax function is applied to convert it into a probability distribution logp(y). This step maps the original geometric distance to the normalized category belonging probability, ensuring that samples with closer distances have a higher probability of being correctly classified.

[0094]

[0095] Among them, p(y=c|z q ) represents a given query sample z q The probability distribution of belonging to category c, is a normalization term that ensures that the sum of the probabilities is 1.

[0096] Next, the gather function is used to extract the corresponding probability value from the converted probability distribution according to the actual target category index, and then the average loss is calculated. This strategy ensures the reasonable feedback of the error during the model training process and provides a clear direction for optimizing the model.

[0097] In another exemplary embodiment of the present application, the above step 205 is replaced by the following steps 501 to 503:

[0098] Step 501: For each query sample, among the probability distributions of the query sample being classified into various categories, the category corresponding to the probability distribution with the largest value is determined as the predicted category of the query sample;

[0099] Step 502: Calculate the loss function value according to the predicted categories and true categories of all the query samples;

[0100] Step 503: updating the model parameters of the human behavior recognition model according to the loss function value to obtain a trained human behavior recognition model.

[0101] The total loss L is calculated using the negative value of the log-likelihood of the true category label y and the predicted probability distribution, that is, the calculation formula of the loss function value is as follows:

[0102]

[0103] Where L represents the loss function; is the indicator function. When the query sample q actually belongs to category c The value is 1, otherwise is 0; Q is the number of samples in the query set; p(y=c|z q ) represents the query sample z q The probability distribution of belonging to class c.

[0104] The human behavior recognition method based on the prototype neural network with fusion attention mechanism also includes: inputting the verification samples in the verification set into the human behavior recognition model trained with the training set to obtain the predicted category of the verification samples; calculating the accuracy rate according to the predicted category and the true category of all the verification samples to verify the model performance.

[0105] Find the maximum value in the probability distribution corresponding to each category output by the classifier, determine the predicted category of each verification sample, and compare it with its corresponding true category label to calculate the accuracy Acc. The accuracy calculation formula is as follows:

[0106]

[0107] in, is the category of the validation sample q predicted by the model. It means that when the prediction of the validation sample q is correct, it is 1, otherwise it is 0. The prediction of the validation sample is correct, which means that the predicted category of the validation sample is the same as the true category, and the prediction of the validation sample is wrong, which means that the predicted category of the validation sample is different from the true category. This process not only verifies the prediction ability of the model, but also provides an important reference indicator for further optimization of the model.

[0108] In order to verify the effectiveness of this application, a large number of experiments were conducted using a data set containing 11 types of human activity information. This embodiment verifies the performance and detection accuracy of the human behavior recognition method based on the fusion attention mechanism prototype neural network provided by this application in two real environments, a bedroom and a conference room. The experimental results show that compared with the existing methods, the reasoning accuracy of the human behavior recognition method based on the fusion attention mechanism prototype neural network provided by this application exceeds 95%, and the model training and reasoning time are greatly reduced.

[0109] This application aims to combine the high efficiency of convolutional neural networks in local feature extraction with the advantages of self-attention mechanisms in capturing global dependencies to solve the lack of accuracy in complex action recognition and cross-dataset generalization, as well as the limitations of real-time performance due to the high computational complexity of the model. This application first applies fast Fourier transform and wavelet transform to the original CSI data to filter, denoise and compress the signal to optimize data quality and improve data analysis accuracy. Then, by fusing the local feature extraction capability of the convolutional neural network and the global dependency capture capability of the self-attention mechanism, parallel processing of feature analysis is achieved. This fusion strategy not only improves the feature representation capability of the model, but also inherits the computational efficiency of the convolutional neural network and the global perception characteristics of the self-attention mechanism. In addition, this application effectively connects the custom module with the traditional convolutional layer in series, expands the depth and breadth of feature extraction, and optimizes the use of computing resources to improve the overall performance of the model.

[0110] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0111] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A human behavior recognition method based on a prototype neural network with fusion attention mechanism, characterized in that: The human behavior recognition method based on the fusion attention mechanism prototype neural network includes: Obtaining original channel state information; Preprocessing the original channel state information to obtain target channel state information corresponding to the original channel state information; The target channel state information is divided into a training set, a test set and a validation set, and a support set and a query set are determined according to the training set; the support set includes a number of support samples; the query set includes a number of query samples; Input the support set and the query set into the encoder to obtain an encoded feature tensor; the encoded feature tensor includes a sample feature vector of each support sample and a sample feature vector of each query sample; the encoder includes a first convolution module, a second convolution module, a self-attention mechanism module, a third convolution module, a fusion module and a flattening layer; the first convolution module includes a plurality of first convolution blocks connected in sequence; the second convolution module includes a plurality of second convolution blocks connected in sequence; the third convolution module includes a plurality of depth-separable convolution blocks; For each category, a category prototype vector of the category is calculated according to the sample feature vectors of all support samples belonging to the category; A classifier is used to calculate the probability distribution of the query sample being classified into each category according to the sample feature vector of the query sample and the category prototype vector of each category; the encoder and the classifier constitute a human behavior recognition model; According to the probability distribution of all the query samples being classified into various categories, the model parameters of the human behavior recognition model are updated to obtain a trained human behavior recognition model; The test samples of the test set are input into the trained human behavior recognition model to obtain the predicted category of the test samples.

2. The human behavior recognition method based on the prototype neural network with fusion attention mechanism according to claim 1 is characterized in that: Preprocessing the original channel state information to obtain target channel state information corresponding to the original channel state information specifically includes: Segmenting and completing the original channel state information to obtain completed data; Calculating the amplitude of the supplemented data to obtain amplitude data; Performing a fast Fourier transform on the amplitude data to convert the signal from the time domain to the frequency domain to obtain first transformed data; Performing wavelet transform on the first transformed data to obtain second transformed data; Performing an inverse wavelet transform on the second transformed data to obtain third transformed data; The third transformed data is converted into a torch tensor; the torch tensor is the target channel state information corresponding to the original channel state information.

3. The human behavior recognition method based on the prototype neural network with fusion attention mechanism according to claim 1 is characterized in that: Calculating the category prototype vector of the category according to the sample feature vectors of all supporting samples belonging to the category specifically includes: An average value is calculated for the sample feature vectors of all supporting samples belonging to the category to obtain an average value of the category; and the average value is determined as a category prototype vector of the category.

4. The human behavior recognition method based on the prototype neural network with fusion attention mechanism according to claim 1 is characterized in that: According to the sample feature vector of the query sample and the category prototype vector of each category, the probability distribution of the query sample being classified into each category is calculated, specifically including: Calculating the Euclidean distance between the sample feature vector of the query sample and the category prototype vector of each category; Taking the negative value of the Euclidean distance between the sample feature vector of the query sample and the category prototype vector of each category to obtain a negative distance; The softmax function is used to convert the negative distance into the probability distribution of the query sample being classified into various categories.

5. The human behavior recognition method based on the prototype neural network with fusion attention mechanism according to claim 1 is characterized in that: According to the probability distribution of all the query samples being classified into various categories, the model parameters of the human behavior recognition model are updated to obtain a trained human behavior recognition model, specifically including: For each of the query samples, determining the category corresponding to the probability distribution with the largest value among the probability distributions of the query samples being classified into various categories as the predicted category of the query sample; Calculate the loss function value according to the predicted category and the true category of all the query samples; The model parameters of the human behavior recognition model are updated according to the loss function value to obtain a trained human behavior recognition model.

6. The human behavior recognition method based on the prototype neural network with fusion attention mechanism according to claim 1 is characterized in that: The human behavior recognition method based on the fusion attention mechanism prototype neural network also includes: Input the verification samples in the verification set into the human behavior recognition model trained with the training set to obtain the predicted category of the verification samples; The accuracy is calculated based on the predicted categories and true categories of all the validation samples to verify the model performance.

7. The human behavior recognition method based on the prototype neural network with fusion attention mechanism according to claim 5 is characterized in that: The calculation formula of the loss function value is as follows: Among them, L represents the loss function; is the indicator function. When the query sample q actually belongs to category c The value is 1, otherwise is 0; Q is the number of samples in the query set; p(y=c|z q ) represents the query sample z q The probability distribution of belonging to class c.

Images