Human body activity recognition method combining residual network and attention mechanism

Abstract

The invention provides a human body activity identification method combining a residual network and an attention mechanism. Constructing a twinning neural network framework which comprises a feature extraction module, a feature distance calculation module and a similarity calculation module, and training the network framework by using the CSI data to obtain a trained twinning neural network framework; cSI data of all human body activity categories are collected to form a support set, and the CSI data of each category comprises multiple pieces of sample data; and forming a sample pair by the to-be-identified CSI sample data and each piece of sample data in the support set, calculating the similarity score of each sample pair through the trained twin neural network framework, and determining the human body activity category corresponding to the highest similarity score as the human body activity category corresponding to the to-be-identified CSI sample data. According to the method, rapid and accurate detection can still be provided under the conditions of limited data samples and limited task time, and deployment requirements in a complex environment are met.

Description

Technical Field

[0001] The present invention relates to the technical field of human activity recognition, and in particular to a human activity recognition method combining a residual network and an attention mechanism. Background Art

[0002] Human activity recognition (Channel State Information, HAR) is a core topic in the field of intelligent sensing. It accurately identifies human activities by fusing multimodal data such as video, sensors, and radar. It has demonstrated significant value in a variety of applications, including smart security, smart healthcare, and smart home appliance interaction. Among various sensing technologies, Channel State Information (CSI) from WiFi signals has garnered significant attention in human sensing tasks, particularly for identifying activities such as walking, running, and sitting. This approach has the advantages of being equipment-free, low-cost, and protecting user privacy, making it an ideal choice for continuous, real-time monitoring in various environments.

[0003] The process for CSI human activity detection based on few-shot learning follows a systematic engineering paradigm: First, a raw CSI signal matrix containing typical movements such as standing, walking, and falling is collected via WiFi transceivers. A hybrid denoising algorithm combining filtering and time-frequency domain transformation is used to eliminate multipath interference and device noise. Next, the research primarily employs meta-learning and transfer learning. By pre-training a feature extraction network in the source domain (laboratory environment), the support set-query set mechanism is used to enable real-time detection in the target domain (real-world scenarios). Performance is verified using N-way K-shot cross-validation. Siamese neural networks have become a mainstream architecture in current CSI human activity detection based on few-shot learning. Siamese neural networks, with their shared-weight dual-branch structure, map paired CSI samples into a high-dimensional embedding space for similarity measurement, and are highly favored for their excellent performance and generalization capabilities.

[0004] The core steps of the twin neural network method mainly include the following aspects: first, construct the input sample pairs, divide the data into positive sample pairs (same type samples) and negative sample pairs (different type samples), and then extract the feature representations of the two input samples respectively through a two-branch convolutional neural network with shared weights. In the feature extraction stage, the two network branches keep the parameters completely synchronized to ensure the consistency of the feature space. In the model training stage, the contrast loss or triplet loss function is used to optimize the network parameters by reducing the feature distance of similar samples and expanding the feature distance of different samples. For new input sample pairs, the network will calculate the similarity between the two feature vectors (such as Euclidean distance or cosine similarity). Finally, the preset similarity threshold is used to determine whether the samples belong to the same category, thereby realizing the classification or verification task based on metric learning.

[0005] In the field of twin neural network technology research, current research progress mainly focuses on the following core directions: First, the feature interaction enhancement mechanism, which embeds attention modules or graph neural networks in the dual-branch feature extraction process to achieve more refined feature extraction capabilities and discrimination; second, the fusion of multi-task loss functions, combining contrast loss, cross entropy loss and boundary constraint loss to optimize the geometric distribution characteristics of the feature space; finally, the optimization of the similarity calculation network, which aims to achieve more accurate calculation of the similarity between sample categories through a deep network architecture. These technological innovations have significantly improved the model's discriminative ability in tasks such as fine-grained classification, but they also bring new challenges: the increase in model complexity not only reduces the inference speed but also affects the model's generalization performance in complex scenarios. Therefore, future research needs to take into account the efficiency and generalization ability of the model while improving performance.

[0006] Existing methods for detecting human activity using CSI signals currently face challenges such as high data annotation costs, computational complexity, real-time application challenges, and difficulty distinguishing similar activities. While few-shot learning aims to quickly and accurately identify new categories, it still faces challenges related to poor generalization and confusion between similar behaviors. Summary of the Invention

[0007] An embodiment of the present invention provides a human activity recognition method that combines a residual network and an attention mechanism to effectively identify human activity category information in video data.

[0008] In order to achieve the above-mentioned purpose, the present invention adopts the following technical solutions.

[0009] A human activity recognition method combining residual network and attention mechanism, including:

[0010] Constructing a twin neural network framework, which includes a feature extraction module, a feature distance calculation module, and a similarity calculation module, and using the preprocessed channel state information (CSI) data to train the twin neural network framework to obtain a trained twin neural network framework;

[0011] Collect CSI data of all human activity categories to form a support set. Each category of CSI data includes multiple sample data.

[0012] The CSI sample data to be identified and each sample data in the support set form a sample pair, the similarity score of each sample pair is calculated through the trained twin neural network framework, and the human activity category corresponding to the highest similarity score is determined as the human activity category corresponding to the CSI sample data to be identified.

[0013] Preferably, the twin neural network framework is constructed, and the twin neural network framework includes a feature extraction module, a feature distance calculation module and a similarity calculation module, including:

[0014] Construct a twin neural network framework including a feature extraction module, a feature distance calculation module, and a similarity calculation module. The feature extraction module is composed of a serial combination of a conv_block module and an RS_block module. The conv_block module includes a sequentially connected convolution block, a batch normalization layer, and a ReLU activation function. The conv_block module is a cascaded block structure, starting with a 7×7 convolution layer with a stride of 2 and a padding of 3 to capture local features. The RS_block module integrates the ResNet and channel attention mechanism SE modules.

[0015] The similarity calculation module consists of a multi-head attention mechanism, which generates Query (Q), Key (K) and Value (V) matrices by applying linear transformation to the input vector D.

[0016] Preferably, the method further comprises:

[0017] The device based on CSI tool is used to collect video data including human activity information in the field of wireless sensing and intelligent environment monitoring. The CSI data irrelevant to the human activity of the subject is cut out from the video data, and the linear interpolation method is used to supplement the video data. The linear interpolation processing formula (1) is shown as follows:

[0018]

[0019] Where φ(x) is the data value of the location where packet loss occurs, x0 and y0 are the location and data value of the previous point adjacent to the location where packet loss occurs, x1 and y1 are the location and data value of the next point adjacent to the location where packet loss occurs, and x is the location where missing data is waiting to be supplemented.

[0020] The Hampel filter is used to remove outliers in the video data by moving the analysis window, while retaining the feature data related to the behavior to obtain the preprocessed CSI data.

[0021] Preferably, the method of training the twin neural network framework using the preprocessed CSI data to obtain a trained twin neural network framework includes:

[0022] Use the train_test_split() function to split the preprocessed CSI data into a training set and a test set. Input the two parallel CSI data (x1, x2) vectors in the training set into the feature extraction module and the convolution layer conv_block module. The conv_block module uses a cascade block structure to extract features E(x) from the two CSI data (x1, x2).

[0023] E(x)=conv_block(input) (4)

[0024] Where conv_block represents the convolution block operation, and input represents the input data consisting of two CSI data (x1, x2);

[0025] The feature E(x) output by the conv_block module is passed to the RS_block module. The RS_block module includes a squeeze operation, an excitation operation, and a scale operation. The input data of the squeeze operation is the feature information E(x) of each channel output by the conv_block module. After the squeeze operation, the output data of the squeeze operation is the information of each channel; the input data of the excitation operation is the information data of all channels output by the squeeze operation, and the output data is the calculated weight corresponding to each channel; the input data of the scale operation is the weight of each channel output by the excitation operation, and the output data is the feature information of each channel;

[0026] The feature extraction module is performed through two weight-sharing feature extraction networks, and after the operations of the conv_block module and the RS_block module, the feature vectors FE(x1) and FE(x2) corresponding to the two input CSI data vectors are generated;

[0027] The feature distance calculation module calculates the feature distance between the two feature vectors FE(x1) and FE(x2) to obtain a vector D as shown below

[0028] D=|FE(x1)-FE(x2)|(10)

[0029] Input vector D into the similarity calculation module to obtain the predicted similarity score sim between the two input CSI data vectors x1,x2 , as shown below:

[0030]

[0031] Where SC(·) represents the calculation process of the similarity calculation module;

[0032] The predicted similarity score sim is simulated using the loss function x1,x2 Compare with the true labels of two CSI data (x1, x2).

[0033]

[0034] where y i is the real label, sim i The similarity score calculated for the Siamese network model, N is the batch size;

[0035] According to the value of the loss function Back propagation is used to update the parameters of the feature extraction module and the similarity calculation module to obtain a trained twin neural network framework.

[0036] Preferably, the vector D is input into the similarity calculation module to obtain the predicted similarity score sim between the two input CSI data vectors. x1,x2 ,include:

[0037] The similarity calculation module consists of a multi-head attention mechanism, which generates Query (Q), Key (K), and Value (V) matrices by applying a linear transformation to the input vector D, as shown below:

[0038] Q=K=V=W qkv ·D(12)

[0039] Where W qkv is a linear transformation matrix with shared weights;

[0040] Convert Q, K, and V into a sequence-compatible format and calculate multi-head attention. The multi-head mechanism divides the input into 8 parallel subspaces for independent calculation and concatenates the results.

[0041]

[0042] Where Q is the query vector, K is the key vector, V is the value vector, softmax is the normalization function, and d k The length of each key (K) vector.

[0043] The fully connected layer projects the features into the classification space;

[0044] x norm =LayerNorm(x+attention_out) (14)

[0045] x is the feature distance calculation vector input to the similarity calculation module, and attention_out is the output data prediction similarity score sim of the multi-head attention calculation x1,x2 , LayerNorm is the normalization operation.

[0046] Preferably, the CSI data of all human activity categories are collected to form a support set, and the CSI data of each category includes multiple sample data, including:

[0047] Collect CSI data for all human activity categories. Each human activity category contains some data samples. The activity categories include 11 actions: waving, squatting, falling, sitting still, walking, sitting down, standing up, picking up objects, running, jumping, and lying down.

[0048] All data samples in all human activity categories constitute the support set S;

[0049] S={S1,S2,…,S n} (16)

[0050] Preferably, the steps of forming sample pairs between the CSI sample data to be identified and each sample data in the support set, calculating a similarity score for each sample pair using the trained twin neural network framework, and determining the human activity category corresponding to the highest similarity score as the human activity category corresponding to the CSI sample data to be identified include:

[0051] The CSI sample data x to be identified sample For each data sample x in each human activity category in the support set S j Form a sample pair, input each sample pair into the trained twin neural network framework, and each sample pair is processed by the above-mentioned feature extraction module, feature distance calculation module and similarity calculation module in turn to obtain a similarity score sim samplej ;

[0052] Calculate the CSI sample data x to be identified sampleThe average similarity score AvgSim between all data samples in each human activity category i , the specific method is as follows:

[0053]

[0054] Where n represents the number of human activity categories in the support set, and m represents the number of data samples for each activity category;

[0055] Select the average similarity AvgSim i The highest activity category is taken as the CSI sample data x to be identified sample The predicted classification Cate sample .

[0056] It can be seen from the technical solutions provided by the above-mentioned embodiments of the present invention that the present invention provides a method for human activity recognition that combines a residual network and an attention mechanism. This method can provide fast and accurate detection even when data samples are limited and task time is limited, and can adapt to deployment requirements in complex environments.

[0057] Additional aspects and advantages of the present invention will be set forth in part in the following description, will become apparent from the following description, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0059] Figure 1 A schematic diagram illustrating the implementation of a method for human activity recognition combining a residual network and an attention mechanism provided by an embodiment of the present invention;

[0060] Figure 2 A processing flow chart of a human activity recognition method combining a residual network and an attention mechanism provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0061] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limiting the present invention.

[0062] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the description of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or couplings. The term "and / or" used herein includes any unit and all combinations of one or more associated listed items.

[0063] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art in the art to which the present invention pertains. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and, unless defined as such herein, will not be interpreted in an idealized or overly formal sense.

[0064] To facilitate understanding of the embodiments of the present invention, several specific embodiments will be further explained below with reference to the accompanying drawings, and each embodiment does not constitute a limitation on the embodiments of the present invention.

[0065] The implementation principle of the human activity recognition method combining residual network and attention mechanism provided by the embodiment of the present invention is as follows Figure 1As shown in the figure, the main processes include data preprocessing, data partitioning, and feature extraction. In the data preprocessing stage, the present invention first normalizes the cropped and interpolated data. Next, the present invention performs time-frequency transformation and filtering on the data, converting the data from the time domain to the frequency domain and removing outliers and noise in the samples to improve the accuracy of recognition between similar actions. In the data partitioning process, the present invention uses a random sampling method to randomly generate similar or dissimilar comparison samples for each instance, thereby creating a training set and a test set. This method enhances generalization ability and can more effectively utilize limited data. For HAR, the present invention creates two datasets: one is a support set for learning category features, and the other is a query set for verifying the model's generalization ability for unseen samples. In the feature extraction process, the present invention proposes a custom module that combines ResNet with an attention mechanism to improve feature extraction by utilizing multi-scale analysis and effectively capturing basic features. The present invention then integrates this custom module into the Siamese network as an encoder. The Siamese network consists of an encoder and two shared parameter branches. The former is used to extract features and the latter is used to calculate similarity, thereby effectively classifying samples.

[0066] The processing flow of a human activity recognition method combining a residual network and an attention mechanism provided by an embodiment of the present invention is as follows: Figure 2 As shown, the processing steps include the following:

[0067] Step S1: After cropping, complementing and filtering the original video data including human activity information, time-frequency transformation is performed to convert the video data into a torch tensor.

[0068] In practical application scenarios such as smart security, smart healthcare, and smart home appliance interaction, in the smart security field, real-time human posture monitoring and fall recognition can be used to trigger automatic alarms and notify family members or community service centers, thereby achieving early warning. In the smart home appliance field, low-latency posture tracking technology can be used to achieve contactless gesture control, enhancing the convenience and energy management efficiency of multi-device collaborative operation. In the smart healthcare field, posture recognition technology can be used to monitor the standardization of patients' rehabilitation training movements or long-term bed rest posture, providing timely warnings of bedsore risks, optimizing care plans, and reducing the incidence of medical accidents. The human posture recognition method based on small-sample learning proposed in this invention uses CSI Tool-based equipment to collect data by collecting raw video data including human activity information of the subject through sensors and cameras. The signal transmitter is a Linux-based computer equipped with an Intel 5300 wireless network card and the CSI Tool installed, with a single transmitting antenna. The signal receiver is a Linux-based computer equipped with an Intel 5300 wireless network card and the CSI Tool installed, with a three-antenna signal transmission rate of 20 packets per second.

[0069] In the field of wireless sensing and intelligent environmental monitoring, CSI data is widely used for activity category recognition. It achieves non-invasive perception of different actions or scenes by capturing the multipath effect changes caused by environmental changes (such as human movement, gestures, or object displacement) during the propagation of wireless signals. The CSI data that is not related to the human activity of the subject is cropped from the above video data to remove redundant information and reduce the amount of data. Then, the linear interpolation method is used to complete the video data. The principle is shown in formula (1):

[0070]

[0071] Where φ(x) is the data value for the location where packet loss occurs, x0 and y0 are the position and data value of the previous point adjacent to the location where packet loss occurs, x1 and y1 are the position and data value of the next point adjacent to the location where packet loss occurs, and x is the location where missing data is waiting to be supplemented. The preprocessed CSI data is obtained.

[0072] Then calculate the amplitude of the cropped and completed video data to highlight the key change points of the signal. The CSI matrix representation of the video data received per unit time is shown in formula (2):

[0073]

[0074] Among them H i,j is the CSI information per unit time on the jth subcarrier for the i-th transmitting antenna and receiving antenna pair. i,j || is CSI information Hi,j The amplitude value is calculated as shown in formula (3):

[0075] ||H i,j ||=abs(H i,j ) (3)

[0076] Where abs(H) is the modulo operation on complex numbers.

[0077] Then, a Hampel filter is used to effectively remove outliers from the video data by moving the analysis window, while retaining feature data related to the behavior. In addition, Hampel filters (such as Chebyshev II filters) help minimize signal distortion when removing irrelevant noise. Finally, the filtered video data is subjected to time-frequency analysis using Fourier transform to extract the frequency domain features of the dynamic changes in the signal, thereby effectively solving the problem of activity confusion caused by time domain similarity. Finally, the data is converted into a torch tensor and the shape is adjusted to meet the input requirements of the feature extraction module. These steps ensure the cleanliness, standardization and applicability of the data before entering the feature extraction module.

[0078] Step S2: Use the train_test_split() function to split the training set and the test set, and then use random sampling to split the support set and the query set.

[0079] Use the train_test_split() function to divide the preprocessed CSI data into training and test sets. In the training and testing stages, random samples are taken from the training and test datasets to divide them into support and query sets, respectively.

[0080] Step S3: Construct a feature extraction module.

[0081] The present invention adopts the Siamese neural network method as the framework. The Siamese neural method includes three parts: feature extraction module, feature distance calculation module and similarity calculation module.

[0082] The feature extraction module consists of a serial combination of the conv_block module and the RS_block module. The conv_block module consists of a cascaded block structure consisting of a convolutional block, a batch normalization layer, a Relu activation function, and a maximum pooling layer. The RS_block module integrates the ResNet and SE (Squeeze-and-Excitation) modules. Two parallel CSI data vectors (x1, x2) are input to the feature extraction module, where the convolutional layer conv_block module extracts features from the two CSI data (x1, x2). The conv_block module begins with a 7×7 convolutional layer with a stride of 2 and padding of 3 to capture local features. After the batch normalization layer and Relu activation function, the size of the feature map is reduced through a 3×3 maximum pooling layer with a stride of 2 and padding of 1.

[0083] The conv_block module above utilizes a cascaded block structure to progressively extract features E(x) from the input data through multiple processing stages. Using an initial large 7×7 convolution kernel to rapidly expand the receptive field of the early stages, this multi-stage feature extraction mechanism transforms low-level signal patterns into high-level semantic descriptions while optimizing computational efficiency and maintaining spatial fidelity.

[0084] E(x)=conv_block(input) (4)

[0085] Where conv_block represents the convolution block operation, and input represents the input data of the feature extraction module composed of the above torch tensors. The feature E(x) contains multi-channel information.

[0086] Step S4: The present invention passes the calculation result of conv_block to the RS_block module, which integrates the ResNet and SE modules. By introducing a bypass connection in this RS_block module, the gradient vanishing problem can be effectively solved. Each RS_block is based on ResNet (residual network).

[0087] Batch normalization layers and ReLU activation functions accelerate model convergence by introducing nonlinear features, enabling the feature extraction module to fit complex functions. The combined use of these techniques improves the feature extraction module's ability to learn and classify in complex situations.

[0088] ResNet is a network that improves on the simple convolutional architecture. Therefore, convolutional layers, batch normalization, and ReLU are also indispensable components of the RS_block module. Typical ordinary convolutional networks suffer from the gradient vanishing problem when the depth increases significantly, which increases the difficulty of training. In contrast, ResNet enables the network parameters to fit the residual map instead of directly fitting the underlying map. Therefore, ResNet retains features with large gradient values, effectively alleviating the gradient vanishing problem. The original required underlying map is represented as H0(x). By introducing a "shortcut" connection, the stacked nonlinear layers fit the residual map as:

[0089] G(x)=H r (x)-x (5)

[0090] Unlike the original mapping H0(x), the mapping from the beginning to the end of a block becomes:

[0091] H r (x)=G(x)+x (6)

[0092] The residual mapping G(x) is easier to optimize than the original mapping H0(x).

[0093] This paper proposes an improved ResNet-based RS_block module by introducing the SE block stack.

[0094] The input data of the RS_block module is the feature E(x) output by the convolution block conv_block. The RS_block module contains three operations: Squeeze, Excitation, and Scale. The Squeeze operation aggregates the summary information of each feature map using adaptive global average pooling, compressing the H×W spatial dimensions of each channel to 1×1. This operation only extracts the most critical information from each channel:

[0095]

[0096] Where h is the height, w is the width of the feature vector, z c is the global information of the cth channel, x c (·) represents the activation function. The input data of the squeeze operation is the output feature E(x) of the convolution block, which is the feature information of each channel. After each squeeze operation, the output data of the squeeze operation can be obtained as the information of each channel.

[0097] The Excitation operation adopts a similar approach, using two fully connected layers and two non-linear activation functions: ReLU and sigmoid to learn the non-linear interdependencies between channels and generate channel weights.

[0098] s c =σ(W2·δ(W1·z)) (8)

[0099] Where W1 and W2 are the weight matrices of the fully connected layer, δ represents ReLU activation, σ represents Sigmoid function, and s c is the channel weight.

[0100] The input data of the Excitation operation is the information data of all channels after the squeeze operation output by the Squeeze operation, and the output data of the Excitation operation is the weight corresponding to each channel after calculation.

[0101] The Scale operation multiplies the channel weights with the original input. This operation recalibrates the importance of channels by suppressing irrelevant features and enhancing key features.

[0102] x′ c =s c ·x c (9)

[0103] where x c represents the cth channel of the original input feature map, x′ c represents the recalibrated c-th channel feature map.

[0104] The input data of the Scale operation is the weight Sc of each channel obtained after the excitation operation, and Xc is the original feature information of each channel.

[0105] By leveraging the complementary strengths of SE and ResNet, the present invention combines the conv_block module and the RS_block module to form the feature extraction module of the present invention. The model architecture of the present invention uses a layered stack of RS_block modules after the conv_block module to construct a deep feature extraction framework: the four layers contain 3, 4, 6, and 3 RS_block modules, respectively. This configuration retains the layered abstraction capabilities of ResNet while leveraging the channel attention mechanism of the SE module to achieve adaptive optimization of multi-scale features.

[0106] The feature extraction module is performed through two weight-sharing feature extraction networks. After the operations of the above-mentioned conv_block module and RS_block module, the feature vectors FE(x1) and FE(x2) corresponding to the two input CSI data vectors are generated.

[0107] Step S5: Calculate the feature distance between the two feature vectors FE(x1) and FE(x2) extracted by the feature extraction module to obtain vector D, as shown below

[0108] D=|FE(x1)-FE(x2)| (10)

[0109] Where FE(·) represents the feature extraction process of the feature extraction network. Then, the vector D is input into the similarity calculation module to obtain the predicted similarity score sim between the two input CSI data vectors. x1,x2 , as shown below:

[0110]

[0111] Where SC(·) represents the calculation process of the similarity calculation module.

[0112] The similarity calculation module of the present invention is composed of a multi-head attention mechanism, which generates Query (Q), Key (K) and Value (V) matrices by applying a linear transformation to the input vector D, as shown below:

[0113] Q=K=V=W qkv ·D(12)

[0114] Where W qkv Is a linear transformation matrix with shared weights. This step projects the vector D into the space required for self-attention calculation.

[0115] Then, Q, K, and V are converted to a sequence-compatible format and multi-head attention is calculated. The multi-head mechanism divides the input into 8 parallel subspaces for independent calculations and concatenates the results. This can capture global dependencies between features and enhance the model's sensitivity to similarity patterns in the following ways.

[0116]

[0117] Where Q is the query vector, K is the key vector, V is the value vector, softmax is the normalization function, and d k The length of each key (K) vector.

[0118] Step S6: Add the original input to the attention output (residual connection) and then perform layer normalization. Then, the fully connected layer projects the features into the classification space.

[0119] x norm =LayerNorm(x+attention_out) (14)

[0120] x is the feature distance calculation vector input to the similarity calculation module, attention_out is the output data of the multi-head attention calculation, and LayerNorm is a normalization operation that requires the feature distance calculation vector input to the similarity calculation module.

[0121] In the human activity classification part, during training, the present invention calculates similarity scores between pairs of samples and compares these predicted scores with their labels using a loss function.

[0122]

[0123] where y i is the real label, sim i The similarity score calculated for the Siamese network model, N is the batch size.

[0124] Based on the loss calculated by the loss function, backpropagation is used to update the feature extraction and similarity calculation parts. The trained model accurately measures the similarity scores between human activity data samples and determines whether two samples represent the same activity category.

[0125] The present invention collects CSI data of all activity categories, where each category contains some data samples, which are organized into a support set S containing 11 different activity categories.

[0126] S={S1,S2,…,S n}(16)

[0127] The activity categories include 11 actions: waving, squatting, falling, sitting still, walking, sitting down, standing up, picking up objects, running, jumping and lying down.

[0128] For a test sample x sample , the present invention and each active subset S in the support set S i Each sample x in j Form a sample pair. Each pair of samples is processed in turn by feature extraction and similarity calculation, and the similarity score sim is calculated. samplej .

[0129] The present invention calculates the average similarity AvgSim between the test sample and all samples of each activity category i . Select the average similarity AvgSim i The highest activity category is used as the predicted classification Cate for the test sample sample , the specific method is as follows:

[0130]

[0131] Where n represents the number of human activity categories in the support set, and m represents the number of data samples for each activity category.

[0132] In summary, the embodiments of the present invention, by integrating the hierarchical feature extraction capability of the ResNet residual network with the adaptive feature calibration characteristics of the SE channel attention mechanism, construct a collaborative architecture with multi-scale receptive field and dynamic channel weighting, aiming to overcome the problem of insufficient feature discrimination of existing action recognition models when generalizing across scene data, while reducing computational complexity while ensuring the accuracy of spatiotemporal correlation modeling. First, the present invention uses Hampel filtering and Chebyshev type II filtering to eliminate noise and outliers in the sample, and transfers data from the time domain to the frequency domain through time-frequency transformation, thereby optimizing data quality and improving data analysis accuracy. Then, the present invention combines Resnet with SE as a new custom feature extraction module. Finally, the present invention introduces a Siamese network that combines a custom module with a multi-head attention mechanism. Multi-head attention achieves robust feature comparison while reducing dependence on large training data sets. The model designed by the present invention retains the residual structure and multi-scale feature extraction advantages of ResNet, while achieving accurate capture of basic features through the attention mechanism.

[0133] To verify the effectiveness of the present invention, we constructed a real-world dataset containing 11 different human activities. Experimental results show that compared with existing algorithms, the present invention has obvious advantages: (1) the accuracy of the model is as high as 98.57%; (2) the inference speed of the model is significantly improved; (3) the generalization ability of the model in complex environments is significantly enhanced, and the confusion problem of similar actions is significantly improved.

[0134] Those skilled in the art will appreciate that the accompanying drawings are merely schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.

[0135] From the above description of the embodiments, it can be seen that those skilled in the art can clearly understand that the present invention can be implemented by means of software plus the necessary general-purpose hardware platform. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present invention or certain parts of the embodiments.

[0136] Each embodiment in this specification is described in a progressive manner. The same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the device or system embodiments, since they are basically similar to the method embodiments, the description is relatively simple. For the relevant parts, refer to the partial description of the method embodiments. The device and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. A person of ordinary skill in the art can understand and implement it without making any creative efforts.

[0137] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A human activity recognition method combining residual network and attention mechanism, characterized in that: include: Constructing a twin neural network framework, which includes a feature extraction module, a feature distance calculation module, and a similarity calculation module, and using the preprocessed channel state information (CSI) data to train the twin neural network framework to obtain a trained twin neural network framework; Collect CSI data of all human activity categories to form a support set. Each category of CSI data includes multiple sample data. The CSI sample data to be identified and each sample data in the support set form a sample pair, the similarity score of each sample pair is calculated through the trained twin neural network framework, and the human activity category corresponding to the highest similarity score is determined as the human activity category corresponding to the CSI sample data to be identified.

2. The method according to claim 1, characterized in that The twin neural network framework is constructed, which includes a feature extraction module, a feature distance calculation module and a similarity calculation module, including: Construct a twin neural network framework including a feature extraction module, a feature distance calculation module, and a similarity calculation module. The feature extraction module is composed of a serial combination of a conv_block module and an RS_block module. The conv_block module includes a sequentially connected convolution block, a batch normalization layer, and a ReLU activation function. The conv_block module is a cascaded block structure, starting with a 7×7 convolution layer with a stride of 2 and a padding of 3 to capture local features. The RS_block module integrates the ResNet and channel attention mechanism SE modules. The similarity calculation module consists of a multi-head attention mechanism, which generates Query (Q), Key (K) and Value (V) matrices by applying linear transformation to the input vector D.

3. The method according to claim 1, characterized in that The method further comprises: The device based on CSI tool is used to collect video data including human activity information in the field of wireless sensing and intelligent environment monitoring. The CSI data irrelevant to the human activity of the subject is cut out from the video data, and the linear interpolation method is used to supplement the video data. The linear interpolation processing formula (1) is shown as follows: Where φ(x) is the data value of the location where packet loss occurs, x0 and y0 are the location and data value of the previous point adjacent to the location where packet loss occurs, x1 and y1 are the location and data value of the next point adjacent to the location where packet loss occurs, and x is the location where missing data is waiting to be supplemented. The Hampel filter is used to remove outliers in the video data by moving the analysis window, while retaining the feature data related to the behavior to obtain the preprocessed CSI data.

4. The method according to claim 3, characterized in that The method of using the preprocessed CSI data to train the twin neural network framework to obtain a trained twin neural network framework includes: Use the train_test_split() function to divide the preprocessed CSI data into a training set and a test set. Input the two parallel CSI data (x1, x2) vectors in the training set into the feature extraction module and the convolution layer conv_block module. The conv_block module uses a cascade block structure to extract features E(x) from the two CSI data (x1, x2). E(x)=conv_block(input) (4) Where conv_block represents the convolution block operation, and input represents the input data consisting of two CSI data (x1, x2); The feature E(x) output by the conv_block module is passed to the RS_block module. The RS_block module includes a squeeze operation, an excitation operation, and a scale operation. The input data of the squeeze operation is the feature information E(x) of each channel output by the conv_block module. After the squeeze operation, the output data of the squeeze operation is the information of each channel; the input data of the excitation operation is the information data of all channels output by the squeeze operation, and the output data is the calculated weight corresponding to each channel; the input data of the scale operation is the weight of each channel output by the excitation operation, and the output data is the feature information of each channel; The feature extraction module is performed through two weight-sharing feature extraction networks, and after the operations of the conv_block module and the RS_block module, the feature vectors FE(x1) and FE(x2) corresponding to the two input CSI data vectors are generated; The feature distance calculation module calculates the feature distance between the two feature vectors FE(x1) and FE(x2) to obtain a vector D as shown below D=| FE(x1)-FE(x2) | (10) Input vector D into the similarity calculation module to obtain the predicted similarity score sim between the two input CSI data vectors x1,x2 , as shown below: Where SC(·) represents the calculation process of the similarity calculation module; The predicted similarity score sim is simulated using the loss function x1,x2 Compare with the true labels of two CSI data (x1, x2); where y i is the real label, sim i The similarity score calculated for the Siamese network model, N is the batch size; According to the value of the loss function Back propagation is used to update the parameters of the feature extraction module and the similarity calculation module to obtain a trained twin neural network framework.

5. The method according to claim 4, characterized in that The vector D is input into the similarity calculation module to obtain the predicted similarity score sim between the two input CSI data vectors. x1,x2 ,include: The similarity calculation module consists of a multi-head attention mechanism, which generates Query (Q), Key (K), and Value (V) matrices by applying a linear transformation to the input vector D, as shown below: Q=K=V=W qkv ·D (12) Where W qkv is a linear transformation matrix with shared weights; Convert Q, K, and V into a sequence-compatible format and calculate multi-head attention. The multi-head mechanism divides the input into 8 parallel subspaces for independent calculation and concatenates the results. Where Q is the query vector, K is the key vector, V is the value vector, softmax is the normalization function, and d k is the length of each key (K) vector; The fully connected layer projects the features into the classification space; x norm =LayerNorm(x+attention_out) (14) x is the feature distance calculation vector input to the similarity calculation module, and attention_out is the output data prediction similarity score sim of the multi-head attention calculation x1,x2 , LayerNorm is the normalization operation.

6. The method according to claim 4 or 5, characterized in that The CSI data of all human activity categories are collected to form a support set. Each category of CSI data includes multiple sample data, including: Collect CSI data for all human activity categories. Each human activity category contains some data samples. The activity categories include 11 actions: waving, squatting, falling, sitting still, walking, sitting down, standing up, picking up objects, running, jumping, and lying down. All data samples in all human activity categories constitute the support set S; S={S1,S2,…,S n } (16)。 7. The method according to claim 6, characterized in that The method of forming sample pairs of the CSI sample data to be identified and each sample data in the support set, calculating the similarity score of each sample pair using the trained twin neural network framework, and determining the human activity category corresponding to the highest similarity score as the human activity category corresponding to the CSI sample data to be identified includes: The CSI sample data x to be identified sample For each data sample x in each human activity category in the support set S j Form a sample pair, input each sample pair into the trained twin neural network framework, and each sample pair is processed by the above-mentioned feature extraction module, feature distance calculation module and similarity calculation module in turn to obtain a similarity score sim samplej ; Calculate the CSI sample data x to be identified sample The average similarity score AvgSim between all data samples in each human activity category i , the specific method is as follows: Where n represents the number of human activity categories in the support set, and m represents the number of data samples for each activity category; Select the average similarity AvgSim i The highest activity category is taken as the CSI sample data x to be identified sample The predicted classification Cate sample .

Images