Human behavior recognition method, device, equipment and medium

Abstract

Embodiments of the present invention disclose a method, apparatus, device, and medium for human behavior recognition. The method converts collected data into a two-dimensional feature map using a Gram angle difference field. Features are then extracted using a visual transformer module combined with a fractional Fourier transform and a recurrent cross-attention module. The features extracted by the two modules are then input into a fusion module for integration, resulting in a more comprehensive and effective feature set. This more comprehensive and effective feature set enables accurate human behavior recognition, meeting practical application requirements.

Description

Technical Field

[0001] The present invention relates to the field of behavior recognition technology, and in particular to a method, device, equipment and medium for human behavior recognition. Background Art

[0002] With the rapid development of artificial intelligence (AI) and the widespread adoption of the Internet of Things (IoT), human behavior recognition is becoming increasingly important in smart environments. In smart homes, identifying occupants' behavior enables automated control of home appliances, improving living comfort and convenience. In health monitoring, it can capture a person's movement and physiological activity in real time, providing data support for disease prevention and treatment. In security scenarios, it can promptly detect abnormal behavior and issue alerts to ensure public safety. In sports tracking, it can accurately analyze athletes' movements, aiding training and improving competitive performance.

[0003] Traditional human activity recognition methods primarily rely on traditional sensors for data collection and analysis. While these methods can achieve a certain degree of behavior recognition, they also have numerous limitations. Human activity recognition based on WiFi channel state information (CSI) protects personal privacy and security, eliminating the need for any equipment to be present, and has become a hot topic of research. With the maturity of deep learning technology, research has applied deep learning methods to CSI-based human activity recognition. For example, deep learning networks such as convolutional neural networks (CNNs) and long short-term memory (LSTMs) have been used in CSI-based activity recognition. While recognition accuracy has improved, the performance of these methods depends heavily on complex feature processing and model architecture. These methods require significant computing resources and have long response times. Using lightweight models can reduce the accuracy of human activity recognition. Summary of the Invention

[0004] Embodiments of the present invention provide a human behavior recognition method, apparatus, device, and medium to solve the technical problem in the prior art of reduced accuracy in human behavior recognition using CSI using a lightweight model.

[0005] In a first aspect, an embodiment of the present invention provides a method for identifying human behavior, comprising:

[0006] Converting the collected human behavior into a two-dimensional image using a Gram angle difference field, so that the two-dimensional image retains the temporal information of the CSI data;

[0007] Dividing the two-dimensional image into multiple token sequences, performing a fractional Fourier transform of the transformation angle on the normalized output of each coding layer when encoding the token sequences, and adding the transformed result to the output of the previous layer as the input of the next layer or the final output of the coding layer;

[0008] Normalize the output of the encoding layer, and input the normalized result into the multi-layer perceptron for processing to obtain the processing result;

[0009] Applying a one-dimensional discrete fractional Fourier transform to the processing result along the hidden dimension to perform a first transformation to obtain a first transformation result, performing a one-dimensional discrete fractional Fourier transform on the first transformation result along the sequence dimension to obtain a second transformation result, and obtaining the real part of the second transformation result as the first output feature;

[0010] The two-dimensional image is input into the repeated cross-attention module to obtain the second output feature with dense and rich context information;

[0011] The first output feature and the second output feature are fused, and the fusion result is input into the fully connected layer to output the human behavior recognition result.

[0012] In a second aspect, an embodiment of the present invention further provides a human behavior recognition device, comprising:

[0013] a two-dimensional image conversion module, configured to convert the collected human behavior into a two-dimensional image using a Gram angle difference field, so that the two-dimensional image retains the temporal information of the CSI data;

[0014] a partitioning module, configured to partition the two-dimensional image into a plurality of token sequences, and when encoding the token sequences, perform a fractional Fourier transform of the transformation angle on the normalized output of each coding layer, and add the transformed result to the output of the previous layer as the input of the next layer or the final output of the coding layer;

[0015] The normalization module is used to normalize the output of the encoding layer and input the normalization result into the multi-layer perceptron for processing to obtain the processing result;

[0016] a transformation module, configured to apply a one-dimensional discrete fractional Fourier transform to the processing result along the hidden dimension for a first transformation to obtain a first transformation result, perform a one-dimensional discrete fractional Fourier transform on the first transformation result along the sequence dimension to obtain a second transformation result, and obtain the real part of the second transformation result as the first output feature;

[0017] An input module, configured to input a two-dimensional image into a repeated cross-attention module to obtain a second output feature with dense and rich context information;

[0018] The output module is used to fuse the first output feature and the second output feature, input the fusion result into the fully connected layer, and output the human behavior recognition result.

[0019] In a third aspect, an embodiment of the present invention further provides a device, including:

[0020] one or more processors;

[0021] a storage device for storing one or more programs;

[0022] When the one or more programs are executed by the one or more processors, the one or more processors implement the human behavior recognition method provided in the above embodiment.

[0023] In a fourth aspect, an embodiment of the present invention further provides a storage medium comprising computer-executable instructions, which, when executed by a computer processor, are used to execute the human behavior recognition method provided in the above embodiment.

[0024] The human behavior recognition method, apparatus, device and medium provided by the embodiments of the present invention convert the collected human behavior into a two-dimensional image by utilizing the Gram angle difference field, so that the two-dimensional image retains the time information of the CSI data; the two-dimensional image is divided to form multiple token sequences, and when encoding the token sequence, the output normalization processing result of each coding layer is subjected to a fractional-order Fourier transform of the transformation angle, and the transformation result is added to the output result of the previous layer as the input of the next layer or the final output of the coding layer; the output of the coding layer is normalized, and the normalization processing result is input into a multi-layer perceptron for processing to obtain a processing result; the processing result is subjected to a one-dimensional discrete fractional-order Fourier transform along the hidden dimension for a first transformation to obtain a first transformation result, and the first transformation result is subjected to a one-dimensional discrete fractional-order Fourier transform along the sequence dimension to obtain a second transformation result, and the real part of the second transformation result is obtained as the first output feature; the two-dimensional image is input into a repeated cross-attention module to obtain a second output feature with dense and rich contextual information; the first output feature and the second output feature are fused, and the fusion result is input into a fully connected layer to output a human behavior recognition result. Utilizing GADF technology, the collected data is converted into two-dimensional feature maps, making the data features more intuitive and easier to analyze. Feature extraction is performed on the two-dimensional feature maps from multiple perspectives, mining rich behavioral feature information. The extracted features are then integrated using a fusion module, making the feature set more comprehensive and effective, enabling accurate recognition of human behavior using lightweight models. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0026] Figure 1 1 is a flow chart of a method for identifying human behavior provided in the first embodiment of the present invention;

[0027] Figure 2 Schematic diagram of the test scenario in the human behavior recognition method provided in the first embodiment;

[0028] Figure 3 1 is a flow chart of a method for identifying human behavior provided in the second embodiment of the present invention;

[0029] Figure 4 Schematic diagram of the structure of the human behavior recognition device provided by the third embodiment of the present invention;

[0030] Figure 5 It is a structural diagram of the device provided in Example 4 of the present invention. DETAILED DESCRIPTION

[0031] The present invention will be further described in detail below with reference to the accompanying drawings and examples. It will be understood that the specific embodiments described herein are intended only to illustrate the present invention and are not intended to limit the present invention. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions relevant to the present invention, not all structures.

[0032] Example 1

[0033] Figure 1 : is a flow chart of a human behavior recognition method provided in Example 1 of the present invention. This embodiment is applicable to situations where human behavior is recognized using CSI. The method can be performed by a human behavior recognition device and specifically includes the following steps:

[0034] Step 110: Convert the collected human behavior into a two-dimensional image using the Gram angle difference field, so that the two-dimensional image retains the time information of the CSI data.

[0035] Exemplarily, the method of converting the collected human behavior into a two-dimensional image using the Gram angle difference field includes: obtaining a CSI data matrix generated when a person performs an action, where the elements in the matrix include an in-phase component, a quadrature component, a phase value, and an amplitude value corresponding to each signal; normalizing all amplitude values, converting the normalized signal into polar coordinates, and calculating the angle of the polar coordinates; and transforming the signal converted into polar coordinates into an image using GADF.

[0036] Specifically, a human behavior monitoring system based on Wi-Fi signals can be built as the basic platform. The tester performs actions at a specified location and collects data to ensure that the data source and collection method are standardized and reliable. The collected data is converted into a two-dimensional feature map through GADF, and the feature map is input into the VIT-FrFT module and the RCCA module respectively to extract the corresponding features. The extracted features are then input into the fusion module for feature fusion. The fused features are input and output into the module for training data, allowing the system to learn the mapping relationship between behavioral features and categories.

[0037] Figure 2 This is a schematic diagram of the test scenario in the human behavior recognition method provided in this embodiment. Figure 2 , you can select an indoor environment for CSI data collection, set up multiple sampling points in fixed positions and evenly arranged in the environment to ensure that the sampling points can fully and reasonably cover the collection area. The tester performs actions such as waving, clapping, squatting, front kicking, side kicking, sitting down, and standing up at each point. Place a mobile device equipped with a network card that supports CSI data collection function in front of the sampling range, and place a router in the back of the sampling range. Use the data collection tool that supports the network card to collect CSI data collection function to collect the CSI data matrix generated when the tester performs the action , is included The information matrix of the data packet is ,in , , represents the imaginary part, and denote the in-phase component and the quadrature component respectively, and Represent the phase value and amplitude value respectively.

[0038] The Gramian Angular Difference Field (GADF) technique is used to convert the collected CSI data into a two-dimensional image. Normalize so that all input values are within In the range, get , the normalized signal Convert to polar coordinates ,in is the signal length, Representation sample The corresponding angle value, then, GADF is used to transform the signal into an image.

[0039] The transformed image can be represented by a matrix:

[0040] , Is the size of The matrix composed of the sine functions of any two angle differences

[0041] in, It reflects the sine value of the difference between different angle values. The matrix composed of these elements is used to represent the transformed two-dimensional image. , , and is the normalized signal value, 、 Represents samples 、 The corresponding angle value, The value range of and j is related to the signal length. Since the number of data packets is related to the measurement time, the generated feature image retains the time information of the CSI data.

[0042] Step 120: Divide the two-dimensional image into multiple token sequences. When encoding the token sequence, perform a fractional Fourier transform of the transformation angle on the output normalization result of each coding layer, and add the transformation result to the output result of the previous layer as the input of the next layer or the final output of the coding layer.

[0043] Optionally, dividing the two-dimensional image to form multiple token sequences may include: dividing the two-dimensional feature map into multiple blocks; and mapping each block through a linear layer to obtain multiple token sequences of fixed size.

[0044] For example, the two-dimensional image obtained by GADF transformation can be used as the input feature map of the VIT-FrFT module. , whose dimensions are expressed as , is the number of channels of the image, is the image size.

[0045] In this embodiment, in order to improve the accuracy of human behavior recognition, it is necessary to extract invisible features as much as possible. Therefore, in this embodiment, the improved Transformer encoder structure can be used. Transformer usually processes sequence data. In order to make the image data meet the input requirements of Transformer, the two-dimensional feature map is divided into a token sequence. After the feature map is subjected to the Patch Embedding operation, a fixed-size token sequence is obtained. The dimension of each token is , the entire sequence length is , and finally reshaped into , which is to divide the input image into multiple tokens .

[0046] Correspondingly, the method can also add the following steps: perform D-dimensional linear mapping on each token, map it to D dimensions, and express the mapped result using a linear layer, where the expression result includes: mapping matrix and position encoding to enhance the perception of element position information in the image.

[0047] For example, a linear layer can be used to represent ,in is a learnable sequence of embeddings, is the mapping matrix, Encode the position to enhance the model's perception of the position information of elements in the image, and then embed the sequence Sent to the first half of the Transformer encoder, the calculation method is , specifically, for each layer ( , is the number of encoders), first the output of the previous layer Perform layer normalization (LN), and then transform the angle to Fractional Fourier transform (FrFT) of , add the changed result to the output of the previous layer to get the output of the current layer Through the above series of operations, the encoding process of the input is gradually completed, and the feature representation containing context information and position information is extracted. The specific calculation method is: given a sequence , the discrete fractional Fourier transform (DFrFT) can be defined as: when , D is an integer, ,but ,when , D is an integer, ,but ;

[0048] In step 130, the output of the coding layer is normalized, and the normalized result is input into a multi-layer perceptron for processing to obtain a processing result.

[0049] The features extracted by the first half of the Transformer encoder It is further normalized and the result is input into a Multi-Layer Perceptron (MLP) block.

[0050] Accordingly, the normalization result is input into the multi-layer perceptron for processing to obtain the processing result, including:

[0051] The multilayer perceptron includes: two linear layers and a nonlinear activation layer Gaussian error linear unit, which performs a two-dimensional discrete fractional Fourier transform operation on the features obtained after being processed by the multilayer perceptron, and transforms them from the sequence dimension and the hidden dimension respectively to obtain the processing results.

[0052] The extracted features are Normalize it one step and input the result into the Multi-Layer Perceptron (MLP) block, which can be expressed as , the MLP block consists of two linear layers and a nonlinear activation layer Gaussian error linear unit.

[0053] In step 140, a one-dimensional discrete fractional Fourier transform is applied to the processed result along the hidden dimension to perform a first transformation to obtain a first transformation result. A one-dimensional discrete fractional Fourier transform is performed on the first transformation result along the sequence dimension to obtain a second transformation result. The real part of the second transformation result is obtained as the first output feature.

[0054] Perform a two-dimensional discrete fractional Fourier transform (2D-DFrFT) operation on the features processed by the MLP block. Specifically, After being processed by LN and MLP blocks, we get ,Then 2D-DFrFT transforms from two dimensions (sequence dimension and hidden dimension) respectively, captures the dynamic change information of human behavior in time and space order, and mines the potential correlation between different feature channels, so as to extract more comprehensive and advanced features, improve the performance of human behavior recognition model, and provide multi-domain context information. Specifically, the system first transforms the input features, that is, after LN and MLP blocks, , and apply one-dimensional discrete fractional Fourier transform (1D-DFrFT) along the hidden dimension to transform, i.e. , apply 1D-DFrFT to the result of the previous step, along the sequence dimension Perform the transformation, and finally, by taking the real part, we get the output feature Z, that is, 2D-DFrFT can be expressed as , since FrFT has additive characteristics, that is, , each encoder block is in the spatial frequency plane with The input features are transformed, so the addition of two FrFTs of different orders results in a larger transformation angle. In each block, the output of the FrFT is multiplied in the MLP sublayer in the Fourier domain. The multiplication in the Fourier domain is the fractional-order convolution in the spatial domain, that is, The 2D-DFrFT block represents a structure that performs spatial domain fractional-order convolution with different transformation angles. The connection between FrFT blocks is used to provide multi-domain context information. Using the above method, the first output feature can be obtained.

[0055] In step 150 , the two-dimensional image is input into a repeated cross-attention module to obtain a second output feature with dense and rich context information.

[0056] For example, the feature map Perform cross attention (CCA) operation, specifically, Applied in two The convolutional layer generates two feature maps and ,in , is the number of channels, dimension ratio Small, for feature maps Every position , extract the feature vector at that position , from the feature map Extract and position All eigenvectors in the same row or column constitute the set , calculate Q u With K u The dot product of each eigenvector in is used to obtain the relevance score. ,in , is Q u With K u The degree of correlation, is the correlation matrix of all positions, Apply the softmax operation on it to get the attention map ,Right now , then, yes Apply another The convolutional layer generates feature maps , for each position ,from Extract and position All eigenvectors in the same row or column constitute the set , yes Center and position A collection of feature vectors in the same row or column, using attention mapping right Perform weighted summation and compare the result with the original feature X u Add together to get the enhanced feature map ,in is the position in the output feature map u The feature vector of context information is added to the local features To enhance the representation of relevant information, therefore, It has a wide contextual receptive field and selectively aggregates contextual information based on the spatial attention map.

[0057] Furthermore, in order to solve the characteristics of the CCA module Figure 1 To solve the connection problem between pixels and their surrounding pixels that are not in the cross path, we introduce a cyclic cross attention (RCCA) module. The RCCA module is equipped with two cycles and can collect full-image context information from all pixels to generate new features with dense and rich context information.

[0058] Step 160: Fuse the first output feature and the second output feature, input the fusion result into a fully connected layer, and output a human behavior recognition result.

[0059] For example, the first output feature and the second output feature can be fused using an existing fusion method using a fusion layer, and the fused feature can be input into a fully connected layer to achieve subsequent classification and regression tasks, thereby outputting the human behavior recognition result.

[0060] This embodiment converts the collected human behavior into a two-dimensional image by utilizing the Gram angle difference field, so that the two-dimensional image retains the time information of the CSI data; the two-dimensional image is divided to form multiple token sequences, and when encoding the token sequence, the output normalization processing result of each coding layer is subjected to a fractional-order Fourier transform of the transformation angle, and the transformation result is added to the output result of the previous layer as the input of the next layer or the final output of the coding layer; the output of the coding layer is normalized, and the normalization processing result is input into a multi-layer perceptron for processing to obtain a processing result; the processing result is subjected to a first transformation by applying a one-dimensional discrete fractional-order Fourier transform along the hidden dimension to obtain a first transformation result, and the first transformation result is subjected to a one-dimensional discrete fractional-order Fourier transform along the sequence dimension to obtain a second transformation result, and the real part of the second transformation result is obtained as the first output feature; the two-dimensional image is input into a repeated cross-attention module to obtain a second output feature with dense and rich contextual information; the first output feature and the second output feature are fused, and the fusion result is input into a fully connected layer to output a human behavior recognition result. Utilizing GADF technology, the collected data is converted into two-dimensional feature maps, making the data features more intuitive and easier to analyze. Feature extraction is performed on the two-dimensional feature maps from multiple perspectives, mining rich behavioral feature information. The extracted features are then integrated using a fusion module, making the feature set more comprehensive and effective, enabling accurate recognition of human behavior using lightweight models.

[0061] Example 2

[0062] Figure 3 This is a flow chart of the human behavior recognition method provided in the second embodiment of the present invention. This embodiment is optimized based on the above embodiment, and the first output feature and the second output feature are integrated. The specific optimization is: the first output feature and the second output feature are connected into a connection matrix, and the connection matrix is linearly transformed using a fully connected layer, and the attention weight is calculated.

[0063] See also Figure 3 , the human behavior recognition method comprises:

[0064] Step 210 : Convert the collected human behavior into a two-dimensional image using the Gram angle difference field, so that the two-dimensional image retains the time information of the CSI data.

[0065] Step 220: Divide the two-dimensional image into multiple token sequences. When encoding the token sequence, perform a fractional Fourier transform of the transformation angle on the output normalization processing result of each coding layer, and add the transformation result to the output result of the previous layer as the input of the next layer or the final output of the coding layer.

[0066] In step 230, the output of the coding layer is normalized, and the normalized result is input into a multi-layer perceptron for processing to obtain a processing result.

[0067] In step 240, the processing result is transformed for the first time by applying a one-dimensional discrete fractional Fourier transform along the hidden dimension to obtain a first transformation result. The first transformation result is transformed for the second time by applying a one-dimensional discrete fractional Fourier transform along the sequence dimension to obtain a second transformation result. The real part of the second transformation result is obtained as the first output feature.

[0068] In step 250, the two-dimensional image is input into a repeated cross-attention module to obtain a second output feature with dense and rich context information.

[0069] Step 260: Connect the first output feature and the second output feature into a connection matrix, perform a linear transformation on the connection matrix using a fully connected layer, calculate the attention weight, and input the fusion result into the fully connected layer to output the human behavior recognition result.

[0070] Connect the features F1 and F2 extracted by the two branches into a matrix ,in , is the dimension of the feature, through a fully connected layer Perform linear transformation to obtain the matrix ,in, W Fis a learnable weight matrix, , and then calculate the attention weight ,in, is the transpose, W ω Is a learnable vector generated by the model. During the training process, the model adjusts W ω The value of , so that it can better focus on the most important part of the input features for the task, and the weight Effect 、 On the fusion feature In this way, the network can adaptively adjust the contribution of RCCA and VIT-FrFT extracted features according to the self-attention mechanism, thereby achieving more effective feature fusion.

[0071] Fusion Features Input the fully connected layer to get the score vector x used to represent each category i , , Represents the set of all behavior categories, the output x of the fully connected layer i The softmax function will be further used to calculate the predicted probability of each category ,Right now ,e is a natural constant; by minimizing the loss function Train the model, where y i Represents the true label of the sample; enables the trained model to make accurate predictions when faced with new data.

[0072] This embodiment optimizes the fusion of the first and second output features by concatenating them into a connection matrix, performing a linear transformation on the connection matrix using a fully connected layer, and calculating attention weights. This approach allows for a flexible weight adjustment mechanism, allowing the two output features to be adaptively adjusted based on the self-attention mechanism. This allows for effective feature fusion, further enhancing the accuracy of human behavior recognition.

[0073] In a preferred implementation of this embodiment, the method may further include the following steps: using a minimum loss function for training to update the transformation angle and attention weight. The model is trained so that it can make accurate predictions when faced with new data. During the training process, the transformation angle α and W ω The value of is adjusted through back propagation, so that the entire model can accurately identify human behavior.

[0074] Example 3

[0075] Figure 4 This is a schematic diagram of the structure of the human behavior recognition device provided by the third embodiment of the present invention. Figure 4 , the human behavior recognition device comprises:

[0076] A two-dimensional image conversion module 310 is configured to convert the collected human behavior into a two-dimensional image using a Gram angle difference field, so that the two-dimensional image retains the time information of the CSI data;

[0077] a partitioning module 320 configured to partition the two-dimensional image into a plurality of token sequences, and when encoding the token sequences, perform a fractional Fourier transform of the transformation angle on the normalized output of each coding layer, and add the transformed result to the output of the previous layer as the input of the next layer or the final output of the coding layer;

[0078] A normalization module 330 is used to normalize the output of the coding layer and input the normalization result into the multi-layer perceptron for processing to obtain a processing result;

[0079] Transformation module 340, configured to apply a one-dimensional discrete fractional Fourier transform to the processing result along the latent dimension to perform a first transformation to obtain a first transformation result, perform a one-dimensional discrete fractional Fourier transform on the first transformation result along the sequence dimension to obtain a second transformation result, and obtain the real part of the second transformation result as the first output feature;

[0080] An input module 350 is configured to input the two-dimensional image into a repeated cross-attention module to obtain a second output feature with dense and rich context information;

[0081] The output module 360 is used to fuse the first output feature and the second output feature, input the fusion result into the fully connected layer, and output the human behavior recognition result.

[0082] The human behavior recognition device provided by this embodiment converts the collected human behavior into a two-dimensional image by using the Gram angle difference field, so that the two-dimensional image retains the time information of the CSI data; the two-dimensional image is divided into multiple token sequences, and when encoding the token sequence, the output normalization processing result of each coding layer is subjected to a fractional-order Fourier transform of the transformation angle, and the transformation result is added to the output result of the previous layer as the input of the next layer or the final output of the coding layer; the output of the coding layer is normalized, and the normalization processing result is input into a multi-layer perceptron for processing to obtain a processing result; the processing result is subjected to a first transformation by applying a one-dimensional discrete fractional-order Fourier transform along the hidden dimension to obtain a first transformation result, and the first transformation result is subjected to a one-dimensional discrete fractional-order Fourier transform along the sequence dimension to obtain a second transformation result, and the real part of the second transformation result is obtained as the first output feature; the two-dimensional image is input into a repeated cross-attention module to obtain a second output feature with dense and rich contextual information; the first output feature and the second output feature are fused, and the fusion result is input into a fully connected layer to output a human behavior recognition result. Utilizing GADF technology, the collected data is converted into two-dimensional feature maps, making the data features more intuitive and easier to analyze. Feature extraction is performed on the two-dimensional feature maps from multiple perspectives, mining rich behavioral feature information. The extracted features are then integrated using a fusion module, making the feature set more comprehensive and effective, enabling accurate recognition of human behavior using lightweight models.

[0083] Based on the above embodiments, the two-dimensional image conversion module includes:

[0084] An acquisition unit is used to acquire the CSI data matrix generated when a person performs an action. The elements in the matrix include the in-phase component, quadrature component, phase value, and amplitude value corresponding to each signal;

[0085] a conversion unit, configured to normalize all amplitude values, convert the normalized signals into polar coordinates, and calculate the angles of the polar coordinates;

[0086] The transformation module is used to transform the signal converted into polar coordinates into an image using GADF.

[0087] Based on the above embodiments, the division module includes:

[0088] A segmentation unit, used to segment the two-dimensional feature map into multiple blocks;

[0089] The mapping unit is used to map each block through a linear layer to obtain multiple token sequences of fixed size.

[0090] Based on the above embodiments, the device further includes:

[0091] The expression module is used to perform D-dimensional linear mapping on each token, map it to D dimensions, and express the mapped result using a linear layer. The expression result includes: mapping matrix and position encoding to enhance the perception of element position information in the image.

[0092] Based on the above embodiments, the normalization module includes:

[0093] The respective transformation units are used to perform a two-dimensional discrete fractional Fourier transform operation on the features obtained after processing by the multi-layer perceptron, and transform them from the sequence dimension and the hidden dimension respectively to obtain the processing results. The multi-layer perceptron includes: two linear layers and a nonlinear activation layer Gaussian error linear unit.

[0094] Based on the above embodiments, the fusion module includes:

[0095] An attention weight calculation unit is used to connect the first output feature and the second output feature into a connection matrix, perform a linear transformation on the connection matrix using a fully connected layer, and calculate the attention weight.

[0096] Based on the above embodiments, the device further includes:

[0097] An updating module is used to perform training using a minimum loss function and to update the transformation angle and attention weight.

[0098] The human behavior recognition device provided in the embodiment of the present invention can execute the human behavior recognition method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0099] Example 4

[0100] Figure 5 A schematic structural diagram of a device provided in Example 4 of the present invention. Figure 5 A block diagram of an exemplary device 12 suitable for use in implementing embodiments of the present invention is shown. Figure 5 The device 12 shown is only an example and should not bring any limitation to the functionality and scope of use of the embodiments of the present invention.

[0101] like Figure 5 As shown, device 12 is implemented as a general-purpose computing device. Components of device 12 may include, but are not limited to, one or more processors or processing units 16, a system memory 28, and a bus 18 that connects various system components (including system memory 28 and processing unit 16).

[0102] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures. Examples of these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0103] Device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by device 12, including volatile and non-volatile media, removable and non-removable media.

[0104] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be configured to read and write non-removable, non-volatile magnetic media ( Figure 5 Not shown, usually called a "hard drive"). Although Figure 5 Although not shown, a magnetic disk drive for reading and writing to a removable non-volatile magnetic disk (e.g., a "floppy disk"), as well as an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. Memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of various embodiments of the present invention.

[0105] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may include an implementation of a network environment. Program modules 42 generally implement the functions and / or methodologies of the embodiments described herein.

[0106] Device 12 may also communicate with one or more external devices 14 (e.g., a keyboard, pointing device, display 24, etc.), one or more devices that enable a user to interact with device 12, and / or any device that enables device 12 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). This communication may occur via input / output (I / O) interface 22. Furthermore, device 12 may communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via network adapter 20. As shown, network adapter 20 communicates with other modules of device 12 via bus 18. It should be understood that, although not shown, other hardware and / or software modules may be used in conjunction with device 12, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0107] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing the human behavior recognition method provided by the embodiment of the present invention.

[0108] Example 5

[0109] The fifth embodiment of the present invention further provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to execute any of the human behavior recognition methods provided in the above embodiments.

[0110] The computer storage medium of the embodiments of the present invention may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or component.

[0111] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0112] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0113] Computer program code for performing the operations of the present invention may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or device. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0114] Note that the above are only preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments and may include many other equivalent embodiments without departing from the concept of the present invention. The scope of the present invention is determined by the scope of the appended claims.

Claims

1. A human behavior recognition method, characterized in that: include: Converting the collected human behavior into a two-dimensional image using a Gram angle difference field, so that the two-dimensional image retains the temporal information of the CSI data; Dividing the two-dimensional image into multiple token sequences, performing a fractional Fourier transform of the transformation angle on the normalized output of each coding layer when encoding the token sequences, and adding the transformed result to the output of the previous layer as the input of the next layer or the final output of the coding layer; Normalize the output of the encoding layer, and input the normalized result into the multi-layer perceptron for processing to obtain the processing result; Applying a one-dimensional discrete fractional Fourier transform to the processing result along the hidden dimension to perform a first transformation to obtain a first transformation result, performing a one-dimensional discrete fractional Fourier transform on the first transformation result along the sequence dimension to obtain a second transformation result, and obtaining the real part of the second transformation result as the first output feature; The two-dimensional image is input into the repeated cross-attention module to obtain the second output feature with dense and rich context information; The first output feature and the second output feature are fused, and the fusion result is input into the fully connected layer to output the human behavior recognition result.

2. The method according to claim 1, characterized in that The method of converting the collected human behavior into a two-dimensional image by using the Gram angle difference field includes: Obtain the CSI data matrix generated when the person performs the action. The elements in the matrix include the in-phase component, quadrature component, phase value, and amplitude value corresponding to each signal; Normalize all amplitude values, convert the normalized signals into polar coordinates, and calculate the angles of the polar coordinates; The signal converted into polar coordinates is transformed into an image using GADF.

3. The method according to claim 1, characterized in that The dividing the two-dimensional image to form multiple token sequences includes: Split the two-dimensional feature map into multiple blocks; Each block is mapped through a linear layer to obtain multiple token sequences of fixed size.

4. The method according to claim 3, characterized in that The method further comprises: Perform a D-dimensional linear mapping on each token, map it to D dimensions, and express the mapped result using a linear layer. The expression result includes: mapping matrix and position encoding to enhance the perception of element position information in the image.

5. The method according to claim 1, wherein The normalization result is input into the multi-layer perceptron for processing to obtain the processing result, including: The multilayer perceptron includes: two linear layers and a nonlinear activation layer Gaussian error linear unit, which performs a two-dimensional discrete fractional Fourier transform operation on the features obtained after being processed by the multilayer perceptron, and transforms them from the sequence dimension and the hidden dimension respectively to obtain the processing results.

6. The method according to claim 1, characterized in that The fusing of the first output feature and the second output feature includes: The first output feature and the second output feature are connected into a connection matrix, and the connection matrix is linearly transformed using a fully connected layer, and the attention weight is calculated.

7. The method according to claim 6, characterized in that The method further comprises: The transformation angle and attention weight are updated by training using the minimum loss function.

8. A human behavior recognition device, characterized in that: include: a two-dimensional image conversion module, configured to convert the collected human behavior into a two-dimensional image using a Gram angle difference field, so that the two-dimensional image retains the temporal information of the CSI data; a partitioning module, configured to partition the two-dimensional image into a plurality of token sequences, and when encoding the token sequences, perform a fractional Fourier transform of the transformation angle on the normalized output of each coding layer, and add the transformed result to the output of the previous layer as the input of the next layer or the final output of the coding layer; The normalization module is used to normalize the output of the encoding layer and input the normalization result into the multi-layer perceptron for processing to obtain the processing result; a transformation module, configured to apply a one-dimensional discrete fractional Fourier transform to the processing result along the hidden dimension for a first transformation to obtain a first transformation result, perform a one-dimensional discrete fractional Fourier transform on the first transformation result along the sequence dimension to obtain a second transformation result, and obtain the real part of the second transformation result as the first output feature; An input module, configured to input a two-dimensional image into a repeated cross-attention module to obtain a second output feature with dense and rich context information; The output module is used to fuse the first output feature and the second output feature, input the fusion result into the fully connected layer, and output the human behavior recognition result.

9. A computer device, characterized in that: The computer device comprises: one or more processors; a storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the human behavior recognition method as described in any one of claims 1 to 7.

10. A storage medium containing computer-executable instructions, characterized in that: When executed by a computer processor, the computer executable instructions are used to execute the human behavior recognition method according to any one of claims 1 to 7.

Images