Human Behavior Recognition System and Method Based on Sparse Three-Channel Hybrid Attention

Abstract

The present invention discloses a human behavior recognition system and method based on sparse three-channel hybrid attention, which relates to the technical field of human behavior recognition in deep learning. It is characterized by including: a human behavior information acquisition module, a human behavior information transmission module, a human behavior information preprocessing module, a human behavior classification and recognition module, and a human behavior information application module connected in sequence; the human behavior information acquisition module is used to collect the behavior data of users in real time. The technical problem to be solved by the present invention is to provide a human behavior recognition system and method based on sparse three-channel hybrid attention, which can independently and parallelly learn spatio-temporal features from the single-channel and inter-channel relationships at different positions of multi-source sensors, solve the deficiencies in the feature extraction method of the existing behavior recognition system and the defect of being unable to fully utilize the contribution rate of sensor positions, and effectively improve the recognition accuracy while reducing the number of parameters.

Description

Technical Field

[0001] The present invention relates to the technical field of human behavior recognition in deep learning, and more specifically, to a human behavior recognition system and method based on sparse three-channel hybrid attention. Background Art

[0002] With the development of wearable devices and deep learning technologies, human activity recognition (HAR) has received extensive attention. The HAR technology refers to the behavior monitoring in a specific field through the data obtained by sensors. In recent years, the HAR technology has been widely applied in many fields such as motion tracking, virtual reality, medical health, work safety, smart home, and abnormal behavior recognition, which can significantly improve production efficiency and quality of life. The general process of a HAR system includes data collection, data preprocessing, feature extraction, and behavior classification.

[0003] The algorithm design in the feature extraction stage is the core of the human behavior recognition system and the key to achieving high accuracy and low memory. Feature extraction aims to understand and judge different behaviors by studying the detailed features of various behaviors in the collected data. Wearable-based HAR uses sensors in portable intelligent devices to collect rich motion information and then accurately recognize human activities. Extracting features reflecting the differences of different activities from massive motion information using a deep learning model is the key to achieving accurate recognition in HAR. The convolutional neural network (CNN) is one of the representative models of deep neural networks and can automatically extract local features in data. The recurrent neural network (RNN) is also one of the widely used deep models at present and can effectively extract global features in sequence data and capture long-term relationships in sequence data. In addition, as a sequence modeling method, the attention mechanism can capture long-range dependencies by learning important time points in sequence data, and more and more attention mechanisms are introduced into neural networks. The fusion of attention mechanisms is of great significance for processing complex sequences and enhancing feature extraction capabilities.

[0004] At present, there are many challenges in the research field of behavior recognition algorithms based on deep learning. For example, most studies usually use RNN and other variants to learn the long-term correlation of latent features, while the utilization of the long-term correlation in the original time series data is insufficient. Secondly, the deep learning models for multi-location sensor data are not very advanced and cannot learn the features from multi-source sensors at different locations in the feature extraction layer, ignoring the location contribution of the sensors. Therefore, analyzing the differences and impacts of multi-location sensor data features in different types of human activities and obtaining the independent contribution of each location to activity recognition are necessary steps. In addition, many networks with attention mechanisms attempt to learn spatio-temporal attention at the image level, ignoring the interpretability of single-channel attention and inter-channel attention in time series analysis. Moreover, increasing the model depth helps to improve the accuracy of the training dataset but reduces the generalization ability. Focusing on learning low-contribution features plays an important role in preventing model overfitting. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a human behavior recognition system and method based on sparse three-channel hybrid attention, which can independently and parallelly learn spatio-temporal features from the single-channel and inter-channel relationships at different positions of multi-source sensors, solve the deficiencies in the feature extraction method of the existing behavior recognition system and the defect of being unable to fully utilize the sensor location contribution rate, effectively improve the recognition accuracy while reducing the number of parameters; the present invention also proposes three attention improvement methods: three-channel dynamic convolution based on improved squeeze-and-excitation (TCDC-ISE), three-channel mixed attention (TCMA), and sparse attention (SA).

[0006] For a human behavior recognition system with sparse three-channel hybrid attention involved in the present invention, first, the user wears a wearable device for data collection, which is built-in with various sensors, and different types can be selected according to different needs and worn on different human body parts. Then, the collected data is sent to a local server or a cloud server for storage in a wireless transmission manner through a data transmission module, and the transmission method can be selected according to different usage scenarios. Next, preprocessing operations are performed on the data, including filtering, normalization / standardization, sliding window segmentation of data, etc. Filtering is to remove the noise in the original sensor signals and the gravitational acceleration signals in the acceleration data; normalization / standardization can transform data with different dimensions and value ranges into the same range, which is beneficial for distance-based calculation models to eliminate the influence caused by different dimensions and value ranges; the sliding window segmentation method divides continuous long-time data into data segments that conform to the input format of the discrimination model. Furthermore, the preprocessed data is input into the human behavior classification and recognition module. Among them, it is first processed by a multi-scale dynamic convolution with sparse three-dimensional mixed attention (MDC-STDMA) module to independently and parallelly learn spatio-temporal features from the single-channel and inter-channel relationships at different positions of multi-source sensors. This operation preprocesses the input of MDC-STCMA through a scanning expansion module and a multi-scale feature segmentation module, focuses on learning time data with long-distance dependence relationships under the same receptive field and groups them according to the sensor positions; pays attention to the single-channel features themselves through the TCDC-ISE module; pays attention to the feature correlation between channels through the TCMA module; and reduces the overfitting tendency of the model through the SA module. Then it is further processed by a global attention based temporal and spatial fusion (GA-TSF) module to achieve the last step of spatio-temporal global information fusion by using the global attention mechanism that focuses on spatio-temporal features. Finally, by calculating the loss function of each category, the model parameters are adjusted during each round of training.

[0007] The present invention is a general human behavior recognition system based on wearable sensors, which can monitor the behavior categories of users in real time. It can be used for the monitoring and management of specific work fields, as well as for the guardianship of the elderly and the rehabilitation management of patients. Aiming at the problems that the existing algorithms mix and process the features of each channel, resulting in the inability to separate the features of sensors at different positions, and the poor interpretability of the attention extraction of each channel and the attention mixing between channels, the present invention provides a multi-scale dynamic convolution method with sparse three-channel hybrid attention to solve this problem, further enhancing the feature representation and improving the accuracy of behavior recognition. In addition, aiming at the problem that the existing methods are prone to overfitting due to the improvement of accuracy, the nodes with low contribution rate are retrained by using the sparse attention method, effectively preventing the model from overfitting, enhancing the universality of the model for complex behaviors, and being more conducive to application in actual scenarios.

[0008] The present invention adopts the following technical solutions to achieve the invention purpose:

[0009] A human behavior recognition system and method based on sparse three-channel hybrid attention, characterized by comprising:

[0010] A human behavior information acquisition module, a human behavior information transmission module, a human behavior information preprocessing module, a human behavior classification and recognition module, and a human behavior information application module that are connected in sequence;

[0011] The human behavior information acquisition module is used to collect the behavior data of the user in real time. The behavior data includes the X, Y, and Z axis data of the acceleration sensor, the X, Y, and Z axis data of the angular velocity sensor, and the X, Y, and Z axis data of the magnetometer;

[0012] The human behavior information transmission module is used to transmit the collected behavior data of the user to the local server or the cloud server;

[0013] The human behavior information preprocessing module is used to store, merge, filter, normalize or standardize, perform sliding window segmentation, and label calibration on the collected behavior data of the user in sequence, so as to obtain smooth behavior data segments that are not affected by the dimension and carry labels;

[0014] The human behavior classification and recognition module is used to input the behavior data of the user preprocessed by the human behavior information preprocessing module into the trained human behavior classification and recognition module for behavior type discrimination;

[0015] The human behavior information application module is used to transmit the obtained human behavior classification and recognition result to various application platforms, so as to realize the corresponding functions.

[0016] As a further limitation of the present technical solution, the human behavior classification and recognition module includes a scanning expansion module, a multi-scale feature segmentation module, a multi-scale dynamic convolution module with sparse three-channel hybrid attention, a global attention spatio-temporal fusion module, and a human behavior classification result output module that are connected in sequence;

[0017] As a further limitation of the present technical solution, the human behavior information acquisition module includes a three-axis acceleration sensor, a three-axis angular velocity sensor, and a three-axis magnetometer for collecting the behavior data of the user.

[0018] As a further limitation of the present technical solution, the human behavior information preprocessing module includes a sensor data storage unit, a sensor data merging unit, a sensor data smoothing and filtering unit, a data normalization / standardization unit, a sliding window segmentation unit, and a data label calibration unit that are connected in sequence.

[0019] As a further limitation of the present technical solution, the collected behavior data of the user is transmitted to a local server or a cloud server through any one of the transmission methods of ultra-wideband, Bluetooth, and 5G.

[0020] A recognition method for a human behavior recognition system based on sparse three-channel hybrid attention, characterized by comprising the following steps:

[0021] S1: Human behavior data acquisition;

[0022] Real-time acquisition of the behavior data of the user, that is, human behavior data, and the behavior data includes the X, Y, and Z axis data of the acceleration sensor, the X, Y, and Z axis data of the angular velocity sensor, and the X, Y, and Z axis data of the magnetometer;

[0023] S2: Human behavior data transmission;

[0024] Transmit the collected behavior data of the user to a local server or a cloud server;

[0025] S3: Human behavior data preprocessing;

[0026] The collected behavior data of the user is sequentially stored, merged, filtered, normalized or standardized, segmented by a sliding window, and labeled, so as to obtain smooth behavior data segments that are not affected by the dimension and carry labels;

[0027] S4: Construct a recognition model and human behavior classification and recognition;

[0028] The behavior data segments preprocessed in S3 are input into the recognition model in batches, and the human behavior recognition output is obtained through training;

[0029] S5: Loss function calculation, update and feedback;

[0030] Calculate the loss function for the classification results generated by S4, and adjust the model parameters through backpropagation to improve the classification accuracy and generalization ability. Use the cross-entropy loss function to calculate the error between the model prediction value and the true value. The cross-entropy loss function is defined as shown in Equation (1):

[0031] (1);

[0032] Where: represents the batch data volume, represents the total number of categories, represents the th sample belonging to the category true label, represents the th sample belonging to the category predicted probability;

[0033] (2);

[0034] Where: represents the model parameters at the th step, represents the model parameters at the th step, represents the learning rate, is a constant added to increase numerical stability, represents the first-order momentum, represents the second-order momentum, and represent the first-order and second-order average coefficients and to the power;

[0035] The calculation process of the first-order momentum can be expressed as Equation (3):

[0036] (3);

[0037] Where: represents the gradient of the loss function , represents the first-order average coefficient, represents the current first-order momentum value, represents the first-order momentum value at the previous moment;

[0038] The calculation process of the second-order momentum can be expressed as Equation (4):

[0039] (4);

[0040] Where: Represents the second-order average coefficient, Represents the current second-order momentum value, Represents the second-order momentum value at the previous moment;

[0041] S6: Application of the human behavior classification output result;

[0042] Transmit the human behavior classification and recognition result to the corresponding application platform in real time.

[0043] As a further limitation of this technical solution, the specific implementation process of S3 is as follows:

[0044] S31: Merging of human behavior data;

[0045] Stitch and merge the behavior data of the collected users. The merged data is in the format of a two-dimensional array, arranged horizontally in the order of the X, Y, and Z axis data of the three-axis acceleration sensor, the X, Y, and Z axis data of the three-axis angular velocity sensor, and the X, Y, and Z axis data of the three-axis magnetometer, and arranged vertically in chronological order;

[0046] S32: Filtering of human behavior data;

[0047] Filtering of human behavior data refers to smoothing filtering of sensor data;

[0048] Smoothing filtering of sensor data uses three-point mean smoothing filtering to eliminate the noise signal inside the sensor. Let the sensor signal sequence be , Represents the number of data points in the sensor signal sequence; The formula for mean smoothing is shown in Formulas (5) and (6):

[0049] (5);

[0050] (6);

[0051] Where: Represents the value after one-time mean smoothing, Represents the value after two-time mean smoothing;

[0052] S33: Normalization / Standardization of human behavior data;

[0053] Use the deviation normalization method for normalization processing. Let the input sequence be , and the output sequence after deviation normalization is , as shown in Formula (7):

[0054] (7);

[0055] Where: and are the maximum and minimum values of all samples in the input sequence respectively;

[0056] Standardization is performed using the z - score standardization method, as shown in Equation (8):

[0057] (8);

[0058] where: is the mean value of all samples in the input sequence ; is the standard deviation of all samples in the input sequence ;

[0059] S34: Sliding window segmentation and tagging;

[0060] Sliding window segmentation uses a window of fixed length to divide continuous sensor data into data segments of fixed length, assigns a corresponding tag to each data segment, that is, the name of the behavior category corresponding to the data segment, and encodes the tag using one - hot encoding.

[0061] As a further limitation of this technical solution, the specific implementation process of S4 is as follows:

[0062] S41: Scan expansion is to perform data augmentation on the input time - series data using scrambled arrangement and flattening operations, expand it into four time - sorting combinations, realize time - feature scrambling, and enhance the feature learning effect. Specifically, it means:

[0063] The time - series data with the size of each feature dimension of is converted to types of image - like data through reshape, where ;

[0064] Each data is flattened in four orders: horizontally from top - left to bottom - right, vertically from top - right to bottom - left, horizontally from bottom - right to top - left, and vertically from bottom - right to top - left, obtaining and The calculation process of its scrambled arrangement and flattening is shown in Equation (9):

[0065] (9);

[0066] The four new time - series sequences and are merged to obtain the data - augmented sample as shown in Equation (10):

[0067] (10);

[0068] Among them: Concat represents the merge connection operation;

[0069] S42: Multi-scale feature segmentation is to separate the features of sensors at different locations. In the subsequent feature extraction stage, the grouped data is input into several parallel similarity block networks to facilitate the extraction of high-level feature representations of sensor data at different locations in the same activity. Specifically, it refers to:

[0070] The data enhancement samples of S41 are According to the body location where the wearable sensor is worn, the multi-source sensor features at different locations are divided into groups: IMU1, IMU2, ..., IMUn;

[0071] S43: Multi-scale dynamic convolution with sparse three-channel mixed attention is used to extract and fuse features in depth;

[0072] S44: The global attention spatiotemporal fusion module is used to achieve full fusion of spatiotemporal attention features:

[0073] The global attention unit combines the gated linear unit and the attention mechanism, and generates the matrix through the gated linear unit mapping process. and , and then use the self-attention mechanism to extract key features. The calculation process is shown in formula (11):

[0074] (11);

[0075] in: Output of the global attention unit; is the weight bias; is the query matrix, and are the keys and values represented by the matrix, is the ReLU activation function;

[0076] After further one-dimensional convolution calculation with a convolution kernel size of 3 and a stride of 1, data with a feature channel of 1 is generated, and the output of the global attention is further extracted to ensure that local information is not lost;

[0077] S45: The human behavior classification result output module is used to reduce the dimension and classify the high-dimensional features obtained by feature extraction and fusion to achieve accurate classification of human behavior, including a flattening layer, layer normalization, fully connected layer and Softmax output layer connected in sequence.

[0078] As a further limitation of this technical solution, the specific implementation process of S43 is as follows:

[0079] S431: The three-channel dynamic convolution unit with improved extrusion excitation includes an initialization convolution block 0 to obtain potential features of each channel, and three convolution blocks 1, 2, and 3 connected in series in sequence;

[0080] S432: The three-channel hybrid attention unit consists of three parts: long and width channel hybrid attention, long and feature channel hybrid attention, and width and feature channel hybrid attention;

[0081] S433: The Patch Merging unit is used to shuffle the features of different sensors, randomize the sensor sources of the features, and achieve the merging of features of different sensors. The specific implementation process is as follows:

[0082] In S432, the finally output attention map passes through the shuffle layer to generate a shuffled feature map with odd and even skip connections; through the splicing layer, data with a feature length 4 times the original and long and width channels 1 / 2 of the original is generated; through a linear layer, the feature dimension is halved, and data with a feature length 2 times the original and long and width channels 1 / 2 of the original is generated;

[0083] S434: The sparse attention unit is used to prevent the model from overfitting during the training of low contribution rate nodes, and it consists of 2 mask layers, an encoding layer, and a decoding layer;

[0084] Each attention vector with a length of passes through a mask layer. The mask module sets the mask rate to 0.75, sets the attention with the top 0.75 weights before sorting to 0, and the rest to 1. Its masking process is shown in equations (12)-(14):

[0085] (12);

[0086] (13);

[0087] (14);

[0088] Where: is the reverse sorting function, is the attention sequence sorted from largest to smallest with a length of , will be used as the input of the encoding module; is the mask function; is the attention sequence after masking;

[0089] The encoding layer first performs two-dimensional convolution calculations with a convolutional kernel size of 3×1 and a stride of 1 to generate data with a feature channel that is 1 / 2 of the original; then it goes through batch normalization and the ReLU activation function; then it performs two-dimensional convolution calculations with a convolutional kernel size of 3×1 and a stride of 1 again to generate data with a feature channel that is 1 / 4 of the original.

[0090] The decoding layer first performs two-dimensional convolution calculations with a convolutional kernel size of 3×1 and a stride of 1 to generate data with a feature channel that is 1 / 2 of the original; then it goes through batch normalization and the ReLU activation function; then it performs two-dimensional convolution calculations with a convolutional kernel size of 3×1 and a stride of 1 again to generate data with a feature channel of the original dimension.

[0091] Compared with the prior art, the advantages and positive effects of the present invention are:

[0092] The present invention proposes a human behavior recognition system with sparse three-channel hybrid attention. By processing and analyzing the motion signals of wearable sensors at different positions, it realizes the accurate recognition of the user's motion state, solves the problems that the video-based human behavior recognition method is vulnerable to the environment, has poor privacy, and the problem that it is difficult to collect data for the radio frequency device-based human behavior recognition method.

[0093] The present invention proposes a human behavior recognition method with sparse three-channel hybrid attention, which can independently and parallelly learn the spatio-temporal features of multi-source sensors at different body positions from single-channel and inter-channel relationships, thereby improving the accuracy of human behavior recognition.

[0094] The present invention designs a multi-scale dynamic convolution module with sparse three-channel hybrid attention to deeply extract and fuse features. To improve the interpretability of single-channel attention and inter-channel attention in the feature extraction layer, this module uses three-channel dynamic convolution with improved squeeze-and-excitation and three-channel hybrid attention to respond to single-channel and inter-channel features respectively, and uses sparse attention to prevent overfitting.

[0095] The present invention introduces a multi-scale feature segmentation module and a scanning expansion module, which effectively utilizes the temporal multi-scale correlation of the original time series data and the position contribution of multi-source sensors, and at the same time relaxes the requirement for the optimal convolutional kernel size of the CNN.. Brief Description of the Drawings

[0096] Figure 1 It is a schematic diagram of the module composition and connection relationship of the wearable human behavior recognition system of the present invention.

[0097] Figure 2 It is a schematic flowchart of the wearable human behavior recognition method of the present invention.

[0098] Figure 3Schematic diagram of the principle of human behavior classification in the wearable human behavior recognition system of the present invention. Detailed implementation mode

[0099] The following combines the accompanying drawings to describe in detail a specific implementation mode of the present invention, but it should be understood that the protection scope of the present invention is not limited by the specific implementation mode.

[0100] Example 1

[0101] In the application of intelligent rehabilitation, there are many problems. For example, the monitoring of the training actions of patients during rehabilitation training is an important criterion for evaluating the rehabilitation training effect. Although cameras are arranged everywhere in the rehabilitation training room, it is very difficult to monitor the rehabilitation training actions of patients in daily scenarios and evaluate them in a timely manner. Moreover, the cameras have problems such as being easily blocked, insufficient light, and dead angles. At this time, the human behavior recognition technology based on wearable devices can make up for the deficiencies of videos. Patients wear wearable devices at corresponding body positions, which are built-in with three-axis acceleration sensors, three-axis angular velocity sensors, and three-axis magnetometers, and can be used to collect the behavior data of patients in real time, and transmit it to the local server or cloud server for storage and calculation in real time, and obtain the corresponding recognition results. The intelligent rehabilitation application platform can display the results in real time, and the attending physicians and family members of the patients can use the results to observe the rehabilitation situation of the patients in real time. In addition, the behavior status of the patients can also be managed in the long term to avoid abnormal situations such as recurrence of diseases in the patients.

[0102] The present invention, as Figure 1 shown, includes:

[0103] A human behavior information acquisition module, a human behavior information transmission module, a human behavior information preprocessing module, a human behavior classification and recognition module, and a human behavior information application module that are connected in sequence.

[0104] The human behavior information acquisition module is used to collect the behavior data of users in real time. The behavior data includes the X, Y, and Z axis data of the acceleration sensor, the X, Y, and Z axis data of the angular velocity sensor, and the X, Y, and Z axis data of the magnetometer. Users can wear multiple wearable devices at different positions on the body according to their needs, and various sensors are built in. The human behavior information acquisition module includes several different sensor units, which are integrated in the wearable device and worn at the corresponding positions of the user's body respectively. The sensor unit includes a three-axis acceleration sensor, a three-axis angular velocity sensor, and a three-axis magnetometer for collecting the behavior data of the user.

[0105] The human body behavior information transmission module is used to transmit the collected behavior data of the user to the local server or the cloud server. According to the scenario where the user is located, a suitable transmission method can be selected to transmit the collected behavior data of the user to the local server or the cloud server through any one of the transmission methods of Ultra Wide Band (UWB), Bluetooth, and 5G. Different transmission methods are different in terms of power consumption, transmission distance, transmission speed, etc., so they are suitable for different application scenarios.

[0106] The human body behavior information preprocessing module is used to sequentially store, merge, filter, normalize or standardize, perform sliding window segmentation and label calibration on the collected behavior data of the user, so as to obtain smooth, dimensionless-influenced and labeled behavior data segments.

[0107] The human body behavior classification and recognition module is used to input the behavior data of the user preprocessed by the human body behavior information preprocessing module into the trained human body behavior classification and recognition module for behavior type discrimination.

[0108] The human body behavior information application module is used to transmit the obtained human body behavior classification and recognition results to various application platforms, so as to realize corresponding functions. The human body behavior information application module faces various application platforms, such as various application platforms like the elderly care management platform and the rehabilitation nursing monitoring management platform. The results of model classification and recognition are transmitted to the databases of each application platform in real time for storage and management, and the behaviors of users are analyzed, managed and visualized in real time.

[0109] This system realizes the real-time monitoring and evaluation of the intelligent rehabilitation training actions of patients through the processing and analysis of sensor data. In the human body behavior information classification module, a new deep learning framework for wearable human body behavior recognition based on a spatio-temporal parallel multi-scale network with sparse three-channel hybrid attention is proposed, which can independently and parallelly learn the spatio-temporal features of multi-source sensors at different body positions from single-channel and inter-channel relationships, thereby improving the accuracy of human body behavior recognition; at the same time, sparse attention is used to retrain the network nodes with lower weights to prevent the model from overfitting, and further improve the robustness and generalization ability of the model.

[0110] Embodiment 2

[0111] The human body behavior classification and recognition module includes a scanning expansion module, a multi-scale feature segmentation module, a multi-scale dynamic convolution module with sparse three-channel hybrid attention, a global attention spatio-temporal fusion module, and a human body behavior classification result output module connected in sequence.

[0112] The scanning expansion module is used to perform data augmentation processing on the input time-series data, expand it into four time sorting combinations, achieve time feature scrambling, and enhance the feature learning effect; the scanning expansion module first converts the time-series data with a feature dimension of into image-like data of class through reshape; the scanning expansion module then performs flattening operations in 4 orders: horizontally from top left to bottom right, vertically from top right to bottom left, horizontally from bottom right to top left, and vertically from bottom right to top left; the scanning expansion module finally combines the 4 flattened time-series data to obtain a data augmentation sample, and the dimension of the new sample is ; the scanning expansion module unfolds data in different orders along four different directions, shuffles the feature sorting, and enables effective learning of numerical features with long-range dependence relationships within the same receptive field. Ensure that each element in the feature integrates information with other positions in different directions, effectively learn the time information with long-range dependence in the input feature, and thus form a global receptive field. Facilitate the subsequent feature extraction module to extract different features from different angles, enriching the diversity of features;

[0113] The multi-scale feature segmentation module is used to separate features of sensors at different positions; the multi-scale feature segmentation module can segment multi-source sensor features at different positions into groups according to the body position grouping method: IMU1, IMU2,..., IMU n ; the grouping method of the multi-scale feature segmentation module can input the grouped data into several parallel similar block networks in the subsequent feature extraction stage, facilitating the extraction of high-level feature representations of sensor data at different positions in the same activity;

[0114] The multi-scale dynamic convolution module with sparse three-channel hybrid attention is used for deep feature extraction; the multi-scale dynamic convolution module with sparse three-channel hybrid attention includes a three-channel dynamic convolution unit based on improved squeeze-and-excitation, a three-channel hybrid attention unit, a Patch Merging unit, and a sparse attention unit connected in sequence; the three-channel dynamic convolution unit based on improved squeeze-and-excitation is a convolutional layer that dynamically and independently learns the attention of each channel (long channel, wide channel, and feature channel) and weights it to improve the independent learning ability of different channel features; the three-channel hybrid attention unit is used to fully mix features across channels and capture the feature relationships between channels; the sparse attention unit is used to prevent model overfitting during the training of low contribution rate nodes.

[0115] The improved squeeze excitation three-channel dynamic convolution unit includes an initial convolution block 0 to obtain potential features of each channel, and three sequentially connected convolution blocks 1, convolution block 2, and convolution block 3. Convolution block 0 performs two-dimensional convolution calculation on the input data with a convolution kernel size of 4×4 and a stride of 1, followed by batch normalization and ReLU activation function to generate data of 64×H0×W0; Convolution block 1 performs two-dimensional convolution calculation on the data of 64×H0×W0 through two groups of convolution kernels with a size of 3×3 and a stride of 1, followed by batch normalization and ReLU activation function to generate data of 32×H1×W1; Convolution block 2 performs two-dimensional convolution calculation on the data of 32×H1×W1 through two groups of convolution kernels with a size of 3×3 and a stride of 1, followed by batch normalization and ReLU activation function to generate data of 64×H2×W2; Convolution block 3 performs two-dimensional convolution calculation on the data of 64×H2×W2 through two groups of convolution kernels with a size of 3×3 and a stride of 1, followed by batch normalization and ReLU activation function to generate data of 128×H3×W3; In addition, the improved squeeze excitation three-channel dynamic convolution unit implements a convolutional layer that dynamically and independently learns the attention of each channel (long channel, wide channel, and feature channel) and weights it to improve the independent learning ability of different channel features; The potential feature 64×H0×W0 output by convolution block 0 is passed through a two-dimensional adaptive pooling with a size of 1×1 and two fully connected layers to generate independent attention for each channel (long channel 1×H0×1, wide channel 1×1×W0, and feature channel 64×1×1); Then, the attention of each channel is summed to obtain the attention 64×H0×W0; Finally, the attention 64×H0×W0 adaptively adjusts the dimension through a fully connected layer and weights the convolutional layers of each module after convolution block 0.

[0116] The three-channel hybrid attention unit consists of three parts: long and wide channel hybrid attention, long and feature channel hybrid attention, and wide and feature channel hybrid attention; for the long and wide hybrid attention, first, the output 128×H3×W3 of the three-channel dynamic convolution unit based on the improved squeeze-and-excitation is subjected to Max-Average self-attention, that is, the feature channel attention is reduced to 2 by combining average pooling operation and max pooling operation, generating data of 2×H3×W3; through two-dimensional convolution calculation with a convolution kernel size of 3×3 and a stride of 1, the feature channel attention is reduced to 1, fixing the feature channel attention, obtaining long and wide channel hybrid attention, and generating data of 1×H3×W3; for the long and feature channel hybrid attention, first, the output 128×H3×W3 of the three-channel dynamic convolution unit based on the improved squeeze-and-excitation is subjected to Max-Average self-attention, that is, the wide channel attention is reduced to 2 by combining average pooling operation and max pooling operation, generating data of 128×H3×2; through two-dimensional convolution calculation with a convolution kernel size of 3×3 and a stride of 1, the wide channel attention is reduced to 1, fixing the wide channel attention, obtaining long and feature channel hybrid attention, and generating data of 128×H3×1; for the wide and feature channel hybrid attention, first, the output 128×H3×W3 of the three-channel dynamic convolution unit based on the improved squeeze-and-excitation is subjected to Max-Average self-attention, that is, the long channel attention is reduced to 2 by combining average pooling operation and max pooling operation, generating data of 128×2×W3; through two-dimensional convolution calculation with a convolution kernel size of 3×3 and a stride of 1, the long channel attention is reduced to 1, fixing the long channel attention, obtaining wide and feature channel hybrid attention, and generating data of 128×1×W3; the output three hybrid attention weights of the three-channel hybrid attention unit at each position are merged and multiplied to generate feature mixed attention data.

[0117] The Patch Merging unit is used to shuffle different sensor features, randomize the sensor sources of the features, and achieve the merging of different sensor features; the attention map is passed through the shuffle layer to generate a shuffled feature map with odd-even skip connections; through the splicing layer, data with a feature length 4 times the original and long and wide channels 1 / 2 of the original is generated; through a linear layer, the feature dimension is halved, generating data with a feature length 2 times the original and long and wide channels 1 / 2 of the original.

[0118] The sparse attention unit is used to prevent the model from overfitting during the training of low contribution rate nodes. It consists of two mask layers, an encoding layer, and a decoding layer; each attention vector length passes through a mask layer. The mask module sets the mask rate to 0.75, sets the attention with the top 0.75 weights before sorting to 0, and the rest to 1; then the encoding layer first performs two-dimensional convolution calculations with a convolutional kernel size of 3×1 and a stride of 1, generating data with a feature channel that is 1 / 2 of the original; passing through batch normalization and the ReLU activation function; performing two-dimensional convolution calculations with a convolutional kernel size of 3×1 and a stride of 1, generating data with a feature channel that is 1 / 4 of the original; then the decoding layer first performs two-dimensional convolution calculations with a convolutional kernel size of 3×1 and a stride of 1, generating data with a feature channel that is 1 / 2 of the original; passing through batch normalization and the ReLU activation function; performing two-dimensional convolution calculations with a convolutional kernel size of 3×1 and a stride of 1, generating data with a feature channel of the original dimension.

[0119] The global attention spatio-temporal fusion module is used to fully fuse spatio-temporal attention features; the global attention spatio-temporal fusion module includes a global attention unit and a one-dimensional standard convolution unit connected in sequence; the global attention unit combines a gated linear unit and an attention mechanism, and the input data is not directly multiplied by the attention parameters, but first mapped and processed by the gated linear unit; the one-dimensional standard convolution unit further extracts the output of the global attention to ensure that local information is not lost.

[0120] The human behavior classification result output module is used to reduce the dimension and classify the high-dimensional features obtained by feature extraction and fusion to achieve accurate classification of human behaviors, and update the parameters of the recognition model according to the loss function. It includes a flattening layer, a layer normalization, a fully connected layer, and a Softmax output layer connected in sequence; the classification result output module calculates and outputs the recognition result through the Softmax layer.

[0121] The human behavior information preprocessing module includes a sensor data storage unit, a sensor data merging unit, a sensor data smoothing and filtering unit, a data normalization / standardization unit, a sliding window segmentation unit, and a data label calibration unit connected in sequence.

[0122] The sensor data storage unit is used to store the real-time behavior data of the user.

[0123] The sensor data merging unit is used to splice and merge the behavior data of different sensor units. The merged data is in the format of a two-dimensional array, and horizontally is arranged in the order of the X, Y, and Z axis data of the three-axis acceleration sensor, the X, Y, and Z axis data of the three-axis angular velocity sensor, and the X, Y, and Z axis data of the three-axis magnetometer, and vertically is arranged in chronological order. The frequency of sensor data acquisition can be set to an appropriate value, such as , that is, the size of the data generated per second is 128 rows and 27 columns.

[0124] The sensor data smoothing and filtering unit is used to remove noise in the original sensor data; usually, the sensor data contains a large amount of high-frequency noise, and mean smoothing filtering can be used to remove these noises.

[0125] The data normalization / standardization unit is used to convert data with different dimensions and value ranges to the same value range; to avoid the adverse effects of different dimensions and value ranges on calculations. Min-Max normalization method can be used for normalization, that is, the values of all the same type of sensors are transformed between [0,1]. Z-Score normalization method can be used for standardization, that is, the data is transformed into data with a mean of 0 and a standard deviation of 1, so that the original incomparable values can be made comparable.

[0126] The sliding window segmentation unit is used to segment the continuous long-time sensor data into several data segments; it is suitable for the input of the recognition model. According to the characteristics of human behavior, the length of one window can be set to 128, that is, it can contain seconds of sensor data, which can basically include a complete set of common human behavior data. The sliding step can be set to 64, that is the data coverage rate.

[0127] The data label calibration unit is used to calibrate the labels of the data after being segmented by the sliding window. The size of one data segment is rows columns, is the number of columns after merging all sensor data. When using triaxial acceleration sensors, triaxial angular velocity sensors, and triaxial magnetometers located on the wrist, waist, and chest, the size of one data segment is 128 rows and 27 columns. Each data segment corresponds to a corresponding label data, and one-hot encoding is performed on the labels.

[0128] The collected user behavior data is transmitted to the local server or cloud server through any one of the transmission methods of ultra-wideband, Bluetooth, and 5G.

[0129] The human behavior information application module faces various application platforms. The classification and recognition results are transmitted to the databases of each application platform in real time for storage and management, and the user's behavior is analyzed, managed, and visualized in real time.

[0130] The recognition method of the human behavior recognition system based on sparse three-channel hybrid attention, as Figure 2 shown, includes the following steps:

[0131] S1: Human body behavior data collection;

[0132] Collect the behavior data of the user in real time, that is, the human body behavior data. The behavior data includes the X, Y, and Z axis data of the acceleration sensor, the X, Y, and Z axis data of the angular velocity sensor, and the X, Y, and Z axis data of the magnetometer. Wearable devices with several built-in different sensor units are respectively worn on different positions of the user's body. The sensor units mainly include a three-axis acceleration sensor, a three-axis angular velocity sensor, and a three-axis magnetometer, which are used to collect the behavior information of the user.

[0133] S2: Human body behavior data transmission;

[0134] Transmit the collected behavior data of the user to the local server or the cloud server. According to different scenarios and user requirements, select a suitable data transmission method, such as UWB, Bluetooth, 5G, etc.

[0135] S3: Human body behavior data preprocessing;

[0136] Store, merge, filter, normalize or standardize, sliding window segmentation, and label calibration are performed on the collected behavior data of the user in sequence, so as to obtain smooth, dimensionless-influenced, and labeled behavior data segments.

[0137] The specific implementation process of S3 is as follows:

[0138] S31: Human body behavior data merging;

[0139] Stitch and merge the behavior data of the user collected at three positions of the wrist, waist, and chest. The merged data is in the format of a two-dimensional array, which is arranged horizontally in the order of the X, Y, and Z axis data of the three-axis acceleration sensor, the X, Y, and Z axis data of the three-axis angular velocity sensor, and the X, Y, and Z axis data of the three-axis magnetometer, and arranged vertically in chronological order. The frequency of sensor data collection Can be set to an appropriate value, such as Hz, that is, the size of the data generated per second is 128 rows and 27 columns;

[0140] S32: Human body behavior data filtering;

[0141] Human body behavior data filtering refers to the smoothing filtering of sensor data;

[0142] The smoothing filtering of sensor data uses three-point mean smoothing filtering to eliminate the noise signal inside the sensor. Let the sensor signal sequence be , Represents the number of data points in the sensor signal sequence; The formula for mean smoothing is shown in formulas (5) and (6):

[0143] (5);

[0144] (6);

[0145] Wherein: represents the value after one-time mean smoothing, represents the value after two-time mean smoothing;

[0146] S33: Normalization / Standardization of human behavior data;

[0147] Convert data with different dimensions and value ranges to the same value range to avoid adverse effects on calculations caused by different dimensions and value ranges. Use the deviation (Min-Max) normalization method for normalization, that is, transform the values of all sensors of the same type to between [0,1]; the deviation (Min-Max) normalization method scales all data values to between [0,1] through a linear transformation of the original data; let the input sequence be , and the output sequence after deviation (Min-Max) normalization is , as shown in Equation (7):

[0148] (7);

[0149] Wherein: and are the maximum and minimum values of all samples in the input sequence respectively;

[0150] Use the standard score (Z-Score) standardization method for standardization, that is, transform the data into data with a mean of 0 and a standard deviation of 1; the Z-Score standardization method transforms the data into data with a mean of 0 and a standard deviation of 1, as shown in Equation (8):

[0151] (8);

[0152] Wherein: is the mean of all samples in the input sequence , is the standard deviation of all samples in the input sequence ;

[0153] S34: Sliding window segmentation and tagging;

[0154] Sliding window segmentation is to use a window with a fixed length to divide continuous sensor data into data segments with a fixed length for input into the recognition model. Set the sensor acquisition frequency to Hz, and select a length of Windows, that is, each window contains seconds of sensor data. The sliding step can be set to 64, that is of the data coverage rate, so as to divide the data into sizes of rows columns, is the number of columns after merging all sensor data. When using triaxial acceleration sensors, triaxial angular velocity sensors, and triaxial magnetometers located at three positions on the wrist, waist, and chest respectively, the size of a data segment is 128 rows and 27 columns. A corresponding label is assigned to each data segment, that is, the name of the behavior category corresponding to the row of this data segment, and the label is encoded using one-hot encoding.

[0155] S4: Construct an identification model for human behavior classification and recognition;

[0156] Input the behavior data segments preprocessed in S3 into the identification model in batches, and obtain the human behavior recognition output after training;

[0157] Such as Figure 3 shown, the specific implementation process of S4 is as follows:

[0158] S41: Scanning expansion is to perform data augmentation processing on the input time series data using scrambled arrangement and flattening operations, expanding it into four time sorting combinations to achieve time feature scrambling and enhance the feature learning effect. Specifically, it means:

[0159] The size of each feature dimension is the size of of the time series data is converted to types of image data through reshape, where ;

[0160] Each data is flattened in four orders: horizontally from top left to bottom right, vertically from top right to bottom left, horizontally from bottom right to top left, and vertically from bottom right to top left, to obtain and The calculation process of its scrambled arrangement and flattening is shown in Equation (9):

[0161] (9);

[0162] The four new time series and are merged to obtain the data augmentation sample as shown in Equation (10):

[0163] (10);

[0164] Among them: Concat represents the merge connection operation;

[0165] S42: Multi-scale feature segmentation separates the features of sensors at different positions. In the subsequent feature extraction stage, the grouped data is input into several parallel similar block networks to facilitate the extraction of high-level feature representations of sensor data at different positions during the same activity. Specifically, it means:

[0166] The data augmentation samples passing through the above S41 According to the grouping method of the body positions where the wearable sensors are worn, the multi-source sensor features at different positions are segmented into groups: IMU1, IMU2,..., IMUn;

[0167] S43: The multi-scale dynamic convolution of sparse three-channel hybrid attention performs in-depth extraction and fusion of features;

[0168] The multi-scale dynamic convolution module of sparse three-channel hybrid attention includes a three-channel dynamic convolution unit based on improved squeeze-and-excitation, a three-channel hybrid attention unit, and a sparse attention unit connected in sequence. The three-channel dynamic convolution unit based on improved squeeze-and-excitation is a convolutional layer that dynamically and independently learns the attention of each channel (long channel, wide channel, and feature channel) and weights it to improve the independent learning ability of different channel features. The three-channel hybrid attention unit is used to fully mix features across channels and capture the feature relationships between channels. The sparse attention unit is used to prevent model overfitting during the training of low contribution rate nodes;

[0169] The multi-scale dynamic convolution module of sparse three-channel hybrid attention in the above S43 includes a three-channel dynamic convolution unit based on improved squeeze-and-excitation, a three-channel hybrid attention unit, a Patch Merging unit, and a sparse attention unit connected in sequence. The specific implementation process is as follows:

[0170] S431: The three-channel dynamic convolution unit based on improved squeeze-and-excitation includes an initial convolutional block 0 to obtain potential features of each channel, and three convolutional blocks 1, 2, and 3 connected in series in sequence;

[0171] Convolutional block 0 realizes the two-dimensional convolution calculation of the input data with a convolution kernel size of 4×4 and a stride of 1, and then passes through batch normalization and the rectified linear unit ReLU (Rectified Linear Unit, ReLU) activation function to generate data of 64×H0×W0;

[0172] Convolutional block 1 realizes the two-dimensional convolution calculation of the 64×H0×W0 data with two convolution kernel sizes of 3×3 and a stride of 1, and then passes through batch normalization and the ReLU activation function to generate data of 32×H1×W1;

[0173] The convolutional block 2 implements two-dimensional convolutional calculations on the data of 32×H1×W1 with a convolutional kernel size of 3×3 and a stride of 1. Then, through batch normalization and the ReLU activation function, data of 64×H2×W2 is generated.

[0174] The convolutional block 3 implements two-dimensional convolutional calculations on the data of 64×H2×W2 with a convolutional kernel size of 3×3 and a stride of 1. Then, through batch normalization and the ReLU activation function, data of 128×H3×W3 is generated.

[0175] The three-channel dynamic convolutional unit based on the improved squeeze-and-excitation realizes a convolutional layer that dynamically and independently learns the attention of each channel (long channel, wide channel, and feature channel) and weights it, so as to improve the independent learning ability for different channel features. The latent features of 64×H0×W0 output by the convolutional block 0 are passed through a two-dimensional adaptive pooling with a size of 1×1 and two fully connected layers to generate independent attention for each channel (long channel 1×H0×1, wide channel 1×1×W0, and feature channel 64×1×1). Then, the attention of each channel is summed to obtain the attention of 64×H0×W0. Finally, the attention of 64×H0×W0 adaptively adjusts the dimension through a fully connected layer and weights the convolutional layers of each module after the convolutional block 0.

[0176] S432: The three-channel hybrid attention unit consists of three parts: long and wide channel hybrid attention, long and feature channel hybrid attention, and wide and feature channel hybrid attention.

[0177] For the long and wide hybrid attention, first, the output 128×H3×W3 of the three-channel dynamic convolutional unit based on the improved squeeze-and-excitation uses Max-Average self-attention, that is, the way of combining average pooling operation and max pooling operation to reduce the feature channel attention to 2, generating data of 2×H3×W3. Through two-dimensional convolutional calculation with a convolutional kernel size of 3×3 and a stride of 1, the feature channel attention is reduced to 1, fixing the feature channel attention, and obtaining the long and wide channel hybrid attention, generating data of 1×H3×W3.

[0178] For the long and feature channel hybrid attention, first, the output 128×H3×W3 of the three-channel dynamic convolutional unit based on the improved squeeze-and-excitation uses Max-Average self-attention, that is, the way of combining average pooling operation and max pooling operation to reduce the wide channel attention to 2, generating data of 128×H3×2. Through two-dimensional convolutional calculation with a convolutional kernel size of 3×3 and a stride of 1, the wide channel attention is reduced to 1, fixing the wide channel attention, and obtaining the long and feature channel hybrid attention, generating data of 128×H3×1.

[0179] The width and feature channel hybrid attention first uses the Max-Average self-attention on the output 128×H3×W3 of the three-channel dynamic convolution unit based on the improved squeeze-and-excitation, that is, the long-channel attention is reduced to 2 by combining the average pooling operation and the max pooling operation, generating data of 128×2×W3; the long-channel attention is reduced to 1 through a two-dimensional convolution calculation with a convolution kernel size of 3×3 and a stride of 1, fixing the long-channel attention, obtaining the width and feature channel hybrid attention, and generating data of 128×1×W3;

[0180] Merge the three hybrid attention weights output by the three-channel hybrid attention unit at each position and multiply them to generate the feature mixed attention data;

[0181] S433: The Patch Merging unit is used to shuffle the features of different sensors, randomize the sensor sources of the features, and achieve the merging of the features of different sensors. The specific implementation process is as follows:

[0182] In S432, the finally output attention map passes through the shuffle layer to generate a shuffled feature map with odd-even skip connections; through the splicing layer, data with a feature length 4 times the original, and long channels and width channels 1 / 2 of the original are generated; through a linear layer, the feature dimension is halved, generating data with a feature length 2 times the original, and long channels and width channels 1 / 2 of the original;

[0183] S434: The sparse attention unit is used to prevent the model from overfitting during the training of low contribution rate nodes. It consists of 2 mask layers, an encoding layer and a decoding layer;

[0184] Each attention vector with a length of passes through a mask layer. The mask module sets the mask rate to 0.75, sets the attention with the top 0.75 weights before sorting to 0, and the rest to 1. Its masking process is shown in equations (12)-(14):

[0185] (12);

[0186] (13);

[0187] (14);

[0188] where: is the reverse sorting function, is the attention sequence sorted from largest to smallest with a length of which will be used as the input of the encoding module to achieve the purpose of retraining the attention nodes with small contributions; ​is a masking function; is the attention sequence after masking;

[0189] The encoding layer first performs two-dimensional convolution calculations with a convolution kernel size of 3×1 and a stride of 1, generating data with a feature channel that is 1 / 2 of the original; then it goes through batch normalization and the ReLU activation function; then it performs two-dimensional convolution calculations with a convolution kernel size of 3×1 and a stride of 1 again, generating data with a feature channel that is 1 / 4 of the original.

[0190] The decoding layer first performs two-dimensional convolution calculations with a convolution kernel size of 3×1 and a stride of 1, generating data with a feature channel that is 1 / 2 of the original; then it goes through batch normalization and the ReLU activation function; then it performs two-dimensional convolution calculations with a convolution kernel size of 3×1 and a stride of 1 again, generating data with a feature channel of the original dimension.

[0191] S44: The global attention spatio-temporal fusion module is used to achieve full fusion of spatio-temporal attention features; the global attention spatio-temporal fusion module includes a globally-attentive unit and a one-dimensional standard convolution unit connected in sequence, and the specific implementation process is as follows:

[0192] The globally-attentive unit combines a gated linear unit and an attention mechanism, and through the mapping process of the gated linear unit, generates matrices and , and then uses the self-attention mechanism for key feature extraction, and its calculation process is shown in Equation (11):

[0193] (11);

[0194] where: is the output of the globally-attentive unit; is the weight bias; is the query matrix, and are the key and value represented by the matrix respectively, is the ReLU activation function;

[0195] Further perform one-dimensional convolution calculation with a convolution kernel size of 3 and a stride of 1, generating data with a feature channel of 1, and further extract the output of the global attention to ensure that local information is not lost;

[0196] S45: The human behavior classification result output module is used to reduce the dimension and classify the high-dimensional features obtained by feature extraction and fusion to achieve accurate classification of human behaviors, and includes a flattening layer, a layer normalization, a fully-connected layer, and a Softmax output layer connected in sequence.

[0197] S5: Loss function calculation, update and feedback;

[0198] Calculate the loss function for the classification results generated by S4, and adjust the model parameters through backpropagation to improve the classification accuracy and generalization ability. Use the cross-entropy loss function to calculate the error between the model prediction value and the true value. The cross-entropy loss function is defined as shown in Equation (1):

[0199] (1);

[0200] Where: represents the batch data volume, represents the total number of classes, represents the th sample belongs to class true label, represents the th sample belongs to class predicted probability;

[0201] (2);

[0202] Where: represents the model parameters at the th step, represents the model parameters at the th step, represents the learning rate, is a constant added to increase numerical stability, represents the first-order momentum, represents the second-order momentum (exponentially weighted average of the gradient squared), and represent the first-order and second-order averaging coefficients and of power;

[0203] The calculation process of the first-order momentum can be expressed as Equation (3):

[0204] (3);

[0205] Where: represents the gradient of the loss function , represents the first-order averaging coefficient, represents the current first-order momentum value, represents the first-order momentum value at the previous moment;

[0206] The calculation process of the second-order momentum can be expressed as Equation (4):

[0207] (4);

[0208] Where: represents the second-order average coefficient, represents the current second-order momentum value, represents the second-order momentum value at the previous moment;

[0209] In each round of training, the data is input into the model in batches for forward propagation and backward propagation in sequence until all the data is trained. During the training process, the performance of the model is evaluated using the validation set data, and the learning rate or other hyperparameters are dynamically adjusted according to the validation set results to avoid overfitting or underfitting problems. After a round of training ends, only the loss function in Equation (2) is used to return to S4 for training until convergence, so as to further adjust the parameters of the recognition model;

[0210] S6: Application of the human behavior classification output result;

[0211] Transmit the human behavior classification recognition result to the corresponding application platform in real time.

[0212] The result output by the human behavior recognition is transmitted to the intelligent rehabilitation training action monitoring platform for patients. Store, process, calculate, and display the results in real time, and now monitor and evaluate the intelligent rehabilitation training actions of patients in real time.

[0213] The above-disclosed are only specific embodiments of the present invention. However, the present invention is not limited thereto, and any changes that can be conceived by those skilled in the art should fall within the protection scope of the present invention.

Claims

1. A recognition method for a human behavior recognition system based on sparsified three-channel hybrid attention, characterized in that, It includes the following steps: S1: Acquisition of human body behavior data; Real-time acquisition of the user's behavior data, i.e., human body behavior data. The behavior data includes the X, Y, and Z axis data of the acceleration sensor, the X, Y, and Z axis data of the angular velocity sensor, and the X, Y, and Z axis data of the magnetometer. S2: Transmission of human body behavior data; Transmit the acquired user behavior data to the local server or cloud server. S3: Preprocessing of human body behavior data; The acquired user behavior data is successively stored, merged, filtered, normalized or standardized, segmented by a sliding window, and labeled, so as to obtain smooth behavior data segments that are not affected by the dimension and carry labels. S4: Construction of an identification model and classification and identification of human body behavior; The behavior data segments preprocessed in S3 are input into the identification model in batches, and the human body behavior recognition output is obtained through training. S5: Calculation, update, and feedback of the loss function; Calculate the loss function for the classification results generated in S4, and adjust the model parameters through backpropagation to improve the classification accuracy and generalization ability. Use the cross-entropy loss function to calculate the error between the model prediction value and the true value. The definition of the cross-entropy loss function is shown in Equation (1): （1）； Wherein: represents the batch data volume, represents the total number of categories, represents the th sample belonging to the category true label, represents the th sample belonging to the category predicted probability; （2）； Wherein: represents the step model parameter, represents the step model parameter, represents the learning rate, is a constant added to increase numerical stability, represents the first-order momentum, represents the second-order momentum, and represent the first-order and second-order average coefficients and to the power; The calculation process of the first-order momentum can be expressed as Equation (3): （3）； Wherein: represents the gradient of the loss function , represents the first-order average coefficient represents the current first-order momentum value represents the first-order momentum value at the previous moment; The calculation process of the second-order momentum can be expressed as Equation (4): （4）； Wherein: represents the second-order average coefficient, represents the current second-order momentum value, represents the second-order momentum value at the previous moment; S6: Application of the output result of human body behavior classification; Transmit the human body behavior classification and identification results to the corresponding application platform in real time.

2. The recognition method according to claim 1, characterized in that: The specific implementation process of S3 is as follows: S31: Merging of human body behavior data; The acquired user behavior data is spliced and merged. The merged data is in the format of a two-dimensional array, and is arranged horizontally in the order of the X, Y, and Z axis data of the three-axis acceleration sensor, the X, Y, and Z axis data of the three-axis angular velocity sensor, and the X, Y, and Z axis data of the three-axis magnetometer, and is arranged vertically in chronological order. S32: Filtering of human body behavior data; Filtering of human body behavior data refers to smoothing filtering of sensor data. The sensor data is smoothed by using three-point mean smoothing to eliminate the noise signals inside the sensor. Let the sensor signal sequence be , representing the number of data points in the sensor signal sequence; the equations for mean smoothing are shown in Equations (5) and (6) as follows: （5）； （6）； Wherein: represents the value after one-time mean smoothing, represents the value after two-time mean smoothing; S33: Normalization / standardization of human body behavior data; Normalization is performed using the deviation normalization method. Let the input sequence be , and the output sequence after deviation normalization is , as shown in Equation (7): （7）； Wherein: and are respectively the maximum value and the minimum value of all samples in the input sequence; Use the standard score normalization method for standardization processing, as shown in Equation (8): （8）； Wherein: is the mean of all samples in the input sequence, is the input sequence the standard deviation of all samples in; S34: Sliding window segmentation and labeling; Sliding window segmentation is to use a window with a fixed length to segment the continuous sensor data into data segments with a fixed length, assign corresponding labels to each data segment, that is, the name of the behavior category corresponding to the data segment, and encode the labels using one-hot encoding.

3. The recognition method according to claim 2, wherein: The specific implementation process of S4 is as follows: S41: Scan expansion is to perform data augmentation processing on the input time series data using scrambled arrangement and flattening operations, expand it into four time sorting combinations, realize time feature scrambling, and enhance the feature learning effect. Specifically: The size of each feature dimension is to reshape the time series data with a size of into image-like data through reshape, where ; ; Flatten each piece of data in four orders: horizontally from the upper left to the lower right, vertically from the upper right to the lower left, horizontally from the lower right to the upper left, and vertically from the lower right to the upper left, to obtain and . The calculation process of their shuffled and flattened arrangement is shown in Equation (9): （9）； Four new time series and are merged to obtain data augmentation samples , as shown in Equation (10): （10）； Where: Concat represents the merge connection operation; S42: Multi-scale feature segmentation is to separate the sensor features at different positions. In the subsequent feature extraction stage, the grouped data is input into several parallel similar block networks to facilitate the extraction of high-level feature representations of sensor data at different positions in the same activity. Specifically: The data augmentation samples that have passed through the above S41 According to the grouping method of the body positions where the wearable sensors are worn, the multi-source sensor features at different positions are segmented into groups: IMU1, IMU2,..., IMUn; S43: The multi-scale dynamic convolution of sparse three-channel hybrid attention is to deeply extract and fuse features; S44: The global attention spatio-temporal fusion module is used to fully fuse spatio-temporal attention features: The global attention unit combines the gated linear unit and the attention mechanism. Through the mapping process of the gated linear unit, matrices and are generated. Then, the self-attention mechanism is used for key feature extraction, and its calculation process is shown in Equation (11): （11）； Wherein: is the output of the global attention unit; is the weight bias; is the query matrix, and are the key and value represented by matrices respectively, is the ReLU activation function; Further, through one-dimensional convolution calculation with a convolution kernel size of 3 and a stride of 1, data with 1 feature channel is generated, and the output of global attention is further extracted to ensure that local information is not lost; S45: The human behavior classification result output module is used to reduce the dimension and classify the high-dimensional features obtained by feature extraction and fusion to achieve accurate classification of human behaviors, including a flattening layer, a layer normalization, a fully connected layer, and a Softmax output layer connected in sequence.

4. The recognition method according to claim 3, wherein: The specific implementation process of S43 is as follows: S431: The improved squeeze-and-excitation three-channel dynamic convolution unit includes an initial convolution block 0 to obtain potential features of each channel, and three convolution blocks 1, 2, and 3 connected in series in sequence; S432: The three-channel hybrid attention unit consists of three parts: long and width channel hybrid attention, long and feature channel hybrid attention, and width and feature channel hybrid attention; S433: The Patch Merging unit is used to shuffle different sensor features, randomize the sensor sources of features, and merge different sensor features. The specific implementation process is as follows: In S432, the finally output attention map passes through the shuffle layer to generate a shuffled feature map with odd-even skip connections; through the splicing layer, data with a feature length 4 times the original, and long and width channels 1 / 2 of the original is generated; through a linear layer, the feature dimension is halved, and data with a feature length 2 times the original, and long and width channels 1 / 2 of the original is generated; S434: The sparse attention unit is used to prevent model overfitting during training of low contribution rate nodes, and it consists of 2 mask layers, an encoding layer, and a decoding layer; Each attention vector with a length of is passed through a masking layer. The masking module sets the masking rate to 0.75, sets the attention with the top 0.75 weights before weight sorting to 0, and the rest to 1. Its masking process is shown in equations (12)-(14): （12）； （13）； （14）； Wherein: is a reverse sorting function, is of length the attention sequence sorted from largest to smallest, will be used as the input of the encoding module; is a masking function; is the attention sequence after masking; The encoding layer first performs two-dimensional convolution calculations with a convolution kernel size of 3×1 and a stride of 1 to generate data with 1 / 2 of the original feature channels; then passes through batch normalization and ReLU activation function; then performs two-dimensional convolution calculations with a convolution kernel size of 3×1 and a stride of 1 to generate data with 1 / 4 of the original feature channels; The decoding layer first performs two-dimensional convolution calculations with a convolution kernel size of 3×1 and a stride of 1 to generate data with 1 / 2 of the original feature channels; then passes through batch normalization and ReLU activation function; then performs two-dimensional convolution calculations with a convolution kernel size of 3×1 and a stride of 1 to generate data with the original dimension of feature channels.

Images