An open set human behavior recognition system, method and device based on multi-resolution fusion convolution and a storage medium

Abstract

The application relates to an open set human behavior recognition system, method, device and storage medium based on a multi-resolution fusion convolution, which comprises a human behavior information acquisition module, a human behavior information transmission module, a human behavior information preprocessing module, a human behavior classification and recognition module, an open set classification and recognition module and a human behavior information application module connected in sequence; the application can monitor the behavior category of a user in real time, can be used for monitoring and management of special personnel, and can also be used for monitoring and rehabilitation management of the elderly or patients. In view of the defect that the existing method only discriminates against a closed set, the application provides an open set recognition method, unknown categories can be correctly discriminated when unknown categories of behaviors occur, and the practicability of the system is enhanced.

Description

Technical Field

[0001] This invention relates to an open set human behavior recognition system, method, device, and storage medium based on multi-resolution fusion convolution, belonging to the field of artificial intelligence technology. Background Technology

[0002] Human behavior recognition is an emerging research field. With the rapid development of deep learning technology, many deep learning methods have been applied to sensor-based human behavior recognition. Human behavior recognition has applications in many aspects of real life, such as the development of systems for health monitoring, gaming, and behavior surveillance. The development process of human behavior recognition algorithms mainly includes three levels: the collection and processing of human behavior information, the extraction of human behavior features and the training of classification models, and the application of human behavior recognition technology.

[0003] Currently, research in the field of human behavior recognition mainly focuses on human behavior feature extraction and classification model training. Traditional human behavior recognition systems often use manual feature extraction methods in the feature extraction and model building stages, that is, extracting temporal and frequency domain features from sensor data based on expert experience. This approach often suffers from insufficient feature depth or the extracted features are not the optimal features required for behavior recognition. With the development of deep learning, deep learning-based human behavior recognition systems can more effectively and automatically extract deep features than traditional methods, recognize more complex behaviors, and improve behavior recognition accuracy. Convolutional Neural Networks (CNNs) are one of the representative models of deep neural networks and are also one of the most widely used models. This model can extract local features from data sequences spatially, and its weight sharing mechanism effectively reduces the number of model parameters, resulting in higher generalization ability. CNN models consist of convolutional layers and pooling layers, which can fully and automatically extract deep features from sensor data. At the same time, by using a self-attention mechanism, different weights can be assigned to different feature data. During the training process of deep network models, the self-attention mechanism can autonomously learn to give higher attention to key features.

[0004] Currently, the field of behavior recognition faces numerous challenges. For example, as the number of behavior categories increases, classification and recognition become more complex. Effectively analyzing and processing raw and feature data of human behavior, better extracting various deep features from sensor sequences, and improving behavior recognition accuracy are all significant challenges in sensor-based human behavior recognition. Furthermore, existing human behavior recognition systems typically only target closed sets. That is, during model training, the types of behaviors in the training set are the same as those used during model testing or actual application. Behaviors not present in the training set cannot be incorrectly identified as known behaviors. Open set recognition, on the other hand, can correctly identify behaviors of unknown categories as unknown classes, thus avoiding the problem of forced misclassification inherent in open set recognition. Summary of the Invention

[0005] To address the shortcomings of existing behavior recognition systems in feature extraction methods and their inability to identify open set categories, this invention proposes an open set human behavior recognition system based on multi-resolution fusion convolution.

[0006] This invention also proposes an open set human behavior recognition method, computer equipment, and storage medium based on multi-resolution fusion convolution. Invention Overview:

[0008] For the open-set human behavior recognition system based on multi-resolution fusion convolution involved in this invention, firstly, the user wears a wearable device for data collection, which has various built-in sensors, and different types can be selected and worn on different parts of the body according to different needs. Then, the collected data is transmitted wirelessly to a local server or cloud server for storage through a data transmission module, and the transmission method can be selected according to different usage scenarios. Next, the data is preprocessed, including filtering, data augmentation, normalization / standardization, and sliding window segmentation. Filtering is used to remove noise from the original sensor signals and gravitational acceleration signals from the acceleration data; data augmentation can expand the number of samples, providing more data for training the model and improving the model's robustness; normalization / standardization can transform data with different dimensions and value ranges to the same range, which is beneficial for distance-based calculation models to eliminate the influence caused by different dimensions and value ranges; the sliding window segmentation method divides continuous long-term data into data segments that conform to the input format of the discriminant model. Preprocessed data is input into the human behavior classification and recognition module, and then processed by the multi-resolution fusion convolution module to obtain feature representations of different resolutions for each axis of the sensor. Data of the same resolution across different axes are fused together, and key features are enhanced through a self-attention mechanism. This operation ensures that the data for each axis remains at a high resolution, and high-resolution features can also be recovered from low-resolution features through upsampling, thereby enhancing the data feature representation. In the case of a closed set, the data can directly yield the corresponding recognition result through the recognition model. However, in the case of an open set, it is necessary to make similar data closer together and dissimilar data farther apart to further narrow the distribution range of known categories in the feature space, which is beneficial for the recognition of unknown categories. A loss function is constructed by calculating the mean and variance of each category, and the model parameters are adjusted in each round of training. Finally, the threshold for open set recognition is determined through experiments, and the unknown category in the open set is then identified based on the threshold.

[0009] This invention is a general-purpose sensor-based human behavior recognition system that can monitor user behavior categories in real time. It can be used for monitoring and managing special personnel, as well as for elderly care and patient rehabilitation management. Addressing the problem that existing algorithms only extract features by reducing resolution and resulting in incomplete feature extraction, this system provides a multi-resolution fusion feature extraction method to solve this problem, further enhancing feature representation and improving behavior recognition accuracy. Furthermore, to address the deficiency of existing methods that only distinguish between closed sets, a method for identifying open sets is proposed. When data of unknown categories appears, it can be correctly identified as belonging to an unknown category, thus avoiding errors caused by forced classification.

[0010] The technical solution of this invention is as follows:

[0011] An open set human behavior recognition system based on multi-resolution fusion convolution includes a human behavior information acquisition module, a human behavior information transmission module, a human behavior information preprocessing module, a human behavior classification and recognition module, an open set classification and recognition module, and a human behavior information application module connected in sequence.

[0012] The human behavior information acquisition module is used to: collect user behavior data in real time, including X, Y, and Z axis data from the accelerometer, X, Y, and Z axis data from the angular velocity sensor, and X, Y, and Z axis data from the magnetometer.

[0013] The human behavior information transmission module is used to transmit the collected user behavior data to a local server or a cloud server.

[0014] The human behavior information preprocessing module is used to: sequentially store, merge, filter, augment, normalize or standardize, segment by sliding window, and label the collected user behavior data, thereby obtaining smooth, diverse, dimensionless, and labeled behavior data fragments.

[0015] The human behavior classification and recognition module is used to input the user's behavior data preprocessed by the human behavior information preprocessing module into the trained human behavior classification and recognition module for behavior type discrimination.

[0016] The open set classification and recognition module is used to: accurately identify behavioral data input into the recognition model in the case of open sets.

[0017] The human behavior information application module is used to transmit the obtained human behavior classification and recognition results to various application platforms, thereby realizing the corresponding functions.

[0018] According to a preferred embodiment of the present invention, the human behavior classification and recognition module includes a multi-resolution fusion convolution module, a multi-resolution attention parallel convolution module, a fusion attention convolution module, and a closed set output module connected in sequence.

[0019] The multi-resolution fusion convolution module is used to sample preprocessed user behavior data to generate data at different resolutions. The multi-resolution fusion convolution module includes a data upsampling unit, a data downsampling unit, and a first data multi-resolution fusion convolution unit connected in sequence. The data upsampling unit doubles the length of the preprocessed user behavior data using a direct copying method. The data downsampling unit reduces the length of the preprocessed user behavior data to 1 / 2 and 1 / 4 of the original data length, respectively, using convolution calculations with a kernel size of 3×1 and strides of 2 and 4. Additionally, the preprocessed user behavior data is subjected to convolution calculations with a kernel size of 3×1 and a stride of 1 to generate feature data at the same resolution as the original data (i.e., the preprocessed user behavior data). The first data multi-resolution fusion convolution unit then performs fusion convolution calculations at different resolutions using upsampling and downsampling methods to further extract features from the sensor data at different resolutions.

[0020] The multi-resolution attention parallel convolution module is used to extract deep features from sensor data of different resolutions. This module includes a same-resolution data stitching unit, a parallel self-attention unit, and a parallel convolution unit connected sequentially. The same-resolution data stitching unit stitches together four different resolutions of sensor data (X, Y, Z axes from the accelerometer, X, Y, Z axes from the angular velocity sensor, and X, Y, Z axes from the magnetometer—a total of nine axes of sensor data processed by the first multi-resolution data fusion convolution unit) according to the principle of the same resolution, ensuring that all data at different resolutions include nine axes of sensor data. The parallel self-attention unit automatically weights important features in the data at different resolutions, giving more attention to these important features. The parallel convolution unit further extracts deep features from the data processed by the self-attention unit.

[0021] The fusion attention convolution module is used to perform deep fusion of feature data at different resolutions. The fusion attention convolution module includes a second data multi-resolution fusion convolution unit, a fusion self-attention unit, and a fusion convolution unit connected in sequence. The second data multi-resolution fusion convolution unit is used to fuse parallel feature data at different resolutions into feature data with a length equal to the size of the original sensor data window. The fusion self-attention unit is used to automatically extract important features from the fused deep feature data. The fusion convolution unit further performs deep feature extraction.

[0022] The closed set output module is used to output the closed set classification and recognition results and the open set feature vector. The closed set output module includes a fully connected unit and a closed set classification and discrimination output unit connected in sequence. The fully connected unit is a multi-layer fully connected layer that performs feature space transformation on the data. The closed set classification and discrimination output unit calculates and outputs the recognition results through a Softmax layer.

[0023] According to a preferred embodiment of the present invention, the open set classification and identification module includes a loss function calculation and update module and an open set output module connected in sequence;

[0024] The loss function calculation and update module is used to calculate the mean and variance of known category behavior data during training, thereby generating a loss function to update the parameters of the recognition model. The loss function calculation and update module includes a mean and variance calculation unit and a loss function construction unit connected in sequence. In the mean and variance calculation unit, the vectors of the second-to-last layer in the fully connected layer are arranged according to the same category, and the mean and variance of each category are calculated respectively. The loss function is constructed using the mean and variance. During training, training is first performed according to the data size B in each batch, using a closed set method. B is set to any positive integer less than the total amount of data. After each round of training, the constructed loss function is used to adjust the parameters of the recognition model. This process is repeated until convergence.

[0025] The open set output module is used to calculate the determination threshold for known and unknown categories and output the determined category. It includes a category threshold determination unit and an open set discrimination output unit connected in sequence. In the category threshold determination unit, the distance between all correctly classified feature data points in each category and the central mean point is calculated, and normalization is performed. The module selects the category from Z... i The threshold ∈ (0,1) is used to calculate the proportion of feature data points contained in each category under different thresholds. The threshold Z is calculated when the proportion exceeds 90%. i This refers to the threshold for classifying open sets, where i represents different behavior categories, i = 1, 2, 3, ..., T, and T is the total number of known behavior categories. When new data is input into the recognition model, the normalized distance between the feature vector of the second-to-last fully connected layer and the mean of the centers of different known categories is calculated. When this distance is less than the threshold Z of a certain category... i If the value is greater than the threshold Z for each category, it is immediately classified as belonging to that category. i If it is, it is determined to be an unknown category; the open set discrimination output unit outputs the result based on the above discrimination.

[0026] According to a preferred embodiment of the present invention, the human behavior information acquisition module includes several different sensor units, which are integrated into the wearable device and worn at corresponding positions on the user's body. The sensor units include a triaxial accelerometer, a triaxial angular velocity sensor, and a triaxial magnetometer, for collecting the user's behavior data.

[0027] According to a preferred embodiment of the present invention, the collected user behavior data is transmitted to a local server or a cloud server via any one of ultra-wideband, Bluetooth, or 5G transmission methods.

[0028] According to a preferred embodiment of the present invention, the human behavior information preprocessing module includes, in sequence, a sensor data storage unit, a sensor data merging unit, an acceleration data gravity filtering unit, a sensor data smoothing filtering unit, a data augmentation unit, a data normalization / standardization unit, a sliding window segmentation unit, and a data label calibration unit;

[0029] The sensor data storage unit is used to store real-time user behavior data;

[0030] The sensor data merging unit is used to merge the behavioral data of different sensor units. The merged data is in a two-dimensional array format. The data is arranged horizontally in the order of X, Y, and Z axis data of the triaxial accelerometer, X, Y, and Z axis data of the triaxial angular velocity sensor, and X, Y, and Z axis data of the triaxial magnetometer. The data is arranged vertically in chronological order.

[0031] The acceleration data gravity filtering unit is used to filter out the inherent gravitational acceleration to obtain the actual behavioral acceleration; specifically, it uses a Butterworth filter to eliminate gravitational acceleration and obtain the true acceleration during human movement.

[0032] The sensor data smoothing and filtering unit is used to remove noise from the raw sensor data;

[0033] The data augmentation unit is used to expand the dataset;

[0034] The data normalization / standardization unit is used to convert data with different dimensions and value ranges to the same value range.

[0035] The sliding window segmentation unit is used to segment continuous long-term sensor data into several data segments.

[0036] The data labeling unit is used to label the data after it has been segmented by a sliding window and to encode the labels using one-hot encoding.

[0037] According to a preferred embodiment of the present invention, the human behavior information application module is oriented towards various application platforms, and the classification and recognition results are transmitted to the database of each application platform in real time for storage and management, thereby analyzing, managing, and visualizing the user's behavior in real time.

[0038] An open-set human behavior recognition method based on multi-resolution fusion convolution is implemented through the aforementioned open-set human behavior recognition system based on multi-resolution fusion convolution, and includes the following steps:

[0039] Step S1: Human Behavioral Data Collection

[0040] Real-time collection of user behavior data, i.e. human behavior data, includes X, Y, and Z axis data from accelerometers, X, Y, and Z axis data from angular velocity sensors, and X, Y, and Z axis data from magnetometers.

[0041] Step S2: Human Behavior Data Transmission

[0042] The collected user behavior data is transmitted to a local server or a cloud server;

[0043] Step S3: Preprocessing of human behavior data

[0044] The collected user behavior data is sequentially stored, merged, filtered, augmented, normalized or standardized, segmented by sliding window, and labeled to obtain smooth, diverse, dimensionless, and labeled behavior data fragments.

[0045] Step S4: Constructing the recognition model and classifying closed sets

[0046] The behavior data fragments preprocessed in step S3 are input into the recognition model in batches, and the recognition output of the closed set is obtained after training.

[0047] The closed set output module includes a Flatten layer, a Dense fully connected layer, a Drop pruning layer, a LayerNorm normalization layer, two Dense fully connected layers, and a Softmax output layer connected in sequence.

[0048] Step S5: Loss Function Calculation, Update, and Feedback

[0049] In the case of open set, the feature vector output by the first Dense fully connected layer of the two Dense fully connected layers in the closed set output module in step S4 is processed according to the principle of the same category. The mean and variance of the feature vector center that belongs to the same category and is finally correctly classified are calculated, as shown in formula (1) and formula (2) respectively. In addition, the variance is normalized, as shown in formula (3).

[0050]

[0051]

[0052]

[0053] Where, μ i Let σ represent the center mean vector of category i. i ) 2 Nor((σ) represents the variance of category i. i ) 2 ) represents the variance of class i after normalization, |C i | represents the number of correctly classified feature vectors in category i. Let represent the feature vector output by the second-to-last fully connected layer for the j-th data in category i, and T be the total number of all known behavior categories;

[0054] The loss function is set as formula (4):

[0055]

[0056] Where max(Δμ) represents the maximum distance between the pairwise mean values ​​of the class centers, calculated using the 1-norm;

[0057] In each round of training, the known category data is first input into the recognition model in batches according to the amount of data B in each batch for parameter updates. B is set to any positive integer less than the total amount of data. At this time, the cross-entropy loss function and Softmax function commonly used in closed set training are used for training and output. After one round of training is completed, only the loss function L in formula (4) is used to return to step S4 for training until convergence, so as to further adjust the recognition model parameters and make the data of the same type closer in the multidimensional space and the data of different types farther apart.

[0058] Step S6: Determine the threshold and output the open set discrimination.

[0059] After the recognition model is trained, the distance between all correctly classified feature data points in each category and the central mean point is calculated, and normalization is performed as shown in formulas (5) and (6):

[0060]

[0061]

[0062] Where, Δz i This represents the average distance between the correctly classified feature vectors in the i-th category and the median mean. represents the normalized distance from the median mean point to the correctly classified j-th feature vector in the i-th category, with a value range of [0,1].

[0063] Choose from Z i For thresholds ∈ (0,1), calculate the proportion of feature data points included in each category under different thresholds, i.e. The percentage of feature data points within the range, and the threshold Z when the percentage exceeds 90%. i The threshold for classifying an open set is denoted as i, where i represents different behavior categories, i = 1, 2, 3, ..., T, and T is the total number of known behavior categories.

[0064] When new data is input into the recognition model, the normalized distance between the feature vector of its penultimate fully connected layer and the mean of the centers of different known classes is calculated. when Less than or equal to the threshold Z of a certain category i If the value is greater than the threshold Z for each category, it is immediately classified as belonging to that category. i It is then classified as an unknown category;

[0065] Step S7: Determine the application of the output result

[0066] The results output from the closed set and the open set are transmitted to the corresponding application platform in real time.

[0067] According to a preferred embodiment of the present invention, step S3 is specifically implemented as follows:

[0068] Step S31: Merging Human Behavioral Data

[0069] The collected user behavior data is concatenated and merged. The merged data is in a two-dimensional array format. Horizontally, it is arranged in the order of X, Y, and Z axis data from the three-axis accelerometer, X, Y, and Z axis data from the three-axis angular velocity sensor, and X, Y, and Z axis data from the three-axis magnetometer. Vertically, it is arranged in chronological order.

[0070] Step S32: Filtering of human behavior data

[0071] Human behavior data filtering includes gravity filtering for acceleration data and smoothing filtering for sensor data;

[0072] Gravity filtering of acceleration data uses a third-order Butterworth filter with a cutoff frequency of 0.3Hz to eliminate gravitational acceleration and extract pure human behavior signals. This third-order Butterworth filter is expressed as formula (7):

[0073]

[0074] Among them, w c w is the cutoff frequency. p Where ω is the edge frequency of the passband, and w is the angular frequency. It is |H(w)| 2 Values ​​at the edge of the passband.

[0075] Sensor data smoothing filtering uses three-point mean smoothing filtering to eliminate noise signals within the sensor. Let the sensor signal sequence be x = [x1, x2, ..., x...]. i ,...,x n ] T , where n represents the number of data points in the sensor signal sequence; the formulas for mean smoothing are shown in formulas (8) and (9):

[0076]

[0077]

[0078] in, This represents the value after one round of mean smoothing. This represents the value after two mean smoothing operations.

[0079] Step S33: Normalization / Standardization of Human Behavioral Data

[0080] The Min-Max normalization method is used for normalization, which transforms the values ​​of all similar sensors to the range [0, 1]. The Min-Max normalization method scales all data values ​​to the range [0, 1] through a linear transformation of the original data. Let the input sequence be x = [x1, x2, ..., x...]. i ,...,x n ] T The output sequence after Min-Max normalization is y = [y1, y2, ..., y]. i ,...,y n ] T As shown in formula (10):

[0081]

[0082] Where, x max and x min Let x be the maximum and minimum values ​​of all samples in the input sequence x, respectively.

[0083] The Z-Score standardization method is used to standardize the data, which transforms the data into data with a mean of 0 and a standard deviation of 1. The Z-Score standardization method transforms the data into data with a mean of 0 and a standard deviation of 1, as shown in formula (11):

[0084]

[0085] Where μ is the mean of all samples in the input sequence x, and σ is the standard deviation of all samples in the input sequence x;

[0086] Step S34: Sliding window splitting and labeling

[0087] Sliding window segmentation uses a fixed-length window to divide continuous sensor data into fixed-length data segments, assigns a corresponding label to each data segment (i.e., the behavior category name corresponding to that data segment), and encodes the label using one-hot encoding.

[0088] According to a preferred embodiment of the present invention, step S4 includes the following steps:

[0089] Step S41: Multi-resolution fusion convolution uses upsampling and downsampling to generate data feature maps of different resolutions, specifically:

[0090] Each 80×1 row data segment is transformed into a 160×1 row data segment, and upsampling is performed using direct copying.

[0091] Each row of data fragments with a length of 80 is transformed into row data fragments with a length of 40 and 20 respectively, and the data lengths are reduced to 1 / 2 and 1 / 4 of the original data lengths respectively. Downsampling is performed using convolution with a kernel size of 3×1 and strides of 2 and 4 respectively.

[0092] In addition, feature data with the same resolution as the original data is generated by convolution calculation with a kernel size of 3×1 and a stride of 1.

[0093] By using upsampling and downsampling methods again, we can perform fusion convolution calculations at different resolutions to further extract features from the single-axis data of the sensor at different resolutions.

[0094] Step S42: Multi-resolution attention parallel convolution is used to extract deep features from sensor data at different resolutions, specifically:

[0095] The X, Y, Z axis data of the accelerometer, the X, Y, Z axis data of the angular velocity sensor, and the X, Y, Z axis data of the magnetometer, which are obtained after the fusion convolution calculation of different resolutions in step S41, are used to generate four types of resolution data. These data are then stitched together according to the principle of the same resolution, so that the data of different resolutions all include the 9-axis sensor data. This results in four types of feature data with different resolutions: high-resolution feature data of size 160×9, original resolution feature data of size 80×9, medium-resolution feature data of size 40×9, and low-resolution feature data of size 20×9.

[0096] Key features are extracted in parallel using a self-attention mechanism on feature data of four different resolutions. The calculation process is shown in formula (12):

[0097]

[0098] Where, d k Q is the key dimension, K and V are the keys and values ​​represented by the matrix, respectively; the more important and classification-friendly features A(Q,K,V) are automatically extracted from the single-axis feature data of the sensor through a self-attention mechanism.

[0099] Subsequently, the feature data at different resolutions were subjected to convolution calculations in parallel with a kernel size of 3×3 and a stride of 1, so as to fully extract the features between the multi-axis sensor data at each resolution.

[0100] Step S43: Fusion attention convolution is a deep fusion of feature data at different resolutions, specifically referring to:

[0101] By upsampling and downsampling, the feature resolution between the multi-axis sensor data obtained in step S42 is converted into feature data of size 80×9, and then spliced ​​into feature data of size 80×36.

[0102] The self-attention mechanism is used to extract important features from multi-axis feature data of different sensors. Then, convolution calculation with a kernel size of 3×3 and a stride of 1 is used to fully extract and fuse the features between multi-axis sensor data of different resolutions.

[0103] Step S44: Classification and recognition involves outputting the classification and recognition results for the closed set and the feature vectors for the open set. Specifically, this means:

[0104] The closed set output module includes a Flatten layer, a Dense fully connected layer, a Drop pruning layer, a LayerNorm normalization layer, two Dense fully connected layers, and a Softmax output layer connected in sequence.

[0105] According to a preferred embodiment of the present invention, in step S41, fusion convolution calculations at different resolutions are performed again using upsampling and downsampling methods to further extract features from the sensor's single-axis data at different resolutions. The specific implementation process is as follows:

[0106] Step S411: The high-resolution data with a length of 160, i.e., the behavioral data segment with a length of 160×1, is processed by convolution with a kernel size of 3×1 and a stride of 1 to generate high-resolution data with the same length of 160; the high-resolution data with a length of 160 is downsampled by convolution with a kernel size of 3×1 and a stride of 2 to generate the original resolution data with a length of 80.

[0107] Step S412: The original resolution data with a length of 80 is upsampled by direct copying to generate high-resolution data with a length of 160; the original resolution data with a length of 80 is calculated by convolution with a kernel size of 3×1 and a stride of 1 to generate original resolution data with the same length of 80; the original resolution data with a length of 80 is downsampled by convolution with a kernel size of 3×1 and strides of 2 and 4 respectively to generate medium-resolution data with a length of 40 and low-resolution data with a length of 20.

[0108] Step S413: The medium-resolution data of length 40 is upsampled by direct copying to generate the original resolution data of length 80; the medium-resolution data of length 40 is calculated by convolution with kernel size 3×1 and stride 1 to generate medium-resolution data of the same length 40; the medium-resolution data of length 40 is downsampled by convolution with kernel size 3×1 and stride 2 to generate low-resolution data of length 20.

[0109] Step S414: The low-resolution data of length 20 is upsampled by direct copying to generate original resolution data and medium resolution data of length 80 and 40, respectively; the low-resolution data of length 20 is generated by convolution with a kernel size of 3×1 and a stride of 1 to generate low-resolution data of the same length 20.

[0110] The beneficial effects of this invention are as follows:

[0111] 1. This invention proposes an open set human behavior recognition system based on multi-resolution fusion convolution. By processing and analyzing sensor data, it achieves real-time classification of human behavior, making up for the shortcomings of video information, such as susceptibility to obstacles and lighting, poor privacy, and high computational load.

[0112] 2. In the human behavior information classification module, this invention proposes a new deep learning algorithm based on multi-resolution fusion convolution to learn the deep features of multi-sensor data at different resolutions, extract the features of sensor data more accurately, and improve the accuracy of behavior recognition.

[0113] 3. This invention proposes an open set human behavior recognition method. By setting an appropriate loss function, adjusting model parameters, and setting an open set threshold, it can solve the problem that existing classification models cannot recognize unknown category data. Attached Figure Description

[0114] Figure 1 This is a schematic diagram showing the module composition and connection relationship of the open set human behavior recognition system based on multi-resolution fusion convolution of the present invention;

[0115] Figure 2This is a flowchart illustrating the open set human behavior recognition method based on multi-resolution fusion convolution of the present invention.

[0116] Figure 3 This is a schematic diagram illustrating the principle of human behavior information classification in the open set human behavior recognition method based on multi-resolution fusion convolution of the present invention. Detailed Implementation

[0117] The present invention will be further defined below with reference to the accompanying drawings and embodiments, but is not limited thereto.

[0118] Example 1

[0119] Many problems exist in the management of special personnel, such as the frequent occurrence of fights and brawls, which are difficult to detect and stop in a timely manner. Although cameras are now deployed throughout special locations requiring monitoring, it is difficult for managers to promptly detect abnormal behavior among the numerous video feeds. Furthermore, cameras are susceptible to obstruction, poor lighting, and blind spots. Wearable device-based human behavior recognition technology can compensate for these shortcomings. Special personnel wear a wristband on their right wrist, equipped with a built-in three-axis accelerometer, three-axis angular velocity sensor, and three-axis magnetometer. This wristband can collect behavioral data in real time and transmit it to a local or cloud server for storage and computation, yielding corresponding recognition results. The special personnel management application platform can display these results in real time and promptly alert authorities to individuals exhibiting abnormal behavior such as fighting. Furthermore, it can also manage the behavior of special personnel long-term, preventing abnormal behavior at night or during inactive hours.

[0120] An open-set human behavior recognition system based on multi-resolution fusion convolution is applied to a special personnel management application platform, such as... Figure 1 As shown, it includes a human behavior information acquisition module, a human behavior information transmission module, a human behavior information preprocessing module, a human behavior classification and recognition module, an open set classification and recognition module, and a human behavior information application module connected in sequence.

[0121] The human behavior information acquisition module is used to collect user behavior data in real time. This data is acquired through sensors, including X, Y, and Z-axis data from an accelerometer, an angular velocity sensor, and a magnetometer. Users can wear multiple wearable devices at different locations on their bodies, each containing various built-in sensors. The human behavior information acquisition module comprises several different sensor units integrated into the wearable device, worn at corresponding locations on the user's body. These sensor units include a three-axis accelerometer, a three-axis angular velocity sensor, and a three-axis magnetometer, used to collect user behavior data.

[0122] The human behavior information transmission module is used to transmit collected user behavior data to a local server or cloud server. Depending on the user's location, a suitable transmission method can be selected to transmit the collected user behavior data to the local server or cloud server via Ultra Wide Band (UWB), Bluetooth, or 5G. Different transmission methods differ in power consumption, transmission distance, and transmission speed, thus making them suitable for different application scenarios.

[0123] The human behavior information preprocessing module is used to: store, merge, filter, augment, normalize or standardize, segment by sliding window, and label the collected user behavior data in sequence, so as to obtain smooth, diverse, dimensionless, and labeled behavior data fragments.

[0124] The human behavior classification and recognition module is used to input the user's behavior data preprocessed by the human behavior information preprocessing module into the trained human behavior classification and recognition module to determine the behavior type.

[0125] The open set classification and recognition module is used to accurately identify behavioral data input into the recognition model in the case of open sets.

[0126] The human behavior information application module is used to transmit the obtained human behavior classification and recognition results to various application platforms, thereby realizing corresponding monitoring, management, and rehabilitation nursing functions. This module is compatible with various application platforms, such as elderly care monitoring and management platforms, rehabilitation nursing monitoring and management platforms, and more. The model classification and recognition results are transmitted in real time to the databases of each application platform for storage and management, enabling real-time analysis, management, and visualization of user behavior.

[0127] This system achieves real-time alarms for abnormal behaviors through the processing and analysis of sensor data. In the human behavior information classification module, a novel deep learning framework based on multi-resolution fusion convolution is proposed to learn the deep features of multi-sensor time series data at different resolutions, effectively improving the accuracy of complex behavior recognition. At the same time, an open set recognition method is proposed, which can correctly identify unknown category data into the model without incorrectly identifying it as a known category.

[0128] Example 2

[0129] The difference between the open set human behavior recognition system based on multi-resolution fusion convolution described in Example 1 and the one described in Example 1 is as follows:

[0130] The human behavior classification and recognition module includes a multi-resolution fusion convolution module, a multi-resolution attention parallel convolution module, a fusion attention convolution module, and a closed set output module connected in sequence.

[0131] The multi-resolution fusion convolution module is used to sample preprocessed user behavior data to generate data at different resolutions. The module includes a data upsampling unit, a data downsampling unit, and a first data multi-resolution fusion convolution unit connected in sequence. The data upsampling unit expands data of length 80 to length 160, doubling the length of the preprocessed user behavior data, using a direct copying method for upsampling. The data downsampling unit samples data of length 80 to lengths of 40 and 20 respectively, reducing the length of the preprocessed user behavior data to half and one-quarter of the original length, respectively, using convolution calculations with kernel sizes of 3×1 and strides of 2 and 4. Additionally, the preprocessed user behavior data is processed through convolution calculations with kernel sizes of 3×1 and strides of 1 to generate feature data at the same resolution as the original data (i.e., the preprocessed user behavior data). The first data multi-resolution fusion convolution unit further performs fusion convolution calculations at different resolutions using upsampling and downsampling methods, further extracting features from the sensor data at different resolutions.

[0132] The multi-resolution attention parallel convolution module is used to extract deep features from sensor data of different resolutions. This module includes a same-resolution data stitching unit, a parallel self-attention unit, and a parallel convolution unit connected sequentially. The same-resolution data stitching unit stitches together four different resolutions of sensor data (X, Y, Z axes from the accelerometer, X, Y, Z axes from the angular velocity sensor, and X, Y, Z axes from the magnetometer—a total of nine axes of sensor data processed by the first multi-resolution data fusion convolution unit) according to the principle of the same resolution, ensuring that all data at different resolutions include nine axes of sensor data. The parallel self-attention unit automatically weights important features in the data at different resolutions, giving more attention to these important features. The parallel convolution unit further extracts deep features from the data processed by the self-attention unit.

[0133] The fusion attention convolution module is used to perform deep fusion of feature data at different resolutions. The fusion attention convolution module includes a second data multi-resolution fusion convolution unit, a fusion self-attention unit, and a fusion convolution unit connected in sequence. The second data multi-resolution fusion convolution unit uses the same calculation method as the first data multi-resolution fusion convolution unit, and is used to fuse parallel feature data at different resolutions into feature data with a length equal to the size of the original sensor data window. The fusion self-attention unit uses the same calculation method as the parallel self-attention unit described above, and is used to automatically extract important features from the fused deep feature data. The fusion convolution unit further performs deep feature extraction.

[0134] The closed set output module is used to output the closed set classification and recognition results and the open set feature vector. The closed set output module includes a fully connected unit and a closed set classification and discrimination output unit connected in sequence. The fully connected unit is a multi-layer fully connected layer that performs feature space transformation on the data. The closed set classification and discrimination output unit calculates and outputs the recognition results through the Softmax layer.

[0135] The open set classification and recognition module includes a loss function calculation and update module and an open set output module connected in sequence;

[0136] The loss function calculation and update module is used to calculate the mean and variance of known category behavior data during training, thereby generating a loss function to update the parameters of the recognition model. This module includes sequentially connected units for calculating the mean and variance of data within the same category, and a loss function construction unit. In the unit for calculating the mean and variance of data within the same category, the vectors of the second-to-last layer in the fully connected layer are arranged according to the same category, and the mean and variance of each category are calculated separately. The loss function is then constructed using the mean and variance. During training, training is performed using a closed set method with a batch size B of data. B is set to any positive integer less than the total number of data points, for example, B = 64. After each round of training, the constructed loss function is used to adjust the parameters of the recognition model, and this process is repeated until convergence.

[0137] The open set output module is used to calculate the decision threshold for known and unknown classes and output the determined class. It includes a class threshold determination unit and an open set discrimination output unit connected in sequence. In the class threshold determination unit, the distances between all correctly classified feature data points in each class and the central mean point are calculated, and normalization is performed. The selection is based on Z... i The threshold ∈ (0,1) is used to calculate the proportion of feature data points contained in each category under different thresholds. The threshold Z is calculated when the proportion exceeds 90%. i This refers to the threshold for classifying open sets, where i represents different behavior categories, i = 1, 2, 3, ..., T, and T is the total number of known behavior categories. When new data is input into the recognition model, the normalized distance between the feature vector of the second-to-last fully connected layer and the mean of the centers of different known categories is calculated. When this distance is less than the threshold Z of a certain category... i If the value is greater than the threshold Z for each category, it is immediately classified as belonging to that category. i If it is, it is determined to be an unknown category; the open set discrimination output unit outputs the result based on the above discrimination.

[0138] The human behavior information preprocessing module includes, in sequence, a sensor data storage unit, a sensor data merging unit, an acceleration data gravity filtering unit, a sensor data smoothing filtering unit, a data augmentation unit, a data normalization / standardization unit, a sliding window segmentation unit, and a data label calibration unit;

[0139] The sensor data storage unit is used to store real-time user behavior data;

[0140] The sensor data merging unit is used to combine behavioral data from different sensor units. The merged data is in a two-dimensional array format, arranged horizontally in the order of X, Y, and Z axis data from the triaxial accelerometer, X, Y, and Z axis data from the triaxial angular velocity sensor, and X, Y, and Z axis data from the triaxial magnetometer, and vertically in chronological order. The sensor data acquisition frequency H can be set to an appropriate value, such as H = 20Hz, which means that the data generated per second is 20 rows and 9 columns.

[0141] The acceleration data gravity filtering unit is used to filter out the inherent gravitational acceleration to obtain the actual behavioral acceleration; to solve this problem, specifically, it means: using a Butterworth filter to eliminate gravitational acceleration and obtain the true acceleration during human movement.

[0142] The sensor data smoothing and filtering unit is used to remove noise from the raw sensor data. Typically, sensor data contains a large amount of high-frequency noise, which can be removed by mean smoothing filtering.

[0143] Data augmentation units are used to expand the dataset. Deep models require a large amount of data during training to learn useful parameters, but currently, the amount of data is often insufficient. Data augmentation of time series can be performed using the average selection method. The average selection method involves randomly selecting a sequence from the original dataset as the initial average vector, with a weight of α, where α∈(0,1); then finding N sequences that are closest to it, where N is any positive integer, and selecting n sequences from these, where n≤N, with weights set to β; the remaining weights 1-(α+nβ) are evenly distributed among the remaining sequences, ensuring that α+nβ<1. For example, α=0.5, N=5, n=2, β=0.15 can be set.

[0144] Data normalization / standardization units are used to transform data with different dimensions and value ranges to the same value range, avoiding the adverse effects of different dimensions and value ranges on calculations. Normalization can use the Min-Max normalization method, which transforms the values ​​of all similar sensors to the range [0,1]. Standardization can use the Z-Score standardization method, which transforms the data into data with a mean of 0 and a standard deviation of 1, thereby making previously incomparable values ​​comparable.

[0145] The sliding window segmentation unit is used to divide continuous, long-term sensor data into several data segments; it is suitable for recognition model input. Based on the characteristics of human behavior, the length W of a window can be set to 80, which can contain sensor data for W / H = 4 seconds, essentially covering a complete set of common human behavioral data. The sliding step size S can be set to 40, i.e., S / W = 50% data coverage.

[0146] The data tagging unit is used to tag the data after it has been segmented by a sliding window. The size of a data segment is W rows and M columns, where M is the number of columns after all sensor data have been merged. When using a triaxial accelerometer, a triaxial angular velocity sensor, and a triaxial magnetometer, the size of a data segment is 80 rows and 9 columns. Each data segment corresponds to a specific tag, and the tag is one-hot encoded.

[0147] Example 3

[0148] An open-set human behavior recognition method based on multi-resolution fusion convolution is proposed, implemented through the open-set human behavior recognition system based on multi-resolution fusion convolution described in either Example 1 or 2. Figure 2 As shown, taking a special personnel management application platform as an example, the steps include the following:

[0149] Step S1: Human Behavioral Data Collection

[0150] Real-time collection of user behavior data, i.e., human behavior data, includes X, Y, and Z-axis data from accelerometers, X, Y, and Z-axis data from angular velocity sensors, and X, Y, and Z-axis data from magnetometers. Several wearable devices with different built-in sensor units are worn on corresponding positions on the user's body. The sensor units mainly include triaxial accelerometers, triaxial angular velocity sensors, and triaxial magnetometers to collect user behavior information.

[0151] Step S2: Human Behavior Data Transmission

[0152] The collected user behavior data is transmitted to a local server or cloud server; depending on the different scenarios and user needs, an appropriate data transmission method is selected, such as UWB, Bluetooth, 5G, etc.

[0153] Step S3: Preprocessing of human behavior data

[0154] The collected user behavior data is sequentially stored, merged, filtered, augmented, normalized or standardized, segmented by sliding window, and labeled to obtain smooth, diverse, dimensionless, and labeled behavior data fragments.

[0155] The specific implementation process is as follows:

[0156] Step S31: Merging Human Behavioral Data

[0157] The collected user behavior data is concatenated and merged. The merged data is in a two-dimensional array format, arranged horizontally in the order of X, Y, and Z axis data from the accelerometer, X, Y, and Z axis data from the angular velocity sensor, and X, Y, and Z axis data from the magnetometer, and vertically in chronological order. The sensor data acquisition frequency H can be set to an appropriate value, such as H = 20Hz, which means that the data generated per second is 20 rows and 9 columns.

[0158] Step S32: Filtering of human behavior data

[0159] Human behavior data filtering includes gravity filtering for acceleration data and smoothing filtering for sensor data;

[0160] Gravity filtering of acceleration data uses a third-order Butterworth filter with a cutoff frequency of 0.3Hz to eliminate gravitational acceleration and extract pure human behavior signals. This third-order Butterworth filter is expressed as formula (7):

[0161]

[0162] Among them, w c w is the cutoff frequency. p Where ω is the edge frequency of the passband, and w is the angular frequency. It is |H(w)| 2 Values ​​at the edge of the passband.

[0163] Sensor data smoothing filtering uses three-point mean smoothing filtering to eliminate noise signals within the sensor. Let the sensor signal sequence be x = [x1, x2, ..., x...]. i ,...,x n ] T , where n represents the number of data points in the sensor signal sequence; the formulas for mean smoothing are shown in formulas (8) and (9):

[0164]

[0165]

[0166] in, This represents the value after one round of mean smoothing. This indicates the value after two mean smoothing operations; two mean smoothing operations can eliminate noise in the signal, which is beneficial for subsequent behavior recognition and analysis.

[0167] Step S33: Normalization / Standardization of Human Behavioral Data

[0168] Data with different dimensions and value ranges are transformed to the same value range to avoid adverse effects on calculations caused by different dimensions and value ranges. Min-Max normalization is used for normalization, transforming the values ​​of all similar sensors to the range [0, 1]. Min-Max normalization scales all data values ​​to the range [0, 1] through a linear transformation of the original data. Let the input sequence be x = [x1, x2, ..., x...]. i ,...,x n ] T The output sequence after Min-Max normalization is y = [y1, y2, ..., y]. i ,...,y n ] T As shown in formula (10):

[0169]

[0170] Where, x max and x min Let x be the maximum and minimum values ​​of all samples in the input sequence x, respectively.

[0171] Z-Score standardization is used to standardize the data, transforming it into data with a mean of 0 and a standard deviation of 1; this makes previously incomparable values ​​comparable. Z-Score standardization transforms data into data with a mean of 0 and a standard deviation of 1, as shown in formula (11):

[0172]

[0173] Where μ is the mean of all samples in the input sequence x, and σ is the standard deviation of all samples in the input sequence x;

[0174] Step S34: Sliding window splitting and labeling

[0175] Sliding window segmentation uses a fixed-length window to divide continuous sensor data into fixed-length data segments for input into the recognition model. The sensor acquisition frequency is set to H = 20Hz. Based on human behavior characteristics, a window of length W = 80 is selected, meaning each window contains W / H = 4 seconds of sensor data. The sliding step size S can be set to 40, resulting in S / W = 50% data coverage. This divides the data into W rows and M columns, where M is the number of columns after merging all sensor data. When using a triaxial accelerometer, triaxial angular velocity sensor, and triaxial magnetometer, a data segment is 80 rows and 9 columns. Each data segment is assigned a corresponding label, representing the behavior category name, and one-hot encoding is used to encode the label.

[0176] Step S4: Constructing the recognition model and classifying closed sets

[0177] The behavior data fragments preprocessed in step S3 are input into the recognition model in batches, and the recognition output of the closed set is obtained after training.

[0178] The closed set output module includes a Flatten layer, a Dense fully connected layer, a Drop pruning layer, a LayerNorm normalization layer, two Dense fully connected layers, and a Softmax output layer connected in sequence.

[0179] like Figure 3 As shown, the steps are as follows:

[0180] Step S41: Multi-resolution fusion convolution uses upsampling and downsampling to generate data feature maps of different resolutions, specifically:

[0181] Each 80×1 row data segment is transformed into a 160×1 row data segment, and upsampling is performed using direct copying.

[0182] Each row of data fragments with a length of 80 is transformed into row data fragments with a length of 40 and 20 respectively, and the data lengths are reduced to 1 / 2 and 1 / 4 of the original data lengths respectively. Downsampling is performed using convolution with a kernel size of 3×1 and strides of 2 and 4 respectively.

[0183] In addition, feature data with the same resolution as the original data is generated by convolution calculation with a kernel size of 3×1 and a stride of 1.

[0184] Further feature extraction at different resolutions is performed by using upsampling and downsampling methods to fuse convolutional calculations at different resolutions for the sensor's single-axis data. The specific implementation process is as follows:

[0185] Step S411: The high-resolution data with a length of 160, i.e., the behavioral data segment with a length of 160×1, is processed by convolution with a kernel size of 3×1 and a stride of 1 to generate high-resolution data with the same length of 160; the high-resolution data with a length of 160 is downsampled by convolution with a kernel size of 3×1 and a stride of 2 to generate the original resolution data with a length of 80.

[0186] Step S412: The original resolution data with a length of 80 is upsampled by direct copying to generate high-resolution data with a length of 160; the original resolution data with a length of 80 is calculated by convolution with a kernel size of 3×1 and a stride of 1 to generate original resolution data with the same length of 80; the original resolution data with a length of 80 is downsampled by convolution with a kernel size of 3×1 and strides of 2 and 4 respectively to generate medium-resolution data with a length of 40 and low-resolution data with a length of 20.

[0187] Step S413: The medium-resolution data of length 40 is upsampled by direct copying to generate the original resolution data of length 80; the medium-resolution data of length 40 is calculated by convolution with kernel size 3×1 and stride 1 to generate medium-resolution data of the same length 40; the medium-resolution data of length 40 is downsampled by convolution with kernel size 3×1 and stride 2 to generate low-resolution data of length 20.

[0188] Step S414: The low-resolution data of length 20 is upsampled by direct copying to generate original resolution data and medium resolution data of length 80 and 40, respectively; the low-resolution data of length 20 is generated by convolution with a kernel size of 3×1 and a stride of 1 to generate low-resolution data of the same length 20.

[0189] Step S42: Multi-resolution attention parallel convolution is used to extract deep features from sensor data at different resolutions, specifically:

[0190] The X, Y, Z axis data of the accelerometer, the X, Y, Z axis data of the angular velocity sensor, and the X, Y, Z axis data of the magnetometer, which are obtained after the fusion convolution calculation of different resolutions in step S41, are used to generate four types of resolution data. These data are then stitched together according to the principle of the same resolution, so that the data of different resolutions all include the 9-axis sensor data. This results in four types of feature data with different resolutions: high-resolution feature data of size 160×9, original resolution feature data of size 80×9, medium-resolution feature data of size 40×9, and low-resolution feature data of size 20×9.

[0191] Key features are extracted in parallel using a self-attention mechanism on feature data of four different resolutions. The calculation process is shown in formula (12):

[0192]

[0193] Where, d kQ is the key dimension, K and V are the keys and values ​​represented by the matrix, respectively; the more important and classification-friendly features A(Q,K,V) are automatically extracted from the single-axis feature data of the sensor through a self-attention mechanism.

[0194] Subsequently, the feature data at different resolutions were subjected to convolution calculations in parallel with a kernel size of 3×3 and a stride of 1, so as to fully extract the features between the multi-axis sensor data at each resolution.

[0195] Step S43: Fusion attention convolution is a deep fusion of feature data at different resolutions, specifically referring to:

[0196] By upsampling and downsampling, the feature resolution between the multi-axis sensor data obtained in step S42 is converted into feature data of size 80×9, and then spliced ​​into feature data of size 80×36.

[0197] The self-attention mechanism is used to extract important features from multi-axis feature data of different sensors. Then, convolution calculation with a kernel size of 3×3 and a stride of 1 is used to fully extract and fuse the features between multi-axis sensor data of different resolutions.

[0198] Step S44: Classification and recognition involves outputting the classification and recognition results for the closed set and the feature vectors for the open set. Specifically, this means:

[0199] The closed set output module comprises a Flatten layer, a Dense fully connected layer, a Drop pruning layer, a LayerNorm normalization layer, two Dense fully connected layers, and a Softmax output layer, connected sequentially. The Flatten layer flattens multidimensional data into one dimension, serving as a transition between convolutional layers and Dense fully connected layers. The Dense fully connected layer transforms and maps the extracted feature data, ultimately used for classification and recognition. The Drop pruning layer randomly deactivates some neurons during training to prevent overfitting. The LayerNorm normalization layer transforms the input data into data with a mean of 0 and a variance of 1, ensuring the stability of the data feature distribution and accelerating model convergence. The Softmax output layer calculates the probability of the behavior corresponding to the feature data, with the highest probability being the closed set behavior recognition result. The number of neurons in the three Dense layers is set to 64, 16, and the total number of known behavior categories T in the closed set, respectively.

[0200] Step S5: Loss Function Calculation, Update, and Feedback

[0201] In the case of open set, the feature vector output by the first Dense fully connected layer of the two Dense fully connected layers in the closed set output module in step S4 is processed according to the principle of the same category. The mean and variance of the feature vector center that belongs to the same category and is finally correctly classified are calculated, as shown in formula (1) and formula (2) respectively. In addition, the variance is normalized, as shown in formula (3).

[0202]

[0203]

[0204]

[0205] Where, μ i Let σ represent the center mean vector of category i. i ) 2 Nor((σ) represents the variance of category i. i ) 2 ) represents the variance of class i after normalization, |C i | represents the number of correctly classified feature vectors in category i. Let represent the feature vector output by the second-to-last fully connected layer for the j-th data in category i, and T be the total number of all known behavior categories;

[0206] The loss function is set as formula (4):

[0207]

[0208] Where max(Δμ) represents the maximum distance between the pairwise mean values ​​of the class centers, calculated using the 1-norm;

[0209] In each round of training, the known category data is first input into the recognition model in batches according to the data volume B of each batch for parameter updates. B is set to any positive integer less than the total amount of data, for example, B = 64. At this time, the cross-entropy loss function and softmax function commonly used in closed set training are used for training and output. After one round of training is completed, only the loss function L in formula (4) is used to return to step S4 for training until convergence, so as to further adjust the recognition model parameters and make the data of the same type closer in the multidimensional space and the data of the opposite type farther apart; thereby achieving the purpose of reducing the feature space of the known category, which is more conducive to the discrimination of the unknown category.

[0210] Step S6: Determine the threshold and output the open set discrimination.

[0211] After the recognition model is trained, the distance between all correctly classified feature data points in each category and the central mean point is calculated, and normalization is performed as shown in formulas (5) and (6):

[0212]

[0213]

[0214] Where, Δz i This represents the average distance between the correctly classified feature vectors in the i-th category and the median mean. represents the normalized distance from the median mean point to the correctly classified j-th feature vector in the i-th category, with a value range of [0,1].

[0215] Choose from Z i For thresholds ∈ (0,1), calculate the proportion of feature data points included in each category under different thresholds, i.e. The percentage of feature data points within the range, and the threshold Z when the percentage exceeds 90%. i The threshold for classifying an open set is denoted as i, where i represents different behavior categories, i = 1, 2, 3, ..., T, and T is the total number of known behavior categories.

[0216] When new data is input into the recognition model, the normalized distance between the feature vector of its penultimate fully connected layer and the mean of the centers of different known classes is calculated. when Less than or equal to the threshold Z of a certain category i If the value is greater than the threshold Z for each category, it is immediately classified as belonging to that category. i It is then classified as an unknown category;

[0217] Step S7: Determine the application of the output result

[0218] The results output from both closed and open sets are transmitted to the corresponding application platforms in real time. These results are then stored, processed, calculated, and displayed in real time. This technology can be used in various application platforms, such as those for managing special personnel, monitoring and managing the elderly, and monitoring and managing rehabilitation and nursing care.

Claims

1. An open set human behavior recognition system based on multi-resolution fusion convolution, characterized in that, The human behavior information collection module, the human behavior information transmission module, the human behavior information preprocessing module, the human behavior classification identification module, the open set classification identification module and the human behavior information application module are sequentially connected. The human behavior information collection module is configured to collect behavior data of a user in real time, wherein the behavior data includes X, Y and Z three-axis data of an acceleration sensor, X, Y and Z three-axis data of an angular velocity sensor and X, Y and Z three-axis data of a magnetometer. The human behavior information transmission module is configured to transmit the collected behavior data of the user to a local server or a cloud server. The human behavior information preprocessing module is configured to sequentially store, merge, filter, augment, normalize or standardize, sliding window segment and label the collected behavior data of the user, so as to obtain smooth, diverse, dimensionless and labeled behavior data segments. The human behavior classification identification module is configured to input the behavior data of the user preprocessed by the human behavior information preprocessing module into the trained human behavior classification identification module to determine the behavior type. The open set classification identification module is configured to accurately determine the behavior data input into the identification model in an open set. The human behavior information application module is configured to transmit the obtained human behavior classification identification result to various application platforms, so as to realize corresponding functions. The open set classification identification module includes a loss function calculation update module and an open set output module connected in sequence. The loss function calculation update module is configured to calculate the mean and variance of the known class behavior data in the training process, so as to generate a loss function to update the parameters of the identification model. The loss function calculation update module includes a same data mean and variance calculation unit and a loss function construction unit connected in sequence. In the same data mean and variance calculation unit, the vectors in the penultimate layer of the full connection layer are arranged according to the same class, the mean and variance of each class are calculated respectively, and the loss function is constructed by the mean and variance. In the training process, the data is trained in a closed set manner according to the data amount of each batch B, B is set to any positive integer less than the total data amount, the parameter of the identification model is adjusted using the constructed loss function after the training of each round is completed, and the process is repeated until convergence. The open set output module is configured to calculate the determination threshold of the known class and the unknown class and output the determined class, and includes a class threshold determination unit and an open set determination output unit connected in sequence. In the category threshold determination unit, the distance between all feature data points classified correctly in each category and the center mean point is calculated, and a normalization operation is performed; a threshold Z i ∈(0,1) is selected, the proportion of feature data points contained in each category under different thresholds is calculated, when the proportion exceeds 90%, the threshold Z i is determined as the open set classification threshold value, where i represents different behavior categories, i = 1, 2, 3,..., T, T is the total number of all known behavior categories; when new data is input into the recognition model, the normalized distance between the feature vector of the second-to-last layer of the full connection layer and the center mean value of different known categories is calculated, when the distance is less than the threshold value Z i of a certain category, it is determined as this category, if they are all greater than the threshold value Z i of each category, it is determined as an unknown category; the open set discrimination output unit outputs according to the above discrimination result.

2. The open-set human behavior recognition system based on multi-resolution fusion convolution of claim 1, wherein, The human behavior classification identification module includes a multi-resolution fusion convolution module, a multi-resolution attention parallel convolution module, a fusion attention convolution module and a closed set output module connected in sequence. The multi-resolution fusion convolution module is used for sampling the preprocessed user behavior data to generate data of different resolutions; the multi-resolution fusion convolution module comprises a data up-sampling unit, a data down-sampling unit and a first data multi-resolution fusion convolution unit connected in sequence; the data length of the preprocessed user behavior data is doubled by the data up-sampling unit, and the up-sampling is performed by using a direct copying method; The data length of the preprocessed user behavior data is respectively changed to 1 / 2 and 1 / 4 of the original data length by the data down-sampling unit, and the down-sampling is performed by using a convolution kernel with a size of 3x1 and a step of 2 and 4; in addition, the preprocessed user behavior data is processed by a convolution kernel with a size of 3x1 and a step of 1 to generate feature data of the same resolution as the preprocessed user behavior data; the first data multi-resolution fusion convolution unit further performs fusion convolution calculation of different resolutions by up-sampling and down-sampling methods, and further extracts features of the sensor data under different resolutions; The multi-resolution attention parallel convolution module is used for deep feature extraction of the sensor data of different resolutions; the multi-resolution attention parallel convolution module comprises a same-resolution data splicing unit, a parallel self-attention unit and a parallel convolution unit connected in sequence; the same-resolution data splicing unit splices four kinds of resolution data generated by the X, Y and Z three-axis data of the acceleration sensor, the X, Y and Z three-axis data of the angular velocity sensor and the X, Y and Z three-axis data of the magnetometer, that is, the data of nine-axis sensors, according to the same resolution principle, so that the data of different resolutions all include nine-axis sensor data; the parallel self-attention unit automatically weights important features in the data of different resolutions to give more attention to important features; and the parallel convolution unit further extracts deep features of the data processed by the self-attention unit; The fusion attention convolution module is used for deep fusion of the feature data of different resolutions; the fusion attention convolution module comprises a second data multi-resolution fusion convolution unit, a fusion self-attention unit and a fusion convolution unit connected in sequence; the second data multi-resolution fusion convolution unit is used for fusing the parallel feature data of different resolutions to become feature data with a length of the original sensor data window size; the fusion self-attention unit is used for automatically extracting important features in the fused deep feature data; and the fusion convolution unit further extracts deep features of the data; The closed set output module is used for outputting the closed set classification recognition result and the open set feature vector; the closed set output module comprises a full connection unit and a closed set classification and discrimination output unit connected in sequence; the full connection unit is a multi-layer full connection layer for performing feature space transformation on the data; The closed set classification and discrimination output unit calculates and outputs the recognition result by a Softmax layer.

3. The open-set human behavior recognition system based on multi-resolution fusion convolution of claim 1, wherein, The human behavior information acquisition module includes a plurality of different sensor units integrated in a wearable device and respectively worn at corresponding positions of the user's body, and the sensor units include a three-axis acceleration sensor, a three-axis angular velocity sensor, and a three-axis magnetometer for acquiring the user's behavior data.

4. The open-set human behavior recognition system based on multi-resolution fusion convolution of claim 1, wherein, The acquired user's behavior data is transmitted to a local server or a cloud server through any one of ultra-wideband, Bluetooth, and 5G transmission modes.

5. The open-set human behavior recognition system based on multi-resolution fusion convolution of claim 1, wherein, The human behavior information preprocessing module includes a sensor data storage unit, a sensor data merging unit, an acceleration data gravity filtering unit, a sensor data smoothing filtering unit, a data augmentation unit, a data normalization / unitization unit, a sliding window segmentation unit, and a data label calibration unit connected in sequence. The sensor data storage unit is configured to store real-time user's behavior data. The sensor data merging unit is configured to splice and merge behavior data of different sensor units, and the merged data is in a two-dimensional array format, with X, Y, and Z axis data of the three-axis acceleration sensor, X, Y, and Z axis data of the three-axis angular velocity sensor, and X, Y, and Z axis data of the three-axis magnetometer arranged in sequence in the horizontal direction, and arranged in time sequence in the vertical direction. The acceleration data gravity filtering unit is configured to filter out inherent gravity acceleration to obtain actual behavior acceleration; specifically, a Butterworth filter is used to eliminate gravity acceleration to obtain the true acceleration in the human motion process. The sensor data smoothing filtering unit is configured to remove noise in the original sensor data. The data augmentation unit is configured to expand the data set. The data normalization / unitization unit is configured to convert data with different dimensions and value ranges to the same value range. The sliding window segmentation unit is configured to segment continuous long-time sensor data into a plurality of data segments. The data label calibration unit is configured to label the data after sliding window segmentation and one-hot encode the labels.

6. An open-set human behavior recognition method based on multi-resolution fusion convolution, implemented by the open-set human behavior recognition system based on multi-resolution fusion convolution of any one of claims 1-5, characterized in that, The method includes the following steps: Step S1: human behavior data acquisition Real-time acquisition of user's behavior data, i.e., human behavior data, including X, Y, and Z three-axis data of the acceleration sensor, X, Y, and Z three-axis data of the angular velocity sensor, and X, Y, and Z three-axis data of the magnetometer; Step S2: human behavior data transmission Transmit the acquired user's behavior data to a local server or a cloud server; Step S3: human behavior data preprocessing Sequentially store, merge, filter, augment, normalize / unitize, sliding window segment, and label the acquired user's behavior data to obtain smooth, diverse, dimension- independent, and labeled behavior data segments; Step S4: constructing an identification model and closed set classification identification Batch the behavior data segments after step S3 preprocessing into the identification model, and obtain the closed set identification output after training. The closed set output module comprises, in sequence, a Flatten layer, a Dense fully connected layer, a Drop layer, a LayerNorm layer, two layers of Dense fully connected layers, and a Softmax output layer; Step S5: loss function calculation, updating and feedback In the open set case, the feature vectors output by the former one of the two layers of Dense fully connected layers in the closed set output module in step S4 are processed according to the same category principle, the center mean and variance of the feature vectors belonging to the same category and finally classified correctly are calculated, as shown in formulas (1) and (2) respectively, and the variance is normalized, as shown in formula (3): where μ i represents the center mean vector of class i, (σ i ) 2 represents the variance of class i, Nor((σ i ) 2 ) represents the normalized variance of class i, |C i | represents the number of correctly classified feature vectors in class i, represents the feature vector of the jth data in class i at the penultimate layer output of the fully connected layer, and T is the total number of known behavior classes. The loss function is set as formula (4): Wherein, max (△μ) represents the maximum distance between the center means of two categories, calculated using the 1-norm; In each round of training, the data of known categories are input into the recognition model in batches according to the data amount of each batch B for parameter updating, B is set to be any positive integer less than the total amount of data, at this time, the cross-entropy loss function and the Softmax function commonly used in closed set training are used for training and output, when a round of training is completed, only the loss function L in formula (4) is used to return to step S4 for training until convergence, so as to further adjust the recognition model parameters, and make the same category data closer in the multi-dimensional space, and the center mean distance of different category data farther; Step S6: determining threshold and open set discrimination output After the training of the recognition model is completed, the distance between all feature data points and the center mean point in each category is calculated, and a normalization operation is performed, as shown in formulas (5) and (6): wherein, Δz i represents the average distance of the correctly classified feature vectors in the i-th class to the center mean point, represents the normalized distance of the j-th correctly classified feature vector in the i-th class to the center mean point, with a value range of [0, 1]; Selecting threshold value Z i ∈(0,1) to calculate the proportion of feature data points contained in each category under different thresholds, i.e. The proportion of feature data points in the range, when the proportion exceeds 90%, the threshold value Z i , that is, it is determined as an open set classification threshold value, where i represents different behavior categories, i = 1, 2, 3,..., T, and T is the total number of all known behavior categories. When new data is input to the recognition model, the normalized distance between the feature vector of the penultimate layer of its fully connected layer and the mean of the different known class centers is calculated When The threshold value Z of a certain class i is determined as the class, if they are all greater than the threshold value Z of each class i then it is determined as an unknown class; Step S7: application of discrimination output result The results of the closed set output and the results of the open set output are transmitted to the corresponding application platform in real time.

7. The open-set human behavior recognition method based on multi-resolution fusion convolution of claim 6, wherein, In step S3, the specific implementation process is as follows: Step S31: merging of human behavior data The collected user behavior data is spliced and merged, and the merged data is in a two-dimensional array format, arranged in the order of X, Y and Z axis data of the three-axis acceleration sensor, X, Y and Z axis data of the three-axis angular velocity sensor, and X, Y and Z axis data of the three-axis magnetometer in the horizontal direction, and arranged in the time sequence in the vertical direction; Step S32: human behavior data filtering Human behavior data filtering includes acceleration data gravity filtering and sensor data smoothing filtering; The acceleration data gravity filtering uses a 3-order Butterworth filter with a cutoff frequency of 0.3 Hz to eliminate the gravity acceleration and extract pure human behavior signals, and the 3-order Butterworth filter is represented by formula (7): where w c is the cutoff frequency, w p is the passband edge frequency, w is the angular frequency, is |H(w)| 2 the value at the passband edge. The sensor data smoothing filter uses a three-point mean value smoothing filter to eliminate the noise signal inside the sensor. Let the sensor signal sequence be x = [x1, x2,..., xn], where n represents the number of data points in the sensor signal sequence. The mean value smoothing formula is shown in Equations (8) and (9): i ,...,x n ] T , n represents the number of data points in the sensor signal sequence. The mean value smoothing formula is shown in Equations (8) and (9): wherein, denotes the value after one mean smoothing, denotes the value after two mean smoothings; Step S33: human behavior data normalization / standardization The Min-Max normalization method is used for normalization, that is, the values of all sensors of the same type are transformed to [0, 1]; the Min-Max normalization method scales all data values to [0, 1] through linear transformation of the original data; assuming that the input sequence is x = [x1, x2,..., x i ,...,x n ] T , and the output sequence after Min-Max normalization is y = [y1, y2,..., y i ,...,y n ] T , as shown in formula (10): where x max and x min are the maximum and minimum values of all samples in the input sequence x, respectively. The Z-Score standardization method is used for standardization processing, that is, the data is changed into data with a mean of 0 and a standard deviation of 1; the Z-Score standardization method changes the data into data with a mean of 0 and a standard deviation of 1, as shown in formula (11): Wherein, μ is the mean of all samples in the input sequence x, and σ is the standard deviation of all samples in the input sequence x. Step S34: sliding window segmentation and labeling Sliding window segmentation is to use a fixed length window to segment continuous sensor data into fixed length data segments, and to assign a corresponding label to each data segment, i.e., the behavior category name corresponding to the data segment, and to encode the label using one-hot encoding.

8. The open-set human behavior recognition method based on multi-resolution fusion convolution of claim 6, wherein, Step S4 includes the following steps: Step S41: multi-resolution fusion convolution is to use upsampling and downsampling to generate data feature maps of different resolutions, specifically: Each behavior data segment with a size of 80x1 is changed to a behavior data segment with a length of 160x1, and upsampling is performed using a direct copy method; Each behavior data segment with a length of 80 is changed to a behavior data segment with a length of 40 and 20, and the data length is changed to 1 / 2 and 1 / 4 of the original data length, respectively, and downsampling is performed using a convolution kernel size of 3x1 and a step size of 2 and 4, respectively; In addition, a feature data with the same resolution as the original data is generated through convolution kernel size of 3x1 and step size of 1; Again, through upsampling and downsampling methods, multi-resolution fusion convolution calculation is performed to further extract features of single-axis sensor data at different resolutions; Step S42: multi-resolution attention parallel convolution is to extract deep features of sensor data at different resolutions, specifically: The 4 kinds of resolution data generated by the 9-axis sensor data of the X, Y, Z three-axis data of the acceleration sensor, the X, Y, Z three-axis data of the angular velocity sensor, and the X, Y, Z three-axis data of the magnetometer after the different resolution fusion convolution calculation in step S41 are spliced according to the same resolution principle, so that the different resolution data all include 9-axis sensor data, and four kinds of feature data with different resolutions are obtained, i.e., high-resolution feature data with a size of 160x9, original-resolution feature data with a size of 80x9, medium-resolution feature data with a size of 40x9, and low-resolution feature data with a size of 20x9; The four kinds of feature data with different resolutions are used in parallel to extract key features using a self-attention mechanism, and the calculation process is shown in formula (12): Wherein, d k is the dimension of the key, Q is the query matrix, K and V are the matrix representation of the key and value respectively; A(Q, K, V) is automatically extracted from the sensor single-axis feature data by the self-attention mechanism, which is more important and more conducive to classification. Subsequently, the feature data at different resolutions is parallelly subjected to convolution kernel size of 3x3 and step size of 1, to fully extract the features between multi-axis sensor data at different resolutions; Step S43: fusion attention convolution is to deeply fuse the feature data at different resolutions, specifically: Through upsampling and downsampling, the features between multi-axis sensor data obtained in step S42 are converted to feature data with a size of 80x9, and spliced into feature data with a size of 80x36; The important features of different sensor multi-axis feature data are extracted using a self-attention mechanism, and then the convolution kernel size of 3x3 and the step size of 1 are calculated to fully extract and fuse the features between multi-axis sensor data at different resolutions; Step S44: classification recognition is to output the closed set classification recognition result and output the open set feature vector, specifically: The closed set output module comprises a Flatten layer, a Dense fully connected layer, a Drop pruning layer, a LayerNorm layer normalization, two layers of Dense fully connected layers and a Softmax output layer connected in sequence.

9. The open-set human behavior recognition method based on multi-resolution fusion convolution according to any one of claims 6-8, characterized in that, In step S41, the different resolution fusion convolution calculations are performed again by the upsampling and downsampling methods, and the feature extraction of the sensor single-axis data is further performed at different resolutions, and the specific implementation process is as follows: In step S411, the high-resolution data with a length of 160, i.e., the behavior data segment with a length of 160x1, is calculated by a convolution kernel with a size of 3x1 and a step of 1 to generate high-resolution data with a length of 160; the high-resolution data with a length of 160 is calculated by a convolution kernel with a size of 3x1 and a step of 2 to generate original resolution data with a length of 80; In step S412, the original resolution data with a length of 80 is upsampled by a direct copying method to generate high-resolution data with a length of 160; the original resolution data with a length of 80 is calculated by a convolution kernel with a size of 3x1 and a step of 1 to generate original resolution data with a length of 80; the original resolution data with a length of 80 is calculated by a convolution kernel with a size of 3x1 and a step of 2 and 4 to generate medium-resolution data with a length of 40 and low-resolution data with a length of 20; In step S413, the medium-resolution data with a length of 40 is upsampled by a direct copying method to generate original resolution data with a length of 80; the medium-resolution data with a length of 40 is calculated by a convolution kernel with a size of 3x1 and a step of 1 to generate medium-resolution data with a length of 40; the medium-resolution data with a length of 40 is calculated by a convolution kernel with a size of 3x1 and a step of 2 to generate low-resolution data with a length of 20; In step S414, the low-resolution data with a length of 20 is upsampled by a direct copying method to generate original resolution data with a length of 80 and medium-resolution data with a length of 40; the low-resolution data with a length of 20 is calculated by a convolution kernel with a size of 3x1 and a step of 1 to generate low-resolution data with a length of 20.

Images