A behavior recognition model training method based on time-frequency fusion enhancement

Abstract

The application provides a behavior recognition model training method based on time-frequency fusion enhancement, comprising: A1, obtaining a training set, wherein original samples are sensing data, and labels indicate the behavior categories of human bodies when the corresponding sensing data is collected; A2, dividing the training set into multiple batches, and iteratively training a feature extractor of a behavior recognition model in batches, wherein each batch training comprises: A21, respectively performing time domain enhancement and frequency domain enhancement on each original sample and then fusing, A22, based on each original sample and the corresponding time-frequency enhanced sample, training the feature extractor to extract sample features according to the input sample through a contrast learning manner, and narrowing the distance between the sample features of the original sample and the corresponding time-frequency enhanced sample and widening the distance between the sample features of the original sample and other samples; A3, obtaining a behavior recognition model comprising the feature extractor trained in step A2 and a classifier, and performing classification training on the behavior recognition model by using the training set.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, in particular to the field of neural networks, and more particularly to a behavior recognition model training method based on time-frequency fusion enhancement. BACKGROUND

[0002] With the increasing popularity of wearable devices such as mobile phones, smart watches, smart bands, and fitness trackers, people are increasingly paying attention to human activity recognition (HAR) and its related application fields such as medical care, assisted living, health monitoring, and smart buildings. Among them, human activity recognition based on wearable devices is an important research direction in the field of ubiquitous computing, aiming to recognize human behavior activities by analyzing the sensor data collected by wearable sensor devices.

[0003] In the past decade, deep learning models have shown superior performance in various applications compared to traditional machine learning methods. The performance improvement of deep learning relies on large-scale labeled data, but obtaining a large amount of labeled data faces significant challenges, requiring a large amount of manpower and economic cost. In addition, the labeling results are easily affected by human subjective factors, leading to labeling errors. To reduce labeling costs, various methods such as semi-supervised learning, weakly supervised learning, and transfer learning have been proposed. By using these technologies, researchers can improve the accuracy and generality of deep learning models while reducing the cost and time associated with manual labeling. However, existing user behavior recognition methods based on supervised learning, semi-supervised learning, weakly supervised learning, and transfer learning face serious challenges, and the model effect is heavily dependent on the quality and scale of the labeled data, hindering the construction of high-precision user behavior recognition models.

[0004] Although some researchers have made efforts to minimize the need for label annotation using various learning paradigms, due to the complexity of accurately annotating raw sensor signals, wearable device-based tasks still face ongoing challenges. Unlike other forms of data such as audio and images, raw sensor signals have poor self-explanatory nature, making manual annotation difficult; resulting in high cost for obtaining a large number of manually annotated samples in this field. The way of directly training and applying the model with a small number of samples may lead to difficulty in ensuring the recognition accuracy of the model. Therefore, how to ensure or improve the accuracy of the trained model in identifying human behavior activities under the condition of limited samples is a problem that needs to be solved. SUMMARY

[0005] Therefore, the purpose of the present application is to overcome the defects of the prior art and provide a behavior recognition model training method based on time-frequency fusion enhancement.

[0006] The purpose of the present application is achieved by the following technical solutions:

[0007] According to a first aspect of the present application, a time-frequency fusion enhanced behavior recognition model training method is provided for identifying the scene of human behavior according to the sensor data collected by the wearable device, comprising: A1, obtaining a training set, which includes a plurality of original samples and a label corresponding to each original sample, the original sample is single or multi-channel sensor data, and the label indicates the behavior category of the human body when the corresponding sensor data is collected; A2, dividing the training set into multiple batches, and iteratively training the feature extractor of the behavior recognition model in batches, wherein the labels corresponding to different original samples in the same batch are different from each other, and each batch is trained based on the original samples contained in the current batch to perform the following self-supervised contrast learning process: A21, performing time domain enhancement and frequency domain enhancement on each original sample respectively, and then fusing to obtain a time-frequency enhanced sample, A22, based on each original sample and the corresponding time-frequency enhanced sample, the feature extractor is trained to extract sample features with behavior recognition information according to the input sample by contrast learning, wherein the goal of contrast learning is to narrow the distance between the sample features of the original sample and its corresponding time-frequency enhanced sample, and to widen the distance between the sample features of the original sample and other samples; A3, obtaining a behavior recognition model including a feature extractor trained in step A2 and a classifier, and performing classification training on the behavior recognition model using the training set to obtain a trained behavior recognition model.

[0008] Optionally, step A21 includes: randomly selecting a corresponding mode from a plurality of preset time domain enhancement modes to enhance each original sample to obtain a time domain enhanced sample; converting each original sample from time domain to frequency domain, enhancing in frequency domain, and converting back to time domain to obtain a frequency domain enhanced sample; and fusing the time domain enhanced sample and the frequency domain enhanced sample corresponding to each original sample to obtain a time-frequency enhanced sample corresponding to the original sample.

[0009] Optionally, the plurality of preset time domain enhancement modes include jitter processing, scaling processing and rearrangement processing, wherein the jitter processing refers to adding random Gaussian noise to the original sample, the scaling processing refers to multiplying the original signal by a scaling factor sampled from a normal distribution, and the rearrangement processing refers to randomly rearranging the original signal after dividing it into multiple segments.

[0010] Optionally, the frequency domain enhanced sample is obtained by first applying fast Fourier transform to the original sample to obtain a frequency domain signal, and then using the FNet network to enhance the frequency domain signal in the frequency domain and convert it back to the time domain.

[0011] Optionally, the time-frequency enhanced sample is obtained by multiplying the signal data corresponding to the time domain enhanced sample and the signal data corresponding to the frequency domain enhanced sample.

[0012] Optionally, the step A22 comprises: in all samples in a batch, forming a positive example pair by each original sample and its corresponding time-frequency enhanced sample, and forming a negative example pair by each original sample and each other sample excluding the original sample and its corresponding time-frequency enhanced sample, wherein the all samples comprise each original sample and its time-frequency enhanced sample in the batch; inputting all samples in a batch into the feature extractor respectively to obtain sample features of each sample with behavior recognition information, and updating the trainable parameters of the feature extractor through contrast learning of the sample features of two samples in the positive example pair and the sample features of two samples in each negative example pair.

[0013] Optionally, the contrast learning adopts a gradient of a loss calculated by a normalized temperature scaling contrastive loss function (NT-Xent) to update the trainable parameters of the feature extractor by back propagation.

[0014] Optionally, the label space corresponding to the label is walking, running, jumping, sitting, lying, standing, cycling, cross-country walking, watching TV, working on a computer, driving, going upstairs, going downstairs, cleaning with a vacuum cleaner, ironing, folding clothes, cleaning a house, playing football, skipping, sitting to standing, lying to standing, standing to lying, sitting down, leaning forward, leaning left, leaning right, leaning backward, leaning on a chair, leaning with protection, falling and hitting an obstacle, fainting, and other behaviors, or a combination thereof.

[0015] According to a second aspect of the present application, a user behavior recognition method is provided, comprising: obtaining sensor data of a user; and inputting the sensor data of the user into a trained behavior recognition model obtained by the method of the first aspect to recognize the behavior of the user.

[0016] According to a third aspect of the present application, an electronic device is provided, comprising: one or more processors; and a memory, wherein the memory is configured to store executable instructions; and the one or more processors are configured to implement the steps of the method of the first aspect and / or the second aspect by executing the executable instructions.

[0017] Compared with the prior art, the present application has the following advantages:

[0018] The embodiments of the present application perform a self-supervised contrast learning process on the feature extractor by using original samples and time-frequency enhanced samples, and add a classifier fine-tuning mode based on the feature extractor obtained by the self-supervised contrast learning, so as to improve the accuracy and robustness of the model for human behavior recognition in the case of a small amount of samples. BRIEF DESCRIPTION OF DRAWINGS

[0019] The embodiments of the present application will be further described below with reference to the accompanying drawings, in which:

[0020] Figure 1 A flowchart of a behavior recognition model training method based on time-frequency fusion enhancement according to an embodiment of the present application is shown in FIG. 1.

[0021] Figure 2 A block diagram of a behavior recognition model training method based on time-frequency fusion enhancement according to an embodiment of the present application is shown in FIG. 2. DETAILED DESCRIPTION

[0022] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application.

[0023] As mentioned in the background section, the self-explanatory nature of the raw sensor signals of wearable devices is poor, which brings difficulties to manual annotation; and the cost of obtaining a large number of manually annotated samples in this field is high. However, the direct training and application of the model using a small number of samples may result in difficulty in ensuring the recognition accuracy of the model. The existing user behavior recognition methods based on supervised learning, semi-supervised learning, weakly supervised learning, transfer learning and the like face severe challenges, which hinder the construction of high-precision user behavior recognition models. To this end, the embodiments of the present application use a self-supervised contrast learning process for the feature extractor (corresponding to step A2, which can use the sub-step of step A2 to construct time-frequency enhanced samples based on original samples, and improve the ability of the feature extractor to extract useful features for user behavior recognition based on the contrast learning of the original samples and the time-frequency enhanced samples) and a way of adding a classifier fine-tuning after the feature extractor obtained based on the self-supervised contrast learning (corresponding to step A3, which allows the behavior recognition model to inherit the parameters of the feature extractor obtained in step A2, and then fine-tune the training to optimize the classification accuracy of behavior classification in the corresponding behavior recognition scenario), so as to achieve the technical effect of improving the accuracy of the model for human behavior recognition in the case of a small number of samples.

[0024] In order to facilitate understanding of the technical solutions of the present application, the application scenarios of the present application will be described first. The present application is used for human activity behavior recognition (or human behavior recognition, user behavior recognition), and the behavior recognition problem can be regarded as a classification task using multivariate time series data (corresponding to sensor data). Illustratively, given a dataset wherein, is a multivariate time series with length T and sensor channel number S, is a label indicating the corresponding behavior category. The goal of human behavior recognition is to learn a mapping function (corresponding to a behavior recognition model): ; and in order to test the effect, a time series for testing is generally given , the prediction label output by the mapping function is able to match the real label y. After learning is completed, the behavior category in which a new time series (such as a newly collected time series of a user) is located can be identified according to the mapping function.

[0025] In order to illustrate the technical solutions of the present application, the following is described from four aspects of data set, model structure, training process and model application.

[0026] I. Data set

[0027] According to an embodiment of the present application, the data set can include a plurality of original signals and a label corresponding to each original signal, the label indicating the behavior category of the human body when the corresponding original signal is collected. The data set can be collected and made by the implementer himself according to the definition. Each original signal can be single-channel sensor data (such as collected by a single-axis motion sensor). Each original signal can be multi-channel sensor data (such as collected by a two-axis or three-axis motion sensor). The label space corresponding to the label (i.e. the set of behavior categories actually to be distinguished) is walking, running, jumping, sitting, lying, standing, cycling, cross-country walking, watching TV, working on a computer, driving, going upstairs, going downstairs, cleaning with a vacuum cleaner, ironing, folding clothes, cleaning the house, kicking a football, jumping rope, sitting to standing, lying to standing, standing to lying, sitting down, leaning forward, leaning left, leaning right, leaning back, leaning on a chair, leaning with protection, falling and hitting an obstacle, fainting and other behaviors or combinations thereof.

[0028] Alternatively, an existing data set can also be used. For example, existing public data sets: USC-HAD data set, PAMAP2 data set, UniMib-SHAR data set, WISDM data set, etc. Among them:

[0029] USC-HAD data set: The USC-HAD data set was created by researchers at the University of Southern California and contains accelerometer and gyroscope data collected from nine different locations on the body, including the wrist, waist, chest, and ankle. The data set includes data activities from 12 different parts, including walking, jogging, jumping, and sitting, and is collected from 14 different subjects. Each activity segment in the data set is labeled with the corresponding activity type and sensor data sampled at 50Hz.

[0030] PAMAP2 Dataset: The PAMAP2 dataset was collected from nine volunteers. The volunteers performed 12 different activities, including walking, cycling, jumping rope, etc. Multiple sensors, including chest sensors, wrist sensors, and ankle sensors, were applied to record the data. The 100 Hz sampling rate was down-sampled to 33.3 Hz for further analysis. The exercise intensity was estimated using a heart rate monitor with a sampling rate of 9 Hz.

[0031] UniMib-SHAR Dataset: The UniMib-SHAR dataset was collected by scholars of the University of Milan-Bicocca, aiming to identify various "falling" activities. Data was collected from 30 participants aged from 18 to 60 using an Android smartphone. During data collection, all participants had to place the smartphone in their left and right pockets. The sensor signals were sampled at 50 Hz.

[0032] WISDM Dataset: The WISDM dataset was collected from 36 different users, including over 1,000,000 sensor data samples, including six different activities, including walking, jogging, climbing stairs, descending stairs, sitting, and standing. The sensor signals were sampled at a frequency of 20 Hz.

[0033] Based on the above datasets, the original signals can be directly taken as original samples, and a subset of the above datasets can be extracted as a training set. Generally, the above datasets are divided into a training set and a test set in a predetermined ratio, such as an 8:2 ratio, or divided into a training set, a validation set, and a test set in a 7:2:1 ratio. Of course, other predetermined ratio division methods can also be customized by the implementer.

[0034] In addition, the original signals of the above datasets can also be segmented into the length required by the implementer, and then the training set is extracted in the aforementioned manner. According to one embodiment of the present application, the behavior recognition model training method based on time-frequency fusion enhancement of the present application includes: obtaining a first dataset (corresponding to the aforementioned dataset), which includes a plurality of original signals and a label corresponding to each original signal, the label indicating the behavior category of the human body when the corresponding original signal is collected; segmenting each original signal in the dataset through a sliding window to obtain a plurality of sensor data of a predetermined length as a plurality of original samples, each original sample inherits the label of the original signal to which it belongs, forming a second dataset; extracting a subset from the second dataset to form a training set. For example, see Figure 1 The segmentation of the original signal assumes that a segment of the original signal includes , , …, , etc. Each sub-signal can be segmented by , , …, The embodiment can adjust the length of the original signal in the data set to the length required by the implementer, and can also increase the number of available samples, simply and efficiently improve the sample quantity, and improve the effect of model training.

[0035] II. Model structure

[0036] According to an embodiment of the present application, the behavior recognition model comprises a feature extractor and a classifier, the feature extractor is used to extract sample features with behavior recognition information (or extract sample features capable of distinguishing human behavior categories) according to input sensor data (i.e. samples), and the classifier is used to recognize behavior categories according to sample features.

[0037] The feature extractor can use an existing convolutional neural network feature extractor, or use an existing convolutional neural network or deep learning model feature extractor (corresponding to Figure 1 The encoder Encoder shown in the figure, for example, can use a three-layer CNN, or other more complex deep learning models, such as ResNet or VGG, etc.) and add a multi-layer perceptron (corresponding to Figure 1 The multi-layer perceptron Projection MLP shown in the figure) as the feature extractor of the embodiment of the present application. The existing convolutional neural network feature extractor or deep learning model is used to extract intermediate sample features, and the multi-layer perceptron is used to process and adjust the dimensions of the intermediate sample features to obtain sample features. That is, the multi-layer perceptron realizes the functions of further feature extraction and dimension adjustment, and reaches the dimensions required by the implementer for corresponding behavior recognition to obtain the corresponding sample features.

[0038] The classifier can be implemented by sequentially connecting multiple fully connected layers and a Softmax function.

[0039] III. Training process

[0040] After the above training set and behavior recognition model are prepared, the behavior recognition model needs to be trained to obtain a behavior recognition model capable of predicting behavior categories, i.e. a trained behavior recognition model.

[0041] According to an embodiment of the present application, referring to Figure 2 , a behavior recognition model training method based on time-frequency fusion enhancement is provided, comprising steps K1 and K2. In order to better understand the present application, each step will be described in detail below in combination with specific embodiments.

[0042] Step K1: dividing the training set into multiple batches, and iteratively training the feature extractor in batches, wherein the labels corresponding to different original samples in the same batch are different from each other, and each batch trains a self-supervised contrast learning process based on the original samples contained in the current batch.

[0043] The self-supervised contrast learning process includes the processes of time domain enhancement, frequency domain enhancement, fusion processing, and contrast learning. This part only trains the feature extractor part of the behavior recognition model. The use of time domain enhancement + frequency domain enhancement + self-supervised manner can more effectively improve the feature extractor of the behavior recognition model to extract features useful for behavior classification, so as to subsequently improve the overall precision of the behavior recognition model. The above-mentioned processes will be described respectively.

[0044] The process of time domain enhancement is as follows:

[0045] According to an embodiment of the present application, a corresponding mode is randomly selected from a plurality of preset time domain enhancement modes to enhance each original sample, and a time domain enhanced sample is obtained. For simplicity, the time domain enhancement mode corresponds to the TNet described below (as shown in Figure 1 ). The plurality of preset time domain enhancement modes include jitter processing, scaling processing, and rearrangement processing, wherein the jitter processing refers to adding random Gaussian noise to the original sample, the scaling processing refers to multiplying the original signal by a scaling factor sampled from a normal distribution, and the rearrangement processing refers to randomly rearranging the original signal after being divided into multiple segments. These three time domain enhancement modes can make the model more effectively capture different time changes in the data, and enhance the ability of the model to distinguish different activities. Illustratively, when performing time domain enhancement, an enhancement library named TNet can be set, which includes three time domain enhancement modes: jitter, scaling, and rearrangement. In order to generate different time-frequency enhanced samples, a random time domain enhancement method is selected for each original sample in each batch. In addition, in addition to this, the implementer can also adjust part of the time domain enhancement mode and / or add other time domain enhancement modes based on other ways (3D rotation, inversion (signal symmetry along the time axis), and / or mask processing, etc.).

[0046] The process of frequency domain enhancement is as follows:

[0047] According to an embodiment of the present application, the frequency domain enhanced sample is obtained by first performing fast Fourier transform on the original sample to obtain a frequency domain signal, and then performing frequency domain enhancement on the frequency domain signal using a preset time-frequency enhancement network (such as the FNet network shown in Figure 1 , hereinafter referred to as FNet network) to convert it back to time domain.

[0048] According to one embodiment of the present application, the preset time-frequency enhancement network comprises: a convolution layer, configured to process an input frequency domain signal through a set of convolution layers to obtain an output of the convolution layer; a skip connection, configured to linearly superimpose an input and an output of the convolution layer to obtain a superimposed result; a deconvolution layer, configured to perform inverse operation of the convolution layer on the superimposed result through a set of deconvolution layers to convert the frequency domain signal back to a time domain to obtain an output of the deconvolution layer; and a final convolution layer, configured to perform a final conversion on the output of the deconvolution layer through the final convolution layer to generate an output frequency domain enhanced sample.

[0049] Wherein, the FNet network aims to convert the frequency domain representation of time series data into a time domain signal. The use of the FNet network enhances the original time domain data with frequency domain information (corresponding to the original sample), which is particularly useful for human activity recognition tasks that can have different frequency components for different activities. The FNet network can be of existing structure. The FNet network consists of four main components: a set of convolution layers, a set of deconvolution layers, a skip connection, and a final convolution layer. The input of the network is the frequency domain representation of time series, and the output is the time domain signal. The role of each part is as follows:

[0050] Convolution layer: the input frequency domain signal is processed through a set of convolution layers to obtain the output of the convolution layer. These layers use a set of learnable filters to extract features from the input signal, which aims to capture local patterns in the frequency domain.

[0051] Skip connection: the input of the convolution layer is also transmitted through the skip connection. Taking the convolution layer as an example, the skip connection directly connects the input and output of the convolution layer. This skip connection aims to help the network learn to retain low-level features of the input signal, which can be lost during convolution and deconvolution operations.

[0052] Deconvolution layer: after the output of the convolution layer and the output of the skip connection are superimposed, a set of deconvolution layers are used to perform inverse operation of the convolution layer to convert the frequency domain signal back to the time domain.

[0053] Final convolution layer: the output of the deconvolution layer is processed through the final convolution layer to perform a final conversion to generate an output frequency domain enhanced sample (also a time domain signal).

[0054] The frequency domain enhancement uses the FNet network architecture to convert the frequency domain representation of time series data (corresponding to the frequency domain signal) into the time domain signal, has the effect of frequency domain enhancement, and enables the model to better understand human activities with different frequency components.

[0055] The process of fusion processing is as follows:

[0056] According to an embodiment of the present application, the time-frequency enhanced sample is obtained by multiplying the time-domain enhanced sample corresponding signal data and the frequency-domain enhanced sample corresponding signal data with corresponding elements. In order to obtain the final enhanced data, a multiplication-based combination can be used to fuse the time series data generated by TNet and FNet. This technique involves performing element-wise multiplication on the two enhanced time series (i.e., the time-domain enhanced sample and the frequency-domain enhanced sample), thereby producing a new time series that contains both time-domain and frequency-domain information from the original signal. This fusion approach can help emphasize certain frequency components in the signal while suppressing others, and can improve the model's ability to distinguish between different activities. Illustratively, given two enhanced time series and the final enhanced data .

[0057] The process of contrastive learning is as follows:

[0058] The embodiments of the present application configure the goal of contrastive learning to narrow the distance of the sample features of the original sample and its corresponding time-frequency enhanced sample, and to widen the distance of the sample features of the original sample and other samples.

[0059] According to one embodiment of the present application, when contrast learning, in all samples in a batch, each original sample and its corresponding time-frequency enhanced sample form a positive example pair, and each original sample and each other sample excluding the original sample and its corresponding time-frequency enhanced sample form a negative example pair, wherein the all samples include each original sample and its time-frequency enhanced sample in the batch; all samples in a batch are respectively input into a feature extractor to obtain sample features of each sample with behavior recognition information, and the trainable parameters of the feature extractor are updated through contrast learning of the sample features of the two samples in the positive example pair and the sample features of the two samples in each negative example pair. Since the time-frequency enhanced sample corresponding to the original sample is actually enhanced based on the original sample by using the aforementioned time-frequency enhancement means, it itself should belong to the same behavior class as the corresponding original sample, and the two can be regarded as a positive example pair. For a certain original sample, in addition to the original sample and its corresponding time-frequency enhanced sample, other samples correspond to different behavior classes, and therefore the original sample and other samples can form a negative example pair. Finally, through contrast learning, the distance (spatial distance, such as cosine similarity) between the sample features of the two samples forming the positive example pair is narrowed, and the distance between the sample features of the two samples forming the negative example pair is widened, so as to realize self-supervised contrast learning using the original sample and its time-frequency enhanced sample without labels, and improve the performance of the feature extractor of the model in extracting features useful for behavior recognition. The contrast learning uses the loss calculated by the loss function NT-Xent (Normalized Temperature-Scaled Cross-Entropy Loss) to update the trainable parameters of the feature extractor through gradient and back propagation.

[0060] Self-supervised learning involves pre-training a feature extractor without using target labels. Contrastive methods are a type of pretext task that aims to match different views of the same instance using a metric learning objective function. Other methods create labels from the data itself. The goal of self-supervised learning is to learn informative and useful data representations that can be used for downstream tasks such as classification and prediction.

[0061] The contrast learning method proposed in the embodiment of the present application takes the final augmented data (corresponding to the time-frequency enhanced sample) and the original data (original sample) as a positive pair (corresponding to the positive example pair, also called the positive example pair), and the negative pair (corresponding to the negative example pair) is generated by sampling other instances in the batch (Batch). Normalized temperature scaled cross entropy loss (NT-Xent) is used as the objective function for training the model. Formally, let and represent the representations of two samples in a batch, where i and j are sample indices. The way to calculate the sub-loss of the samples in any one batch in the NT-Xent loss is defined as follows:

[0062]

[0063] wherein N is the number of sample pairs, is the sum of the number of positive pairs and negative pairs in a batch, represents the index of the original sample (the number in a batch), represents the index of the time-frequency enhanced sample corresponding to the i-th original sample.

[0064] wherein: is calculated as follows:

[0065]

[0066] wherein, is a preset temperature parameter, is a function with an indication function, which equals to 1 when and equals to 0 otherwise, represents a cosine similarity function. The NT Xent loss makes the representations of two samples in a positive pair close in the embedding space, while the representations of two samples in a negative pair far apart. In the training process, the model is trained to minimize the average NT Xent loss of all positive pairs and negative pairs in a batch. The formula calculates the average value of the NT Xent loss of all positive pairs and negative pairs in a batch. By optimizing the NT Xent loss, the model learns to extract useful information features from the data, which can be used for downstream tasks such as classification and prediction.

[0067] It should be noted that the embodiments of the present application improve the samples, that is, even if the neural network for extracting time domain features and the neural network for extracting frequency domain features are not respectively arranged in the model structure, the model can still improve the extraction ability of frequency domain features and time domain features based on the contrastive learning of the original samples and the corresponding time-frequency enhanced samples designed by the embodiments of the present application.

[0068] K2, obtaining a behavior recognition model including the feature extractor trained in step A2 and the classifier, performing classification training on the behavior recognition model using the training set to obtain a trained behavior recognition model.

[0069] According to an embodiment of the present application, in step K2, a classification loss is calculated according to the confidence of each behavior category output by the behavior recognition model for the corresponding original sample of the training set and the corresponding label using a classification cross-entropy loss function, and the parameters of the classifier and the feature extractor are updated by gradient and back propagation according to the classification loss. The classification loss calculated according to the classification cross-entropy loss function in the embodiments of the present application further guides the classification training of the behavior recognition model, and improves the accuracy of the model in behavior classification.

[0070] Through the above steps, the training of the behavior recognition model can be realized by using a small amount of samples, because different physical activities usually exhibit different frequency components. The method of the embodiment of the application effectively captures the time and spectral characteristics of human activities, reduces the need for a large amount of annotated data, and achieves better prediction accuracy and robustness compared with the prior art, thereby improving the performance of the model.

[0071] IV. Model application

[0072] According to an embodiment of the application, a user behavior recognition method is provided, including: obtaining sensor data of a user; and inputting the sensor data of the user into a trained behavior recognition model obtained by the method described in the foregoing embodiments, to recognize the behavior of the user.

[0073] According to an embodiment of the application, an electronic device is provided, including one or more processors; and a memory, wherein the memory is configured to store executable instructions; and the one or more processors are configured to implement the steps of the method described in the foregoing embodiments by executing the executable instructions.

[0074] In addition, it should be understood that the above step numbers are only added for the convenience of understanding, and can be removed, or other step numbers are used to implement the technology of the application. Some illustrative other embodiments are given below:

[0075] Other Embodiment 1

[0076] According to an embodiment of the application, a behavior recognition training method based on time-frequency fusion enhancement is provided, which is used in a scenario of recognizing human behavior by collecting sensor data of a human body by a sensor of a wearable device, and includes the following steps:

[0077] obtaining a first data set including a plurality of original signals and a label corresponding to each original signal, the label indicating a behavior category of the human body when the corresponding original signal is collected; segmenting each original signal in the data set by a sliding window to obtain a plurality of original samples of a preset length as a plurality of sensor data, each original sample inheriting the label of the original signal to which it belongs, to form a second data set; extracting a subset from the second data set to form a training set;

[0078] The training set is divided into multiple batches, and the feature extractor is iteratively trained in batches, wherein the labels corresponding to each original sample in each batch are different from each other, and each batch training includes: separately performing time domain enhancement and frequency domain enhancement on each original sample to obtain the corresponding time-frequency enhanced sample; in all samples in a batch, each original sample and its corresponding time-frequency enhanced sample form a positive example pair, and each original sample and each other sample excluding the original sample and its corresponding time-frequency enhanced sample form a negative example pair, wherein the all samples include each original sample in the batch and its time-frequency enhanced sample; all samples in a batch are input into the feature extractor to obtain sample features of each sample with behavior recognition information, and the trainable parameters of the feature extractor are updated through contrast learning of sample features of two samples in a positive example pair and sample features of two samples in each negative example pair; an initial behavior recognition model is obtained, which includes a classifier and a feature extractor trained in step, and the behavior recognition model is fine-tuned using each original sample in the training set and the corresponding label.

[0079] Other embodiments 2:

[0080] According to an embodiment of the present application, a behavior recognition training method based on time-frequency fusion enhancement is provided, comprising: S1, obtaining a training set, which includes a plurality of original samples and a label corresponding to each original sample, the original sample being the sensor data, and the label indicating the behavior category of the human body when the corresponding sensor data is collected; S2, dividing the training set into multiple batches, and iteratively training the feature extractor in batches, wherein the labels corresponding to different original samples in the same batch are different from each other, and the following self-supervised learning process is performed based on the original samples contained in each batch: S21, randomly selecting a corresponding mode from a plurality of preset time domain enhancement modes for each original sample to perform enhancement, to obtain a time domain enhanced sample, S22, converting each original sample from a time series signal to a frequency domain signal, and performing frequency domain enhancement on the frequency domain signal through a preset frequency domain enhancement mode and converting it back to time domain to obtain a frequency domain enhanced sample, S23, fusing the time domain enhanced sample and the frequency domain enhanced sample corresponding to each original sample to obtain a time-frequency enhanced sample corresponding to the original sample; S24, training the feature extractor to extract sample features with behavior recognition information according to the input sample based on each original sample and the corresponding time-frequency enhanced sample through a contrast learning manner, wherein the goal of contrast learning is to narrow the distance between the sample features of the original sample and its corresponding time-frequency enhanced sample and to widen the distance between the sample features of the original sample and other samples; A3, obtaining a behavior recognition model, which includes a classifier and a feature extractor trained in step A2, and performing classification training on the behavior recognition model using each original sample in the training set and the corresponding label to obtain a trained behavior recognition model.

[0081] Other embodiments 3:

[0082] According to one embodiment of the application, a time-frequency fusion enhanced behavior recognition training method (or parameter adaptive fine-grained transfer learning method) is provided, comprising:

[0083] 1) The original signal is segmented by a sliding window to obtain a plurality of original samples;

[0084] 2) The original sample is copied into two copies to obtain the first and second original signals;

[0085] 3) The first original signal is randomly subjected to one or more of a plurality of preset time domain enhancement methods (such as jitter, scaling, arrangement, etc.) for time domain enhancement to obtain a time domain enhanced signal (corresponding to a time domain enhanced sample);

[0086] 4) The second original signal is subjected to a fast Fourier transform to obtain a frequency domain signal;

[0087] 5) The frequency domain signal is converted using a time-frequency conversion network (FNet) to obtain a denoised time domain signal (corresponding to a frequency domain enhanced sample)

[0088] 6) The time domain enhanced signal and the denoised time domain signal are fused by multiplying the corresponding elements to obtain the final enhanced signal (corresponding to the time-frequency enhanced sample)

[0089] 9) The final time-frequency enhanced sample and the original signal are compared to obtain a feature extractor

[0090] 10) A classifier is added to the feature extractor obtained in step 9), and the behavior recognition model is trained to obtain a trained behavior recognition model.

[0091] In order to verify the effect of the embodiments of the application, the inventors also conducted related comparative experiments. The following are the related contents of the comparative experiments:

[0092] a) Purpose of the experiment

[0093] To solve the problem that a large amount of labeled time series data is needed to build a general recognition model in the field of behavior recognition, and the behavior recognition data labeling is difficult, an embodiment of the present application proposes a method (or a self-supervised contrastive behavior recognition method based on time-frequency fusion enhancement, hereinafter referred to as FusionCL). FusionCL is composed of two parts of behavior recognition self-supervised pre-training and small data fine-tuning. In the self-supervised pre-training process, FusionCL realizes two different data enhancement methods, including a time domain enhancement method (such as jitter, scaling and permutation) and a frequency domain based enhancement method, which are named as TNet and FNet respectively, and the two enhancement methods are fused to realize more diverse data enhancement. The goal of TNet is to capture various time features of the initial sensor data, including but not limited to jitter, scaling and permutation, while FNet aims to enhance the original time domain data with frequency domain information. In the pre-training stage, TNet is used for frequency domain enhancement and FNet is used for noise reduction of the original time domain data, and the final enhanced data is obtained by element-wise multiplication. The final enhanced data and the original data are used as positive samples, and a feature extraction model is pre-trained by applying contrastive learning. In the fine-tuning process, a three-layer multilayer perceptron is connected behind the model obtained by pre-training to form a classification model, and the new classification model is fine-tuned using a limited labeled data set, so that it can adapt to the label system of the data set. The purpose of the comparative experiment of the present application is to verify the effect of FusionCL. FusionCL adopts a three-layer CNN.

[0094] b) Data set and pre-processing

[0095] Table 1 Four kinds of public benchmark data sets

[0096]

[0097] c) Contrastive method and parameter details

[0098] To verify the experimental effect of the FuseCL method, the present application selects 10 existing methods as contrastive methods, including:

[0099] SVM (Support Vector Machine)

[0100] KNN (K-Nearest Neighbors)

[0101] CNN baseline

[0102] Resnet baseline

[0103] AutoEncoder

[0104] TF-C (Time-Frequency Consistency)

[0105] TS2Vec

[0106] Mixing-up

[0107] TS-TCC (Time-Series representation learning framework via Temporal and Contextual Contrasting)

[0108] SimCLR

[0109] The above table shows the experimental results of various methods on different datasets. The FusionCL method proposed in the study outperforms all other self-supervised methods and traditional machine learning methods such as KNN and SVM, achieving the highest accuracy score on all four datasets. In addition, the accuracy of the FusionCL method on the Unimib-SHAR dataset reached 81.25%, higher than the accuracy achieved by the CNN baseline and Resnet baseline, and even close to the state-of-the-art result. On the other three datasets, the accuracy of the FusionCL method is close to that of supervised learning methods, indicating that the FusionCL method can achieve performance comparable to supervised learning methods with only 20% of the labels for fine-tuning

[0110] d) Comparison of experimental results

[0111] Table 2 Comparison of experimental results

[0112]

[0113] e) Ablation experiment results

[0114] Table 3 Ablation experiment results

[0115]

[0116] The ablation experiments in this study aim to investigate the contribution of TNet and FNet to the performance of the FusionCL method. To this end, two experiments were conducted: one involving the deletion of TNet and the use of FNet only, and the other involving the deletion of FNet and the use of TNet only. The experimental results are shown in Table III, where the TNet and FNet modules are evaluated separately and compared with FusionCL. The results show that the FusionCL method, which uses both TNet and FNet, achieves the highest accuracy scores on all four datasets. Removing TNet or FNet from the method results in lower accuracy scores, indicating that both TNet and FNet significantly contribute to the performance of the FusionCL method. These findings suggest that the proposed FusionCL method effectively leverages the complementarity between the two networks, thereby improving the performance of self-supervised learning for human activity recognition.

[0117] It should be noted that although the above describes various steps in a particular order, it does not mean that the steps must be performed in the above-mentioned particular order. In fact, some of the steps can be performed concurrently or even in reverse order, as long as the desired function can be achieved.

[0118] The present application can be a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present application.

[0119] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or punched tape, a

[0120] Having described various embodiments of the application, it is to be understood that the above description is meant not to limit and not to encompass all of the possible embodiments covered by the claims. Many modifications and variations of this application can be apparent to those of ordinary skill in the art without departing from the spirit and scope of the described embodiments. It is intended that the scope of the application should only be limited by the appended claims.

Claims

1. A behavior recognition model training method based on time-frequency fusion enhancement is used to identify human behavior scenarios based on sensor data collected by sensors of wearable devices on the human body, characterized in that: include: A1. Obtain a training set, which includes multiple original samples and labels corresponding to each original sample. The original sample is single-channel or multi-channel sensor data, and the label indicates the human behavior category when the corresponding sensor data is collected; A2. Divide the training set into multiple batches and iteratively train the feature extractor of the behavior recognition model in batches. Different raw samples in the same batch have different labels. Each training batch performs the following self-supervised contrastive learning process based on the raw samples in the current batch: A21. Each original sample is subjected to time domain enhancement and frequency domain enhancement and then fused to obtain a time-frequency enhanced sample, wherein step A21 includes: randomly selecting a corresponding method from a plurality of preset time domain enhancement methods to enhance each original sample to obtain a time-domain enhanced sample; converting each original sample from the time domain to the frequency domain and then converting it back to the time domain after frequency domain enhancement to obtain a frequency-domain enhanced sample; fusing the time-domain enhanced sample and the frequency-domain enhanced sample corresponding to each original sample to obtain a time-frequency enhanced sample corresponding to the original sample; the frequency-domain enhanced sample is obtained by first performing fast Fourier transform on the original sample to obtain a frequency domain signal. signal, and then using a preset time-frequency enhancement network to convert the frequency domain signal back to the time domain after frequency domain enhancement; the preset time-frequency enhancement network includes: a convolution layer, which is used to process the input frequency domain signal through a group of convolution layers to obtain the output of the convolution layer; a jump connection, which is used to linearly superimpose the input and output of the convolution layer to obtain a superposition result; a deconvolution layer, which is used to perform an inverse operation of the convolution layer on the superposition result through a group of deconvolution layers, convert the frequency domain signal back to the time domain, and obtain the output of the deconvolution layer; a final convolution layer, which is used to pass the output of the deconvolution layer through the final convolution layer for final conversion to generate an output sample with frequency domain enhancement; A22. Based on each original sample and the corresponding time-frequency enhanced sample, train the feature extractor to extract sample features containing behavior recognition information from the input sample through contrastive learning, wherein the goal of contrastive learning is configured to narrow the distance between the sample features of the original sample and its corresponding time-frequency enhanced sample and to widen the distance between the sample features of the original sample and other samples; A3. Obtain a behavior recognition model, which includes the feature extractor and classifier trained in step A2, and use the training set to perform classification training on the behavior recognition model to obtain a trained behavior recognition model.

2. The method according to claim 1, characterized in that The multiple preset time domain enhancement methods include dithering processing, scaling processing and rearrangement processing, wherein, Dithering refers to adding random Gaussian noise to the original samples. Scaling refers to multiplying the original signal by a scaling factor sampled from a normal distribution. Rearrangement processing refers to dividing the original signal into multiple segments and then randomly rearranging them.

3. The method according to claim 2, characterized in that The time-frequency enhanced samples are obtained by fusing the signal data corresponding to the samples enhanced in the time domain and the signal data corresponding to the samples enhanced in the frequency domain by multiplying the corresponding elements.

4. The method according to any one of claims 1 to 3, characterized in that Step A22 includes: Among all samples in a batch, each original sample and its corresponding time-frequency enhanced sample are combined into a positive pair, and each original sample and every other sample excluding the original sample and its corresponding time-frequency enhanced sample are combined into a negative pair, wherein all samples include each original sample in the batch and its time-frequency enhanced sample; All samples in a batch are input into the feature extractor respectively to obtain the sample features of each sample with behavior recognition information. The trainable parameters of the feature extractor are updated by comparing the sample features of the two samples in the positive example pair and the sample features of the two samples in each negative example pair.

5. The method according to claim 4, characterized in that The contrastive learning uses the loss calculated by the normalized temperature scaling contrast loss function to find the gradient and back-propagate to update the trainable parameters of the feature extractor.

6. The method according to any one of claims 1 to 5, characterized in that The tag space corresponding to the tag is walking, running, jumping, sitting, lying down, standing, cycling, Nordic walking, watching TV, working on a computer, driving, going upstairs, going downstairs, vacuuming, ironing, folding clothes, cleaning the house, playing football, jumping rope, sitting to standing, lying down to standing, standing to lying down, sitting down, falling forward, falling to the left, falling to the right, falling backward, falling while sitting on a chair, falling with protection, falling and hitting an obstacle, fainting and other behaviors or their combinations.

7. A user behavior identification method, characterized in that: include: Obtaining user sensor data The user's sensor data is input into a trained behavior recognition model obtained by the method according to any one of claims 1 to 6 to recognize the user's behavior.

8. A computer-readable storage medium, characterized in that A computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of the method according to any one of claims 1 to 7.

9. An electronic device, characterized in that: include: one or more processors; as well as a memory, wherein the memory is used to store executable instructions; The one or more processors are configured to implement the steps of the method of any one of claims 1 to 7 by executing the executable instructions.

Images