A method for constructing a behavior recognition model, a behavior recognition model and a method

Abstract

The application provides a behavior recognition model construction method based on wearable device data, and the method comprises the following steps: S1, obtaining data of a wearable device and performing preprocessing to obtain a training set, wherein the training set comprises part of behavior categories in known behavior categories, each behavior category corresponds to a group of behavior attributes, and each behavior category comprises a plurality of samples with behavior category labels; S2, training a basic behavior recognition model to convergence by using the training set, wherein the basic behavior recognition model comprises a feature extraction network, an attribute branch network, a feature branch network and a classifier, the attribute branch network comprises an attribute encoder and an attribute decoder, and the feature branch network comprises a feature encoder and a feature decoder; S3, constructing a behavior recognition model by using the feature extraction network, the attribute decoder, the feature encoder and the classifier in the basic behavior recognition model trained to convergence in step S2.

Description

Technical Field

[0001] This invention relates to the field of behavior recognition in machine learning, specifically to the field of behavior recognition based on wearable devices, and more specifically, to a method for constructing a behavior recognition model based on wearable device data, a behavior recognition model, and a method thereof. Background Technology

[0002] Today, wearable devices equipped with sensors can recognize human behavior, meeting the needs of daily health monitoring and making it more convenient to monitor one's own physiological signs. Therefore, wearable devices have gradually become an important part of many areas related to human health, such as fall detection, healthcare, and smart homes. Compared to traditional medical monitoring equipment, wearable devices offer advantages such as greater portability, less intrusiveness, better protection of user privacy, the ability to collect more behavioral data from daily life, and smaller storage space required for the collected data. Therefore, large-scale datasets with behavioral data labels can be obtained through wearable devices to train models with intelligent behavior recognition capabilities. However, most existing research on human behavior recognition generally faces two problems: First, the labeled datasets used to train most existing research models often only contain a subset of known behavior categories. This limits the ability of most existing research techniques to identify other known behavior categories that were not present during training. Moreover, the effectiveness of most existing research techniques depends on the massive amount of labeled data corresponding to the behavior categories in the labeled dataset, meaning that most existing research techniques cannot effectively identify behavior categories with only a small amount of labeled data. Second, collecting behavioral data on all human activities in real life is almost impossible, and the cost of labeling the massive amounts of collected behavioral data is extremely expensive and prohibitive.

[0003] Therefore, how to improve the effectiveness of existing technologies in identifying known behavior categories with insufficient labeled data, and how to overcome the limitation that existing technologies can only identify behavior categories seen during training but not those not seen during training, have become urgent problems to be solved. Summary of the Invention

[0004] Therefore, the purpose of this invention is to overcome the shortcomings of the prior art and provide a method for constructing a behavior recognition model based on wearable device data, a behavior recognition model, and a behavior recognition method.

[0005] The objective of this invention is achieved through the following technical solution:

[0006] According to a first aspect of the present invention, a method for constructing a behavior recognition model based on wearable device data is provided. The method includes the following steps: S1, acquiring data from the wearable device and preprocessing it to obtain a training set, wherein the training set includes a subset of known behavior categories, each behavior category corresponds to a set of behavior attributes, and each behavior category includes multiple samples labeled with behavior categories; S2, training a basic behavior recognition model to convergence using the training set, wherein the basic behavior recognition model includes a feature extraction network, an attribute branch network, a feature branch network, and a classifier, and the attribute branch network includes an attribute encoder and an attribute decoder, and the feature branch network includes a feature encoder and a feature decoder, and the following steps are performed during the training process: S21, using the attribute encoder in the attribute branch network to extract the latent attribute vector of each sample based on the behavior attributes corresponding to the behavior category to which each sample belongs, and using the attribute decoder to obtain the predicted behavior attribute corresponding to each sample based on the latent attribute vector of each sample; S22 S23. The feature extraction network is used to extract the spatiotemporal features of each sample in the training set; S24. The feature decoder in the feature branch network is used to extract the latent feature vector of each sample based on the spatiotemporal features of each sample, and the feature decoder is used to obtain the predicted feature corresponding to each sample based on the latent feature vector of each sample; S25. The classifier is used to obtain the behavior category classification prediction result corresponding to each sample based on the latent feature vector of each sample; S26. The attribute prediction loss is calculated based on the behavior attributes corresponding to the behavior category of all samples and the predicted behavior attributes of all samples; the feature prediction loss is calculated based on the spatiotemporal features and predicted features of all samples; the classification prediction loss is calculated based on the behavior category classification prediction result of all samples and the behavior category label corresponding to all samples; S27. The parameters of the basic behavior recognition model are updated based on the comprehensive loss of all losses calculated in step S25; S28. The behavior recognition model is constructed using the feature extraction network, attribute decoder, feature encoder and classifier in the basic behavior recognition model trained to convergence in step S2.

[0007] In some embodiments of the present invention, step S1 includes: S11, acquiring historical time-series data recorded by multiple wearable devices, wherein all historical time-series data recorded by wearable devices correspond to continuous behavioral actions composed of a series of known behavioral categories; S12, simultaneously truncating all historical time-series data recorded by wearable devices acquired in step S11 using a preset sliding time window to obtain multiple samples, each sample containing data from all wearable devices within the same window time period, and using the behavior being performed at the last moment within the window time period as the behavior category label of the sample corresponding to that window time period; S13, forming a dataset from all samples obtained in step S12, and dividing the dataset into a training set and a test set according to preset rules, wherein the training set is used to train a basic behavior recognition model, and the test set is used to test whether the accuracy of the basic behavior recognition model on the test set reaches a preset accuracy requirement, and the training set includes samples corresponding to some behavior categories among the known behavior categories, and the test set includes different samples of the same behavior categories as those in the training set and samples corresponding to behavior categories different from those in the training set.

[0008] In some embodiments of the present invention, the feature extraction network includes a long short-term memory network and a convolutional neural network.

[0009] In some embodiments of the present invention, in step S22, a feature extraction network is used to extract the spatiotemporal features of each sample in the training set in the following manner: a long short-term memory network is used to extract the temporal features of each sample, and a long short-term memory network and a convolutional neural network are used to extract the spatial features of each sample respectively; based on the temporal features and spatial features of each sample, the spatiotemporal features of each sample are determined respectively.

[0010] In some embodiments of the present invention, the spatiotemporal characteristics of each sample are determined as follows:

[0011] x i =E([h1, h2, h3]),

[0012] Where, x i Let h1 represent the spatiotemporal features of the i-th sample in the training set, E represent the mathematical expectation, h1 represent the temporal features of the i-th sample extracted by the Long Short-Term Memory Network, h2 represent the spatial features of the i-th sample extracted by the Long Short-Term Memory Network, and h3 represent the spatial features of the i-th sample extracted by the Convolutional Neural Network.

[0013] In some embodiments of the present invention, in step S25, the attribute prediction loss is calculated as follows:

[0014]

[0015] Where 'a' represents the behavioral attribute corresponding to the behavioral category to which the sample belongs, and 'z'... a Represents the latent vector of attributes of a sample. Indicate z a The approximate inference function of the posterior distribution corresponding to a, p(z) a ) represents z a The Gaussian distribution function is given by E, where E represents the expected value and KL(·) represents the divergence function.

[0016] In some embodiments of the present invention, in step S25, the feature prediction loss is calculated as follows:

[0017]

[0018] Where x represents the spatiotemporal feature corresponding to the sample, z x Represents the latent feature vector of a sample. Indicate z x The approximate inference function of the posterior distribution corresponding to x, p(z) x ) represents z x The Gaussian distribution function is given by E, where E represents the expected value and KL(·) represents the divergence function.

[0019] In some embodiments of the present invention, in step S25, the classification prediction loss is calculated as follows:

[0020]

[0021] in, The latent feature vector z of the sample x Corresponding feature labels The latent feature vector z of the sample x The corresponding classification prediction results, The latent vector z of the sample represents the attribute. a The corresponding attribute tags, The latent vector z of the sample represents the attribute. a The corresponding classification prediction results.

[0022] In some embodiments of the present invention, step S24 further includes: using an attribute decoder to obtain the cross-prediction attribute of each sample based on the feature latent vector of each sample, and using a feature decoder to obtain the cross-prediction feature of each sample based on the attribute latent vector of each sample; step S25 further includes: calculating the cross-alignment loss based on the behavioral attributes, cross-prediction attributes, spatiotemporal features and cross-prediction features of all samples.

[0023] In some embodiments of the present invention, in step S25, the cross-alignment loss is calculated as follows:

[0024]

[0025] Where 'a' represents the behavioral attribute corresponding to the behavioral category to which the sample belongs, 'x' represents the spatiotemporal feature corresponding to the sample, and 'D' represents the behavior attribute corresponding to the behavioral category to which the sample belongs. s (z x ) represents the attribute decoder D s Based on the latent feature vector z of the sample x The obtained cross-prediction attribute, D x (z a ) represents the feature decoder D x Based on the latent attribute vector z of the sample a The obtained cross-prediction features, where E represents the mathematical expectation.

[0026] In some embodiments of the present invention, step S25 further includes: calculating the contrastive alignment loss based on the attribute latent vectors and feature latent vectors of all samples in the following manner:

[0027]

[0028] Where M represents the number of samples in the training set, y i Let y represent the label of the i-th sample. j Let y represent the label of the j-th sample. k Let c represent the label of the k-th sample. i，j Let c represent the cosine similarity between the i-th sample and the j-th sample. i，k Represents the cosine similarity between the i-th sample and the k-th sample, 1 i≠j The value is 1 when the i-th sample and the j-th sample are different, and 0 when they are the same. The value is 1 when the label of the i-th sample is different from that of the j-th sample, and 0 when they are the same. t represents the hyperparameter, and E represents the expected value.

[0029] In some embodiments of the present invention, in step S26, the comprehensive loss is calculated as follows:

[0030]

[0031] in, This represents the attribute prediction loss. This represents the feature prediction loss. Indicates the cross-alignment loss. Indicates the contrast alignment loss. λ1, λ2, and λ3 represent the classification prediction loss, and λ1, λ2, and λ3 are all preset parameters.

[0032] According to a second aspect of the present invention, a behavior recognition model is provided for recognizing behavior categories corresponding to wearable device data. The model includes: a feature extraction network, a feature encoder, an attribute decoder, and a classifier, all trained based on the method described in the first aspect of the present invention. The feature extraction network extracts spatiotemporal features from the wearable device data; the feature encoder encodes the spatiotemporal features extracted by the feature extraction network to obtain the latent feature vectors of the wearable device data; the attribute decoder decodes the latent feature vectors obtained by the feature encoder to obtain the cross-prediction attributes of the wearable device data; and the classifier classifies data that, after being determined by the behavior category judgment module, belong to the model training process. The system includes: a wearable device data set containing known behavior categories seen in the training set, and behavior category prediction based on their corresponding feature latent vectors; a behavior category judgment module, used to determine whether the wearable device data belongs to a known behavior category seen in the training set or a known behavior category not seen in the test set during model training, based on the distance between the feature latent vectors of the wearable device data and the behavior attributes corresponding to all behavior categories seen during model training; and an invisible classifier, used to determine the behavior category of wearable device data belonging to a known behavior category not seen in the test set during model training, based on their corresponding cross-prediction attributes, after the behavior category judgment module has determined it to be such that it belongs to a known behavior category not seen during model training.

[0033] In some embodiments of the present invention, the behavior category determination module is configured to determine whether the wearable device data belongs to a known behavior category seen in the training set during model training in the following manner:

[0034]

[0035]

[0036] Among them, z te Represents the latent feature vector of wearable device data. This represents the set of manifold centers corresponding to all known behavior categories seen in the training set during model training. Indicates the calculation of z te and The cosine similarity between the manifold centers of each known behavior category is denoted by max(·), where max(·) represents the function that takes the maximum value. `seen` represents the known behavior categories seen in the training set during model training, and `unseen` represents the known behavior categories not seen in the test set during model training. `η` refers to the popular boundary corresponding to each known behavior category seen in the training set during model training. This indicates that the wearable device data belongs to the known behavior categories seen in the training set during model training. This represents known behavior categories in the test set that were not seen during model training.

[0037] In some embodiments of the present invention, the invisible classifier is configured to perform behavior category determination on wearable device data that, after being determined by the behavior category determination module, belong to a known behavior category in the test set, in the following manner:

[0038]

[0039] in, This indicates that the wearable device data belongs to a known behavior category in the test set that was not seen during model training. This represents the cross-prediction attribute of wearable device data. This represents the set of behavioral attributes corresponding to all known behavioral categories in the test set that were not seen during model training. Indicates calculation and The cosine similarity between the behavioral attributes corresponding to each known behavioral category, where argmax(·) represents the selection. and The known behavior categories corresponding to the maximum cosine similarity between them This represents the classification prediction results for wearable device data belonging to known behavioral categories in the test set that were not seen during model training.

[0040] According to a third aspect of the present invention, a behavior recognition method based on the behavior recognition model described in the second aspect of the present invention is provided, the method comprising the following steps: T1, acquiring wearable device data to be processed; T2, processing the wearable device data to be processed using the behavior recognition model to identify its behavior category.

[0041] Compared with the prior art, the advantages of the present invention are as follows:

[0042] 1. In the training process, this invention improves the effectiveness of the behavior recognition model in recognizing user behavior by introducing corresponding behavior attributes for all known behavior categories and by jointly training the features of the samples in the training set and the behavior attributes corresponding to the behavior categories to which the samples belong.

[0043] 2. Based on the basic behavior recognition model trained to convergence, this invention constructs a new behavior recognition model by introducing a behavior category judgment module and an invisible classifier. Based on the behavior attributes corresponding to the behavior categories in the unseen category set, it can recognize not only the behavior categories in the seen category set but also the behavior categories in the unseen category set without training an additional model.

[0044] 3. In the training process, this invention constructs new attribute prediction loss, feature prediction loss, cross alignment loss and contrast alignment loss, which realizes the alignment of the two data modalities of behavioral attributes and features in the latent space, and enables the behavior recognition model constructed by this invention to effectively distinguish whether the data to be identified belongs to the seen class set or the unseen class set during the application process. Attached Figure Description

[0045] The embodiments of the present invention will be further described below with reference to the accompanying drawings, wherein:

[0046] Figure 1 This is a schematic diagram illustrating the behavioral attributes of a known behavioral category according to an embodiment of the present invention;

[0047] Figure 2 This is a schematic diagram illustrating the construction process of a behavior recognition model according to an embodiment of the present invention;

[0048] Figure 3 This is a schematic diagram of a training method for a basic behavior recognition model according to an embodiment of the present invention;

[0049] Figure 4 This is a schematic diagram of the structure of the behavior recognition model according to an embodiment of the present invention; Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the invention is further described in detail below through specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0051] It should be noted that, for ease of description, this invention refers to the behavior categories seen during model training as seen classes, and all seen classes constitute the seen class set; and the behavior categories not seen during model training are referred to as unseen classes, and all unseen classes constitute the unseen class set.

[0052] As mentioned in the background section, existing methods for recognizing human behavior generally suffer from problems such as lack of ability to identify unseen behavior categories, poor effectiveness in recognizing seen behavior categories with insufficient labeled data, and high cost of labeling massive amounts of behavior data. To address these issues, this invention provides a behavior recognition model construction scheme based on wearable device data that can effectively identify unseen behavior categories, has high effectiveness in recognizing seen behavior categories with insufficient labeled data, and does not require massive amounts of labeled data. In summary, the technical approach of this invention is as follows: During the training process of the basic behavior recognition model, behavioral attributes corresponding to the behavior category of each sample in the training set are introduced, and the latent vectors of attributes and features of each sample are extracted. Then, classification prediction is performed based on the latent vectors of the samples, and the predicted behavioral attributes, predicted features, cross-prediction attributes, and cross-prediction features of the samples are obtained based on the latent vectors of attributes and features, respectively. Finally, the parameters of the basic behavior recognition model are updated until convergence is achieved through the constructed total loss. The partially converged basic behavior recognition model is then combined with the introduced behavior category judgment module and the invisible classifier to form a behavior recognition model. In particular, introducing behavioral attributes corresponding to the behavioral categories of all samples in the training set can supplement the category information of the samples, which greatly improves the effectiveness of the behavior recognition model in recognizing behavior categories with insufficient labeled data. Furthermore, the invisible classifier in the behavior recognition model can identify the specific behavior category of wearable device data in the unseen category set through the cross-prediction attributes corresponding to wearable device data. This solves the problem that existing models can only recognize behavior categories in the seen category set but cannot recognize behavior categories in the unseen category set, while also reducing the cost of data labeling.

[0053] In this invention, the so-called behavioral attribute is a semantic feature that defines a behavioral category. When constructing the behavior recognition model, during the preparation phase of the wearable device dataset, this invention defines a set of behavioral attributes for all known behavioral categories (including all behavioral categories in both the seen and unseen class sets) so that the trained model can obtain more complete category information from the wearable device data. According to one example of this invention, such as... Figure 1As shown, it illustrates some known behavior categories and their corresponding behavioral attributes, including five behavior categories: lying down, sitting, standing, walking, and running. Each of these five behavior categories corresponds to a set of behavioral attributes. Taking walking as an example, its corresponding set of behavioral attributes includes: dynamic behavior attributes, cyclical movement behavior attributes, translational movement behavior attributes, vertical body behavior attributes, forward body behavior attributes, arm movement behavior attributes, arm bending and straightening transition behavior attributes, leg movement behavior attributes, leg bending and straightening transition behavior attributes, alternating leg transition behavior attributes, indoor attributes, and outdoor attributes. Subsequently, during the training phase of the basic behavior recognition model, the behavioral attributes introduced in this invention will participate in the training in a semantic form to improve the optimization effect of the trained basic behavior recognition model.

[0054] To better understand the present invention, the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0055] According to one embodiment of the present invention, such as Figure 2 As shown, this invention provides a method for constructing a behavior recognition model based on wearable device data, including: S1, acquiring data from wearable devices and preprocessing it to obtain a training set; S2, training a basic behavior recognition model to convergence using the training set; S3, constructing a behavior recognition model using the feature extraction network, attribute decoder, feature encoder, and classifier in the basic behavior recognition model trained to convergence in step S2. It should be noted that this invention uses various wearable devices to monitor and record user behavior data. Different wearable devices are used to monitor and record different user behavior data. For example, smartwatches are mainly used to monitor users' vital signs data (such as body temperature data, heart rate data, etc.), while fitness trackers are mainly used to monitor users' motion data (such as 3D acceleration data, 3D gyroscope data, etc.).

[0056] According to one embodiment of the present invention, such as Figure 3As shown, the basic behavior recognition model used in the training process of this invention includes: a feature extraction network, a feature branch network, an attribute branch network, and a classifier. The attribute branch network includes an attribute encoder and an attribute decoder, and the feature branch network includes a feature encoder and a feature decoder. Specifically, the feature extraction network extracts the spatiotemporal features of samples in the training set; the feature encoder in the feature branch network encodes the spatiotemporal features of the samples to extract the latent feature vectors of the samples; the feature decoder in the feature branch network decodes the latent feature vectors and attribute latent vectors of the samples to obtain the predicted features and cross-prediction features of the samples, respectively; the attribute encoder in the attribute branch network encodes the behavioral attributes corresponding to the samples (i.e., the semantics introduced to describe the behavioral category to which the sample belongs) to extract the latent attribute vectors of the samples; the attribute decoder in the attribute branch network decodes the latent attribute vectors and feature latent vectors of the samples to obtain the predicted behavioral attributes and cross-prediction attributes of the samples, respectively; and the classifier performs behavioral category classification prediction on the samples based on their latent feature vectors. According to an embodiment of this invention, both the attribute branch network and the feature branch network used in this invention are variational autoencoder models with encoding and decoding structures. It should be noted that the specific model structures used in the attribute branch network and feature branch network in this invention can be set according to the actual scenario, and this invention does not impose any specific limitations.

[0057] To better understand the present invention, steps S1 to S3 described above will be explained in detail below.

[0058] In step S1, the present invention obtains a training set by acquiring and preprocessing data from wearable devices. According to one embodiment of the present invention, the present invention acquires and preprocesses wearable device data through the following steps: S11, acquiring historical time-series data recorded by multiple wearable devices, wherein all historical time-series data recorded by all wearable devices correspond to continuous behavioral actions composed of a series of known behavioral categories; S12, simultaneously truncating all historical time-series data recorded by all wearable devices acquired in step S11 using a preset sliding time window to obtain multiple samples, each sample containing data from all wearable devices within the same window time period, and using the behavior being performed at the last moment within the window time period as the behavioral category label of the sample corresponding to that window time period; S13, processing all samples obtained in step S12... A dataset is constructed and divided into a training set and a test set according to preset rules. The training set is used to train a basic behavior recognition model, and the test set is used to test whether the accuracy of the basic behavior recognition model on the test set meets the preset accuracy requirements. The training set includes samples corresponding to some behavior categories from known behavior categories (the behavior categories in the training set are seen categories relative to the basic behavior recognition model). The test set includes different samples of the same behavior categories as those in the training set, as well as samples corresponding to behavior categories different from those in the training set (the behavior categories in the test set include a small number of seen categories and unseen categories relative to the basic behavior recognition model). It should be noted that the preset sliding window used in this invention employs an overlapping window strategy when truncating all historical time-series data recorded by wearable devices. This overlapping strategy effectively avoids the problem of losing boundary data between adjacent windows in the obtained sample data, ensuring the integrity of each sample data.

[0059] In step S2, the present invention uses the training set obtained in step S1 to train the basic behavior recognition model to convergence. According to an embodiment of the present invention, the following steps are performed during the training process: S21, using an attribute encoder in an attribute branch network to extract the latent attribute vector of each sample based on the behavioral attributes corresponding to the behavioral category to which each sample belongs, and using an attribute decoder to obtain the predicted behavioral attribute corresponding to each sample based on the latent attribute vector of each sample; S22, using a feature extraction network to extract the spatiotemporal features of each sample in the training set; S23, using a feature decoder in a feature branch network to extract the latent feature vector of each sample based on the spatiotemporal features of each sample, and using a feature decoder to obtain the predicted feature corresponding to each sample based on the latent feature vector of each sample; S24, using a classifier to obtain the behavioral category classification prediction result corresponding to each sample based on the latent feature vector of each sample; S25, calculating the attribute prediction loss based on the behavioral attributes corresponding to the behavioral categories to which all samples belong and the predicted behavioral attributes of all samples, calculating the feature prediction loss based on the spatiotemporal features and predicted features of all samples, and calculating the classification prediction loss based on the behavioral category classification prediction results of all samples and the behavioral category labels corresponding to all samples; S26, updating the parameters of the basic behavior recognition model based on the comprehensive loss of all losses calculated in step S25. According to an embodiment of the present invention, during the training process, step S24 further includes: obtaining the cross-prediction attribute of each sample based on the feature latent vector of each sample using an attribute decoder, and obtaining the cross-prediction feature of each sample based on the attribute latent vector of each sample using a feature decoder; step S25 further includes: calculating the cross-alignment loss based on the behavioral attributes, cross-prediction attributes, spatiotemporal features, and cross-prediction features of all samples, and calculating the contrast alignment loss based on the attribute latent vectors and feature latent vectors of all samples. It should be noted that the feature prediction loss, attribute prediction loss, cross-alignment loss, and contrast alignment loss in the present invention, during the training process, aim to group samples of the same behavioral category together as much as possible and separate samples of different behavioral categories as much as possible, so that the finally constructed behavior recognition model can have a better recognition effect on user behavior.

[0060] According to an embodiment of the present invention, in step S21, the latent attribute vector corresponding to each sample is obtained as follows: z a =E S (a), where z a Let E represent the latent attribute vector of a sample, where 'a' represents the behavioral attribute corresponding to the behavioral category to which the sample belongs. S E represents the attribute encoder. S (a) indicates that 'a' is used as the input to the attribute encoder. According to one embodiment of the present invention, the predicted behavioral attribute corresponding to each sample is obtained in the following manner: in, z represents the predicted behavior attribute corresponding to the sample. a D represents the latent vector of attributes of a sample. S D represents the attribute decoder. S (z a ) indicates that z a As input to the attribute decoder.

[0061] According to one embodiment of the present invention, in step S22, the feature extraction network includes Long Short-Term Memory Networks (LSTMs) and Convolutional Neural Networks (CNNs). Specifically, the LSTMs are used to extract the temporal features of each sample, and the LSTMs and CNNs are used respectively to extract the spatial features of each sample. According to one embodiment of the present invention, the present invention extracts historical temporal data corresponding to multiple wearable devices through LSTMs [v] i1 …v iL The temporal characteristics h1=f t ([v i1 …v iL ]), where v i1 This represents the historical time-series data recorded by the first wearable device within a sliding time window, v iL This represents the historical time-series data recorded by the Lth wearable device within a sliding time window, [v i1 …v iL ] represents the historical time-series data recorded by all wearable devices from the 1st to the Lth wearable device within the same sliding time window (i.e., a single sample data point), h1 represents the temporal characteristics of the sample, and f t (·) represents a function that calculates the temporal features of a sample using a Long Short-Term Memory (LSTM) network. Furthermore, this invention extracts spatial features of samples using both LSTM and convolutional neural networks. According to one embodiment of this invention, LSTM is used to extract spatial features of samples. in, This represents the data recorded by the i-th sample on the first wearable device. Let f represent the data recorded by the i-th sample on the L-th wearable device, where T represents the transpose. sl (·) represents a function that calculates the spatial features of a sample using a Long Short-Term Memory (LSTM) network. According to one embodiment of the present invention, a convolutional neural network is used to extract the spatial features h3 = f of the sample. sc (X i ), where X i Let f represent the i-th sample. sc(·) represents a function that calculates the spatial features of a sample using a convolutional neural network.

[0062] The temporal and spatial features of all samples can be obtained using the methods described in the above embodiments. According to one embodiment of the present invention, after obtaining the temporal and spatial features of all samples, the spatiotemporal features of each sample are determined as follows: x i =E([h1, h2, h3]), where x i Let h1 represent the spatiotemporal features of the i-th sample in the training set, E represent the mathematical expectation, h1 represent the temporal features of the i-th sample extracted by the Long Short-Term Memory Network, h2 represent the spatial features of the i-th sample extracted by the Long Short-Term Memory Network, and h3 represent the spatial features of the i-th sample extracted by the Convolutional Neural Network.

[0063] According to an embodiment of the present invention, in step S23, the present invention obtains the latent feature vector corresponding to each sample in the following manner: z x =E x (x), where z x Let E represent the latent feature vector of the sample, x represent the spatiotemporal features of the sample, and E represent the latent feature vector of the sample. x E represents the feature encoder. x (x) indicates that x is used as the input to the feature encoder. According to one embodiment of the present invention, the predicted features corresponding to each sample are obtained in the following manner: in, z represents the predicted feature corresponding to the sample. x D represents the latent feature vector of a sample. x D represents the feature decoder. x (z x ) indicates that z x As input to the feature decoder.

[0064] According to an embodiment of the present invention, in step S24, the present invention obtains the cross-prediction attribute corresponding to each sample in the following manner: in, z represents the cross-prediction attribute corresponding to the sample. x D represents the latent feature vector of a sample. S D represents the attribute decoder. S (z x ) indicates that z x As input to the attribute decoder. According to one embodiment of the present invention, the cross-prediction features corresponding to each sample are obtained in the following manner: in, z represents the cross-prediction feature corresponding to the sample. a D represents the latent vector of attributes of a sample. xD represents the feature decoder. x (z a ) indicates that z a As the input to the feature decoder. Since the use of a classifier to classify and predict samples in this invention is a well-known technique in the art, it will not be described in detail here.

[0065] According to an embodiment of the present invention, in step S25, the attribute prediction loss is calculated as follows:

[0066]

[0067] Where 'a' represents the behavioral attribute corresponding to the behavioral category to which the sample belongs, and 'z'... a Represents the latent vector of attributes of a sample. Indicate z a The approximate inference function of the posterior distribution corresponding to a, p(z) a ) represents z a The Gaussian distribution function is given by E, where E represents the expected value and KL(·) represents the divergence function.

[0068] According to one embodiment of the present invention, the feature prediction loss is calculated as follows:

[0069]

[0070] Where x represents the spatiotemporal feature corresponding to the sample, z x Represents the latent feature vector of a sample. Indicate z x The approximate inference function of the posterior distribution corresponding to x, p(z) x ) represents z x The Gaussian distribution function is given by E, where E represents the expected value and KL(·) represents the divergence function.

[0071] According to an embodiment of the present invention, in step S25, the classification prediction loss is calculated as follows:

[0072]

[0073] in, The latent feature vector z of the sample x Corresponding feature labels The latent feature vector z of the sample x The corresponding classification prediction results, The latent vector z of the sample represents the attribute. a The corresponding attribute tags, The latent vector z of the sample represents the attribute. a The corresponding classification prediction results.

[0074] According to one embodiment of the present invention, in step S25, the cross-alignment loss is calculated as follows:

[0075]

[0076] Where 'a' represents the behavioral attribute corresponding to the behavioral category to which the sample belongs, 'x' represents the spatiotemporal feature corresponding to the sample, and 'D' represents the behavior attribute corresponding to the behavioral category to which the sample belongs. s (z x ) represents the attribute decoder D s Based on the latent feature vector z of the sample x The obtained cross-prediction attribute, D x (z a ) represents the feature decoder D x Based on the latent attribute vector z of the sample a The obtained cross-prediction features, where E represents the mathematical expectation.

[0077] According to an embodiment of the present invention, in step S25, the contrast alignment loss is calculated as follows:

[0078]

[0079] Where M represents the number of samples in the training set, y i Let y represent the label of the i-th sample. j Let y represent the label of the j-th sample. k Let c represent the label of the k-th sample. i，j Let c represent the cosine similarity between the i-th sample and the j-th sample. i，k Represents the cosine similarity between the i-th sample and the k-th sample, 1 i≠j The value is 1 when the i-th sample and the j-th sample are different, and 0 when they are the same. The value is 1 when the label of the i-th sample is different from that of the j-th sample, and 0 when they are the same. t represents the hyperparameter, and E represents the expected value.

[0080] According to one embodiment of the present invention, in step S25, the present invention calculates the comprehensive loss in the following manner:

[0081]

[0082] in, This represents the attribute prediction loss. This represents the feature prediction loss. Indicates the cross-alignment loss. Indicates the contrast alignment loss. λ1, λ2, and λ3 represent the classification prediction loss, and λ1, λ2, and λ3 are all preset parameters.

[0083] In step S3, the feature extraction network, attribute decoder, feature encoder, classifier, and the introduced behavior category judgment module and invisible classifier in the basic behavior recognition model trained to convergence in step S2 are used to construct the final behavior recognition model.

[0084] When the behavior recognition model constructed by the method described in the above embodiments is used to identify the behavior category corresponding to wearable device data, the feature extraction network is used to extract the spatiotemporal features of the wearable device data, the feature encoder is used to encode the spatiotemporal features extracted by the feature extraction network to obtain the feature latent vector of the wearable device data, and the attribute decoder is used to decode the feature latent vector obtained by the feature encoder to obtain the cross-prediction attribute of the wearable device data; the classifier is used to perform behavior category classification prediction based on the corresponding feature latent vector of wearable device data that belongs to the known behavior category seen in the training set during model training after being judged by the behavior category judgment module; the behavior category judgment module is used to determine whether the wearable device data belongs to the known behavior category seen in the training set during model training or the known behavior category not seen in the test set during model training based on the distance between the feature latent vector of the wearable device data and the behavior attributes corresponding to all behavior categories seen during model training; the invisible classifier is used to perform behavior category judgment based on the corresponding cross-prediction attribute of wearable device data that belongs to the known behavior category not seen in the test set during model training after being judged by the behavior category judgment module.

[0085] It should be noted that before applying the behavior recognition model constructed by this invention, this invention calculates the manifold centers and boundaries of all behavior categories in the training set (which is also the set of seen categories) to help the behavior category judgment module better distinguish whether the behavior category corresponding to the wearable device data is a seen category or an unseen category during application. According to one embodiment of the invention, this invention encodes the behavior attributes corresponding to the behavior categories of all samples in the training set into the latent space using the converged basic behavior recognition model to obtain the attribute latent vectors corresponding to each behavior category in the training set. The attribute latent vectors corresponding to each behavior category are then used as the manifold centers of each behavior category. Then, the cosine similarity between the feature latent vectors corresponding to all samples in each behavior category and the manifold centers of that behavior category is calculated to determine the popular boundary of each behavior category based on the obtained cosine similarity between all samples in each behavior category. According to one embodiment of the invention, this invention determines the popular boundary of each behavior category in the following manner: First, a similarity set is formed by combining the feature latent vectors of all samples in the training set with the cosine similarity between them and the manifold centers of their corresponding behavior categories. Among them, C iLet S represent the set of cosine similarities between all samples in the i-th behavior category in the training set and the manifold center of that behavior category, and let S represent the number of all behavior categories in the training set; then, for The cosine similarity sets for each behavior category are sorted in descending order according to the magnitude of the cosine similarity, and then... Located in the |C i The cosine similarity at position |×γ serves as the manifold boundary for each behavior category, where |C i | represents the number of cosine similarities in the i-th behavior category in the training set, and γ represents a preset parameter with a value of 0.95. It should be noted that the value of γ can be set according to the actual scenario, and this invention does not impose specific limitations on it.

[0086] According to one embodiment of the present invention, the behavior category determination module is configured to determine whether the wearable device data belongs to a known behavior category seen in the training set during model training in the following manner:

[0087]

[0088]

[0089] Among them, z te Represents the latent feature vector of wearable device data. This represents the set of manifold centers corresponding to all known behavior categories seen in the training set during model training. Indicates the calculation of z te and The cosine similarity between the manifold centers of each known behavior category is denoted by max(·), where max(·) represents the function that takes the maximum value. `seen` represents the known behavior categories seen in the training set during model training, and `unseen` represents the known behavior categories not seen in the test set during model training. `η` refers to the popular boundary corresponding to each known behavior category seen in the training set during model training. This indicates that the wearable device data belongs to the known behavior categories seen in the training set during model training. This represents known behavior categories in the test set that were not seen during model training.

[0090] According to one embodiment of the present invention, the invisible classifier is configured to determine the behavior category of wearable device data belonging to known behavior categories in the test set that have not been seen during model training, as determined by the behavior category determination module:

[0091]

[0092] in, This indicates that the wearable device data belongs to a known behavior category in the test set that was not seen during model training. This represents the cross-prediction attribute of wearable device data. This represents the set of behavioral attributes corresponding to all known behavioral categories in the test set that were not seen during model training. Indicates calculation and The cosine similarity between the behavioral attributes corresponding to each known behavioral category, where argmax(·) represents the selection. and The known behavior categories corresponding to the maximum cosine similarity between them This represents the classification prediction results for wearable device data belonging to known behavioral categories in the test set that were not seen during model training.

[0093] In summary, such as Figure 4 As shown, when using a behavior recognition model to identify user behavior, the behavior category of the user behavior is obtained through the following steps: First, the wearable device worn by the user obtains wearable device data corresponding to the user behavior; second, the feature extraction network extracts the spatiotemporal features corresponding to the wearable device data; then, the feature encoder extracts the latent feature vector corresponding to the wearable device data based on the spatiotemporal features; finally, the behavior category judgment module determines whether the wearable device data belongs to the seen class set or the unseen class set based on the latent feature vector and the manifold centers and boundaries of all behavior categories in the seen class set. When the wearable device data belongs to the seen class set, the classifier performs classification prediction on the wearable device data belonging to the seen class set to obtain the classification probability (e.g., ...). Figure 4 The values ​​shown are 0.11, 0.05, 0.20, and 0.13, and the classification result is obtained based on the classification probability. When the wearable device data belongs to the unseen class set, the semantic decoder will decode the feature latent vectors corresponding to the wearable device data belonging to the unseen class set to obtain the cross-prediction attribute, and obtain the classification result based on the cross-prediction attribute through the unseen classifier.

[0094] Compared with the prior art, the advantages of the present invention are as follows:

[0095] 1. Based on the basic behavior recognition model trained to convergence, this invention constructs a new behavior recognition model by introducing a behavior category judgment module and an invisible classifier. Based on the behavior attributes corresponding to the behavior categories in the unseen category set, it can recognize not only the behavior categories in the seen category set but also the behavior categories in the unseen category set without training an additional model.

[0096] 2. In the training process, this invention improves the effectiveness of the behavior recognition model in recognizing user behavior by introducing corresponding behavior attributes for all known behavior categories and by jointly training the features of the samples in the training set and the behavior attributes corresponding to the behavior categories to which the samples belong.

[0097] 3. In the training process, this invention constructs new attribute prediction loss, feature prediction loss, cross alignment loss and contrast alignment loss, which realizes the alignment of the two data modalities of behavioral attributes and features in the latent space, and enables the behavior recognition model constructed by this invention to effectively distinguish whether the data to be identified belongs to the seen class set or the unseen class set during the application process.

[0098] It should be noted that although the steps are described in a specific order above, it does not mean that the steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently, or even in a different order, as long as the required function can be achieved.

[0099] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0100] Computer-readable storage media can be tangible devices that hold and store instructions for use by an instruction execution device. Computer-readable storage media can be, for example, including but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof.

[0101] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for constructing a behavior recognition model based on wearable device data, the method comprising: The method comprises the following steps: S1, obtaining data of a wearable device and preprocessing to obtain a training set, wherein the training set comprises part of behavior categories in known behavior categories, each behavior category corresponds to a set of behavior attributes, and each behavior category comprises a plurality of samples with behavior category labels; S2, training a basic behavior recognition model to convergence using the training set, wherein the basic behavior recognition model comprises a feature extraction network, an attribute branch network, a feature branch network, and a classifier, the attribute branch network comprises an attribute encoder and an attribute decoder, the feature branch network comprises a feature encoder and a feature decoder, and the following steps are performed in the training process: S21, using the attribute encoder in the attribute branch network to extract an attribute latent vector of each sample based on the behavior attributes corresponding to the behavior category to which each sample belongs, and using the attribute decoder to obtain predicted behavior attributes corresponding to each sample based on the attribute latent vector of each sample; S22, using the feature extraction network to extract the spatio-temporal features of each sample in the training set; S23, using the feature encoder in the feature branch network to extract a feature latent vector of each sample based on the spatio-temporal features of each sample, and using the feature decoder to obtain predicted features corresponding to each sample based on the feature latent vector of each sample; S24, using the classifier to obtain a behavior category classification prediction result corresponding to each sample based on the feature latent vector of each sample, using the attribute decoder to obtain cross-predicted attributes of each sample based on the feature latent vector of each sample, and using the feature decoder to obtain cross-predicted features of each sample based on the attribute latent vector of each sample; S25, calculating an attribute prediction loss based on the behavior attributes corresponding to the behavior categories of all samples and the predicted behavior attributes of all samples, calculating a feature prediction loss based on the spatio-temporal features and the predicted features of all samples, calculating a classification prediction loss based on the behavior category classification prediction results of all samples and the behavior category labels corresponding to all samples, and calculating a cross-alignment loss based on the behavior attributes, the cross-predicted attributes, the spatio-temporal features, and the cross-predicted features of all samples; S26, updating the parameters of the basic behavior recognition model based on the comprehensive loss of all losses calculated in step S25; S3, constructing a behavior category judgment module and an invisible classifier to construct a behavior recognition model using the feature extraction network, the attribute decoder, the feature encoder, the classifier, and the behavior category judgment module and the invisible classifier in the basic behavior recognition model trained to convergence in step S2, wherein: The behavior category judgment module is used to determine whether the wearable device data belongs to a known behavior category in the training set seen in the model training process or a known behavior category in a test set not seen in the model training process according to the distance between the feature latent vector of the wearable device data and the behavior attributes corresponding to all behavior categories seen in the model training process. An unseen classifier is configured to determine the behavior category of the wearable device data belonging to the known behavior category in the test set that is not seen in the model training process based on the corresponding cross-predicted attribute.

2. The method of claim 1, wherein, The step S1 includes: S11, obtaining historical time series data recorded by a plurality of wearable devices, wherein the historical time series data recorded by all wearable devices corresponds to continuous behavior actions composed of a series of known behavior categories; S12, truncating all historical time series data recorded by wearable devices obtained in step S11 using a preset sliding time window to obtain a plurality of samples, each sample containing data of all wearable devices in the same window time period, and taking the behavior being performed at the last moment in the window time period as the behavior category label of the sample corresponding to the window time period; S13, grouping all samples obtained in step S12 into a data set, and dividing the data set into a training set and a test set according to a preset rule, wherein the training set is used to train a basic behavior recognition model, the test set is used to test whether the accuracy of the basic behavior recognition model on the test set reaches a preset accuracy requirement, and the training set includes samples corresponding to part of the behavior categories in the known behavior categories, and the test set includes different samples of the same behavior categories as those in the training set and samples corresponding to behavior categories different from those in the training set.

3. The method of claim 2, wherein, The feature extraction network includes a long short-term memory network and a convolutional neural network.

4. The method of claim 3, wherein, In the step S22, the feature extraction network extracts the spatio-temporal features of each sample in the training set in the following manner: The long short-term memory network is used to extract the time series features of each sample, and the long short-term memory network and the convolutional neural network are used to extract the spatial features of each sample, respectively; Based on the time series features and the spatial features of each sample, the spatio-temporal features of each sample are determined, respectively.

5. The method of claim 4, wherein, The spatio-temporal features of each sample are determined in the following manner: in, Indicates the first training set The spatiotemporal characteristics of each sample Represents the mathematical expectation. This indicates the first [item] retrieved from the Long Short-Term Memory network. Temporal features of each sample, This indicates the first [item] retrieved from the Long Short-Term Memory network. Spatial features of a sample This represents the first extraction by the convolutional neural network. Spatial characteristics of each sample.

6. The method of claim 5, wherein, In the step S25, the attribute prediction loss is calculated in the following manner: wherein, represents a behavior attribute corresponding to a behavior category to which the sample belongs, represents an attribute latent vector of the sample, represents and an approximate inference function of a posterior distribution corresponding thereto, represents a Gaussian distribution function of represents a mathematical expectation, represents a divergence function.

7. The method of claim 6, wherein, In the step S25, the feature prediction loss is calculated in the following manner: wherein, denotes a spatio-temporal feature corresponding to a sample, denotes a feature latent vector of a sample, denotes and an approximate inference function of the corresponding posterior distribution, denotes a Gaussian distribution function of denotes a mathematical expectation, denotes a divergence function.

8. The method of claim 7, wherein, In the step S25, the classification prediction loss is calculated in the following manner: wherein, a feature latent vector representing a sample a corresponding feature label, a feature latent vector representing a sample a corresponding classification prediction, an attribute latent vector representing a sample a corresponding attribute label, an attribute latent vector representing a sample a corresponding classification prediction.

9. The method of claim 8, wherein, In the step S25, the cross-alignment loss is calculated in the following manner: wherein, denotes a behavior attribute corresponding to a behavior class to which the sample belongs, denotes a spatio-temporal feature corresponding to the sample, denotes an attribute decoder based on a feature latent vector of the sample obtained cross-predicted attribute, denotes a feature decoder based on an attribute latent vector of the sample obtained cross-predicted feature, denotes a mathematical expectation.

10. The method of claim 9, wherein, In the step S25, the contrast alignment loss is calculated in the following manner based on the attribute latent vector and the feature latent vector of all samples: wherein, denotes the number of samples in the training set, denotes the label of the th sample, denotes the label of the th sample, denotes the label of the th sample, denotes the cosine similarity between the th sample and the th sample, denotes the cosine similarity between the th sample and the th sample, denotes the indicator function of the th sample and the th sample, which takes the value 1 if they are different and 0 if they are the same, denotes the indicator function of the label of the th sample and the label of the th sample, which takes the value 1 if they are different and 0 if they are the same, denotes a hyperparameter, denotes the mathematical expectation.

11. The method of claim 10, wherein, In the step S26, the comprehensive loss is calculated in the following manner: wherein, represents an attribute prediction loss, represents a feature prediction loss, represents a cross-alignment loss, represents a contrast-alignment loss, represents a classification prediction loss, 、 、 all represent preset parameters.

12. A behavior recognition model based on the method of any one of claims 1-11, for identifying a behavior category corresponding to wearable device data, characterized in that, The model includes: A feature extraction network is configured to extract the spatio-temporal features of the wearable device data; A feature encoder is configured to encode the spatio-temporal features extracted by the feature extraction network to obtain the feature latent vector of the wearable device data; An attribute decoder is configured to decode the feature latent vector obtained by the feature encoder to obtain the cross-predicted attribute of the wearable device data; A classifier is configured to classify the wearable device data belonging to the known behavior category in the training set that has been seen in the model training process based on the corresponding feature latent vector. a behavior category judgment module, configured to judge whether the wearable device data belongs to a known behavior category in a training set seen in a model training process or a known behavior category in a test set not seen in the model training process according to a feature latent vector of the wearable device data and distances between behavior attributes corresponding to all behavior categories seen in the model training process; an unseen classifier, configured to judge a behavior category of the wearable device data belonging to the known behavior category in the test set not seen in the model training process based on corresponding cross-prediction attributes thereof after being judged by the behavior category judgment module.

13. The behavior recognition model of claim 12, wherein, The behavior category judgment module is configured to judge whether the wearable device data belongs to the known behavior category in the training set seen in the model training process in the following manner: ， ， wherein, represents a feature latent vector of the wearable device data, represents a set of manifold centers corresponding to all known behavior classes in the training set seen during the model training process, represents a computation represents a cosine similarity between a manifold center of each known behavior class in the set, represents a function that takes a maximum value, represents a known behavior class in the training set seen during the model training process, represents a known behavior class in the test set not seen during the model training process, generically refers to a popular boundary corresponding to each known behavior class in the training set seen during the model training process, represents that the wearable device data belongs to a known behavior class in the training set seen during the model training process, represents a known behavior class in the test set not seen during the model training process.

14. The behavior recognition model of claim 13, wherein, The unseen classifier is configured to judge a behavior category of the wearable device data belonging to the known behavior category in the test set in the following manner after being judged by the behavior category judgment module: wherein, represents that the wearable device data belongs to a known behavior category in a test set that is unseen during the model training process, represents a cross-predicted attribute of the wearable device data, represents a set of behavior attributes corresponding to all known behavior categories in a test set that is unseen during the model training process, represents computing a cosine similarity between each known behavior category and a behavior attribute corresponding to each known behavior category, represents selecting a known behavior category corresponding to the maximum cosine similarity between and a behavior attribute corresponding to each known behavior category, represents a classification prediction result of the wearable device data belonging to a known behavior category in a test set that is unseen during the model training process.

15. A behavior recognition method based on the behavior recognition model according to any one of claims 12-14, characterized by, The method comprises the following steps: T1, acquiring wearable device data to be processed; T2, processing the wearable device data to be processed by using the behavior recognition model to recognize a behavior category thereof.

16. A computer-readable storage medium, characterized in that, A computer program is stored thereon, and the computer program can be executed by a processor to implement steps of the method of any one of claims 1-11 and 15.

17. An electronic device, comprising: comprise: one or more processors; a storage device, configured to store one or more programs, which, when executed by the one or more processors, cause the electronic device to implement steps of the method of any one of claims 1-11 and 15.

Images