An open-set human activity recognition method based on two-stage feature alignment

Abstract

The application discloses an open set human activity recognition method based on two-stage feature alignment, first extracts global features of source domain and target domain sensor data, performs global feature alignment, then divides source domain global features and target domain global features into source domain instance features and target domain instance features, pre-trains a category division module using the source domain instance features, then inputs the target domain instance features into the category division module, judges whether the target domain instance features are common class instance features or private class instance features, and then performs instance feature alignment of the common class. For the target domain private class instance features, private class constraint is performed. The application also uses a discrete time sequence self-attention module, mines key time sequence information in the instance features through a self-attention mechanism, extracts time sequence features with stronger migration through local feature fusion and weighting, and improves the accuracy and robustness of open set activity recognition.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of human activity recognition, and particularly relates to an open set human activity recognition method based on two-stage feature alignment. BACKGROUND

[0002] With the rapid development of video technology, smart home, robot interaction and other fields, human activity recognition as a key technology has wide application value in health monitoring, intelligent security, sports analysis and other aspects. Traditional activity recognition methods usually assume that the label space of source data and target data is consistent, that is, all behavior categories to be recognized are known in the training stage. However, in real-world scenarios, human activity patterns have high diversity and unpredictability. Different users may include personalized unknown actions or intermediate states when performing the same complex activity, making it difficult for traditional closed set assumptions to be applicable. To address this challenge, open set activity recognition has gradually become a research hotspot. Unlike traditional methods that can only recognize known categories in the training set, open set activity recognition requires the model not only to accurately recognize known behaviors, but also to have the ability to detect and handle unknown behaviors, thereby improving the generalization ability and robustness of the system in dynamic environments.

[0003] Existing open set activity recognition methods usually include the following steps: first, classifying classes in the source domain and the target domain, dividing the classes into public classes and private classes. Then, remove the private classes and only keep the public classes. Finally, align the features of the public classes to convert the open set problem into a closed set problem. In the process of aligning the features of the public classes, traditional methods usually assume that the contributions of global features and instance features to feature alignment are the same. However, in actual situations, the different proportions of private classes in sensor data have different effects on the alignment of global features and instance features. And in the open set human activity recognition task, not all source domain data has a positive transfer effect on the target domain. The data distribution of the source domain and the target domain may be significantly different, and it is difficult to verify whether the features of the source domain and the target domain share similar feature distributions in actual applications. When the correlation between the source domain data and the target domain data is low or even conflicting, directly transferring the source domain features may introduce irrelevant information or noise, thereby affecting the recognition performance of the model. If the source domain data contains abnormal samples or potential malicious information, it may further exacerbate the generalization error of the model on the target domain. Therefore, in the open set human activity recognition task, how to effectively filter and use the source domain information to minimize the negative transfer effect is a challenge that needs to be solved. SUMMARY

[0004] The application aims to provide an open set human activity recognition method based on two-stage feature alignment. In the first stage, the global features of the source domain and the target domain are aligned, and in the second stage, the instance features in the common categories of the source domain and the target domain are aligned. In the instance feature alignment, the discrete temporal self-attention module is used to suppress the interference of non-transferable features on the model training, thereby reducing the negative transfer risk and improving the robustness and accuracy of the open set human activity recognition.

[0005] To achieve the above-mentioned purpose, the technical scheme of the application is as follows:

[0006] An open set human activity recognition method based on two-stage feature alignment, which adopts source domain sensor data and target domain sensor data to train a constructed human activity recognition network model, and then uses the trained human activity recognition network model for human activity recognition, comprising:

[0007] Input the source domain sensor data sample and the target domain sensor data sample into a feature extraction module to extract source domain global features and target domain global features;

[0008] Input the source domain global features and the target domain global features into a global feature alignment discriminator respectively, and calculate the global feature alignment loss through adversarial learning;

[0009] Divide the source domain global features and the target domain global features into source domain instance features and target domain instance features, pre-train a category division module using the source domain instance features, and then input the target domain instance features into the category division module to determine whether the target domain instance features are common category instance features or private category instance features;

[0010] Pre-train a discrete temporal self-attention module using the source domain instance features, and freeze the parameters of the discrete temporal self-attention module, input the common category instance features into the discrete temporal self-attention module to obtain transferable features, and calculate the instance feature alignment loss through adversarial learning;

[0011] Calculate the feature center and the probability center of the target domain private category instance features, and then calculate the deviation of each target domain private category instance feature relative to the feature center and the deviation relative to the probability center through the cosine similarity distance, and calculate the private category constraint loss;

[0012] Jointly calculate the global feature alignment discriminator loss, the common category instance alignment loss and the private category constraint loss, and update the parameters of the human activity recognition network model;

[0013] Input the sensor data to be recognized into the trained human activity recognition network model to obtain the recognition result.

[0014] As preferred, the source domain global feature and the target domain global feature are respectively input into the global feature alignment discriminator, and a global feature alignment loss is calculated through adversarial learning, including:

[0015] The source domain global feature and the target domain global feature are respectively input into the global feature alignment discriminator to obtain a global feature alignment discriminator output;

[0016] The global feature alignment loss is calculated by the following formula :

[0017]

[0018] Wherein, The global feature alignment discriminator output corresponding to the source domain global feature is represented as The global feature alignment discriminator output corresponding to the target domain global feature is represented as The expected value corresponding to the source domain global feature is represented as The expected value corresponding to the target domain global feature is represented as

[0019] As preferred, the source domain instance feature is used to pretrain the category division module, and then the target domain instance feature is input into the category division module to determine whether the target domain instance feature is a public class instance feature or a private class instance feature, including:

[0020] The classifier of the category division module is pre-trained using the source domain instance feature, and the parameter matrix of the classifier of the category division module is singular value decomposed;

[0021] The right singular matrix obtained by singular value decomposition is divided to obtain a source domain known category feature subspace and a source domain unknown category feature subspace;

[0022] The target domain instance feature is projected on the source domain known category feature subspace and the source domain unknown category feature subspace to obtain corresponding known features and unknown features;

[0023] The Euclidean norm corresponding to the unknown features obtained by projection is calculated, and then compared with a distribution threshold to determine whether the target domain instance feature is a public class instance feature or a private class instance feature.

[0024] As preferred, the discrete time sequence self-attention module includes time discrete sampling layers, and the public class instance feature is input into the discrete time sequence self-attention module to obtain a migratable feature, including:

[0025] For the i-th time discrete sampling layer, the instance feature is divided into ​a local instance feature, the local instance feature being fused by a multi-layer perception to obtain a local discrete feature;

[0026] performing class prediction on the local discrete feature through an auxiliary classifier to obtain a class prediction result;

[0027] calculating an entropy of the class prediction, and then obtaining a weight of the local discrete feature based on the entropy of the class prediction;

[0028] performing weighted summation on all the local discrete features to obtain a transferable feature.

[0029] Preferably, the instance feature alignment loss calculated through the adversarial learning comprises:

[0030] inputting the transferable feature corresponding to the source domain instance feature into a human activity recognition classifier, and calculating a human activity recognition classification loss;

[0031] calculating a weight of the source domain instance feature with respect to the target domain instance feature;

[0032] inputting the transferable feature corresponding to the source domain instance feature and the transferable feature corresponding to the target domain instance feature into an instance feature alignment discriminator, and calculating an instance feature alignment discriminator loss corresponding to each target domain instance feature;

[0033] combining the human activity recognition classification loss and all the instance feature alignment discriminator losses to obtain the instance feature alignment loss.

[0034] Preferably, the private class constraint loss is represented by the following formula:

[0035]

[0036] wherein, represents the private class constraint loss, is the number of target domain private class instance feature samples, represents a deviation with respect to a feature center, represents a deviation with respect to a probability center.

[0037] The open set human activity recognition method based on two-stage feature alignment provided in the present application uses a two-stage feature alignment framework in the process of open set activity recognition. In the first stage, the global feature is aligned to preliminarily eliminate the distribution difference between domains. In the second stage, the instance features of public classes are further aligned to narrow the distribution between domains. The discrete time sequential self-attention module is used to mine the key time sequence information in the instance feature through the self-attention mechanism, and the time sequence feature with stronger transferability is extracted through local feature fusion and weighting, thereby improving the accuracy and robustness of open set activity recognition. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 A flow chart of an open-set human activity recognition method based on two-stage feature alignment of the present application.

[0039] Figure 2 A schematic diagram of a network model structure for training of the present application.

[0040] Figure 3 A schematic diagram of a feature extraction module structure of an embodiment of the present application.

[0041] Figure 4 A schematic diagram of a human activity recognition network model structure of the present application. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0043] One embodiment of the present application, as shown in Figure 1 , provides an open-set human activity recognition method based on two-stage feature alignment, which adopts source domain sensor data and target domain sensor data to train a human activity recognition network model constructed, and then uses the trained human activity recognition network model to recognize human activities in the target domain. The open-set human activity recognition method based on two-stage feature alignment comprises the following steps.

[0044] Step S1, inputting source domain sensor data samples and target domain sensor data samples into a feature extraction module to extract source domain global features and target domain global features.

[0045] Steps S1-S6 of the present embodiment describe the training process, Figure 2 is a network model structure diagram corresponding to the training process, and two-stage adversarial learning is performed, i.e., global feature alignment stage and instance feature alignment stage, and respective discriminators are used in the adversarial learning, i.e., global feature alignment discriminator and instance feature alignment discriminator.

[0046] In the present embodiment, the original sensor data samples are denoted as , the source domain original sensor data samples are denoted as , and the target domain original sensor data samples are denoted as . The source domain original sensor data samples and the target domain original sensor data samples are subjected to a feature extraction module to obtain source domain global features and target domain global features .

[0047] The feature extraction network is used in the present application to extract a representative feature vector from the original data. The design of the network can be selected according to the specific application scenario and data characteristics. For example, convolutional neural network, recurrent neural network, etc.

[0048] In the present embodiment, a space-time-frequency domain feature extraction network is used as the feature extraction network, which can effectively capture the space-time-frequency domain features in the sensor data. As shown in Figure 3 The feature extraction module includes a time feature extraction module, a space feature extraction module and a frequency domain feature extraction module. The time feature extraction module, the space feature extraction module and the frequency domain feature extraction module are all composed of a plurality of feature extraction layers. The sensor original data is first passed through the Transformer encoder to obtain high-dimensional features. For the time feature extraction module, the high-dimensional features are passed through the time attention module and the feedforward network, and then connected in residual connection with the features of the previous layer to obtain the time features; for the space feature extraction module, the high-dimensional features are passed through the space attention module and the feedforward network, and then connected in residual connection with the features of the previous layer to obtain the space features; for the frequency domain feature extraction module, the high-dimensional features are first passed through the discrete Fourier transform, and then passed through the frequency domain attention module and the feedforward network, and then connected in residual connection with the features of the previous layer to obtain the frequency domain features. The time features, the space features and the frequency domain features are spliced to obtain the global features corresponding to the sensor data.

[0049] Step S2, input the source domain global features and the target domain global features into the global feature alignment discriminator respectively, and calculate the global feature alignment loss through adversarial learning.

[0050] This step performs global feature alignment, i.e. the first stage feature alignment, the source domain global features and the target domain global features are aligned through the global feature alignment discriminator in adversarial learning.

[0051] The target of the global feature alignment discriminator is to distinguish whether the global features come from the source domain or the target domain, which can be specifically expressed as the following formula:

[0052] (1)

[0053] wherein, is a sigmoid activation function, which maps the output of the global feature alignment discriminator to 0 to 1, indicating the probability that the input features belong to the target domain. is the weight parameter matrix of the global feature alignment discriminator, is the bias term.

[0054] The global feature alignment loss function is shown as follows:

[0055] (2)

[0056] wherein, represents an expected value. By minimizing , the discriminator can gradually improve the distinguishing ability of the target domain features. This adversarial learning mechanism can effectively reduce the distribution difference between the source domain and the target domain, and improve the generalization performance of the model on the target domain. represents the global feature alignment discriminator output corresponding to the source domain global feature, represents the global feature alignment discriminator output corresponding to the target domain global feature, represents the expected value corresponding to the source domain global feature, represents the expected value corresponding to the target domain global feature.

[0057] Step S3, the source domain global feature and the target domain global feature are divided into source domain instance features and target domain instance features, the source domain instance features are used to pretrain the category division module, and then the target domain instance features are input into the category division module to determine whether the target domain instance features are public class instance features or private class instance features.

[0058] In this embodiment, the global features are divided by a sliding time window to extract instance features with finer granularity. The source domain global features and the target domain global features are divided by a sliding window to obtain source domain instance features and target domain instance features .

[0059] In this embodiment, the sliding window size is set to 200, and the overlap rate is set to 50%. In this way, it can ensure that each instance sample contains sufficient behavior information, and also ensure the continuity between instance samples. Through this segmentation method, local dynamic changes in time series data can be effectively captured, while avoiding information loss caused by too sparse window division.

[0060] Traditional methods usually separate public class features and private class features by manually setting threshold standards or introducing time-consuming and unstable clustering strategies. However, these methods often lack robustness and are difficult to adapt to complex data distribution. This application divides public classes and private classes from the perspective of feature decomposition.

[0061] In one specific embodiment, the use of source domain instance features to pretrain the category division module, and then inputting the target domain instance features into the category division module to determine whether the target domain instance features are public class instance features or private class instance features, includes:

[0062] Step 3.1, pre-training the classifier of the category division module using the source domain instance feature, and singular value decomposition is performed on the parameter matrix of the classifier of the category division module.

[0063] This embodiment only pre-trains the classifier of the category division module using the source domain instance feature , and freezes the parameters of the classifier , and the parameter matrix of the classifier is , where is the number of known categories, is the dimension of the feature, and singular value decomposition (SVD) is performed on the weight matrix , and the following formula can be obtained:

[0064] (3)

[0065] where singular value decomposition decomposes into left singular matrix , diagonal matrix , and right singular matrix .

[0066] Step 3.2, dividing the right singular matrix obtained by singular value decomposition to obtain the source domain known category feature subspace and the source domain unknown category feature subspace.

[0067] The right singular matrix obtained by singular value decomposition is divided, and the formula is as follows:

[0068] (4)

[0069] (5)

[0070] where span represents that the matrix is divided by column, and the column vector of the right singular matrix constitutes a set of orthogonal bases of the feature space. represents the source domain known category feature subspace, which is composed of the first feature tensors of the right singular matrix , and represents the main representation space of the model for the known category in the training process. represents the source domain unknown category feature subspace, which is composed of the remaining dimension feature vectors of the right singular matrix , and represents the data distribution of the unknown category that may appear.

[0071] Step 3.3, project the target domain instance feature on the source domain known class feature subspace and the source domain unknown class feature subspace to obtain corresponding known features and unknown features.

[0072] Target domain instance feature In projecting obtains corresponding known features In projecting obtains corresponding unknown features The corresponding formula is as follows:

[0073] (6)

[0074] (7)

[0075] (8)

[0076] Wherein, is the projection of the target domain instance feature on , that is, the known features obtained by projecting; is the projection of the target domain instance feature on , that is, the unknown features obtained by projecting; denotes the projection coefficient of the target domain instance feature on the orthogonal basis .

[0077] Step 3.4, calculate the Euclidean norm corresponding to the unknown features obtained by projecting, and then compare it with the distribution threshold to determine whether the target domain instance feature is a public class instance feature or a private class instance feature.

[0078] In order to further distinguish the public class instance feature and the private class instance feature, in the embodiment, the Euclidean norm of the target domain instance feature is modeled by using a Gaussian mixture model (GMM), which can be expressed as the following formula:

[0079] (9)

[0080] Wherein, denotes the Euclidean norm of the target domain instance feature, which is used to measure the intensity of the target data in the source domain unknown space .The smaller the value is, the smaller the component of the feature of the sample in the source domain unknown space is, and the more likely the sample is a public class feature. On the contrary, the larger the value is, the larger the component of the sample in the unknown space is, and the more likely the sample is a private class feature. ​​​represents a Gaussian distribution. and are the mixing coefficients of the public feature distribution and the private feature distribution respectively, satisfying . Thus, the public feature distribution and the private feature distribution . and respectively represent the mean of the public feature distribution and the mean of the private feature distribution, and respectively represent the standard deviation of the public feature distribution and the standard deviation of the private feature distribution.

[0081] The embodiment uses as the dividing line between the public class and the private class, i.e. the distribution threshold, when is less than , the target domain instance feature is determined as the public class instance feature; when is greater than , the target domain instance feature is determined as the private class instance feature.

[0082] Through this strategy based on feature decomposition and Gaussian mixture model, the separation of public class instance features and private class instance features can be effectively realized, providing a reliable feature representation basis for subsequent cross-domain adaptation tasks.

[0083] Step S4, pre-training the discrete time self-attention module using the source domain instance feature, and freezing the parameters of the discrete time self-attention module, inputting the public class instance feature into the discrete time self-attention module to obtain the transferable feature, and calculating the instance feature alignment loss through adversarial learning.

[0084] In the embodiment, the instance feature is further extracted using the discrete time self-attention module to obtain the transferable feature, and then the feature alignment is performed through adversarial learning.

[0085] The embodiment first pre-trains the discrete time self-attention module using the source domain instance feature , and freezes the parameters of the discrete time self-attention module, which includes time discrete sampling layers.

[0086] In one specific embodiment, the public class instance feature is input into the discrete time self-attention module to obtain the transferable feature, including:

[0087] Step 4.1.1, for the time discrete sampling layer, the instance feature is divided into The local instance features are fused by a multi-layer perception machine to obtain local discrete features.

[0088] In this embodiment, the source domain instance features are all regarded as common class instance features. The source domain instance features are first used to train the discrete time self-attention module. The source domain instance features and the target domain common class instance features are input into the discrete time self-attention module to obtain corresponding transferable features. Taking the target domain instance features as an example, the input target domain instance features belong to the common class instance features.

[0089] The target domain instance features are input into time discrete sampling layers. For the i-th time discrete sampling layer, the target domain instance features are divided into local instance features by sliding segmentation. The local instance features can be represented by the following formula:

[0090] (10)

[0091] Then, the local instance features are fused by a multi-layer perception machine (MLP) to obtain local discrete features , which can be represented by the following formula:

[0092] (11)

[0093] wherein, f represents the multi-layer perception machine for feature fusion, xi represents the i-th local instance feature in the i-th time discrete sampling layer.

[0094] Step 4.1.2, the local discrete features are classified by an auxiliary classifier to obtain a class prediction result.

[0095] The local discrete features are classified by an auxiliary classifier to obtain a class prediction result , which can be represented by the following formula:

[0096] (12)

[0097] wherein, f represents the auxiliary classifier, the loss function of which adopts a cross-entropy loss, and yi represents the class prediction result of the local discrete features xi in the source domain label space. ​​​​​​​​​

[0098] It should be noted that three classifiers are mentioned in the present application, including the classifier of the category division module, the auxiliary classifier of the discrete temporal self-attention module, and the human activity recognition classifier for human activity recognition , and the human activity recognition classifier for human activity recognition. As for the classifier, softmax can be used to realize it, and the following will not be described in detail.

[0099] Step 4.1.3, calculate the entropy of the category prediction, and then obtain the weight of the local discrete feature based on the entropy of the category prediction.

[0100] The category prediction result is passed , the entropy of the category prediction can be calculated to measure the certainty of the model prediction, so as to obtain the weight of the local discrete feature . , the formula is as follows:

[0101] (13)

[0102] (14)

[0103] Wherein, represents the entropy of the category prediction, the smaller the entropy value, the higher the prediction certainty of the model, and the greater the weight. represents the probability of the th activity category, represents the total number of source domain categories. represents the hyperbolic tangent function, which is used to limit the weight in the range of [0, 1] to avoid over amplification or reduction of feature influence.

[0104] Step 4.1.4, weighted sum of all local discrete features is obtained.

[0105] Finally, by weighted sum of all local discrete features , the transferable feature can be obtained, and the formula is as follows:

[0106] (15)

[0107] Similarly, the source domain instance feature is input into the discrete temporal self-attention module, and the corresponding transferable feature can also be obtained.

[0108] In this embodiment, the instance feature alignment loss is calculated by adversarial learning, including:

[0109] Step 4.2.1, input the transferable feature corresponding to the source domain instance feature into the human activity recognition classifier, and calculate the human activity recognition classification loss.

[0110] The calculation formula is as follows:

[0111] (16)

[0112] wherein, represents a human activity recognition classification loss, is the total number of categories, represents the true probability that the activity belongs to the category , and represents the predicted probability that the behavior output by the classifier belongs to the category .

[0113] Step 4.2.2, calculate the weight of the source domain instance feature to the target domain instance feature.

[0114] The weight of the source domain instance feature to the target domain instance feature can be expressed as the following formula:

[0115] (17)

[0116] wherein, the calculation of the weight is the average result of the output of the target domain sample in multiple time discrete layer auxiliary classifiers, which measures the similarity of the target domain instance feature and the source domain instance feature.

[0117] Step 4.2.3, input the transferable feature corresponding to the source domain instance feature and the transferable feature corresponding to the target domain instance feature into the instance feature alignment discriminator, and calculate the instance feature alignment discriminator loss corresponding to each target domain instance feature.

[0118] In this embodiment, the transferable feature corresponding to the source domain instance feature and the transferable feature corresponding to the target domain instance feature are input into the instance feature alignment discriminator, and the instance feature alignment discriminator outputs discrimination results and respectively, and then calculates the instance feature alignment discriminator loss:

[0119] (18)

[0120] wherein, represents the instance feature alignment discriminator loss function corresponding to the target domain instance feature , and represents the number of source domain instance feature samples, represents the instance feature alignment discriminator, represents the source domain transferable feature.

[0121] ​Step 4.2.4. Combine the human activity recognition classification loss and all instance feature alignment discriminator losses to obtain the instance feature alignment loss.

[0122] The formula is as follows:

[0123] (19)

[0124] wherein, denotes the instance feature alignment loss, denotes the number of target domain public class instance feature samples.

[0125] Step S5, calculate the feature center and probability center of the target domain private class instance feature, and then calculate the deviation of each target domain private class instance feature relative to the feature center and the deviation relative to the probability center through the cosine similarity distance, to calculate the private class constraint loss.

[0126] The present application reduces the category confusion and feature distribution deviation caused by private categories in the target domain by constraining the Euclidean distance of private class sample features and category probabilities. The deviation degree of the instance is measured by the Euclidean distance, so as to ensure that the samples of the same category maintain compactness in the feature space. The matching degree of the samples in the target domain is calculated by the cosine similarity, so as to ensure that the category structure is not destroyed in the process of aligning the instance features.

[0127] In the present embodiment, first, the feature center of the target domain instance feature is calculated, and the formula is as follows:

[0128] (20)

[0129] wherein, denotes the feature center of all private class samples in the target domain, denotes the number of target domain private class instance feature samples.

[0130] Subsequently, the probability center of the target domain private class sample is calculated, and the formula is as follows:

[0131] (21)

[0132] (22)

[0133] wherein, is the probability that the target domain instance sample is a private category, denotes the total number of categories, is the probability center of the target domain private class sample, is the number of private class instance feature samples.

[0134] Subsequently, the deviation of the private class instance features in each target domain in the feature space with respect to the feature center and the deviation in the probability space with respect to the probability center are calculated by the cosine similarity distance, respectively The deviation of the sample from the center can be expressed as the following formula:

[0135] (23)

[0136] (24)

[0137] wherein, denotes the cosine similarity distance of the target domain sample in the feature space with respect to the feature center, denotes the cosine similarity distance of the target domain sample in the probability space with respect to the probability center.

[0138] In order to minimize the mean of the intra-domain distance, the loss function of the private class constraint network can be obtained as follows:

[0139] (25)

[0140] wherein, denotes the private class constraint loss, is the number of private class instance feature samples of the target domain, denotes the deviation with respect to the feature center, denotes the deviation with respect to the probability center.

[0141] Step S6, the global feature alignment discriminator loss, the public class instance feature alignment loss and the private class constraint loss are combined to update the parameters of the human activity recognition network model.

[0142] Finally, the total loss function of the present application is:

[0143] (26)

[0144] By minimizing the loss function , the parameters of the human activity recognition network model are updated, and the recognition effect and stability of the network model in the present embodiment are improved.

[0145] Step S7, the sensor data to be identified is input into the trained human activity recognition network model to obtain the recognition result.

[0146] In practical application, the human activity recognition network model is as shown in Figure 4 The sensor data to be identified obtains the global feature through the feature extraction network, and then is segmented into instance features​​​ , judged by the category classification module Is it a public class? If not, output unknown behavior. If it is a public class, instance characteristics Transferable features are extracted through discrete temporal self-attention modules ,Finally, the classification results are obtained through the human activity recognition classifier.

[0147] In this application, the technical solution of this application is also verified through experiments. The experiments are based on the PAMAP2 dataset and the UCI-HAR dataset, and compared with the current mainstream cross-domain behavior recognition algorithms. The accuracy is used to measure the recognition performance. The experimental results are shown in Table 1:

[0148] Table 1

[0149]

[0150] As shown in the experimental data in Table 1, our proposed method significantly outperforms other methods on both the PAMAP2 and UCI-HAR datasets. On the PAMAP2 dataset, our proposed method achieved an accuracy of 84.35%, a 6.5% improvement over the next-best method (Mdk-ResNet); on the UCI-HAR dataset, our proposed method achieved an accuracy of 77.46%, a 1.85% improvement over the next-best method (ResNet+TA). Our proposed method demonstrated significant advantages in open-set activity recognition on both datasets, demonstrating particularly strong robustness and generalization capabilities for highly complex tasks.

[0151] In summary, the proposed method has certain advantages over other algorithms. It can more comprehensively understand the behavior reflected by sensor data and efficiently extract behavioral features from sensor data, thereby significantly improving the accuracy and robustness of cross-domain behavior recognition.

[0152] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. An open set human activity recognition method based on two-stage feature alignment, characterized in that, The two-stage feature alignment-based open-set human activity recognition method trains a human activity recognition network model constructed by source domain sensor data and target domain sensor data, and then uses the trained human activity recognition network model to recognize human activities, including: inputting the source domain sensor data sample and the target domain sensor data sample into a feature extraction module to extract source domain global features and target domain global features; inputting the source domain global features and the target domain global features into a global feature alignment discriminator respectively, and calculating a global feature alignment loss through adversarial learning; dividing the source domain global features and the target domain global features into source domain instance features and target domain instance features, pre-training a category division module using the source domain instance features, and then inputting the target domain instance features into the category division module to determine whether the target domain instance features are common class instance features or private class instance features; pre-training a discrete time sequence self-attention module using the source domain instance features, freezing the parameters of the discrete time sequence self-attention module, inputting the common class instance features into the discrete time sequence self-attention module to obtain transferable features, and calculating an instance feature alignment loss through adversarial learning; calculating a feature center and a probability center of the target domain private class instance features, and then calculating the deviation of each target domain private class instance feature relative to the feature center and the deviation relative to the probability center through cosine similarity distance, to calculate a private class constraint loss; updating the parameters of the human activity recognition network model by combining the global feature alignment discriminator loss, the common class instance alignment loss, and the private class constraint loss; inputting the sensor data to be recognized into the trained human activity recognition network model to obtain a recognition result.

2. The two-stage feature alignment based open-set human activity recognition method of claim 1, wherein, The method includes: inputting the source domain global features and the target domain global features into a global feature alignment discriminator respectively to obtain global feature alignment discriminator outputs; The global feature alignment loss is calculated by the following equation : ； wherein, denotes the global feature alignment discriminator output corresponding to the source domain global feature, denotes the global feature alignment discriminator output corresponding to the target domain global feature, denotes the expected value corresponding to the source domain global feature, denotes the expected value corresponding to the target domain global feature.

3. The two-stage feature alignment based open-set human activity recognition method of claim 1, wherein, The method includes: pre-training a classifier of the category division module using the source domain instance features, and performing singular value decomposition on the parameter matrix of the classifier of the category division module; dividing the right singular matrix obtained through the singular value decomposition to obtain a source domain known category feature subspace and a source domain unknown category feature subspace; projecting the target domain instance features on the source domain known category feature subspace and the source domain unknown category feature subspace to obtain corresponding known features and unknown features; calculating the Euclidean norm of the unknown features obtained through the projection, and then comparing the Euclidean norm with a distribution threshold to determine whether the target domain instance features are common class instance features or private class instance features.

4. The two-stage feature alignment based open-set human activity recognition method of claim 1, wherein, The discrete timing self-attention module comprises a time discrete sampling layer, which inputs the common class instance features into the discrete timing self-attention module to obtain the transferable features, comprising: For the first time-discrete sampling layer, the instance features are slidingly divided into local instance features, and the local instance features are fused by a multilayer perceptron to obtain local discrete features; performing category prediction on the local discrete features through an auxiliary classifier to obtain a category prediction result; calculating the entropy of the category prediction, and then obtaining the weight of the local discrete features based on the entropy of the category prediction; performing weighted summation on all the local discrete features to obtain transferable features.

5. The two-stage feature alignment based open-set human activity recognition method of claim 1, wherein, The instance feature alignment loss is calculated by the adversarial learning, and the instance feature alignment loss comprises: inputting the transferable feature corresponding to the source domain instance feature into a human activity recognition classifier, and calculating a human activity recognition classification loss; calculating a weight of the source domain instance feature to the target domain instance feature; inputting the transferable feature corresponding to the source domain instance feature and the transferable feature corresponding to the target domain instance feature into an instance feature alignment discriminator, and calculating an instance feature alignment discriminator loss corresponding to each target domain instance feature; combining the human activity recognition classification loss and all the instance feature alignment discriminator losses to obtain the instance feature alignment loss.

6. The two-stage feature alignment based open-set human activity recognition method of claim 1, wherein, The private class constraint loss is represented by the following formula: ； wherein, denotes the private class constraint loss, is the number of target domain private class instance feature samples, denotes the deviation from the feature center, denotes the deviation from the probability center.

Images