Cross-domain activity recognition method and system based on base model enhanced semantic prototype

Abstract

This invention relates to a cross-domain activity recognition method and system based on a base model-enhanced semantic prototype. The method includes: acquiring textual descriptions of activity categories; extracting semantic features from the textual descriptions using a pre-trained base model; normalizing the semantic features; and constructing a cross-domain shared semantic prototype space; acquiring raw sensor data collected by multimodal sensors; inputting the raw sensor data into a decoupled variational autoencoder; and decomposing the raw sensor data into domain latent variables representing domain style and activity latent variables representing activity content using a domain encoder and an activity encoder, respectively; mapping the activity latent variables to the semantic prototype space using a projection head to obtain mapped feature vectors; and calculating the cosine similarity between the feature vectors and semantic prototypes of each category in the semantic prototype space to obtain the activity recognition result. This method can eliminate domain noise interference and achieve high-precision activity recognition in an unsupervised target domain.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method and system for cross-domain activity recognition based on a base model-enhanced semantic prototype. Background Technology

[0002] With the rapid development of IoT technology and wearable devices, Human Activity Recognition (HAR) has become an important research direction in ubiquitous computing and artificial intelligence. HAR technology automatically identifies user behavior patterns by analyzing data from sensors such as accelerometers and gyroscopes, and has significant application value in areas such as smart home control, elderly health monitoring, rehabilitation training assessment, and human-computer interaction. However, in practical applications, due to individual user differences, heterogeneity of sensor devices, and variations in wearing position, there are often significant distribution differences between training data (source domain) and test data (target domain), a problem known as "domain drift." This inconsistency makes it difficult for high-precision models trained in specific environments to be directly generalized to new scenarios or users, severely limiting the large-scale deployment and practical application of HAR technology.

[0003] To address cross-domain recognition challenges, existing technologies primarily employ transfer learning and domain adaptation methods. Early research relied heavily on manually extracting statistical features and utilizing metrics such as maximum mean difference (MMD) to reduce the distributional distance between the source and target domains. With the rise of deep learning, methods based on deep neural networks have gradually become mainstream. For example, convolutional neural networks (CNNs) and recurrent neural networks (RNNs) are used to extract deep features and combined with adversarial training, typically introducing a domain discriminator. Through adversarial learning, the feature extractor is forced to generate features that cannot distinguish the source from the target. Furthermore, generative model-based methods have also attracted considerable attention, such as using variational autoencoders (VAEs) or generative adversarial networks (GANs) to reconstruct input data, attempting to align the distributions of different domains in the latent space. Simultaneously, decoupled representation learning has also been introduced into the HAR field.

[0004] However, most existing domain adaptation methods only align at the feature statistics level, lacking utilization of deep semantic information about activity categories. Secondly, existing decoupled representation learning methods often struggle to achieve thorough feature separation. Due to the lack of effective supervision signals or constraint mechanisms, in complex cross-domain scenarios, domain-specific style information is easily leaked into the activity content representation, causing the classifier to still be affected by domain differences. Furthermore, traditional methods typically ignore the rich semantic information contained in the labeled text, resulting in extremely limited model generalization ability and difficulty adapting to dynamic changes in open environments.

[0005] Therefore, traditional human activity recognition methods often suffer from low recognition accuracy due to weak cross-domain generalization ability, as they only perform distribution alignment at the feature statistics level, lack deep semantic guidance for activity categories, and struggle to completely decouple domain-specific styles from activity content features. Summary of the Invention

[0006] In order to solve the above-mentioned technical problems, a cross-domain activity recognition method and system based on a basic model-enhanced semantic prototype is provided. This method can eliminate the interference of domain noise and achieve high-precision activity recognition in an unsupervised target domain.

[0007] A method for cross-domain activity identification based on a base model enhanced with semantic prototypes, the method comprising:

[0008] Obtain the text description of the activity category, extract the semantic features of the text description using a pre-trained base model, normalize the semantic features, and construct a cross-domain shared semantic prototype space.

[0009] The raw sensing data collected by the multimodal sensor is acquired and input into the decoupled variational autoencoder. The raw sensing data is decomposed into domain latent variables representing the domain style and activity latent variables representing the activity content by the domain encoder and activity encoder, respectively.

[0010] The activity latent variables are mapped to the semantic prototype space through a projection head to obtain the mapped feature vector. The cosine similarity between the feature vector and each category of semantic prototype in the semantic prototype space is calculated to obtain the activity recognition result.

[0011] Calculate the reconstruction loss, KL divergence loss, and auxiliary classification loss. Construct a total loss function based on the weighted sum of the reconstruction loss, KL divergence loss, and auxiliary classification loss. Perform joint optimization of the decoupled variational autoencoder and the projector head using the total loss function.

[0012] In one embodiment, a textual description of the activity category is obtained, and semantic features of the textual description are extracted using a pre-trained base model, including:

[0013] Define a set of categories containing various activity categories, and obtain the natural language text description corresponding to each activity category;

[0014] The text description is input into a pre-trained base model, which performs word segmentation, context encoding, and forward inference on the text description to obtain the original semantic prototype vector as semantic features.

[0015] In one embodiment, the semantic features are normalized, and a cross-domain shared semantic prototype space is constructed, including:

[0016] The original semantic prototype vector is subjected to L2 normalization to obtain semantic prototypes distributed on the unit hypersphere for cross-domain alignment.

[0017] The semantic prototypes are concatenated and stacked in order of category to construct a semantic prototype matrix;

[0018] Based on the semantic prototype matrix, a semantic prototype space that is independent of the acquisition domain and shared across domains is constructed.

[0019] In one embodiment, raw sensing data collected by a multimodal sensor is acquired, and the raw sensing data is input to a decoupled variational autoencoder. The raw sensing data is then decomposed into domain latent variables representing domain style and activity latent variables representing activity content by a domain encoder and an activity encoder, respectively, including:

[0020] The raw sensing data collected by the multimodal sensor is preprocessed to obtain various time-series input data in a fixed format;

[0021] Each of the aforementioned time-series input data is input in parallel to the domain encoder and active encoder of the decoupled variational autoencoder;

[0022] The posterior distribution parameters of the domain latent variables are output through the domain encoder, and the posterior distribution parameters of the active latent variables are output through the active encoder.

[0023] Based on the posterior distribution parameters of the domain latent variable and the posterior distribution parameters of the activity latent variable, the domain latent variable and the activity latent variable are obtained.

[0024] In one embodiment, the raw sensor data is decomposed into domain latent variables representing domain style and activity latent variables representing activity content by a domain encoder and an activity encoder, respectively, including:

[0025] The domain encoder employs a first multi-layer convolutional neural network structure to extract features from the temporal input data and output the mean vector and log-variance vector of the posterior distribution of the domain latent variables.

[0026] The active encoder employs a second multi-layer convolutional neural network structure to extract features from the temporal input data and output the mean vector and log-variance vector of the posterior distribution of the active latent variables.

[0027] Based on the mean and log-variance vectors of the posterior distribution of the domain latent variable and the mean and log-variance vectors of the posterior distribution of the active latent variable, the posterior distribution parameters of the domain latent variable and the active latent variable are obtained, and the decomposed domain latent variable and active latent variable are output.

[0028] In one embodiment, the activity latent variable is mapped to the semantic prototype space through a projection head to obtain a mapped feature vector, and the cosine similarity between the feature vector and each category of semantic prototype in the semantic prototype space is calculated to obtain the activity recognition result, including:

[0029] The active latent variables are input into a linear projection head for linear dimension transformation and mapped to the semantic prototype space to obtain the mapped feature vector.

[0030] Calculate the cosine similarity between the feature vector and each category of semantic prototype in the semantic prototype space, and introduce a learnable temperature coefficient to scale the cosine similarity;

[0031] The scaled cosine similarity is converted into a probability distribution using the Softmax function, and the category with the highest probability is taken as the activity recognition result based on the probability distribution.

[0032] In one embodiment, the decoupled variational autoencoder further includes a decoder; the method further includes:

[0033] The domain latent variables and activity latent variables are input into the decoder. The decoder concatenates the domain latent variables and activity latent variables to reconstruct the original sensing data, thus obtaining the reconstructed data.

[0034] In one embodiment, the reconstruction loss, KL divergence loss, and auxiliary classification loss are calculated, and a total loss function is constructed based on the weighted sum of the reconstruction loss, KL divergence loss, and auxiliary classification loss, including:

[0035] Calculate the mean square error between the original sensing data and the reconstructed data, and calculate the reconstruction loss based on the mean square error;

[0036] Calculate the KL divergence between the posterior distribution and the corresponding conditional prior distribution of the domain latent variable and the activity latent variable respectively, and weight each KL divergence to obtain the KL divergence loss;

[0037] Based on the domain latent variables and activity latent variables, the domain classification loss and activity classification loss are calculated using the cross loss function, and the auxiliary classification loss is obtained by weighting the domain classification loss and activity classification loss.

[0038] The total loss function is obtained by weighted summing of the reconstruction loss, KL divergence loss, and auxiliary classification loss.

[0039] In one embodiment, the method further includes:

[0040] Calculate the reconstruction error between the original sensing data and the reconstructed data, and calculate the semantic deviation between the projected active latent variables of the sample and the known category semantic prototype;

[0041] After normalizing the reconstruction error, it is weighted and combined with the semantic deviation to obtain a comprehensive anomaly score. When the comprehensive anomaly score exceeds a preset threshold, the sample is determined to be an abnormal activity.

[0042] A cross-domain activity recognition system based on a base model enhanced with semantic prototypes, the system comprising:

[0043] The semantic prototype construction module is used to obtain text descriptions of activity categories, extract semantic features of the text descriptions using a pre-trained base model, normalize the semantic features, and construct a cross-domain shared semantic prototype space.

[0044] The data decomposition module is used to acquire the raw sensing data collected by the multimodal sensor, input the raw sensing data into the decoupled variational autoencoder, and decompose the raw sensing data into domain latent variables representing the domain style and activity latent variables representing the activity content through the domain encoder and the activity encoder, respectively.

[0045] The semantic alignment and activity recognition module is used to map the activity latent variables to the semantic prototype space through the projection head to obtain the mapped feature vector, and calculate the cosine similarity between the feature vector and each category of semantic prototype in the semantic prototype space to obtain the activity recognition result.

[0046] The training module is optimized to calculate the reconstruction loss, KL divergence loss, and auxiliary classification loss. A total loss function is constructed based on the weighted sum of the reconstruction loss, KL divergence loss, and auxiliary classification loss. The decoupled variational autoencoder and the projector are jointly optimized using the total loss function.

[0047] The aforementioned cross-domain activity recognition method and system based on a base model enhanced with semantic prototypes extracts textual semantic features of activity categories and constructs a cross-domain shared semantic prototype space. This leverages the natural semantic relationships between activity categories as cross-domain invariant anchors, enhancing the model's generalization ability across different sensors, users, and wearing positions. By decoupling the variational autoencoder, the original sensor data is decomposed into domain style domain latent variables and activity content latent variables, effectively separating interference information from device noise, acquisition environment, and individual differences. Mapping the activity latent variables to the semantic prototype space and using cosine similarity for classification achieves precise alignment between sensor features and textual semantics, strengthening category boundary discrimination and outputting stable and accurate activity recognition results even in unlabeled target domains. Constructing and jointly optimizing the total loss function simultaneously ensures data reconstruction integrity, latent variable decoupling effectiveness, and classification accuracy, improving model training convergence speed and overall recognition accuracy. Attached Figure Description

[0048] Figure 1 This is an application environment diagram of a cross-domain activity recognition method based on a base model-enhanced semantic prototype in one embodiment;

[0049] Figure 2 This is a flowchart illustrating a cross-domain activity identification method based on a base model-enhanced semantic prototype in one embodiment.

[0050] Figure 3 This is a block diagram of a cross-domain activity recognition system based on a base model-enhanced semantic prototype in one embodiment.

[0051] Figure 4 This is a schematic diagram of a cross-domain activity recognition system based on a base model-enhanced semantic prototype in another embodiment;

[0052] Figure 5 This is a schematic diagram of the semantic prototype construction module structure in one embodiment;

[0053] Figure 6 This is a schematic diagram of the structure of a multimodal decoupling variational autoencoder module in one embodiment;

[0054] Figure 7 This is a schematic diagram of a cross-domain activity recognition framework based on a base model and enhanced semantic prototype in one embodiment.

[0055] Figure 8 This is a schematic diagram of a process applied to elderly monitoring in a smart home, as shown in one embodiment.

[0056] Figure 9 This is a schematic diagram of a process applied to the assessment of rehabilitation training movements in one embodiment;

[0057] Figure 10 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0059] It is understood that the terms "first," "second," etc., used in this application may be used to describe multilayer convolutional neural network (CNN) structures, but these CNN structures are not limited by these terms. These terms are only used to distinguish one CNN structure from another. For example, without departing from the scope of this application, the first CNN structure may be referred to as the second CNN structure, and similarly, the second CNN structure may be referred to as the first CNN structure. Both the first and second CNN structures are CNN structures, but they are not the same CNN structure.

[0060] The cross-domain activity identification method based on a base model-enhanced semantic prototype provided in this application can be applied to, for example... Figure 1 The application environment shown. For example... Figure 1 As shown, the application environment includes computer device 110. Computer device 110 can acquire textual descriptions of activity categories, extract semantic features of the textual descriptions using a pre-trained base model, normalize the semantic features, and construct a cross-domain shared semantic prototype space. Computer device 110 can acquire raw sensing data collected by multimodal sensors, input the raw sensing data into a decoupled variational autoencoder, and decompose the raw sensing data into domain latent variables representing domain style and activity latent variables representing activity content through a domain encoder and an activity encoder, respectively. Computer device 110 can map the activity latent variables to the semantic prototype space through a projection head to obtain the mapped feature vectors, and calculate the cosine similarity between the feature vectors and the semantic prototypes of each category in the semantic prototype space to obtain the activity recognition result. Computer device 110 can calculate reconstruction loss, KL divergence loss, and auxiliary classification loss, construct a total loss function based on the weighted sum of reconstruction loss, KL divergence loss, and auxiliary classification loss, and jointly optimize the decoupled variational autoencoder and projection head through the total loss function. Among them, computer equipment 110 may include, but is not limited to, various personal computers, laptops, smartphones, robots, unmanned aerial vehicles, tablets, and other devices.

[0061] In one embodiment, such as Figure 2As shown, a cross-domain activity identification method based on a base model and enhanced semantic prototypes is provided, including the following steps:

[0062] Step 202: Obtain the text description of the activity category, extract the semantic features of the text description using the pre-trained base model, normalize the semantic features, and construct a cross-domain shared semantic prototype space.

[0063] Traditional activity recognition methods typically treat category labels as discrete one-hot encodings, ignoring the rich semantic associations inherent in the label text itself. In this embodiment, a pre-trained base model (such as SentenceTransformer) can be introduced to extract deep text embeddings of activity category names as semantic prototypes of the category. These prototypes have fixed relative positions in the feature space and can serve as cross-domain invariant reference points to guide the learning of visual / sensory features.

[0064] Specifically, computer devices can utilize pre-trained foundation models to transform textual descriptions of activity categories into high-dimensional semantic vectors, constructing a cross-domain shared semantic prototype space and providing stable semantic anchors for subsequent feature alignment. Specifically, the SentenceTransformer model can be used to extract deep semantic features and perform normalization processing, ensuring that the distribution of prototypes in the semantic space has clear geometric meaning.

[0065] In one embodiment, a cross-domain activity recognition method based on a base model-enhanced semantic prototype may further include a process of extracting semantic features. The specific process includes: defining a set of categories containing various activity categories, obtaining a natural language text description corresponding to each activity category; inputting the text description into a pre-trained base model, and performing word segmentation, context encoding, and forward inference processing on the text description through the base model to obtain the original semantic prototype vector as semantic features.

[0066] The computer device can pre-determine all categories of human activities to be identified, constructing a text set of activity categories. Each category corresponds to a natural language description, such as "walking," "running," "sitting," "standing," "going upstairs," "going downstairs," and "falling," forming a discrete and mutually exclusive category text system. Next, the computer device can sequentially input the activity category text descriptions into a pre-trained language model. The model performs word segmentation, embedding encoding, and contextual semantic modeling on the text, outputting a high-dimensional, dense feature vector containing deep semantic information. This feature vector is then processed through multiple Transformer layers to capture semantic relationships between words and similarities and differences between actions, mapping the discrete text into continuous high-dimensional semantic feature vectors. Fixed-dimensional semantic feature vectors are extracted as the original semantic features for that activity category, used for subsequent semantic prototype construction.

[0067] Specifically, computer devices can define a set of activity categories. ,in, This represents the total number of activity categories; for each category... Using a pre-trained base model (SentenceTransformer) Extracting the corresponding text embedding vectors, the basic model, pre-trained on a large-scale corpus, is able to capture subtle semantic differences between words. Specifically, for the first... Text description of each activity category Its semantic prototype vector The calculation formula is: ;in, Indicates the first Semantic prototype vectors of each category, This represents the output dimension of the pre-trained language model. The natural language description representing this category is used, such as "Walking", "Sitting", "Standing", etc. The pre-trained base model can adopt the all-mpnet-base-v2 architecture.

[0068] In one embodiment, a cross-domain activity recognition method based on a base model-enhanced semantic prototype may further include feature processing and the construction of a semantic prototype space. The specific process includes: performing L2 normalization on the original semantic prototype vectors to obtain semantic prototypes distributed on a unit hypersphere for cross-domain alignment; concatenating and stacking the semantic prototypes in order of category to construct a semantic prototype matrix; and constructing a semantic prototype space that is independent of the acquisition domain and shared across domains based on the semantic prototype matrix.

[0069] The computer device can perform L2 normalization on the extracted semantic features to eliminate the influence of feature magnitude, projecting all category semantic features onto a unit hypersphere. This ensures that subsequent semantic similarity is measured only by differences in vector directions, improving the stability of cross-domain matching. Next, the computer device can arrange the normalized semantic features of all activity categories in category order to form a semantic prototype matrix, which serves as a cross-domain invariant semantic anchor point to guide the alignment and classification of sensor features.

[0070] Specifically, in order to eliminate the influence of vector magnitude and make similarity calculation focus more on directional consistency, i.e., semantic content similarity, computer devices can perform... Normalization process: in, This represents the normalized semantic prototype vector. This represents the Euclidean norm of the vector. These normalized prototype vectors form the baseline anchors in the semantic space, with prototype vectors of all categories distributed on the unit hypersphere.

[0071] Next, a system containing [a specific component] was built into the computer device. Semantic matrix of prototypes The semantic matrix not only encodes the semantic information of each category, but also implicitly preserves the semantic distance between categories. For example, semantically similar activities (such as "going upstairs" and "going downstairs") have high cosine similarity of prototype vectors in the semantic space, while activities with large semantic differences (such as "lying down" and "running") have low similarity.

[0072] The semantic topology structure in this embodiment provides strong prior knowledge for cross-domain recognition. During model training, the features extracted from the sensor data are mapped to the same semantic space, and the sensor features of the same category are required to be as close as possible to their corresponding semantic prototypes. Since the semantic prototypes are generated by a domain-independent language model, they naturally have cross-domain invariance, which can effectively alleviate the domain drift problem, so that the model can still accurately classify based on semantic consistency when facing new sensor environments.

[0073] Step 204: Obtain the raw sensing data collected by the multimodal sensor, input the raw sensing data into the decoupled variational autoencoder, and decompose the raw sensing data into domain latent variables representing the domain style and activity latent variables representing the activity content through the domain encoder and activity encoder, respectively.

[0074] Computer devices incorporate Disentangled Variational Autoencoders (D-VAEs) to extract clean activity features from complex sensor data. A D-VAE architecture comprises a domain-specific encoder and an activity-specific encoder, rigorously decomposing raw sensor data into latent variables representing domain style and activity content, structurally decoupling features. Specifically, the D-VAE employs a convolutional neural network as the feature extractor and combines reparameterization techniques for variational inference, effectively separating task-irrelevant noise. The D-VAE can extract clean activity features from input data. Decomposed into two independent latent variables: domain latent variables (Including information in areas such as sensor noise and wearing position) and activity latent variables (Includes specific action details).

[0075] In one embodiment, a cross-domain activity recognition method based on a base model-enhanced semantic prototype may further include a process of decomposing data into latent variables. Specifically, this process includes: preprocessing the raw sensing data acquired by the multimodal sensor to obtain various time-series input data in a fixed format; inputting these time-series input data in parallel to the domain encoder and activity encoder of a decoupled variational autoencoder; outputting the posterior distribution parameters of the domain latent variables through the domain encoder and the posterior distribution parameters of the activity latent variables through the activity encoder; and obtaining the domain latent variables and the activity latent variables based on the posterior distribution parameters of the domain latent variables and the activity latent variables.

[0076] Computer equipment can acquire raw human activity sensor data collected by multimodal sensors such as accelerometers and gyroscopes. This raw sensor data undergoes denoising, sliding window segmentation, resampling, and standardization to obtain time-series input data of fixed length and uniform format. Next, the computer equipment can simultaneously input each time-series input data into the domain encoder and activity encoder of a decoupled variational autoencoder. The two encoders operate in parallel and independently, without sharing parameters. The domain encoder outputs the posterior distribution parameters of the domain latent variables, and the activity encoder outputs the posterior distribution parameters of the activity latent variables, thus obtaining the domain latent variables and the activity latent variables.

[0077] In one embodiment, a cross-domain activity recognition method based on a base model-enhanced semantic prototype may further include a process of specifically decomposing the original sensor data, including: a domain encoder employing a first multi-layer convolutional neural network structure to extract features from the temporal input data and output the mean vector and log-variance vector of the posterior distribution of the domain latent variables; an activity encoder employing a second multi-layer convolutional neural network structure to extract features from the temporal input data and output the mean vector and log-variance vector of the posterior distribution of the activity latent variables; based on the mean vector and log-variance vector of the posterior distribution of the domain latent variables and the mean vector and log-variance vector of the posterior distribution of the activity latent variables, the posterior distribution parameters of the domain latent variables and the posterior distribution parameters of the activity latent variables are obtained, and the decomposed domain latent variables and activity latent variables are output.

[0078] The domain encoder uses a first-layer convolutional neural network to extract features from the original sensor data, outputting the mean and log-variance of the posterior distribution of the domain latent variables. Based on a reparameterization technique, it samples the distribution of the domain latent variables to obtain domain latent variables, which specifically represent domain-related information, including sensor type, wearing position, device noise, sampling differences, and other domain-specific characteristics. The activity encoder uses a second-layer convolutional neural network, structurally independent of the domain encoder, to extract features from the same original sensor data, outputting the mean and log-variance of the posterior distribution of the activity latent variables. Based on a reparameterization technique, it samples the distribution of the activity latent variables to obtain activity latent variables, which specifically represent the content information of the action itself, including action type, motion amplitude, temporal pattern, and other activity content features.

[0079] Domain encoder of decoupled variational autoencoder and active encoder For a given time-series input data Two encoders predict the posterior distribution parameters (mean and log-variance) of the latent variables. Both encoders employ a multi-layer convolutional neural network (CNN) structure, specifically containing four convolutional blocks. The first convolutional layer maps the low-dimensional sensor input to a high-dimensional feature space (e.g., 1024-dimensional). Subsequently, abstract features are extracted and the dimensionality is reduced through layer-by-layer convolution and pooling operations (the number of channels changes sequentially to 1024, 512, 128, and 64). Finally, the data is flattened and passed through a fully connected layer to generate the mean and variance of the latent variables.

[0080] Specifically, for domain latent variables Its posterior distribution Assume it follows a multivariate Gaussian distribution: ; ;in, and Let represent the mean vector and standard deviation vector of the domain latent variable distribution, respectively. It is a domain encoder network. For its network parameters. Similarly, for the activity latent variables... Its posterior distribution Represented as: ; ;in, and Let represent the mean vector and standard deviation vector of the distribution of the latent variables of activity, respectively. It is an active encoder network. Its network parameters.

[0081] To achieve backpropagation of the gradient, this embodiment employs the reparameterization trick for sampling, making the random sampling process differentiable: in, This represents element-wise multiplication. and It is an auxiliary noise variable sampled from the standard normal distribution. It is an identity matrix.

[0082] In one embodiment, the decoupled variational autoencoder further includes a decoder; the provided cross-domain activity recognition method based on a base model-enhanced semantic prototype may also include a data reconstruction process, specifically including: inputting domain latent variables and activity latent variables into the decoder, and reconstructing the original sensing data by concatenating the domain latent variables and activity latent variables through the decoder to obtain reconstructed data.

[0083] decoder The goal is to utilize the decoupled hidden variables and Reconstruct the original input data This ensures that the decomposed latent variables retain the complete information of the original data, preventing information loss.

[0084] The refactoring process can be represented as: ;in, It is the reconstructed sensor data. It is a decoder network composed of deconvolution / ConvTranspose2d layers. Its parameters. The decoder first... and The data is concatenated along the feature dimension, then mapped to the initial feature map size through a fully connected layer. Subsequently, it undergoes four deconvolutional blocks (including unpooling operations) to progressively restore the spatiotemporal dimensions of the data. The final output is the same as the input. Reconstructed data with consistent shape.

[0085] To constrain the distribution of latent variables and introduce prior knowledge of the domain and category, a conditional prior network is also designed in this embodiment. For domain latent variables, their prior distribution... Domain tags Generated via a fully connected network; for the activity latent variables, their prior distribution... By activity tag Generation; further strengthening the semantic meaning of latent variables, making Tendency to cluster in specific domains They tend to cluster around specific activities.

[0086] Step 206: Map the activity latent variables to the semantic prototype space through the projection head to obtain the mapped feature vector, and calculate the cosine similarity between the feature vector and each category of semantic prototype in the semantic prototype space to obtain the activity recognition result.

[0087] To ensure the decoupling of active hidden variables To truly incorporate activity semantics and align them with predefined semantic prototypes, computer devices can design semantic alignment mechanisms based on projection heads and introduce auxiliary classification tasks for joint optimization. The computer device can use the projection head to map the decoupled activity content latent variables to the semantic space, and then match them with the semantic prototype using cosine similarity to obtain the activity recognition result.

[0088] In one embodiment, a cross-domain activity recognition method based on a base model-enhanced semantic prototype may further include a semantic alignment and activity recognition process. The specific process includes: inputting the activity latent variables into a linear projection head for linear dimension transformation, mapping them to a semantic prototype space, and obtaining the mapped feature vector; calculating the cosine similarity between the feature vector and each category of semantic prototypes in the semantic prototype space, and introducing a learnable temperature coefficient to scale the cosine similarity; using a Softmax function to convert the scaled cosine similarity into a probability distribution, and using the category with the highest probability as the activity recognition result based on the probability distribution.

[0089] The computer device can extract the latent activity variables generated by the decoupled variational autoencoder and input them into a preset projection head. The projection head uses one or more fully connected neural networks (linear layers) to increase or map the dimension of the latent activity variables through matrix multiplication and weight parameter operations, ensuring that its feature dimension is consistent with the dimension of the semantic prototype space, thus aligning the feature space dimensions. After the projection head transformation, the resulting mapped feature vectors are unified into a high-dimensional space identical to the semantic prototypes. At this point, the mapped feature vectors no longer merely represent the original action sequence features but possess semantic expressive power, enabling them to establish a direct semantic association with the linguistic description of the activity category. Since the constructed semantic prototype space stores the normalized semantic prototype vectors corresponding to all activity categories, the mapped feature vectors are used as query vectors and paired with each category's semantic prototype vector (Key) in the semantic prototype space, constructing a computational graph containing the similarity relationships of all categories. Then, the computer device can use a cosine similarity metric function to calculate the angular distance between the query vector and each prototype vector pairwise. The cosine similarity value ranges from (−1, 1). A value closer to 1 indicates that the two vectors are more aligned in direction, meaning they represent more similar semantic activities; a value closer to -1 indicates that the semantics are more opposite. Based on the calculated cosine similarity values ​​for all categories, a preliminary classification score distribution is constructed.

[0090] In this embodiment, in order to adjust the sharpness of the classification distribution and enhance the generalization ability of the model, a learnable temperature coefficient τ can be introduced to scale the similarity score. The similarity score vector after temperature coefficient adjustment is applied to the Softmax activation function to convert it into a probability distribution vector. Each element in the vector corresponds to the probability value of the activity. The activity category corresponding to the dimension with the highest probability value is the final activity recognition result.

[0091] Specifically, due to activity latent variables The dimension is usually related to the semantic prototype. Due to the inconsistency in dimensions, this embodiment introduces a linear projection head. Will Mapping to a high-dimensional semantic space: ;in, It is the mapped feature vector. and These are learnable projection weight matrices and bias vectors. Next, the mapping features are calculated. semantic prototypes of all categories The cosine similarity between them is used as the basis for classification: in, This indicates that the current sample belongs to the first... Semantic similarity scores for class activities. To enhance the discriminative power of classification and control the entropy of the probability distribution, a learnable temperature coefficient is introduced. The similarity is scaled, and the probability of belonging to each class is calculated using the Softmax function: ;in, It is a learnable temperature parameter (usually initialized to 0.07) used to control the smoothness of the probability distribution. When When the size is smaller, the distribution is sharper, and the model is more confident in its predictions.

[0092] Step 208: Calculate the reconstruction loss, KL divergence loss, and auxiliary classification loss. Construct a total loss function based on the weighted sum of the reconstruction loss, KL divergence loss, and auxiliary classification loss. Then, perform joint optimization of the decoupled variational autoencoder and the projector head using the total loss function.

[0093] The computer device can perform joint optimization by combining reconstruction loss, KL divergence loss and auxiliary classification loss; it adjusts the sharpness of the classification distribution by introducing a learnable temperature coefficient, and strengthens the semantic consistency of latent variables through a multi-task learning strategy.

[0094] In one embodiment, a cross-domain activity recognition method based on a base model-enhanced semantic prototype may further include a process of constructing a total loss function. Specifically, this process includes: calculating the mean squared error between the original sensing data and the reconstructed data, and calculating the reconstruction loss based on the mean squared error; calculating the KL divergence between the posterior distributions of the domain latent variable and the activity latent variable and their corresponding conditional prior distributions, and weighting each KL divergence to obtain the KL divergence loss; calculating the domain classification loss and the activity classification loss using a cross-loss function based on the domain latent variable and the activity latent variable, and weighting the domain classification loss and the activity classification loss to obtain the auxiliary classification loss; and weighting and summing the reconstruction loss, the KL divergence loss, and the auxiliary classification loss to obtain the total loss function.

[0095] In order to force Includes domain information. Including activity information, two auxiliary classifiers are introduced: a domain classifier. (Based on fully connected layers) and an activity classifier. In this embodiment, the total loss function... It consists of three parts: reconstruction loss, KL divergence loss, and auxiliary classification loss, which are jointly optimized by weighted summation.

[0096] Specifically, the computer equipment can input the domain latent variables and active latent variables from the decoupled variational autoencoder into the decoder to obtain the reconstruction result of the original sensing data; the mean squared error (MSE) between the original sensing data and the reconstructed data is used as the reconstruction loss to constrain the model to retain complete data information and avoid feature loss. The reconstruction loss... Mean squared error (MSE) is used to measure the input. With Restructuring The difference between them, the reconstruction loss can be expressed as: .

[0097] The computer equipment can separately calculate the KL divergence between the posterior distribution of the domain latent variables and the domain conditional prior distribution generated based on the domain labels, and the KL divergence between the posterior distribution of the active latent variables and the active conditional prior distribution generated based on the activity labels. The two KL divergences are then weighted according to preset weighting coefficients to obtain the total KL divergence loss, which is used to normalize the latent variable distribution and enhance feature decoupling. The KL divergence loss... The constraint that the posterior distribution approximates the prior distribution serves a regularization function; the KL divergence between the domain-conditional prior distributions is: The KL divergence between the prior distributions of the activity conditions is: in, Indicates the Kullback-Leibler divergence. and It is a conditional prior distribution generated based on labels. and These are the corresponding weighting coefficients (usually set to 1.0). The two KL divergences are weighted according to the preset weighting coefficients to obtain the total KL divergence loss: .

[0098] The computer device can construct domain classification loss and activity classification loss separately: Domain latent variables are input into the domain classifier, and the cross-entropy loss is calculated using real domain labels as supervision, constraining the domain latent variables to learn only domain style information; the similarity between the activity features mapped to the semantic space and the semantic prototype is calculated and classified, and the cross-entropy loss is calculated using real activity labels as supervision, constraining the activity latent variables to learn only activity content information; the two auxiliary classification losses are weighted according to preset weights to obtain the total auxiliary classification loss. The domain classification loss is: The activity-classified loss is: ;in, Based on Domain classifier, and These are the real domain tag and the activity tag (in one-hot format); and These are the weight coefficients of the auxiliary classification loss; larger weights help accelerate the learning of decoupled features. The final total auxiliary classification loss is: .

[0099] The computer equipment can perform a weighted summation of the reconstruction loss, KL divergence loss, and auxiliary classification loss to obtain the total loss function used for overall model optimization. By minimizing the total loss through backpropagation, accurate data reconstruction, effective decoupling of latent variables, semantic alignment classification, and domain differentiation are simultaneously achieved, completing the joint optimization of the model. The total loss function is: .

[0100] In one embodiment, a cross-domain activity identification method based on a base model-enhanced semantic prototype may further include an anomaly detection process. The specific process includes: calculating the reconstruction error between the original sensing data and the reconstructed data, and calculating the semantic deviation between the projected activity latent variable of the sample and the known category semantic prototype; normalizing the reconstruction error and weighting it with the semantic deviation to obtain a comprehensive anomaly score; when the comprehensive anomaly score exceeds a preset threshold, the sample is determined to be an abnormal activity.

[0101] Computer equipment can utilize a dual scoring mechanism combined with sensor reconstruction quality and semantic bias for anomaly detection. After the decoupled variational autoencoder and projector head are trained, for test samples... Whether it is abnormal activity can be evaluated through two dimensions: reconstruction error and semantic deviation.

[0102] Specifically, reconstruction error This measures the difference between the input data and the model's reconstructed output. Outlier samples, due to their significantly different distribution characteristics from normal activity in the training set, are often difficult to reconstruct accurately, thus resulting in higher reconstruction errors. in, and Let represent the posterior distribution means of the domain latent variable and the activity latent variable, respectively. This represents the decoder function.

[0103] Semantic bias The measure is the maximum similarity between the projected latent variables of a sample's activity and the semantic prototypes of all known categories. Latent variables for normal activity should be highly aligned with the prototypes of their corresponding categories, while latent variables for anomalous activity are difficult to match with any known prototypes. in, This represents the result of mapping the mean of the activity's latent variables to the semantic space via the projection head. Represents the cosine similarity function. Indicates the first Semantic prototype vectors of each category, This represents the total number of known activity categories.

[0104] Final anomaly score This is obtained by weighted combination of reconstruction error and semantic bias, taking into account information from both the sensor and semantic levels: in, It is a balancing hyperparameter used to adjust the relative importance of the two scores; Represents the minimum-maximum normalization function, calibrated based on the training set data to ensure that the two scores are on the same scale; when When the sample exceeds the preset threshold, It was determined to be abnormal activity.

[0105] The dual scoring mechanism in this embodiment fully utilizes the reconstruction capability and semantic prototype discrimination capability of the variational autoencoder, which can accurately identify out-of-distribution abnormal behavior in cross-domain scenarios, significantly improving the robustness and practicality of the model.

[0106] This application provides a cross-domain activity identification method based on a base model enhanced with semantic prototypes.

[0107] This application addresses the lack of semantic guidance in traditional alignment methods by introducing semantic prototypes generated by a pre-trained Foundation Model as "anchors" for cross-domain sharing. Specifically, it utilizes a SentenceTransformer to extract embedding vectors from activity description text, constructing class prototypes with clear semantic meaning. Simultaneously, it combines a Disentangled Variational Autoencoder (VAE) to design a specialized encoding structure that rigorously decomposes the sensor data into domain-invariant activity content latent variables and domain-specific style latent variables. To ensure the effectiveness of decoupling, an auxiliary classification task is introduced, using domain labels and activity labels to supervise the decomposed latent variables. Finally, a semantic alignment-based classification mechanism maps the extracted activity content latent variables to the semantic space and calculates their cosine similarity to the semantic prototype for classification. This method not only leverages multimodal information to enhance the discriminative power of features but also effectively eliminates domain noise interference through explicit decoupling and semantic constraints, thereby achieving high-precision activity recognition in the unsupervised target domain and significantly improving the model's generalization ability and robustness.

[0108] It should be understood that although the steps in the flowchart above are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart above may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0109] In one embodiment, such as Figure 3 As shown, a cross-domain activity recognition system based on a base model enhanced with semantic prototypes is provided, including: a semantic prototype construction module 310, a data decomposition module 320, a semantic alignment and activity recognition module 330, and an optimization training module 340, wherein:

[0110] The semantic prototype construction module 310 is used to obtain text descriptions of activity categories, extract semantic features of the text descriptions using a pre-trained base model, normalize the semantic features, and construct a cross-domain shared semantic prototype space.

[0111] The data decomposition module 320 is used to acquire the raw sensing data collected by the multimodal sensor, input the raw sensing data into the decoupled variational autoencoder, and decompose the raw sensing data into domain latent variables representing the domain style and activity latent variables representing the activity content through the domain encoder and the activity encoder, respectively.

[0112] The semantic alignment and activity recognition module 330 is used to map the activity latent variables to the semantic prototype space through the projection head, obtain the mapped feature vector, and calculate the cosine similarity between the feature vector and each category of semantic prototype in the semantic prototype space to obtain the activity recognition result.

[0113] The training module 340 is optimized to calculate the reconstruction loss, KL divergence loss, and auxiliary classification loss. The total loss function is constructed based on the weighted sum of the reconstruction loss, KL divergence loss, and auxiliary classification loss. The decoupled variational autoencoder and the projector are jointly optimized through the total loss function.

[0114] In one embodiment, the semantic prototype construction module 310 is further configured to define a set of categories containing various activity categories, obtain a natural language text description corresponding to each activity category, input the text description into a pre-trained base model, and perform word segmentation, context encoding and forward inference processing on the text description through the base model to obtain the original semantic prototype vector as semantic features.

[0115] In one embodiment, the semantic prototype construction module 310 is further used to perform L2 normalization on the original semantic prototype vector to obtain semantic prototypes distributed on the unit hypersphere for cross-domain alignment; to concatenate and stack the semantic prototypes in order of category to construct a semantic prototype matrix; and to construct a semantic prototype space that is independent of the acquisition domain and shared across domains based on the semantic prototype matrix.

[0116] In one embodiment, the data decomposition module 320 is further configured to preprocess the raw sensing data acquired by the multimodal sensor to obtain various time-series input data in a fixed format; input the various time-series input data in parallel to the domain encoder and the active encoder of the decoupled variational autoencoder; output the posterior distribution parameters of the domain latent variables through the domain encoder and the posterior distribution parameters of the active latent variables through the active encoder; and obtain the domain latent variables and the active latent variables based on the posterior distribution parameters of the domain latent variables and the posterior distribution parameters of the active latent variables.

[0117] In one embodiment, the data decomposition module 320 is further configured to: extract features from the temporal input data using a first multi-layer convolutional neural network structure via a domain encoder; output the mean vector and log-variance vector of the posterior distribution of the domain latent variable; extract features from the temporal input data using a second multi-layer convolutional neural network structure via an activity encoder; output the mean vector and log-variance vector of the posterior distribution of the activity latent variable; and, based on the mean vector and log-variance vector of the posterior distribution of the domain latent variable and the posterior distribution vector of the activity latent variable, obtain the posterior distribution parameters of the domain latent variable and the posterior distribution parameters of the activity latent variable, and output the decomposed domain latent variable and activity latent variable.

[0118] In one embodiment, the semantic alignment and activity recognition module 330 is further configured to input the activity latent variables into a linear projection head for linear dimension transformation, map them to a semantic prototype space, and obtain the mapped feature vector; calculate the cosine similarity between the feature vector and each category of semantic prototypes in the semantic prototype space, and introduce a learnable temperature coefficient to scale the cosine similarity; use the Softmax function to convert the scaled cosine similarity into a probability distribution, and use the category with the highest probability as the activity recognition result based on the probability distribution.

[0119] In one embodiment, the decoupled variational autoencoder further includes a decoder; the data decomposition module 320 is also used to input the domain latent variables and active latent variables into the decoder, and reconstruct the original sensing data by splicing the domain latent variables and active latent variables through the decoder to obtain the reconstructed data.

[0120] In one embodiment, the optimization training module 340 is further used to calculate the mean squared error between the original sensing data and the reconstructed data, and to calculate the reconstruction loss based on the mean squared error; to calculate the KL divergence between the posterior distribution of the domain latent variable and the activity latent variable and the corresponding conditional prior distribution, and to weight each KL divergence to obtain the KL divergence loss; to calculate the domain classification loss and the activity classification loss based on the domain latent variable and the activity latent variable using the cross loss function, and to weight the domain classification loss and the activity classification loss to obtain the auxiliary classification loss; and to weight and sum the reconstruction loss, the KL divergence loss, and the auxiliary classification loss to obtain the total loss function.

[0121] In one embodiment, the optimization training module 340 is further used to calculate the reconstruction error between the original sensing data and the reconstructed data, and to calculate the semantic deviation between the projected activity latent variable of the sample and the known category semantic prototype; after normalizing the reconstruction error, it is weighted and combined with the semantic deviation to obtain a comprehensive anomaly score; when the comprehensive anomaly score exceeds a preset threshold, the sample is determined to be an abnormal activity.

[0122] In another embodiment, a cross-domain activity recognition system based on a base model enhanced semantic prototype is provided, such as... Figure 4 As shown, the process includes several main stages: data input, feature extraction and decoupling, semantic alignment, and output. Input data includes time-series data from multimodal sensors and natural language descriptions of activity categories. The semantic prototype construction module is responsible for establishing a cross-domain shared semantic space. The decoupled variational autoencoder module is responsible for extracting clean activity features from heterogeneous sensor data. The semantic alignment and multi-task optimization module is responsible for mapping the extracted features to the semantic space and performing classification, while simultaneously strengthening feature decoupling through auxiliary tasks.

[0123] like Figure 4 As shown, the system mainly consists of three core modules: a semantic prototype construction module based on a base model, a multimodal decoupled variational autoencoder module, and a semantic alignment and multi-task optimization module. The system first receives textual descriptions of activity categories and raw sensor data as input. The semantic prototype construction module uses a pre-trained language model to transform the textual descriptions into semantic prototype vectors, providing semantic anchors for classification. The multimodal decoupled variational autoencoder module decomposes the sensor data into domain latent variables representing domain style and activity latent variables representing activity content through an encoder. The semantic alignment and multi-task optimization module receives the activity latent variables and semantic prototypes, performs classification prediction through projection and similarity matching, and simultaneously uses domain latent variables for domain classification auxiliary tasks to optimize the decoupling effect of the entire network. Finally, the system outputs activity recognition results, domain classification results, and reconstructed data.

[0124] In one embodiment, the semantic prototype building block structure is as follows: Figure 5 As shown, in practical applications, the semantic prototype construction module first inputs the activity category name (such as "Walking") into the pre-trained model to obtain high-dimensional text embeddings. To eliminate the influence of vector magnitude, the embedding vectors are subjected to L2 normalization, ultimately resulting in a semantic prototype matrix distributed on a unit hypersphere with clear geometric meaning. These prototype vectors serve as cross-domain shared anchor points, guiding subsequent feature learning.

[0125] In one embodiment, the structure of the multimodal decoupled variational autoencoder module is as follows: Figure 6 As shown, this aims to achieve feature decoupling and reconstruction. (As illustrated...) Figure 6 As shown, the input sensor data is fed into the domain encoder and the activity encoder, respectively. The encoder outputs the posterior distribution parameters (mean and log-variance) of the latent variables, which are sampled using the reparameterization trick to obtain the domain latent variable (zd) and the activity latent variable (zy). These two latent variables are then concatenated and input into the decoder to reconstruct the original input data, thereby ensuring the integrity of the features.

[0126] The cross-domain activity recognition system based on a base model enhanced with semantic prototypes, as described in this application, can be applied to, for example... Figure 7 The SemanticAnchor-VAE framework is shown below. Figure 7 As shown, the SemanticAnchor-VAE framework includes a process of generating semantic prototypes from activity labels using a pre-trained language model (SentenceTransformer); a decoupled network structure based on VAE, including a domain encoder and an activity encoder, and a decoder for reconstruction; and a semantic alignment-based classification mechanism that predicts by calculating the cosine similarity between the activity latent variables and the semantic prototypes. Based on the SemanticAnchor-VAE framework, the language model first generates semantic prototypes; then, the data preprocessing module inputs sensor data into the encoder, which generates latent variables, which are then passed to the decoder for reconstruction and to the classifier for semantic alignment and domain classification; finally, the system calculates the total loss, including reconstruction loss, KL divergence loss, and classification loss, and backpropagates to update the model parameters.

[0127] In one embodiment, such as Figure 8 As shown, a cross-domain activity recognition method and system based on a base model enhanced semantic prototype will be applied to a smart home scenario to build a non-invasive elderly safety monitoring system. This system aims to solve the problem of large differences in sensor data distribution and difficulty in universalization among different brands of smart devices (such as wristbands, watches, and mobile phones), achieving accurate identification and real-time alarm for the daily activities (such as walking, sitting, and lying down) and abnormal behaviors (such as falls) of the elderly. The specific implementation process is as follows:

[0128] Multi-source sensor data acquisition: The system receives real-time data from accelerometers and gyroscopes of smart bracelets and smartwatches worn by the elderly, as well as environmental sensors deployed in the room. This data may have different sampling rates (e.g., 50Hz or 100Hz) and measurement ranges.

[0129] Semantic prototype library construction: Using a pre-trained base model (SentenceTransformer, specifically all-mpnet-base-v2), input natural language descriptions of behavioral categories such as "falling down", "walking", "sitting down", and "lying down" to generate corresponding 768-dimensional standardized semantic prototype vectors, and construct a semantic space shared across devices.

[0130] Data preprocessing and alignment: The acquired heterogeneous sensor data is divided into sliding window segments (window length set to 128, overlap rate 50%), and interpolation resampling and Z-score normalization are performed to make it conform to the input format requirements of the model.

[0131] Feature decoupling encoding: The preprocessed data is input into a decoupled variational autoencoder (D-VAE) deployed on an edge gateway; the encoder decomposes the data into "domain latent variables" (50-dimensional) that characterize the device wearing position and sensor noise, and "activity latent variables" (50-dimensional) that characterize the actual limb movements.

[0132] Semantic space mapping: The extracted 50-dimensional "active latent variables" are mapped to a 768-dimensional space identical to the semantic prototype constructed in the linear projection head, in preparation for semantic matching;

[0133] Behavior recognition and matching: Calculate the cosine similarity between the mapped activity feature vector and the semantic prototypes of all behavior categories, and scale it by combining a temperature coefficient (set to 0.07) to output the probability distribution of elderly people's behavior within the current time window and determine whether a fall has occurred;

[0134] Anomaly detection and alarm: In addition to classification and recognition, the system also calculates the reconstruction error of the data and the distance between the features and the nearest semantic prototype. If the reconstruction error is too large or the similarity with all known prototypes is below the threshold, it is judged as abnormal behavior (such as sudden illness and convulsions) and an alarm is immediately triggered to notify the guardian.

[0135] System feedback and optimization: The guardian confirms or corrects the alarm result (such as a false alarm); the system collects this feedback data, fine-tunes the distribution of semantic prototypes or updates the projection head parameters, and realizes the online continuous evolution of the model.

[0136] In one embodiment, such as Figure 9 As shown, a cross-domain activity recognition method and system based on a base model enhanced with semantic prototypes will be applied to the field of medical rehabilitation to assist patients in home-based rehabilitation training. Addressing the issues of inconsistent patient wearing positions and significant differences in movement accuracy, cross-domain recognition capabilities are utilized to achieve accurate identification and quality assessment of specific rehabilitation movements (such as upper limb extension and squats). The specific implementation process is as follows:

[0137] Rehabilitation movement data acquisition: Patients wear medical-grade IMU sensors or consumer-grade smart wearable devices for rehabilitation training; the system collects inertial sensor data of limb movement (including 3-axis acceleration and 3-axis angular velocity).

[0138] Action semantic definition: The rehabilitation physician inputs a detailed text description of the standard rehabilitation action (such as "squat: stand with feet apart, back straight, squat down until thighs are parallel to the ground"); the system uses the basic model to generate high-quality semantic prototypes of these standard actions;

[0139] Style noise filtering: Data is input into a decoupled network, and the system automatically identifies and separates "style noise" (domain latent variables) caused by wearing position deviations (such as wearing it on the inside or outside of the wrist), preserving pure motion content features;

[0140] Core movement feature extraction: Extract "activity latent variables" from the decoupled latent space. These variables contain only kinematic information related to rehabilitation movements and are not affected by the patient's body shape or wearing method.

[0141] Action classification and recognition: The extracted features are mapped to the semantic space and matched with standard action semantic prototypes to identify the type of action that the patient is currently performing in real time and prevent action confusion;

[0142] Action quality quantification: Calculate the cosine similarity between the patient's actual action feature vector and the standard action semantic prototype vector; the higher the similarity, the more standard the action; the lower the similarity, the greater the deviation in the action. The system can set a threshold (e.g., 0.8) to determine whether the action is qualified.

[0143] Training report generation: The system counts the number of times each movement is completed and the average quality score, and generates a visual rehabilitation training report that points out the specific aspects of the movement that are not performed correctly;

[0144] Personalized model updates: As rehabilitation progresses, patients' motor abilities change; the system dynamically adjusts the weight of semantic prototypes or introduces new rehabilitation movement descriptions based on doctors' assessments of recent data to achieve personalized adaptation.

[0145] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 10 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When executed by the processor, the computer program implements a cross-domain activity recognition method based on a base model-enhanced semantic prototype. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0146] Those skilled in the art will understand that Figure 10The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0147] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of a cross-domain activity recognition method based on a base model-enhanced semantic prototype.

[0148] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program being executed by a processor to implement the steps of a cross-domain activity identification method based on a base model-enhanced semantic prototype.

[0149] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0150] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0151] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for cross-domain activity recognition based on a base model enhanced with semantic prototypes, characterized in that, The method includes: Obtain the text description of the activity category, extract the semantic features of the text description using a pre-trained base model, normalize the semantic features, and construct a cross-domain shared semantic prototype space. The raw sensing data collected by the multimodal sensor is acquired and input into the decoupled variational autoencoder. The raw sensing data is decomposed into domain latent variables representing the domain style and activity latent variables representing the activity content by the domain encoder and activity encoder, respectively. The activity latent variables are mapped to the semantic prototype space through a projection head to obtain the mapped feature vector. The cosine similarity between the feature vector and each category of semantic prototype in the semantic prototype space is calculated to obtain the activity recognition result. Calculate the reconstruction loss, KL divergence loss, and auxiliary classification loss. Construct a total loss function based on the weighted sum of the reconstruction loss, KL divergence loss, and auxiliary classification loss. Perform joint optimization of the decoupled variational autoencoder and the projector head using the total loss function.

2. The cross-domain activity identification method based on a base model-enhanced semantic prototype according to claim 1, characterized in that, Obtain textual descriptions of activity categories, and extract semantic features of the textual descriptions using a pre-trained base model, including: Define a set of categories containing various activity categories, and obtain the natural language text description corresponding to each activity category; The text description is input into a pre-trained base model, which performs word segmentation, context encoding, and forward inference on the text description to obtain the original semantic prototype vector as semantic features.

3. The cross-domain activity identification method based on a base model-enhanced semantic prototype according to claim 2, characterized in that, The semantic features are normalized, and a cross-domain shared semantic prototype space is constructed, including: The original semantic prototype vector is subjected to L2 normalization to obtain semantic prototypes distributed on the unit hypersphere for cross-domain alignment. The semantic prototypes are concatenated and stacked in order of category to construct a semantic prototype matrix; Based on the semantic prototype matrix, a semantic prototype space that is independent of the acquisition domain and shared across domains is constructed.

4. The cross-domain activity identification method based on a base model-enhanced semantic prototype according to claim 1, characterized in that, The raw sensing data acquired by the multimodal sensor is obtained and input into a decoupled variational autoencoder. The raw sensing data is then decomposed into domain latent variables representing domain style and activity latent variables representing activity content by a domain encoder and an activity encoder, respectively. The raw sensing data collected by the multimodal sensor is preprocessed to obtain various time-series input data in a fixed format; Each of the aforementioned time-series input data is input in parallel to the domain encoder and active encoder of the decoupled variational autoencoder; The posterior distribution parameters of the domain latent variables are output through the domain encoder, and the posterior distribution parameters of the active latent variables are output through the active encoder. Based on the posterior distribution parameters of the domain latent variable and the posterior distribution parameters of the activity latent variable, the domain latent variable and the activity latent variable are obtained.

5. The cross-domain activity identification method based on a base model-enhanced semantic prototype according to claim 4, characterized in that, The raw sensor data is decomposed into domain latent variables representing domain style and activity latent variables representing activity content using a domain encoder and an activity encoder, respectively, including: The domain encoder employs a first multi-layer convolutional neural network structure to extract features from the temporal input data and output the mean vector and log-variance vector of the posterior distribution of the domain latent variables. The active encoder employs a second multi-layer convolutional neural network structure to extract features from the temporal input data and output the mean vector and log-variance vector of the posterior distribution of the active latent variables. Based on the mean and log-variance vectors of the posterior distribution of the domain latent variable and the mean and log-variance vectors of the posterior distribution of the active latent variable, the posterior distribution parameters of the domain latent variable and the active latent variable are obtained, and the decomposed domain latent variable and active latent variable are output.

6. The cross-domain activity identification method based on a base model-enhanced semantic prototype according to claim 1, characterized in that, The activity latent variables are mapped to the semantic prototype space through a projection head to obtain the mapped feature vectors. The cosine similarity between the feature vectors and the semantic prototypes of each category in the semantic prototype space is then calculated to obtain the activity recognition result, including: The active latent variables are input into a linear projection head for linear dimension transformation and mapped to the semantic prototype space to obtain the mapped feature vector. Calculate the cosine similarity between the feature vector and each category of semantic prototype in the semantic prototype space, and introduce a learnable temperature coefficient to scale the cosine similarity; The scaled cosine similarity is converted into a probability distribution using the Softmax function, and the category with the highest probability is taken as the activity recognition result based on the probability distribution.

7. The cross-domain activity identification method based on a base model-enhanced semantic prototype according to claim 1, characterized in that, The decoupled variational autoencoder further includes a decoder; the method further includes: The domain latent variables and activity latent variables are input into the decoder. The decoder concatenates the domain latent variables and activity latent variables to reconstruct the original sensing data, thus obtaining the reconstructed data.

8. The cross-domain activity identification method based on a base model-enhanced semantic prototype according to claim 7, characterized in that, Calculate the reconstruction loss, KL divergence loss, and auxiliary classification loss. Construct a total loss function based on the weighted sum of the reconstruction loss, KL divergence loss, and auxiliary classification loss, including: Calculate the mean square error between the original sensing data and the reconstructed data, and calculate the reconstruction loss based on the mean square error; Calculate the KL divergence between the posterior distribution and the corresponding conditional prior distribution of the domain latent variable and the activity latent variable respectively, and weight each KL divergence to obtain the KL divergence loss; Based on the domain latent variables and activity latent variables, the domain classification loss and activity classification loss are calculated using the cross loss function, and the auxiliary classification loss is obtained by weighting the domain classification loss and activity classification loss. The total loss function is obtained by weighted summing of the reconstruction loss, KL divergence loss, and auxiliary classification loss.

9. The cross-domain activity identification method based on a base model-enhanced semantic prototype according to claim 1, characterized in that, The method further includes: Calculate the reconstruction error between the original sensing data and the reconstructed data, and calculate the semantic deviation between the projected active latent variables of the sample and the known category semantic prototype; After normalizing the reconstruction error, it is weighted and combined with the semantic deviation to obtain a comprehensive anomaly score. When the comprehensive anomaly score exceeds a preset threshold, the sample is determined to be an abnormal activity.

10. A cross-domain activity recognition system based on a base model enhanced with semantic prototypes, characterized in that, The system includes: The semantic prototype construction module is used to obtain text descriptions of activity categories, extract semantic features of the text descriptions using a pre-trained base model, normalize the semantic features, and construct a cross-domain shared semantic prototype space. The data decomposition module is used to acquire the raw sensing data collected by the multimodal sensor, input the raw sensing data into the decoupled variational autoencoder, and decompose the raw sensing data into domain latent variables representing the domain style and activity latent variables representing the activity content through the domain encoder and the activity encoder, respectively. The semantic alignment and activity recognition module is used to map the activity latent variables to the semantic prototype space through the projection head to obtain the mapped feature vector, and calculate the cosine similarity between the feature vector and each category of semantic prototype in the semantic prototype space to obtain the activity recognition result. The training module is optimized to calculate the reconstruction loss, KL divergence loss, and auxiliary classification loss. A total loss function is constructed based on the weighted sum of the reconstruction loss, KL divergence loss, and auxiliary classification loss. The decoupled variational autoencoder and the projector are jointly optimized using the total loss function.

Images