A method and system for skeletal behavior recognition based on continuous learning

Abstract

This invention relates to a skeletal behavior recognition method and system based on continuous learning, belonging to the field of behavior recognition technology, and solves the problems of catastrophic forgetting and classification bias in existing technologies. The method includes: constructing a skeletal behavior recognition model, which is used to extract features from input skeletal sequence data and predict the corresponding behavior type; constructing incremental tasks; for each incremental task, constructing a current training sample set based on a memory pool and current incremental task samples, training the skeletal behavior recognition model on the current training sample set using mutual information loss based on a kernel function, and updating the prototype features of each type according to the current incremental task; inputting the skeletal sequence data to be detected into the skeletal behavior recognition model to obtain the features of the skeletal sequence data to be detected; and obtaining the behavior recognition result of the skeletal sequence data to be detected based on the features of the skeletal sequence data to be detected and the prototype features of each type. This achieves fast and accurate behavior recognition.

Description

Technical Field

[0001] This invention relates to the field of behavior recognition technology, and in particular to a skeletal behavior recognition method and system based on continuous learning. Background Technology

[0002] In real-world scenarios such as smart education and intelligent surveillance, human interaction detection is a crucial technology. Skeletal data, due to its advantages such as small size, resistance to noise interference, and strong privacy protection, has been widely applied in various computer vision tasks in recent years, especially in the field of human behavior recognition. Skeletal behavior recognition aims to determine the specific actions or intentions represented by analyzing the spatial positions and movement trajectories of key human body nodes.

[0003] In the field of smart education, skeletal data analyzing student behaviors such as raising hands, writing, and leaning on desks can be used to intelligently assess classroom status and analyze student engagement. In the field of security monitoring, risk warnings can be issued by identifying abnormal pedestrian behaviors (such as falls or fights).

[0004] Incremental learning aims to enable models to learn new knowledge while retaining old knowledge, much like humans do. However, deep neural networks commonly suffer from "catastrophic forgetting" during incremental learning, meaning that the model's performance on older tasks drops sharply after learning new ones. Furthermore, in complex environments such as classrooms, human interaction data often exhibits both high intra-class variability (e.g., different hand-raising gestures) and high inter-class similarity (e.g., "raising a hand" and "adjusting glasses" appear similar from certain angles). This further exacerbates the model's classification confusion problem, especially in incremental learning, where the model tends to incorrectly classify new samples into previously learned categories—a phenomenon known as "classification bias." Summary of the Invention

[0005] Based on the above analysis, the embodiments of the present invention aim to provide a skeletal behavior recognition method and system based on continuous learning, in order to solve the problems of catastrophic forgetting and classification bias in existing skeletal behavior recognition.

[0006] On one hand, embodiments of the present invention provide a skeletal behavior recognition method based on continuous learning, comprising the following steps:

[0007] A skeletal behavior recognition model is constructed, which is used to extract features from input skeletal sequence data and predict the corresponding behavior type;

[0008] Construct incremental tasks; for each incremental task, construct the current training sample set based on the memory pool and the current incremental task samples, train the skeletal behavior recognition model on the current training sample set based on the mutual information loss of the kernel function, and update the prototype features of each type according to the current incremental task;

[0009] The bone sequence data to be detected is input into the bone behavior recognition model to obtain the features of the bone sequence data to be detected; based on the features of the bone sequence data to be detected and the prototype features of each type, the behavior recognition result of the bone sequence data to be detected is obtained.

[0010] Based on the further improvement of the above method, the mutual information loss is calculated using the following formula:

[0011] ;

[0012] Where N represents the number of samples in the current training batch. This represents the feature extracted from the i-th sample by the skeletal behavior recognition model. The prototype feature represents the type to which the i-th sample belongs. Let C represent the prototype features of the type that does not belong to the i-th sample, and let C represent the number of types. This indicates temperature hyperparameters. This represents the similarity kernel function.

[0013] Based on the further improvements to the above method, the prototype features of each type are updated according to the current incremental task, including:

[0014] After each training batch of the current incremental task is completed, the prototype features of each type corresponding to that training batch are smoothly updated.

[0015] After the current incremental task is completed, the prototype features of the newly added types are corrected.

[0016] Based on the above method, a further improvement is made, and the following formula is used to smoothly update the prototype features of each type corresponding to the training batch:

[0017] ;

[0018] in, This represents the prototype feature of the i-th type corresponding to this training batch. This represents the prototype feature after a smooth update. This represents the temporary prototype feature of the i-th type calculated based on the samples in this training batch. This represents the momentum coefficient.

[0019] Based on further improvements to the above method, the following formula is used to correct the prototype of the newly added type:

[0020] ;

[0021] in, This represents the prototype feature of the i-th newly added type before correction. This represents the prototype feature after prototype correction for the i-th newly added type. This represents the prototype feature of the k-th old type. This represents the weight of the k-th old type, and M represents the number of old types. This represents the balance coefficient.

[0022] Based on the further improvement of the above method, the weight of the j-th old type is calculated using the following formula:

[0023] ;

[0024] in, This represents the prototype feature of the j-th old type. Represents the similarity kernel function. This indicates the temperature hyperparameter.

[0025] Based on further improvements to the above method, the similarity kernel function The calculation formula is:

[0026] ;

[0027] ;

[0028] ;

[0029] in, This represents the kernel function hyperparameters. This represents the L2 norm of a vector.

[0030] Based on the further improvement of the above method, the temporary prototype feature of the i-th type calculated from the samples of this training batch is obtained in the following way. :

[0031] The features of samples belonging to the i-th type in the training batch are averaged to obtain the feature values. .

[0032] Based on further improvements to the above method, behavior recognition results of the skeletal sequence data to be detected are obtained based on the features of the skeletal sequence data to be detected and the prototype features of various types, including:

[0033] Calculate the distance between the features of the bone sequence data to be detected and the prototype features of each type;

[0034] The closest type is the behavior type of the skeletal sequence data to be detected.

[0035] On the other hand, embodiments of the present invention provide a skeletal behavior recognition system based on continuous learning, comprising:

[0036] The model building module is used to build a skeletal behavior recognition model, which is used to extract features from the input skeletal sequence data and predict the corresponding behavior type.

[0037] The incremental training module is used to construct incremental tasks. For each incremental task, a current training sample set is constructed based on the memory pool and the current incremental task samples. The skeletal behavior recognition model is trained on the current training sample set based on the mutual information loss of the kernel function. The prototype features of each type are updated according to the current incremental task.

[0038] The recognition module is used to input the bone sequence data to be detected into the bone behavior recognition model to obtain the features of the bone sequence data to be detected; and to obtain the behavior recognition result of the bone sequence data to be detected based on the features of the bone sequence data to be detected and the prototype features of various types.

[0039] Compared with existing technologies, this invention constructs a skeletal behavior recognition model. On each incremental task, it builds the current training sample set based on a memory pool, thus avoiding catastrophic forgetting. The skeletal behavior recognition model is trained using mutual information loss based on kernel functions, and the prototype features of each type are updated, thereby reducing the difference in intra-class features and increasing the distance between inter-class features. This prevents new categories from being identified as old categories, effectively alleviating classification bias and achieving more accurate and effective sample classification. The skeletal sequence data to be detected is input into the trained skeletal behavior recognition model to obtain the features of the skeletal sequence data. Based on the features of the skeletal sequence data and the prototype features of each type, the behavior recognition result of the skeletal sequence data can be obtained quickly and accurately. This improves the accuracy and stability of skeletal behavior recognition in dynamic environments.

[0040] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description

[0041] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0042] Figure 1 This is a flowchart of a skeletal behavior recognition method based on continuous learning, as described in an embodiment of the present invention.

[0043] Figure 2 This is a block diagram of a skeletal behavior recognition system based on continuous learning, according to an embodiment of the present invention. Detailed Implementation

[0044] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0045] A specific embodiment of the present invention discloses a skeletal behavior recognition method based on continuous learning, such as... Figure 1 As shown, it includes the following steps:

[0046] S1. Construct a skeletal behavior recognition model, which is used to extract features from the input skeletal sequence data and predict the corresponding behavior type;

[0047] S2. Construct incremental tasks; For each incremental task, construct the current training sample set based on the memory pool and the current incremental task samples, train the skeletal behavior recognition model on the current training sample set based on the mutual information loss of the kernel function, and update the prototype features of each type according to the current incremental task.

[0048] S3. Input the bone sequence data to be detected into the bone behavior recognition model to obtain the features of the bone sequence data to be detected; based on the features of the bone sequence data to be detected and the prototype features of each type, obtain the behavior recognition result of the bone sequence data to be detected.

[0049] Compared with existing technologies, the skeletal behavior recognition method based on continuous learning provided in this embodiment avoids catastrophic forgetting by constructing a skeletal behavior recognition model and building the current training sample set based on a memory pool for each incremental task. It trains the skeletal behavior recognition model based on mutual information loss using kernel functions and updates the prototype features of each type, thereby reducing the gap between intra-class features and increasing the distance between inter-class features. This prevents new categories from being identified as old categories, effectively alleviating classification bias and achieving more accurate and effective sample classification. The skeletal sequence data to be detected is input into the trained skeletal behavior recognition model to obtain the features of the skeletal sequence data. Based on the features of the skeletal sequence data and the prototype features of each type, the behavior recognition result of the skeletal sequence data can be obtained quickly and accurately. This improves the accuracy and stability of skeletal behavior recognition in dynamic environments.

[0050] In practice, the skeletal sequence data is a tensor containing the three-dimensional coordinates of multiple joints in consecutive frames.

[0051] In implementation, the constructed skeletal behavior recognition model includes a feature extractor. To extract skeletal sequence features, the feature extractor in this application adopts a spatio-temporal Transformer (STTransformer) network structure.

[0052] The Spatio-Temporal Transformer (STTransformer) network structure is specifically designed to process skeletal sequence data to extract feature embeddings that simultaneously characterize human spatial pose and temporal dynamics. A concrete implementation of this STTransformer network may include the following components:

[0053] 1. Input Mapping Layer:

[0054] The original skeletal data is typically a tensor containing the 3D coordinates (X, Y, Z) of multiple joints across consecutive frames. This input mapping layer first passes the input 3-channel coordinate data through a 2D convolutional layer (Conv2d) to map it to a higher-dimensional feature space, for example, to 64 feature channels. Subsequently, the features are processed through a batch normalization layer (BatchNorm2d) and activation functions (such as LeakyReLU) to enhance the model's non-linear expressiveness and training stability.

[0055] 2. Stacking of spatiotemporal attention modules:

[0056] The features, after being mapped from the input, are fed into a backbone network consisting of multiple cascaded spatiotemporal attention modules. These modules are the core of the network and are responsible for learning spatiotemporal dependencies.

[0057] In one embodiment, each spatiotemporal attention module contains:

[0058] (1) Position encoding unit: adds position information to the input features to help the model understand the sequential relationship between key points and time frames.

[0059] (2) QKV generation layer: Through a convolutional layer, the input features are converted into queries, keys and values ​​required by the attention mechanism.

[0060] (3) Attention calculation and feedforward network: The dependency relationship between features is calculated through the self-attention mechanism, and further feature transformation is performed using feedforward networks, etc.

[0061] Multiple spatiotemporal attention modules are stacked together, enabling the model to extract features layer by layer, from shallow to deep. For example, the first few modules may learn local limb movements, while the later modules can learn overall, complex full-body movements. The number of channels in the feature map also increases accordingly (e.g., from 64 dimensions to 128 dimensions, and then to 256 dimensions) to carry richer semantic information.

[0062] 3. Output layer (fc):

[0063] After processing by all the spatiotemporal attention modules, the network outputs a fixed-dimensional feature vector, for example, a 256-dimensional vector. This vector is the feature embedding we need, which represents the core semantics of the input skeletal sequence.

[0064] During implementation, a prototype set is constructed, which stores the prototype features of each type. Each prototype feature is the mean of the sample features of that type in the latent space (feature space). Initially, the prototype set is empty.

[0065] Construct a memory pool to store old samples. During implementation, the memory pool stores the sample data of the old samples along with the features extracted by the feature extractor. Initially, the memory pool is empty.

[0066] Construct incremental tasks. In practice, online continuous category learning considers a sequence of consecutive tasks from a single data stream. ,in Let T be the dataset for the t-th incremental task, where T is the total number of incremental tasks. Dataset Include A labeled sample, For the sample Class label, ,in Let be the set of class labels for the t-th incremental task. During implementation, the set of class labels for different incremental tasks do not overlap. For example, the first incremental task is used to classify "raising hand", "writing", and "adjusting glasses", while the second incremental task is used to classify "falling down" and "fighting".

[0067] This invention uses a replay method, with a memory pool (example set) used to store a small portion of previously seen data and dynamically maintain the class balance of samples in the memory pool.

[0068] For each incremental task, the current training sample set is constructed based on the memory pool and the current incremental task samples.

[0069] During implementation, the memory pool is empty before the first incremental task begins. Therefore, all samples from the first incremental task constitute the current training sample set. For subsequent incremental tasks, a portion of samples are extracted from the memory pool and combined with the samples from that incremental task to form the current training sample set.

[0070] After obtaining the current training sample set, for each training batch of the current training sample set, the parameters of the skeletal behavior recognition model are updated based on the mutual information loss of the kernel function.

[0071] In implementation, to reduce feature differences within the same class and increase feature differences between different classes, this invention employs kernel-based mutual information loss to update model parameters. The mutual information loss for the i-th sample is calculated using the following formula:

[0072] ;

[0073] Where N represents the number of samples in the current training batch. This represents the feature extracted from the i-th sample by the skeletal behavior recognition model. The prototype feature represents the type to which the i-th sample belongs. Let C represent the prototype feature of the k-th sample that is not of the i-th sample, and let C represent the number of types. This indicates temperature hyperparameters. This represents the similarity kernel function.

[0074] It should be noted that C represents the number of types, which refers to the number of types that the model has learned or seen, and is also the number of types stored in the current prototype set.

[0075] The mutual information loss method based on kernels aims to optimize the feature extractor so that the features of a sample approximate their corresponding prototype feature embeddings in the latent space. Since prototype features are generally regarded as centroids of types in the latent space, they can serve as common anchors for features of the same class to reduce intra-class variance. Similarly, inter-class variance can be increased by maintaining a greater distance metric from prototype feature embeddings of other classes, thereby enhancing the feature differences between different classes.

[0076] For the i-th sample, construct a pair of features between the sample and the prototype features of its type. Together with other types of prototype features, they form negative sample pairs. This invention eliminates the need to construct additional negative sample pairs, thereby reducing the algorithm's time overhead.

[0077] To combine the constructed positive and negative sample pairs, we consider the mutual information between the two sample pair sets, i.e., the mutual information between the positive prototype feature pair sets. Mutual information between sets of negative prototype features The formula is as follows:

[0078] ;

[0079] ;

[0080] in, Represents the mutual information function.

[0081] ;

[0082] in, and It represents probability.

[0083] The goal of kernel-based mutual information loss is to maximize the mutual information of the set of positive prototype feature pairs. Minimize the negative prototype features for set mutual information That is, to allow the set of prototype features of the same class as the sample to contain more sample dependency information.

[0084] Because the InfoNCE loss is a lower bound of mutual information, the formula is as follows:

[0085] ;

[0086] in, This represents the similarity measurement function.

[0087] Therefore, the objective is transformed into a form that can be solved by gradient descent, and the loss formula is given:

[0088] ;

[0089] Where, sim( ) represents similarity. This represents the expectation of the probability density function. It is InfoNCE loss (Information Noise-Contrastive Estimation). Although it is different from mutual information in form, it indirectly optimizes the mutual information between positive sample pairs by maximizing the joint probability for positive sample pairs and minimizing the marginal probability for negative sample pairs.

[0090] Existing similarity functions are usually cosine functions. In order to better utilize the distribution of feature data in different high-dimensional spaces, the similarity function of this invention adopts a kernel function.

[0091] The similarity kernel function defined in this invention The kernel function formula is as follows:

[0092] ;

[0093] ;

[0094] ;

[0095] in, This represents the kernel function hyperparameters. This represents the L2 norm of a vector.

[0096] It combines two symmetric positive definite kernel functions: the Gaussian kernel and the Laplace kernel, thus satisfying symmetric positive definiteness. The kernel function maps the characteristics... and Each feature is mapped to a different space. This kernel function is designed to enhance the separability of features. Feature embeddings that are difficult to distinguish in the original feature space can exhibit clearer distribution differences in different latent spaces after being mapped by this kernel function.

[0097] Similarity kernel functions can map features to a higher-dimensional Hilbert space, in which features that were originally difficult to distinguish become easier to separate, thereby effectively improving the model's recognition accuracy.

[0098] Therefore, the mutual information loss is calculated using the following formula:

[0099] ;

[0100] Where N represents the number of samples in the current training batch. This represents the feature extracted from the i-th sample by the skeletal behavior recognition model. The prototype feature represents the type to which the i-th sample belongs. Let C represent the prototype features of the type that does not belong to the i-th sample, and let C represent the number of types. This indicates temperature hyperparameters. This represents the similarity kernel function.

[0101] It should be noted that in the first batch of the first task, the initial prototype features of the corresponding category are calculated based on the data of the current batch. The calculation method is the mean of the sample features of each type in the current batch. Then, the loss is calculated to optimize the model. Subsequent batches continue to optimize the prototype features and encoder parameters based on these initial prototype features.

[0102] For a batch of training samples, after calculating the mutual information loss, the Stochastic Gradient Descent (SGD) algorithm is used to optimize the parameters of the skeletal behavior recognition model.

[0103] Since the incremental tasks of this invention are non-repeating tasks, and the label types differ for different tasks, after the first training batch of each incremental task, the mean feature value of each type of sample in that batch is calculated to obtain the prototype feature for that type. Subsequent training batches then smoothly update the prototype feature. After training for an incremental task is completed, the prototype feature for the new type is also corrected to reduce confusion with the old category when classifying the new category. The role of the prototype feature is to prevent the encoded semantic information of that category from becoming outdated as the encoder learns in continuous tasks.

[0104] Specifically, based on the current incremental task, update the prototype features of each type, including:

[0105] After each training batch of the current incremental task is completed (except for the first training batch), the prototype features of each type corresponding to that training batch are smoothly updated.

[0106] After the current incremental task training is completed, the prototype features of the newly added types are corrected.

[0107] To stabilize the prototype and prevent it from fluctuating drastically due to noise in a single batch, the prototype features for each type corresponding to each training batch are smoothly updated after each training batch ends.

[0108] Specifically, the following formula is used to smoothly update the prototype features of each type corresponding to this training batch:

[0109] ;

[0110] in, This represents the prototype feature of the i-th type corresponding to this training batch. This represents the prototype feature after a smooth update. This represents the temporary prototype feature of the i-th type calculated based on the samples in this training batch. This represents the momentum coefficient, which is a value between 0 and 1.

[0111] It should be noted that the type corresponding to this training batch refers to all types to which the training samples in this batch belong.

[0112] The temporary prototype features of the i-th type calculated from the samples of this training batch are obtained in the following manner. :

[0113] The features of samples belonging to the i-th type in the training batch are averaged to obtain the feature values. .

[0114] During implementation, the average value of the features belonging to the i-th type in the training batch obtained by the feature extractor is taken. .

[0115] During implementation, the corresponding prototype features in the prototype set are replaced with the smoothed features.

[0116] To explicitly calibrate the prototype features of the new category using the prototype features of the old category, and to address the issue of the new category being easily identified as the old category under the new task, the prototype features of the newly added type are corrected using the following formula after training on an incremental task:

[0117] ;

[0118] in, This represents the prototype feature of the i-th newly added type before correction. This represents the prototype feature after prototype correction for the i-th newly added type. This represents the prototype feature of the k-th old type. This represents the weight of the k-th old type, and M represents the number of old types. This represents the balance coefficient.

[0119] The new type refers to the type of samples included in the current incremental task, while the old type refers to the type of samples included in previous incremental tasks.

[0120] During implementation, the correction used all the old types of prototype features. By assigning weights to each old prototype feature To achieve correction in this feature direction. This is a hyperparameter, typically set to 0.3.

[0121] Specifically, the weight of the j-th old type is calculated using the following formula:

[0122] ;

[0123] in, This represents the prototype feature of the j-th old type. Represents the similarity kernel function. This indicates the temperature hyperparameter.

[0124] In implementation, weights are determined based on the similarity between the prototype features of the new category and the prototype features of the old category under the current task. The more similar an old category prototype feature is to the new category prototype feature, the greater its relative weight in the correction vector should be compared to other old category prototype features. These weights are used to perform "co-directional correction" on the new type. By correcting the new type based on the old type, and using old knowledge to correct new knowledge, the confusion of old categories during new category classification can be reduced. This also reduces the deviation between the prototype features of the new category and the overall features of the new category.

[0125] After each incremental task is completed, a portion of the samples will be placed into the memory pool, and the memory pool will be updated.

[0126] Instead of directly storing high-dimensional raw data (such as complete skeletal sequences), the memory pool stores it in a more efficient way. In one embodiment, the memory pool maintains a separate storage unit for each learned type.

[0127] Each storage unit can store the following simultaneously:

[0128] (1) Original sample data: A small portion of the original skeletal sequence data, used to re-extract features when needed.

[0129] (2) Sample feature embedding: The feature embeddings corresponding to these original sample data when they were last updated. Storing this embedding can avoid repeated feature extraction in some playback strategies and improve efficiency.

[0130] During implementation, after each incremental task is completed, a portion of the samples corresponding to that incremental task can be randomly extracted and stored in the memory pool.

[0131] In one embodiment, to accommodate scenarios where the number of categories is constantly increasing, the total capacity of the memory pool is fixed, but the capacity allocated to each category is dynamic. As a new category is added, the system recalculates and evenly distributes the storage quota for each category, and may randomly discard some old samples that exceed the quota to ensure that the total memory usage remains constant.

[0132] During implementation, except for the first incremental task, each additional incremental task requires the extraction of a portion of the samples from the memory pool before starting to form the training sample set for that incremental task.

[0133] In implementation, a class-balanced random sampling strategy can be adopted. This strategy first counts all available classes in the memory pool, then randomly and evenly determines how many samples to draw from each class to achieve the preset replay batch size. Finally, a corresponding number of samples are randomly drawn from each selected class to form the replay batch. This helps prevent older classes with larger sample sizes from dominating the training process during replay.

[0134] Once the model has completed training for all incremental tasks, it yields a trained skeletal behavior recognition model, as well as a final, fully optimized and corrected global prototype feature set.

[0135] The bone sequence data to be detected is input into the bone behavior recognition model to obtain the features of the bone sequence data to be detected. Based on the features of the bone sequence data to be detected and the prototype features of each type, the behavior recognition result of the bone sequence data to be detected is obtained.

[0136] Based on the features of the skeletal sequence data to be detected and the prototype features of various types, the behavior recognition results of the skeletal sequence data to be detected are obtained, including:

[0137] Calculate the distance between the features of the bone sequence data to be detected and the prototype features of each type;

[0138] The closest type is the behavior type of the skeletal sequence data to be detected.

[0139] During implementation, the distance between the features of the skeletal sequence data to be detected and the prototype features of each type is calculated, such as Euclidean distance. The type with the closest distance is the behavior type of the skeletal sequence data to be detected.

[0140] The skeletal behavior recognition method based on continuous learning in this invention alleviates the intra-class diversity and inter-class similarity of data samples in classroom behavior learning, achieving effective aggregation of features of samples of the same category and promoting effective classification of samples of different categories. Simultaneously, it effectively mitigates catastrophic forgetting and classification bias in the continuous learning process of skeletal behavior recognition, improving the accuracy and stability of skeletal behavior recognition in dynamic environments.

[0141] A specific embodiment of the present invention discloses a skeletal behavior recognition system based on continuous learning, such as... Figure 2 As shown, it includes:

[0142] The model building module is used to build a skeletal behavior recognition model, which is used to extract features from the input skeletal sequence data and predict the corresponding behavior type.

[0143] The incremental training module is used to construct incremental tasks. For each incremental task, a current training sample set is constructed based on the memory pool and the current incremental task samples. The skeletal behavior recognition model is trained on the current training sample set based on the mutual information loss of the kernel function. The prototype features of each type are updated according to the current incremental task.

[0144] The recognition module is used to input the bone sequence data to be detected into the bone behavior recognition model to obtain the features of the bone sequence data to be detected; and to obtain the behavior recognition result of the bone sequence data to be detected based on the features of the bone sequence data to be detected and the prototype features of various types.

[0145] The above-described method and system embodiments are based on the same principles, and their related aspects can be referenced from each other to achieve the same technical effects. For specific implementation processes, please refer to the foregoing embodiments, which will not be repeated here.

[0146] Electronic device example:

[0147] One specific implementation of this application discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the skeletal behavior recognition method based on continuous learning in the method embodiment.

[0148] Examples of readable storage media:

[0149] One specific implementation of this application discloses a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the skeletal behavior recognition method based on continuous learning in the method embodiment.

[0150] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0151] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A skeletal behavior recognition method based on continuous learning, characterized in that, Includes the following steps: A skeletal behavior recognition model is constructed, which is used to extract features from input skeletal sequence data and predict the corresponding behavior type; Construct incremental tasks; for each incremental task, construct the current training sample set based on the memory pool and the current incremental task samples, train the skeletal behavior recognition model on the current training sample set based on the mutual information loss of the kernel function, and update the prototype features of each type according to the current incremental task; The bone sequence data to be detected is input into the bone behavior recognition model to obtain the features of the bone sequence data to be detected; the behavior recognition results of the bone sequence data to be detected are obtained based on the features of the bone sequence data to be detected and the prototype features of each type. The mutual information loss is calculated using the following formula: ； Where N represents the number of samples in the current training batch. This represents the feature extracted from the i-th sample by the skeletal behavior recognition model. The prototype feature represents the type to which the i-th sample belongs. Let C represent the prototype features of the type that does not belong to the i-th sample, and let C represent the number of types. This indicates temperature hyperparameters. Represents the similarity kernel function; Update the prototype features of each type based on the current incremental task, including: After each training batch of the current incremental task is completed, the prototype features of each type corresponding to that training batch are smoothly updated. After the current incremental task is completed, the prototype features of the newly added types are corrected.

2. The skeletal behavior recognition method based on continuous learning according to claim 1, characterized in that, The prototype features for each type corresponding to this training batch are updated smoothly using the following formula: ； in, This represents the prototype feature of the i-th type corresponding to this training batch. This represents the prototype feature after a smooth update. This represents the temporary prototype feature of the i-th type calculated based on the samples in this training batch. This represents the momentum coefficient.

3. The skeletal behavior recognition method based on continuous learning according to claim 1, characterized in that, The prototype of the new type is corrected using the following formula: ； in, This represents the prototype feature of the i-th newly added type before correction. This represents the prototype feature after prototype correction for the i-th newly added type. This represents the prototype feature of the k-th old type. This represents the weight of the k-th old type, and M represents the number of old types. This represents the balance coefficient.

4. The skeletal behavior recognition method based on continuous learning according to claim 3, characterized in that, The weight of the j-th old type is calculated using the following formula: ； in, This represents the prototype feature of the j-th old type. Represents the similarity kernel function. This indicates the temperature hyperparameter.

5. The skeletal behavior recognition method based on continuous learning according to claim 1, characterized in that, Similarity kernel function The calculation formula is: ； ； ； in, This represents the kernel function hyperparameters. This represents the L2 norm of a vector.

6. The skeletal behavior recognition method based on continuous learning according to claim 2, characterized in that, The temporary prototype features of the i-th type calculated from the samples of this training batch are obtained in the following manner. : The features of samples belonging to the i-th type in the training batch are averaged to obtain the feature values. .

7. The skeletal behavior recognition method based on continuous learning according to claim 1, characterized in that, Based on the features of the skeletal sequence data to be detected and the prototype features of various types, the behavior recognition results of the skeletal sequence data to be detected are obtained, including: Calculate the distance between the features of the bone sequence data to be detected and the prototype features of each type; The closest type is the behavior type of the skeletal sequence data to be detected.

8. A skeletal behavior recognition system based on continuous learning, characterized in that, include: The model building module is used to build a skeletal behavior recognition model, which is used to extract features from the input skeletal sequence data and predict the corresponding behavior type. The incremental training module is used to construct incremental tasks. For each incremental task, a current training sample set is constructed based on the memory pool and the current incremental task samples. The skeletal behavior recognition model is trained on the current training sample set based on the mutual information loss of the kernel function. The prototype features of each type are updated according to the current incremental task. The recognition module is used to input the bone sequence data to be detected into the bone behavior recognition model to obtain the features of the bone sequence data to be detected; and to obtain the behavior recognition result of the bone sequence data to be detected based on the features of the bone sequence data to be detected and the prototype features of various types. The mutual information loss is calculated using the following formula: ； Where N represents the number of samples in the current training batch. This represents the feature extracted from the i-th sample by the skeletal behavior recognition model. The prototype feature represents the type to which the i-th sample belongs. Let C represent the prototype features of the type that does not belong to the i-th sample, and let C represent the number of types. This indicates temperature hyperparameters. Represents the similarity kernel function; Update the prototype features of each type based on the current incremental task, including: After each training batch of the current incremental task is completed, the prototype features of each type corresponding to that training batch are smoothly updated. After the current incremental task is completed, the prototype features of the newly added types are corrected.

Images