Bone behavior recognition method and system based on continuous learning

Abstract

The invention relates to a bone behavior recognition method and system based on continuous learning, belongs to the technical field of behavior recognition, and solves the problems of disastrous forgetting and classification bias in the prior art. The method comprises the following steps: constructing a skeleton behavior recognition model, wherein the skeleton behavior recognition model is used for extracting features of input skeleton sequence data and predicting a corresponding behavior type; constructing an increment task; for each incremental task, constructing a current training sample set based on a memory pool and a current incremental task sample, training the skeleton behavior recognition model on the current training sample set based on mutual information loss of a kernel function, and updating prototype features of various types according to the current incremental task; inputting the to-be-detected skeleton sequence data into the skeleton behavior recognition model to obtain features of the to-be-detected skeleton sequence data; and obtaining a behavior recognition result of the to-be-detected skeleton sequence data based on the features of the to-be-detected skeleton sequence data and the prototype features of each type. And rapid and accurate behavior identification is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of behavior recognition, and in particular to a skeletal behavior recognition method and system based on continuous learning. Background Art

[0002] Human interaction detection is a key technology in real-world scenarios such as smart education and intelligent surveillance. Skeletal data, due to its small size, low susceptibility to noise, and strong privacy protection, has been widely used in various computer vision tasks in recent years, particularly in the field of human action recognition. Skeletal action recognition aims to determine the specific actions or intentions represented by key human nodes by analyzing their spatial positions and motion trajectories.

[0003] In the field of smart education, skeletal data analysis of student behaviors such as raising hands, writing, and leaning on desks can be used to intelligently assess classroom status and analyze student learning engagement. In the field of security monitoring, risk warnings can be issued by identifying abnormal pedestrian behavior (such as falls and fighting).

[0004] Incremental learning aims to enable models to continuously learn new knowledge while retaining previous knowledge, just like humans do. However, deep neural networks commonly suffer from a phenomenon called "catastrophic forgetting" during incremental learning. This means that after learning a new task, the model's performance on old tasks drops sharply. Furthermore, in complex environments like classrooms, interactive behavior data often exhibits both "large intra-class variability" (e.g., "raising a hand" has varying postures) and "high inter-class similarity" (e.g., "raising a hand" and "adjusting glasses" are similar at certain angles). This further exacerbates the model's classification confusion problem. In particular, in incremental learning, the model tends to incorrectly classify new samples into previously learned categories, a phenomenon known as "classification bias." Summary of the Invention

[0005] In view of the above analysis, the embodiments of the present invention aim to provide a skeletal behavior recognition method and system based on continuous learning, so as to solve the problems of catastrophic forgetting and classification bias in existing skeletal behavior recognition.

[0006] On the one hand, an embodiment of the present invention provides a skeletal behavior recognition method based on continuous learning, comprising the following steps: Constructing a skeleton behavior recognition model, wherein the skeleton behavior recognition model is used to extract features of input skeleton sequence data and predict corresponding behavior types; Constructing incremental tasks; for each incremental task, constructing a current training sample set based on the memory pool and the current incremental task sample, training the skeletal behavior recognition model based on the mutual information loss of the kernel function on the current training sample set, and updating each type of prototype features according to the current incremental task; The skeleton sequence data to be detected is input into the skeleton behavior recognition model to obtain the features of the skeleton sequence data to be detected; and the behavior recognition result of the skeleton sequence data to be detected is obtained based on the features of the skeleton sequence data to be detected and various types of prototype features.

[0007] Based on the further improvement of the above method, the mutual information loss is calculated using the following formula: ; Among them, N represents the number of samples in the current training batch, Represents the features of the i-th sample extracted by the skeletal behavior recognition model, Indicates the prototype feature of the type to which the i-th sample belongs, Indicates the prototype feature of the type that is not the i-th sample, C indicates the number of types, represents the temperature hyperparameter, Represents the similarity kernel function.

[0008] Based on the further improvement of the above method, various types of prototype features are updated according to the current incremental task, including: After each training batch of the current incremental task is completed, the prototype features of each type corresponding to the training batch are smoothly updated; After the current incremental task is completed, the prototype features of the newly added types are corrected.

[0009] Based on the further improvement of the above method, the following formula is used to smoothly update the prototype features of each type corresponding to the training batch: ; in, Represents the prototype feature of the i-th type corresponding to the training batch, represents the prototype feature after smooth update, represents the temporary prototype features of the i-th type calculated based on the samples of the training batch, represents the momentum coefficient.

[0010] Based on the further improvement of the above method, the following formula is used to calibrate the newly added types of prototypes: ; in, represents the prototype feature of the i-th newly added type before correction, represents the prototype features of the ith newly added type prototype after correction, represents the prototype feature of the k-th old type, represents the weight of the kth old type, M represents the number of old types, Indicates the balance coefficient.

[0011] Based on the further improvement of the above method, the weight of the j-th old type is calculated using the following formula: ; in, represents the prototype feature of the j-th old type, represents the similarity kernel function, represents the temperature hyperparameter.

[0012] Based on the further improvement of the above method, the similarity kernel function The calculation formula is: ; ; ; in, represents the kernel function hyperparameters, Represents the L2 norm of the vector.

[0013] Based on the further improvement of the above method, the following method is used to obtain the temporary prototype features of the i-th type calculated based on the samples of the training batch: : Take the average value of the features of the samples belonging to the i-th type in the training batch and get the feature .

[0014] Based on the further improvement of the above method, the behavior recognition results of the skeleton sequence data to be detected are obtained based on the features of the skeleton sequence data to be detected and the prototype features of each type, including: Calculate the distance between the features of the bone sequence data to be detected and the prototype features of each type; The type with the closest distance is the behavior type of the skeleton sequence data to be detected.

[0015] On the other hand, an embodiment of the present invention provides a skeletal behavior recognition system based on continuous learning, comprising: A model building module is used to build a skeleton behavior recognition model, wherein the skeleton behavior recognition model is used to extract features of input skeleton sequence data and predict corresponding behavior types; An incremental training module is used to construct incremental tasks. For each incremental task, a current training sample set is constructed based on the memory pool and the current incremental task sample. The skeletal behavior recognition model is trained on the current training sample set based on the mutual information loss of the kernel function, and each type of prototype feature is updated according to the current incremental task. The recognition module is used to input the skeleton sequence data to be detected into the skeleton behavior recognition model to obtain the characteristics of the skeleton sequence data to be detected; and obtain the behavior recognition result of the skeleton sequence data to be detected based on the characteristics of the skeleton sequence data to be detected and various types of prototype features.

[0016] Compared with the prior art, the present invention constructs a skeletal behavior recognition model, constructs a current training sample set based on a memory pool for each incremental task, thereby avoiding catastrophic forgetting, trains the skeletal behavior recognition model based on the mutual information loss of a kernel function, and updates the prototype features of the type, thereby reducing the gap between intra-class features and increasing the distance between inter-class features, avoiding new categories from being identified as old categories, effectively alleviating classification bias, and achieving more accurate and effective sample classification. The skeletal sequence data to be detected is input into the trained skeletal behavior recognition model to obtain the features of the skeletal sequence data to be detected; based on the features of the skeletal sequence data to be detected and the prototype features of each type, the behavior recognition results of the skeletal sequence data to be detected can be quickly and accurately obtained. The accuracy and stability of skeletal behavior recognition in a dynamic environment are improved.

[0017] In the present invention, the above-mentioned technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of the present invention will be described in the following description, and some advantages will become apparent from the description or be learned through practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings are only used for the purpose of illustrating specific embodiments and are not to be considered as limiting the present invention. Throughout the drawings, the same reference symbols denote the same components. Figure 1 Flowchart of a skeletal behavior recognition method based on continuous learning according to an embodiment of the present invention; Figure 2 This is a block diagram of a skeletal behavior recognition system based on continuous learning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The preferred embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, and are not used to limit the scope of the present invention.

[0020] A specific embodiment of the present invention discloses a skeleton behavior recognition method based on continuous learning, such as Figure 1 As shown, the following steps are included: S1. Constructing a skeleton behavior recognition model, wherein the skeleton behavior recognition model is used to extract features of input skeleton sequence data and predict corresponding behavior types; S2. Construct incremental tasks; for each incremental task, construct a current training sample set based on the memory pool and the current incremental task sample, train the skeletal behavior recognition model based on the mutual information loss of the kernel function on the current training sample set, and update the prototype features of each type according to the current incremental task; S3. Input the skeleton sequence data to be detected into the skeleton behavior recognition model to obtain the features of the skeleton sequence data to be detected; obtain the behavior recognition result of the skeleton sequence data to be detected based on the features of the skeleton sequence data to be detected and various types of prototype features.

[0021] Compared with the prior art, the skeletal behavior recognition method based on continuous learning provided by this embodiment, by constructing a skeletal behavior recognition model, constructs a current training sample set based on a memory pool on each incremental task, thereby avoiding catastrophic forgetting, training the skeletal behavior recognition model based on the mutual information loss of the kernel function, and updating the prototype features of the type, thereby reducing the gap between intra-class features, increasing the distance between inter-class features, avoiding new categories from being identified as old categories, effectively alleviating classification bias, and achieving more accurate and effective sample classification. The skeletal sequence data to be detected is input into the trained skeletal behavior recognition model to obtain the features of the skeletal sequence data to be detected; based on the features of the skeletal sequence data to be detected and the prototype features of each type, the behavior recognition results of the skeletal sequence data to be detected can be quickly and accurately obtained. The accuracy and stability of skeletal behavior recognition in a dynamic environment are improved.

[0022] In implementation, the skeleton sequence data is a tensor containing the 3D coordinates of multiple joint points in consecutive frames.

[0023] During implementation, the constructed skeletal behavior recognition model includes a feature extractor. To extract skeletal sequence features, the feature extractor of this application adopts a spatio-temporal transformer (STTransformer) network structure.

[0024] The Spatio-Temporal Transformer (STTransformer) network structure is specifically designed to process skeletal sequence data to extract feature embeddings that can simultaneously characterize the spatial posture and temporal dynamics of the human body. The specific implementation of the STTransformer network includes the following parts: 1. Input mapping layer: Raw skeletal data is typically a tensor containing the three-dimensional coordinates (X, Y, Z) of multiple joints in consecutive frames. The input mapping layer first passes through a two-dimensional convolutional layer (Conv2d) to map the input three-channel coordinate data into a higher-dimensional feature space, for example, 64 feature channels. Subsequently, the features are processed through a batch normalization layer (BatchNorm2d) and an activation function (such as LeakyReLU) to enhance the model's nonlinear representation capabilities and training stability.

[0025] 2. Stacking of spatiotemporal attention modules: After input mapping, the features are fed into a backbone network consisting of multiple cascaded spatiotemporal attention modules. These modules are the core of the network and are responsible for learning spatiotemporal dependencies.

[0026] In one embodiment, each spatiotemporal attention module internally includes: (1) Position encoding unit: adds position information to the input features to help the model understand the sequential relationship between joint points and time frames.

[0027] (2) QKV generation layer: Through a convolutional layer, the input features are converted into the query (Query), key (Key) and value (Value) required by the attention mechanism.

[0028] (3) Attention calculation and feedforward network: The dependency between features is calculated through the self-attention mechanism, and further feature transformation is performed using feedforward networks.

[0029] Multiple spatiotemporal attention modules are stacked together, enabling the model to extract features layer by layer, from shallow to deep. For example, the first few modules may learn local limb movements, while the later modules can learn holistic, complex whole-body movements. The number of channels in the feature map also increases (for example, from 64 dimensions to 128 dimensions, and then to 256 dimensions) to carry richer semantic information.

[0030] 3. Output layer (fc): After all the spatiotemporal attention modules have processed the network, it will output a fixed-dimensional feature vector, for example, a 256-dimensional vector. This vector is the feature embedding we need to represent the core semantics of the input bone sequence.

[0031] During implementation, a prototype set is constructed. The prototype set stores the prototype features of each type. The prototype features of each type are the mean of the sample features of that type in the latent space (feature space). Initially, the prototype set is an empty set.

[0032] Construct a memory pool to store old samples. During implementation, the memory pool stores the sample data of old samples and the features extracted by the feature extractor. Initially, the memory pool is empty.

[0033] Construct incremental tasks. When implemented, online continuous category learning considers a continuous sequence of tasks from a single data stream. ,in is the dataset of the tth incremental task, and T is the total number of incremental tasks. Include labeled samples, For samples The class label of ,in is the class label set for the tth incremental task. During implementation, the class label sets of different incremental tasks are disjoint. For example, the first incremental task is used to classify "raising hand", "writing", and "adjusting glasses", while the second incremental task is used to classify "falling" and "fighting".

[0034] The present invention uses a replay method, a memory pool (example set) is used to store a small portion of the data seen in the past, and dynamically maintains the category balance of samples in the memory pool.

[0035] For each incremental task, the current training sample set is constructed based on the memory pool and the current incremental task sample.

[0036] During implementation, before the first incremental task begins, the memory pool is empty. Therefore, all samples of the first incremental task constitute the current training sample set. For subsequent incremental tasks, some samples are extracted from the memory pool and combined with the samples of the incremental task to form the current training sample set.

[0037] After obtaining the current training sample set, for each training batch of the current training sample set, the parameters of the skeletal behavior recognition model are updated based on the mutual information loss of the kernel function.

[0038] During implementation, in order to reduce the feature differences of the same class and increase the feature differences of different classes, the present invention uses the mutual information loss based on the kernel function to update the model parameters. The mutual information loss of the i-th sample is calculated using the following formula: ; Among them, N represents the number of samples in the current training batch, Represents the features of the i-th sample extracted by the skeletal behavior recognition model, Indicates the prototype feature of the type to which the i-th sample belongs, Indicates the prototype feature of the type to which the kth non-ith sample belongs, C indicates the number of types, represents the temperature hyperparameter, Represents the similarity kernel function.

[0039] It should be noted that C represents the number of types, which refers to the number of types that the model has currently learned and seen, and is also the number of types stored in the current prototype set.

[0040] The goal of the kernel-based mutual information loss is to optimize the feature extractor so that the sample's features approximate the corresponding prototype feature embedding in the latent space. Since prototype features are generally viewed as the centroid of a class in the latent space, they can serve as a common anchor for similar features to reduce intra-class variance. Similarly, by maintaining a large distance metric with the prototype feature embeddings of other classes, inter-class variance can be increased, thereby enhancing the feature differentiation between different classes.

[0041] For the i-th sample, the features of the sample and the prototype features of its type form a pair , and other types of prototype features form negative sample pairs The present invention does not require additional construction of negative sample pairs, thereby reducing the time overhead of the algorithm.

[0042] In order to combine the constructed positive and negative sample pairs, consider the mutual information of the two sample pairs, that is, the mutual information of the positive prototype feature pair set. Mutual information of the set of negative prototype feature pairs , the formula is as follows: ; ; in, represents the mutual information function.

[0043] ; in, and Represents probability.

[0044] The goal of the mutual information loss based on the kernel method is to maximize the mutual information of the set of positive prototype feature pairs. , minimize the mutual information of negative prototype feature pairs , that is, let the prototype feature set of the same type as the sample contain more sample dependency information.

[0045] Because InfoNCE loss is a lower bound of mutual information, the formula is as follows: ; in, Represents the similarity measurement function.

[0046] Therefore, the goal is converted into a goal form that can be solved by gradient descent and the Loss loss formula is given: ; Among them, sim( ) represents the similarity, represents the expected probability density function. InfoNCE loss (Information Noise-Contrastive Estimation) is different from mutual information in form, but it indirectly optimizes the mutual information between positive sample pairs by maximizing the joint probability of positive sample pairs and minimizing the marginal probability of negative sample pairs.

[0047] The existing similarity function is usually a cosine function. In order to better utilize the distribution of feature data in different high-dimensional spaces, the similarity function of the present invention adopts a kernel function.

[0048] Similarity kernel function customized by the present invention The kernel function formula is as follows: ; ; ; in, represents the kernel function hyperparameters, Represents the L2 norm of the vector.

[0049] It combines two symmetric positive definite kernel functions: Gaussian kernel and Laplace kernel, so it satisfies the symmetric positive definite property. and Mapped to different spaces. This kernel function is designed to enhance feature separability. Feature embeddings that are difficult to distinguish in the original feature space can show clearer distribution differences in different latent spaces after being mapped by this kernel function.

[0050] The similarity kernel function can map features to a higher-dimensional Hilbert space, where features that were originally difficult to distinguish become easier to separate, thereby effectively improving the recognition accuracy of the model.

[0051] Therefore, the mutual information loss is calculated using the following formula: ; Among them, N represents the number of samples in the current training batch, Represents the features of the i-th sample extracted by the skeletal behavior recognition model, Indicates the prototype feature of the type to which the i-th sample belongs, Indicates the prototype feature of the type that is not the i-th sample, C indicates the number of types, represents the temperature hyperparameter, Represents the similarity kernel function.

[0052] It should be noted that in the first batch of the first task, the initial prototype features of the corresponding category are calculated based on the data of the current batch. The calculation method is the mean of the sample features of each type in the current batch, and then the loss optimization model is calculated. Subsequent batches continue to optimize the prototype features and encoder parameters based on this initial prototype feature.

[0053] For a batch of training samples, after calculating the mutual information loss, the stochastic gradient descent (SGD) algorithm is used to optimize the parameters of the skeletal behavior recognition model.

[0054] Since the incremental tasks of the present invention are non-repetitive tasks, the label types of different tasks are different. Therefore, after the first training batch of each incremental task is completed, the feature mean of each type of sample contained in the batch is calculated to obtain the prototype feature of the type, and the prototype feature is smoothly updated in subsequent training batches. After the training of an incremental task is completed, the prototype feature of the new type is also corrected to reduce the confusion of the old category when the new category is classified. The role of the prototype feature is to prevent the encoded semantic information of the category from becoming outdated as the encoder learns in the continuous task.

[0055] Specifically, various types of prototype features are updated according to the current incremental task, including: After each training batch of the current incremental task is completed (except the first training batch), the prototype features of each type corresponding to the training batch are smoothly updated; After the current incremental task training is completed, the prototype features of the newly added types are corrected.

[0056] In order to stabilize the prototype and avoid it from fluctuating violently due to the noise of a single batch, after each training batch, the prototype features of each type corresponding to the training batch are smoothly updated.

[0057] Specifically, the following formula is used to smoothly update the prototype features of each type corresponding to the training batch: ; in, Represents the prototype feature of the i-th type corresponding to the training batch, represents the prototype feature after smooth update, represents the temporary prototype features of the i-th type calculated based on the samples of the training batch, represents the momentum coefficient, which is a value between 0 and 1.

[0058] It should be noted that the type corresponding to the training batch refers to all types of training samples in the batch.

[0059] The temporary prototype features of the i-th type calculated based on the samples of the training batch are obtained in the following way: : Take the average value of the features of the samples belonging to the i-th type in the training batch and get the feature .

[0060] During implementation, the features of the samples belonging to the i-th type in the training batch obtained by the feature extractor are averaged to obtain .

[0061] During implementation, the corresponding prototype features in the prototype set are replaced with the smoothed features.

[0062] In order to use the prototype features of the old category to explicitly calibrate the prototype features of the new category, and to solve the problem that the new category is easily identified as the old category under the new task, after the training of an incremental task is completed, the prototype features of the newly added type are calibrated using the following formula: ; in, represents the prototype feature of the i-th newly added type before correction, represents the prototype features of the ith newly added type prototype after correction, represents the prototype feature of the k-th old type, represents the weight of the kth old type, M represents the number of old types, Indicates the balance coefficient.

[0063] The new type is the type of samples included in the current incremental task, and the old type is the type of samples included in the previous incremental task.

[0064] When implemented, the correction uses all old type prototype features , by assigning a weight to each old prototype feature Achieve correction in this feature direction. It is a hyperparameter and is generally set to 0.3.

[0065] Specifically, the weight of the j-th old type is calculated using the following formula: ; in, represents the prototype feature of the j-th old type, represents the similarity kernel function, represents the temperature hyperparameter.

[0066] During implementation, weights are determined based on the similarity between the prototype features of the new category and the prototype features of the old category for the current task. The more similar the prototype features of the old category are to the prototype features of the new category, the greater their relative proportion in the correction vector should be. These weights are used to perform "same-direction correction" on the new category. By correcting the new category based on the old category, old knowledge is used to correct new knowledge. This can reduce the confusion of the old category when classifying the new category. It also reduces the deviation between the prototype features of the new category and the overall characteristics of the new category.

[0067] After each incremental task is completed, some samples will be placed in the memory pool and the memory pool will be updated.

[0068] Rather than directly storing high-dimensional raw data (e.g., a complete skeleton sequence), the memory pool stores it in a more efficient manner. In one embodiment, the memory pool maintains a separate storage unit for each learned type.

[0069] Each storage unit can store: (1) Original sample data: A small amount of original skeleton sequence data, used to re-extract features when needed.

[0070] (2) Sample feature embedding: The feature embedding corresponding to the original sample data when it was last updated. Storing this embedding can avoid repeated feature extraction in certain playback strategies and improve efficiency.

[0071] During implementation, after each incremental task is completed, some samples can be randomly extracted from the samples corresponding to the incremental task and stored in the memory pool.

[0072] In one embodiment, to accommodate the growing number of categories, the total capacity of the memory pool is fixed, but the capacity allocated to each category is dynamic. As new categories are added, the system recalculates and evenly distributes the storage quota for each category, and may randomly discard older samples that exceed the quota to ensure that the total memory usage remains constant.

[0073] During implementation, except for the first incremental task, before starting each other incremental task, it is necessary to extract some samples of the incremental task from the memory pool to form the training sample set of the incremental task.

[0074] During implementation, a class-balanced random sampling strategy can be employed. This strategy first counts all available classes in the memory pool, then randomly and evenly determines how many samples to draw from each class to achieve the preset replay batch size. Finally, a corresponding number of samples are randomly drawn from each selected class to form the replay batch. This helps prevent older classes with large sample sizes from dominating the training process during replay.

[0075] When the model completes the training of all incremental tasks, a trained skeletal behavior recognition model and a final, fully optimized and corrected global prototype feature set are obtained.

[0076] The skeleton sequence data to be detected is input into the skeleton behavior recognition model to obtain the features of the skeleton sequence data to be detected, and the behavior recognition results of the skeleton sequence data to be detected are obtained based on the features of the skeleton sequence data to be detected and various types of prototype features.

[0077] Based on the features of the skeleton sequence data to be detected and the prototype features of each type, the behavior recognition results of the skeleton sequence data to be detected are obtained, including: Calculate the distance between the features of the bone sequence data to be detected and the prototype features of each type; The type with the closest distance is the behavior type of the skeleton sequence data to be detected.

[0078] During implementation, the distance between the features of the skeleton sequence data to be detected and the prototype features of each type is calculated, such as the Euclidean distance, and the type with the closest distance is the behavior type of the skeleton sequence data to be detected.

[0079] The continuous learning-based skeletal behavior recognition method of the present invention alleviates the intra-class diversity and inter-class similarity of data samples in classroom behavior learning, effectively aggregates the features of samples of the same category, and promotes the effective classification of samples of different categories. At the same time, it can effectively alleviate catastrophic forgetting and classification bias in continuous learning of skeletal behavior recognition, and improve the accuracy and stability of skeletal behavior recognition in dynamic environments.

[0080] A specific embodiment of the present invention discloses a skeleton behavior recognition system based on continuous learning, such as Figure 2 Shown, including: A model building module is used to build a skeleton behavior recognition model, wherein the skeleton behavior recognition model is used to extract features of input skeleton sequence data and predict corresponding behavior types; An incremental training module is used to construct incremental tasks. For each incremental task, a current training sample set is constructed based on the memory pool and the current incremental task sample. The skeletal behavior recognition model is trained on the current training sample set based on the mutual information loss of the kernel function, and each type of prototype feature is updated according to the current incremental task. The recognition module is used to input the skeleton sequence data to be detected into the skeleton behavior recognition model to obtain the characteristics of the skeleton sequence data to be detected; and obtain the behavior recognition result of the skeleton sequence data to be detected based on the characteristics of the skeleton sequence data to be detected and various types of prototype features.

[0081] The above method embodiment and system embodiment are based on the same principle, and their related parts can be used for reference, and can achieve the same technical effect. The specific implementation process can be found in the above embodiment and will not be repeated here.

[0082] Electronic device embodiment: A specific implementation of the present application discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the skeletal behavior recognition method based on continuous learning in the method embodiment are implemented.

[0083] Readable storage medium embodiment: A specific implementation of the present application discloses a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the skeletal behavior recognition method based on continuous learning in the method embodiment are implemented.

[0084] Those skilled in the art will appreciate that all or part of the process steps of the above-described embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, such as a magnetic disk, an optical disk, a read-only memory, or a random access memory.

[0085] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed by the present invention should be covered by the scope of protection of the present invention.

Claims

1. A skeleton behavior recognition method based on continuous learning, characterized in that: The following steps are involved: Constructing a skeleton behavior recognition model, wherein the skeleton behavior recognition model is used to extract features of input skeleton sequence data and predict corresponding behavior types; Constructing incremental tasks; for each incremental task, constructing a current training sample set based on the memory pool and the current incremental task sample, training the skeletal behavior recognition model based on the mutual information loss of the kernel function on the current training sample set, and updating each type of prototype features according to the current incremental task; The skeleton sequence data to be detected is input into the skeleton behavior recognition model to obtain the features of the skeleton sequence data to be detected; and the behavior recognition result of the skeleton sequence data to be detected is obtained based on the features of the skeleton sequence data to be detected and various types of prototype features.

2. The skeleton behavior recognition method based on continuous learning according to claim 1, characterized in that The mutual information loss is calculated using the following formula: ； Among them, N represents the number of samples in the current training batch, Represents the features of the i-th sample extracted by the skeletal behavior recognition model, Indicates the prototype feature of the type to which the i-th sample belongs, Indicates the prototype feature of the type that is not the i-th sample, C indicates the number of types, represents the temperature hyperparameter, Represents the similarity kernel function.

3. The skeleton behavior recognition method based on continuous learning according to claim 1 is characterized in that Update various types of prototype features based on the current incremental task, including: After each training batch of the current incremental task is completed, the prototype features of each type corresponding to the training batch are smoothly updated; After the current incremental task is completed, the prototype features of the newly added types are corrected.

4. The skeleton behavior recognition method based on continuous learning according to claim 3 is characterized in that: The following formula is used to smoothly update the prototype features of each type corresponding to the training batch: ； in, Represents the prototype feature of the i-th type corresponding to the training batch, represents the prototype feature after smooth update, represents the temporary prototype features of the i-th type calculated based on the samples of the training batch, represents the momentum coefficient.

5. The skeleton behavior recognition method based on continuous learning according to claim 3 is characterized in that: The following formula is used to calibrate the newly added prototype: ； in, represents the prototype feature of the i-th newly added type before correction, represents the prototype features of the ith newly added type prototype after correction, represents the prototype feature of the k-th old type, represents the weight of the kth old type, M represents the number of old types, Indicates the balance coefficient.

6. The skeleton behavior recognition method based on continuous learning according to claim 5, characterized in that: The weight of the j-th old type is calculated using the following formula: ； in, represents the prototype feature of the j-th old type, represents the similarity kernel function, represents the temperature hyperparameter.

7. The skeleton behavior recognition method based on continuous learning according to any one of claims 2 or 5, characterized in that: Similarity kernel function The calculation formula is: ； ； ； in, represents the kernel function hyperparameters, Represents the L2 norm of the vector.

8. The skeleton behavior recognition method based on continuous learning according to claim 4 is characterized in that: The temporary prototype features of the i-th type calculated based on the samples of the training batch are obtained in the following way: : Take the average value of the features of the samples belonging to the i-th type in the training batch and get the feature .

9. The skeleton behavior recognition method based on continuous learning according to claim 1, characterized in that: Based on the features of the skeleton sequence data to be detected and the prototype features of each type, the behavior recognition results of the skeleton sequence data to be detected are obtained, including: Calculate the distance between the features of the bone sequence data to be detected and the prototype features of each type; The type with the closest distance is the behavior type of the skeleton sequence data to be detected.

10. A skeleton behavior recognition system based on continuous learning, characterized in that: include: A model building module is used to build a skeleton behavior recognition model, wherein the skeleton behavior recognition model is used to extract features of input skeleton sequence data and predict corresponding behavior types; An incremental training module is used to construct incremental tasks. For each incremental task, a current training sample set is constructed based on the memory pool and the current incremental task sample. The skeletal behavior recognition model is trained on the current training sample set based on the mutual information loss of the kernel function, and each type of prototype feature is updated according to the current incremental task. The recognition module is used to input the skeleton sequence data to be detected into the skeleton behavior recognition model to obtain the characteristics of the skeleton sequence data to be detected; and obtain the behavior recognition result of the skeleton sequence data to be detected based on the characteristics of the skeleton sequence data to be detected and various types of prototype features.

Images