A key point detection continual learning method based on analytical matrix

Abstract

This invention discloses a continuous learning method for keypoint detection based on analytic matrix regression. It includes: 1. Given an initial dataset; establishing a target keypoint detection model, with the keypoint prediction layer constructed using pointwise convolution; and performing fully supervised training on the detection model using the initial dataset. 2. Reconstructing the keypoint prediction layer and simultaneously constructing an analytic matrix regression loss function. 3. Constructing a joint optimization loss function based on sequential data and designing a recursive solution strategy for output weights consistent with continuous learning. This invention solves the joint optimization loss function based on sequential data in stages, ensuring zero forgetting of learned keypoint knowledge. It reconstructs the pixel-level regression loss of keypoints into a quadratic convex optimization problem with an analytical solution, compressing training time from hours to seconds, significantly reducing computational and time overhead. This invention does not require access to historical image samples, possesses excellent privacy protection characteristics, and achieves secure data isolation.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and target key point detection technology, and relates to a key point parsing continuous learning method that takes into account data privacy protection, efficient training and zero forgetting of old knowledge. Background Technology

[0002] Keypoint detection, a fundamental task in computer vision, has wide applications in areas such as human pose estimation, medical image analysis, and human-computer interaction. Currently, on fixed datasets, detection models based on supervised learning paradigms exhibit superior detection performance. However, in real-world applications, keypoint definitions often evolve dynamically. For example, in medical diagnosis, as research progresses and the disease develops, more critical lesion sites need to be analyzed; in human-computer interaction, the task objective may gradually expand from core torso recognition to more refined limb extremity recognition. These practical needs demand the continuous learning ability of keypoint detection models, particularly the ability to accurately detect newly defined keypoints without forgetting previously learned keypoint knowledge.

[0003] To mitigate the catastrophic forgetting problem in continuous learning of key points, existing technical solutions can be mainly divided into three categories: 1) Data replay-based methods: including storing a small number of historical samples or reconstructing the distribution of old task data using generative models. However, this approach incurs additional storage overhead and poses a risk of data privacy leakage; 2) Regularization-based methods: including parameter penalty and knowledge distillation. However, this approach faces a trade-off between stability and plasticity. In long-sequence incremental tasks, conflicts in gradient update directions can lead to performance degradation and representation drift; 3) Architecture-based methods: including isolating task-specific parameters or dynamically expanding network capacity. However, as the number of new key points increases, the model size and complexity become difficult to adapt to resource-constrained computing environments.

[0004] In recent years, parsing continuous learning has performed well in incremental classification learning. However, its closed-loop solution mechanism based on global vectorized features is difficult to adapt to high-resolution keypoint heatmap regression, easily leading to the curse of dimensionality and disrupting the spatial topology upon which accurate localization depends. Therefore, this invention proposes a parsing continuous learning method for keypoints that balances data privacy protection, efficient training, and zero forgetting of prior knowledge by constructing a method suitable for high-dimensional keypoint heatmap regression. Summary of the Invention

[0005] To address the critical issues of catastrophic forgetting and data privacy leakage in existing methods for continuous learning of keypoints, this invention proposes a keypoint detection continuous learning method based on an analytical matrix approach. This method achieves zero-forgetting continuous learning of newly added keypoints without using historical data. This approach effectively solves the continuous learning obstacles caused by the dynamic evolution of keypoint definitions (e.g., the need for new lesion analysis due to disease progression, or the expansion of interaction targets to finer areas) in a wide range of applications such as human pose estimation, medical image analysis, and human-computer interaction. It also meets the practical needs of rapid model adaptation and privacy data isolation. Specifically, this invention first reconstructs the keypoint prediction layer using matrix expressions, modifying the originally complex iterative optimization loss function into a quadratic convex optimization problem with a closed-form solution. Based on this, a recursive closed-form analytical solution optimization method is designed to solve the joint optimization loss function based on sequential data in stages, thereby achieving continuous learning of newly added keypoints.

[0006] This technical solution mainly includes the following steps:

[0007] Step 1: Given a... An initial dataset of predefined key points is used; a target key point detection model is established, and the key point prediction layer is constructed using point-by-point convolution; the detection model is trained in a fully supervised manner using the initial dataset, and feature representations with generalizability are extracted.

[0008] The target keypoint detection models include, but are not limited to, mainstream deep learning backbone networks such as HRNet, ViTPose, SimpleBaseline, Stacked Hourglass, and TokenPose. Preferably, this invention primarily uses the high-resolution network HRNet and the visual Transformer architecture ViTPose as the target keypoint detection models for specific implementation and experimental verification.

[0009] Step 2: Reconstruct the key point prediction layer and construct the analytical matrix regression loss function.

[0010] The pointwise convolutional keypoint prediction layer is reconstructed into a fully connected matrix layer and then subjected to random mapping to increase its dimensionality. A pixel-level keypoint regression loss function based on matrix expression is established, and an analytical solution is obtained according to the Karush-Kuhn-Tucker (KKT) optimality condition.

[0011] Step 3: Construct a joint optimization loss function based on sequential data and design a recursive solution strategy for output weights that conforms to the continuous learning approach.

[0012] During the streaming continuous learning phase, the target key point detection model only receives newly added key point images from the current phase as input, constructs autocorrelation feature matrices and cross-correlation feature matrices, and uses closed-form solutions to obtain the output weights of the prediction layer that include newly added key point detection.

[0013] The specific implementation of step 1 is as follows:

[0014] 1-1 Given a predefined initial dataset (containing (1 key point), select the target key point detection model, where the prediction layer is built using pointwise convolution.

[0015] 1-2 The selected target keypoint detection model is trained under full supervision using the initial dataset. After training, the model is decomposed into a feature extractor. and prediction layer (by Pointwise convolution It consists of two parts. In the subsequent analytical continuous learning phase, it is frozen. The weight parameters make This is only used to extract image features with general applicability. Given the i-th input image in the initial dataset. Through feature extractor Obtain the output feature map Where C, H, and W represent the number of channels, height, and width of the feature map, respectively; and the output of the prediction layer is denoted as... The output of the ground truth labeled key point heatmap of the image is: .

[0016] The specific implementation of step 2 is as follows:

[0017] 2-1 Extracted feature maps Perform spatial dimension flattening operation The flattened feature matrix is ​​obtained. :

[0018] ,

[0019] In the prediction layer The pointwise convolutions are arranged and concatenated column-wise to obtain the output weight matrix of the prediction layer. Therefore, the original prediction layer convolution operation is equivalently replaced by a fully connected matrix layer:

[0020] .

[0021] 2-2 To achieve better representation performance, a feature dimensionality increase operation of random mapping is introduced. A random mapping matrix consisting of L randomly generated pointwise convolutions is constructed. Flattening the feature matrix using a random mapping matrix. Perform high-dimensional mapping and use activation functions The new feature representation is obtained:

[0022] .

[0023] Simultaneously, the prediction layer output weight matrix is ​​re-expressed as... .

[0024] After the dimensionality increase operation is completed in steps 2-3, the output of the prediction layer is represented in the following fully connected matrix form:

[0025] .

[0026] Let the output form of the heatmap matrix of the ground truth labeled key points of the image be:

[0027] .

[0028] Based on the above mapping, the pixel-level regression loss on the initial dataset is equivalently reconstructed into the following equality-constrained optimization problem:

[0029]

[0030] in To balance the hyperparameters of the regularization term, Denotes the Frobenius norm. The number of samples in the initial dataset. This represents the error between the predicted value and the actual value.

[0031] Based on the KKT optimality conditions, the closed-form analytical solution on the initial dataset is obtained as follows:

[0032] ;

[0033] in, Let represent the identity matrix. This calculation process completely abandons the traditional backpropagation iterative optimization, directly obtaining the globally optimal prediction layer output weights in one step through matrix inversion and multiplication. For ease of representation later, let Define the autocorrelation feature matrix:

[0034] ;

[0035] Simultaneously, define the cross-correlation feature matrix:

[0036] ;

[0037] Therefore, the optimal solution for the prediction layer output weights can be equivalently expressed as:

[0038] ;

[0039] The specific implementation of step 3 is as follows:

[0040] 3-1 Let the initial dataset be stage 0, and let the datasets for each stage be: The images and corresponding key points are different at each stage.

[0041] 3-2 In the t-th stage, the image Input frozen feature extractor In the middle, and after spatial flattening processing, it is obtained Perform random mapping to obtain the feature representation of the current stage. Therefore, the joint optimization loss function over all t+1 datasets is expressed as:

[0042]

[0043] in, , , For regularization hyperparameters, Let be the number of samples in the j-th dataset.

[0044] For the aforementioned joint optimization loss function, through the optimality condition, the closed-form solution can be obtained as follows:

[0045] ;

[0046] The autocorrelation matrix With cross-correlation mapping matrix They are respectively:

[0047] ;

[0048] .

[0049] 3-3 Based on the closed-form solution above, when entering the t-th continuous learning stage, the system only loads samples from the current stage. This dataset Calculate the feature representation of the current stage t. and the corresponding truth matrix of newly added key points. Calculate the current stage independently Thus, the keypoint prediction layer weights are constructed recursively:

[0050]

[0051] .

[0052] The beneficial effects of this invention are as follows:

[0053] This invention addresses the critical issues of catastrophic forgetting and data privacy leakage in keypoint continuous learning tasks by proposing a keypoint detection continuous learning method based on an analytical matrix. This invention achieves zero-forgetting continuous learning of newly added keypoints without using historical data, and it eliminates the need for iterative optimization based on backpropagation, enabling rapid adaptation of newly added keypoint data within a single training round. Compared with existing incremental learning techniques that rely on parameter fine-tuning, knowledge distillation, or data replay, this invention has the following three innovative aspects: (1) This invention designs a recursive closed-form analytical solution optimization method, solving the joint optimization loss function based on sequential data in stages, ensuring zero forgetting of learned keypoint knowledge. (2) The pixel-level regression loss of keypoints is reconstructed into a quadratic convex optimization problem with an analytical solution, compressing the training time from several hours to seconds, significantly reducing computational and time overhead. (3) The constructed recursive closed-form solution only depends on the training data of newly added keypoints in the current stage, without accessing historical image samples, exhibiting excellent privacy protection characteristics and achieving secure data isolation. Attached Figure Description

[0054] Figure 1 The present invention presents a task-driven incremental learning process framework for key epileptic action points designed for epilepsy monitoring scenarios.

[0055] Figure 2 The reconstruction principle and closed-form solution process of the linear analytical architecture in this invention are described in detail. Detailed Implementation

[0056] The present invention will be further described below with reference to the accompanying drawings and embodiments. It should be noted that the content of the present invention is not limited to this specific implementation.

[0057] Example:

[0058] This invention describes specific implementation schemes to address the need for video analysis of patients' epileptic seizure movements in a home environment. Epilepsy is a common chronic disease requiring long-term, even lifelong, management. Video monitoring, due to its non-contact nature, enables 24 / 7 home monitoring, making it a crucial tool for long-term care of epilepsy patients. Compared to traditional epilepsy patient diaries, video monitoring provides more detailed and traceable records. In particular, the brain is responsible for the vast majority of bodily activities; the symptoms and limb movements during a seizure are strongly correlated with the abnormal electrical discharge areas in the brain, the extent and speed of the discharge propagation. Therefore, detailed records of the patient's movements during a seizure are essential for assisting doctors in diagnosis and accurate treatment during follow-up visits.

[0059] Target key point detection can assist doctors in quickly anchoring key limb areas of observation in massive amounts of daily monitoring videos, thereby providing support for subsequent analysis of seizure movements. However, as the condition progresses and treatment advances, the key areas of observation often differ at each follow-up visit, which necessitates the continuous learning capability of the key point detection model. Furthermore, the real-time interactivity of follow-up visits requires the model to quickly and accurately detect newly added key points based on the doctor's needs. The content of this invention perfectly meets this practical requirement.

[0060] like Figure 1 and 2 As shown, the technical solution of the present invention mainly includes the following steps:

[0061] Step 1: By integrating existing public human pose datasets and a large amount of daily pose data of epilepsy patients in home environments, an initial dataset of key points of typical parts is constructed; HRNet-W32 is used as the backbone network, and the input and output feature dimensions and prediction layer structure are set to complete the fully supervised training of the model on the initial dataset.

[0062] Step 2: HRNet-W32 backbone network prediction layer reconstruction. Remove the original output convolutional prediction layers of the backbone network, retaining only the backbone as a feature extractor; introduce random mapping to increase dimensionality, reconstruct the prediction layer mapping relationship into a pixel-level keypoint regression loss function based on matrix expression, and solve for the initial analytical solution.

[0063] Step 3: During follow-up visits for epilepsy patients, the model expands the prediction layer in real-time for newly identified limb areas of interest to the doctor through a parsing process. After the doctor completes the annotation of new key points on a small sample of follow-up data, the model only receives this incremental dataset as input, calculates the current local statistics, jointly optimizes the loss function to obtain the incremental weight blocks of the newly parsed prediction layer, and recursively constructs the global prediction layer weights that include historical task data.

[0064] The specific implementation of step 1 is as follows:

[0065] 1-1 We collected existing general human posture datasets as basic feature sources, and also collected a large number of videos of daily activities and seizure postures of multiple epilepsy patients in real home environments. After frame extraction and cleaning, we constructed a home epilepsy posture dataset.

[0066] 1-2 The high-resolution network HRNet-W32 was used as the backbone network of the detection model; the images in the initial dataset were uniformly scaled and normalized to a standard aspect ratio; the input image tensor was set. After multi-stage parallel convolution and multi-scale fusion of HRNet-W32, its highest resolution branch output feature map The original prediction layer has a 32-channel input and an output of... channel Two-dimensional convolutional layer, outputting a predicted heatmap tensor. The residual between the predicted heatmap and the target ground truth is calculated using the standard mean squared error loss function. The weights of the HRNet-W32 network are then updated using the backpropagation algorithm to enable it to train and converge on the initial dataset.

[0067] The specific implementation of step 2 is as follows:

[0068] 2-1 After training in step 1, freeze all convolutional layer parameters of the HRNet-W32 backbone for feature extraction only; for the connection matrix analytical operation, the feature map Flattening along the spatial dimension, the features of a single image are reconstructed into a two-dimensional feature matrix. .

[0069] 2-2 Introducing a pre-defined high-dimensional space-preserving mapping matrix To ensure a balance between analytical accuracy and computational efficiency, a mapping dimension is defined. Using matrix multiplication to transform the characteristic matrix Projected onto a 4096-dimensional space and connected to the GeLU activation function, the design feature matrix is ​​generated. The calculation formula is as follows:

[0070] ;

[0071] 2-3 Simultaneously add standard heatmap labels Flattening is performed to obtain the target matrix. Define the prediction weight matrix to be solved. The model prediction process is transformed from convolution operations to matrix multiplication: The acquisition of the model prediction layer weights in the original HRNet-W32 network can be equivalently reconstructed into an optimization problem constrained by equality constraints:

[0072]

[0073] Based on the KKT optimality conditions, the closed-form analytical solution on the initial dataset is obtained as follows:

[0074] ;

[0075] That is, the initial autocorrelation matrix and the initial cross-correlation matrix are:

[0076] ;

[0077] ;

[0078] Thus, the initial analytical prediction layer weights are obtained. .

[0079] The specific implementation of step 3 is as follows:

[0080] 3-1 As epilepsy progresses, during follow-up visits, doctors often need to focus on specific areas for newly appearing symptoms. Based on the limited data on typical seizures collected at the current stage, doctors need to address the newly emerging symptoms... The key points are labeled, and this new data with new labels constitutes the incremental dataset for the current stage. .

[0081] 3-2 Incremental Image Input the frozen HRNet-W32 backbone network to obtain feature maps. And perform a flattening operation to obtain the feature matrix. By reusing the specific implementation 2-2 in step 2, the design matrix for the current task is obtained. .

[0082] 3-3 Model in All The optimization problem for performing joint optimization on multiple datasets is expressed as:

[0083] ;

[0084] in, Predict layer weights for global key points in all stages. This is the heatmap label matrix for the newly added key points. The autocorrelation matrix and cross-correlation mapping matrix are calculated using the incremental data from the current stage.

[0085] ;

[0086] .

[0087] The weights of the newly added key point prediction layer can be obtained. .

[0088] 3-4 Extract the global prediction layer weights saved by the model in the previous stage This allows for the recursive construction of the updated keypoint prediction layer weights. :

[0089]

[0090] .

[0091] Example 2: Results of streaming incremental experiments on a large-scale public human keypoint dataset (this example uses the classic human pose dataset MPII dataset).

[0092] Table 1. Experimental results at each stage under different networks

[0093]

[0094] It should be noted that Stage 0 is the fully supervised training stage of the target keypoint detection model on the initial dataset. Its core purpose is to build and freeze a feature extractor with generalizability. At this stage, the streaming expansion and recursive closed-form solution of the new keypoint prediction layer have not yet been introduced. Since this invention focuses on solving the problems of continuous learning, rapid adaptation, and zero forgetting of historical knowledge for newly added keypoints under streaming sequential data, in order to more intuitively and centrally reflect the evolution performance and accuracy stability of the model in the subsequent streaming continuous learning stages, the initial benchmark results of Stage 0 are omitted in Table 1, and the experimental data of each incremental learning stage from Stage 1 to Stage 4 are presented in detail.

[0095] Table 1 shows the experimental results on a large-scale public human keypoint dataset, with random dimensionality increase. Regularization hyperparameters As shown in Table 1, the proposed linear parsing detection architecture exhibits excellent keypoint detection performance across target keypoint detection models of different scales (such as the HRNet series and ViTPose series as backbone networks). Experiments demonstrate that as the incremental learning phase progresses, the features extracted by each backbone network can achieve accurate coordinate mapping through the linear parsing head of this invention. Furthermore, the detection accuracy remains stable in subsequent incremental phases, without any significant catastrophic forgetting phenomenon. This fully demonstrates that the architecture of this invention has good universal adaptability and can effectively support the continuous evolution of feature extractors of different scales in streaming incremental scenarios.

[0096] Table 2. Efficiency Comparison Based on Incremental Phase of the Same Backbone Network (Table 2)

[0097]

[0098] Table 2 compares the efficiency of incremental learning on the MPII dataset under the same HRNet-W32 backbone network and batch size of 32. The LWF and EWC methods use 15 training epochs per task, while the AIKL method uses only one parsing update. Peak memory statistics do not include memory usage during the fine-tuning phase of the basic tasks. Experimental results show that, thanks to the mechanism of requiring only one parsing update, this invention achieves a speed improvement of over 30 times and a memory saving of nearly 50% compared to traditional methods requiring 15 training epochs, significantly improving the real-time performance and deployment feasibility of incremental learning.

[0099] Figure 1This paper presents a task-driven incremental learning framework for epilepsy action key points designed for epilepsy monitoring scenarios. The diagram simulates new diagnostic needs arising over time during home monitoring through an incremental key point training data stream. After mastering predefined general human posture knowledge in the initial stage (model t), the model evolves into model t+n when faced with newly annotated specific seizure symptom data from doctors during follow-up visits. The core of this architecture lies in the dynamic expansion of the prediction layer. Through an old knowledge retention mechanism, the learned task weights are saved and concatenated with newly acquired incremental weight blocks. This ensures that the model acquires new site monitoring capabilities while simultaneously preventing the forgetting of historical key point knowledge, without revisiting historical data.

[0100] Figure 2 The reconstruction principle and closed-form solution process of the linear analytical architecture in this invention are described in detail. This process consists of two parallel logics: initialization (stage 0) and incremental expansion (stage t). In stage 0, the feature map obtained by the feature extractor is flattened into a feature matrix. Design matrix constructed by random mapping matrix The initial weights are directly calculated using the closed-form solution of the optimization problem. Subsequently, the feature extractor enters a frozen state. In the subsequent stage t, the model only needs to perform local design matrix calculations on the newly added keypoint data and parse out the corresponding incremental weight blocks. Finally, by recursively updating the weights of the prediction layers at each stage, a continuously evolving globally optimal weight matrix is ​​constructed. .

[0101] The above description, in conjunction with specific / preferred embodiments, provides a further detailed explanation of the present invention and should not be construed as limiting the specific implementation of the invention to these descriptions. Those skilled in the art can make various substitutions or modifications to these described embodiments without departing from the inventive concept, and all such substitutions or modifications should be considered within the scope of protection of the present invention. The parts of the present invention not described in detail are well-known to those skilled in the art.

Claims

1. A continuous learning method for keypoint detection based on analytic matrix, characterized in that, Includes the following steps: Step 1: Given a... An initial dataset of predefined key points; a target key point detection model is established, and the key point prediction layer is constructed using point-by-point convolution; the detection model is trained under full supervision using the initial dataset; Step 2: Reconstruct the keypoint prediction layer and construct the analytical matrix regression loss function; Step 3: Construct a joint optimization loss function based on sequential data and design a recursive solution strategy for output weights that conforms to the continuous learning approach.

2. The continuous learning method for keypoint detection based on analytic matrix as described in claim 1, characterized in that, The specific implementation of step 1 is as follows: The selected target keypoint detection model was trained under full supervision using the initial dataset; after training, the model was decomposed into feature extractors. It consists of two parts: a prediction layer and a static layer; in the subsequent analytical continuous learning phase, the static layer is frozen. The weight parameters make It is only used to extract image features with general applicability; Given the i-th input image in the initial dataset Through feature extractor Obtain the output feature map Meanwhile, let the output of the prediction layer be... The output of the ground truth labeled key point heatmap of the image is: .

3. The continuous learning method for keypoint detection based on analytic matrix as described in claim 1, characterized in that, The specific implementation of step 2 is as follows: The pointwise convolutional keypoint prediction layer is reconstructed into a fully connected matrix layer and then subjected to random mapping to increase its dimensionality. A pixel-level keypoint regression loss function based on matrix expression is established, and an analytical solution is obtained according to the KKT optimality condition.

4. The continuous learning method for keypoint detection based on analytic matrix as described in claim 2, characterized in that, The specific implementation of step 2 is as follows: 2-1 Extracted feature maps Perform spatial dimension flattening operation The flattened feature matrix is ​​obtained. ; In the prediction layer The pointwise convolutions are arranged and concatenated column-wise to obtain the output weight matrix of the prediction layer. The original prediction layer convolution operation is equivalently replaced by a fully connected matrix layer: ； 2-2 Introducing a feature dimensionality-up operation using random mapping, a random mapping matrix is ​​constructed consisting of L randomly generated pointwise convolutions. ; Flattening the feature matrix using a random mapping matrix Perform high-dimensional mapping and use activation functions The new feature representation is obtained: ； Simultaneously, the prediction layer output weight matrix is ​​re-expressed as... ; 2-3 The output of the prediction layer is represented in the following fully connected matrix form: ； Let the output form of the heatmap matrix of the ground truth labeled key points of the image be: ； Based on the above mapping, the pixel-level regression loss on the initial dataset is equivalently reconstructed into the following equality-constrained optimization problem: ； in To balance the hyperparameters of the regularization term, Denotes the Frobenius norm. The number of samples in the initial dataset. This represents the error between the predicted value and the actual value; Based on the KKT optimality conditions, the closed-form analytical solution on the initial dataset is obtained as follows: ； in, Let the identity matrix be an integer, and the globally optimal prediction layer output weights can be obtained directly in one step through matrix inversion and multiplication. Define the autocorrelation feature matrix: ； Simultaneously, define the cross-correlation feature matrix: ； Therefore, the optimal solution for the prediction layer output weights is equivalently represented as: 。 5. The continuous learning method for keypoint detection based on analytic matrix as described in claim 2, characterized in that, The specific implementation of step 3 is as follows: 3-1 Let the initial dataset be stage 0, and let the datasets for each stage be: The images and corresponding key points are different at each stage; 3-2 In the t-th stage, the image Input frozen feature extractor In the middle, and after spatial flattening processing, it is obtained Perform random mapping to obtain the feature representation of the current stage. Therefore, the joint optimization loss function over all t+1 datasets is expressed as: ； in, , , For regularization hyperparameters, Let be the number of samples in the j-th dataset; For the aforementioned joint optimization loss function, through the optimality condition, the formal closed-form solution is obtained as follows: ， The autocorrelation matrix With cross-correlation mapping matrix They are respectively: ； ； 3-3 Based on the closed-form solution above, when entering the t-th continuous learning phase, the system only loads the current phase's content. A dataset of samples ; Calculate the feature representation of the current stage t and the corresponding truth matrix of newly added key points. Calculate the current stage independently Thus, the keypoint prediction layer weights are constructed recursively: ； 。

Images