A lightweight lipreading continual learning method based on fine-grained spatiotemporal perception

Abstract

The application provides a kind of light-weighted lip speech continual learning method based on fine-grained spatiotemporal perception, and relates to the technical field of artificial intelligence.The method uses Sigmoid activation function to capture multiple key pronunciation postures through feature difference enhancement processing;Adopt hierarchical difference distillation strategy, transfer old model knowledge from three dimensions of original features, key frames and class prediction stability, effectively alleviate catastrophic forgetting;Combined with two-stage training strategy, the orthogonality of inter-class features is constrained by orthogonal remapping in the second stage to correct the bias of the classifier with minimal overhead.The application does not need to store the original video, can effectively alleviate the problem of catastrophic forgetting, and realizes efficient lip speech class incremental learning while protecting user privacy.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, and particularly relates to the field of lip reading technology, especially a lightweight continuous lip reading learning method based on fine-grained spatiotemporal perception. Background Technology

[0002] Lip reading is a technology that identifies speech content by analyzing the movement of a speaker's lips. It has broad application prospects in areas such as silent speech recognition, speech enhancement in noisy environments, and assisting communication for the hearing impaired. With the development of deep learning technology, neural network-based lip reading methods have made significant progress. Their core principle lies in extracting the lip region features of a speaker from a video sequence and mapping them to the corresponding text or phoneme sequence. However, in practical applications, lip reading systems often need to continuously adapt to new speakers, new vocabulary, or new languages, thus involving the problem of incremental learning.

[0003] Incremental learning, a key form of incremental learning, aims to enable models to continuously learn new categories while maintaining their ability to recognize already learned categories. A core challenge in incremental learning is the catastrophic forgetting problem, where the model's performance on older categories often significantly decreases when learning new ones. This problem is particularly pronounced in lip-reading because different speakers exhibit significantly different lip movement patterns, and the introduction of new vocabulary requires the model to flexibly expand its recognition capabilities.

[0004] Unlike typical image or video recognition tasks, lip-reading is a highly fine-grained spatiotemporal recognition task. In image classification, a single frame typically contains sufficient static features for recognition; however, in lip-reading, a single frame contains almost no linguistic information, and the identity information of words is encoded in the subtle temporal dynamics of lip movements. This fine-grained characteristic manifests as high inter-class similarity and significant intra-class variability, directly leading to the instability of standard incremental learning methods. Because these methods are not specifically designed to handle such subtle visual differences, the inherently fragile decision boundaries between similar words are easily disrupted during model updates, thus accelerating catastrophic forgetting.

[0005] Existing incremental learning methods mainly fall into three categories: playback-based methods, regularization-based methods, and architecture-based methods. Playback-based methods mitigate the forgetting problem by storing samples or features of some old categories for joint training during new task training. However, in lip-reading scenarios, due to privacy concerns, storing the user's original video data is often not feasible, limiting the application of traditional playback methods. Regularization-based methods protect old knowledge by constraining model parameter updates, but these methods typically struggle to handle large category increments. Architecture-based methods dynamically expand the network structure to accommodate new categories, but this can lead to a continuous increase in model size, hindering practical deployment.

[0006] Therefore, how to achieve efficient incremental learning for lip reading recognition while protecting user privacy, and at the same time effectively alleviate the problem of catastrophic forgetting, has become a pressing technical challenge. Summary of the Invention

[0007] The purpose of this invention is to address the shortcomings or deficiencies of the existing technologies by proposing a lightweight continuous lip-reading learning method based on fine-grained spatiotemporal awareness. Through three core components—Feature Distinctiveness Boosting (FDB), Hierarchical Discrepancy Distillation (HDD), and Orthogonal Remapping (OR)—this method effectively alleviates the catastrophic forgetting problem without requiring the storage of all historical video data, only needing to retain a very small number of representative samples.

[0008] The technical solution to achieve the purpose of this invention is: a lightweight continuous lip-reading learning method based on fine-grained spatiotemporal awareness, the method comprising the following steps:

[0009] Step 1: Acquire lip-reading video data and preprocess it to obtain lip-reading video sequences, while maintaining a memory buffer containing representative samples of the learned categories;

[0010] Step 2: Construct a feature extraction network to extract short-term and long-term features from the lip-reading video sequence;

[0011] Step 3: Use a two-stage training strategy to incrementally train the lip-reading model:

[0012] The first stage of training involves extracting fused features using feature difference enhancement processing and combining it with hierarchical difference distillation processing to constrain the knowledge consistency between the old and new models, thereby collaboratively optimizing the feature extraction network and the classifier.

[0013] Second-stage training: Based on the features obtained in the first stage, a balanced dataset is constructed, and the feature space distribution is optimized and the classifier is updated through orthogonal remapping.

[0014] Specifically, the hierarchical differential distillation process and the second-stage training will only be started for non-first tasks, that is, when the number of tasks t>1, the hierarchical differential distillation process and the second-stage training process will be triggered.

[0015] Step 4: Save the parameters of the trained lip-reading model and update the memory buffer;

[0016] Step 5: For the lip-reading video data to be learned, return to step 1.

[0017] Furthermore, step 1 specifically includes:

[0018] Step 1-1: Obtain video data containing the speaker's lip movements and their corresponding labels, and convert the video into a grayscale image;

[0019] Steps 1-2: Perform face detection and locate lip key points for each frame of the video, and calculate the bounding box containing the complete lip region;

[0020] Steps 1-3: Crop the lip region image according to the bounding box and adjust it to a uniform size, and perform pixel normalization processing;

[0021] Steps 1-4: During the training phase, perform data augmentation on the training data, including at least random horizontal flipping, random cropping, and standard color normalization.

[0022] Steps 1-5: Construct and update the memory buffer storing representative samples for knowledge distillation in subsequent tasks; wherein, for the first task, the memory buffer is empty by default; the representative samples have custom constraints.

[0023] Furthermore, the feature extraction network in step 2 includes:

[0024] Front-end feature extraction network It consists of 3D convolutional layers and residual networks, and is used to extract short-term temporal features;

[0025] Backend Temporal Modeling Network It consists of a bidirectional recurrent neural network, which extracts long-term time-dependent features based on short-term features.

[0026] Furthermore, step 3, before performing incremental task training, also includes adaptive expansion of the classifier structure:

[0027] Get the number of new categories in the current task;

[0028] Expand the output layer dimension of the classifier, retain the weight parameters corresponding to the old category, and initialize the weight parameters corresponding to the new category.

[0029] Furthermore, the feature difference enhancement processing in step 3, the first stage of training, specifically includes:

[0030] Step 3-1, for short-term characteristics Perform global information aggregation and use the Sigmoid activation function to calculate the importance score for each frame;

[0031] Step 3-2: Select the top K frames as keyframe features based on their importance scores in descending order. ;

[0032] Step 3-3: Concatenate and fuse the keyframe features with the average pooling representation of long-term features to obtain the fused features. ;

[0033] Steps 3-4: Use weighted supervision to compare losses Constraints are imposed on the fusion features by introducing weights based on the number of valid samples for each class to balance the contributions of the old and new classes.

[0034] Furthermore, the formula for calculating the number of valid samples for each category in steps 3-4 is as follows:

[0035]

[0036] In the formula, This represents the number of valid samples in category c. This indicates the number of samples of category c in the current dataset. This indicates the strength of the control weight.

[0037] Furthermore, the hierarchical difference distillation process in step 3, the first stage of training, specifically includes:

[0038] Original Feature Consistency Distillation: Calculate the mean squared error loss between the new model and the old model on short-term and long-term features, denoted as the original feature consistency loss. ;

[0039] Keyframe knowledge inheritance distillation: Using the temporal index of keyframes identified by the old model, the new model is constrained to reproduce the feature cues of the old model at the same positions, and the keyframe knowledge inheritance loss is calculated. ;

[0040] Class prediction stability distillation: By using binary cross-entropy in conjunction with the Sigmoid activation function, the output probability distribution of the new model for the old class is constrained to remain consistent with that of the old model, resulting in class prediction stability loss. ;

[0041] Total loss of stratified distillation :

[0042]

[0043] In the formula, , and These are the weight coefficients for the original feature consistency loss, keyframe knowledge inheritance loss, and class prediction stability loss, respectively.

[0044] The total loss function for the first stage of training in step 3 is: :

[0045]

[0046] In the formula, It is the standard cross-entropy loss used for classification.

[0047] Furthermore, the specific process of step 3, the second phase of training, includes:

[0048] Steps 3-5: Construct a balanced dataset by selecting an equal amount of sample features for each new and old category from the current task data and the memory buffer; specifically: obtain old category features from the memory buffer through a feature extraction network, and obtain an equal amount of new category features from the current task data;

[0049] Steps 3-6: Perform orthogonal remapping to project the features to an orthogonal subspace;

[0050] Steps 3-7, using orthogonal loss Explicitly constrain the orthogonality between feature representations of different categories, and update only the orthogonal mapping layer and classifier parameters.

[0051] Furthermore, the total loss function for the second stage of training in step 3 is: :

[0052]

[0053] In the formula, It is the standard cross-entropy loss used for classification;

[0054] in,

[0055]

[0056] In the formula, It is about balancing the number of inter-class pairings in the dataset. It is the label corresponding to the i-th sample. It is the label corresponding to the j-th sample. It is the first The long-term features corresponding to each sample are concatenated after average pooling and time-maximum pooling, and then normalized by L2. It is the first The long-term features corresponding to each sample are concatenated after average pooling and time-maximum pooling, and then normalized by L2. This indicates the matrix transpose.

[0057] Furthermore, in steps 1-5 and step 4, a distance-based selection strategy is used to update the memory buffer, specifically including:

[0058] Calculate the feature vector of all samples in each category;

[0059] Calculate the average feature vector of this category and use it as the class center;

[0060] Sort the samples according to their Euclidean distance from the class center from closest to furthest, and select the top m samples as the most representative samples to store in the memory buffer.

[0061] Compared with the prior art, the significant advantages of this invention are:

[0062] (1) This invention utilizes feature difference enhancement processing and employs the Sigmoid activation function to calculate the importance score of keyframes, enabling multiple non-adjacent frames to simultaneously obtain high importance scores, thereby effectively capturing multiple key pronunciation postures in lip-reading videos. At the same time, combined with weighted supervised contrast loss, it enhances the distinguishability between different categories of lip-reading features and improves the model's ability to identify similar lip movement patterns.

[0063] (2) This invention employs a hierarchical differential distillation method to perform knowledge distillation at three levels: consistency of original features, inheritance of keyframe knowledge, and stability of class prediction. Among these, the class prediction stability distillation method uses a combination of binary cross-entropy and the Sigmoid activation function to independently evaluate the activation intensity of each category, effectively avoiding the suppression of old category activation by new category, fully preserving the knowledge of the old model, and effectively alleviating the catastrophic forgetting problem.

[0064] (3) The present invention utilizes a two-stage training strategy to achieve the orthogonal remapping objective. The first stage focuses on feature learning and knowledge preservation, while the second stage freezes the feature extractor and trains only the lightweight mapping layer and classifier. Orthogonal loss is used to constrain the orthogonality between features of different categories, effectively reducing the interference between new and old category features, while introducing only minimal computational overhead.

[0065] (4) This invention does not require storing the user's original video data. It achieves incremental learning of classes only through knowledge distillation and orthogonal constraints at the feature level, which effectively protects user privacy and is suitable for application scenarios with high requirements for privacy protection.

[0066] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0067] Figure 1 This is an overall flowchart of a lightweight lip-reading continuous learning method based on fine-grained spatiotemporal awareness in one embodiment.

[0068] Figure 2 This is a schematic diagram of the process for differential distillation in one embodiment, wherein... Figure 2 (a) in the text represents the original characteristic consistency distillation. Figure 2 (b) in the diagram represents the distillation of keyframe knowledge inheritance. Figure 2 (c) in the example represents the predictive stability distillation.

[0069] Figure 3 This is a comparison chart of algorithm accuracy under three different scenarios for a subset i-LRW of 200 classes of the public dataset LRW in one embodiment. Figure 3 (a) represents scenario B0-Task20. Figure 3 (b) in the diagram represents the B0-Task10 scenario. Figure 3 (c) in the text refers to scenario B100-Task10.

[0070] Figure 4 This is a comparison chart of algorithm accuracy under three different scenarios for a 200-class subset i-LRW1000 of the public dataset LRW1000 in one embodiment. Figure 4 (a) represents scenario B0-Task20. Figure 4 (b) in the diagram represents the B0-Task10 scenario. Figure 4 (c) in the text refers to scenario B100-Task10. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0072] It should be noted that if the embodiments of the present invention involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of the components in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.

[0073] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0074] In one embodiment, a lightweight lip-reading continuous learning method based on fine-grained spatiotemporal awareness is provided, the method comprising the following steps:

[0075] Step 1: Acquire lip-reading video data and preprocess it to obtain lip-reading video sequences, while maintaining a memory buffer containing representative samples of the learned categories;

[0076] Step 2: Construct a feature extraction network to extract short-term and long-term features from the lip-reading video sequence;

[0077] Step 3: Use a two-stage training strategy to incrementally train the lip-reading model:

[0078] The first stage of training involves extracting fused features using feature difference enhancement processing and combining it with hierarchical difference distillation processing to constrain the knowledge consistency between the old and new models, thereby collaboratively optimizing the feature extraction network and the classifier.

[0079] Second-stage training: Based on the features obtained in the first stage, a balanced dataset is constructed, and the feature space distribution is optimized and the classifier is updated through orthogonal remapping.

[0080] Specifically, the hierarchical differential distillation process and the second-stage training will only be started for non-first tasks, that is, when the number of tasks t>1, the hierarchical differential distillation process and the second-stage training process will be triggered.

[0081] Step 4: Save the parameters of the trained lip-reading model and update the memory buffer;

[0082] Step 5: For the lip-reading video data to be learned, return to step 1.

[0083] Furthermore, in one embodiment, step 1 specifically includes:

[0084] Step 1-1: Obtain video data containing the speaker's lip movements and their corresponding labels, and convert the video into a grayscale image;

[0085] Steps 1-2: Perform face detection and locate lip key points for each frame of the video, and calculate the bounding box containing the complete lip region;

[0086] Steps 1-3: Crop the lip region image according to the bounding box and adjust it to a uniform size, and perform pixel normalization processing;

[0087] Steps 1-4: During the training phase, perform data augmentation on the training data, including at least random horizontal flipping, random cropping, and standard color normalization.

[0088] Steps 1-5: Construct and update the memory buffer storing representative samples for knowledge distillation in subsequent tasks; wherein, for the first task, the memory buffer is empty by default; the representative samples have custom constraints.

[0089] Furthermore, in one embodiment, the feature extraction network in step 2 includes:

[0090] Front-end feature extraction network It consists of 3D convolutional layers and residual networks, and is used to extract short-term temporal features;

[0091] Backend Temporal Modeling Network It consists of a bidirectional recurrent neural network, which extracts long-term time-dependent features based on short-term features.

[0092] Preferably, in some embodiments, step 2 specifically includes the following process:

[0093] Step 2-1: Construct a front-end feature extraction network consisting of 3DCNN and ResNet-34 to extract local temporal patterns;

[0094] Step 2-2: Construct a backend temporal modeling network composed of bidirectional GRUs to capture long-term temporal dependencies;

[0095] Steps 2-3 involve inputting the preprocessed video into a front-end feature extraction network to extract short-term features. ;

[0096] Steps 2-4: Input short-term features into the back-end temporal modeling network to extract long-term features. ;

[0097] Steps 2-5 involve performing average pooling on the long-term features along the time dimension to obtain the global representation. .

[0098] Furthermore, in one embodiment, step 3 further includes adaptive expansion of the classifier structure before performing incremental task training:

[0099] Get the number of new categories in the current task;

[0100] Expand the output layer dimension of the classifier, retain the weight parameters corresponding to the old category, and initialize the weight parameters corresponding to the new category.

[0101] Furthermore, in one embodiment, the feature differential enhancement processing in step 3, the first stage of training, specifically includes:

[0102] Step 3-1, for short-term characteristics Perform global information aggregation and use the Sigmoid activation function to calculate the importance score for each frame;

[0103] Step 3-2: Select the top K frames as keyframe features based on their importance scores in descending order. ;

[0104] Step 3-3: Concatenate and fuse the keyframe features with the average pooling representation of long-term features to obtain the fused features. ;

[0105] Steps 3-4: Use weighted supervision to compare losses Constraints are imposed on the fusion features by introducing weights based on the number of valid samples for each class to balance the contributions of the old and new classes.

[0106] Preferably, in some embodiments, the formula for calculating the number of valid samples by category in steps 3-4 is:

[0107]

[0108] In the formula, This represents the number of valid samples in category c. This indicates the number of samples of category c in the current dataset. This indicates the strength of the control weight.

[0109] Preferably, in some embodiments, step 3-4 utilizes a weighted supervised contrast loss. Represented as:

[0110]

[0111] in, Indicates the first The fusion features of each sample Indicates the first The fusion features of each sample Indicates the first The fusion features of each sample Indicates matrix transpose. This indicates temperature hyperparameters. Indicates the label corresponding to the i-th sample. The sample-level weights are used to combat class imbalance by giving higher importance to underrepresented classes; This represents a combined dataset.

[0112] Here, preferably, sample-level weights Based on its category-level weight The calculated category-level weights The calculation formula is:

[0113]

[0114] In the formula, This indicates the total number of categories that have been learned; Indicates category The number of valid samples, Indicates category The number of valid samples.

[0115] Specifically, in some embodiments, the feature difference enhancement process in step 3 includes:

[0116] S3011, for short-term characteristics Average pooling is performed along the feature dimension, and the importance score for each frame is calculated independently using the Sigmoid activation function. ;

[0117] S3012, select the highest score based on importance score. The frame is used as a keyframe to obtain keyframe features. ;

[0118] S3013, Sort the selected keyframes in descending order of their importance scores;

[0119] S3014, keyframe features Flattened and global representation By concatenating the data and obtaining fused features through a linear layer, the desired features can be obtained. ;

[0120] S3015, Calculating Weighted Supervision and Contrast Loss Based on Fuded Features Sample-level weights are introduced to compensate for class imbalance.

[0121] Furthermore, in one embodiment, the hierarchical differential distillation process in step 3, the first stage of training, specifically includes:

[0122] Original Feature Consistency Distillation: Calculate the mean squared error loss between the new model and the old model on short-term and long-term features, denoted as the original feature consistency loss. ;

[0123] Keyframe knowledge inheritance distillation: Using the temporal index of keyframes identified by the old model, the new model is constrained to reproduce the feature cues of the old model at the same positions, and the keyframe knowledge inheritance loss is calculated. ;

[0124] Class prediction stability distillation: By using binary cross-entropy in conjunction with the Sigmoid activation function, the output probability distribution of the new model for the old class is constrained to remain consistent with that of the old model, resulting in class prediction stability loss. ;

[0125] Total loss of stratified distillation :

[0126]

[0127] In the formula, , and These are the weight coefficients for the original feature consistency loss, keyframe knowledge inheritance loss, and class prediction stability loss, respectively.

[0128] The total loss function for the first stage of training in step 3 is: :

[0129]

[0130] In the formula, It is the standard cross-entropy loss used for classification.

[0131] It should be noted that when the task ordinal number is greater than 1, the hierarchical difference distillation process in the first stage of training is initiated.

[0132] Specifically, in some embodiments, the hierarchical distillation process specifically includes:

[0133] S3021, calculate the mean squared error between the short-term and long-term features of the new and old models to obtain the original feature consistency loss. ;

[0134] S3022, Keyframe Index Identified Using the Older Model The mean squared error between the features of the new model at the corresponding position and the keyframe features of the old model is calculated to obtain the keyframe knowledge inheritance loss. ;

[0135] S3023, apply Sigmoid activation to the output logits of the new model for the old class and the output logits of the old model respectively, and calculate the binary cross-entropy loss of the two to obtain the class prediction stability loss. ;

[0136] S3024, the total loss is obtained by weighted summation of the above three losses. .

[0137] Furthermore, in one embodiment, the specific process of step 3, the second stage of training, includes:

[0138] Steps 3-5: Construct a balanced dataset by selecting an equal amount of sample features for each new and old category from the current task data and the memory buffer; specifically: obtain old category features from the memory buffer through a feature extraction network, and obtain an equal amount of new category features from the current task data;

[0139] Steps 3-6: Perform orthogonal remapping to project the features to an orthogonal subspace;

[0140] Steps 3-7, using orthogonal loss Explicitly constrain the orthogonality between feature representations of different categories, and update only the orthogonal mapping layer and classifier parameters.

[0141] Furthermore, in one embodiment, the total loss function for the second stage of training in step 3 is: :

[0142]

[0143] In the formula, It is the standard cross-entropy loss used for classification;

[0144] in,

[0145]

[0146] In the formula, It is about balancing the number of inter-class pairings in the dataset. It is the label corresponding to the i-th sample. It is the label corresponding to the j-th sample. It is the first The long-term features corresponding to each sample are concatenated after average pooling and time-maximum pooling, and then normalized by L2. It is the first The long-term features corresponding to each sample are concatenated after average pooling and time-maximum pooling, and then normalized by L2. This indicates the matrix transpose.

[0147] Specifically, in some embodiments, constructing a balanced dataset specifically includes:

[0148] S3041, features are extracted by the feature extraction network trained in the first stage;

[0149] S3042: Select samples from the memory buffer for the old category; select samples closest to the class mean for the new category. Each instance ensures that each class contains an equal number of feature representations;

[0150] S3043, apply time average pooling and time max pooling to long-term features in sequence, and then concatenate the two results;

[0151] S3044, perform L2 normalization on the concatenated features to obtain the feature vector h projected onto the orthogonal subspace.

[0152] Specifically, in some embodiments, the orthogonal remapping specifically includes:

[0153] S3051, freeze the feature extraction network, allowing only the orthogonal mapping layer and classifier parameters to be updated;

[0154] S3052 calculates the orthogonal loss by calculating the cross-correlation between feature representations of different categories in the balanced dataset. To constrain the orthogonality between classes.

[0155] Preferably, in some embodiments, steps 1-5 and step 4, which update the memory buffer using a distance-based selection strategy, specifically include:

[0156] Calculate the feature vector of all samples in each category;

[0157] Calculate the average feature vector of this category and use it as the class center;

[0158] Sort the samples according to their Euclidean distance from the class center from closest to furthest, and select the top m samples as the most representative samples to store in the memory buffer.

[0159] Compared with the prior art, the present invention has the following characteristics:

[0160] (1) Significantly improved system reliability and stability under fine-grained recognition tasks.

[0161] For the highly fine-grained temporal task of lip reading recognition, standard incremental learning methods in the background art are unable to handle subtle visual differences, leading to easily corrupted decision boundaries. This invention achieves a quantitative breakthrough in performance metrics through the synergistic effect of three components: FDB, HDD, and OR.

[0162] Accuracy Improvement: In the B100-Task10 scenario of the i-LRW dataset, the average reliability (average accuracy) of the present invention is improved by 12.35% compared with the state-of-the-art (SOTA) method TPAN; on the i-LRW1000 dataset, the average accuracy is 8.27% higher than TPAN.

[0163] Enhanced stability: This invention significantly reduces the performance degradation rate, decreasing it by 16.08% compared to TPAN on the i-LRW dataset and by 12.56% compared to FOSTER on the i-LRW1000 dataset. This demonstrates the invention's strong resistance to forgetting in long sequence incremental tasks.

[0164] (2) It achieves a balance between lightweight model and resource efficiency, making it suitable for edge deployment.

[0165] In the background, architecture-based methods typically learn new classes by dynamically expanding the network, resulting in a linear increase in the number of parameters with the task. This invention employs a strategy with a fixed backbone network:

[0166] Constant model size: After completing all incremental tasks, the number of parameters in this invention is only 30.82M, which is only a slight increase compared to the lightest baseline method (28.51M), thus avoiding the memory explosion problem.

[0167] High-performance inference: Since the orthogonal remapping layer is only a single-layer linear structure, the inference latency is kept at around 9.22ms, and the throughput reaches 204.92FPS, which is on par with the basic lightweight model and can meet the real-time requirements of interactive devices.

[0168] The training cost is extremely low: the second-stage orthogonal remapping training completely bypasses the deep video backbone network and operates only on the lightweight one-dimensional feature vector, with its actual training time accounting for only about 8% of the total training time.

[0169] (3) It solves the problems of classifier bias and catastrophic forgetting from the structural and theoretical levels.

[0170] Robustness of keyframe extraction: Unlike the competitive weight suppression generated by Softmax in the background technology, this invention uses the Sigmoid activation function to independently evaluate the importance of each frame, which theoretically allows the capture of multiple non-adjacent key articulation gestures (such as the start and end actions of articulation), thereby extracting more discriminative features.

[0171] Geometric separation of feature space: By introducing an orthogonal remapping stage, the orthogonal loss is used to explicitly constrain the feature representations of different categories to be orthogonal to each other in geometric space, which theoretically maximizes the inter-class separation and effectively corrects the classifier bias caused by data imbalance in incremental learning.

[0172] Completeness of knowledge transfer: Hierarchical differential distillation not only preserves the final prediction logic, but also forces the new model to reproduce the features of the old model at specific time anchors through keyframe knowledge inheritance, ensuring accurate memory of fine-grained visual cues.

[0173] (4) Enhanced privacy protection capabilities

[0174] Compared to methods in the background that may require replaying a large number of original video samples, this invention primarily relies on feature-level differential enhancement and hierarchical distillation. By rigorously screening the stored samples (retaining only a very small number of representative samples) and combining this with feature remapping techniques, the dependence on original data storage is reduced, making it more conducive to achieving edge-side online learning while protecting privacy.

[0175] In one embodiment, a lightweight lip-reading continuous learning system based on fine-grained spatiotemporal awareness is provided, the system comprising sequentially executed:

[0176] The first module is used to acquire lip-reading video data and preprocess it to obtain lip-reading video sequences, while maintaining a memory buffer containing representative samples of the learned categories.

[0177] The second module is used to construct a feature extraction network to extract short-term and long-term features of the lip-reading video sequence;

[0178] The third module is used to perform incremental task training on the lip-reading model using a two-stage training strategy:

[0179] The first stage of training involves extracting fused features using feature difference enhancement processing and combining it with hierarchical difference distillation processing to constrain the knowledge consistency between the old and new models, thereby collaboratively optimizing the feature extraction network and the classifier.

[0180] Second-stage training: Based on the features obtained in the first stage, a balanced dataset is constructed, and the feature space distribution is optimized and the classifier is updated through orthogonal remapping.

[0181] The fourth module is used to save the parameters of the trained lip-reading model and update the memory buffer;

[0182] The fifth module is used to return to the execution of the first module for the lip-reading video data to be learned.

[0183] Specific limitations regarding the lightweight lip-reading continuous learning system based on fine-grained spatiotemporal awareness can be found in the limitations of the lightweight lip-reading continuous learning method based on fine-grained spatiotemporal awareness mentioned above, and will not be repeated here. Each module in the aforementioned lightweight lip-reading continuous learning system based on fine-grained spatiotemporal awareness can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0184] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements:

[0185] Step 1: Acquire lip-reading video data and preprocess it to obtain lip-reading video sequences, while maintaining a memory buffer containing representative samples of the learned categories;

[0186] Step 2: Construct a feature extraction network to extract short-term and long-term features from the lip-reading video sequence;

[0187] Step 3: Use a two-stage training strategy to incrementally train the lip-reading model:

[0188] The first stage of training involves extracting fused features using feature difference enhancement processing and combining it with hierarchical difference distillation processing to constrain the knowledge consistency between the old and new models, thereby collaboratively optimizing the feature extraction network and the classifier.

[0189] Second-stage training: Based on the features obtained in the first stage, a balanced dataset is constructed, and the feature space distribution is optimized and the classifier is updated through orthogonal remapping.

[0190] Step 4: Save the parameters of the trained lip-reading model and update the memory buffer;

[0191] Step 5: For the lip-reading video data to be learned, return to step 1.

[0192] For specific limitations on each step, please refer to the limitations of the lightweight lip-reading continuous learning method based on fine-grained spatiotemporal awareness mentioned above, which will not be repeated here.

[0193] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program being implemented when executed by a processor:

[0194] Step 1: Acquire lip-reading video data and preprocess it to obtain lip-reading video sequences, while maintaining a memory buffer containing representative samples of the learned categories;

[0195] Step 2: Construct a feature extraction network to extract short-term and long-term features from the lip-reading video sequence;

[0196] Step 3: Use a two-stage training strategy to incrementally train the lip-reading model:

[0197] The first stage of training involves extracting fused features using feature difference enhancement processing and combining it with hierarchical difference distillation processing to constrain the knowledge consistency between the old and new models, thereby collaboratively optimizing the feature extraction network and the classifier.

[0198] Second-stage training: Based on the features obtained in the first stage, a balanced dataset is constructed, and the feature space distribution is optimized and the classifier is updated through orthogonal remapping.

[0199] Step 4: Save the parameters of the trained lip-reading model and update the memory buffer;

[0200] Step 5: For the lip-reading video data to be learned, return to step 1.

[0201] For specific limitations on each step, please refer to the limitations of the lightweight lip-reading continuous learning method based on fine-grained spatiotemporal awareness mentioned above, which will not be repeated here.

[0202] As a specific example, the invention will be described in detail in one embodiment.

[0203] This embodiment provides a lightweight lip-reading continuous learning method based on fine-grained spatiotemporal awareness. The method includes four main steps: data acquisition and preprocessing, feature extraction, model training, and model storage. The model training step employs a two-stage training strategy: the first stage includes feature difference enhancement and hierarchical difference distillation, and the second stage includes orthogonal remapping. Through this clear hierarchical structure, this invention can achieve efficient incremental learning for lip-reading recognition without storing the original video data, effectively mitigating the catastrophic forgetting problem.

[0204] The method of this invention is applicable to incremental learning tasks in lip-reading scenarios. It is assumed that during training, the model needs to learn sequentially... Each task Includes a new set of categories In the first During training for a task, the model can access the training data for the current task. and accumulated sample memory buffer The goal of this invention is to achieve incremental class learning by maintaining the ability to recognize already learned classes while learning new classes.

[0205] (1) Data acquisition and preprocessing steps (step S100)

[0206] Data acquisition and preprocessing are fundamental to incremental learning in lip-reading recognition. Their purpose is to acquire high-quality lip-reading video data and convert it into a format suitable for neural network processing. The specific implementation process of this step is as follows:

[0207] Step S101: Lip Reading Video Data Acquisition. Acquire video data containing the speaker's lip movements. The video data can be a real-time video stream or a pre-recorded video file. Each video sample corresponds to a label representing the word or sentence spoken by the speaker. In the class incremental learning scenario, the video data for the current task contains new class labels that do not overlap with the class labels from previous tasks. Input video samples ,in Indicates the number of frames. and These represent the height and width of the frame, respectively. This indicates the number of channels. To reduce computational load, the input video is usually converted to a grayscale image. =1.

[0208] Step S102: Lip Region Detection and Localization. Face detection is performed on each frame of the video. After locating the face region, lip keypoints are further detected. Lip keypoints typically include the contour points of the upper and lower lips. Based on the detected lip keypoints, the bounding box of the lip region is calculated. The bounding box should contain the entire lip region and have appropriate edge margins.

[0209] Step S103: Lip Region Cropping and Normalization. Based on the lip region bounding box obtained in Step S102, the lip region image is cropped from the original video frame. The cropped lip image is adjusted to a uniform size, for example, 96×96 pixels. The image is normalized to map the pixel values ​​to a standard range to accelerate the training convergence of the neural network.

[0210] Step S104: Data Augmentation. During the training phase, data augmentation is performed on the training data to improve the model's generalization ability. Data augmentation methods include: random horizontal flipping to increase data diversity; random cropping, such as cropping from 96×96 pixels to 88×88 pixels; and standard color normalization. During the inference phase, the input frames are centered and cropped to 88×88 pixels and then color normalized.

[0211] Step S105: Sample Memory Buffer Construction. To mitigate the catastrophic forgetting problem in incremental class learning, a sample memory buffer strategy is employed. Specifically, for each task, a sample memory buffer is stored for each new class learned. Representative samples are selected using a distance-based selection strategy, choosing the sample closest to the average feature vector of each class. An accumulated sample memory buffer is used for knowledge distillation in subsequent tasks. For the first task ( When =1), the sample memory buffer is empty by default.

[0212] (2) Feature extraction step (step S200)

[0213] The purpose of the feature extraction step is to construct a feature extraction network and extract short-term and long-term features from the preprocessed lip-reading video sequence. This invention employs a standard lip-reading recognition architecture, including a 3D CNN front-end, a ResNet-34, and a bidirectional GRU back-end. The specific implementation process of this step is as follows:

[0214] Step S201: Construction of the front-end feature extraction network. Construct the front-end feature extraction network. It is used to process spatial features and capture local temporal patterns. The front-end network adopts a 3D CNN combined with a ResNet-34 structure, which can extract short-term spatiotemporal features from consecutive frames.

[0215] Step S202: Construction of the backend temporal modeling network. Construct the backend temporal modeling network. It is used to integrate temporal features in a global temporal context. The backend network adopts a bidirectional GRU structure, which can capture long-term temporal dependencies.

[0216] Step S203: Short-term feature extraction. The input video... Input front-end feature extraction network Extracting short-term features The calculation formula is:

[0217]

[0218] Step S204: Long-term feature extraction. Extract short-term features... Input backend time series modeling network Capture long-term temporal dependencies to obtain long-term features. The calculation formula is:

[0219]

[0220] Step S205: Global feature aggregation. For long-term features... Average pooling is performed along the time dimension to obtain the global representation. The calculation formula is:

[0221]

[0222] in, This represents the average pooling operation along the time dimension. The complete feature extractor is defined as follows: Through the above steps, the lip-reading video sequence is converted into short-term features. and long-term characteristics These features will serve as inputs for subsequent model training steps.

[0223] (3) Model training steps (step S300)

[0224] The model training step is the core step of this invention. A two-stage training strategy is used to train the lip-reading recognition model. The overall training process is as follows: Figure 1 As shown. The first stage of training includes feature differential enhancement, hierarchical differential distillation, and first-stage parameter updates; the second stage of training includes constructing a balanced dataset, orthogonal remapping, and second-stage parameter updates. The two-stage training strategy is designed to decouple feature learning from bias correction. The first stage focuses on learning discriminative features and retaining old knowledge, while the second stage focuses on classifier bias removal and feature space optimization.

[0225] Step S300a: Adaptive expansion of the classifier structure. Before each new incremental learning task begins, the output dimension of the classifier is dynamically expanded according to the number of new classes. Let the weight matrix of the old classifier be... (Its dimension corresponds to the number of old categories) The number of new categories is Then the weight matrix of the classifier is expanded into an adaptation matrix. There are 10 categories. The weight parameters for the old categories are retained, while the weight parameters for the new categories are randomly initialized. After the structure expansion is completed, a two-stage training process begins.

[0226] (3.1) First stage training

[0227] The goal of the first stage of training is to train from the combined dataset. Discriminative spatiotemporal features are learned, and baseline classification performance is established on all known categories. The first stage of training includes three sub-steps: feature difference enhancement, hierarchical difference distillation, and first-stage parameter update.

[0228] (3.1.1) Feature difference enhancement processing (step S301)

[0229] The purpose of feature discriminative enhancement is to improve the discriminative power between different categories of lip-reading features through a keyframe selection strategy and weighted supervised contrastive loss. In lip-reading recognition, the lip movement patterns of different words can be very similar, posing a challenge to classification. Feature discriminative enhancement effectively improves the inter-class separability of features by selecting the most discriminative keyframes and imposing contrastive learning constraints. The specific implementation process is as follows:

[0230] Step S3011: Calculate the keyframe importance score. For short-term features... Average pooling is performed along the feature dimension to obtain the representation of each frame, and then the importance score of each frame is calculated using the Sigmoid activation function. The calculation formula is:

[0231]

[0232] in, This represents the average pooling operation along the feature dimension.

[0233] The reason this invention chooses the Sigmoid activation function instead of the Softmax activation function is as follows: Softmax forces the sum of weights for all frames to be 1, which means that amplifying the importance of one frame will suppress the scores of other frames. However, the pronunciation of a single word often involves multiple different and equally important articulation gestures (e.g., the mouth opening action when pronouncing the initial phoneme and the mouth closing action when pronouncing the final phoneme). Sigmoid independently evaluates the importance of each frame, allowing multiple non-adjacent frames to simultaneously obtain high importance scores without inhibiting each other, thereby ensuring that all key temporal articulation cues are effectively captured.

[0234] Step S3012: Keyframe selection. Based on the importance score calculated in step S3011. Choose the one with the highest score. Frames are used as keyframes. Keyframe index set. The calculation formula is:

[0235]

[0236] Keyframe features Selected via index:

[0237]

[0238] In the formula, This indicates that the sorted position number is returned. Indicates descending order;

[0239] Step S3013: Keyframe Sort. The selected keyframes are strictly sorted in descending order of their importance scores, rather than in chronological order in the original video. This score-based sorting ensures that the most prominent vocal gestures across different video samples are consistently aligned to the same feature subspace, enabling subsequent linear layers to reliably process primary and secondary visual cues regardless of their absolute temporal position in the original video.

[0240] Step S3014: Feature Fusion. Combine keyframe features... Flattened and global representation The data is concatenated and then dimensionality is adjusted using a linear layer to obtain the fused features. The calculation formula is:

[0241]

[0242] Fusion features It contains the most distinctive keyframe information and global context information.

[0243] Step S3015: Calculation of weighted supervised contrast loss. This invention proposes a weighted supervised contrast loss method. A class-balanced weighting strategy is introduced based on the supervised contrast loss. The formula for calculating the weighted supervised contrast loss is:

[0244]

[0245] in, Indicates the first Fusion features of individual samples , This indicates temperature hyperparameters. Indicates label Sample-level weights are used to combat class imbalance by giving higher importance to underrepresented classes.

[0246] Step S3016: Calculate category balancing weights. Sample-level weights. Based on its category-level weight Calculated. Category-level weights. The calculation formula is:

[0247]

[0248] in, Indicates category The number of valid samples is an indicator for measuring the actual information value of a category; This indicates the total number of categories that have been learned; Indicates category The number of samples in the current dataset; Control the weight strength.

[0249] (3.1.2) Differential distillation treatment (step S302) (Note: This step is triggered when t > 1)

[0250] The purpose of hierarchical differential distillation is to transfer knowledge from the old model to the new model through a multi-level distillation strategy, thereby mitigating the catastrophic forgetting problem. Traditional knowledge distillation methods typically only perform distillation at the output layer, while the hierarchical differential distillation strategy proposed in this invention performs distillation at three levels: original feature consistency, keyframe knowledge inheritance, and class prediction stability, thus more comprehensively preserving the knowledge of the old model. Figure 2 As shown, the specific implementation process of this treatment is as follows:

[0251] Step S3021: Original Feature Consistency Distillation. To ensure the spatiotemporal coherence and contextual consistency of the lip-reading video, minimize the task... and tasks The mean squared error (MSE) between the short-term and long-term features extracted by the model. Original feature consistency loss. The calculation formula is:

[0252]

[0253] in, and Representing tasks and tasks The long-term features extracted by the model and Representing tasks and tasks The model extracts short-term features.

[0254] Step S3022: Keyframe Knowledge Inheritance Distillation. Keyframes play a crucial role in distinguishing visually similar lip shapes, therefore their knowledge transfer is essential. Keyframe knowledge inheritance distillation strictly uses the keyframe index identified by the old model. Instead of aligning keyframes selected independently by the two models, this approach distills keyframes selected independently by the new model with those of the old model, potentially leading to severe temporal mismatches. By forcing the new model to reproduce features of the old model at precise temporal anchors defined by the old model, we can explicitly prevent the network from overriding specific discriminative visual cues that define the old categories. (Keyframe knowledge inheritance loss) The calculation formula is:

[0255]

[0256] in, This represents the short-term features of the new model at the keyframe index of the old model. This represents the keyframe features of the old model.

[0257] Step S3023: Class Prediction Stability Distillation. This invention employs Binary Cross-Entropy (BCE) combined with a Sigmoid activation function, instead of the classic Softmax temperature distillation. The reason for choosing BCE over Softmax is as follows: Softmax enforces a competitive probability distribution, where the sum of probabilities must be 1. In incremental class learning scenarios, introducing a new class inevitably compresses and distorts the relative probability space of the old classes. Conversely, by using class-independent Sigmoid activation combined with binary cross-entropy, the logit activation strength of each class is independently evaluated. This independent constraint explicitly preserves the absolute activation strength of the learned classes, thereby preventing newly introduced classes from suppressing the activation of the learned classes. Class Prediction Stability Loss The calculation formula is:

[0258]

[0259] in, This represents the new model's output logits for the old categories. This represents the output logits of the old model. This represents the Sigmoid activation function.

[0260] Step S3024: Calculate the total loss of hierarchical difference distillation. The weighted sum of the original feature consistency loss, keyframe knowledge inheritance loss, and class prediction stability loss yields the total loss of hierarchical difference distillation:

[0261]

[0262] in, , and These are the weight coefficients for the original feature consistency loss, keyframe knowledge inheritance loss, and class prediction stability loss, respectively. The present invention preferably sets... =0.05、 =10、 =1, ensuring that distillation losses are on a consistent scale.

[0263] (3.1.3) First stage parameter update (step S303)

[0264] The purpose of the first-stage parameter update is to update the parameters of the feature extraction network and the classifier based on the first-stage loss function. The training objective of the first stage is defined as:

[0265]

[0266] in, The standard cross-entropy loss is calculated on the combined dataset:

[0267]

[0268] The Adam optimizer was used to update the parameters of the feature extraction network and the classifier via backpropagation. The learning rate was set to 0.0003, the weight decay factor to 0.0001, and the batch size to 32. The basic task was trained for 20 epochs, and subsequent incremental tasks were trained for 10 epochs. After the first stage of training, the feature extraction network had learned discriminative feature representations.

[0269] (3.2) Second stage training (Note: This step is triggered when t > 1)

[0270] The purpose of the second-stage training is to address the classifier bias problem caused by data imbalance in incremental learning. This problem is even more pronounced in fine-grained recognition scenarios where the decision boundary itself is complex. Directly integrating regularization techniques into the main training stage to address classifier bias essentially couples feature learning with bias correction, introducing a competitive optimization objective. This invention employs a two-stage training strategy, explicitly decoupling these two objectives. By isolating the bias correction process and performing the final refinement step, it simultaneously alleviates classifier bias and maximizes geometric separation in the feature space without compromising the fundamental feature representation. The second-stage training includes three sub-steps: constructing a balanced dataset, orthogonal remapping processing, and second-stage parameter updating.

[0271] (3.2.1) Construct a balanced dataset (step S304)

[0272] The purpose of constructing a balanced dataset is to provide class-balanced feature representations for the second-stage training. The specific implementation process for this step is as follows:

[0273] Step S3041: Feature Extraction. The feature extraction network trained in the first stage extracts long-term features.

[0274] Step S3042: Sample Selection. Construct a balanced dataset. The dataset contains each class Each feature represents: Features extracted by a feature extraction network and their mean. For the old category Use the memory buffer One sample; for the new category Since more samples are available, carefully select those closest to the class mean. An example.

[0275] Step S3043: Feature pooling and concatenation. Long-term features are analyzed by applying temporal average pooling and max pooling sequentially. Average pooling captures the overall statistical trend and global temporal context of the articulation sequence; max pooling isolates the most significant and extreme activations, which typically correspond to key lip deformations, including abrupt closure or large opening. Concatenating the results of the two pooling methods ensures that the final features simultaneously contain comprehensive global context and peak discriminative signals.

[0276] Step S3044: Feature L2 Normalization. To prevent trivial solutions that minimize the loss by simply reducing the feature amplitude, the concatenated features are L2 normalized and projected into an orthogonal subspace. The normalized feature vector is denoted as h.

[0277] (3.2.2) Orthogonal remapping processing (step S305)

[0278] The purpose of orthogonal remapping is to constrain the orthogonality between features of different categories through orthogonal loss, reduce interference between new and old category features, and improve the stability and efficiency of class increment learning. The specific implementation process is as follows:

[0279] Step S3051: Freeze the feature extraction network. In the second stage, the feature extraction network is frozen, and only the orthogonal mapping layer and the final classifier are updated. This design allows the dual benefits of classifier bias correction and feature space optimization to be achieved with minimal computational overhead.

[0280] Step S3052: Orthogonal Loss Calculation. Orthogonal loss explicitly constrains the orthogonality between feature representations of different categories. The calculation formula is:

[0281]

[0282] in, yes The number of pairings between classes This represents the L2-normalized feature vector. This loss encourages inter-class separation in the mapped feature space.

[0283] (3.2.3) Second stage parameter update (step S306)

[0284] The purpose of the second-stage parameter update is to update the parameters of the orthogonal mapping layer and the classifier based on the second-stage loss function. The training objective of the second stage is defined as:

[0285]

[0286] in, It is the standard cross-entropy loss used for classification, orthogonal loss. Encourage inter-class separation in the mapped feature space. Using an optimizer (such as Adam), with the backbone network frozen, only compute and update the network parameters of the lightweight orthogonal mapping layer and the classifier whose structure has been expanded, thereby efficiently correcting the data imbalance bias caused by class increments.

[0287] (4) Model saving steps (step S400)

[0288] The purpose of the model saving step is to save the parameters of the trained lip-reading model for subsequent deployment and use. The specific implementation process of this step is as follows:

[0289] Step S401: Save Model Parameters. After training is complete, save the model's parameters and structure information. The model can be saved as a PyTorch .pt file for subsequent deployment and use.

[0290] Step S402: Sample Memory Buffer Update. After training for each task, the sample memory buffer is updated using a distance-based selection strategy, storing data for each class. A representative sample is used for knowledge distillation in subsequent tasks.

[0291] The present invention will be further verified and illustrated through specific embodiments below.

[0292] Example 1: A lightweight continuous lip-reading learning system based on fine-grained spatiotemporal awareness using the method of the present invention.

[0293] This embodiment implements a complete lightweight continuous lip-reading learning system based on fine-grained spatiotemporal awareness, including a data acquisition and preprocessing module, a feature extraction module, a model training module, and a model storage module. The system is implemented using the PyTorch deep learning framework and runs on a server configured with an NVIDIA RTX 3090 GPU.

[0294] The system employs a standard lip-reading architecture, including a 3D CNN front-end, a ResNet-34 back-end, and a bidirectional GRU back-end. (Number of keyframes...) Set to 2, temperature over-parameter Set to 0.1, category balance weight parameter Set to 0.9. The hierarchical difference distillation weight is set to... =0.05、 =10、 =1. Sample memory buffer stores each class. One sample.

[0295] The system was tested on the i-LRW and i-LRW1000 datasets. Figure 3 and Figure 4 The results are shown in Tables 1 and 2 below. The experiments employed a class-based incremental learning setup, including three scenarios: B0-Task10 (no basic task, 10 incremental tasks, 20 new classes per task), B0-Task20 (no basic task, 20 incremental tasks, 10 new classes per task), and B100-Task10 (100 basic tasks followed by 10 incremental tasks). Experimental results demonstrate that the proposed method achieves state-of-the-art performance across all scenarios. In the B100-Task10 scenario of the i-LRW dataset, the proposed method achieves an average accuracy of 73.40%, a 12.35% improvement over the strongest baseline, TPAN; the performance degradation rate is only 19.31%, a 16.08% reduction compared to TPAN. Significant performance improvements were also achieved on the i-LRW1000 dataset.

[0296] Table 1. Performance comparison of algorithms in three different scenarios using the 200-class subset i-LRW of the public dataset LRW.

[0297]

[0298] Note: The statistics include three metrics: average accuracy (%), final accuracy (%), and performance degradation rate (%). Higher average accuracy and final accuracy are better, and lower performance degradation rate is better. Bold and underlined text indicate best and second-best performance, respectively. The improvement value is the difference between the best and second-best values.

[0299] Table 2. Comparison of algorithm performance in three different scenarios using the 200-class subset i-LRW1000 of the public dataset LRW1000.

[0300]

[0301] Note: The statistics include three metrics: average accuracy (%), final accuracy (%), and performance degradation rate (%). Higher average accuracy and final accuracy are better, and lower performance degradation rate is better. Bold and underlined text indicate best and second-best performance, respectively. The improvement value is the difference between the best and second-best values.

[0302] Example 2: Comparative Experiment of the Method of the Invention and the Traditional Method

[0303] To verify the technical advantages of this invention, a comparative experiment was conducted between the method of this invention and traditional incremental learning methods. The comparative methods include: iCaRL (classical sample replay method), BiC (bias correction method), TPAN (current state-of-the-art method for incremental learning of lip reading), DER (dynamic expansion method), FOSTER (feature consolidation method), and JIOC (adaptive input coordination method).

[0304] Experimental results show that the method of the present invention is significantly better than all the comparison methods in terms of system reliability (average accuracy) and stability (performance degradation rate). The advantages of the method of the present invention are mainly reflected in the following aspects: (1) Feature difference enhancement processing uses the Sigmoid activation function to calculate the key frame importance score, allowing multiple key frames to obtain high scores at the same time, effectively capturing multiple key pronunciation postures in lip reading videos; (2) Hierarchical difference distillation processing uses BCE in conjunction with the Sigmoid activation function to perform class prediction stability distillation, independently evaluating the activation intensity of each category, effectively avoiding the suppression of the activation of old categories by new categories; (3) The two-stage training strategy decouples feature learning from bias correction. The second stage only requires about 8% of the total training time, achieving classifier bias removal and feature space optimization with minimal computational overhead.

[0305] In summary, the proposed solution of this invention: (1) By enhancing feature difference, the Sigmoid activation function is used to capture multiple key pronunciation postures, and combined with weighted supervised contrast loss, the model's ability to distinguish fine-grained, high-similarity lip movement patterns is significantly improved; (2) A hierarchical difference distillation strategy is adopted to transfer knowledge from three dimensions: original features, key frame knowledge, and class prediction stability. In particular, the use of binary cross-entropy in conjunction with the Sigmoid activation function protects the activation intensity of the old category and prevents it from being suppressed by the new category; (3) A two-stage training strategy decouples feature learning from bias correction. In the second stage, under the premise of freezing the backbone network, orthogonal remapping constrains the orthogonality between classes with minimal computational overhead, reducing interference between the old and new categories; (4) The system does not need to store the user's original video data, but only achieves incremental learning through feature-level knowledge distillation and sample memory buffer, which is suitable for application scenarios with strict requirements for privacy protection. Experimental results show that this invention is superior to the prior art in terms of average accuracy and performance degradation rate, and can continuously adapt to new categories without significantly reducing the ability to recognize old categories.

[0306] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention without departing from its spirit and scope should be included within the protection scope of the present invention.

Claims

1. A lightweight, continuous lip-reading learning method based on fine-grained spatiotemporal awareness, characterized in that, The method includes the following steps: Step 1: Acquire lip-reading video data and preprocess it to obtain lip-reading video sequences, while maintaining a memory buffer containing representative samples of the learned categories; Step 2: Construct a feature extraction network to extract short-term and long-term features from the lip-reading video sequence; Step 3: Use a two-stage training strategy to incrementally train the lip-reading model: The first stage of training involves extracting fused features using feature difference enhancement processing and combining it with hierarchical difference distillation processing to constrain the knowledge consistency between the old and new models, thereby collaboratively optimizing the feature extraction network and the classifier. Second-stage training: Based on the features obtained in the first stage, a balanced dataset is constructed, and the feature space distribution is optimized and the classifier is updated through orthogonal remapping. Specifically, the hierarchical differential distillation process and the second-stage training will only be started for non-first tasks, that is, when the number of tasks t>1, the hierarchical differential distillation process and the second-stage training process will be triggered. Step 4: Save the parameters of the trained lip-reading model and update the memory buffer; Step 5: For the lip-reading video data to be learned, return to step 1.

2. The lightweight lip-reading continuous learning method based on fine-grained spatiotemporal awareness according to claim 1, characterized in that, Step 1 specifically includes: Step 1-1: Obtain video data containing the speaker's lip movements and their corresponding labels, and convert the video into a grayscale image; Steps 1-2: Perform face detection and locate lip key points for each frame of the video, and calculate the bounding box containing the complete lip region; Steps 1-3: Crop the lip region image according to the bounding box and adjust it to a uniform size, and perform pixel normalization processing; Steps 1-4: During the training phase, perform data augmentation on the training data, including at least random horizontal flipping, random cropping, and standard color normalization. Steps 1-5: Construct and update the memory buffer storing representative samples for knowledge distillation in subsequent tasks; wherein, for the first task, the memory buffer is empty by default; the representative samples have custom constraints.

3. The lightweight lip-reading continuous learning method based on fine-grained spatiotemporal awareness according to claim 1, characterized in that, The feature extraction network in step 2 includes: Front-end feature extraction network It consists of 3D convolutional layers and residual networks, and is used to extract short-term temporal features; Backend Temporal Modeling Network It consists of a bidirectional recurrent neural network, which extracts long-term time-dependent features based on short-term features.

4. The lightweight lip-reading continuous learning method based on fine-grained spatiotemporal awareness according to claim 1, characterized in that, Step 3, before performing incremental task training, also includes adaptive expansion of the classifier structure: Get the number of new categories in the current task; Expand the output layer dimension of the classifier, retain the weight parameters corresponding to the old category, and initialize the weight parameters corresponding to the new category.

5. The lightweight lip-reading continuous learning method based on fine-grained spatiotemporal awareness according to claim 1, characterized in that, Step 3, the feature differential enhancement process in the first stage of training, specifically includes: Step 3-1, for short-term characteristics Perform global information aggregation and use the Sigmoid activation function to calculate the importance score for each frame; Step 3-2: Select the top K frames as keyframe features based on their importance scores in descending order. ; Step 3-3: Concatenate and fuse the keyframe features with the average pooling representation of long-term features to obtain the fused features. ; Steps 3-4: Use weighted supervision to compare losses Constraints are imposed on the fusion features by introducing weights based on the number of valid samples for each class to balance the contributions of the old and new classes.

6. The lightweight lip-reading continuous learning method based on fine-grained spatiotemporal awareness according to claim 5, characterized in that, The formula for calculating the number of valid samples for each category in steps 3-4 is as follows: In the formula, This represents the number of valid samples in category c. This indicates the number of samples of category c in the current dataset. This indicates the strength of the control weight.

7. The lightweight lip-reading continuous learning method based on fine-grained spatiotemporal awareness according to claim 6, characterized in that, Step 3, the hierarchical difference distillation process in the first stage of training, specifically includes: Original Feature Consistency Distillation: Calculate the mean squared error loss between the new model and the old model on short-term and long-term features, denoted as the original feature consistency loss. ; Keyframe knowledge inheritance distillation: Using the temporal index of keyframes identified by the old model, the new model is constrained to reproduce the feature cues of the old model at the same positions, and the keyframe knowledge inheritance loss is calculated. ; Class prediction stability distillation: By using binary cross-entropy in conjunction with the Sigmoid activation function, the output probability distribution of the new model for the old class is constrained to remain consistent with that of the old model, resulting in class prediction stability loss. ; The total loss of differential distillation is : In the formula, , and These are the weight coefficients for the original feature consistency loss, keyframe knowledge inheritance loss, and class prediction stability loss, respectively. The total loss function for the first stage of training in step 3 is: : In the formula, It is the standard cross-entropy loss used for classification.

8. The lightweight lip-reading continuous learning method based on fine-grained spatiotemporal awareness according to claim 1, characterized in that, Step 3, the second phase of training, includes the following specific processes: Steps 3-5: Construct a balanced dataset by selecting an equal amount of sample features for each new and old category from the current task data and the memory buffer; specifically: obtain old category features from the memory buffer through a feature extraction network, and obtain an equal amount of new category features from the current task data; Steps 3-6: Perform orthogonal remapping to project the features to an orthogonal subspace; Steps 3-7, using orthogonal loss Explicitly constrain the orthogonality between feature representations of different categories, and update only the orthogonal mapping layer and classifier parameters.

9. The lightweight lip-reading continuous learning method based on fine-grained spatiotemporal awareness according to claim 8, characterized in that, The total loss function for the second stage of training in step 3 is: : In the formula, It is the standard cross-entropy loss used for classification; in, In the formula, It is about balancing the number of inter-class pairings in the dataset. It is the label corresponding to the i-th sample. It is the label corresponding to the j-th sample. It is the first The long-term features corresponding to each sample are concatenated after average pooling and time-maximum pooling, and then normalized by L2. It is the first The long-term features corresponding to each sample are concatenated after average pooling and time-maximum pooling, and then normalized by L2. This indicates the matrix transpose.

10. The lightweight lip-reading continuous learning method based on fine-grained spatiotemporal awareness according to claim 2, characterized in that, In steps 1-5 and step 4, a distance-based selection strategy is used to update the memory buffer, specifically including: Calculate the feature vector of all samples in each category; Calculate the average feature vector of this category and use it as the class center; Sort the samples according to their Euclidean distance from the class center from closest to furthest, and select the top m samples as the most representative samples to store in the memory buffer.

Images