A distribution preserving type structure distillation method for continuous named entity recognition

Abstract

The application discloses a distribution preserving type structure distillation method for continuous named entity recognition, and comprises the following steps: S1, initializing a model; S2, training the model; S3, after the training is completed, a final inference model is obtained through a semantic perception parameter fusion mechanism VSPF; and S4, outputting a result by using the final model. The method can effectively alleviate the catastrophic forgetting problem in long-term online learning of a voice assistant and improve continuous learning performance.

Description

Technical Field

[0001] This invention relates to the field of knowledge distillation technology, and more specifically to a distributed structure distillation method for persistent named entity recognition. Background Technology

[0002] Intelligent voice assistants are widely used in various terminal devices, and their interaction accuracy relies on Continuous Named Entity Recognition (CNER) technology. CNER, combined with the core goal of voice assistants, is to solve the key problem of how voice assistants can continuously learn corresponding new entities while retaining their original recognition capabilities when adding new functions, new devices, and new vocabulary during the functional iteration process.

[0003] In its initial stages, voice assistants only need to recognize basic entities such as date, location, and contact person (PER) to support basic functions like making calls and navigation. However, as functionality expands, it needs to recognize new entity types such as music genre, app name, smart home device, or point of interest (POI). Traditional voice assistants are prone to "catastrophic forgetting" after adding new entities; that is, after learning the name of a new smart device, they lose the ability to recognize existing entities such as contacts, locations, and organizations, severely impacting the user experience.

[0004] Continuous learning is the core idea for solving this problem, aiming to allow the model to retain its historical learning performance while continuously receiving new tasks. Among existing continuous learning methods, regularization-based methods have become mainstream because they require no additional storage, do not modify the model structure, and are suitable for the lightweight and privacy protection requirements of voice assistants. Among these, knowledge distillation methods based on output logits are widely used.

[0005] However, existing CNER-based knowledge distillation methods mostly rely on logits for point-by-point alignment, which makes it difficult to characterize entity category relationships and internal distributions, resulting in insufficient knowledge retention. Some improved methods are either unsuitable for sequence labeling scenarios or rely on additional storage, conflicting with the deployment requirements of voice assistant terminals. In summary, existing technologies cannot effectively solve the catastrophic forgetting problem in long-term online learning for voice assistants, and are insufficient to meet practical application needs. Summary of the Invention

[0006] This invention aims to solve the technical problems existing in the prior art, and innovatively proposes a distribution-preserving structure distillation method for continuous named entity recognition, which can effectively alleviate the catastrophic forgetting problem in the long-term online learning of voice assistants and improve continuous learning performance.

[0007] To achieve the above objectives, this invention provides a distribution-preserving structure distillation method for persistent named entity recognition, comprising the following steps:

[0008] S1: Initialize the teacher and student models;

[0009] S2: Train the teacher model and the student model;

[0010] S2-1: The teacher model processes the input data containing old entity types required by the voice assistant model;

[0011] S2-2: Process the input data containing new entity types required by the voice assistant model using the student model;

[0012] S2-3: Calculate the cross-entropy loss by comparing the student model classifier output with the true labels;

[0013] The formula is as follows:

[0014] ;

[0015] in, Represents cross-entropy loss, It is for real tags The indicator function is used to label the true category of the training data for the voice assistant model. This indicates that the student model, on the input voice command sample data, for the first... The probability that each token is predicted to belong to the true label category; The number of entity types that the voice assistant has learned. The total number of new entity types that the voice assistant model needs to learn;

[0016] S2-4: Calculate the in-token class relationship distillation loss between the teacher-student model and the student model based on Z-score;

[0017] ;

[0018] in, This represents the class distillation loss within a token, used to preserve the relative class relationships between old entities along the token dimension when the voice assistant model incrementally learns new entities. For temperature coefficient, For the teacher model in the voice command The token corresponds to the first The logits value of the token-class dimension after standardization and Softmax calculation for item categories. For student models in the voice command The token corresponds to the first The logits value of the token-class dimension after standardization and Softmax calculation for item categories; For the entity types that the voice assistant has already learned;

[0019] S2-5: Calculate the distillation loss of all tokens within a single class based on Z-score for the teacher-student model and the student model;

[0020] ;

[0021] in, The distillation loss, representing the distribution relationship within categories, is used to preserve the positional distribution pattern of old entity categories along the token dimension when a voice assistant incrementally learns new entities. For temperature coefficient, In the teacher model, the first The first category The logits value of the class-token dimension after standardization and Softmax calculation of the voice command token. In the student model, the first The first category The logits value of the class-token dimension after standardization and Softmax calculation of the voice command token;

[0022] S2-6: Calculate the total loss;

[0023] The calculation is performed using the following formula:

[0024] ;

[0025] in, This represents the total loss during the incremental learning phase of the voice assistant. Cross-entropy loss is used to supervise the student model's recognition performance on the newly added entity categories. This represents the distillation loss of the category relationship within the token. Distillation loss is a distribution within the category. , They are respectively and Weighting coefficients;

[0026] S3: After training, the model is fused using the semantically aware parameter fusion mechanism VSPF to obtain the final inference model;

[0027] S4: Output the results using the final inference model.

[0028] In the above scheme, step S2-1 further includes:

[0029] S2-1-1: Sample of input voice command Input voice command samples are extracted using the TCSD module of the teacher model. Discrimination structure relationships between different entity categories within a single token;

[0030] S2-1-2: Extracting input voice command samples using the CTDD module of the teacher model. Global response distribution structure among all tokens within a single entity category;

[0031] S2-1-3: After Z-score normalization of the category probability distribution extracted by the teacher model, the distribution is split into two paths: one path is input into the teacher model classifier, and the other path is retained as a knowledge supervision signal.

[0032] In the above scheme, step S2-1-1 further includes:

[0033] S2-1-1-1: Sample input voice commands For each token, extract the raw logits of all entity categories of the teacher model on that token. ;

[0034] S2-1-1-2: Perform Z-score normalization on all categories of logits within the same voice command token;

[0035] The formula is as follows:

[0036] ;

[0037] in, For the standardized first Logits prediction results of the category teacher model under the item category. This represents the mean of logits for all categories in the teacher model under the current voice command token. This represents the standard deviation of logits for all categories in the teacher model under the current voice command token. Indicates the first Logits prediction results of the teacher model under each item category;

[0038] S2-1-1-3: Apply temperature scaling to the standardized logits and calculate the class probability distribution;

[0039] A distillation temperature coefficient is introduced into the logits after Z-score normalization. Scaling is applied, and then the old entity category probability distribution of the teacher model on the voice command token is calculated using Softmax;

[0040] ;

[0041] in, For temperature coefficient, For the teacher model in the first voice command The token corresponds to the first The logits value of the token-class dimension after standardization and Softmax calculation for the old entity category. For the standardized first Logits prediction results of the teacher model under each item category. For the standardized first Logits prediction results for the teacher model under each item category.

[0042] In the above scheme, step S2-1-2 also includes:

[0043] S2-1-2-1: Transpose the logit matrix obtained during the distillation process of the voice command samples;

[0044] Let the number of tokens selected during the distillation process be... For the first There are entities, and their teacher model logit vectors on all tokens are denoted as follows: ;

[0045] in, The logits vector representing the teacher model of the first token of the voice command; Indicates the first voice command The logits vector of the teacher model for each token;

[0046] S2-1-2-2: Perform Z-score standardization on all tokens within the same category of logits;

[0047] The formula is as follows:

[0048] ;

[0049] in, For the class-token matrix in the teacher model, the first... The result after Z-score standardization, This represents the mean logits of all voice command tokens in the teacher model for the current category. This represents the standard deviation of the logits of all voice command tokens in the teacher model for the current category. This represents the first class-token in the teacher model. The value of the item;

[0050] S2-1-2-3: Apply temperature scaling to the standardized logits and calculate the class probability distribution;

[0051] A distillation temperature coefficient is introduced into the logits after Z-score normalization. Scaling is applied, and then the class probability distribution of teachers is calculated using Softmax;

[0052] ;

[0053] in, For temperature coefficient, In the teacher model, the first The first category The logits value of the class-token dimension after standardization and Softmax calculation of the voice command token. For the teacher model class-token matrix, the first The result after Z-score standardization, For the teacher model class-token matrix, the first The result after Z-score standardization.

[0054] In the above scheme, step S2-2 further includes:

[0055] S2-2-1: Extract input voice command samples using the TCSD module of the student model. Discrimination structure relationships between different entity categories within a single token;

[0056] Distillation retains only the relative distribution structure between categories within the same token, completely shielding the interference of the overall scale and global offset of logits;

[0057] S2-2-2: Extract input speech command samples using the CTDD module of the student model. Global response distribution structure among all tokens within a single entity category;

[0058] S2-2-3: Apply temperature scaling to the standardized logits and calculate the class probability distribution;

[0059] S2-2-4: After normalizing the class probability distribution extracted from the student model using Z-score, input it into the student model classifier.

[0060] In the above scheme, step S2-2-1 further includes:

[0061] S2-2-1-1: Sample input voice commands For each token, extract the raw logits of all entity categories of the student model associated with that token. ;

[0062] S2-2-1-2: Perform Z-score normalization on the logits of all old entity categories within the same voice command token;

[0063] The formula is as follows:

[0064] ;

[0065] in, For the standardized first Logits prediction results of the student model under each item category. This represents the mean of logits for all categories in the student model under the current voice command token. This represents the standard deviation of logits for all categories in the student model under the current voice command token. Indicates the first Logits prediction results of the student model under each item category;

[0066] S2-2-1-3: Apply temperature scaling to the standardized logits and calculate the class probability distribution;

[0067] A distillation temperature coefficient is introduced into the logits after Z-score normalization. Scaling is performed, and then the entity category probability distribution of the student model on the voice command token is calculated using Softmax;

[0068] ;

[0069] in, For temperature coefficient, For the student model in the voice command of the voice command The token corresponds to the first The standardized logits values ​​of the token-class dimension of the student model by category, calculated using Softmax. For the standardized first Logits prediction results of the student model under each item category. For the standardized first Logits prediction results of the student model under each item category. This represents the total number of old entity categories that the voice assistant has already learned. This represents the total number of new entity types that the voice assistant model needs to learn.

[0070] In the above scheme, step S2-2-2 further includes:

[0071] S2-2-2-1: Transpose the logit matrix obtained during the distillation process of the voice command samples;

[0072] Let the total number of voice command tokens in the voice command sample be... For the first Let there be entity categories, whose logit vector for the student model across all tokens is denoted as . ;

[0073] in, This represents the logits vector of the student model for the first token of the voice command. Indicates the first voice command The logits vector of a student model with one token;

[0074] S2-2-2-2: Perform Z-score standardization on all tokens within the same category of logits;

[0075] The formula is as follows:

[0076] ;

[0077] in, For the class-token matrix in the student model, the first... The result after Z-score standardization, This represents the mean logits of all voice command tokens in the student model for the current category. This represents the standard deviation of the logits of all voice command tokens in the student model for the current category. This represents the first element in the class-token matrix of the student model. The value of the item;

[0078] Step S2-2-3 also includes:

[0079] A distillation temperature coefficient is introduced into the logits after Z-score normalization. The scale is then applied, and then Softmax is used to calculate the probability distribution of each voice command sample category for the student model under that entity category;

[0080] ;

[0081] in, For temperature coefficient, In the student model, the first Category 1 The logits value of the class-token dimension after standardization and Softmax calculation for each voice command token category. For the class-token matrix in the student model, the first... The result after Z-score standardization, For the class-token matrix in the student model, the first... The result after Z-score standardization.

[0082] In the above scheme, step S3 also includes:

[0083] S3-1: Validation dataset for a given voice assistant scenario Simultaneously input the teacher model and the trained student model, and obtain the full prediction logits of the teacher model and the student model on the validation dataset;

[0084] ;

[0085] in, This represents the complete logits vector output by the old model on this voice command validation sample. This indicates the teacher model classification headers that were frozen and saved in the previous stage. This represents the complete logits vector output by the new model on this voice command validation sample. This represents the student model classification head that has completed training for the new task in the current stage;

[0086] S3-2: Dimension clipping, retaining only the prediction information from the old category space;

[0087] S3-3: Concatenate and vectorize the prediction results of all voice command verification samples in the old class space to form an overall prediction distribution representation;

[0088] S3-4: Calculate the semantic consistency of the global prediction distribution of the old and new models using cosine similarity to obtain the dynamic fusion weights;

[0089] ;

[0090] in, Indicates the fusion weights; Cosine similarity is used to measure the global predictive consistency between the old and new models on the old entity recognition task of voice commands.

[0091] S3-5: Update the final model parameters by merging all parameters of the teacher model and the student model through semantic weighting;

[0092] The formula is as follows:

[0093] ;

[0094] in, This represents the model parameters that were frozen and saved in the previous stage. This represents the model parameters obtained during the current stage of training on the new task. This represents the model obtained after updating the parameters, i.e., the final model parameters of the voice assistant;

[0095] S3-6: Output the final model for continuous entity recognition tasks using voice commands.

[0096] In summary, the beneficial effects of this invention are as follows: This technical solution, from a structural modeling perspective, extends the distillation objective from category-level relational constraints to dual structural modeling at both the token and category levels. The TCSD module characterizes the discriminative relationships between different entity categories within a single voice command token; simultaneously, the CTDD module models the response distribution across different voice command tokens from a category perspective, thereby constraining the internal structure of categories. Both approaches provide complementary perspectives for structural modeling of the entity types already learned by the voice assistant, enabling the method to better adapt to sequence labeling scenarios and achieve more stable and robust knowledge retention during continuous learning. By combining the TCSD and CTDD modules, the discriminative relationships between categories and the distribution information within categories are maintained. This solves the problem of "catastrophic forgetting" that traditional voice assistants easily experience after adding new entity types. That is, they learn the names of new smart devices but forget the original entity recognition capabilities such as contacts, locations, and organizations. This effectively alleviates the catastrophic forgetting problem in long-term online learning of voice assistants and improves the stability of knowledge transfer. The parameter fusion strategy VSPF is set up to achieve dynamic parameter fusion of semantic perception through the consistency of validation set predictions. This enhances the stability and adaptability of the model in the incremental learning process, balances new and old knowledge, and reduces knowledge conflicts in continuous learning. Attached Figure Description

[0097] Figure 1 This is the framework diagram of this model.

[0098] Figure 2(a) is a comparison chart of the stepwise micro-average F1 score of the present technical solution with that of CFNER, CPFD, EDAW, and FALCON based on FG-1-PG-1 on the I2B2 dataset; (b) is a comparison chart of the stepwise micro-average F1 score of the present technical solution with that of CFNER, CPFD, EDAW, and FALCON based on FG-2-PG-2 on the I2B2 dataset; (c) is a comparison chart of the stepwise micro-average F1 score of the present technical solution with that of CFNER, CPFD, EDAW, and FALCON based on FG-8-PG-1 on the I2B2 dataset; (d) is a comparison chart of the stepwise micro-average F1 score of the present technical solution with that of CFNER, CPFD, EDAW, and FALCON based on FG-8-PG-2 on the I2B2 dataset.

[0099] Figure 3 (e) is a comparison chart of the stepwise macro-average F1 score of this technical solution with that of CFNER, CPFD, and FALCON based on FG-1-PG-1 on the I2B2 dataset; (f) is a comparison chart of the stepwise macro-average F1 score of this technical solution with that of CFNER, CPFD, and FALCON based on FG-2-PG-2 on the I2B2 dataset; (g) is a comparison chart of the stepwise macro-average F1 score of this technical solution with that of CFNER, CPFD, and FALCON based on FG-8-PG-1 on the I2B2 dataset; (h) is a comparison chart of the stepwise macro-average F1 score of this technical solution with that of CFNER, CPFD, and FALCON based on FG-8-PG-2 on the I2B2 dataset.

[0100] Figure 4 (i) is a comparison chart of the stepwise micro-average F1 score of the present technical solution with that of CFNER, CPFD, EDAW, and FALCON based on FG-1-PG-1 on the Onto dataset; (j) is a comparison chart of the stepwise micro-average F1 score of the present technical solution with that of CFNER, CPFD, EDAW, and FALCON based on FG-2-PG-2 on the Onto dataset; (k) is a comparison chart of the stepwise micro-average F1 score of the present technical solution with that of CFNER, CPFD, EDAW, and FALCON based on FG-8-PG-1 on the Onto dataset; and (l) is a comparison chart of the stepwise micro-average F1 score of the present technical solution with that of CFNER, CPFD, EDAW, and FALCON based on FG-8-PG-2 on the Onto dataset.

[0101] Figure 5(m) is a comparison chart of the progressive macro-average F1 score of this technical solution with that of CFNER, CPFD, and FALCON based on FG-1-PG-1 on the Onto dataset; (n) is a comparison chart of the progressive macro-average F1 score of this technical solution with that of CFNER, CPFD, and FALCON based on FG-2-PG-2 on the Onto dataset; (o) is a comparison chart of the progressive macro-average F1 score of this technical solution with that of CFNER, CPFD, and FALCON based on FG-8-PG-1 on the Onto dataset; (p) is a comparison chart of the progressive macro-average F1 score of this technical solution with that of CFNER, CPFD, and FALCON based on FG-8-PG-2 on the Onto dataset.

[0102] Figure 6 (a) is a comparison of the stepwise micro-average F1 scores of the overall model of this technical solution with the models with VSPF removed, CTDD removed, TCSD removed, and both CTDD and TCSD removed, based on the FG-8-PG-1 setting on the I2B2 dataset; (b) is a comparison of the stepwise macro-average F1 scores of the overall model of this technical solution with the models with VSPF removed, CTDD removed, TCSD removed, and both CTDD and TCSD removed, based on the FG-8-PG-1 setting on the I2B2 dataset. (c) is a comparison graph of the stepwise micro-average F1 score of the overall model of this technical solution with the model with VSPF removed, the model with CTDD removed, the model with TCSD removed, and the model with both CTDD and TCSD removed, based on the FG-8-PG-2 setting on the I2B2 dataset; (d) is a comparison graph of the stepwise macro-average F1 score of the overall model of this technical solution with the model with VSPF removed, the model with CTDD removed, the model with TCSD removed, and the model with both CTDD and TCSD removed, based on the FG-8-PG-2 setting on the I2B2 dataset.

[0103] Figure 7 (a) shows the settings based on FG-8-PG-1 on the Onto dataset. and Heatmaps of micro-average F1 scores for different values, (b) showing the FG-8-PG-1 settings on the Onto dataset. and Heatmap of macroscopic average F1 score for different values. Detailed Implementation

[0104] The present invention will be further described below with reference to the embodiments and accompanying drawings:

[0105] like Figure 1As shown, a distributed structure distillation method for persistent named entity recognition includes the following steps:

[0106] S1: Initialize the teacher and student models;

[0107] S1-1: Define the teacher model ;

[0108] Define student model The structure is the same as the teacher model, and it is the module to be trained; wherein, the teacher model in this embodiment With student model All architectures employ the Transformer encoder architecture based on the BERT-base-cased pre-trained language model, and their complete structure is as follows:

[0109] Input layer: The input is a token sequence processed by the built-in token segmenter of BERT-base-cased, which includes: word embedding: each token is mapped to a 768-dimensional vector; position embedding: sequence position information is added to each token; segment embedding: sentence pairs are distinguished; the sum of the three is used as the input representation of the model, with dimensions [8, sequence_length, 768].

[0110] Encoder body: Composed of 12 identical Transformer encoder layers stacked together, each layer has the following structure:

[0111] Multi-head self-attention layer: Contains 12 attention heads, each with a dimension of 64, maintaining a total dimension of 768; calculates the contextual attention weights between tokens and outputs a weighted feature vector;

[0112] Residual connection + layer normalization: Add the output of the attention layer to the input, and then perform layer normalization;

[0113] Feedforward network: The structure is Linear(768→3072)→GELU activation→Linear(3072→768); nonlinear feature transformation is performed on the attention output; residual connection and layer normalization are performed again to obtain the final output of the layer;

[0114] The output layer (adapted for NER tasks) adds a linear classification layer on top of the BERT encoder output:

[0115] Input: A 768-dimensional feature vector for each token output by the encoder;

[0116] Output: A logits vector mapped to the entity label space (dimension is the number of label categories), used for subsequent cross-entropy loss calculation and knowledge distillation.

[0117] Training method: During training, the model is optimized independently for each consecutive learning phase. The number of training epochs, learning rate, and batch size are set to 30, 9×10⁻, and 9×10⁻, respectively. 4 And 8.

[0118] Initialization: Load the pre-trained BERT-base-cased weights and randomly initialize the linear classifier of the output layer; Training data: Use only the training set where only new entities have labels and all other entities are non-entity types during continuous learning.

[0119] Training objective: Optimize model parameters using TCSD, CTDD, and cross-entropy as the target loss functions, enabling the model to learn new knowledge and retain old knowledge; update the student model after training. The parameters.

[0120] In the current incremental phase, let's assume there is already... The voice assistant has learned the entity types and introduced them. A voice assistant model needs to learn new entity types, and the model's goal is to expand the category space. The above are classified uniformly;

[0121] The current stage of new voice command sample data is recorded as follows: ,in, Indicates the number of samples. and They represent the first One sample of input voice commands and its corresponding label;

[0122] During training, the student model On the one hand, through classification loss Learning new categories of knowledge from current data; on the other hand, the teacher model trained in the previous stage... Under the guidance of this approach, knowledge of old categories is preserved through distillation loss;

[0123] Unlike traditional distillation methods that rely solely on logits alignment, this technical solution constrains knowledge through a structural distillation mechanism to improve the stability of the model during continuous learning.

[0124] Input voice command sample Let the output logits of the teacher model (old model) and the student model (new model) be respectively:

[0125] ;

[0126] in, This represents the logits prediction results of the teacher model on the entity type space already learned by the voice assistant. This represents the logits prediction results of the student model on the new entity type space to be learned by the voice assistant model;

[0127] S1-2: Define the classification loss (Cross-Entropy, CE) as follows:

[0128] ;

[0129] in, Indicates input voice command sample Classification loss, It is for real tags Indicator functions, This represents the student model's predicted probability for the correct category. The number of entity types that the voice assistant has learned. The total number of new entity types to be learned by the voice assistant model.

[0130] Meanwhile, we use the softmax function to transform the logits of the old categories into a probability distribution. and This is used for knowledge distillation, and the calculation method is as follows:

[0131] ;

[0132] in, For temperature coefficient, For the teacher model in the voice command The token corresponds to the first The logits value of the token-class dimension after standardization and Softmax calculation for item categories. For student models in the voice command The token corresponds to the first The logits value of the token-class dimension after standardization and Softmax calculation for item categories; For the student model Logits prediction results for the token-class dimension. For the student model Logits prediction results for the token-class dimension. For the teacher model Logits prediction results for the token-class dimension. For the teacher model Logits prediction results for the token-class dimension;

[0133] It is worth noting that in continuous incremental learning (CIL) scenarios, both cross-entropy (CE) and Kullback-Leibler divergence are widely used to calculate knowledge distillation loss. .

[0134] The knowledge distillation loss based on cross-entropy is defined as follows:

[0135] ;

[0136] in, This represents the knowledge distillation loss of the teacher model and the student model. For the teacher model in the voice command The token corresponds to the first The logits value of the token-class dimension after standardization and Softmax calculation for item categories. For student models in the voice command The token corresponds to the first The logits value of the token-class dimension after standardization and Softmax calculation for item categories;

[0137] KD loss based on KL The calculation is performed using the following formula:

[0138] ;

[0139] in, This represents the KD loss of the teacher model and the student model, where the teacher model performs the following on the first speech command. The token corresponds to the first The logits value of the token-class dimension after standardization and Softmax calculation for item categories. For student models in the voice command The token corresponds to the first The logits value of the token-class dimension after standardization and Softmax calculation for item categories. Temperature coefficient;

[0140] Finally, combining the cross-entropy loss with the knowledge distillation (KD) loss, the overall loss is defined as follows:

[0141] ;

[0142] in, This represents the total loss during the incremental learning phase of the voice assistant. Indicates input voice command sample Classification loss, KD loss based on KL;

[0143] S2: Train the teacher model and the student model;

[0144] Input voice command sample data is processed through teacher model and student model respectively;

[0145] S2-1: The teacher model processes the input data containing old entity types required by the voice assistant model;

[0146] S2-1-1: Sample of input voice command Input speech command samples are extracted using the TCSD (Token-wise Class-Structure Distillation) module of the teacher model. The discriminative structural relationship between different entity categories within a single token; the TCSD module is used to perform structured modeling of category logits within each token of the voice command, constraining only the relative ordering relationship between categories, while remaining insensitive to the scale and offset of their absolute values, thereby avoiding optimization conflicts caused by numerical alignment, and achieving more stable knowledge transfer.

[0147] Distillation preserves only the relative distribution structure between voice commands and categories within tokens, completely shielding the interference of the overall scale and global offset of logits;

[0148] S2-1-1-1: Sample input voice commands For each token, extract the raw logits of all entity categories on that token from the teacher model (old model). ;

[0149] S2-1-1-2: Perform Z-score normalization on all categories of logits within the same voice command token;

[0150] The formula is as follows:

[0151] ;

[0152] in, For the standardized first Logits prediction results of the teacher model under each item category. This represents the mean of logits for all categories in the teacher model under the current voice command token. This represents the standard deviation of logits for all categories in the teacher model under the current voice command token. Indicates the first Logits prediction results of the teacher model under each item category;

[0153] Z-score normalization is applied to the category logits of each voice command token to eliminate category-level global activation shifts while preserving the token's equi-order semantic structure. The normalized logits effectively characterize the discriminative structure of the same token across different entity categories, unaffected by changes in absolute activation amplitude; it eliminates global activation shifts and global numerical scaling caused by continuous learning; and it completely cancels out the overall upward / downward shifts and overall scaling caused by the introduction of new categories. Simultaneously, it strictly preserves the relative size order and relative distribution structure of all category logits within the same token.

[0154] S2-1-1-3: Apply temperature scaling to the standardized logits and calculate the class probability distribution;

[0155] A distillation temperature coefficient is introduced into the logits after Z-score normalization. Scaling is applied, and then the old entity category probability distribution of the teacher model on the voice command token is calculated using Softmax;

[0156] ;

[0157] in, For temperature coefficient, For the teacher model in the first voice command The token corresponds to the first The logits value of the token-class dimension after standardization and Softmax calculation for item categories. For the standardized first Logits prediction results of the teacher model under each item category. For the first Logits prediction results of the teacher model under each item category; The total number of old entity categories that the voice assistant has learned;

[0158] S2-1-2: Extract input speech command samples using the CTDD (Class-wise Token-Distribution Distillation) module of the teacher model. Global response distribution structure among all tokens within a single entity category;

[0159] S2-1-2-1: Transpose the logit matrix obtained during the distillation process of the voice command samples; transpose the token-class form into the class-token form, so that each row corresponds to the response trajectory of an entity class on all selected tokens;

[0160] Let the total number of tokens for the voice command samples be... For the first There are 1 entity categories, and their teacher model logit vectors on all voice command tokens are denoted as follows: ;

[0161] in, The logits vector representing the teacher model of the first token of the voice command; Indicates the first voice command The logits vector of the teacher model for each token;

[0162] S2-1-2-2: Perform Z-score standardization on all voice command tokens within the same category of logits;

[0163] The formula is as follows:

[0164] ;

[0165] in, For the class-token matrix in the teacher model, the first... The result after Z-score standardization, This represents the mean logits of all voice command tokens in the teacher model for the current category. This represents the standard deviation of the logits of all voice command tokens in the teacher model for the current category. This represents the first class-token in the teacher model. The value of the item;

[0166] S2-1-2-3: Apply temperature scaling to the standardized logits and calculate the class probability distribution;

[0167] A distillation temperature coefficient is introduced into the logits after Z-score normalization. Scaling is performed, and then the probability distribution of each category in the teacher's voice commands is calculated using Softmax.

[0168] ;

[0169] in, For temperature coefficient, In the teacher model, the first The first category The logits value of the class-token dimension after standardization and Softmax calculation of the voice command token. For the class-token matrix in the teacher model, the first... The result after Z-score standardization, For the teacher model class-token matrix, the first The result after Z-score standardization.

[0170] S2-1-3: After Z-score normalization of the category probability distribution extracted by the teacher model, it is split into two paths: one path is input into the teacher model classifier, and the other path is retained as a knowledge supervision signal.

[0171] S2-2: Process the input data containing new entity types required by the voice assistant model using the student model;

[0172] S2-2-1: Extract input voice command samples using the TCSD module of the student model. Discrimination structure relationships between different entity categories within a single token;

[0173] This distillation method retains only the relative distribution structure of old entity categories within the same voice command token, completely shielding the interference of the overall scale and global offset of logits between voice commands. This allows the student model to maintain stable class discrimination of old entities when learning new voice command entity categories.

[0174] S2-2-1-1: Sample input voice commands For each token, extract the original logits of all entity categories of the student model (new model) on that token. ;

[0175] S2-2-1-2: Perform Z-score standardization on the logits of all old entity categories within the same token;

[0176] The formula is as follows:

[0177] ;

[0178] in, For the standardized first Logits prediction results of the student model under each item category. This represents the mean of logits for all categories in the student model under the current voice command token. This represents the standard deviation of logits for all categories in the student model under the current voice command token. Indicates the first Logits prediction results of the student model under each item category;

[0179] Z-score normalization is applied to the category logits of each voice command token to eliminate category-level global activation shifts while preserving the token's equi-order semantic structure. The normalized logits effectively characterize the discriminative structure of the same voice command token across different entity categories, unaffected by changes in absolute activation amplitude. It eliminates global activation shifts and global numerical scaling caused by continuous learning, and cancels out the overall upward / downward shifts and overall scaling caused by the introduction of new categories. Simultaneously, it strictly preserves the relative size order and distribution structure of all category logits within the same voice command token.

[0180] S2-2-1-3: Apply temperature scaling to the standardized logits and calculate the class probability distribution;

[0181] A distillation temperature coefficient is introduced into the logits after Z-score normalization. Scaling is performed, and then the entity category probability distribution of the student model on the voice command token is calculated using Softmax;

[0182] ;

[0183] in, For temperature coefficient, For student models in the first voice command The token corresponds to the first The logits value of the token-class dimension after standardization and Softmax calculation for item categories. For the standardized first Logits prediction results of the student model under each item category. For the standardized first The logits prediction results of the student model. This represents the total number of old entity categories that the voice assistant has already learned. The total number of new entity types that the voice assistant model needs to learn;

[0184] S2-2-2: Extract input speech command samples using the CTDD (Class-wise Token-Distribution Distillation) module of the student model. The global response distribution structure among all tokens within a single entity category is defined. Specifically, the CTDD module models and constrains the response distribution of the same category across different tokens from a category perspective to maintain its cross-token distribution structure. This CTDD module complements the TCSD module in terms of modeling dimensions: the TCSD module focuses on the category discrimination structure within a single token, while the CTDD module focuses on the category distribution structure in the sequence. Together, they achieve robust preservation of old knowledge from both local discrimination and global distribution perspectives.

[0185] S2-2-2-1: Transpose the logit matrix obtained during the distillation process of the voice command samples; transpose it from token-class form to class-token form, so that each row corresponds to the response trajectory of an entity class on all selected tokens;

[0186] Let the number of tokens in the voice command sample be... For the first Let there be entity categories, whose logit vector for the student model across all tokens is denoted as . ;

[0187] in, This represents the logits vector of the student model for the first token of the voice command. Indicates the first voice command The logits vector of a student model with one token;

[0188] S2-2-2-2: Perform Z-score standardization on all voice command tokens within the same category of logits;

[0189] The formula is as follows:

[0190] ;

[0191] in, For the class-token matrix in the student model, the first... The result after Z-score standardization, This represents the mean logits of all voice command tokens in the student model for the current category. This represents the standard deviation of the logits of all voice command tokens in the student model for the current category. This represents the first element in the class-token matrix of the student model. The value of the item.

[0192] S2-2-3: Apply temperature scaling to the standardized logits and calculate the class probability distribution;

[0193] A distillation temperature coefficient is introduced into the logits after Z-score normalization. The scale is then applied, and then Softmax is used to calculate the probability distribution of each voice command sample category for the student model under that entity category;

[0194] ;

[0195] in, The temperature coefficient helps student models learn the distribution patterns of old entity categories in voice commands. In the student model, the first Category 1 The logits value of the class-token dimension after standardization and Softmax calculation for each voice command token category. For the class-token matrix in the student model, the first... The results of the item categories after Z-score standardization For the class-token matrix in the student model, the first... The results of the item categories after Z-score standardization.

[0196] S2-2-4: After normalizing the category probability distribution extracted from the student model, input it into the student model classifier;

[0197] S2-3: Calculate the cross-entropy loss by comparing the student model classifier output with the true labels;

[0198] The formula is as follows:

[0199] ;

[0200] in, Represents cross-entropy loss, It is for real tags The indicator function, which is used to label the true category of the training data for the voice assistant model, For the student model, on the input voice command sample data, for the first The probability that each token is predicted to belong to the true label category; For the entity types that the voice assistant has already learned, This represents the new entity type that the voice assistant model needs to learn.

[0201] S2-4: Calculate the in-token class relationship distillation loss between the teacher-student model and the student model based on Z-score;

[0202] The KL relative entropy (KL divergence) measures the difference in class probability distributions between the teacher and student models, and serves as the final distillation loss function to constrain the student model's learning.

[0203] ;

[0204] in, This represents the class distillation loss within a token, used to preserve the relative class relationships between old entities along the token dimension when the voice assistant model incrementally learns new entities; This represents the distillation loss based on KL divergence. The standard binary operator for representing KL divergence is specifically used to write the KL distance between two probability distributions. This is a temperature coefficient used to control the smoothness of the probability distribution; For the teacher model in the voice command The token corresponds to the first The logits value of the token-class dimension after standardization and Softmax calculation for item categories. For student models in the voice command The token corresponds to the first The logits value of the token-class dimension after standardization and Softmax calculation; O is the total number of old entity categories learned by the voice assistant model; This represents the logits distribution of the teacher model. Represents the logits distribution of the student model;

[0205] S2-5: Calculate the distillation loss of all tokens within a single class based on Z-score for the teacher-student model and the student model;

[0206] ;

[0207] in, Distillation loss, representing the distribution relationship within categories, is used to preserve the positional distribution pattern of old entity categories along the token dimension when the voice assistant incrementally learns new entities (e.g., time-related entities often appear at the end of sentences, and location-related entities often appear after prepositions). This represents the distillation loss based on KL divergence. The standard binary operator for representing KL divergence is specifically used to write the KL distance between two probability distributions. For temperature coefficient, In the teacher model, the first The first category The logits value of the class-token dimension after standardization and Softmax calculation of the voice command token. For the student model The logits value of the class-token dimension; t is the total number of tokens in the voice command sample data; Teacher model The probability distribution of the class-token dimension. Student model Probability distribution of the class-token dimension;

[0208] S2-6: Calculate the total loss;

[0209] The calculation is performed using the following formula:

[0210] ;

[0211] in, This represents the total loss during the incremental learning phase of the voice assistant. Cross-entropy loss is used to supervise the student model's recognition performance on the newly added entity categories. This represents the distillation loss of the category relationship within the token. Distillation loss is a distribution within a category; , They are respectively and The weighting coefficients.

[0212] S3: After training, the model is fused using the semantic-aware parameter fusion mechanism VSPF (Validation-GuidedSemantic-AwareParameterFusion) to obtain the final inference model;

[0213] The consistency between the teacher model and the student model in the old semantic space is directly extracted by the prediction behavior of the teacher model and the student model on the validation set, and this is used to guide parameter fusion.

[0214] S3-1: Validation dataset for a given voice assistant scenario Simultaneously input the teacher model and the trained student model, and obtain the full prediction logits of the teacher model and the student model on the validation dataset;

[0215] ;

[0216] in, This represents the complete logits vector output by the old model on this voice command validation sample. This indicates the classification header of the teacher model (old model) that was frozen and saved in the previous stage. This represents the complete logits vector output by the new model on this voice command validation sample. This represents the classification head of the student model (new model) that has completed training for the new task in the current stage;

[0217] S3-2: Dimensional clipping, retaining only the prediction information of the old category space and excluding interference from the new category;

[0218] S3-3: Concatenate and vectorize the prediction results of all voice command verification samples in the old entity category space to form a global prediction distribution representation of the old and new models;

[0219] S3-4: Calculate the semantic consistency of the global prediction distribution of the old and new models using cosine similarity to obtain the dynamic fusion weights;

[0220] ;

[0221] in, Indicates the fusion weights; Cosine similarity is used to measure the global predictive consistency between the old and new models on the old entity recognition task of voice commands.

[0222] This design allows the fusion weights to be explicitly correlated with the model's actual predictive behavior on the validation set, avoiding the introduction of additional nonlinear mappings or heuristic adjustment strategies. A high consistency score indicates that the new model's prediction distribution in the old category semantic space maintains high consistency with the old model. In this case, the weights of the new model's parameters in the fusion process can be appropriately increased to enhance the model's adaptability to new tasks. Conversely, a low consistency score indicates that the new model has significantly shifted from the old semantic space. Reducing its fusion weights can effectively suppress semantic drift, thereby reducing the risk of catastrophic forgetting.

[0223] S3-5: Update the final model parameters by merging all parameters of the teacher model and the student model through semantic weighting;

[0224] The formula is as follows:

[0225] ;

[0226] in, This represents the model parameters that were frozen and saved in the previous stage. This represents the model parameters obtained during the current stage of training on the new task. This indicates the final model parameters of the voice assistant obtained after updating the parameters;

[0227] S3-6: Output the final model for continuous entity recognition tasks using voice commands;

[0228] S4: Output the results using the final inference model. In use, the text sentence from the speech recognition is input into the final inference model of this technical solution to identify entities, understand, and execute actions.

[0229] To comprehensively verify the effectiveness and generalization ability of this technical solution in continuous named entity recognition tasks across different domains, this paper conducted experiments on three widely used public NER datasets: CoNLL, I2B2, and OntoNotes (Onto). These three datasets cover different application scenarios such as news, medical, and general text, and have significantly different language styles and entity distribution characteristics, which can better simulate cross-domain and cross-task continuous learning scenarios in real applications. The specific statistical information of each dataset is summarized in Table 1.

[0230] Table 1: Dataset Statistics (sent is an abbreviation for sentence)

[0231]

[0232] In Table 1, the entity types in the CoNLL dataset, translated into Chinese in the following order, are: Location, Miscellaneous, Institution, and Person's Name; the I2B2 dataset's are: Age, City, Country, Date, Doctor, Hospital, ID Number, Medical Record, Institution, Patient, Phone Number, Occupation, State, Username / Account Name, and Postal Code; and the Onto dataset's are: Cardinal Number, Date, Event, Facility, Geopolitical Entity, Language, Law / Regulations, Location / Geographical Entity, Currency Amount, Ethnic / Religious / Political Group, Ordinal Number, Organization / Institution, Percentage, Person / Name, Product / Item, Quantity, Time Point, and Work of Art.

[0233] In our experiments, we adopted and followed the continuous named entity recognition experimental setup proposed by Zhengetal to construct a standard and reproducible continuous learning process. Entity types in each dataset were divided into several non-overlapping entity groups, introduced sequentially in alphabetical order to simulate a continuous learning scenario where entity categories gradually expand over time. In the first stage, the model was trained only on the first group of entity types, denoted as FG. In subsequent stages, the model, while retaining the learned entity types, progressively introduced a new group of entity types, denoted as PG, in each stage. The overall setup is represented in the form FG–A–PG–B, where A and B represent the number of entity types introduced in the first and subsequent stages, respectively.

[0234] The specific settings used on different datasets are consistent with Zheng et al., with CoNLL2003 using FG-1-PG-1 and FG-2-PG-1, and I2B2 and OntoNotes using FG-1-PG-1, FG-2-PG-2, FG-8-PG-1, and FG-8-PG-2. In terms of data construction, the training set for each dataset is divided into multiple stages, corresponding to different steps in the continuous learning process. In each stage, only the label information of entity types introduced in the current stage is retained in the sentences, and the labels of entity types that have not appeared are uniformly replaced with 'O', thus strictly avoiding the leakage of future entity information. The selection of training samples adopts the greedy sampling strategy proposed by Zheng et al. to ensure that all entity types in the current stage are covered as broadly as possible within each stage. The construction method of the validation set is consistent with that of the training set, used for model selection and transition between stages; while in the testing stage, all entity type labels that have appeared in the learned stages are retained, and entity types representing future stages are uniformly replaced with 'O', to evaluate the model's overall ability to retain historical entity knowledge in the current stage. In each stage, the model that performs best on the validation set is selected as the old model for the next stage and used in subsequent continuous learning processes.

[0235] We employed the BIO annotation scheme for entity labeling on all three datasets. The backbone model used was a BERT-based cased pre-trained model implemented using the HuggingFace framework. During training, each consecutive learning phase was optimized independently, with the number of training epochs, learning rate, and batch size set to 30, 9×10⁻, and 9×10⁻, respectively. 4 And 8. The natural language description form of entity labels follows the design proposed by Maetal. Our model is implemented using an NVIDIA GeForce RTX 3090 GPU within the PyTorch framework.

[0236] Based on existing practices in continuous named entity recognition research, this paper uses two evaluation metrics, MicroF1 and MacroF1, to comprehensively evaluate the model performance.

[0237] The definitions of these two fractions are as follows:

[0238] ;

[0239] ;

[0240] ;

[0241] Among them, a total Entity categories, It represents the actual number of positive samples that were predicted as positive samples. It represents the number of negative samples that were actually predicted as positive samples. It represents the actual number of negative samples that are predicted as negative samples.

[0242] Micro-F1 calculates the precision and recall of all entity predictions at an overall level, focusing more on the recognition performance of high-frequency entity types; while Macro-F1 uses an equal-weighted average of the F1 scores for each entity type, which can better reflect the overall performance of the model on different entity types, especially low-frequency entity types.

[0243] The baselines compared in the experiment are summarized as follows:

[0244] The OnlyFinetuning model is documented in Devlin's 2019 paper, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding." This model does not introduce any anti-forgetting mechanisms; it only performs supervised fine-tuning of the model in each consecutive learning phase.

[0245] The PODNet model is documented in "Pooled outputs distillation for small-tasks incremental learning" published by Douillard, Cord, Ollion, Robert, and Valle in 2020. This model obtains a flexible and stable class representation by introducing a spatial structure-based distillation loss constraint on the model parameter evolution process and maintaining multiple proxies for each class.

[0246] The LUCIR model is documented in "Learning a unified classifier incrementally via rebalancing" published by Hou, Pan, Loy, Wang, and Lin in 2019. This model addresses the imbalance problem between old and new class samples in continuous learning by employing three strategies: constraining the magnitude distribution of weights for different classes, preserving the geometric structure information of the old class, and increasing the distance between the old and new classes in the feature space to mitigate class drift.

[0247] The Self-Training model is described in "Semi-supervised self-training of object detection models" published by Rosenberg, Hebert, and Schneiderman in 2005. This model uses the old model to generate pseudo-labels for old entity types in the current training data at each stage, and trains a new model based on the updated labeled data, thereby preserving old knowledge without additional labels.

[0248] The ExtendNER model, described in Monaikul's 2021 paper "Continual learning for name identity recognition," dynamically expands the output dimension of the classifier at each continuous learning stage and mitigates catastrophic forgetting at both the structural and objective levels by constraining the new model's predicted distribution of old classes through distillation loss based on KL divergence.

[0249] The CFNER model is documented in "Distilling causal effect from miscellaneous other-class for continuous name identity recognition" published by Zheng in 2022. Starting from the causal relationship between entity types, the model introduces a course learning mechanism to guide the model to adapt to new entity types, so as to reduce the performance degradation caused by label noise during continuous learning.

[0250] The ExtendNER+DLD model is described in "Decomposing logits distillation for incremental name identity recognition" published by Zhang Yu in 2023. This model introduces Decomposing Logits Distillation (DLD) on the basis of ExtendNER. By decomposing logits and applying distillation constraints, the model enhances its ability to distinguish old entity types.

[0251] The CFNER+DLD model is documented in "Decomposing logits distillation for incremental name identity recognition" published by Zhang Yu in 2023. This model combines the DLD distillation strategy with CFNER, which improves the model's ability to distinguish semantics of old entity types while maintaining the advantages of course learning.

[0252] The RDP model is documented in "Task Relation Distillation and Prototypical Pseudolabel for Incremental Name Identity Recognition" published by Zhang Li in 2023. This model explicitly distills the relationships between different tasks in each successive learning phase and relabels tokens marked as 0 through a prototype-driven pseudolabel strategy, thereby mitigating the catastrophic forgetting problem.

[0253] The CPFD model is documented in "Continual namedentity recognition without catastrophic forgetting" published by Zhang Cong in 2023. The CPFD model uses pooling feature distillation loss to preserve linguistic knowledge and balance model stability and plasticity. It proposes a confidence-based pseudo-label strategy to identify past entity types and solve the semantic shift problem.

[0254] The EDAW model is documented in "Enhanced knowledge distillation and adaptive pseudolabel weights for continuous name identity recognition" published by Sheng Zhang in 2024. This model designs an enhanced knowledge distillation method to significantly reduce catastrophic forgetting by aligning the probability and feature distributions of continuous models. It also proposes an entropy-based adaptive pseudolabel weighting mechanism to reduce the accumulation of pseudolabel errors and effectively handle the semantic offset problem.

[0255] The RFSI model is documented in "Reduced Forgetfulness in Continual Learning for Named Entity Recognition Through Confident Soft-Label Imitation" published by Zhang Zhou in 2024. This model proposes a feature imitation-based continuous learning method for named entity recognition, designs an improved version of knowledge distillation loss to enhance the learning of old task knowledge to optimize the original CPFD, proposes a confidence-based soft pseudo-label method to retain old task knowledge, and introduces a balancing factor to distinguish between hard labels and soft labels to optimize the original pseudo-label strategy.

[0256] The FALCON model is documented in "Concept-driven knowledge distillation and pseudolabel generation for continuous name identity recognition" published by LiuXin in 2025. This model uses FCA concept lattice to represent the deep knowledge structure of the old model and proposes a concept-driven knowledge distillation scheme and pseudolabel generation strategy to solve the problems of catastrophic forgetting caused by the generalization ability of neural networks and background shift, respectively.

[0257] The experimental results comparing our model with baselines on three datasets are shown in Tables 2-6. Our method achieves superior or competitive performance in most continuous learning settings. The following analysis examines these results from three aspects.

[0258] Analysis of overall model performance: Experimental results in Tables 2-6 show that our method outperforms existing baseline methods in various continuous learning settings across the three datasets. Under several typical settings, our method surpasses existing baseline methods in both micro-F1 and macro-F1 average metrics. For example, considering the FG-1 setting on the CoNLL, I2B2, and Onto datasets, our model improves the MicroF1 score by 0.29%, 1.35%, and 1.50% compared to the second-best model, respectively; and improves the MacroF1 score by 0.25%, 2.18%, and 4.03%, respectively. This demonstrates that our method effectively maintains the discriminative structure of existing knowledge while continuously introducing new entity types, thereby improving learning performance. We will further analyze the specific contributions of each component in ablation experiments.

[0259] On the other hand, as shown in Table 4, we also observed that under the FG-8-PG-1 setting on I2B2 (a scenario where entity types in some datasets have high token-level similarity), a certain degree of confusion may occur between different entity types. These entities are often highly similar in word form or structure, such as containing a large number of numbers or sharing similar contextual patterns, thus increasing the difficulty for the model to distinguish them. In this case, our proposed method still maintains a high Micro-F1 score (e.g., 80.91%) in the later stages, improving it by 2.27% compared to the second-best model. This result indicates that the distribution-preserving structure distillation strategy can effectively alleviate the category confusion problem by constraining the class discriminant relation (TCSD) within tokens and the token distribution structure (CTDD) between categories when the categories are highly similar, thereby improving the stability and robustness of the model during continuous learning.

[0260] It's worth noting that the performance improvement on the CoNLL dataset is relatively small. We believe this is because CoNLL only includes four entity types, resulting in a smaller number of entity categories and less significant forgetting of old knowledge during continuous learning. Our proposed approach shows greater advantage with increasing number of types; for example, on the Onto dataset with the most types (18 types), our results consistently outperform the baseline. Even so, our proposed approach still achieves performance improvements of 0.29% and 0.25% over the baseline on the FG-1-PG-1 setting of the CoNLL dataset, demonstrating that our proposed structural distillation and semantic-aware parameter fusion strategy has good generalization ability under different data complexities.

[0261] Table 2: Comparison results of this technical solution with other baseline models on the CoNLL dataset based on FG-1-PG-1 and FG-2-PG-1 settings.

[0262]

[0263] Table 3: Comparison results of this technical solution with other baseline models on the I2B2 dataset based on FG-1-PG-1 and FG-2-PG-2 settings.

[0264]

[0265] Table 4: Comparison results of this technical solution with other baseline models based on FG-8-PG-1 and FG-8-PG-2 settings on the I2B2 dataset.

[0266]

[0267] Table 5: Comparison results of this technical solution with other baseline models on the Onto dataset based on FG-1-PG-1 and FG-2-PG-2 settings.

[0268]

[0269] Table 6: Comparison results of this technical solution with the baseline model based on FG-8-PG-1 and FG-8-PG-2 settings on the Onto dataset.

[0270]

[0271] The baseline results in Tables 2-6, excluding those of this technical solution, are from Liu Xin's 2025 publication, "Concept-driven knowledge distillation and pseudolabel generation for continual name identity recognition".

[0272] To further analyze the model's ability to handle catastrophic forgetting, we examined its performance at each step on three datasets. Our model is compared with experimental results from several baselines, for example... Figure 2-5 As shown in the results, our model achieves optimal performance in most steps.

[0273] For example, in the FG-8-PG-1 setting of the I2B2 dataset, the EDAW model's MicroF1 score drops to approximately 72% in step 9, a decrease of about 16% from the initial value; while our model only drops from approximately 88% to approximately 78%, a decrease of about 10% from the initial value. In the FG-8-PG-2 setting, the EDAW model's MicroF1 score drops to approximately 76% in step 5, a decrease of about 12%; while our model drops from approximately 88% to approximately 82%, a decrease of only about 6%. This result shows that our method can more effectively suppress performance degradation during the continuous introduction of new categories. This advantage mainly stems from the structural distillation mechanism proposed in this paper. TCSD constrains the discriminative relationships between different categories within a single token, enabling the model to maintain a stable category ranking even when the logits distribution undergoes scaling changes, thereby avoiding discrimination boundary degradation caused by numerical shifts; CTDD constrains the response distribution on different tokens from a category perspective, preventing the compression or expansion of the overall activation pattern of old categories during the training of new tasks, thus mitigating implicit semantic drift. The two work together to enable the model to maintain consistency between local discriminative structure and global distribution during the category expansion process.

[0274] Furthermore, we analyzed the performance changes of the old and new entity types at each step. Our model exhibits excellent recognition ability for the new entity type, while maintaining higher recognition accuracy for the old entity type at each step, demonstrating stronger knowledge inheritance. For example, in the FG-1-PG-1 setting on I2B2, our model achieved a MacroF1 score of approximately 53% for the old entity type in the final step, while the CFNER model only achieved approximately 28%, a significant difference. Even compared with baselines such as FALCON and CPFD, our model outperformed them by approximately 8% and 10%, respectively. This indicates that our model inherited more knowledge from the old model, enhancing its robustness. Further, this result demonstrates that TCSD effectively alleviates the confusion between the old and new categories by maintaining the class discriminative relationship at the token level; while CTDD avoids the overall degradation of the semantic representation of the old categories by maintaining the response distribution of the categories along the token dimension. By combining VSPF's semantic-aware fusion mechanism in the parameter space, the model can dynamically adjust knowledge weights based on the predictive consistency between the old and new models, thereby maximizing the retention of historical information while learning new knowledge and improving the model's stability and generalization ability during continuous learning.

[0275] In this section, we conduct ablation experiments to verify the effectiveness of each proposed component. The variables used in the ablation experiments are summarized below:

[0276] w / oTCSD: Remove the Token-wise Class-Structure Distillation (TCSD) module from the complete model, that is, no longer apply distillation constraints to the discriminative structure between different entity categories within the token.

[0277] w / oCTDD: Remove the Class-wise Token Distribution Distillation (CTDD) module from the complete model, meaning that the distribution structure of the same entity category across different token dimensions is no longer distilled.

[0278] w / oTCSD+CTDD: Simultaneously remove both TCSD and CTDD distillation modules, retaining only supervised training and parameter update strategies, to analyze the overall contribution of the structural distillation mechanism to model performance.

[0279] w / oVSPF: Remove the Validation-based Semantic-aware Parameter Fusion (VSPF) parameter fusion strategy from the complete model, that is, no longer use the consistency of the validation set predictions to perform semantic-aware fusion of the parameters of the old and new models.

[0280] The ablation experiment results on the Onto datasets FG-8-PG-1 and FG-8-PG-2 are shown in Table 7. When different modules were removed from the complete model, the model's Micro-F1 and Macro-F1 scores decreased to varying degrees, indicating that each component played a key role in the continuous learning process.

[0281] Table 7: Ablation Experiment Results of this Technical Solution on FG-8-PG-1 and FG-8-PG-2 Datasets of the Onto Dataset

[0282]

[0283] As shown in Table 7 and Figure 6As shown, under the FG-8-PG-1 setting, removing the VSPF, CTDD, and TCSD components respectively resulted in a decrease in the model's Micro-F1 (micro-average F1) value of 0.59%, 0.92%, and 1.27%. Removing both TCSD and CTDD simultaneously led to an overall decrease in Micro-F1 of 1.58%. The removal of TCSD caused the most significant performance degradation, indicating that the class discriminative relationships at the token level play a crucial role in maintaining model stability during continuous learning. When the logits distribution exhibits scale shift, TCSD effectively maintains the relative ranking relationships between different classes, thus preventing the new task training from directly damaging the old class discriminative boundaries. The removal of CTDD also caused model performance degradation, demonstrating that relying solely on the discriminative relationships within tokens is insufficient to fully preserve old knowledge. CTDD, by constraining the response distribution across different tokens from a class perspective, effectively alleviates the problem of compression or expansion of the overall activation patterns of old classes during new task training, thereby reducing the occurrence of class-level semantic drift. This result further confirms that the distribution structure information within a category is of paramount importance for the long-term retention of old knowledge.

[0284] Furthermore, although the performance degradation caused by removing VSPF is relatively mild, it still has a stable effect on improving model performance. This means that, based on structural distillation, semantically aware fusion at the parameter level can further reconcile the conflicts between new and old knowledge. Compared with static fusion strategies based on the number of categories, VSPF dynamically adjusts the fusion weights by leveraging prediction consistency on the validation set, thus achieving a more reasonable knowledge balance at different incremental stages.

[0285] Notably, the model performance degraded most significantly when both TCSD and CTDD were removed simultaneously, indicating a clear complementary role between the two in the modeling process: TCSD primarily maintains the class discrimination structure at the token level, while CTDD constrains the distribution pattern across tokens at the class level. They function from both local and global perspectives, jointly helping the model stably retain old knowledge during class expansion, ultimately significantly improving the model's continuous learning performance.

[0286] To analyze the role of each module in the continuous learning process, we conducted a stepwise ablation experiment on the I2B2 dataset. The results are as follows: Figure 6As shown, the complete model achieved optimal performance at each stage, while removing any module led to a performance decline, with this difference becoming more pronounced in later stages, indicating that each module plays a crucial role in mitigating catastrophic forgetting. For example, in the FG-8-PG-1 setting, the Micro-F1 score of the complete model was 82.0% in the final stage (Step 5). Removing TCSD resulted in a performance drop to 79.5%, the largest decrease among all single modules; removing CTDD resulted in a drop to 80.2%; and removing VSPF resulted in a drop to only 81.1%. When both TCSD and CTDD were removed simultaneously, the performance further declined to 79.0%.

[0287] Further analysis from a stage evolution perspective reveals that TCSD's impact is particularly significant in the mid-to-late stages (e.g., Step 3–5), indicating that it effectively stabilizes the decision boundary by constraining the relative discriminative relationships of categories within tokens, thereby mitigating the ranking perturbation problem caused by category expansion. In contrast, CTDD's performance decline is more gradual, but accumulates slowly in later stages, suggesting that it effectively suppresses semantic drift and maintains overall consistency at the category level by constraining the response distribution of categories on different tokens.

[0288] The significant performance degradation upon removing TCSD and CTDD indicates that they complement each other by modeling historical knowledge from two orthogonal structural perspectives: token-class and class-token, respectively. In contrast, the removal of VSPF only resulted in a small but stable performance degradation, suggesting that it primarily mitigates the conflict between old and new knowledge at the parameter level. While its effect is not as direct as structural distillation, it provides additional stability assurance in the later stages of continuous learning.

[0289] Therefore, TCSD and CTDD constitute the core structural constraint mechanism for mitigating catastrophic forgetting, while VSPF provides auxiliary support from the perspective of parameter fusion. The three work together to enable the model to maintain the stability of the discriminative structure and the consistency of semantic distribution during the class expansion process, thereby improving the overall performance of continuous learning.

[0290] To analyze the impact of distillation loss weights on model performance, we tested the hyperparameters on the Onto dataset. and A two-dimensional grid search was performed, in which and The values ​​range for all values ​​are {0.1, 0.3, 0.5, 0.7, 1.0}. The experimental results are as follows: Figure 7 As shown, the Micro-F1 (micro-average F1) and Macro-F1 (macro-average F1) performances of the model under different parameter combinations are illustrated. It can be observed that when... and As the value gradually increases from a smaller value, the overall model performance shows a trend of first increasing and then slightly decreasing. This is because this phenomenon mainly stems from the trade-off between distillation constraints and learning new tasks. When and When the value is small, the structural distillation constraint is weak, and the model tends to rely more on the data of the current stage for optimization, resulting in insufficient ability to retain old knowledge and thus low performance. With... and As the number of tokens increases, the structural constraints of TCSD and CTDD on historical knowledge gradually become stronger, enabling the model to better maintain the class discrimination relationship within tokens and the distribution consistency within classes, thereby effectively mitigating catastrophic forgetting and improving overall performance.

[0291] However, when and As the distillation loss increases further, its proportion in the overall objective function becomes too high, excessively limiting the model's adaptability to new tasks and making it difficult for the model to fully learn newly introduced entity types, resulting in a slight performance decrease. Therefore, the model performance exhibits a trend of first increasing and then decreasing in relation to the distillation intensity and the ability to learn new tasks. and When the value is in the range of 0.3–0.7, the model performance changes relatively steadily. and At that time, the model achieved good results on both Micro-F1 and Macro-F1. Therefore, in subsequent experiments, we will... , As the default hyperparameter setting.

[0292] We selected a representative case from the second step of the FG-1-PG-1 setup on CoNLL to illustrate the effectiveness of our model. In this step, MISC is the only new entity type, while [LOCATION](LOC) is the old entity type.

[0293] Table 8: Classification of the same input sentence in CoNLL by this technical solution and traditional methods

[0294]

[0295] As shown in Table 8, a real-world case from the CoNLL test set, B- and i- distinguish the start / inside of an entity. O, LOC, and MISC represent non-entity category, location, and miscellaneous, respectively. All prediction results correspond to the second step of FG-1-PG-1.

[0296] As shown in Table 8, errors in Continuous Named Entity Recognition (CNER) mainly stem from two aspects: first, category discrimination errors; and second, inconsistent token-level responses within the same entity. Specifically, during continuous learning, when the model learns a new category (such as MISC), previously learned entity categories (such as LOC) are uniformly labeled as O, introducing significant semantic conflicts. This background shift causes confusion in category discrimination; for example, the token "Asian" might oscillate between LOC and MISC, reflecting a disruption of the category discrimination structure. Simultaneously, due to the lack of constraints on token distribution within categories, the model may produce inconsistent predictions within the same entity, leading to incorrect entity boundary segmentation (e.g., "AsianCup" being partially recognized).

[0297] To address the aforementioned issues, this paper proposes TCSD, which constrains the relative ranking relationships between different categories at the token level, maintaining a stable category discrimination structure even when distributions change, thereby reducing confusion between semantically similar categories. CTDD, on the other hand, models and constrains the response distribution of the same entity category across different tokens from a category perspective, ensuring consistency in predictions within entities and maintaining the integrity of entity boundaries. The synergistic effect of these two approaches enables the model to more stably and accurately identify complex entities (such as AsianCup) in continuous learning scenarios.

[0298] Table 9: Classification of the same input sentence in the OntoNotes5 test set by this technical solution and traditional methods

[0299]

[0300] As shown in Table 9, in the current learning step, the model introduces a new entity type, PER, while DATE, EVENT, NORP, and ORG are all previously learned entity types. Traditional methods are prone to entity structure disruption and category confusion during continuous learning, especially with complex entities (such as EVENT) and time-based entities (such as DATE), where significant catastrophic forgetting is more likely to occur, manifesting as broken entity boundaries or category prediction shifts. In contrast, our proposed method constrains the category discrimination structure within tokens using TCSD and maintains the consistency of category distribution across tokens using CTDD, thereby effectively preserving the semantic structure and boundary information of old categories while learning new tasks, thus mitigating the catastrophic forgetting problem.

[0301] In summary, this technical solution proposes a structural distillation and semantically aware parameter fusion framework for Continuous Named Entity Recognition (CNER) to alleviate the catastrophic forgetting problem in long-term online learning of voice assistants. Specifically, we design Token-wise Class-Structure Distillation (TCSD) to maintain the discriminative structural relationships between different entity categories; simultaneously, we propose Class-wise Token Distribution Distillation (CTDD) to constrain the distribution information of the same entity category across different token dimensions, thereby preserving historical knowledge from both structural relationship and distribution characteristics perspectives. Furthermore, we introduce a Validation-guided Semantic-aware Parameter Fusion (VSPF) parameter fusion strategy, using validation set prediction consistency to perform semantically aware fusion of new and old model parameters, achieving a dynamic balance between new and old knowledge. Extensive experimental results demonstrate that the proposed method achieves superior Micro-F1 and Macro-F1 performance compared to existing methods on multiple datasets and CNER settings, effectively mitigating the catastrophic forgetting problem in long-term online learning of voice assistants and improving continuous learning performance.

Claims

1. A distributed structure distillation method for persistent named entity recognition, characterized in that: Includes the following steps: S1: Initialize the teacher and student models; S2: Train the teacher model and the student model; S2-1: The teacher model processes the input data containing old entity types required by the voice assistant model; S2-2: Process the input data containing new entity types required by the voice assistant model using the student model; S2-3: Calculate the cross-entropy loss by comparing the student model classifier output with the true labels; The formula is as follows: ； in, Represents cross-entropy loss, It is for real tags The indicator function is used to label the true category of the training data for the voice assistant model. This indicates that the student model, on the input voice command sample data, for the first... The probability that each token is predicted to belong to the true label category; This represents the total number of entity types that the voice assistant has learned. The total number of new entity types that the voice assistant model needs to learn; S2-4: Calculate the in-token class relationship distillation loss between the teacher-student model and the student model based on Z-score; ； in, This represents the class distillation loss within a token, used to preserve the relative class relationships between old entities along the token dimension when the voice assistant model incrementally learns new entities. For temperature coefficient, For the teacher model in the voice command The token corresponds to the first The logits value of the token-class dimension after standardization and Softmax calculation for item categories. For student models in the voice command The token corresponds to the first The logits value of the token-class dimension after standardization and Softmax calculation for item categories; The total number of old entity categories that the voice assistant has learned; S2-5: Calculate the distillation loss of all tokens within a single class based on Z-score for the teacher-student model and the student model; ； in, The distillation loss, representing the distribution relationship within categories, is used to preserve the positional distribution pattern of old entity categories along the token dimension when a voice assistant incrementally learns new entities. For temperature coefficient, In the teacher model, the first The first category The logits value of the class-token dimension after standardization and Softmax calculation of the voice command token. In the student model, the first The first category The logits value of the class-token dimension after standardization and Softmax calculation of the voice command token; S2-6: Calculate the total loss; The calculation is performed using the following formula: ； in, This represents the total loss during the incremental learning phase of the voice assistant. Cross-entropy loss is used to supervise the student model's recognition performance on the newly added entity categories. This represents the distillation loss of the category relationship within the token. Distillation loss is a distribution within the category. , They are respectively and Weighting coefficients; S3: After training, the model is fused using the semantically aware parameter fusion mechanism VSPF to obtain the final inference model; S4: Output the results using the final inference model.

2. The distributed structure distillation method for persistent named entity recognition according to claim 1, characterized in that: Step S2-1 also includes: S2-1-1: Sample of input voice command Input voice command samples are extracted using the TCSD module of the teacher model. Discrimination structure relationships between different entity categories within a single token; S2-1-2: Extracting input voice command samples using the CTDD module of the teacher model. Global response distribution structure among all tokens within a single entity category; S2-1-3: After Z-score normalization of the category probability distribution extracted by the teacher model, the distribution is split into two paths: one path is input into the teacher model classifier, and the other path is retained as a knowledge supervision signal.

3. The distributed structure distillation method for persistent named entity recognition according to claim 2, characterized in that: Step S2-1-1 also includes: S2-1-1-1: Sample input voice commands For each token, extract the raw logits of all entity categories of the teacher model on that token. ; S2-1-1-2: Perform Z-score normalization on all categories of logits within the same voice command token; The formula is as follows: ； in, For the standardized first Logits prediction results of the category teacher model under the item category. This represents the mean of logits for all categories in the teacher model under the current voice command token. This represents the standard deviation of logits for all categories in the teacher model under the current voice command token. Indicates the first Logits prediction results of the teacher model under each item category; S2-1-1-3: Apply temperature scaling to the standardized logits and calculate the class probability distribution; A distillation temperature coefficient is introduced into the logits after Z-score normalization. Scaling is applied, and then the old entity category probability distribution of the teacher model on the voice command token is calculated using Softmax; ； in, For temperature coefficient, For the teacher model in the first voice command The token corresponds to the first The logits value of the token-class dimension after standardization and Softmax calculation for the old entity category. For the standardized first Logits prediction results of the teacher model under each item category. For the standardized first Logits prediction results for the teacher model under each item category.

4. The distributed structure distillation method for persistent named entity recognition according to claim 2, characterized in that: Step S2-1-2 also includes: S2-1-2-1: Transpose the logit matrix obtained during the distillation process of the voice command samples; Let the number of tokens selected during the distillation process be... For the first There are entities, and their teacher model logit vectors on all tokens are denoted as follows: ; in, The logits vector representing the teacher model of the first token of the voice command; Indicates the first voice command The logits vector of the teacher model for each token; S2-1-2-2: Perform Z-score standardization on all tokens within the same category of logits; The formula is as follows: ； in, For the class-token matrix in the teacher model, the first... The result after Z-score standardization, This represents the mean logits of all voice command tokens in the teacher model for the current category. This represents the standard deviation of the logits of all voice command tokens in the teacher model for the current category. This represents the first class-token in the teacher model. The value of the item; S2-1-2-3: Apply temperature scaling to the standardized logits and calculate the class probability distribution; A distillation temperature coefficient is introduced into the logits after Z-score normalization. Scaling is applied, and then the class probability distribution of teachers is calculated using Softmax; ； in, For temperature coefficient, In the teacher model, the first The first category The logits value of the class-token dimension after standardization and Softmax calculation of the voice command token. For the teacher model class-token matrix, the first The result after Z-score standardization, For the teacher model class-token matrix, the first The result after Z-score standardization.

5. The distributed preserving structure distillation method for persistent named entity recognition according to claim 1, characterized in that: Step S2-2 also includes: S2-2-1: Extract input voice command samples using the TCSD module of the student model. Discrimination structure relationships between different entity categories within a single token; Distillation retains only the relative distribution structure between categories within the same token, completely shielding the interference of the overall scale and global offset of logits; S2-2-2: Extract input speech command samples using the CTDD module of the student model. Global response distribution structure among all tokens within a single entity category; S2-2-3: Apply temperature scaling to the standardized logits and calculate the class probability distribution; S2-2-4: After normalizing the class probability distribution extracted from the student model using Z-score, input it into the student model classifier.

6. The distributed preserving structure distillation method for persistent named entity recognition according to claim 5, characterized in that: Step S2-2-1 also includes: S2-2-1-1: Sample input voice commands For each token, extract the raw logits of all entity categories of the student model associated with that token. ; S2-2-1-2: Perform Z-score normalization on the logits of all old entity categories within the same voice command token; The formula is as follows: ； in, For the standardized first Logits prediction results of the student model under each item category. This represents the mean of logits for all categories in the student model under the current voice command token. This represents the standard deviation of logits for all categories in the student model under the current voice command token. Indicates the first Logits prediction results of the student model under each item category; S2-2-1-3: Apply temperature scaling to the standardized logits and calculate the class probability distribution; A distillation temperature coefficient is introduced into the logits after Z-score normalization. Scaling is performed, and then the entity category probability distribution of the student model on the voice command token is calculated using Softmax; ； in, For temperature coefficient, For the student model in the voice command of the voice command The token corresponds to the first The standardized logits values ​​of the token-class dimension of the student model by category, calculated using Softmax. For the standardized first Logits prediction results of the student model under each item category. For the standardized first Logits prediction results of the student model under each item category. This represents the total number of old entity categories that the voice assistant has already learned. This represents the total number of new entity types that the voice assistant model needs to learn.

7. The distributed preserving structure distillation method for persistent named entity recognition according to claim 5, characterized in that: Step S2-2-2 also includes: S2-2-2-1: Transpose the logit matrix obtained during the distillation process of the voice command samples; Let the total number of voice command tokens in the voice command sample be... For the first Let there be entity categories, whose logit vector for the student model across all tokens is denoted as . ; in, This represents the logits vector of the student model for the first token of the voice command. Indicates the first voice command The logits vector of a student model with one token; S2-2-2-2: Perform Z-score standardization on all tokens within the same category of logits; The formula is as follows: ； in, For the class-token matrix in the student model, the first... The result after Z-score standardization, This represents the mean logits of all voice command tokens in the student model for the current category. This represents the standard deviation of the logits of all voice command tokens in the student model for the current category. This represents the first element in the class-token matrix of the student model. The value of the item; Step S2-2-3 also includes: A distillation temperature coefficient is introduced into the logits after Z-score normalization. The scale is then applied, and then Softmax is used to calculate the probability distribution of each voice command sample category for the student model under that entity category; ； in, For temperature coefficient, In the student model, the first Category 1 The logits value of the class-token dimension after standardization and Softmax calculation for each voice command token category. For the class-token matrix in the student model, the first... The result after Z-score standardization, For the class-token matrix in the student model, the first... The result after Z-score standardization.

8. The distributed preserving structure distillation method for persistent named entity recognition according to claim 1, characterized in that: Step S3 also includes: S3-1: Validation dataset for a given voice assistant scenario Simultaneously input the teacher model and the trained student model, and obtain the full prediction logits of the teacher model and the student model on the validation dataset; ； in, This represents the complete logits vector output by the old model on this voice command validation sample. This indicates the teacher model classification headers that were frozen and saved in the previous stage. This represents the complete logits vector output by the new model on this voice command validation sample. This represents the student model classification head that has completed training for the new task in the current stage; S3-2: Dimension clipping, retaining only the prediction information from the old category space; S3-3: Concatenate and vectorize the prediction results of all voice command verification samples in the old class space to form an overall prediction distribution representation; S3-4: Calculate the semantic consistency of the global prediction distribution of the old and new models using cosine similarity to obtain the dynamic fusion weights; ； in, Indicates the fusion weights; Cosine similarity is used to measure the global predictive consistency between the old and new models on the old entity recognition task of voice commands. S3-5: Update the final model parameters by merging all parameters of the teacher model and the student model through semantic weighting; The formula is as follows: ； in, This represents the model parameters that were frozen and saved in the previous stage. This represents the model parameters obtained during the current stage of training on the new task. This represents the model obtained after updating the parameters, i.e., the final model parameters of the voice assistant; S3-6: Output the final model for continuous entity recognition tasks using voice commands.

Images