Anti-forgetting multi-field Chinese spelling error correction knowledge migration method

Abstract

An anti-forgetting multi-domain Chinese spelling error correction knowledge migration method comprises the following steps: initializing a teacher model for storing and integrating cross-domain Chinese spelling error correction knowledge; for each new domain, training a student model, and learning and absorbing knowledge of the new domain and the old domain from the teacher model through knowledge distillation; then, dynamically adjusting the distillation weight, balancing the learning process according to the proportion of new and old data, and slowing down forgetting; and continuously updating the student model, and copying the current state of the student model to update the teacher model so as to contain new domain knowledge, thereby realizing continuous learning. The dynamic teacher model and the distillation weight adjustment mechanism of the method ensure the balanced learning of new and old knowledge, and significantly improve the performance of multi-field spelling error correction. Experimental results show that effective migration and long-term retention of knowledge are realized in multi-field tasks, and the method has the advantages of plug and play, high flexibility and surpassing the prior art, and is of great significance in improving the practicability of Chinese spelling error correction and promoting the development of a natural language processing technology.

Description

Technical Field

[0001] The present invention relates to artificial intelligence natural language processing technology, and in particular to a multi-domain Chinese spelling error correction knowledge transfer method for resisting forgetting. Background Art

[0002] The task of Chinese typos detection and correction refers to automatically identifying and correcting spelling errors in Chinese sentences. This task plays an important role in application fields such as search engines, intelligent writing assistants, speech recognition, and optical character recognition, and has therefore attracted widespread attention from researchers. The current mainstream technology uses the Transformer-based Bidirectional Encoder Representation (BERT) method, which first performs unsupervised training on a large-scale text corpus, and then performs supervised training on a dedicated Chinese spelling correction dataset to achieve a high correction effect. Recent studies have also introduced confusion sets to enable the model to learn the graphic features and phonetic information of Chinese characters during the pre-training stage, further improving the detection and correction performance of Chinese typos.

[0003] Based on the BERT pre-trained language model, the semantic information of each Chinese character in the input sentence is obtained, the Chinese character strokes and pinyin sequences are encoded through a neural network, the voice information and graphic information of the sentence are extracted, and the adaptive gating mechanism is used to fuse these three types of Chinese character information, and finally the corrected sentence is output, thereby improving the model's language comprehension ability and spelling correction ability, achieving results that exceed previous advanced methods.

[0004] However, the existing technologies have the following areas to be improved: the current mainstream methods address this challenge by sharing cross-domain knowledge and improving generalization capabilities, but due to the complexity of domain-specific knowledge, these methods still have difficulty maintaining performance in multiple domains, especially in continuous learning scenarios, where there is a phenomenon of catastrophic forgetting. Existing research has not attempted to allow the model to have specific knowledge in multiple domains at the same time, thereby solving the problem of catastrophic forgetting in continuous learning.

[0005] It should be noted that the information disclosed in the above background technology section is only used for understanding the background of the present application, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the invention

[0006] The main purpose of the present invention is to overcome the defects existing in the above-mentioned background technology and provide a multi-domain Chinese spelling error correction knowledge transfer method for anti-forgetting.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] A multi-domain Chinese spelling error correction knowledge transfer method for anti-forgetting includes the following steps:

[0009] Initialization and dynamic evolution of the teacher model: Initialize a teacher model that has the ability to process Chinese texts from multiple fields and is used to store cross-domain knowledge. With the introduction of new domain data, the teacher model is continuously updated to integrate and retain key Chinese spelling correction knowledge from previous fields.

[0010] Training and knowledge distillation of student models: For each new domain, a student model is initialized and trained to learn the specific Chinese spelling correction knowledge of the current domain. During the training process, the student model learns from the teacher model through the knowledge distillation loss function, absorbing and integrating the Chinese spelling correction knowledge of the new and old domains.

[0011] Dynamically adjust learning weights: According to the ratio of new domain data to accumulated old data, dynamically adjust the distillation weights to balance the learning of new and old knowledge and slow down catastrophic forgetting;

[0012] Continuous learning and model updating: After training in each field is completed, the student model continues to be updated, and the current state of the student model is copied to update the teacher model to include knowledge in the new field, realizing the dynamic evolution of the teacher model and providing support for continuous learning.

[0013] Furthermore, in the scenario of multi-domain continuous learning, a sequence consisting of multiple domains is processed, and each domain is trained in turn; the model receives Chinese sentences containing typos as input and provides correct sentences as output; when training each domain, the model's access to previous domain data is restricted, or only limited previous domain data is allowed to be accessed to ensure that the model does not forget the knowledge of the old domain when learning the new domain.

[0014] Furthermore, a multi-stage knowledge transfer framework is adopted, and each stage is used to learn data from a specific field. Each stage learns a sample set in the field, and each sample includes a sentence with a spelling error and its corresponding correct sentence.

[0015] The goal of the model at each stage is to learn a function that can map an input sentence containing typos to a correct sentence while retaining knowledge from previous stages to prevent catastrophic forgetting;

[0016] The loss function at each stage consists of two parts: one is the knowledge distillation loss, which is used to measure the difference between the predictions of the teacher model and the student model; the other is the cross entropy loss, which is used to measure the difference between the output of the student model and the true label;

[0017] By dynamically adjusting the weight between knowledge distillation loss and cross entropy loss, the learning of new and old knowledge is balanced, thereby slowing down the forgetting of old domain knowledge in new domain learning;

[0018] After each stage, the student model continues to update to learn new domain knowledge, and at the same time copies the current state of the student model to update the teacher model to include knowledge in the new domain, thereby achieving dynamic evolution of the teacher model and cross-domain knowledge transfer.

[0019] Furthermore, the weight between knowledge distillation loss and cross entropy loss is dynamically determined based on the ratio of current domain data to accumulated old data. When the amount of current domain data is large, the change in the weight increases the impact of knowledge distillation loss accordingly, so that the model pays more attention when learning new domain knowledge; when the amount of old data is large, the change in the weight increases the impact of cross entropy loss accordingly, so as to maintain the importance of old domain knowledge, thereby effectively slowing down forgetting.

[0020] Furthermore, a dynamic teacher model strategy is adopted to ensure that the model can retain knowledge from early stages and adapt to new domains during the continuous learning process. In each training stage, the teacher model serves as a knowledge base, retains and integrates knowledge from all previous domains, acts as a core knowledge source, and guides the student model to learn and absorb cross-domain Chinese spelling correction knowledge by calculating the distillation loss with the student model. After the training stage is completed, the state of the student model is copied to update the teacher model, and the student model continues to be updated to prepare for learning the next new domain, thereby realizing the dynamic evolution of the teacher model.

[0021] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the multi-domain Chinese spelling error correction knowledge transfer method.

[0022] A computer program product includes a computer program, and when the computer program is executed by a processor, the multi-domain Chinese spelling error correction knowledge transfer method is implemented.

[0023] The present invention has the following beneficial effects:

[0024] The present invention proposes a method for anti-forgetting multi-domain Chinese spelling correction knowledge transfer to solve the model performance bottleneck caused by knowledge differences between domains in multi-domain Chinese spelling correction tasks. The method designs an anti-forgetting multi-domain Chinese spelling correction knowledge transfer framework, and significantly improves the knowledge retention and transfer capabilities of the model in a multi-domain environment through a dynamically evolving teacher model and dynamically adjusted distillation weights. The learned knowledge is transferred to the student model through a dynamically evolving teacher model, and a performance balance is achieved between new and old knowledge by introducing dynamically changing distillation weights. This method not only effectively solves the problem of catastrophic forgetting in multi-domain tasks, realizes efficient learning of new domain knowledge and long-term retention of old domain knowledge, but also can directly adapt to a variety of spelling correction model structures through model-independent plug-and-play characteristics, and can improve performance without additional large-scale retraining. In addition, the method of the present invention only retains the student model that has undergone multi-stage learning in the reasoning stage, avoiding the additional overhead of the teacher model and ensuring reasoning efficiency.

[0025] Experiments have shown that the present invention has shown significant performance improvement in multi-domain Chinese spelling correction tasks, surpassing the current advanced multi-domain spelling correction methods, as shown in the following:

[0026] 1. Through the dynamically evolving teacher model and the dynamically adjusted distillation weight mechanism, the present invention effectively solves the catastrophic forgetting problem in multi-domain tasks, and achieves efficient learning of new domain knowledge and long-term retention of old domain knowledge. Compared with existing methods, the performance of the model in cross-domain tasks is greatly improved.

[0027] 2. The framework is both model-independent and plug-and-play, and can be directly adapted to existing spelling correction models, improving their correction capabilities without the need for additional large-scale retraining. Experiments have shown that this feature significantly reduces deployment and application costs, while also being extremely practical and flexible.

[0028] 3. The dynamic distillation and teacher model strategy of the present invention surpasses the current advanced multi-domain spelling correction methods in performance, and provides new ideas for knowledge transfer and multi-domain learning in the field of natural language processing.

[0029] The present invention is of great significance for improving the robustness and practicality of Chinese spelling error correction tasks and promoting the application of natural language processing technology in multiple fields.

[0030] Other beneficial effects of the embodiments of the present invention will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a flow chart of a method for transferring knowledge of Chinese spelling correction in multiple fields for resisting forgetting according to an embodiment of the present invention.

[0032] Figure 2 Schematic diagram of a multi-domain Chinese spelling error correction knowledge transfer framework for anti-forgetting according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope and application of the present invention.

[0034] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0035] This paper proposes a model-independent dynamic knowledge transfer method. For the multi-domain Chinese spelling correction task, a multi-domain Chinese spelling correction knowledge transfer framework with anti-forgetting is designed. Combined with the dynamically evolving teacher model and dynamically adjusted distillation weights, the multi-domain knowledge retention and transfer capabilities of the model are significantly improved. Figure 1 and Figure 2 , the framework consists of two dynamic mechanisms: a dynamic evolution mechanism of the teacher model and a dynamic adjustment mechanism of the distillation weights. In the training stage, firstly, the knowledge of the previous field is accumulated stage by stage through the dynamically evolving teacher model, and it is passed to the student model as guidance information; secondly, the learning priority of new and old knowledge is balanced according to the proportion of domain data through the dynamically adjusted distillation weights, which effectively alleviates the catastrophic forgetting problem in multi-domain training. The entire framework relies on the idea of ​​knowledge distillation. The teacher model and the student model complement each other during the training process and continuously optimize the model performance. In the reasoning stage, the present invention only retains the student model that has undergone multi-stage learning, avoiding the additional overhead of the teacher model and ensuring the reasoning efficiency. The knowledge transfer framework of the present invention has the characteristics of plug-and-play, can be adapted to a variety of model architectures, and significantly improves the performance of spelling correction tasks in multi-domain scenarios while ensuring training efficiency.

[0036] See also Figure 1 and Figure 2 The embodiment of the present invention provides a multi-domain Chinese spelling error correction knowledge transfer method for anti-forgetting, comprising the following steps:

[0037] S1. Initialization and dynamic evolution of the teacher model: Initialize a teacher model that has the ability to process Chinese texts in multiple fields and is used to store cross-domain knowledge. With the introduction of new domain data, the teacher model is continuously updated to integrate and retain key Chinese spelling correction knowledge from previous fields.

[0038] S2. Training and knowledge distillation of student models: For each new domain, a student model is initialized and trained to learn the specific Chinese spelling correction knowledge of the current domain. During the training process, the student model learns from the teacher model through the knowledge distillation loss function, and absorbs and integrates the Chinese spelling correction knowledge of the new and old domains.

[0039] S3. Dynamically adjust learning weights: According to the ratio of new domain data to accumulated old data, dynamically adjust the distillation weights to balance the learning of new and old knowledge and slow down catastrophic forgetting.

[0040] S4. Continuous learning and model updating: After the training in each field is completed, the parameters of the student model continue to be updated, and the current state of the student model is copied to update the teacher model to include knowledge in the new field, realizing the dynamic evolution of the teacher model and providing support for continuous learning.

[0041] In this method, the student model is continuously updated, and the teacher model is updated by copying the current state of the student model after training in each new field, so that the teacher model of the previous field becomes the teacher model of the current field, so as to realize the transfer of knowledge and the dynamic evolution of the model. This method realizes cross-domain knowledge transfer and the mitigation of catastrophic forgetting through the interaction between the teacher model and the student model, ensuring the stability and adaptability of the model in a multi-domain environment.

[0042] For the multi-domain Chinese spelling correction task, the method of the present invention designs a dynamic knowledge transfer framework, which revolves around two key technical points. First, it introduces a dynamically evolving teacher model and dynamically adjusted distillation weights, and gradually accumulates and transfers knowledge in historical fields in stages, effectively retains the core knowledge of previous fields at each stage, and balances the comprehensive learning ability of new and old knowledge. This mechanism not only optimizes the efficiency of knowledge transfer in multi-domain spelling correction, but also significantly alleviates the problem of catastrophic forgetting. Second, the model-independent plug-and-play framework can directly adapt to a variety of spelling correction model structures. Through the interactive optimization of the dynamic teacher model and the student model, the present invention modularizes the complex cross-domain knowledge transfer process to form a plug-and-play solution, which can achieve performance improvement on the existing model without additional large-scale retraining.

[0043] In some embodiments, in a scenario of multi-domain continuous learning, the model processes a sequence consisting of multiple domains, and each domain is trained in turn; the model receives Chinese sentences containing typos as input, and provides correct sentences as output; when training each domain, the model's access to previous domain data is restricted, or only limited previous domain data is allowed to be accessed to ensure that the model does not forget the knowledge of the old domain when learning the new domain.

[0044] In a preferred embodiment, a multi-stage knowledge transfer framework is adopted, and each stage is used to learn data from a specific domain, wherein each stage learns a sample set of the domain, and each sample includes a sentence containing a spelling error and its corresponding correct sentence; the goal of the model at each stage is to learn a function that can map an input sentence containing a typo to a correct sentence while retaining the knowledge from the previous stage to prevent catastrophic forgetting; the loss function of each stage includes two parts: one part is the knowledge distillation loss, which is used to measure the difference between the predictions of the teacher model and the student model; the other part is the cross entropy loss, which is used to measure the difference between the output of the student model and the true label; by dynamically adjusting the weight between the knowledge distillation loss and the cross entropy loss, the learning of new and old knowledge is balanced, thereby slowing down the forgetting of old domain knowledge in new domain learning; after each stage, the student model continues to be updated to learn the new domain knowledge, and at the same time, the current state of the student model is copied to update the teacher model to include the new domain knowledge, thereby realizing the dynamic evolution of the teacher model and cross-domain knowledge transfer.

[0045] In a more preferred embodiment, the weight between knowledge distillation loss and cross entropy loss is dynamically determined based on the ratio of current domain data to accumulated old data. When the amount of current domain data is large, the change in the weight increases the impact of knowledge distillation loss accordingly, so that the model pays more attention when learning new domain knowledge; when the amount of old data is large, the change in the weight increases the impact of cross entropy loss accordingly, so as to maintain the importance of old domain knowledge, thereby effectively slowing down forgetting.

[0046] In a preferred embodiment, a dynamic teacher model strategy is adopted to ensure that the model can retain knowledge from early stages and adapt to new fields during the continuous learning process. In each training stage, the teacher model serves as a knowledge base, retains and integrates knowledge from all previous fields, acts as a core knowledge source, and guides the student model to learn and absorb cross-domain Chinese spelling correction knowledge by calculating the distillation loss with the student model. After the training stage is completed, the state of the student model is copied to update the teacher model, and the student model continues to be updated to prepare for learning the next new field, thereby realizing the dynamic evolution of the teacher model.

[0047] The specific embodiments of the present invention and algorithm examples thereof are further described below.

[0048] A multi-domain Chinese spelling correction knowledge transfer method is implemented based on the anti-forgetting multi-domain Chinese spelling correction knowledge transfer framework, which combines two dynamic mechanisms and is designed for scenarios involving continuous training across multiple domains. The main mechanism includes a dynamically evolving teacher model that continuously updates its knowledge base to cover the most critical knowledge in the previous training domain. The second mechanism involves dynamic distillation weights that can better balance the losses between the teacher model and the student model. This design provides a simple and effective continuous learning solution for the multi-domain Chinese spelling correction task.

[0049] Task definition:

[0050] The Chinese text error correction task involves detecting and correcting spelling errors in Chinese sentences. The model receives a sentence X = {x1, x2, ..., x n}, where there are n characters as input, and the corresponding correct sentence Y = {y1, y2, ..., y n In the multi-domain continuous learning scenario, consider a sequence D = {D1, D2, ..., D n}, each field D k It is trained sequentially and does not access or can only access a small portion of the data of previous domains during training. The goal of the present invention is to ensure that after training a new domain, not only the performance of the current domain is evaluated, but also the performance of all previously trained domains.

[0051] MKT frame structure:

[0052] The multi-stage knowledge transfer framework operates on each training stage. At each stage k, the model learns data from the current domain Where T k represents the number of samples in the kth stage, each sample includes a misspelled sentence And the corresponding correct sentence The goal of the model is to learn a function g(·;ω) that transforms each input Mapping to the correct output And retain knowledge from previous stages to prevent catastrophic forgetting.

[0053] The loss function at each stage is defined as:

[0054] in: is the knowledge distillation loss, measuring the teacher model g(·;ω k-1 ) and the student model g(·;ω k ) predictions, is the cross entropy loss between the student model output and the true label.

[0055] Knowledge distillation loss:

[0056] Knowledge Distillation Loss The calculation method is:

[0057]

[0058] This term ensures that the student model learns from the teacher model, which incorporates all prior domain knowledge.

[0059] Cross Entropy Loss:

[0060] Cross Entropy Loss The calculation method is:

[0061]

[0062] This loss ensures that the student model can accurately predict the corrected sentence for the current domain.

[0063] Dynamic distillation weight hyperparameter λ:

[0064] The dynamic distillation weight hyperparameter λ controls the balance between distillation loss and cross entropy loss. It is determined by the current domain data S d and the accumulated old data S o The ratio is determined by: This dynamic weighting helps prevent catastrophic forgetting by adjusting the influence of the teacher model based on the ratio of new and old data.

[0065] Dynamic Teacher Model Strategy:

[0066] A key challenge in continual learning is to ensure that the model can retain knowledge from earlier stages while adapting to new domains. To this end, this problem is addressed through a dynamic teacher model strategy.

[0067] At each stage k, the teacher model g(·;ω k-1 ) retains knowledge from all previous domains. The teacher model acts as a knowledge base and is used to calculate the distillation loss of the current student model. This dynamic teacher model effectively stores and transfers the core knowledge of all domains so far, guiding the training process of the current stage. After the training of this stage is completed, the student model serves as the teacher model of the next stage, and the teacher model completes dynamic evolution.

[0068] Experiments have shown that the dynamic knowledge transfer framework proposed in this paper has shown significant performance improvement in multi-domain Chinese spelling correction tasks, specifically:

[0069] 1. The knowledge transfer effect is significant

[0070] Through the dynamically evolving teacher model and the dynamically adjusted distillation weight mechanism, the present invention effectively solves the catastrophic forgetting problem in multi-domain tasks, and achieves efficient learning of new domain knowledge and long-term retention of old domain knowledge. Compared with existing methods, the performance of the model in cross-domain tasks is greatly improved.

[0071] 2. Plug and play to improve flexibility

[0072] The framework is both model-independent and plug-and-play, and can be directly adapted to existing spelling correction models, improving their correction capabilities without the need for additional large-scale retraining. Experiments have shown that this feature significantly reduces deployment and application costs while being extremely practical and flexible.

[0073] 3. Innovative approaches that go beyond existing technologies

[0074] The dynamic distillation and teacher model strategy of the present invention surpasses the current advanced multi-domain spelling correction methods in performance, and provides new ideas for knowledge transfer and multi-domain learning in the field of natural language processing.

[0075] The present invention is of great significance for improving the robustness and practicality of Chinese spelling error correction tasks and promoting the application of natural language processing technology in multiple fields.

[0076] An embodiment of the present invention further provides a storage medium for storing a computer program, which at least performs the above method when executed.

[0077] An embodiment of the present invention further provides a control device, comprising a processor and a storage medium for storing a computer program; wherein the processor is configured to execute at least the method described above when executing the computer program.

[0078] An embodiment of the present invention further provides a processor, wherein the processor executes a computer program and at least executes the method described above.

[0079] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a ferromagnetic random access memory (Flash Memory), a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The storage medium described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0080] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0081] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0082] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0083] A person skilled in the art can understand that: all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, etc. Various media that can store program codes.

[0084] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0085] The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0086] The features disclosed in several product embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0087] The features disclosed in several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0088] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art of the present invention, several equivalent substitutions or obvious variations can be made without departing from the concept of the present invention, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. A multi-domain Chinese spelling error correction knowledge transfer method for anti-forgetting, characterized in that: The following steps are involved: Initialization and dynamic evolution of the teacher model: Initialize a teacher model that has the ability to process Chinese texts in multiple fields and is used to store cross-domain knowledge; As new domain data is introduced, the teacher model is continuously updated to integrate and retain key Chinese spelling correction knowledge from previous domains; Training and knowledge distillation of student models: For each new domain, a student model is initialized and trained to learn the specific Chinese spelling correction knowledge of the current domain. During the training process, the student model learns from the teacher model through the knowledge distillation loss function, absorbing and integrating the Chinese spelling correction knowledge of the new and old domains. Dynamically adjust learning Weight: Dynamically adjust the distillation weight according to the ratio of new domain data to accumulated old data to balance the learning of new and old knowledge and slow down catastrophic forgetting; Continuous learning and model updating: After training in each field is completed, the student model continues to be updated, and the current state of the student model is copied to update the teacher model to include knowledge in the new field, realizing the dynamic evolution of the teacher model and providing support for continuous learning.

2. The multi-domain Chinese spelling error correction knowledge transfer method as claimed in claim 1, characterized in that: In the scenario of multi-domain continuous learning, a sequence consisting of multiple domains is processed, and each domain is trained in turn; The model receives Chinese sentences with typos as input and provides correct sentences as output; When training each domain, limit the model’s access to data from previous domains, or only allow access to limited data from previous domains, to ensure that the model does not forget knowledge from old domains when learning new domains.

3. The multi-domain Chinese spelling error correction knowledge transfer method as claimed in claim 1 or 2, characterized in that: A multi-stage knowledge transfer framework is adopted. Each stage is used to learn data from a specific field. Each stage learns a sample set in the field. Each sample includes a sentence with a spelling error and its corresponding correct sentence. The goal of the model at each stage is to learn a function that can map an input sentence containing typos to a correct sentence while retaining knowledge from previous stages to prevent catastrophic forgetting; The loss function at each stage consists of two parts: one is the knowledge distillation loss, which is used to measure the difference between the predictions of the teacher model and the student model; the other is the cross entropy loss, which is used to measure the difference between the output of the student model and the true label; By dynamically adjusting the weight between knowledge distillation loss and cross entropy loss, the learning of new and old knowledge is balanced, thereby slowing down the forgetting of old domain knowledge in new domain learning; After each stage, the student model continues to update to learn new domain knowledge, and at the same time copies the current state of the student model to update the teacher model to include knowledge in the new domain, thereby achieving dynamic evolution of the teacher model and cross-domain knowledge transfer.

4. The multi-domain Chinese spelling error correction knowledge transfer method as claimed in claim 3, characterized in that: The weight between knowledge distillation loss and cross entropy loss is dynamically determined based on the ratio of current domain data to accumulated old data. When the amount of current domain data is large, the change in the weight increases the impact of knowledge distillation loss accordingly, so that the model pays more attention when learning new domain knowledge; when the amount of old data is large, the change in the weight increases the impact of cross entropy loss accordingly, so as to maintain the importance of old domain knowledge, thereby effectively slowing down forgetting.

5. The multi-domain Chinese spelling error correction knowledge transfer method according to any one of claims 1 to 4, characterized in that: A dynamic teacher model strategy is adopted to ensure that the model can retain knowledge from earlier stages and adapt to new domains during continuous learning. In each training stage, the teacher model, as a knowledge base, retains and integrates knowledge from all previous domains and acts as a core knowledge source. By calculating the distillation loss with the student model, the teacher model is guided to learn and absorb cross-domain Chinese spelling correction knowledge. After the training phase is completed, the state of the student model is copied to update the teacher model, and the student model continues to be updated in preparation for learning the next new domain, realizing the dynamic evolution of the teacher model.

6. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the multi-domain Chinese spelling error correction knowledge transfer method as described in any one of claims 1 to 5 is implemented.

7. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the multi-domain Chinese spelling error correction knowledge transfer method as described in any one of claims 1 to 5 is implemented.

Images