A training method for apnea knowledge question answering model based on knowledge distillation
Through the training method of apnea knowledge question-and-answer model based on knowledge distillation, the reconstruction loss and classification loss of the teacher model are used to reduce the computational complexity of the natural language processing model, and the problem of difficulty in applying the model in real scenarios in the existing technology is solved, and the problem of improving the problem of The response speed and quality of medical services of the Q&A system.
Patent Information
- Application Number
- CN202411414096.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-11
- Publication Date
- 2024-12-20
- Estimated Expiration
- 2044-10-11
AI Technical Summary
Due to the high computational complexity of existing natural language processing models, it is difficult to apply in real scenarios, especially in the popularization of knowledge related to apnea and rapid diagnosis.
The apnea knowledge question-and-answer model training method based on knowledge distillation is adopted to supervise the feature learning of the student model through the reconstruction loss of the teacher model, and combine the classification loss to improve the generalization ability of the student model, reducing the computational complexity and deployment cost of the model.
While maintaining high accuracy, the response speed of the Q&A system is significantly improved, the rational allocation of medical resources and the improvement of medical service quality is promoted, and the practical application value of the model is realized.
Smart Images
Figure CN118916701B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of computer systems based on specific computing models, and in particular to a method for training an apnea knowledge question-answering model based on knowledge distillation. Background Art
[0002] Recently, by training Transformer models (often containing hundreds of billions of parameters) on large-scale corpora, they have shown strong capabilities in solving various natural language processing tasks. However, hundreds of billions of parameters greatly increase the computational complexity of the model and put forward more stringent requirements on the hardware configuration on which the model is deployed, making it difficult to apply it in real scenarios.
[0003] Therefore, under the premise of ensuring model performance, in order to effectively reduce the computational complexity of the model and solve various natural language processing problems in different scenarios, the field of natural language processing method technology urgently needs an apnea knowledge question-answering model based on knowledge distillation. Summary of the invention
[0004] The present invention is to solve the problem of model calculation complexity and provide a method for training an apnea knowledge question-answering model based on knowledge distillation. The teacher model supervises the feature learning of the student model through reconstruction loss, so that it has the feature expression ability of the teacher model; the student model combines classification loss to improve the generalization ability of the student model in downstream tasks. Compared with previous language model training methods, the present invention improves the response speed of the question-answering system while maintaining a high accuracy rate. This has significant beneficial effects on popularizing apnea-related knowledge, assisting rapid diagnosis, and providing personalized treatment recommendations, thereby promoting the rational allocation of medical resources and improving the quality of medical services.
[0005] The present invention provides a method for training an apnea knowledge question-answering model based on knowledge distillation, comprising the following steps:
[0006] S1. Establish a sleep apnea knowledge dataset based on the AI language large model question and answer results and divide it into training set, validation set and test set;
[0007] S2. Establish a masked language module based on knowledge distillation and a masked language training module based on self-supervision. The masked language module based on knowledge distillation guides the feature reconstruction of the student model through the masked language model and the reconstruction loss so that the student model has the same apnea knowledge feature expression ability as the teacher model. The teacher model includes an encoder ViT-H and a word embedding module. The number of encoder parameters of the student model is less than the number of encoder ViT-H parameters. The student model also includes a word embedding module.
[0008] The language training module based on self-supervised masking is an asymmetric encoder-decoder network structure. The encoder is the encoder ViT-T of the student model, and the decoder is a multi-layer perceptron. By combining the masked language module based on knowledge distillation and classification loss, the encoder ViT-T's implicit apnea knowledge expression ability and generalization ability for downstream tasks are improved.
[0009] S3. Establish reconstruction loss , classification loss and total loss ;
[0010] S4. Use the training set, validation set and test set to train, validate and test the student model respectively. After reaching the model convergence threshold, the apnea knowledge question-answering model based on knowledge distillation is obtained. The training method of the apnea knowledge question-answering model based on knowledge distillation is completed.
[0011] The apnea knowledge question-answering model training method based on knowledge distillation described in the present invention, as a preferred embodiment, step S1 comprises:
[0012] S11. Extract apnea knowledge-related text data from medical literature, professional databases, and online health forums;
[0013] S12. Create a template based on text data related to sleep apnea knowledge and use the AI language big model to conduct question and answer between patients and smart assistants. Record the question and answer results of the AI language big model and divide them into training set, validation set and test set.
[0014] The apnea knowledge question-answering model training method based on knowledge distillation described in the present invention is, as a preferred embodiment, in step S2, the encoder ViT-H is a pre-trained encoder, and the training corpus of the encoder ViT-H is larger than the training corpus of the student model encoder.
[0015] The present invention discloses a method for training an apnea knowledge question-answering model based on knowledge distillation. As a preferred embodiment, the basic modules of the encoder ViT-H and the encoder ViT-T include a first-layer standardization module, a multi-head attention module, a second-layer standardization module and a multi-layer perceptron connected in sequence. The input port of the basic module is simultaneously input to the first-layer standardization module and the second-layer standardization module, and the output of the multi-head attention module and the output of the multi-layer perceptron are simultaneously output to the outside.
[0016] The sleep apnea knowledge question-answering model training method based on knowledge distillation described in the present invention is, as a preferred embodiment, the number of basic modules of the encoder ViT-H is 32, and the number of basic modules of the encoder ViT-T is 6.
[0017] The apnea knowledge question-answering model training method based on knowledge distillation described in the present invention is, as a preferred embodiment, in step S2, the processing method of the mask language module based on knowledge distillation is:
[0018] The input apnea knowledge sentence is segmented to obtain a text token sequence And output to the teacher model, the teacher model performs token sequence based on the word embedding module Vectorization, output And extract the target features of the teacher model , the teacher model is in a frozen state;
[0019] ;
[0020] Sequence text tokens After random masking, the output is sent to the student model, and the student model performs token sequence based on the word embedding module. Vectorization, output And extract the student model reconstruction features ;
[0021] ;
[0022] According to the teacher model target characteristics and student model reconstruction features Get the reconstruction loss , reconstruct the features of the student model Output to the decoder.
[0023] The apnea knowledge question-answering model training method based on knowledge distillation of the present invention is preferably used in step S2, based on the self-supervised mask language training module, including: processing the apnea knowledge sentence into a text token sequence through word segmentation, generating mask area label information based on the word embedding module and random masking ,in is the index of the mask token, is the length of word embedding.
[0024] The present invention discloses a method for training a apnea knowledge question-answering model based on knowledge distillation. As a preferred method, in step S3, the reconstruction loss for:
[0025] ;
[0026] Where N is the vector dimension, for norm.
[0027] The apnea knowledge question-answering model based on knowledge distillation described in the present invention is preferably configured such that in step S3, the classification loss for:
[0028] ;
[0029] in, For the The first mask token The prediction of the class position, The number of mask tokens.
[0030] The present invention discloses a method for training a apnea knowledge question-answering model based on knowledge distillation. As a preferred method, in step S3, the total loss for:
[0031] .
[0032] The present invention has the following advantages:
[0033] (1) Based on the excellent feature extraction ability of Vision Transformer-Huge (ViT-H) pre-trained on a large-scale corpus, combined with reconstruction loss, the knowledge distillation structure is used to guide the feature reconstruction of models with fewer parameters such as Vision Transformer-Tiny (ViT-T) in downstream tasks at the feature level, and explore the knowledge transfer ability of the student model to the teacher model; in addition, the student model is combined with the masked language model and classification loss to further improve the encoder's implicit knowledge expression ability and the decoder's generalization ability for downstream tasks;
[0034] (2) The self-supervised masked language training module enables the student model to complete and correct text sentences based on the corpus information provided by downstream tasks and the context information provided by it;
[0035] (3) The simple network structure of the student model significantly reduces its computational complexity and the deployment cost of the model, making it possible to apply it to real scenarios and having certain practical value;
[0036] (4) The present invention adopts a masked language training module based on knowledge distillation and a self-supervised masked language training module, combined with reconstruction loss and classification loss, which not only enables the student model to have rich knowledge of the teacher model, but also improves the generalization ability of the student model for downstream tasks; in addition, the simple network structure of the student model significantly reduces its computational complexity, reduces the deployment cost of the model, and makes it possible to apply it to real scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 This is a flow chart of a method for training an apnea knowledge question-answering model based on knowledge distillation;
[0038] Figure 2 This is a network structure diagram of a model for a method of training a apnea knowledge question-answering model based on knowledge distillation;
[0039] Figure 3 The network structure diagram of VisionTransformer, a knowledge distillation-based apnea knowledge question-answering model training method;
[0040] Figure 4 This is a network structure diagram of the encoder basic module of a knowledge distillation-based apnea knowledge question-answering model training method. DETAILED DESCRIPTION
[0041] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Example 1
[0042] like Figures 1 to 4 As shown, a method for training a apnea knowledge question-answering model based on knowledge distillation is implemented in the following steps:
[0043] Step 1: Establish an apnea knowledge dataset and divide it into a training set, a validation set, and a test set;
[0044] 1) Extract apnea knowledge-related text data from medical literature, professional databases, and online health forums.
[0045] 2) Using the powerful language comprehension and text generation capabilities of the ChatGLM4 model, based on the text data related to apnea knowledge, the prompt is "Please use the following template based on the knowledge in this article to create a question and answer session between the patient and the smart assistant. Note that the user and assistant must speak alternately."
[0046] { "messages": [
[0047] { "role": "user", "content": "What is the specific mechanism by which snoring causes pneumonia?"},
[0048] { "role": "assistant", "content": "When you snore, the vibration of the soft tissue in your throat will generate airflow, blowing the liquid in your throat into tiny droplets. These droplets may carry pathogens such as bacteria and viruses and enter the lungs with breathing. If the lungs' resistance decreases, these pathogens will cause infection and lead to pneumonia."},
[0049] { "role": "user", "content": "Will these droplets enter the lungs directly?"},
[0050] { "role": "assistant", "content": "Not necessarily. Most droplets will be deposited in the airways, but some of them will enter the lungs, especially those with a diameter of less than 5 microns, which can enter the alveoli and cause infection."}
[0051] ]}"""
[0052] 3) Record the questions and answers of the ChatGLM4 model and divide them into training set, validation set and test set, and save them as json files.
[0053] Step 2: Build the model;
[0054] 1) Masked language training module based on knowledge distillation, the network structure is as follows Figure 2 As shown in the figure, the teacher model includes the encoder ViT-H and Word Embeding pre-trained on a large-scale corpus. By freezing the teacher model and based on a small-scale corpus in the downstream task, the feature reconstruction of the student model is guided by the masked language model and reconstruction loss. The student model includes the encoder ViT-T and Word Embeding, so that the small model has the same feature expression ability as the large model. The network structure of Vision Transformer (ViT) is shown in the figure Figure 2 As shown, L=32 and L=6 correspond to the network structures of ViT-H and ViT-T respectively;
[0055] 2) Based on the self-supervised masked language training module, the network structure is as follows Figure 2 As shown in the figure, this module adopts an asymmetric encoder-decoder network structure, the encoder is the ViT-T of the student model, and the decoder is a multi-layer perceptron (MLP). Combined with the masked language model and classification loss, the encoder's implicit knowledge expression ability and the decoder's generalization ability for downstream tasks are further improved.
[0056] Step 3: Establish a loss function;
[0057] The loss function is established as follows:
[0058] 1) Reconstruction loss: Based on the excellent performance of the teacher model, we explore the knowledge transfer ability of the student model to the teacher model in downstream tasks, as follows:
[0059] (1) The input sentence is processed into a text token sequence after word segmentation, represented as , Indicates the length of the text token sequence;
[0060] (2) The teacher model is based on its Word Embedding to achieve The vectorization of , ViT-H is expressed as , the corresponding target feature is expressed as:
[0061] (1)
[0062] (3) The mask ratio used by the student model is 0.75, achieving The random mask of , and based on its WordEmbeding, realize its vectorization, expressed as , ViT-T is expressed as , the corresponding reconstruction feature is expressed as:
[0063] (2)
[0064] (4) The reconstruction loss is expressed as:
[0065] (3)
[0066] In the above formula, express norm;
[0067] 2) Classification loss, which aims to improve the generalization ability of the student model in downstream tasks. Specifically, the input sentence is processed into a text token sequence after word segmentation, and the label information of the mask area is generated based on its word embedding and random mask. ,in Indicates the index of the mask token. represents the length of Word Embedding, and the classification loss is expressed as:
[0068] (4)
[0069] In the above formula, Expressing the The first mask token The prediction of the class position, Indicates the number of mask tokens;
[0070] 3) Total loss, which aims to keep the student model focused on the knowledge transfer of the teacher model through reconstruction loss, and improve the generalization ability of the student model for downstream tasks through classification loss. The total loss can be expressed as:
[0071] (5)
[0072] Step 4: Use the training set, validation set, and test set to train, validate, and test the model, respectively, to obtain an apnea knowledge question-and-answer model.
[0073] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.
Claims
1. A method for training a sleep apnea knowledge question-answering model based on knowledge distillation, characterized in that: The following steps are involved: S1. Establish a sleep apnea knowledge dataset based on the AI language large model question and answer results and divide it into training set, validation set and test set; S2. Establishing a masked language module based on knowledge distillation and a masked language training module based on self-supervision, wherein the masked language module based on knowledge distillation guides the feature reconstruction of the student model through the masked language model and the reconstruction loss so that the student model has the apnea knowledge feature expression capability equivalent to that of the teacher model, wherein the teacher model includes an encoder ViT-H and a word embedding module, wherein the number of encoder parameters of the student model is less than the number of parameters of the encoder ViT-H, and the student model also includes a word embedding module; The masked language training module based on self-supervision is an asymmetric encoder-decoder network structure, the encoder is the encoder ViT-T of the student model, and the decoder is a multi-layer perceptron. By combining the masked language module based on knowledge distillation and classification loss, the encoder ViT-T's implicit apnea knowledge expression ability and generalization ability for downstream tasks are improved; The processing method of the mask language module based on knowledge distillation is: The input apnea knowledge sentence is segmented to obtain a text token sequence And output to the teacher model, the teacher model performs token sequence based on the word embedding module Vectorization, output And extract the target features of the teacher model , the teacher model is in a frozen state; ； Sequence the text token After random masking, the output is sent to the student model, and the student model performs token sequence based on the word embedding module. Vectorization, output And extract the student model reconstruction features ; ； According to the teacher model target characteristics and the student model reconstructs the features Get the reconstruction loss , the student model reconstructs the features Output to the decoder; The self-supervised masked language training module processes the apnea knowledge sentence into a text token sequence through word segmentation, and generates masked region label information based on the word embedding module and random masking. ,in is the index of the mask token, is the length of word embedding; S3. Establish reconstruction loss , classification loss and total loss ; S4. The student model is trained, verified and tested respectively using the training set, the validation set and the test set. After reaching the model convergence threshold, an apnea knowledge question-answering model based on knowledge distillation is obtained. Thus, a method for training an apnea knowledge question-answering model based on knowledge distillation is completed.
2. The method for training a sleep apnea knowledge question-answering model based on knowledge distillation according to claim 1, characterized in that: Step S1 includes: S11. Extract apnea knowledge-related text data from medical literature, professional databases, and online health forums; S12. Create a template based on the apnea knowledge-related text data and use the AI language big model to conduct question and answer between the patient and the intelligent assistant, record the question and answer results of the AI language big model and divide them into the training set, the verification set and the test set.
3. The method for training a sleep apnea knowledge question-answering model based on knowledge distillation according to claim 1, characterized in that: In step S2, the encoder ViT-H is a pre-trained encoder, and the training corpus of the encoder ViT-H is larger than the training corpus of the student model encoder.
4. The method for training a sleep apnea knowledge question-answering model based on knowledge distillation according to claim 3, characterized in that: The basic modules of the encoder ViT-H and the encoder ViT-T both include a first-layer standardization module, a multi-head attention module, a second-layer standardization module and a multi-layer perceptron connected in sequence. The input port of the basic module is simultaneously input to the first-layer standardization module and the second-layer standardization module, and the output of the multi-head attention module and the output of the multi-layer perceptron are simultaneously output to the outside.
5. The method for training a sleep apnea knowledge question-answering model based on knowledge distillation according to claim 4, characterized in that: The number of basic modules of the encoder ViT-H is 32, and the number of basic modules of the encoder ViT-T is 6.
6. The method for training a sleep apnea knowledge question-answering model based on knowledge distillation according to claim 1, characterized in that: In step S3, the reconstruction loss for: ； Where N is the vector dimension, for norm.
7. The method for training a sleep apnea knowledge question-answering model based on knowledge distillation according to claim 1, characterized in that: In step S3, the classification loss for: ； in, For the The first mask token The prediction of the class position, The number of mask tokens.
8. The method for training a sleep apnea knowledge question-answering model based on knowledge distillation according to claim 1, characterized in that: In step S3, the total loss for: 。