Mask feature progressive knowledge distillation completion method for user portrait of nasopharyngeal carcinoma patient

Abstract

The invention provides a mask feature progressive knowledge distillation completion method for a user portrait of a nasopharyngeal carcinoma patient, and relates to the technical field of image data processing, and the method comprises the following steps: a pre-distillation stage: carrying out the pre-distillation of a pre-training language model, so as to obtain a high-quality teacher model, generating feature distillation through a mask, and carrying out the feature distillation; applying a mask operation in the input data, and setting a mask proportion according to the number of student model parameters; and S2, mask generation feature distillation: the teacher model and the student model perform mask operation on input data including the triad and text description thereof in a data input stage. In the masking generation characteristic distillation, masking operation is applied to input marks, and a proper masking rate is set in a pre-distillation stage according to student model parameter counting, so that the performance of an initial model is improved. In the progressive distillation stage, a multi-grade student model is designed, and the parameter count and the mask rate are gradually reduced. Each student model is trained under the three kinds of supervision described above.

Description

Technical Field

[0001] The present invention relates to the technical field of image data processing, and specifically to a mask feature progressive knowledge distillation completion method for user portraits of nasopharyngeal carcinoma patients. Background Art

[0002] Since its concept was proposed, knowledge graph has become an important tool for enhancing search functions and promoting information management, and it has also shown great potential in the medical field. With the rapid accumulation of medical data, the knowledge graph in the field of nasopharyngeal carcinoma patients has also developed rapidly. Although the academic community has conducted some research on the epidemiology and treatment methods of nasopharyngeal carcinoma, the user portrait of nasopharyngeal carcinoma patients in the nasopharyngeal carcinoma knowledge graph still has problems with incomplete data and insufficient timeliness, which brings challenges to practical applications such as medical decision support and formulating personalized nutrition plans for patients. Therefore, both academia and industry are actively exploring knowledge graph completion technology in this area, in order to improve its practicality and accuracy in the nutritional management of nasopharyngeal carcinoma patients.

[0003] The construction of the knowledge graph of user portraits of NPC patients usually includes an ontology layer and an instance layer. The ontology layer provides a conceptual model and a logical basis, while the instance layer is a concrete expression of the ontology. In the field of NPC, the breadth and complexity of professional knowledge make it particularly difficult to build a comprehensive and structured ontology layer, which usually requires in-depth participation of domain experts and a lot of time investment. Therefore, many knowledge graphs choose to directly build the instance layer, but this leads to a lack of effective ontology layer constraints. At the same time, the existing technology also has the problem of data sparsity. In the completion of user portraits of NPC patients, some patients may have only a small amount of related information, which makes it difficult for the model to accurately learn the relationship between these patient information. Summary of the invention

[0004] 1. Technical issues to be resolved

[0005] In view of the shortcomings of the existing technology, the present invention provides a masked feature progressive knowledge distillation completion method for nasopharyngeal carcinoma patient user portraits, which solves the problem of the medical industry's demand for a nasopharyngeal carcinoma patient user portrait knowledge graph with accurate learning capabilities.

[0006] (II) Technical solution

[0007] To achieve the above objectives, the present invention is implemented through the following technical scheme: a mask feature progressive knowledge distillation completion method for user portraits of nasopharyngeal carcinoma patients, comprising the following steps:

[0008] S1, pre-distillation stage: pre-distill the pre-trained language model to obtain a high-quality teacher model, then use mask generation feature distillation, apply masking operations to the input data, and set the mask ratio according to the number of student model parameters;

[0009] S2. Masked feature distillation: The teacher model and the student model perform a mask operation on the input data including triples and their text descriptions at the data input stage, wherein the mask operation is a predetermined mask rate determined based on the teacher model parameter size;

[0010] S3, the teacher model encodes the mask mark to generate a mask feature vector containing rich semantic information, and the student model encodes the mask mark during training to generate the corresponding student feature vector;

[0011] S4, progressive distillation framework: By gradually reducing the mask ratio and model parameters, the knowledge of the teacher model is effectively transferred to the student model. The framework is divided into two stages: pre-distillation stage and progressive distillation stage;

[0012] S5. Multi-level student model design: In the progressive distillation stage, different levels of student models are designed, and the number of parameters of each level of student model gradually decreases;

[0013] S6. Use true labels for distillation to ensure that the model learns the key triple feature information in the dataset;

[0014] S7, using the potential prior knowledge of the teacher model, evaluates the global information of the triples through the scoring module, and uses the mask generation feature distillation module to generate inferred feature vectors through random mask operations, prompting the student model to learn additional information and enhance its expressiveness;

[0015] S8, the ability to capture global and local information of triples using α and β balance models;

[0016] S9. Experimental setup: Extensive experiments were conducted on a dataset of nasopharyngeal carcinoma patients to verify the effectiveness of the progressive distillation completion method based on mask features. The model in the progressive distillation stage significantly reduced the model parameters while maintaining a certain level of performance.

[0017] Preferably, redundant parameters are eliminated during the pre-distillation of the pre-trained language model to reduce the size and complexity of the pre-trained language model. The redundant parameters include excessive repeated input data, data sets related to nasopharyngeal carcinoma patients, and data sets with low collection relevance. If there are redundant parameters, weight penalties are imposed.

[0018] Preferably, the data set with low collection relevance is a data value obtained by comparing experimentally irrelevant data collection of a data set of nasopharyngeal carcinoma patients.

[0019] Preferably, the number of student models is determined by an initial setting in the progressive distillation stage, and the mask ratio is reduced once the initial number setting is exceeded.

[0020] Preferably, in S3, the corresponding student feature vectors generated have a uniform modulus, and the modulus is the average value of the mask feature vector containing rich semantic information.

[0021] Preferably, the loss value corresponding to the true label is realized by soft and hard label losses, specifically by Kullback-Leibler divergence (KL divergence) and cross entropy loss, respectively.

[0022] Preferably, in S8, the loss value corresponding to the true label is compared by Ltotal=αLsoft+βLhard, where Lsoft is the soft label loss and Lhard is the hard label loss.

[0023] (III) Beneficial effects

[0024] The present invention provides a mask feature progressive knowledge distillation completion method for user portraits of nasopharyngeal carcinoma patients. It has the following beneficial effects:

[0025] 1. The existing traditional Logits distillation usually involves complex and deep learning models, which require multiple iterations to generate the final result. In comparison, the method of the present invention focuses on reducing the sampling steps required for the generative model in the reasoning stage, thereby improving the reasoning efficiency, and can quickly generate and feedback information in real time. Through the progressive distillation strategy, the method based on mask features not only simplifies the model structure and reduces the complexity, but also significantly reduces the model parameters, which is significantly reduced compared to the baseline, achieving model lightweight. This simplification reduces resource consumption without sacrificing performance, allowing the model to run efficiently and adapt to medical environments with limited computing resources. In addition, the simplified model speeds up the processing speed of patient user portraits, improves the real-time update capability of the knowledge graph, while maintaining the generalization capability of new situations, providing fast and accurate support for personalized dietary recommendations for nasopharyngeal carcinoma patients.

[0026] 2. Traditional distillation technology requires training a teacher model and then transferring knowledge from the teacher model to the student model. This process usually takes a long time, especially on large-scale data sets, and both the teacher model and the student model need to store a large number of parameters, which increases the storage cost. Progressive distillation technology significantly reduces the computational cost of the inference phase by converting complex models into models that require only a small number of sampling steps. The computational overhead of the progressive distillation process does not exceed the overhead of training the original model. By iteratively distilling the teacher model into a student model, the sampling steps are halved each iteration until the required number of sampling steps is reached. This approach not only reduces the time required to generate high-quality samples, but also reduces the computational cost of running diffusion models in practical applications. In addition, since the student model generated by the progressive distillation technology has fewer parameters, the storage cost is also reduced accordingly.

[0027] 3. By utilizing masked feature technology, our model can accurately identify and focus on key symptom information of nasopharyngeal carcinoma. This targeted feature extraction significantly improves the accuracy of the model, and thus significantly improves the model's ability to predict whether an individual has nasopharyngeal carcinoma and accurately stage nasopharyngeal carcinoma; using a progressive learning method, the model can gradually and deeply master complex features. This refined learning process greatly promotes the generation of high-quality complementary information. In addition, by integrating medical data from different sources, the model can construct a more comprehensive and multi-dimensional representation of patient features, which not only enriches the level of patient user portraits, but also deepens the depth of user portraits.

[0028] 4. It can flexibly deal with medical data of various sources and qualities, and can properly handle even incomplete or noisy records. Using progressive distillation technology, the model continues to evolve during training and finely learns complex feature representations, thereby improving sensitivity and adaptability to individual differences among patients.

[0029] 5. In terms of design, the model is highly adjustable and scalable, and can quickly respond to new medical research or clinical needs. By integrating more dimensional features and treatment strategies, the depth and breadth of the model can be further enriched. In addition, the model can comprehensively analyze multi-source medical data such as sociodemographic information related to the dietary behavior of NPC patients, personal dietary preferences, dietary knowledge and health behavior levels, accompanying NIS and physical signs and disease characteristics, and construct a more comprehensive knowledge graph of NPC patient user portraits. DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the present invention are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] Nasopharyngeal carcinoma is a malignant tumor of the head and neck that originates from nasopharyngeal epithelial cells. Due to the toxic side effects of radiotherapy and chemotherapy and the obstruction of eating due to the location of the tumor, nasopharyngeal carcinoma patients experience significant nutritional symptoms during treatment and recovery. Studies have shown that reasonable dietary adjustments can effectively improve the nutritional status of these patients. However, nasopharyngeal carcinoma patients are often accompanied by multiple and changing nutritional symptoms, and complex dietary intake principles make it difficult for them to manage nutrition. Existing medical resources are insufficient to provide long-term personalized dietary customization for each patient, forcing some patients and their caregivers to obtain dietary nutrition knowledge through online searches, but this information is usually scattered, redundant or unscientific, causing them to fall into trouble.

[0032] Therefore, we hope to propose a personalized dietary recommendation system to help NPC patients solve this dilemma. One of the key links of the system is to complete the knowledge graph of NPC patient user portraits. Currently, these user portraits are incomplete, lack of timeliness, lack of personalized information, lack of dynamic and time information, and challenges in entity and relationship recognition.

[0033] The embodiment of the present invention provides a mask feature progressive knowledge distillation completion method for a user portrait of a nasopharyngeal carcinoma patient, comprising the following steps:

[0034] S1, pre-distillation stage: pre-distill the pre-trained language model to obtain a high-quality teacher model, then use mask generation feature distillation, apply mask operation to the input data, and set the mask ratio according to the number of student model parameters. In the process, compress the trained language model network to obtain a multi-level student model;

[0035] S2. Masked feature distillation: The teacher model and the student model perform a mask operation on the input data including triples and their text descriptions at the data input stage, wherein the mask operation is a predetermined mask rate determined based on the teacher model parameter size;

[0036] S3, the teacher model encodes the mask mark to generate a mask feature vector containing rich semantic information, and the student model encodes the mask mark during training to generate the corresponding student feature vector;

[0037] S4, Progressive Distillation Framework: By gradually reducing the mask ratio and model parameters, the knowledge of the teacher model is ensured to be effectively transferred to the student model. This framework is divided into two stages: the pre-distillation stage and the progressive distillation stage. By gradually reducing the mask ratio and model parameters, the knowledge of the teacher model is ensured to be effectively transferred to the student model.

[0038] S5. Multi-level student model design: In the progressive distillation stage, student models of different scales are designed. The number of parameters of the student model of each scale is gradually reduced. In the above, student models of different parameter scales are designed to adapt to different application scenarios and performance requirements.

[0039] S6. Use true labels for distillation to ensure that the model learns the key triple feature information in the dataset;

[0040] S7, using the potential prior knowledge of the teacher model, evaluates the global information of the triples through the scoring module, and uses the mask generation feature distillation module to generate inferred feature vectors through random mask operations, prompting the student model to learn additional information and enhance its expressiveness;

[0041] S8, the ability to capture the global and local information of the triplet using the α and β balance models, that is, using the α and β balance models as weights to evaluate the true label loss, respectively used to evaluate the weight of the soft label loss and the weight of the hard label loss;

[0042] S9. Experimental setup: Extensive experiments were conducted on a dataset of nasopharyngeal carcinoma patients to verify the effectiveness of the progressive distillation completion method based on mask features. The model in the progressive distillation stage significantly reduced the model parameters while maintaining a certain level of performance.

[0043] In the process of pre-distillation of the pre-trained language model, redundant parameters are eliminated to reduce the size and complexity of the pre-trained language model. Redundant parameters include too much repeated input data, data sets related to nasopharyngeal carcinoma patients, and data sets with low collection relevance. If there are redundant parameters, weight penalties are performed. The above can reduce the complexity of the pre-trained language model and the waste of computing resources.

[0044] The data set with low correlation is collected by comparing the data values ​​obtained by collecting and comparing experimentally irrelevant data from the data set of nasopharyngeal carcinoma patients.

[0045] The number of student models is determined by the initial setting in the progressive distillation stage, and the mask ratio is reduced once it exceeds the initial number setting.

[0046] In S3, the corresponding student feature vectors generated have a uniform modulus, which is the average value of the mask feature vector containing rich semantic information.

[0047] The loss value corresponding to the true label is realized by using Kullback-Leibler divergence (KL divergence) for soft label loss and cross entropy loss for hard label loss.

[0048] In S8, the loss value corresponding to the true label is compared by Ltotal=αLsoft+βLhard, where Lsoft is the soft label loss and Lhard is the hard label loss.

[0049] In the above, the real label distillation: use the real label to supervise the model to ensure that the model learns the key triple feature information in the dataset. Scoring module supervision: the global information of the triple is evaluated through the scoring module, so that the student model learns the implicit knowledge in the teacher model. Masked generation feature distillation supervision: through the masked generation feature distillation strategy module, the student model learns the representation information of the inferred entity set generated by the teacher model.

[0050] Pre-distillation and masked feature distillation strategy: The pre-distillation stage enhances the performance of the baseline model through a masked feature distillation strategy and serves as a teacher model to guide the advanced student model.

[0051] Progressive distillation framework: Efficient knowledge transfer from the teacher model to the student model is achieved by gradually reducing the masking ratio and model parameters.

[0052] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. The masked feature progressive knowledge distillation completion method for user portraits of nasopharyngeal carcinoma patients is characterized by: The following steps are involved: S1, pre-distillation stage: pre-distill the pre-trained language model to obtain a high-quality teacher model, then use mask generation feature distillation, apply masking operations to the input data, and set the mask ratio according to the number of student model parameters; S2. Masked feature distillation: The teacher model and the student model perform a mask operation on the input data including triples and their text descriptions at the data input stage, wherein the mask operation is a predetermined mask rate determined based on the teacher model parameter size; S3, the teacher model encodes the mask mark to generate a mask feature vector containing rich semantic information, and the student model encodes the mask mark during training to generate the corresponding student feature vector; S4, progressive distillation framework: By gradually reducing the mask ratio and model parameters, the knowledge of the teacher model is effectively transferred to the student model. The framework is divided into two stages: pre-distillation stage and progressive distillation stage; S5. Multi-level student model design: In the progressive distillation stage, different levels of student models are designed, and the number of parameters of each level of student model gradually decreases; S6. Use true labels for distillation to ensure that the model learns the key triple feature information in the dataset; S7, using the potential prior knowledge of the teacher model, evaluates the global information of the triples through the scoring module, and uses the mask generation feature distillation module to generate inferred feature vectors through random mask operations, prompting the student model to learn additional information and enhance its expressiveness; S8, the ability to capture global and local information of triples using α and β balance models; S9. Experimental setup: Extensive experiments were conducted on a dataset of nasopharyngeal carcinoma patients to verify the effectiveness of the progressive distillation completion method based on mask features. The model in the progressive distillation stage significantly reduced the model parameters while maintaining a certain level of performance.

2. The mask feature progressive knowledge distillation completion method for user portraits of nasopharyngeal carcinoma patients according to claim 1 is characterized by: In the process of pre-distilling the pre-trained language model, redundant parameters are eliminated to reduce the size and complexity of the pre-trained language model. The redundant parameters include excessive repeated input data, data sets related to nasopharyngeal carcinoma patients, and data sets with low collection relevance. If there are redundant parameters, weight penalties are performed.

3. The mask feature progressive knowledge distillation completion method for user portraits of nasopharyngeal carcinoma patients according to claim 2 is characterized by: The data set with low acquisition correlation is obtained by comparing the experimentally irrelevant data acquisition of the data set of nasopharyngeal carcinoma patients.

4. The mask feature progressive knowledge distillation completion method for user portraits of nasopharyngeal carcinoma patients according to claim 3 is characterized by: The number of student models is determined by the initial setting of the progressive distillation stage, and the mask ratio is reduced once the initial number setting is exceeded.

5. The mask feature progressive knowledge distillation completion method for user portraits of nasopharyngeal carcinoma patients according to claim 1 is characterized by: In S3, the corresponding student feature vectors generated have a uniform modulus, which is the average value of the mask feature vectors containing rich semantic information.

6. The mask feature progressive knowledge distillation completion method for user portraits of nasopharyngeal carcinoma patients according to claim 1 is characterized by: The loss value corresponding to the true label is realized through soft and hard label losses, respectively, through Kullback-Leibler divergence (KL divergence) and cross entropy loss.

7. The mask feature progressive knowledge distillation completion method for user portraits of nasopharyngeal carcinoma patients according to claim 6 is characterized by: In S8, the loss value corresponding to the true label is compared by Ltotal=αLsoft+βLhard, where Lsoft is the soft label loss and Lhard is the hard label loss.