A training method of a clinical auxiliary diagnosis model

Abstract

The present application relates to a kind of training methods of clinical auxiliary diagnosis model, belong to medical information processing technical field, solve the problem of poor coordination of size model in prior art affects the accuracy of diagnosis decision.Method includes: obtaining the medical text of multiple patients and constructs sample set;Text compression model and medical diagnosis large model are constructed, and the pre-training of text compression model and medical diagnosis large model constructed respectively is carried out based on sample set;The parameter of medical diagnosis large model is fixed, and the compressed text output by text compression model is input to medical large model, and text compression model is optimized based on the diagnosis task loss of medical diagnosis large model;The parameter of text compression model is fixed, and the compressed text output by text compression model is input to medical large model, and medical diagnosis large model is fine-tuned based on the diagnosis task loss of medical diagnosis large model, and the trained clinical auxiliary diagnosis model is obtained.The accuracy of diagnosis in complex clinical scenario is improved.

Description

Technical Field

[0001] This invention relates to the field of medical information processing technology, and in particular to a training method for a clinical auxiliary diagnostic model. Background Technology

[0002] Clinical decision support systems are a core component of modern smart healthcare. Their goal is to provide doctors with decision support such as diagnostic suggestions and treatment recommendations by analyzing patient medical information. With the development of artificial intelligence technologies such as reinforcement learning and deep learning, clinical decision support systems based on natural language processing have gradually evolved from traditional rule-based systems to data-driven deep learning models.

[0003] Current technical approaches mainly fall into two categories: **Specialized Small Model Approach:** This involves training streamlined models for specific departments or diseases (such as entity recognition models based on BiLSTM-CRF or classification models based on CNN). These models typically have millions to tens of millions of parameters and are characterized by lightweight deployment and fast inference speed. **General Large Model Approach:** This approach utilizes pre-trained large-scale language models (such as the GPT series and medically fine-tuned versions of LLaMA), which are then fine-tuned through instructions to achieve functions such as clinical question answering and diagnostic reasoning. These models have billions or even hundreds of billions of parameters and possess powerful semantic understanding and logical reasoning capabilities.

[0004] Existing research attempts to directly train end-to-end models by simply concatenating small and large models. However, due to their different training objectives and poor synergy, the errors of the small model are directly transmitted and amplified in the concatenated architecture, and cannot be corrected by the loss of downstream tasks, thus affecting the accuracy of diagnostic decisions. Summary of the Invention

[0005] Based on the above analysis, the embodiments of the present invention aim to provide a training method for a clinical auxiliary diagnostic model to solve the problem that poor synergy between existing large and small models affects the accuracy of diagnostic decisions.

[0006] On one hand, embodiments of the present invention provide a training method for a clinical auxiliary diagnostic model, comprising the following steps:

[0007] Collect medical texts from multiple patients to construct a sample set;

[0008] A text compression model and a large-scale medical diagnosis model are constructed, and the constructed text compression model and the large-scale medical diagnosis model are pre-trained based on the sample set; the large-scale medical diagnosis model adopts a large language model structure.

[0009] With the parameters of the fixed medical diagnosis model fixed, the compressed text output by the text compression model is used as the input of the medical diagnosis model. The text compression model is adapted and optimized based on the diagnostic task loss of the medical diagnosis model.

[0010] With the parameters of the fixed text compression model as the input of the medical large model, the medical diagnostic large model is fine-tuned based on the diagnostic task loss of the medical diagnostic large model to obtain a trained clinical auxiliary diagnostic model.

[0011] Based on a further improvement of the above method, the text compression model includes:

[0012] The input layer is used for preprocessing the input text.

[0013] A shared encoder is used to extract shared features from preprocessed text.

[0014] The compression task header is used to decode shared features and predict compressed text.

[0015] The key information extraction header is used to decode shared features and predict key field information.

[0016] Based on a further improvement of the above method, the input layer preprocesses the input text by adaptively segmenting the input text into blocks; the input layer includes:

[0017] Chunking unit; used to divide input text into chunks based on similarity;

[0018] The overlap adjustment unit is used to adjust the block division based on the overlapping area between adjacent blocks determined by multi-factor fusion, so as to obtain the final block division result.

[0019] Based on further improvements to the above method, the overlap adjustment unit determines the overlapping area between adjacent blocks using the following method based on multi-factor fusion:

[0020] ;

[0021] in, This represents the size of the overlapping region between the i-th block and the (i+1)-th block determined based on semantic units. This represents the size of the overlapping region between the i-th block and the (i+1)-th block, determined based on the block length. This represents the size of the overlapping region between the i-th block and the (i+1)-th block, determined based on entity density. , and All represent weighting coefficients. This represents the length of the overlapping area between the final determined i-th block and the (i+1)-th block.

[0022] Based on the above method, the shared encoder extracts shared features from the blocks in the following way:

[0023] For each block, feature extraction is performed to obtain the basic features of each block;

[0024] For overlapping regions of adjacent blocks, the basic features corresponding to the overlapping regions are fused based on bidirectional attention to obtain the fused features of the overlapping regions;

[0025] The technical features of the non-overlapping regions of each block and the fusion features of the overlapping regions of adjacent blocks are sequentially spliced ​​together to form shared features.

[0026] Based on the above method, a further improvement is made to obtain the fused features of the overlapping regions by fusing the basic features corresponding to the overlapping regions using bidirectional attention, including:

[0027] Project the basic features of the overlapping regions of two adjacent blocks onto the query, key, and value spaces, respectively;

[0028] Bidirectional cross attention is calculated using multi-head attention based on the projected query matrix, key matrix, and value matrix;

[0029] The fusion features of the overlapping region are obtained by position-weighted fusion of bidirectional cross-attention.

[0030] Based on the above method, a further improvement is made to calculate the pre-training loss of the text compression model in the following way:

[0031] ;

[0032] in, This represents the compression loss of the text compression model. This represents the key information prediction loss of the text compression model. and All of these represent weighting coefficients.

[0033] Based on further improvements to the above method, the diagnostic task loss of the large-scale medical diagnostic model is calculated using the following formula:

[0034] ;

[0035] in, This represents the real token at position t. The input sequence of the sample. This represents the sequence of tokens generated before position t in the large-scale medical diagnostic model. These represent the parameters of a large-scale medical diagnostic model. This represents the probability distribution predicted by the large-scale medical diagnostic model, where T represents the total length of the output sequence. These represent the parameters of the text compression model. Denotes the 2-norm of a matrix. and This represents the weighting coefficient.

[0036] Based on further improvements to the above methods, a large-scale medical diagnostic model is pre-trained using dynamic prompt optimization and key field weight optimization.

[0037] Further improvements to the above methods include pre-training a large-scale medical diagnostic model based on dynamic prompting optimization and key field weight optimization, including:

[0038] Build a basic prompt word project and reserve dynamic insertion points within it;

[0039] Build an error knowledge base;

[0040] During each round of training, the parameters of the large medical diagnostic model are adjusted based on inference loss and key field extraction loss;

[0041] After each round of training, dynamic insertion information is constructed based on the key fields extracted from the medical diagnosis model and the error knowledge base, and inserted into the dynamic insertion points to obtain the prompt word project for the next round of training.

[0042] Based on the above method, a further improvement is made to calculate the key field extraction loss using the following method:

[0043] ;

[0044] in, The true value of the i-th key field, This represents the i-th key field extracted from the large medical diagnostic model. The number of key fields This represents the weight of the i-th key field in the t-th training round. This represents the loss function.

[0045] Based on the above method, the weight of the i-th key field is calculated in the following way:

[0046] ;

[0047] in, This represents the weight of the i-th key field during the (t-1)th round of training. Indicates the attenuation factor. This represents the normalized error rate of the i-th key field in round t-1.

[0048] Based on further improvements to the above method, the chunking unit adaptively chunks the input text in the following way:

[0049] S11. Extract the sentence sequence of the input text and obtain the embedding vector of each sentence;

[0050] S12. Set the current sequence number to 1; set the block number of the first sentence to the current sequence number; take the second sentence in the sentence sequence as the current sentence;

[0051] S13. Calculate the similarity between the current sentence and the block corresponding to the current sequence number; if the similarity is greater than the similarity threshold and the length of the block corresponding to the current sequence number and the total length of the current sentence do not exceed the length threshold, then set the block number of the current sentence to the current sequence number; otherwise, increment the current sequence number by one and set the block number of the current sentence to the current sequence number.

[0052] S14. If there is a next sentence, then the next sentence is the current sentence, and return to step S13; otherwise, end the block segmentation.

[0053] Based on further improvements to the above method, the following formula is used to calculate... :

[0054] ;

[0055] in, Indicates the basic proportion. This represents the length of the i-th block. This represents the length of the (i+1)th block. Indicates the minimum overlap length. This indicates the maximum overlap length.

[0056] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0057] 1. Through progressive training, the compressed model learns to generate representations that are easy for large models to reason about, while the large model learns to understand the essence of the compressed semantics. The representation spaces of the two gradually align, overcoming the information loss and optimization conflicts of simple concatenated architectures and improving the accuracy of reasoning. The task orientation of the compressed model is enhanced (preserving key diagnostic information), and the reasoning targeting of the large model is improved, thereby enhancing the diagnostic accuracy in complex clinical scenarios.

[0058] 2. Through a dedicated division of labor architecture of text compression model + medical diagnosis big model, the text compression model is responsible for extracting ultra-long text into compact semantic representation, while the medical diagnosis big model focuses on deep reasoning based on compressed representation, which solves the contradiction that the small model is not capable of processing long text and the large model is limited by input length.

[0059] 3. Alternating training provides a stable learning environment for both models, significantly improving the training success rate and convergence speed.

[0060] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description

[0061] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0062] Figure 1 This is a flowchart of the training method for the clinical auxiliary diagnostic model according to an embodiment of the present invention. Detailed Implementation

[0063] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0064] A specific embodiment of the present invention discloses a training method for a clinical auxiliary diagnostic model, such as... Figure 1 As shown, it includes the following steps:

[0065] S1. Obtain medical texts from multiple patients to construct a sample set;

[0066] S2. Construct a text compression model and a large-scale medical diagnosis model. Based on the sample set, pre-train the constructed text compression model and the large-scale medical diagnosis model respectively. The large-scale medical diagnosis model adopts a large language model structure.

[0067] S3. Fix the parameters of the medical diagnosis model, use the compressed text output by the text compression model as the input of the medical diagnosis model, and adapt and optimize the text compression model based on the diagnostic task loss of the medical diagnosis model.

[0068] S4. Fix the parameters of the text compression model, use the compressed text output by the text compression model as the input of the medical large model, and fine-tune the medical diagnostic large model based on the diagnostic task loss of the medical diagnostic large model to obtain a trained clinical auxiliary diagnostic model.

[0069] It should be noted that the obtained clinical auxiliary diagnostic model is a cascaded architecture of a text compression model and a medical diagnostic model.

[0070] It should be noted that in steps S3 and S4, the compressed text output by the text compression model is part of the data input to the large medical model.

[0071] Compared with existing technologies, the training method of the clinical auxiliary diagnostic model provided in this embodiment constructs a sample set and adopts a progressive collaborative training framework. First, it pre-trains the text compression model and the large medical diagnostic model separately. Then, it fixes the parameters of the large medical diagnostic model and uses the compressed text output by the text compression model as the input of the large medical model. Based on the diagnostic task loss of the large medical diagnostic model, it adapts and optimizes the text compression model. Finally, it fixes the parameters of the text compression model and uses the compressed text output by the text compression model as the input of the large medical model. Based on the diagnostic task loss of the large medical diagnostic model, it fine-tunes the large medical diagnostic model, enabling the text compression model to learn to generate representations that are "easy for the large model to reason about". The large model learns to understand the essence of the compressed semantics. The compressed text generated by the text compression model through diagnostic task loss optimization naturally adapts to the input requirements of the large model, reducing information loss and redundancy, improving the reasoning efficiency of the large model, and thus overcoming the information loss and optimization conflict of the simple serial architecture, improving the diagnostic accuracy in complex clinical scenarios.

[0072] In practice, the patient's medical texts include medical text data from multiple sources such as outpatient medical records and examination reports. For each patient, a summary of the medical text, key field information, and doctor's diagnosis information are obtained as tags corresponding to that patient's medical text, and a sample is constructed to build a sample set.

[0073] Key fields include symptom fields, examination fields, medical history fields, and medication fields.

[0074] In practice, the text compression model uses a small model architecture, such as the BERT model. It should be noted that the small model is designed for compressing long medical text information and does not include basic structured data such as patient names and ages.

[0075] In the first phase, a text compression model and a large medical diagnosis model were pre-trained based on the constructed sample set.

[0076] In implementation, to improve the reasoning capabilities of the large-scale medical diagnostic model, it is used not only for diagnostic reasoning but also for key field extraction. During implementation, large-scale models such as Qwen and GPT can be used.

[0077] During implementation, the large-scale medical diagnosis model is pre-trained based on dynamic prompt optimization and key field weight optimization.

[0078] Specifically, the pre-training of large-scale medical diagnostic models includes:

[0079] Build a basic prompt word project and reserve dynamic insertion points within it;

[0080] Build an error knowledge base;

[0081] During each round of training, the parameters of the large medical diagnostic model are adjusted based on inference loss and key field extraction loss;

[0082] After each round of training, dynamic insertion information is constructed based on the key fields extracted from the medical diagnosis big data model and the error knowledge base and inserted into the dynamic insertion point to obtain the prompt word project for the next round of training.

[0083] When training a large-scale medical diagnostic model, the input data includes basic structured patient data and prompts summarizing medical texts. The output consists of extracted key fields and clinical decision reasoning results. During implementation, existing large-scale model fine-tuning techniques (e.g., LoRA, QLoRA, adapter techniques) are used to adjust the parameters of the medical diagnostic model.

[0084] During implementation, first design the basic prompt word project and reserve dynamic insertion points, for example:

[0085] """Document Medical Records:

[0086] {input_text}

[0087] JSON used to describe the surgery: [

[0089] {

[0090] "Surgery time": "",

[0091] "Combined surgical procedure name": "",

[0092] "Sub-operation": [

[0093] {

[0094] "Surgery Name": "",

[0095] "Surgical Site": "",

[0096] "Surgery Status": "",

[0097] Surgical procedure: "",

[0098] "Surgical approach": "",

[0099] "Extended surgical resection area": ​​"",

[0100] "Assemblage method": "",

[0101] "Nature of surgery": ""

[0102] }

[0103] ],

[0104] Intraoperative blood loss: "",

[0105] "Intraoperative blood loss in units": "",

[0106] "Transfusion Status": "",

[0107] "Details of intraoperative blood transfusion": [

[0108] {

[0109] "Intraoperative blood transfusion components": "",

[0110] Intraoperative blood transfusion volume: "",

[0111] "Intraoperative blood transfusion volume (units): ""

[0112] }

[0113] ],

[0114] "Anesthesia method": "",

[0115] ASA Score: "",

[0116] "Intraoperative / postoperative complications": [

[0117] {

[0118] Symptoms / Diagnosis: "",

[0119] "Symptoms / Diagnostic Status": "",

[0120] "Symptoms / Diagnosis Time": ""

[0121] }

[0122] ],

[0123] "Intraoperative medication": [

[0124] {

[0125] "Drug Name": "",

[0126] "Method of administration": "",

[0127] Dosage: "

[0128] } ]

[0130] } ]

[0132] Field description:

[0133] Please note that multiple surgery times may occur, and all of them must be output.

[0134] Combined surgical procedure name: Sometimes a single surgery involves multiple procedures. The medical record will provide a combined name for this surgery, extracted from the original medical record's 'Surgery / Procedure Name'. For example: AAA+BBB+CCC, etc.

[0135] Surgical name: Take the names corresponding to the surgical steps contained in the 'Combined Surgical Name' separately and in sequence. For example, when the combined surgical name is 'Distal gastrectomy, D2 lymph node dissection, Billroth II + Braun anastomosis', the sub-surgical names are 'Distal gastrectomy', 'D2 lymph node dissection', and 'Billroth II + Braun anastomosis', respectively.

[0136] Surgical site: Only take the value from the name of the surgery, such as: the surgical site of total gastrectomy is the whole stomach.

[0137] Surgery status: Value range: confirmed / suspected / denied.

[0138] Surgical procedure: Examples include open surgery, laparoscopic surgery, endoscopic surgery, robotic surgery, etc. If no specific procedure is mentioned, output a null value.

[0139] Surgical nature: Exploration, diagnosis, radical treatment, modified radical treatment, palliative. If the specific surgical nature is not explicitly mentioned, output a null value.

[0140] Surgical approach: refers to the path through which surgery enters the surgical area. For example, the surgical approach for laparoscopy is the abdomen, the surgical approach for thoracoscopic surgery is the chest, the surgical approach for open surgery is the abdomen, and the surgical approach for thoracolaparoscopic surgery is the chest and abdomen.

[0141] Anastomosis method: refers to the reconstruction method that connects the severed ends after surgery. Examples include: Billroth I, Billroth II, Billroth II + Braun, Roux-en-Y, Uncut Roux-en-Y, and esophageal-gastric remnant anastomosis.

[0142] Surgical resection scope: Only the value is taken from the name of the surgery, such as the surgical site of total gastrectomy is the whole stomach.

[0143] ### [Dynamic Insertion Point - START]

[0144] Dynamic content includes:

[0145] Error correction rules based on an error knowledge base

[0146] - Field weight adjustment instructions

[0147] ### [Dynamic Insertion Point - END]

[0148] Read the surgical names in the above medical records and describe the other surgeries mentioned in the records, excluding those mentioned in the postoperative status description, using the given JSON format. If no surgeries meet the criteria, output an empty list.

[0149] During implementation, an error knowledge base is built based on experience. The error knowledge base stores the error types that may occur for each key field, as well as the corresponding prompts for that error.

[0150] During implementation, unrecorded error types are clustered to generate candidate knowledge entries. Human-machine collaborative verification is then implemented, automatically recommending new error patterns, which are then confirmed by domain experts to update the error knowledge base.

[0151] During each round of training of the large medical diagnostic model, the parameters of the large medical diagnostic model are adjusted based on inference loss and key field extraction loss.

[0152] Specifically, the inference loss can use existing loss functions, such as the cross-entropy loss function and the mean squared error loss function.

[0153] Specifically, the loss for extracting key fields is calculated using the following method:

[0154] ;

[0155] in, The true value of the i-th key field, This represents the i-th key field extracted from the large medical diagnostic model. The number of key fields This represents the weight of the i-th key field in the t-th training round. This represents the loss function.

[0156] Specifically, the weight of the i-th key field is calculated using the following method:

[0157] ;

[0158] in, This represents the weight of the i-th key field during the (t-1)th round of training. Indicates the attenuation factor. This represents the normalized error rate of the i-th key field in round t-1.

[0159] By using a key field weighted loss function, the weights of each key field are calculated based on the error rate of the key fields in the previous round, so that the model training resources are tilted towards fields with high error rates, thereby improving the accuracy of key field extraction.

[0160] The total loss is obtained by summing the inference loss and the key field extraction loss. Based on the total loss, the parameters of the large medical diagnostic model are adjusted using existing large model fine-tuning techniques.

[0161] After each round of training, dynamic insertion information is constructed based on the key fields extracted from the medical diagnosis big data model and the error knowledge base and inserted into the dynamic insertion point to obtain the prompt word project for the next round of training.

[0162] During implementation, after each training round, the key field information of errors extracted in that round is extracted. It is then determined whether this erroneous key field information exists in the error knowledge base. If it does, the corresponding prompt content is inserted into the dynamically reserved insertion point in the prompt word project. For example, if an error in the format of the surgical time key field is found after the current training round, the corresponding prompt content is inserted into the aforementioned prompt word project. The prompt word project then becomes:

[0163] """Document Medical Records:

[0164] {input_text}

[0165] JSON used to describe the surgery: [

[0167] {

[0168] "Surgery time": "",

[0169] "Combined surgical procedure name": "",

[0170] "Sub-operation": [

[0171] {

[0172] "Surgery Name": "",

[0173] "Surgical Site": "",

[0174] "Surgery Status": "",

[0175] Surgical procedure: "",

[0176] "Surgical approach": "",

[0177] "Extended surgical resection area": ​​"",

[0178] "Assemblage method": "",

[0179] "Nature of surgery": ""

[0180] }

[0181] ],

[0182] Intraoperative blood loss: "",

[0183] "Intraoperative blood loss in units": "",

[0184] "Transfusion Status": "",

[0185] "Details of intraoperative blood transfusion": [

[0186] {

[0187] "Intraoperative blood transfusion components": "",

[0188] Intraoperative blood transfusion volume: "",

[0189] "Intraoperative blood transfusion volume (units): ""

[0190] }

[0191] ],

[0192] "Anesthesia method": "",

[0193] ASA Score: "",

[0194] "Intraoperative / postoperative complications": [

[0195] {

[0196] Symptoms / Diagnosis: "",

[0197] "Symptoms / Diagnostic Status": "",

[0198] "Symptoms / Diagnosis Time": ""

[0199] }

[0200] ],

[0201] "Intraoperative medication": [

[0202] {

[0203] "Drug Name": "",

[0204] "Method of administration": "",

[0205] Dosage: "

[0206] } ]

[0208] } ]

[0210] Field description:

[0211] Please note that multiple surgery times may occur, and all of them must be output.

[0212] Combined surgical procedure name: Sometimes a single surgery involves multiple procedures. The medical record will provide a combined name for this surgery, extracted from the original medical record's 'Surgery / Procedure Name'. For example: AAA+BBB+CCC, etc.

[0213] Surgical name: Take the names corresponding to the surgical steps contained in the 'Combined Surgical Name' separately and in sequence. For example, when the combined surgical name is 'Distal gastrectomy, D2 lymph node dissection, Billroth II + Braun anastomosis', the sub-surgical names are 'Distal gastrectomy', 'D2 lymph node dissection', and 'Billroth II + Braun anastomosis', respectively.

[0214] Surgical site: Only take the value from the name of the surgery, such as: the surgical site of total gastrectomy is the whole stomach.

[0215] Surgery status: Value range: confirmed / suspected / denied.

[0216] Surgical procedure: Examples include open surgery, laparoscopic surgery, endoscopic surgery, robotic surgery, etc. If no specific procedure is mentioned, output a null value.

[0217] Surgical nature: Exploration, diagnosis, radical treatment, modified radical treatment, palliative. If the specific surgical nature is not explicitly mentioned, output a null value.

[0218] Surgical approach: refers to the path through which surgery enters the surgical area. For example, the surgical approach for laparoscopy is the abdomen, the surgical approach for thoracoscopic surgery is the chest, the surgical approach for open surgery is the abdomen, and the surgical approach for thoracolaparoscopic surgery is the chest and abdomen.

[0219] Anastomosis method: refers to the reconstruction method that connects the severed ends after surgery. Examples include: Billroth I, Billroth II, Billroth II + Braun, Roux-en-Y, Uncut Roux-en-Y, and esophageal-gastric remnant anastomosis.

[0220] Surgical resection scope: Only the value is taken from the name of the surgery, such as the surgical site of total gastrectomy is the whole stomach.

[0221] ### [Dynamic Insertion Point - START]

[0222] Dynamic content includes:

[0223] Surgical time: The output format for surgical time is yyyy-mm-dd

[0224] ### [Dynamic Insertion Point - END]

[0225] Read the surgical names in the above medical records and describe the other surgeries mentioned in the records, excluding those mentioned in the postoperative status description, using the given JSON format. If no surgeries meet the criteria, output an empty list.

[0226] It should be noted that the dynamic reserved points in this solution are not in the large model structure or parameters, but in the prompt word project, that is, dynamically inserting information into the data of the input large model.

[0227] Specifically, the constructed text compression model includes:

[0228] The input layer is used for preprocessing the input text.

[0229] A shared encoder is used to extract shared features from preprocessed text.

[0230] The compression task header is used to decode shared features and predict compressed text.

[0231] The key information extraction head is used to decode shared features and predict key information.

[0232] Because medical texts, especially those concerning chronic diseases, are often very long, exceeding the processing capacity of large models, a text compression model is employed. To ensure that extremely long medical texts are compressed into concise representations while retaining information crucial for clinical diagnosis, the text compression model uses a multi-task learning architecture to predict key information during text compression.

[0233] In implementation, the input layer preprocesses the input text, including adaptive text segmentation. Specifically, the input layer includes:

[0234] Chunking unit; used to divide input text into chunks based on similarity;

[0235] The overlap adjustment unit is used to determine the overlapping area between adjacent blocks based on information density and semantic importance, and then adjust the block segmentation to obtain the final segmentation result.

[0236] In implementation, the chunking unit adaptively chunks the input text in the following way:

[0237] S11. Extract the sentence sequence of the input text and obtain the embedding vector of each sentence;

[0238] S12. Set the current sequence number to 1; set the block number of the first sentence to the current sequence number; take the second sentence in the sentence sequence as the current sentence;

[0239] S13. Calculate the similarity between the current sentence and the block corresponding to the current sequence number; if the similarity is greater than the similarity threshold and the length of the block corresponding to the current sequence number and the total length of the current sentence do not exceed the length threshold, then set the block number of the current sentence to the current sequence number; otherwise, increment the current sequence number by one and set the block number of the current sentence to the current sequence number.

[0240] S14. If there is a next sentence, then the next sentence is the current sentence, and return to step S13; otherwise, end the block segmentation.

[0241] To ensure the integrity of semantic units, the block segmentation of the present invention is based on sentences of the input text.

[0242] During implementation, the sentence sequence of the input text is first extracted. ,in, Let represent the i-th sentence, and n represent the number of sentences. The embedding vector of the i-th sentence is represented as... In practice, the embedding vector of the sentence can be obtained through an existing sentence encoder (such as BERT's [CLS] token).

[0243] During implementation, initially, the current sequence number is one. (First sentence) The block number is the current block number, meaning the initial block number is one. The current sentence is the second sentence. ,calculate Similarity to the first block (currently there is only one sentence) If the similarity is greater than the similarity threshold and the length of the block corresponding to the current sequence number is... If the total length does not exceed the length threshold, then... Set the block number to the current number; otherwise, increment the current number by one before setting the current number. The block number is set to the current number, that is... This marks the beginning of a new block. Then, with... Return to step S13 for the current sentence, and so on, until the segmentation ends. The length of the segment corresponding to the current sequence number is... The total length refers to the number of blocks and sentences. The character length. The similarity between the current sentence and the block corresponding to the current sequence number is calculated based on the embedding vector of the current sentence and the embedding vector of the block corresponding to the current sequence number.

[0244] In practice, when calculating the similarity between the current sentence and the block corresponding to the current sequence number, the embedding vector of the block corresponding to the current sequence number is defined as the average pooling of the embedding vectors of the sentences within the block corresponding to the current sequence number. Then, the similarity between the embedding vector of the current sentence and the embedding vector of the block corresponding to the current sequence number is calculated. For example, cosine similarity can be calculated.

[0245] During implementation, the similarity threshold can be set according to the similarity calculation method used. The length threshold can be set according to the processable length of the model.

[0246] To maintain semantic continuity, the overlapping areas between adjacent blocks are determined by the overlap adjustment unit based on multi-factor fusion, and the block segmentation is adjusted to obtain the final segmentation result.

[0247] Specifically, the block division is adjusted based on the overlapping areas between adjacent blocks determined by multi-factor fusion, resulting in the final block division result, including:

[0248] ;

[0249] in, This represents the size of the overlapping region between the i-th block and the (i+1)-th block determined based on semantic units. This represents the size of the overlapping region between the i-th block and the (i+1)-th block, determined based on the block length. This represents the size of the overlapping region between the i-th block and the (i+1)-th block, determined based on entity density. , and All represent weighting coefficients. This represents the length of the overlapping region between the final determined i-th block and the (i+1)-th block. (The result is...) Then, search backwards from the end of the i-th block. The position of the length is used as the starting point of the (i+1)th block, that is, the length of the i-th block is searched from the end to the beginning. The length portion is added to the (i+1)th block, thus adjusting the starting position of the (i+1)th block in the entire text. To maintain sentence coherence, if the starting position of the search is located in the middle of a sentence, then the starting position of that sentence is used as the starting position of the (i+1)th block.

[0250] Specifically, the following formula is used for calculation. :

[0251] ;

[0252] in, Indicates the minimum overlap length. Indicates the maximum overlap length. Let represent the set of the last k sentences of the i-th block. `s` represents the length of sentence `s`. `k` represents the minimum number of overlapping sentences, for example, 2 overlapping sentences, 3 overlapping sentences. In implementation, and The overlap length can be adjusted based on the length of the medical text. Longer medical texts have weaker global context dependencies, so the overlap length can be appropriately reduced to improve computational efficiency. Shorter texts have stronger context sparsity, so the overlap length needs to be increased to ensure the cross-block coherence of key information. For example, for medical texts less than 10,000 characters, the minimum overlap length is set to 70 characters and the maximum overlap length to 100 characters; for medical texts greater than 10,000 but less than 20,000 sentences, the minimum overlap length is set to 50 characters and the maximum overlap length to 80 characters; for medical texts greater than 20,000 characters, the minimum overlap length is set to 30 characters and the maximum overlap length to 60 characters.

[0253] Specifically, the following formula is used for calculation. :

[0254] ;

[0255] in, This represents the basic ratio, for example, it can be 0.2. This represents the length of the i-th block. This represents the length of the (i+1)th block.

[0256] Specifically, the following formula is used for calculation. :

[0257] ;

[0258] ;

[0259] ;

[0260] ;

[0261] in, This represents the number of entities within the last k statements of the i-th block. Indicates the minimum overlap length. This represents the length of the last k statements in the i-th block. This represents the number of entities in the i-th block. This represents the density adjustment function.

[0262] During implementation, entities can be identified using existing NER models.

[0263] In implementation, the density adjustment function can be a piecewise function, for example:

[0264] 。

[0265] By considering the semantic units, length, and entity density of adjacent blocks, the semantics of overlapping areas are made smoother and the connections are more reasonable.

[0266] After the input text is segmented, the shared encoder extracts features from the segments to obtain shared features.

[0267] Specifically, the shared encoder uses the following method to extract shared features from the blocks:

[0268] For each block, feature extraction is performed to obtain the basic features of each block;

[0269] For overlapping regions of adjacent blocks, the basic features corresponding to the overlapping regions are fused based on bidirectional attention to obtain the fused features of the overlapping regions;

[0270] The technical features of the non-overlapping regions of each block and the fusion features of the overlapping regions of adjacent blocks are sequentially spliced ​​together to form shared features.

[0271] In implementation, the shared encoder can use an existing pre-trained natural language processing (NLP) model to extract the basic features of each block. For each text block, it is input into the pre-trained NLP model, and the high-dimensional semantic vector output by the model is obtained as the basic features of that block. The pre-trained NLP model is a lightweight Transformer model, such as the ALBERT model, and its parameters are frozen during the training of the shared encoder.

[0272] For overlapping regions of adjacent blocks, feature fusion is performed based on bidirectional attention.

[0273] Specifically, based on bidirectional attention, the following formula is used to fuse the basic features corresponding to overlapping regions. This includes:

[0274] Project the basic features of the overlapping regions of two adjacent blocks onto the query, key, and value spaces, respectively:

[0275] ;

[0276] ;

[0277] ;

[0278] ;

[0279] ;

[0280] ;

[0281] in, This represents the basic features of the part that overlaps with the (i+1)th block in the i-th block. , and The query matrix, key matrix, and value matrix represent the query matrix, key matrix, and value matrix of the j-th attention head corresponding to the basic features of the part overlapping with the (i+1)-th block in the i-th block; , and These represent the query projection matrix, key projection matrix, and value projection matrix of the j-th attention head corresponding to the basic features of the part overlapping with the (i+1)-th block in the i-th block, respectively. , and The query matrix, key matrix, and value matrix represent the j-th attention head corresponding to the basic features of the part overlapping with the ith block in the (i+1)-th block; , and Let $\mathbf{i}$ and $\mathbf{j}$ represent the query projection matrix, key projection matrix, and value projection matrix of the j-th attention head corresponding to the basic features of the part overlapping with the i-th block in the (i+1)-th block. $\mathbf{i}$ represents the basic features of the part overlapping with the i-th block in the (i+1)-th block.

[0282] Bidirectional cross attention is calculated using multi-head attention based on the projected query matrix, key matrix, and value matrix;

[0283] ;

[0284] ;

[0285] ;

[0286] ;

[0287] in, This represents the attention calculated by the j-th attention head; and This represents the attention mask matrix, used to control which locations can pay attention to each other. The superscript T indicates transpose. The dimension of the attention head is represented by h. h represents the number of attention heads. Indicates the output projection matrix. Indicates splicing, This represents the softmax function.

[0288] Position-weighted fusion of bidirectional cross-attention yields fusion features of overlapping regions:

[0289] ;

[0290] ;

[0291] ;

[0292] ;

[0293] in, This refers to the dropout function. The layer normalization function is represented. This represents a multilayer perceptron. This represents the sigmoid function. This represents the fusion feature of the overlapping region between the i-th block and the (i+1)-th block; This represents the fusion feature of the region overlapping with the ith block in the (i+1)th block. express The fusion feature at position p in the middle, express The fusion feature at position p in the middle, This represents the weight of the p-th position. This represents the fused feature at the p-th position in the final fused feature set. This represents the final fusion feature obtained.

[0294] Since calculations in two directions may produce different results, adaptive weighting is required during the final fusion process. This involves calculating the weights for each direction at each location to improve feature accuracy and maintain global coherence.

[0295] Then, the technical features of the non-overlapping regions of each block and the fusion features of the overlapping regions of adjacent blocks are sequentially spliced ​​together to form shared features.

[0296] For example, the block sequence obtained from the input layer is .

[0297] In practice, during stitching, only the right overlapping region of the previous block needs to be extracted (because the right overlapping region of the subsequent block is the same region as the right overlapping region of the previous block, and they have already been merged into the same features). For example, in block stitching... Middle and Block The overlapping part is represented as The other parts are represented as Segmentation Middle and Block The overlapping part is represented as The other parts are represented as Segmentation Middle and Block The overlapping part is represented as The other parts are represented as The feature sequence formed by sequentially splicing together is .

[0298] During implementation, the compression task header decodes the obtained shared features to output a summary of the medical text. The decoding module can employ a single-layer Transformer decoder.

[0299] The key information extraction head decodes shared features to predict key field information.

[0300] Specifically, the pre-training loss of the text compression model is calculated using the following method:

[0301] ;

[0302] in, This represents the compression loss of the text compression model. This represents the key information prediction loss of the text compression model. and All of these represent weighting coefficients.

[0303] When pre-training the text compression model, backpropagation is performed based on the pre-training loss of the text compression model to update the parameters of the text compression model.

[0304] In implementation, the compression loss of the text compression model can use existing loss functions, such as cross-entropy loss function and mean squared error loss function, to calculate the loss between the medical text summary in the sample labels and the medical text summary output by the model. The key information prediction loss can also use existing loss functions, such as mean squared error loss and cross-entropy loss function, to calculate the loss between the key field information in the sample labels and the predicted key field information.

[0305] It should be noted that the uncompressed text includes irrelevant or potentially misleading information such as precautions, differential diagnoses, and suggestions provided by the diagnosis and treatment team. The compressed text will remove this information to avoid misleading the large model's reasoning. The compressed medical information is shorter than the uncompressed medical information.

[0306] The first stage of training is completed when the pre-training loss of both the text compression model and the large-scale medical diagnosis model is less than a preset threshold. The threshold can be set according to the training accuracy requirements.

[0307] Through a dedicated architecture combining a text compression model and a large-scale medical diagnostic model, the text compression model extracts extremely long texts into compact semantic representations, while the large-scale medical diagnostic model focuses on deep reasoning. This resolves the contradiction between the insufficient ability of small models to handle long texts and the limited input length of large models. The compression model reduces texts containing thousands of tokens to vectors of hundreds of dimensions, preserving key information while adapting to the input limitations of large models.

[0308] To enable the text compression model to better adapt to diagnostic reasoning, in the second stage, the parameters of the large medical diagnostic model are fixed, and the compressed text output by the text compression model and the patient's basic structured data are used as inputs to the large medical diagnostic model. Based on the diagnostic task loss of the large medical diagnostic model, the text compression model is adapted and optimized so that its output compressed text can directly meet the input requirements of the large medical diagnostic model without the large model having to process redundant information.

[0309] In the second stage, with the goal of minimizing the diagnostic task loss of the large-scale medical diagnostic model, gradients are backpropagated to update only the parameters of the text compression model. This process forces the compression model to learn to "generate compressed text that directly contributes to the diagnostic results" while filtering out irrelevant information.

[0310] During implementation, in the second phase, the diagnostic task loss of the large-scale medical diagnostic model is calculated using the following formula:

[0311] ;

[0312] in, This represents the real token at position t. The input sequence representing the sample. This represents the sequence of tokens generated before position t in the large-scale medical diagnostic model. These represent the parameters of a large-scale medical diagnostic model. This represents the probability distribution predicted by the large-scale medical diagnostic model, where T represents the total length of the output sequence. These represent the parameters of the text compression model. Denotes the 2-norm of a matrix. and This represents the weighting coefficient.

[0313] To avoid overfitting in the text compression model, regularization loss is introduced during training. The parameters of the large medical diagnostic model are only used for constraints and are not updated.

[0314] When the diagnostic task loss of the large medical diagnostic model converges (or reaches the preset number of iterations), the adaptation optimization of the text compression model is stopped, all its parameters are fixed, and the final text compression model adapted to the input of the large model is generated, completing the second stage of training.

[0315] In the third stage, the parameters of the text compression model optimized in the second stage are frozen, and training is performed independently only on the large-scale medical diagnostic model. The input sources for the large-scale medical diagnostic model are the output of the optimized text compression model from the second stage and the patient's basic structured data. Based on the diagnostic task loss, the parameters of the large-scale medical diagnostic model are updated, and the parameter adjustment method can be the same as in the pre-training stage. To reduce training costs and avoid destroying pre-trained knowledge, only the parameters of the last few layers of the large-scale medical diagnostic model (such as partial weights of the output layer, fully connected layer, and attention layer) can be updated.

[0316] When the diagnostic task loss converges and the model performance (such as accuracy, recall, F1 score, etc.) reaches the preset indicators, training stops, and the final clinical auxiliary diagnostic model (a series architecture of text compression model + medical diagnostic big model) is obtained.

[0317] Through progressive, phased training, the compressed model learns to generate representations that are easy for large models to reason about, and the large model learns to understand the essence of the compressed semantics, overcoming the information loss and optimization conflicts of simple concatenated architectures.

[0318] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0319] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A training method for a clinical auxiliary diagnostic model, characterized in that, Includes the following steps: Collect medical texts from multiple patients to construct a sample set; A text compression model and a large-scale medical diagnosis model are constructed, and the constructed text compression model and the large-scale medical diagnosis model are pre-trained based on the sample set; the large-scale medical diagnosis model adopts a large language model structure. With the parameters of the fixed medical diagnosis model fixed, the compressed text output by the text compression model is used as the input of the medical diagnosis model. The text compression model is adapted and optimized based on the diagnostic task loss of the medical diagnosis model. With the parameters of the fixed text compression model, the compressed text output by the text compression model is used as the input of the large medical model. The large medical diagnostic model is fine-tuned based on the diagnostic task loss of the large medical diagnostic model to obtain a trained clinical auxiliary diagnostic model. The text compression model includes: The input layer is used for preprocessing the input text. A shared encoder is used to extract shared features from preprocessed text. The compression task header is used to decode shared features and predict compressed text. The key information extraction header is used to decode shared features and predict key field information; The input layer preprocesses the input text, including adaptive text segmentation; the input layer includes: Chunking unit; used to divide input text into chunks based on similarity; The overlap adjustment unit is used to adjust the blocks based on the overlapping area between adjacent blocks determined by multi-factor fusion, so as to obtain the final block division result; The pre-training loss of the text compression model is calculated using the following method: ； in, This represents the compression loss of the text compression model. This represents the key information prediction loss of the text compression model. and All represent weighting coefficients; A large-scale medical diagnostic model is pre-trained based on dynamic prompting optimization and key field weight optimization.

2. The training method for the clinical auxiliary diagnostic model according to claim 1, characterized in that, The overlap adjustment unit determines the overlapping area between adjacent blocks based on multi-factor fusion using the following method: ； in, This represents the size of the overlapping region between the i-th block and the (i+1)-th block determined based on semantic units. This represents the size of the overlapping region between the i-th block and the (i+1)-th block, determined based on the block length. This represents the size of the overlapping region between the i-th block and the (i+1)-th block, determined based on entity density. , and All represent weighting coefficients. This represents the length of the overlapping area between the final determined i-th block and the (i+1)-th block.

3. The training method for the clinical auxiliary diagnostic model according to claim 1, characterized in that, The shared encoder extracts shared features from blocks using the following method: For each block, feature extraction is performed to obtain the basic features of each block; For overlapping regions of adjacent blocks, the basic features corresponding to the overlapping regions are fused based on bidirectional attention to obtain the fused features of the overlapping regions; The technical features of the non-overlapping regions of each block and the fusion features of the overlapping regions of adjacent blocks are sequentially spliced ​​together to form shared features.

4. The training method for the clinical auxiliary diagnostic model according to claim 3, characterized in that, The fused features of the overlapping regions are obtained by fusing the basic features corresponding to the overlapping regions based on bidirectional attention, including: Project the basic features of the overlapping regions of two adjacent blocks onto the query, key, and value spaces, respectively; Bidirectional cross attention is calculated using multi-head attention based on the projected query matrix, key matrix, and value matrix; The fusion features of the overlapping region are obtained by position-weighted fusion of bidirectional cross-attention.

5. The training method for the clinical auxiliary diagnostic model according to claim 1, characterized in that, The diagnostic task loss of the large-scale medical diagnostic model is calculated using the following formula: ； in, This represents the real token at position t. The input sequence representing the sample. This represents the sequence of tokens generated before position t in the large-scale medical diagnostic model. These represent the parameters of a large-scale medical diagnostic model. This represents the probability distribution predicted by the large-scale medical diagnostic model, where T represents the total length of the output sequence. These represent the parameters of the text compression model. Denotes the 2-norm of a matrix. and This represents the weighting coefficient.

6. The training method for the clinical auxiliary diagnostic model according to claim 1, characterized in that, Pre-training of a large-scale medical diagnostic model based on dynamic prompting optimization and key field weight optimization includes: Build a basic prompt word project and reserve dynamic insertion points within it; Build an error knowledge base; During each round of training, the parameters of the large medical diagnostic model are adjusted based on inference loss and key field extraction loss; After each round of training, dynamic insertion information is constructed based on the key fields extracted from the medical diagnosis big data model and the error knowledge base and inserted into the dynamic insertion point to obtain the prompt word project for the next round of training.

7. The training method for the clinical auxiliary diagnostic model according to claim 6, characterized in that, The loss for extracting key fields is calculated using the following method: ； in, The true value of the i-th key field, This represents the i-th key field extracted from the large medical diagnostic model. The number of key fields This represents the weight of the i-th key field in the t-th training round. This represents the loss function.

8. The training method for the clinical auxiliary diagnostic model according to claim 7, characterized in that, The weight of the i-th key field is calculated using the following method: ； in, This represents the weight of the i-th key field during the (t-1)th round of training. Indicates the attenuation factor. This represents the normalized error rate of the i-th key field in round t-1.

9. The training method for the clinical auxiliary diagnostic model according to claim 1, characterized in that, The chunking unit adaptively chunks the input text using the following method: S11. Extract the sentence sequence of the input text and obtain the embedding vector of each sentence; S12. Set the current sequence number to one; Set the block number of the first sentence to the current block number; set the second sentence in the sentence sequence as the current sentence; S13. Calculate the similarity between the current sentence and the block corresponding to the current sequence number; If the similarity is greater than the similarity threshold and the length of the block corresponding to the current sequence number does not exceed the length threshold with the total length of the current sentence, then the block sequence number of the current sentence is set to the current sequence number; otherwise, the current sequence number is incremented by one, and the block sequence number of the current sentence is set to the current sequence number. S14. If there is a next sentence, then the next sentence is the current sentence, and return to step S13; Otherwise, end the block segmentation.

10. The training method for the clinical auxiliary diagnostic model according to claim 2, characterized in that, Calculate using the following formula : ； in, Indicates the basic proportion. This represents the length of the i-th block. This represents the length of the (i+1)th block. Indicates the minimum overlap length. This indicates the maximum overlap length.

Images