An electronic medical record text normalization method based on large model fine tuning

Abstract

The application belongs to the field of artificial intelligence technology application, and relates to an electronic medical record text normalization method based on large model fine tuning, which comprises the following steps: constructing a medical field specific corpus, which contains unnormalized electronic medical record text x k and its normalized electronic medical record text y k ; constructing an Lora block for each layer of a pre-trained large language model M pre ; constructing a hierarchical strength confirmation model; inputting the text x k into the hierarchical strength confirmation model to obtain hierarchical strength G k ; inserting a corresponding Lora block at each layer of the model M k according to the strength G pre , to obtain a large language model M lora with inserted Lora blocks; inputting the text x k into the model M lora to obtain a predicted text y' k ; updating the parameters of the model according to the text y' k and the text y k until a trained Lora block is obtained; inserting the trained Lora block into the corresponding layer of the model M pre to obtain a trained large language model; and the application dynamically selects the strength of the Lora block inserted in the model M pre by using the hierarchical strength confirmation model, thereby reducing the calculation overhead.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of medical information processing and artificial intelligence technology application, and relates to an electronic medical record text normalization method based on large model fine-tuning. BACKGROUND

[0002] The existing electronic medical record text normalization technology mainly relies on simple logical rewriting means such as regular matching. These methods often lack depth understanding and adaptability when dealing with complex and variable medical texts, resulting in limited normalization effect. In recent years, the emergence of large language models (LLM) has brought breakthrough progress in text processing, but its application in the medical field still faces the problems of adaptability and high cost. Although the Lora (Low-Rank Adaptation) fine-tuning technology can reduce training overhead, its traditional approach usually only inserts fine-tuning parameters in the fixed layers of the model, which cannot fully meet the adaptation needs of different complexity texts, and does not combine efficient training strategies. SUMMARY

[0003] To solve the above technical problems, the application adopts an electronic medical record text normalization method based on large model fine-tuning, which includes: obtaining an unnormalized electronic medical record text x in the medical field, inputting the text x into a trained large language model to obtain a normalized electronic medical record text; the training process of the large language model includes:

[0004] S1, constructing a medical field specific corpus, the medical field specific corpus containing unnormalized electronic medical record texts x k and its normalized electronic medical record texts y k ; wherein k represents the index of the unnormalized electronic medical record text;

[0005] S2, obtaining a pre-trained large language model, constructing a Lora block for each layer of the pre-trained large language model; constructing a hierarchical strength confirmation model;

[0006] S3, inputting the unnormalized electronic medical record text x k into the hierarchical strength confirmation model to obtain the predicted hierarchical strength G k of the text x k ;

[0007] S4, inserting the corresponding Lora block in each layer of the pre-trained large language model according to the hierarchical strength G k , to obtain a large language model with inserted Lora blocks;

[0008] S5, inputting the unnormalized electronic medical record text x k into the large language model with inserted Lora blocks to obtain the predicted text x k normalized electronic medical record text;

[0009] S6、 according to the predicted text x k Normalized electronic medical record text and its corresponding normalized electronic medical record text y k Calculate the loss function value, update the parameters of the Lora block and the hierarchical strength confirmation model according to the loss function value, and obtain the trained Lora block when the loss function value is minimum;

[0010] S7, insert the trained Lora block into the corresponding layer of the pre-trained large language model to obtain a trained large language model.

[0011] Beneficial effects:

[0012] 1. The Lora fine-tuning technology is used to fine-tune the pre-trained large language model, which can improve the adaptability of the large language model in the medical field electronic medical record text normalization task, so that the model can understand the medical record text content more deeply and accurately perform normalization operation, thereby improving the accuracy and utilization efficiency of medical record information; 2. The hierarchical strength of the Lora block inserted in the pre-trained large language model is adaptively selected by the hierarchical strength confirmation model, which improves the accuracy and utilization efficiency of medical record information while reducing unnecessary computing overhead; 3. The medical dictionary is used to assist the context complexity analysis, which improves the complex text discrimination ability. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 A flowchart of an electronic medical record text normalization method based on large model fine-tuning is provided for the embodiments of the present application;

[0014] Figure 2 A Lora principle diagram is provided for the embodiments of the present application;

[0015] Figure 3 A training process loss Loss trend chart is provided for the embodiments of the present application. DETAILED DESCRIPTION

[0016] The technical solutions in the embodiments of the present application will be described in detail below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0017] As Figure 1 shown, the present application adopts an electronic medical record text normalization method based on large model fine-tuning, which comprises: obtaining medical field non-normalized electronic medical record text X, inputting the text X into the trained large language model to obtain normalized electronic medical record text;

[0018] The training process of a large language model includes:

[0019] S1. Construct a medical domain-specific corpus, which contains multiple unnormalized electronic medical record texts. k and its standardized electronic medical record text y k Where k represents the index of the unnormalized electronic medical record text;

[0020] To train a large model for the standardization of electronic medical records in the medical field, it is first necessary to build a medical-specific corpus.

[0021] Unstandardized electronic medical record text data in the medical field typically contains information such as a patient's medical history, diagnosis, and treatment, but there may be differences in the wording. For example, in the sentence "Had been vaccinated against hepatitis B and BCG, denies history of infectious diseases such as tuberculosis and hepatitis," the words "had been vaccinated" and "denies" are not clearly marked for each item. Similarly, in the sentence "No subcutaneous nodules or scars, no subcutaneous bleeding or rash, no petechiae or ecchymosis," the word "none" is not repeated for each item.

[0022] The standardized electronic medical record text follows a unified format in its expression. For example, each piece of information is clearly marked with verbs such as "previously vaccinated" and "denied," and negative words such as "none" are repeatedly marked for each item.

[0023] S2. Obtain a pre-trained large language model and construct a LoRa block for each layer of the pre-trained large language model; construct a hierarchical strength confirmation model.

[0024] The pre-trained large language model is Chatglm-6B, which has been trained on large-scale text data and possesses powerful text understanding and generation capabilities. However, due to the specialized and unique nature of the medical field, directly using this model for the standardization of electronic medical records may not be effective. Therefore, this invention designs and employs an adaptive multi-layer LoRa fine-tuning technique to specifically train this model.

[0025] like Figure 2 As shown, the principle of LoRa fine-tuning is to add a bypass next to the pre-trained large language model, performing a dimensionality reduction and then dimensionality increase operation. During training, the parameters of the pre-trained language model are fixed, and only the dimensionality reduction matrix A and the dimensionality increase matrix B are trained. The input and output dimensions of the model remain unchanged, and the output is the superposition of BA and the parameters of the pre-trained large language model. A mean of 0 and a variance of δ are used. 2 The random Gaussian distribution N(0,δ) 2) initialization A, matrix B is initialized with 0 matrix. This can guarantee that when the training starts, the new path BA=0 has no effect on the model results. In inference, as long as the matrix product BA of the completed training is added to the original weight matrix W as a new weight parameter to replace the pre-trained large language model W.

[0026] For convenience of representation, the present application uses matrix D to represent the matrix product BA in the Lora fine-tuning technology, and the weight of the Lora block corresponding to each layer of the pre-trained large language model is represented as: ΔW i = D i ·W pre,i +b i , D i = B i A i , where ΔW i is the weight of the Lora block corresponding to the i-th layer of the pre-trained large language model, D i is the low-rank matrix of the Lora block ΔW i , the rank r i of the low-rank matrix D i = 2i, b i is the bias vector of the Lora block ΔW i , i is the index of the number of layers of the pre-trained large language model, A i is the dimension reduction matrix, and B i is the dimension increasing matrix.

[0027] S3, input the unnormalized electronic medical record text x k into the hierarchical strength confirmation model to obtain the predicted hierarchical strength G k of the text x k .

[0028] Specifically, the hierarchical strength confirmation model includes: a pre-trained Transformer encoder, an aggregation module, a medical dictionary auxiliary analysis module, and a gating module; the processing of the unnormalized electronic medical record text by the hierarchical strength confirmation model includes:

[0029] S31, input the unnormalized electronic medical record text x k into the pre-trained Transformer encoder to obtain a text embedding vector H k .

[0030] S32, input the text embedding vector H k into the aggregation module to obtain a global context vector F k .

[0031] The aggregation module is a MeanPooling module or an extraction of a [CLS] token, and the global context vector Fk carrying semantic information of the whole text context, including complexity and ambiguity characteristics.

[0032] S33, the non-normalized electronic medical record text x k Input medical dictionary auxiliary analysis module to obtain the number of medical terms of text x k The encoding vector N k of medical terms of text x k ;

[0033] In order to more accurately represent the complexity of the context, medical dictionary auxiliary analysis is introduced to count the number of medical terms appearing in the text. Specifically, the medical dictionary auxiliary analysis module processes the non-normalized electronic medical record text, including: obtaining a medical dictionary, the medical dictionary including a plurality of medical terms; matching the non-normalized electronic medical record text x k with the medical terms in the medical dictionary to obtain the number of medical terms of text x k , and encoding the number of medical terms of text x k to obtain the encoding vector N k .

[0034] The more medical terms appearing in the text, the higher the professional and complexity of the text content, thereby dynamically affecting the Lora insertion strategy.

[0035] S34, splicing the global context vector F k and the encoding vector N k of the number of medical terms to obtain the complexity feature vector C k ;

[0036] S35, inputting the complexity feature vector C k into the gate module to obtain the predicted hierarchical strength of text x k .

[0037] For complex text (high context complexity), Lora is inserted at a deep layer and the size of A matrix and b vector is increased; for simple text (low context complexity), Lora is inserted at a shallow layer and the size of A and b is reduced.

[0038] The specific method is: the gate module generates a set of gate coefficients for each layer according to the distribution characteristics of the context complexity feature vector C k = [g k,1 , g k,2 ,...g k,L ] = softmax(W g ·C k +b g ); wherein g k,i ∈ [0, 1] is the text x kThe predicted i-th layer of the pre-trained large language model inserts the strength of the Lora block, L is the number of layers of the pre-trained large language model, W g , b g is a gating weight, G k is a text x k The predicted pre-trained large language model inserts the hierarchical strength matrix of the Lora block, that is, the gating coefficient.

[0039] The formula realizes adaptive matching of model structure and task difficulty by dynamically mapping the context complexity and the Lora insertion position. Through the above formula, when the complexity feature vector C k has higher semantic complexity or professional term density (for example, high frequency of medical terms, C k The norm of the vector is large), the gating module allocates higher strength to the deep layer (close to the output) of the pre-trained large language model, that is, increases the insertion strength of the deep Lora and the scale of the low-rank matrix and the bias vector; when the complexity feature vector C k is low (for example, C k The norm represented is small or the professional term is sparse), the gating module allocates higher strength to the shallow layer (close to the input) of the pre-trained large language model, that is, increases the insertion strength of the shallow Lora, reduces the scale of the low-rank matrix and the bias vector, thereby improving the inference efficiency and saving the computing resources.

[0040] S4, according to the hierarchical strength G k The corresponding Lora block is inserted into each layer of the pre-trained large language model to obtain the large language model M lora inserted with the Lora block.

[0041] According to the hierarchical strength G k , each Lora block is inserted into the corresponding layer of the pre-trained large language model, which is represented as: W lora,i =W pre,i +g k,i ·ΔW i ; wherein, ΔW i is the i-th layer of the pre-trained large language model M pre , W pre,i is the weight of the i-th layer of the pre-trained large language model M pre , and W lora,i is the weight of the i-th layer of the large language model M lora inserted with the Lora block.

[0042] Through the above adaptive multi-layer Lora fine-tuning technology, the model is trained in a targeted manner, which can improve the accuracy of text normalization and reduce the amount of calculation.

[0043] S5, standardizing the unstandardized electronic medical record text x in the medical field k inputting the large language model of the inserted Lora block, to obtain the predicted text x k standardized electronic medical record text

[0044] S6, according to the predicted text x k standardized electronic medical record text and its corresponding standardized electronic medical record text y k calculating the loss function value, updating the parameters of the Lora block and the hierarchical strength confirmation model according to the loss function value, and obtaining the trained Lora block when the loss function value is the smallest;

[0045] updating the parameters of the Lora block and the hierarchical strength confirmation model according to the loss function value includes: freezing the parameters of the pre-trained large language model and the pre-trained Transform encoder, and updating the parameters D of the Lora block according to the loss function value i , b i and the parameters W of the gate module g , b g .

[0046] Preferably, the loss function is a cross-entropy loss function.

[0047] As shown in Figure 3 , during the training process, as the number of iterations increases, the loss Loss converges quickly, which improves the accuracy of standardization and shortens the training time.

[0048] As shown in Table 1, the effect of the fine-tuned large language model is better than that of the pre-trained (i.e., pre-trained) large language model.

[0049] Table 1: Performance of large language models before and after fine-tuning

[0050]

[0051] The above examples further illustrate the purpose, technical solutions and advantages of the present application. It should be understood that the above examples are only preferred embodiments of the present application and do not limit the present application. Any modification, equivalent replacement, improvement, etc. made to the present application within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A method for electronic medical record text normalization based on large model fine-tuning, characterized in that, include: Obtain unstandardized electronic medical record text x from the medical field, input text x into a trained large language model, and obtain standardized electronic medical record text; The training process of a large language model includes: S1, constructing a medical field specific corpus, the medical field specific corpus comprising un-normalized electronic medical record texts and normalized electronic medical record texts thereof ; wherein k represents an index of the un-normalized electronic medical record texts. S2. Obtain a pre-trained large language model and construct a LoRa block for each layer of the pre-trained large language model; construct a hierarchical strength confirmation model. S3, unnormalized electronic medical record text an input hierarchical strength confirmation model, to obtain text predicted hierarchical strength ; The hierarchical strength confirmation model comprises: a pre-trained encoder, an aggregation module, a medical dictionary assisted analysis module, and a gating module; the hierarchical strength confirmation model is used for confirming the unnormalized electronic medical record text The processing comprises: S31, normalizing the unnormalized electronic medical record text inputting the pre-trained encoder to obtain a text embedding vector; S32. Embed the text into the vector input aggregation module to obtain the global context vector; S33, normalizing the electronic medical record text inputting a medical dictionary assisted analysis module to obtain an encoding vector of the number of medical terms in the text of the medical record text S34. Concatenate the global context vector and the encoding vector of the number of medical terms to obtain the complexity feature vector; S35. Input the complexity feature vector into the gating module to obtain the text. Predicted level strength ; S4、 according to the hierarchical strength inserting a corresponding Lora block in each layer of the pre-trained large language model to obtain a large language model with inserted Lora blocks; Inserting corresponding LoRa blocks into each layer of the pre-trained large language model includes: ； in, Let be the weights of the Lora block corresponding to the i-th layer of the pre-trained large language model. The weights of the i-th layer of the pre-trained large language model. The weights of the i-th layer of the large language model into which LoRa blocks are inserted. , The strength of inserting Lora blocks into the i-th layer of the pre-trained large language model, where L is the number of layers in the pre-trained large language model; S5, unnormalized electronic medical record text Input the large language model inserted into the Lora block to obtain the predicted text Normalized electronic medical record text ; S6. Based on the predicted text Standardized electronic medical record text and its corresponding standardized electronic medical record text Calculate the loss function value, update the LoRa block and the layer strength to confirm the model parameters based on the loss function value, and obtain the trained LoRa block when the loss function value is minimized; S7. Insert the trained LoRa blocks into the corresponding layers of the pre-trained large language model to obtain the trained large language model.

2. The electronic medical record text normalization method based on large model fine-tuning according to claim 1, characterized in that, The weights of the LoRa blocks corresponding to each layer of the pre-trained large language model are represented as follows: ,in, This is a low-rank matrix of the Lora block. This is the bias vector for the Lora block.

3. The electronic medical record text normalization method based on large model fine-tuning according to claim 2, characterized in that, low-rank matrix rank .

4. The electronic medical record text normalization method based on large model fine-tuning according to claim 1, characterized in that, The aggregation module is an average pooling module.

5. The electronic medical record text normalization method based on large model fine-tuning according to claim 1, characterized in that, The medical dictionary-assisted analysis module analyzes unstandardized electronic medical record texts. The processing includes: obtaining a medical dictionary containing multiple medical terms; and converting unstandardized electronic medical record text. The text is obtained by matching it with medical terms in a medical dictionary. The number of medical terms in the text The number of medical terms is used to encode features, resulting in an encoding vector.

6. The electronic medical record text normalization method based on large model fine-tuning according to claim 1, characterized in that, The gating module processes the complexity feature vectors in the following ways: in, , For gating weights, For text The complexity of the feature vector, For text The predicted strength of the Lora block inserted at the i-th layer of the pre-trained large language model.

7. The electronic medical record text normalization method based on large model fine-tuning according to claim 1, characterized in that, Updating the parameters of the LoRa block and the hierarchical strength confirmation model based on the loss function value includes: freezing the parameters of the pre-trained large language model and the pre-trained encoder, and updating the parameters of the LoRa block and the gating module based on the loss function value.

8. The electronic medical record text normalization method based on large model fine-tuning according to claim 1, characterized in that, The loss function is the cross-entropy loss function.

Images