Medical image report auxiliary generation method and device based on large language model
By constructing and fine-tuning the large language model, an assisted generation model suitable for medical imaging reports is generated, which solves the problems of low efficiency of image reporting generation and inconsistent reports in the prior art, and achieves efficient and consistent image reporting generation.
Patent Information
- Application Number
- CN202411113560.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-14
- Publication Date
- 2025-03-25
AI Technical Summary
In the prior art, the generation efficiency of medical imaging reports is low and reports written by different physicians are inconsistent.
Using a large language model-based approach, a medical large language model suitable for image report-assisted generation is generated by constructing medical corpus pre-trained data sets, large language model architecture design, model pre-training and image report instruction fine-tuning data sets. This model assists in generating the contents of the examination in the image report based on the patient's age, gender, examination items and diagnostic results.
This improves the efficiency of image report generation and avoids the problem of inconsistency in writing reports from different doctors.
Smart Images
Figure CN119694475A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of smart medical care and artificial intelligence, and in particular to a method for assisting the generation of medical imaging reports based on a large language model, and a device for assisting the generation of medical imaging reports based on a large language model. Background Art
[0002] The large language model is a deep learning model with a large number of parameters and strong learning ability. It is pre-trained with a large general corpus dataset and then fine-tuned with supervised data, thus having powerful capabilities in text generation.
[0003] Writing medical imaging reports is a very serious application scenario, and the authenticity and accuracy of each word generated are very high. Medical imaging reports mainly include three parts: patient information, examination findings, and diagnosis results. Examination findings are standardized medical descriptions of various disease signs, and the diagnosis result is the conclusion of this examination based on the examination findings. At present, medical imaging reports mainly rely on manual writing by radiologists, which is not only time-consuming and labor-intensive, but also lacks consistency in the reports written by different physicians. Summary of the invention
[0004] In order to overcome the defects of the prior art, the technical problem to be solved by the present invention is to provide a method for assisting the generation of medical imaging reports based on a large language model, which can obtain a medical large language model suitable for assisting the generation of imaging reports. The model assists in generating the examination findings in the imaging report according to information such as the patient's age, gender, examination items, and diagnosis results, thereby effectively improving the efficiency of imaging report generation and avoiding the problem of inconsistency in reports written by different doctors.
[0005] The technical solution of the present invention is: a method for auxiliary generation of medical imaging reports based on a large language model, which comprises the following steps:
[0006] (1) Construct a medical corpus pre-training dataset, filter the original medical text, remove the content that does not match the medical content, remove the redundant parts and desensitize the sensitive information to obtain a medical corpus pre-training dataset;
[0007] (2) Design a large language model architecture, design, build, and train a Transformer-based base model, or use an existing open source model as the base model;
[0008] (3) Pre-training the large language model: After the base model is determined, the words in the input medical data are replaced with special symbols, the replaced words are predicted by the base model, and compared with the real words, the loss value is calculated, and the parameters of the base model are updated until the base model converges or reaches the predetermined number of rounds, and finally the pre-trained medical large language model is obtained;
[0009] (4) Constructing an imaging report instruction fine-tuning dataset: The patient's gender, age, examination items, and diagnosis results are combined as the input part of the instruction fine-tuning dataset, and the examination findings are used as the output part of the instruction fine-tuning dataset to form the imaging report instruction fine-tuning dataset;
[0010] (5) Fine-tune the medical large language model using full parameter fine-tuning or fine-tuning based on a low-rank adaptation module;
[0011] (6) Deploy the fine-tuned medical language model in server / client mode or local deployment mode;
[0012] (7) Use the deployed medical big language model for reasoning: The doctor fills in the diagnosis result based on the image generated by the patient's imaging examination. The client combines the diagnosis result with the patient's gender, age, and examination items to infer the required information.
[0013] The present invention obtains a medical big language model suitable for assisting the generation of image reports through the construction of a medical corpus pre-training data set, the design of a big language model architecture, the pre-training of the model, the construction of an image report instruction fine-tuning data set, and the fine-tuning of a big language model based on the data set. The model assists in generating the examination findings in the image report according to information such as the patient's age, gender, examination items, and diagnosis results, thereby effectively improving the efficiency of image report generation and avoiding the problem of inconsistency in reports written by different doctors.
[0014] A medical imaging report auxiliary generation device based on a large language model is also provided, comprising:
[0015] The pre-training data set module is configured to construct a medical corpus pre-training data set, filter the original medical text, remove the content that does not match the medical content, remove the redundant parts and desensitize the sensitive information to obtain the medical corpus pre-training data set;
[0016] The architecture design module is configured to design a large language model architecture, design, build and train a Transformer-based base model, or use an existing open source model as a base model;
[0017] The large language model pre-training module is configured to pre-train the large language model: after the base model is determined, the words in the input medical data are replaced with special symbol markers, the replaced words are predicted by the base model, and compared with the real words, the loss value is calculated, and the parameters of the base model are updated until the base model converges or reaches the predetermined number of rounds, and finally the pre-trained medical large language model is obtained;
[0018] A fine-tuning dataset module is configured to construct an imaging report instruction fine-tuning dataset: the patient's gender, age, examination items and diagnosis results are combined as the input part of the instruction fine-tuning dataset, and the examination findings are used as the output part of the instruction fine-tuning dataset to form an imaging report instruction fine-tuning dataset;
[0019] A fine-tuning training module is configured to perform fine-tuning training on a large medical language model. The fine-tuning training uses full parameter fine-tuning or fine-tuning based on a low-rank adaptation module.
[0020] The deployment module is configured to deploy the fine-tuned medical large language model, and the deployment mode can be selected in server / client mode or local deployment mode;
[0021] The reasoning module is configured to use the deployed medical big language model for reasoning: the doctor fills in the diagnosis result based on the image generated by the patient's imaging examination, and the client combines the diagnosis result with the patient's gender, age, and examination items to infer the required information. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Figure 1 A flow chart of a method for assisting in generating a medical imaging report based on a large language model according to the present invention is shown.
[0023] Figure 2 A flowchart of step (1) of the method for assisting in generating a medical imaging report based on a large language model according to the present invention is shown.
[0024] Figure 3 A flowchart of step (2) of the method for auxiliary generation of medical imaging reports based on a large language model according to the present invention is shown.
[0025] Figure 4 A flowchart of step (3) of the method for auxiliary generation of medical imaging reports based on a large language model according to the present invention is shown.
[0026] Figure 5 A flowchart of step (4) of the method for auxiliary generation of medical imaging reports based on a large language model according to the present invention is shown.
[0027] Figure 6A flowchart of step (5) of the method for auxiliary generation of medical imaging reports based on a large language model according to the present invention is shown.
[0028] Figure 7 A flowchart of step (6) of the method for auxiliary generation of medical imaging reports based on a large language model according to the present invention is shown.
[0029] Figure 8 A flowchart of step (7) of the method for auxiliary generation of medical imaging reports based on a large language model according to the present invention is shown. DETAILED DESCRIPTION
[0030] like Figure 1 As shown, a method for assisting in generating a medical imaging report based on a large language model comprises the following steps:
[0031] (1) Construct a medical corpus pre-training dataset, filter the original medical text, remove the content that does not match the medical content, remove the redundant parts and desensitize the sensitive information to obtain a medical corpus pre-training dataset;
[0032] (2) Design a large language model architecture, design, build, and train a Transformer-based base model, or use an existing open source model as the base model;
[0033] (3) Pre-training the large language model: After the base model is determined, the words in the input medical data are replaced with special symbols, the replaced words are predicted by the base model, and compared with the real words, the loss value is calculated, and the parameters of the base model are updated until the base model converges or reaches the predetermined number of rounds, and finally the pre-trained medical large language model is obtained;
[0034] (4) Constructing an imaging report instruction fine-tuning dataset: The patient's gender, age, examination items, and diagnosis results are combined as the input part of the instruction fine-tuning dataset, and the examination findings are used as the output part of the instruction fine-tuning dataset to form the imaging report instruction fine-tuning dataset;
[0035] (5) Fine-tune the medical large language model using full parameter fine-tuning or fine-tuning based on a low-rank adaptation module;
[0036] (6) Deploy the fine-tuned medical language model in server / client mode or local deployment mode;
[0037] (7) Use the deployed medical big language model for reasoning: The doctor fills in the diagnosis result based on the image generated by the patient's imaging examination. The client combines the diagnosis result with the patient's gender, age, and examination items to infer the required information.
[0038] The present invention obtains a medical big language model suitable for assisting the generation of image reports through the construction of a medical corpus pre-training data set, the design of a big language model architecture, the pre-training of the model, the construction of an image report instruction fine-tuning data set, and the fine-tuning of a big language model based on the data set. The model assists in generating the examination findings in the image report according to information such as the patient's age, gender, examination items, and diagnosis results, thereby effectively improving the efficiency of image report generation and avoiding the problem of inconsistency in reports written by different doctors.
[0039] Preferably, if Figure 2 As shown, in step (1), public medical texts of Chinese and Western medicine textbooks, exercises, and medical expert guides are collected, and the public medical texts are filtered to remove non-medical content and duplicate content between different texts, such as directory, chapter, publisher, date, and other information. Sensitive information is desensitized to obtain a medical corpus pre-training data set. Identifiers are added at the beginning and end of each sentence to indicate the beginning and end of the sentence, as shown in Table 1:
[0040] Table 1
[0041]
[0042] Preferably, if Figure 3 As shown, in step (2), the Transformer is mainly composed of an encoder and a decoder, and different architectures are designed according to different requirements; when selecting an open source model, the applicability in the vertical field and the support level of computer hardware devices are considered.
[0043] Preferably, if Figure 4 As shown, in step (3), the commonly used English characters, numbers, symbols or continuous high-frequency Chinese characters in the medical corpus pre-training data set are combined into new subwords to form a dictionary, and the collected text is segmented according to the dictionary and mapped into a text vector, where the smallest text unit is called a token, as shown in Table 2:
[0044] Table 2
[0045]
[0046] The text vector is mapped according to the dimension of the basic large language model to obtain the input token set. The model randomly hides some tokens in the set, and then the large language model is calculated to output the result, which is compared with the real token, the loss value is calculated, and the model parameters are updated to maximize the likelihood function of the model until the model converges or reaches the predetermined number of rounds. Finally, the pre-trained medical large language model is obtained. The likelihood function L MBLM The calculation formula is:
[0047]
[0048] Where θ is the parameter of the model, X is the input token, X = {x1, x2, ..., x n}.
[0049] Preferably, if Figure 5 As shown, in step (4), the data set is divided into a training set, a validation set and a test set. The data is in the form of the following Table 3:
[0050] Table 3
[0051]
[0052] In the step (4), an image report instruction fine-tuning dataset is constructed, and the format of the image examination diagnosis report is shown in Table 4. The examination number, patient type, patient number, name, gender, age, department, ward, bed, examination date, equipment type and examination items are generated by the PACS system, and the main work of the physician is to fill in the examination findings and diagnosis results based on the image pictures. The examination findings are the standardized description of the patient's examination content in the image report, which is jointly determined by the examination items and the patient's gender, age and diagnosis results. The method and device for auxiliary generation of medical image reports based on a large language model provided by the present invention assists in generating the examination findings part of the report based on the patient's gender, age, examination items and diagnosis results. Therefore, in the process of constructing the dataset, the patient's gender, age, examination items and diagnosis results are combined as the input part of the instruction fine-tuning dataset, and the examination findings are used as the output part of the instruction fine-tuning dataset to form the image report instruction fine-tuning dataset.
[0053] Table 4
[0054]
[0055]
[0056] Preferably, if Figure 6 As shown, in step (5), when full parameter fine-tuning is adopted, the objective function of the standard language model is adopted as the objective function, the input part in the training set, including the patient's gender, age and examination findings, is input into the medical big language model, the corresponding examination findings are generated, and compared with the actual examination findings in the data set, the cross entropy loss value is calculated and all parameters of the medical big language model are updated until the model converges or reaches the predetermined number of rounds, thereby obtaining a fine-tuned medical big language model.
[0057] Preferably, in step (5), when fine-tuning based on the low-rank adaptation module is adopted, a low-rank adaptation matrix module is added next to the weight matrix of the pre-trained medical large language model as a trainable parameter, the objective function of the standard language model is adopted as the objective function, the input part in the training set, including the patient's gender, age and examination findings, is input into the medical large language model, the corresponding examination findings are generated, and compared with the actual examination findings in the data set, the cross-entropy loss value is calculated and the parameters of the low-rank adaptation module of the medical large language model are updated until the model converges or reaches a predetermined number of rounds, and finally the low-rank adaptation matrix and the pre-trained medical large language model are merged to obtain a fine-tuned medical large language model; wherein the low-rank adaptation matrix W MBLM The calculation formula is:
[0058] W MBLM =W θ +BA
[0059] Among them W θ is the weight matrix of the pre-trained medical language model, BA is a trainable parameter, and W MBLM ∈R d×k For the fine-tuned medical language model, B∈R d×r , A∈R r×k , low rank r<<min(d,k).
[0060] Preferably, if Figure 7 As shown, in the step (6), the floating-point data of the medical large language model is quantized into integers to speed up the reasoning speed. When the server / client mode is adopted, the trained medical large language model is loaded on the server side, and the client accesses the server side through the network to obtain the auxiliary generation result; when the local deployment mode is adopted, the medical large language model is loaded locally, and the client obtains the auxiliary generation result through local call.
[0061] Preferably, if Figure 8 As shown, in step (7), when the server / client mode is adopted, the merged data is sent to the server, and the server-side medical big language model assists in generating the content seen during the inspection and then returns it to the client; when the local deployment mode is adopted, the merged information is sent to the local medical big language model, and the local medical big language model assists in generating the content seen during the inspection and then returns it to the client.
[0062] Those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes the steps of the above-mentioned embodiment method, and the storage medium can be: ROM / RAM, disk, CD, memory card, etc. Therefore, corresponding to the method of the present invention, the present invention also includes a medical imaging report auxiliary generation device based on a large language model, which is usually represented in the form of functional modules corresponding to the steps of the method. The device includes:
[0063] The pre-training data set module is configured to construct a medical corpus pre-training data set, filter the original medical text, remove the content that does not match the medical content, remove the redundant parts and desensitize the sensitive information to obtain the medical corpus pre-training data set;
[0064] The architecture design module is configured to design a large language model architecture, design, build and train a Transformer-based base model, or use an existing open source model as a base model;
[0065] The large language model pre-training module is configured to pre-train the large language model: after the base model is determined, the words in the input medical data are replaced with special symbol markers, the replaced words are predicted by the base model, and compared with the real words, the loss value is calculated, and the parameters of the base model are updated until the base model converges or reaches the predetermined number of rounds, and finally the pre-trained medical large language model is obtained;
[0066] A fine-tuning dataset module is configured to construct an imaging report instruction fine-tuning dataset: the patient's gender, age, examination items and diagnosis results are combined as the input part of the instruction fine-tuning dataset, and the examination findings are used as the output part of the instruction fine-tuning dataset to form an imaging report instruction fine-tuning dataset;
[0067] A fine-tuning training module is configured to perform fine-tuning training on a large medical language model. The fine-tuning training uses full parameter fine-tuning or fine-tuning based on a low-rank adaptation module.
[0068] The deployment module is configured to deploy the fine-tuned medical large language model, and the deployment mode can be selected in server / client mode or local deployment mode;
[0069] The reasoning module is configured to use the deployed medical big language model for reasoning: the doctor fills in the diagnosis result based on the image generated by the patient's imaging examination, and the client combines the diagnosis result with the patient's gender, age, and examination items to infer the required information.
[0070] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any simple modification, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention are still within the protection scope of the technical solution of the present invention.
Claims
1. A method for assisting the generation of medical imaging reports based on a large language model, characterized in that: It includes the following steps: (1) Construct a pre-training dataset of medical corpus, filter the original medical text, and remove the content that does not match the medical content; Eliminate redundant parts and desensitize sensitive information to obtain a medical corpus pre-training dataset; (2) Design a large language model architecture, design, build, and train a Transformer-based base model, or use an existing open source model as the base model; (3) Pre-training the large language model: After the base model is determined, the words in the input medical data are replaced with special symbols, the replaced words are predicted by the base model, and compared with the real words, the loss value is calculated, and the parameters of the base model are updated until the base model converges or reaches the predetermined number of rounds, and finally the pre-trained medical large language model is obtained; (4) Constructing an imaging report instruction fine-tuning dataset: The patient's gender, age, examination items, and diagnosis results are combined as the input part of the instruction fine-tuning dataset, and the examination findings are used as the output part of the instruction fine-tuning dataset to form the imaging report instruction fine-tuning dataset; (5) Fine-tune the medical large language model using full parameter fine-tuning or fine-tuning based on a low-rank adaptation module; (6) Deploy the fine-tuned medical language model in server / client mode or local deployment mode; (7) Use the deployed medical big language model for reasoning: The doctor fills in the diagnosis result based on the image generated by the patient's imaging examination. The client combines the diagnosis result with the patient's gender, age, and examination items to infer the required information.
2. The method for assisting in generating medical imaging reports based on a large language model according to claim 1, characterized in that: In the step (1), publicly available medical texts of Chinese and Western medicine textbooks, exercises, and medical expert guidelines are collected, the publicly available medical texts are filtered, non-medical content is removed, duplicate content between different texts is removed, and sensitive information therein is desensitized to obtain a medical corpus pre-training data set.
3. The method for assisting in generating medical imaging reports based on a large language model according to claim 2, characterized in that: In step (2), the Transformer is mainly composed of an encoder and a decoder, and different architectures are designed according to different requirements. When selecting an open source model, the applicability in the vertical field and the support level of computer hardware devices are considered.
4. The method for assisting in generating medical imaging reports based on a large language model according to claim 3, characterized in that: In the step (3), commonly used English characters, numbers, symbols or continuous high-frequency Chinese characters in the medical corpus pre-training data set are combined into new subwords to form a dictionary, and the collected text is segmented according to the dictionary and mapped into a text vector, wherein the smallest text unit is called a token, and the text vector is mapped according to the basic large language model dimension to obtain an input token set. The model randomly hides some tokens in the set, and then the large language model is allowed to calculate the output result, compare it with the real token, calculate the loss value, and update the model parameters to maximize the model's likelihood function until the model converges or reaches a predetermined number of rounds, and finally obtain a pre-trained medical large language model, wherein the likelihood function L MBLM The calculation formula is: Where θ is the parameter of the model, X is the input token, X = {x1, x2, ..., x n }.
5. The method for assisting in generating medical imaging reports based on a large language model according to claim 4, characterized in that: In the step (4), the data set is divided into a training set, a validation set and a test set.
6. The method for assisting in generating medical imaging reports based on a large language model according to claim 5, characterized in that: In the step (5), when full parameter fine tuning is used, the objective function of the standard language model is used as the objective function, the input part in the training set, including the patient's gender, age and examination findings, is input into the medical large language model, the corresponding examination findings are generated, and compared with the actual examination findings in the data set, the cross entropy loss value is calculated and all parameters of the medical large language model are updated until the model converges or reaches a predetermined number of rounds, thereby obtaining a fine-tuned medical large language model.
7. The method for assisting in generating medical imaging reports based on a large language model according to claim 6, characterized in that: In the step (5), when fine-tuning based on the low-rank adaptation module is adopted, a low-rank adaptation matrix module is added next to the weight matrix of the pre-trained medical large language model as a trainable parameter, the objective function of the standard language model is adopted as the objective function, the input part in the training set, including the patient's gender, age and examination findings, is input into the medical large language model, the corresponding examination findings are generated, and compared with the actual examination findings in the data set, the cross-entropy loss value is calculated and the parameters of the low-rank adaptation module of the medical large language model are updated until the model converges or reaches a predetermined number of rounds, and finally the low-rank adaptation matrix and the pre-trained medical large language model are merged to obtain a fine-tuned medical large language model; wherein the low-rank adaptation matrix W MBLM The calculation formula is: W MBLM =W θ +BA Where W θ is the weight matrix of the pre-trained medical language model, BA is a trainable parameter, and W MBLM ∈R d×k For the fine-tuned medical language model, B∈R d×r , A∈R r×k , low rank r<<min(d,k).
8. The method for assisting in generating medical imaging reports based on a large language model according to claim 7, characterized in that: In the step (6), the floating-point data of the medical large language model is quantized into integers to speed up the reasoning speed. When the server / client mode is adopted, the trained medical large language model is loaded on the server side, and the client accesses the server side through the network to obtain the auxiliary generation result; when the local deployment mode is adopted, the medical large language model is loaded locally, and the client obtains the auxiliary generation result through local call.
9. The method for assisting in generating medical imaging reports based on a large language model according to claim 8, characterized in that: In the step (7), when the server / client mode is adopted, the merged data is sent to the server, and the server-side medical big language model assists in generating the inspection content and then returns it to the client; when the local deployment mode is adopted, the merged information is sent to the local medical big language model, and the local medical big language model assists in generating the inspection content and then returns it to the client.
10. A medical imaging report auxiliary generation device based on a large language model, characterized in that: It includes: The pre-training dataset module is configured to build a medical corpus pre-training dataset, filter the original medical text, and remove the content that does not match the medical content; Eliminate redundant parts and desensitize sensitive information to obtain a medical corpus pre-training dataset; The architecture design module is configured to design a large language model architecture, design, build and train a Transformer-based base model, or use an existing open source model as a base model; The large language model pre-training module is configured to pre-train the large language model: after the base model is determined, the words in the input medical data are replaced with special symbol markers, the replaced words are predicted by the base model, and compared with the real words, the loss value is calculated, and the parameters of the base model are updated until the base model converges or reaches the predetermined number of rounds, and finally the pre-trained medical large language model is obtained; A fine-tuning dataset module is configured to construct an imaging report instruction fine-tuning dataset: the patient's gender, age, examination items and diagnosis results are combined as the input part of the instruction fine-tuning dataset, and the examination findings are used as the output part of the instruction fine-tuning dataset to form an imaging report instruction fine-tuning dataset; A fine-tuning training module is configured to perform fine-tuning training on a large medical language model. The fine-tuning training uses full parameter fine-tuning or fine-tuning based on a low-rank adaptation module. The deployment module is configured to deploy the fine-tuned medical large language model, and the deployment mode can be selected in server / client mode or local deployment mode; The reasoning module is configured to use the deployed medical big language model for reasoning: the doctor fills in the diagnosis result based on the image generated by the patient's imaging examination, and the client combines the diagnosis result with the patient's gender, age, and examination items to infer the required information.