Method for automatically generating medical digital slice report based on mixed mode

Abstract

The invention relates to a method for automatically generating a medical digital slice report based on a mixed mode, which comprises the following steps of: acquiring a digital slice-report data set, processing a report part in the digital slice-report data set, and acquiring a digital slice-question and answer data set; the multi-modal model is trained; deploying the multi-modal model on a server; obtaining question and answer pairs for the digital slices; finely adjusting the large language model according to the question and answer pairs of the digital slices; and generating a medical diagnosis report according to the fine-tuned large model. The method has the beneficial effects that the method does not depend on a large number of template designs, but combines a template method and a neural network method, so that the model can process more types of medical slices and generate a targeted diagnosis report, and the flexibility and adaptability of the model are improved. Moreover, on the basis of automatically generating the diagnosis report by the deep neural network, the details of the generated report are controlled.

Description

Technical Field

[0001] The present invention relates to the field of medical imaging technology, and more specifically, to a method for automatically generating a medical digital slice report based on a hybrid mode. Background Art

[0002] It is not only time-consuming and laborious for pathologists to write reports based on stained medical slides, but also prone to errors. This results in a long period of time for generating diagnostic reports and a lack of accuracy. Therefore, artificial intelligence can effectively assist doctors in their diagnosis efficiency and detect some potential symptoms at the first time.

[0003] At present, commonly used pathology report generation models can be roughly divided into two categories: traditional template-based methods and deep neural network-based methods. The advantage of traditional template-based methods is that the template structure is clear and has a strong purpose for the disease, but the disadvantage is that a large number of templates need to be designed for different diseases. Most of the automatic report generation models based on deep learning only focus on the fluency of generated reports, but lack the evaluation of the accuracy of generated details. Summary of the invention

[0004] The purpose of the present invention is to address the deficiencies of the prior art and to propose a method for automatically generating a medical digital slice report based on a hybrid mode.

[0005] In a first aspect, a method for automatically generating a medical digital slice report based on a hybrid mode is provided, comprising:

[0006] S1. Obtain a digital slice-report dataset, process the report part in the digital slice-report dataset through a large language model, and obtain a digital slice-question-answering dataset;

[0007] S2. Training a multimodal model using the digital slice-question-answering dataset; the multimodal model includes a visual feature extractor, a region perception module, a text encoder, an attention gate structure, and a decoder;

[0008] S3, deploying the multimodal model on a server; repeatedly inputting different digital slices and corresponding questions, obtaining corresponding answers, and acquiring question-answer pairs for the digital slices;

[0009] S4. Fine-tune the large language model according to the question-answer pairs of the digital slices; and generate a medical diagnosis report according to the fine-tuned large model.

[0010] Preferably, S2 comprises:

[0011] S201, extracting features from the digital slice using a pre-trained visual feature extractor to obtain visual features I;

[0012] S202, the visual feature I is passed through the regional perception module to obtain enhanced visual information I h ;

[0013] S203, extracting the question part in the digital slice-question-answering dataset, and passing the question part through a text encoder to obtain a text feature T;

[0014] S204, text features T and enhanced visual information I h Through the attention gate structure, the fusion feature c is obtained;

[0015] S205: Put the fused feature c into the decoder to obtain the answer corresponding to the question part.

[0016] Preferably, in S2, the text encoder and decoder are Flan-T5 encoder and Flan-T5 decoder respectively.

[0017] Preferably, S4 includes:

[0018] S401, according to the report part of the digital slice-report dataset in S1, find the corresponding question-answer pair in the digital slice-question-answer dataset;

[0019] S402, writing the report part, the question-answer pair and the corresponding relationship between the two into a document;

[0020] S403, writing the question-answer pair obtained in S3 at the end of the document;

[0021] S404: Use the document as input to the large model to obtain a medical diagnosis report corresponding to the question-answer pair in S3.

[0022] In a second aspect, a medical digital slide report automatic generation system based on a hybrid mode is provided, which is used to execute any method described in the first aspect, including:

[0023] The first acquisition module is used to acquire a digital slice-report dataset, and process the report part in the digital slice-report dataset through a large language model to acquire a digital slice-question-answer dataset;

[0024] A training module, used to train a multimodal model using the digital slice-question-answering dataset; the multimodal model includes a visual feature extractor, a region perception module, a text encoder, an attention gate structure and a decoder;

[0025] The second acquisition module is used to deploy the multimodal model on the server; repeatedly input different digital slices and corresponding questions, obtain corresponding answers, and obtain question-answer pairs for the digital slices;

[0026] A generation module is used to fine-tune the large language model according to the collection of question-answer pairs of the digital slices; and generate a medical diagnosis report according to the fine-tuned large model.

[0027] According to a third aspect, a computer storage medium is provided, wherein a computer program is stored in the computer storage medium; when the computer program is executed on a computer, the computer executes any method described in the first aspect.

[0028] In a fourth aspect, an electronic device is provided, including:

[0029] Memory, used to store computer programs;

[0030] A processor is used to execute the computer program to implement any method as described in the first aspect.

[0031] The beneficial effects of the present invention are:

[0032] 1. Compared with the traditional template-based method, the present invention does not rely on a large number of template designs, but combines the template method with the neural network method, which enables the model to process more types of medical slices and generate targeted diagnostic reports, thereby improving the flexibility and adaptability of the model. In addition, based on the automatic generation of diagnostic reports by deep neural networks, the present invention has some control over the details of the generated reports.

[0033] 2. The present invention combines multiple technical means such as visual feature extraction, regional perception, text encoding and attention gate structure to ensure that the generated report is not only smooth but also accurate in details. Through the training and optimization of multimodal models, the pathological features in the slices can be captured more accurately and accurately reflected in the diagnosis report, reducing errors caused by human factors.

[0034] 3. The present invention greatly simplifies the diagnostic report process that originally required pathologists to write a time-consuming and laborious process through an automated method. By using a large language model and a multimodal model, digital slice information can be quickly processed and analyzed, and a corresponding diagnostic report can be generated, significantly shortening the time cycle for generating diagnostic reports. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A flowchart of a method for automatically generating a medical digital slice report based on a hybrid mode provided by the present invention;

[0036] Figure 2 A schematic diagram of a multimodal visual question answering model provided by the present invention;

[0037] Figure 3 A schematic diagram of the intra-group sensing structure provided by the present invention;

[0038] Figure 4A structural diagram of a gate control module provided by the present invention;

[0039] Figure 5 This is a schematic diagram of the structure of the medical digital slice report automatic generation system provided by the present invention. DETAILED DESCRIPTION

[0040] The present invention is further described below in conjunction with embodiments. The description of the following embodiments is only used to help understand the present invention. It should be noted that for ordinary persons in the art, without departing from the principle of the present invention, the present invention can also be modified in some ways, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

[0041] Embodiment 1:

[0042] To solve the problems of the prior art, such as Figure 1 As shown, Example 1 of the present application provides a method for automatically generating a medical digital slice report based on a hybrid mode, comprising:

[0043] S1. Obtain a digital slice-report dataset, process the report part in the digital slice-report dataset through a large language model, and obtain a digital slice-question and answer dataset.

[0044] Among them, the digital slice-report dataset includes an image part and a text part, the image part refers to the digital slice, the text part refers to the diagnostic report, and the slice corresponds to the diagnostic report. It should be noted that in the diagnostic report, there are usually some descriptions of cell morphology, size and other traits. This application uses the prompt words of the large language model to extract these trait details and obtain corresponding question-answer pairs. In addition, unlike the digital slice-report dataset, usually one digital slice corresponds to multiple question-answer pairs, thereby forming a digital slice-question-answer dataset.

[0045] S2. Use the digital slice-question-answering dataset to train the multimodal model. Figure 2 As shown, the multimodal model includes a visual feature extractor, a region perception module, a text encoder, an attention gate structure and a decoder. The text encoder and the decoder can be a Flan-T5 encoder and a Flan-T5 decoder, respectively.

[0046] Specifically, S2 includes:

[0047] S201, extracting features from the digital slice using a pre-trained visual feature extractor to obtain visual features I;

[0048] S202, the visual feature I is passed through the regional perception module to obtain enhanced visual information I h ;

[0049] S203, extracting the question part in the digital slice-question-answering dataset, and passing the question part through a text encoder to obtain a text feature T;

[0050] S204, text features T and enhanced visual information I h Through the attention gate structure, the fusion feature c is obtained;

[0051] S205: Put the fused feature c into the decoder to obtain the answer corresponding to the question part.

[0052] S3. Deploy the multimodal model on a server; repeatedly input different digital slices and corresponding questions, obtain corresponding answers, and acquire question-answer pairs for the digital slices.

[0053] S4. Fine-tune the large language model according to the question-answer pairs of the digital slices; and generate a medical diagnosis report according to the fine-tuned large model. It should be noted that the large language model is a commercial mature model (such as a GPT model).

[0054] Specifically, S4 includes:

[0055] S401. According to the report part of the digital slice-report dataset described in S1, find the corresponding question-answer pairs in the digital slice-question-answer dataset.

[0056] S402, writing the report part, the question-answer pair and the corresponding relationship between the two into a document.

[0057] For example, it is written into the document in the form of "question-answer pair => report".

[0058] S403. Write the question-answer pair obtained in S3 at the end of the document.

[0059] For example, in the last line of the document, the question-answer pair obtained from S3 is written into the document in the form of “question-answer pair=>?”.

[0060] S404: Use the document as input to the large model to obtain a medical diagnosis report corresponding to the question-answer pair in S3.

[0061] Embodiment 2:

[0062] Based on Example 1, Example 2 of the present application provides a more specific method for automatically generating a medical digital slice report based on a hybrid mode, including:

[0063] S1. Obtain a digital slice-report dataset, process the report part in the digital slice-report dataset through a large language model, and obtain a digital slice-question and answer dataset.

[0064] S2. Using the digital slice-question-answering dataset to train a multimodal model; the multimodal model includes a visual feature extractor, a region perception module, a text encoder, an attention gate structure and a decoder.

[0065] S2 includes:

[0066] S201, using a pre-trained visual feature extractor to extract features from the digital slice to obtain visual features I.

[0067] Specifically, in S201, the pre-trained model DINOv2 ViT Large is used as a visual extractor to extract features from the input slices and obtain visual features. where n and d i They refer to the length of the visual features and the dimension of the hidden layer, respectively. They are limited by the pre-trained model. i =1024.

[0068] S202, the visual feature I is passed through the regional perception module to obtain enhanced visual information I h .

[0069] Specifically, Figure 3 As shown, Where K represents the number of groups that divide the visual feature I. Since the length of the visual feature n may not completely divide K, it is necessary to fill the insufficient number of groups. in is the number of region groups. The padded visual features are then applied in the self-attention layer h = SelfAttention (W q p,W k p,W v p), where h represents the attention head, W q , W k , W v represents the learnable parameter. Finally, the h vector is operated so that And add the original visual feature I to obtain the enhanced visual information I h =I+h, and

[0070] S203: extract the question part in the digital slice-question-answering dataset, and pass the question part through a text encoder to obtain a text feature T.

[0071] Specifically, the encoder part of the pre-trained model Flan-T5 Large is used to train the input question Q = {w1,w2,…,w m}, where w represents the word and m represents the word length. Get the text features d t=1024 represents the hidden dimension of text features.

[0072] S204, text features T and enhanced visual information I h Through the attention gate structure, the fusion feature c is obtained.

[0073] Specifically, a gating module is used to fuse multimodal features, such as Figure 4 As shown. The gate control module is mainly composed of two parts. The mathematical representation of the first part, the cross-modal attention mechanism, is as follows:

[0074] M=CrossAttention(T,I h ,I h )

[0075] in represents the cross-modal feature, CrossAttention(·) represents the cross-modal attention mechanism, and the specific formula is as follows:

[0076]

[0077] Among them, Q represents query, K represents key, and K T represents the transpose of the key vector, V represents the value, and d k Indicates that the module receives dimensions.

[0078] The specific formula of the second part of the gating function is as follows:

[0079]

[0080] The fusion features Indicates averaging the text feature vectors. It means that the enhanced visual vector is averaged and then transposed, GeLU(·) is represented as the activation function, and its specific formula is as follows:

[0081]

[0082] S205: Put the fused feature c into the decoder to obtain the answer corresponding to the question part.

[0083] Specifically, the encoder of Flan-T5 Large is used as the initial weight. In the training stage, the answer data A and the fusion feature c obtained by S24 are used as input, where t represents the word data feature, and e represents the length. The prediction stage obtains the result And use cross entropy as the loss function loss = CrossEntropy(A,O), which is expressed as follows:

[0084]

[0085] Where l represents the label value and r represents the actual calculated value.

[0086] S3. Deploy the multimodal model on a server; repeatedly input different digital slices and corresponding questions, obtain corresponding answers, and acquire question-answer pairs for the digital slices.

[0087] S4. Fine-tune the large language model according to the question-answer pairs of the digital slices; and generate a medical diagnosis report according to the fine-tuned large model.

[0088] It should be noted that the parts in this embodiment that are the same or similar to those in Embodiment 1 can be referenced to each other and will not be described in detail in this application.

[0089] Embodiment 3:

[0090] Based on Examples 1 and 2, Example 3 of the present application provides a medical digital slice report automatic generation system based on a hybrid mode, such as Figure 5 As shown, including:

[0091] The first acquisition module is used to acquire a digital slice-report dataset, and process the report part in the digital slice-report dataset through a large language model to acquire a digital slice-question-answer dataset;

[0092] A training module, used to train a multimodal model using the digital slice-question-answering dataset; the multimodal model includes a visual feature extractor, a region perception module, a text encoder, an attention gate structure and a decoder;

[0093] The second acquisition module is used to deploy the multimodal model on the server; repeatedly input different digital slices and corresponding questions, obtain corresponding answers, and obtain question-answer pairs for the digital slices;

[0094] A generation module is used to fine-tune the large language model according to the question-answer pairs of the digital slices; and generate a medical diagnosis report according to the fine-tuned large model.

[0095] Specifically, the system provided in this embodiment is a system corresponding to the method provided in Embodiments 1 and 2. Therefore, the parts in this embodiment that are the same or similar to Embodiments 1 and 2 can be referenced to each other and will not be repeated in this application.

Claims

1. A method for automatically generating medical digital slice reports based on a hybrid mode, characterized in that: include: S1. Obtain a digital slice-report dataset, process the report part in the digital slice-report dataset through a large language model, and obtain a digital slice-question-answering dataset; S2. Training a multimodal model using the digital slice-question-answering dataset; the multimodal model includes a visual feature extractor, a region perception module, a text encoder, an attention gate structure, and a decoder; S3, deploying the multimodal model on a server; repeatedly inputting different digital slices and corresponding questions, obtaining corresponding answers, and acquiring question-answer pairs for the digital slices; S4. Fine-tuning the large language model according to the question-answer pairs of the digital slices; And generate a medical diagnosis report based on the fine-tuned large model.

2. The method for automatically generating medical digital slice reports based on hybrid mode according to claim 1, characterized in that S2 include: S201, extracting features from the digital slice using a pre-trained visual feature extractor to obtain visual features I; S202, the visual feature I is passed through the regional perception module to obtain enhanced visual information I h ; S203, extracting the question part in the digital slice-question-answering dataset, and passing the question part through a text encoder to obtain a text feature T; S204, text features T and enhanced visual information I h Through the attention gate structure, the fusion feature c is obtained; S205: Put the fused feature c into the decoder to obtain the answer corresponding to the question part.

3. The method for automatically generating medical digital slide reports based on hybrid mode according to claim 2, characterized in that: In S2, the text encoder and decoder are Flan-T5 encoder and Flan-T5 decoder respectively.

4. The method for automatically generating medical digital slide reports based on hybrid mode according to claim 3 is characterized in that: S4 include: S401, according to the report part of the digital slice-report dataset in S1, find the corresponding question-answer pair in the digital slice-question-answer dataset; S402, writing the report part, the question-answer pair and the corresponding relationship between the two into a document; S403, writing the question-answer pair obtained in S3 at the end of the document; S404: Use the document as input to the large model to obtain a medical diagnosis report corresponding to the question-answer pair in S3.

5. A medical digital slice report automatic generation system based on hybrid mode, characterized in that: Used to perform the method according to any one of claims 1 to 4, comprising: The first acquisition module is used to acquire a digital slice-report dataset, and process the report part in the digital slice-report dataset through a large language model to acquire a digital slice-question-answer dataset; A training module, used to train a multimodal model using the digital slice-question-answering dataset; the multimodal model includes a visual feature extractor, a region perception module, a text encoder, an attention gate structure and a decoder; The second acquisition module is used to deploy the multimodal model on the server; repeatedly input different digital slices and corresponding questions, obtain corresponding answers, and obtain question-answer pairs for the digital slices; A generation module is used to fine-tune the large language model according to the question-answer pairs of the digital slices; and generate a medical diagnosis report according to the fine-tuned large model.

6. A computer storage medium, characterized in that: The computer storage medium stores a computer program; when the computer program is executed on a computer, the computer executes the method for automatically generating a medical digital slide report based on a hybrid mode according to any one of claims 1 to 4.

7. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor is used to execute the computer program to implement the method for automatically generating a medical digital slice report based on a hybrid mode as described in any one of claims 1 to 4.

Images