Gastroscope diagnosis and treatment report generation method and system based on fine-tuning multi-modal large model

Abstract

The invention provides a gastroscope diagnosis and treatment report generation method and system based on a fine-tuning multi-modal large model, and belongs to the technical field of medical image processing and natural processing, and the gastroscope diagnosis and treatment report generation method comprises the following steps: obtaining image text pair data of gastrointestinal tracts of a first-stage data set as a first-stage fine-tuning data set; performing first-stage fine tuning on a multi-modal large model by adopting the first-stage fine tuning data set to obtain a first-stage fine tuning model; acquiring image text pair data of the gastrointestinal tract of the second-stage data set, and performing templating processing to obtain a second-stage fine adjustment data set; performing second-stage fine tuning on the first-stage fine tuning model by adopting the second-stage fine tuning data set to obtain a second-stage fine tuning model; and inputting a gastroscope image to be diagnosed and treated into the second-stage fine tuning model, giving diagnosis and treatment problems to the second-stage fine tuning model, and generating a gastroscope diagnosis and treatment report. The effectiveness and accuracy of the gastroscope diagnosis and treatment report generated by the model are effectively improved.

Description

Technical Field

[0001] The present invention belongs to the technical fields of medical image processing and natural processing technology, and particularly relates to a gastroscopy diagnosis and treatment report generation method and system based on fine-tuning a multi-modal large model. Background Technique

[0002] The statements in this part only provide background technical information related to the present invention and do not necessarily constitute prior art.

[0003] Digestive endoscopy (referred to as endoscopy, Endoscopy) images are real-time images of the digestive tract (such as the esophagus, stomach, duodenum, colon, etc.) obtained through endoscope equipment, and play an important role in the screening, diagnosis, and follow-up of gastrointestinal diseases. Endoscopy images usually have high resolution and multiple perspectives, and can clearly show the subtle changes in the inner wall of the digestive tract, such as the location, shape, and color of lesions. Endoscopy images have rich color and texture information, and can reveal the blood vessel distribution, morphological characteristics of the mucosa, etc. In addition, the lighting conditions have a significant impact on the image quality, and the light source of the endoscope itself may cause uneven illumination of the image, so further image processing is required to improve its quality. Since the analysis of endoscopy images usually relies on the experience of professional doctors, the accurate annotation of the lesion area is usually completed by doctors, which also makes the cost of data acquisition and annotation relatively high.

[0004] A gastroscopy diagnosis and treatment report is a description and analysis of the examination results by a doctor after completing an endoscopy examination, and is an important basis for clinical decision-making. It usually includes patient information, examination process, lesion description, diagnosis conclusion, and doctor's suggestions, etc. The core content of the report mainly includes image description, diagnosis conclusion, and doctor's suggestions. Among them, the image description details the lesion conditions found during the examination process, including key features such as the location, size, shape, color, and boundary of the lesion, providing an intuitive basis for subsequent diagnosis. The diagnosis conclusion is the preliminary judgment made by the doctor based on image analysis, such as whether there are gastric polyps, ulcers, or other abnormal lesions. The doctor's suggestion part combines the examination results and proposes subsequent treatment plans, such as whether further biopsy, treatment measures, or regular follow-up are needed to ensure that patients receive timely and effective medical intervention.

[0005] Traditional gastroscopy diagnosis and treatment reports rely on doctors to manually write after observing endoscopy images manually, which not only consumes a lot of time and energy, but also the quality and accuracy of the report largely depend on the experience and professional level of doctors. Different doctors may have different judgments, resulting in difficulty in ensuring the consistency of the report. With the development of medical informatization, hospitals need to efficiently and accurately process and manage a large amount of medical data, including endoscopy examination data. Automatically generating gastroscopy diagnosis and treatment reports helps to improve the digital level of medical information, facilitate data storage, retrieval, and sharing, and provide richer data resources for medical research and clinical decision support systems.

[0006] Recently, the application of artificial intelligence in the medical field has become increasingly widespread, aiming to assist doctors in disease diagnosis and treatment decision-making, and improve the efficiency and quality of medical care. The method of generating gastroscopy diagnosis and treatment reports using multi-modal large models is an application of artificial intelligence in the field of medical endoscopy, and is expected to become a powerful assistant for doctors, helping doctors generate gastroscopy diagnosis and treatment reports faster and more accurately, and improving the diagnosis efficiency. Large models have powerful language understanding and generation capabilities, as well as feature extraction and analysis capabilities for data such as images. Through pre-training on large-scale data, large models have achieved remarkable results in tasks such as text generation and question-answering systems, providing technical reference for the generation of gastroscopy diagnosis and treatment reports.

[0007] However, there are the following technical problems in the existing technology: First of all, high-quality labeled endoscopic images are extremely scarce, which limits the application of data-driven methods in the medical field and also poses a huge challenge to the construction of specialized models with efficient diagnostic and analysis capabilities. Due to the difficulty in obtaining medical data, high annotation costs, and the need for professional medical knowledge, it is difficult to construct a high-quality endoscopic image dataset, thus affecting the training effect and generalization ability of the model.

[0008] Secondly, although multi-modal large models have demonstrated powerful understanding and reasoning capabilities in general tasks, there are still certain limitations in medical applications, especially in the understanding and accurate reasoning of specific medical domain knowledge. This is mainly because the training data of general large models are mainly text and images in the open domain, while the use of endoscopic data is relatively less, resulting in weak adaptability in the task of generating professional gastroscopy diagnosis and treatment reports.

[0009] In addition, directly training a multi-modal large model dedicated to medical tasks from scratch not only requires a large amount of high-quality data support, but also requires extremely high computing resources and training costs, which are unbearable for most medical institutions and research teams. Summary of the Invention

[0010] To overcome the above deficiencies of the existing technology, the present invention proposes a method and system for generating gastroscopy diagnosis and treatment reports based on fine-tuning a multi-modal large model. The present invention injects domain knowledge in the first stage and generates gastroscopy diagnosis and treatment reports in the second stage. Both stages of fine-tuning integrate the prompt-based LoRA fine-tuning strategy, which can enable the model to better capture domain-specific information while maintaining the generality of the pre-trained model.

[0011] To achieve a more ideal adaptation effect and deeply fit the task of generating gastroscopy diagnosis and treatment reports, the present invention plans a two-stage implementation strategy. In the first stage, given that the public digestive tract data has broad generality and certain domain representativeness, such data is used to conduct comprehensive and in-depth domain knowledge learning. Through systematic analysis and mining of a large amount of public digestive tract data, the model initially constructs a cognitive framework for knowledge related to digestive endoscopy, laying a solid foundation for subsequent fine optimization.

[0012] Entering the second stage, to achieve the goal of generating highly templatized language, the reports marked by a certain tertiary hospital in Shandong Province are selected as the key data resources. However, in the actual investigation, it is found that during the marking process by physicians, due to factors such as the high intensity of clinical work and individual language habit differences, some text descriptions have defects, such as being too colloquial, some sentence structures being chaotic, being unsmooth, and even having incorrect expressions in some places. In response to the above problems, the collected original marked reports are input into the GPT-4 model for language templatization processing. Using GPT-4 can standardize and rewrite colloquial expressions, sort out chaotic sentence structures, and correct misspelled words, thus forming templatized language. To ensure semantic invariance, a screening algorithm based on cosine distance is used to carefully screen the optimized sentences. This algorithm can effectively screen out high-quality sentences that meet the medical report standards and have consistent semantics by accurately calculating the semantic similarity between the sentences before and after optimization.

[0013] To achieve the above object, one or more embodiments of the present invention provide the following technical solutions: In the first aspect, a method for generating a gastroscopy diagnosis and treatment report based on fine-tuning a multi-modal large model is disclosed, including: Obtaining the image-text pair data of the gastrointestinal tract in the first-stage dataset as the first-stage fine-tuning dataset; Using the first-stage fine-tuning dataset to perform first-stage fine-tuning on the multi-modal large model to obtain a first-stage fine-tuned model; Obtaining the image-text pair data of the gastrointestinal tract in the second-stage dataset and performing templatization processing to obtain a second-stage fine-tuning dataset; Using the second-stage fine-tuning dataset to perform second-stage fine-tuning on the first-stage fine-tuned model to obtain a second-stage fine-tuned model; Inputting the gastroscopy image to be diagnosed and treated into the second-stage fine-tuned model and giving the second-stage fine-tuned model diagnosis and treatment questions to generate a gastroscopy diagnosis and treatment report.

[0014] In the second aspect, a system for generating a gastroscopy diagnosis and treatment report based on fine-tuning a multi-modal large model is disclosed, including: A data acquisition module, which is configured to: acquire image-text pair data of the gastrointestinal tract in the first-phase dataset as the first-phase fine-tuning dataset; A first-phase fine-tuning module, which is configured to: perform first-phase fine-tuning on the multi-modal large model using the first-phase fine-tuning dataset to obtain a first-phase fine-tuned model; A data templatization module, which is configured to: acquire image-text pair data of the gastrointestinal tract in the second-phase dataset and perform templatization processing to obtain a second-phase fine-tuning dataset; A second-phase fine-tuning module, which is configured to: perform second-phase fine-tuning on the first-phase fine-tuned model using the second-phase fine-tuning dataset to obtain a second-phase fine-tuned model; A gastroscopy diagnosis and treatment report generation module, which is configured to: input the gastroscopy image to be diagnosed and treated into the second-phase fine-tuned model, and give the second-phase fine-tuned model diagnosis and treatment questions to generate a gastroscopy diagnosis and treatment report.

[0015] In a third aspect, an electronic device is disclosed, including a memory, a processor, and computer instructions stored on the memory and running on the processor. When the computer instructions are run by the processor, the steps of the above-mentioned method for generating a gastroscopy diagnosis and treatment report based on fine-tuning a multi-modal large model are completed.

[0016] In a fourth aspect, a computer-readable storage medium is disclosed, which is used to store computer instructions. When the computer instructions are executed by a processor, the steps of the above-mentioned method for generating a gastroscopy diagnosis and treatment report based on fine-tuning a multi-modal large model are completed.

[0017] Compared with the prior art, the beneficial effects of the present invention are: The present invention proposes a method for templatizing gastroscopy diagnosis and treatment reports through GPT-4, which is beneficial for subsequent fine-tuning, solves the problems of chaotic manual annotation statements and word errors by manual physicians, and overcomes the problems of inconsistent traditional report formats, redundant information, and difficult extraction of key features. It can effectively improve the consistency and quality of data, thereby providing a more accurate fine-tuning basis for automatic report generation and diagnostic assistance systems based on large models.

[0018] The present invention uses a sentence screening method based on cosine distance to obtain texts with high language similarity after templatization, avoiding the deviation of sentence semantics and the noise problems caused by inconsistent descriptions and diverse language styles in the data.

[0019] The present invention proposes a two-stage fine-tuning method to adapt to the generation of gastroscopy diagnosis and treatment reports, overcoming the deficiencies of traditional single-stage fine-tuning methods in the medical field, such as insufficient generalization ability and limited understanding of professional knowledge. Through the pre-training of general medical knowledge in the first stage, the model's understanding of medical terms and expressions is enhanced; in the second stage, fine-tuning is combined with high-quality templated report data, enabling the model to generate gastroscopy diagnosis and treatment reports that more accurately meet clinical needs, improving the accuracy and consistency of report generation while reducing the dependence on large-scale labeled data.

[0020] The present invention proposes a LoRA fine-tuning method combined with prompts, overcoming the problems of insufficient adaptability, large training data requirements, and unstable generated text quality in traditional LoRA fine-tuning in medical scenarios. By introducing a prompt mechanism, the model can efficiently learn medical knowledge under low-resource conditions and better understand and generate gastroscopy diagnosis and treatment reports that meet clinical needs.

[0021] Advantages of additional aspects of the present invention will be given in part in the following description, become apparent in part from the following description, or be learned through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings forming a part of this specification are used to provide a further understanding of the present invention. The schematic embodiments and descriptions thereof of the present invention are used to explain the present invention and do not constitute an improper limitation of the present invention.

[0023] Figure 1 It is a structural diagram of the method for generating gastroscopy diagnosis and treatment reports based on fine-tuning a multi-modal large model described in Embodiment 1 of the present invention.

[0024] Figure 2 It is an example diagram of data templating and data screening described in Embodiment 1 of the present invention.

[0025] Figure 3 It is a structural diagram of the LoRA fine-tuning method based on prompts described in Embodiment 1 of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0026] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention belongs.

[0027] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention.

[0028] In the case of no conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other.

[0029] Example 1 In one or more embodiments, a method for generating a gastroscopy diagnosis and treatment report based on fine-tuning a multimodal large model is disclosed. As Figure 1 shown, it includes the following steps: Step S1: Obtain the image-text pair data of the gastrointestinal tract in the first-stage dataset as the first-stage fine-tuning dataset; The first-stage dataset is descriptive data related to the gastrointestinal tract, that is, image-text pair data, as the first-stage fine-tuning dataset. In this embodiment, a total of 6K image-text pair data are screened from the open-source medical datasets PubMed, MedTrinity-25M, and VQA-Med.

[0030] Step S2: Use the first-stage fine-tuning dataset to perform the first-stage fine-tuning on DeepSeek-VL-7B to obtain the first-stage fine-tuning model; In this embodiment, the multimodal large model uses DeepSeek-VL-7B as the base model and is trained using the LoRA fine-tuning method based on prompts to obtain the first-stage fine-tuning model with the ability to learn general gastroenterology knowledge.

[0031] The DeepSeek-VL-7B base model includes a visual encoder and a text encoder. The pre-trained visual encoder SigLIP is used to extract image features. For each input RGB image , where and represent the height and width of the image respectively. The visual encoder processes the input image to obtain the visual token sequence , where represents the sequence length of the visual tokens; the pre-trained text encoder DeepSeek LLM is used to process the text features to obtain the text token sequence , where represents the sequence length of the text tokens. Subsequently, the visual token sequence and the text token sequence are concatenated together. Then the concatenated sequence is input into DeepSeek-VL-7B to generate the language response , where represents the text response length.

[0032] Specifically, it includes the following detailed steps: Step S201: Input the image-text pair data into the base model DeepSeek-VL-7B. The visual encoder SigLIP extracts the image features, and the text encoder DeepSeek LLM obtains the text features.

[0033] Step S202: Freeze the visual encoder and only fine-tune the text encoder DeepSeek LLM to obtain the first-stage fine-tuned model. Specifically: Step S2021: Insert trainable first prompt parameters into each transformers layer of the text encoder of the base model. The trainable parameters are merged with the input sequence before each layer. Specifically, a set of special first prompt words is added to the front end of the input sequence of each transformers layer. The embedding parameters of these prompt words are learnable and are continuously adjusted during training to adapt to the task requirements, while ensuring that other pre-trained weights are frozen. The prompt words are soft prompts, i.e., Soft Prompt, also known as continuous prompts: It refers to guiding the model to generate text that meets specific requirements by inputting a parameterizable prompt template to the model. The parameters in the prompt template can be adjusted according to specific tasks and requirements to achieve the best generation effect. Using prompt words does not require explicitly specifying what each token in these templates is, but only needs to represent a vector in the semantic space.

[0034] Step S2022: Concatenate these trainable first prompt parameters with the first text sequence where the first text sequence is the text data in the image-text pair of the first-stage fine-tuning dataset, to form the extended first input sequence which is used as the input to each transformers layer of the text encoder DeepSeek LLM. This concatenation operation enables the model to access this additional prompt information during subsequent attention calculations by adding the prompt to the front of the input sequence. During self-attention calculation, the modified input sequence is used to calculate Query, Key, and Value, thereby integrating the information contained in the inserted trainable prompt into the attention mechanism. This method helps the attention layer better capture information relevant to the target task, and at the same time, the dimension of the output result remains the same as that of the original input sequence.

[0035] Step S2023: Apply the LoRA method in the first, middle, and last transformers layers of the text encoder DeepSeek LLM, as shown in (b) of Figure 3 while maintaining the merger of the input sequence and adjustable parameters in each layer. Specifically, at the first fully connected layer, the input dimension is expanded from to a larger dimension , that is, a low-rank matrix is inserted at the weight matrix of the fully connected layer. The original weight matrix that needs to be optimized , in the traditional fine-tuning method All the parameters in need to be updated, while LoRA proposes a more efficient way, that is, keeping unchanged and only adding a low-rank adjustment term , so that the fine-tuned weight matrix is as shown in formula (1).

[0036] (1) where is decomposed into low rank as , where , , is the rank of the decomposition, and it should satisfy .

[0037] Next, the intermediate result is non-linearly transformed through the ReLU activation function to further enhance the expression ability of the model. In the second fully connected layer, mapping the dimension back to the original input dimension , the more complex representation obtained after the non-linear transformation can be mapped back to the dimension consistent with the input, so as to be consistent with other parts of the model. The feed-forward layer can fuse the information learned by the non-linear transformation and the attention layer, retain the key context information, and make the representation of the whole model more expressive.

[0038] Step S3: Obtain the image-text pair data of the gastrointestinal tract in the second-stage dataset and perform data templatization to obtain the second-stage fine-tuning dataset; Step S301: Collect the image-text pairs of the gastrointestinal tract labeled by hospital physicians as the second-stage dataset.

[0039] Step S302: Use the GPT-4 model to perform text templatization on the second-stage dataset; Step S3021: Input the second-stage dataset into the GPT-4 model and give the GPT-4 positive information prompt: "You are an assistant who is happy to answer various questions. Your task is to templatize this sentence for the user. The template is: This picture shows (part), describing the features (color, texture, lesion, etc.)." Step S3022: Introduce a reliable semantic consistency judgment method, input the templatized text description into the Sentence-BERT model, obtain the sentence vector representation, and then screen the optimized sentence vectors of the model.

[0040] Suppose is the vector representation of the original sentence, is the vector representation of the templatized sentence, calculate the norms of the two sentence vectors and , that is and , where is the i-th component of the vector , and is the i-th component of the vector . is the number of components. Then, the similarity value sim between the original sentence and the optimized sentence is obtained, as shown in formula (2).

[0041] (2) During the screening process, the similarity value threshold is set to 0.85, and the generated texts are screened one by one. If some texts do not meet the set threshold, new sentences are regenerated until the requirements are met, thus achieving the goal of strengthening the data. As Figure 2 shown, the sentences before and after are compared.

[0042] The image and the optimized sentence vector are used as the dataset for the second-stage fine-tuning.

[0043] Through the above optimization and screening process, it is ensured that during the process of optimizing the sentence, a core principle is always followed, that is, the optimized sentence must be semantically consistent with the original sentence to ensure that the transmitted information does not shift or get lost.

[0044] Step S4: Use the second-stage fine-tuning dataset to perform the second-stage fine-tuning on the model fine-tuned in the first stage to obtain the second-stage fine-tuned model.

[0045] The second-stage fine-tuning is implemented based on the first stage. The second-stage fine-tuning dataset obtained in step S3 is used, and the method is similar to that of the first-stage fine-tuning.

[0046] Specifically, it includes the following detailed steps: Step S401: Input the second-stage fine-tuning dataset into the model fine-tuned in the first stage. The visual encoder SigLIP extracts the image features, and the text encoder DeepSeek LLM obtains the text features.

[0047] Step S402: Freeze the visual encoder and fine-tune the text encoder to obtain the second-stage fine-tuned model.

[0048] Step S4021: Insert trainable second prompt word parameters into each transformers layer of the text encoder in the first-stage fine-tuned model, as Figure 3 shown in (a) of . Specifically, a set of special second prompt words

[0049] Step S4022: Concatenate these trainable second prompt parameters with the second text sequence . The second text sequence is the text data in the image-text pairs of the second-stage fine-tuning dataset, forming an extended second input sequence , which is used as the input for each transformers layer of the text encoder DeepSeek LLM. When performing self-attention calculation, the modified input sequence is used to calculate Query, Key, and Value, thereby incorporating the information contained in the inserted trainable prompt into the attention mechanism.

[0050] Step S4023: Apply the LoRA method in the first, middle, and last transformers layers of the text encoder obtained after the first-stage fine-tuning, as shown in Figure 3 (b). At the same time, keep the combination of the input sequence and the tunable parameters in each layer. Specifically, in the first fully connected layer, expand the input dimension from to a larger dimension , that is, insert a low-rank matrix at the weight matrix of the fully connected layer. The original weight matrix that needs to be optimized , the parameters in in the traditional fine-tuning method all need to be updated, while LoRA proposes a more efficient way, that is, keep unchanged and only add a low-rank adjustment term , so that the fine-tuned weight matrix is as shown in formula (3).

[0051] (3) where is low-rank decomposed into , where , , is the rank of the decomposition and needs to satisfy .

[0052] Then, perform a non-linear transformation on the intermediate result through the ReLU activation function to further enhance the expressive power of the model. In the second fully connected layer, map the dimension back to the original input dimension , which can map the more complex representation obtained after the non-linear transformation back to the dimension consistent with the input, so as to be consistent with other parts of the model.

[0053] Step S5: Input the gastroscope image to be diagnosed and treated into the second-stage fine-tuning model, and give the second-stage fine-tuning model the diagnosis and treatment questions. The system calls the second-stage fine-tuning model completed by two-stage fine-tuning to generate a gastroscope diagnosis and treatment report.

[0054] Among them, the diagnosis and treatment questions are positive, active, and guiding questions, such as: "Carefully observe this endoscopic image. What significant features are presented in the image?"

[0055] Embodiment 2 In one or more embodiments, a gastroscopy diagnosis and treatment report generation system based on a fine-tuned multi-modal large model is disclosed, specifically including: A data acquisition module, which is configured to: acquire the image-text pair data of the gastrointestinal tract in the first-stage dataset as the first-stage fine-tuning dataset; A first-stage fine-tuning module, which is configured to: perform first-stage fine-tuning on the multi-modal large model using the first-stage fine-tuning dataset to obtain a first-stage fine-tuned model; A data templatization module, which is configured to: acquire the image-text pair data of the gastrointestinal tract in the second-stage dataset and perform templatization processing to obtain a second-stage fine-tuning dataset; A second-stage fine-tuning module, which is configured to: perform second-stage fine-tuning on the first-stage fine-tuned model using the second-stage fine-tuning dataset to obtain a second-stage fine-tuned model; A gastroscopy diagnosis and treatment report generation module, which is configured to: input the gastroscopy image to be diagnosed and treated into the second-stage fine-tuned model, and give the second-stage fine-tuned model diagnosis and treatment questions to generate a gastroscopy diagnosis and treatment report.

[0056] Embodiment 3 This embodiment provides an electronic device, including a memory, a processor, and computer instructions stored on the memory and running on the processor. When the computer instructions are run by the processor, the steps of the above-mentioned gastroscopy diagnosis and treatment report generation method based on a fine-tuned multi-modal large model are completed.

[0057] Embodiment 4 This embodiment provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the steps of the above-mentioned gastroscopy diagnosis and treatment report generation method based on a fine-tuned multi-modal large model are completed.

[0058] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, and the combination of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate for implementing in the process Figure 1 one process or multiple processes and / or blocks Figure 1Apparatus for the functions specified in one or more boxes.

[0059] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a particular manner, such that the instructions stored in the computer-readable memory produce a manufacture including an instruction device that implements the functions specified in one Figure 1 one or more processes and / or boxes Figure 1 or more boxes.

[0060] These computer program instructions may also be loaded onto a computer or other programmable data processing device to perform a series of operational steps on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one Figure 1 one or more processes and / or boxes Figure 1 or more boxes.

[0061] In the above embodiments, the descriptions of the various embodiments have their own emphases. For parts not detailed in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0062] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for generating gastroscopy diagnosis and treatment reports based on fine-tuning multi-modal large models, characterized in that, Including: Obtain the image-text pair data of the gastrointestinal tract in the first-stage dataset as the first-stage fine-tuning dataset; Use the first-stage fine-tuning dataset to perform the first-stage fine-tuning on the multi-modal large model to obtain the first-stage fine-tuned model; Obtain the image-text pair data of the gastrointestinal tract in the second-stage dataset and perform templating processing to obtain the second-stage fine-tuning dataset; Use the second-stage fine-tuning dataset to perform the second-stage fine-tuning on the first-stage fine-tuned model to obtain the second-stage fine-tuned model; Input the gastroscope image to be diagnosed and treated into the second-stage fine-tuned model, and give the diagnosis and treatment questions to the second-stage fine-tuned model to generate a gastroscope diagnosis and treatment report.

2. The gastroscopy diagnosis and treatment report generation method based on the fine-tuned multi-modal large model according to claim 1, characterized in that, The multi-modal large model includes a visual encoder and a text encoder. The pre-trained visual encoder extracts image features to obtain a visual token sequence, and the pre-trained text encoder processes text features to obtain a text token sequence. The visual token sequence and the text token sequence are concatenated and input into the multi-modal large model to generate a language response.

3. The endoscopic diagnosis and treatment report generation method based on fine-tuning a multi-modal large model according to claim 1, wherein, The specific first-stage fine-tuning is as follows: Insert a prompt word parameter at the front section of the first text sequence input to each transformers layer of the text encoder of the multi-modal large model to obtain an extended first input sequence, and the extended first input sequence is used for self-attention calculation.

4. The gastroscopy diagnosis and treatment report generation method based on fine-tuning a multi-modal large model according to claim 3, wherein, Insert a low-rank matrix at the weight matrix of the first fully-connected layer in the first, middle, and last transformers layers of the multi-modal large model to obtain the weight matrix of the fine-tuned transformers layer, expressed as: Among them, is the weight matrix after fine-tuning; is the low-rank adjustment term, which is decomposed into low rank as where , , is the rank of the decomposition; is the original weight matrix; In the second fully-connected layer, map the dimension back to the original input dimension.

5. The gastroscopy diagnosis and treatment report generation method based on fine-tuning a multi-modal large model according to claim 1, wherein, The specific templating processing is as follows: Input the second-stage dataset into the GPT-4 model and give information prompts to obtain templated language; Input the templated language into the Sentence-BERT model, obtain the optimized sentence vector representation of the model and perform screening; calculate the norm of the templated language and the optimized sentence vector of the model, calculate their similarity. If the similarity is lower than the preset threshold, regenerate the optimized sentence vector until the similarity value reaches the preset threshold to obtain the second-stage fine-tuning dataset.

6. The gastroscopy diagnosis and treatment report generation method based on fine-tuning a multi-modal large model according to claim 1, wherein, The specific second-stage fine-tuning is as follows: Insert a prompt word parameter at the front section of the second text sequence input to each transformers layer of the text encoder of the first-stage fine-tuned model to obtain an extended second input sequence, and the extended second input sequence is used for self-attention calculation.

7. The method for generating a gastroscopy diagnosis and treatment report based on a fine-tuned multi-modal large model according to claim 1, wherein, Insert a low-rank matrix at the weight matrix of the first fully-connected layer in the first, middle, and last transformers layers of the first-stage fine-tuned model to obtain the weight matrix of the fine-tuned transformers layer, expressed as: Among them, is the weight matrix after fine-tuning; is the low-rank adjustment term, which is decomposed into low rank as where , , is the rank of the decomposition; is the original weight matrix; In the second fully-connected layer, map the dimension back to the original input dimension.

8. A gastroscopy diagnosis and treatment report generation system based on fine-tuning a multimodal large model, characterized in that, Including: A data acquisition module, which is configured to: obtain the image-text pair data of the gastrointestinal tract in the first-stage dataset as the first-stage fine-tuning dataset; The first-stage fine-tuning module is configured to: perform first-stage fine-tuning on the multi-modal large model using the first-stage fine-tuning dataset to obtain a first-stage fine-tuned model; The data templatization module is configured to: obtain the image-text pair data of the gastrointestinal tract of the second-stage dataset and perform templatization processing to obtain a second-stage fine-tuning dataset; The second-stage fine-tuning module is configured to: perform second-stage fine-tuning on the first-stage fine-tuned model using the second-stage fine-tuning dataset to obtain a second-stage fine-tuned model; The gastroscopy diagnosis report generation module is configured to: input the gastroscopy image to be diagnosed into the second-stage fine-tuned model, and give the second-stage fine-tuned model diagnosis questions to generate a gastroscopy diagnosis report.

9. An electronic device, characterized in that, It includes a memory, a processor, and computer instructions stored on the memory and running on the processor. When the computer instructions are run by the processor, the method for generating a gastroscopy diagnosis report based on a fine-tuned multi-modal large model according to any one of claims 1-7 is completed.

10. A computer-readable storage medium, characterized in that, For storing computer instructions, when the computer instructions are executed by the processor, the method for generating a gastroscopy diagnosis report based on a fine-tuned multi-modal large model according to any one of claims 1-7 is completed.

Images