Auxiliary examination method of large language and vision assistant based on digestive endoscopy

Abstract

The invention discloses an auxiliary examination method of a large language and visual assistant based on a digestive endoscope, which comprises the following steps: acquiring original data of an endoscopic examination case, and cleaning and preprocessing the data; performing endoscope visual instruction data generation based on GPT assistance according to the processed original data to obtain an endoscope multi-mode instruction tracking data set; on the basis of a progressive transfer learning method, performing fine-tuning training on the constructed multi-modal large-scale language model to realize knowledge transfer of the multi-modal large-scale language model and the existing pre-training model to the endoscopic examination data, and meanwhile, avoiding disastrous forgetting; and finally, obtaining an endoscope multi-mode large language model, and performing evaluation based on GPT-4 and manual work. According to the method, excellent advantages can be shown in endoscopy, the problem of data shortage can be relieved, the inquiry intention can be accurately captured, medical terms can be accurately explained, reasoning is performed based on focus images, inquiry fuzziness is eliminated, and the application precedent of a multi-modal large-scale model in the field is created.

Description

Technical Field

[0001] The present invention relates to the technical field of endoscope examination, and in particular to an auxiliary examination method based on a large-scale language and visual assistant of a digestive endoscope. Background Art

[0002] Endoscopy plays a vital role in the diagnosis and treatment of gastrointestinal diseases, helping to detect abnormalities, classify lesions, and determine treatment. During gastrointestinal endoscopy, doctors may encounter some practical obstacles such as fatigue, stress, or limited experience, which may lead to errors in the examination results. To address these limitations, artificial intelligence (AI)-assisted endoscopy technology has emerged, which can provide higher accuracy, consistency, and speed, making the endoscopy process more efficient and higher quality.

[0003] Existing AI-assisted inspection methods rely on different visual models to complete various tasks independently, resulting in fixed interfaces, limited interactivity, and adaptability to user instructions. Recently, large language models such as GPT, Gemini, and LLaMA have reached or approached human levels in various language tasks, which has promoted the development of general large language models in the biomedical field. They are initialized using pre-trained large oracle models and fine-tuned according to customized biomedical instructions following datasets. The resulting large language models have shown great potential in various biomedical-related fields, such as helping doctors assist in diagnosis, understand patients' needs, and provide reasonable diagnosis and treatment recommendations.

[0004] The progress of large language models has prompted researchers to incorporate them as components or core elements of visual tasks into large visual-language models or multimodal large language models, such as LLaVA, GPT-4V, etc. These large models have shown amazing results in complex reasoning and multi-round dialogues on natural images. However, in related fields such as biomedicine, they cannot be directly used effectively due to the professionalism and inconsistency of images.

[0005] On the other hand, in biomedical and other related fields, a large amount of data exists in the form of images, especially endoscopic images, tissue sections and various medical images. These images carry rich biological information and are crucial for disease diagnosis, treatment and research. By combining the semantic understanding ability of the language model and the visual information of the image modality, the large language model can more accurately infer the characteristics of lesions in medical images and provide accurate diagnosis and treatment recommendations.

[0006] The development of multimodal large-scale models in the biomedical vertical is limited by the lack of large-scale systematic training data. Although some significant progress has been made, their applicability in the field of endoscopy remains challenging. This is mainly because the endoscopic image data that the visual part relies on is limited to acquisition under specific examination conditions, and its data distribution is significantly offset from natural images. On the other hand, the language model part faces the problem of lack of expertise specific to the field of endoscopy, which is difficult to directly extract from existing public data. Summary of the invention

[0007] The purpose of the present invention is to overcome the shortcomings of the prior art. To achieve the above purpose, an auxiliary inspection method based on a large-scale language and visual assistant of a digestive endoscope is adopted to solve the problems raised in the above background technology.

[0008] An auxiliary examination method based on large-scale language and visual assistants of digestive endoscopy, comprising the following steps:

[0009] Step S1, obtaining the original data of the endoscopic examination case, and performing data cleaning and preprocessing;

[0010] Step S2, generating data based on the GPT-assisted endoscope visual instructions according to the processed raw data to obtain a multimodal instruction tracking endoscope data set;

[0011] Step S3, constructing a multimodal large language model, wherein the framework of the model consists of a visual encoder, a large language model, and a trainable projection matrix; based on a progressive transfer learning method, adapting the multimodal large language model and the existing pre-trained model to the field of endoscopy;

[0012] Step S4: Based on the obtained multimodal large language model, GPT-4 is used to manually score the orderliness and accuracy of the model's answers to obtain the final model, and an endoscopic-assisted examination is performed.

[0013] 2. According to claim 1, a large-scale language and visual assistant-assisted inspection method based on digestive endoscopy is characterized in that the specific steps in step S1 include:

[0014] Obtain hospital endoscopy examination cases and collect raw data from real hospital endoscopy diagnosis scenarios;

[0015] The original data with privacy and ethical issues are cleaned, and the inspection conclusions are cleaned, and a corresponding inspection conclusion is assigned to each endoscopic inspection image.

[0016] 3. According to the large-scale language and visual assistant auxiliary examination method based on digestive endoscopy in claim 1, the specific steps in step S2 include:

[0017] Step S21, creating a set of questions about the image content, to enhance the ability of the large language model to describe the endoscope image content;

[0018] Step S22: For each endoscopic image, a question is randomly selected as a language instruction, and the answer is the image description and brief conclusion after cleaning in step S21, so as to construct a single-round dialogue dataset of the image and its corresponding description text;

[0019] Step S23: using the single-round question-answering about the endoscopic examination images generated in step S22, using the pure language large model GPT-3.5-turbo, generate a multi-round instruction follow-up data set, and inject medical common sense into the data set;

[0020] Step S24: Generate different multi-round dialogues for endoscopic data with lesions to alleviate model bias caused by unbalanced data sets.

[0021] 4. According to the large-scale language and visual assistant auxiliary inspection method based on digestive endoscopy in claim 3, the specific architecture of the multimodal large-scale language model constructed in step S3 includes:

[0022] The multimodal large language model consists of a visual encoder, a large language model, and a projection matrix;

[0023] A visual encoder using a pre-trained model of contrastive text-image pairs is used to process the input image X v , generate the corresponding visual feature Z v ;

[0024] Among them, the visual features before and after the last Transformer layer in the visual encoder use a simple trainable multi-layer perceptron to construct the projection matrix W and transform the visual features Z v Converted to an embedded token H aligned with the language model q , align the image to the word embedding space, denoted as H v =W·Z v ;

[0025] Generate target answer X a The probability calculation is denoted as p(X a |X v , X instruct ); At this time, X instruct is a combination term. When t=1, The instruction is defined as When t>1, it is defined as or

[0026] 5. According to the large-scale language and visual assistant auxiliary inspection method based on digestive endoscopy in claim 1, the specific steps of the progressive transfer learning in step S3 include:

[0027] Step S31, based on the progressive transfer learning method, in the first stage, LoRA is used in combination with the public biomedical dataset from LLaVA-Med to establish a strong biomedical baseline that contains medical common sense and expertise in certain specific biomedical fields, while understanding and generating coherent multimodal dialogues;

[0028] Step S32, constructing the endoscope command tracking data, and using the same LoRA method to fine-tune the model trained in the first stage in the vertical professional field of endoscopy;

[0029] Step S33: Fuse the LoRA weights and some untrained weights to obtain the final model weights.

[0030] 6. According to claim 1, a large-scale language and visual assistant-assisted inspection method based on digestive endoscopy, characterized in that the specific steps in step S4 include:

[0031] Step S41, randomly select several images from the validation set, and use the proposed data generation pipeline to generate detailed dialogues for the lesions;

[0032] The constructed Endoscopic Multi-Model Large Prediction Model predicts the answer based on the question and the visual input image; GPT-4 makes reference predictions based on the question and title as an upper bound for the teacher model;

[0033] After obtaining the responses from the two models, the question, visual information, and the responses generated by the two models are provided to GPT-4, which is asked to give an overall score;

[0034] Step S42: Perform manual scoring based on professional knowledge and experience, requiring high-quality and accurate evaluation of the orderliness, accuracy and relevance of the model questions and answers to obtain an overall score;

[0035] Step S43: average the scores of the two evaluation methods to obtain a final evaluation score.

[0036] Compared with the prior art, the present invention has the following technical effects:

[0037] The above technical solution was adopted to ensure the accuracy and availability of the data by acquiring and preprocessing the original case data. Subsequently, GPT was used to assist in the generation of a multimodal instruction tracking dataset and fine-tune a large language model to achieve accurate understanding and generation of complex endoscopic visual instructions. Through progressive transfer learning, this technology effectively integrates advanced models into the field of endoscopic examination, enhancing the adaptability and practicality of the model. Finally, strict evaluation and manual verification based on GPT-4 ensured the accuracy and reliability of the model, providing strong technical support for endoscopic assisted examination. This solution not only improves the efficiency of inspection, but also brings significant beneficial effects to medical diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings:

[0039] Figure 1 A schematic diagram of the steps of the auxiliary inspection method of the embodiment disclosed in this application;

[0040] Figure 2 A flowchart of an embodiment disclosed in this application;

[0041] Figure 3 This is a model framework diagram of the embodiment disclosed in this application. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] Please refer to Figure 1 and Figure 2 In an embodiment of the present invention, a large-scale language and visual assistant-assisted inspection method based on digestive endoscopy comprises the following steps:

[0044] Step S1: obtaining the original data of the endoscopic examination cases, and performing data cleaning and preprocessing. The specific steps include:

[0045] Obtain hospital endoscopy examination cases and collect raw data from real hospital endoscopy diagnosis scenarios;

[0046] The original data with privacy and ethical issues are cleaned, and the inspection conclusions are cleaned, and the corresponding inspection conclusions are assigned to each endoscopic inspection image.

[0047] In this embodiment, in order to solve the problem of severe lack of multimodal endoscopic data, a new data set was developed in this embodiment of the present invention. The data set consists of 3423 cases, each of which includes information such as patient gender, age, examination date, colonoscopy images and examination conclusions.

[0048] Endoscopic examination cases were obtained from hospitals. The original data was collected from real hospital endoscopic diagnosis scenarios and cleaned for privacy and ethical issues to ensure that the data used did not contain any patient personal information. There were only lesion images and reports with personal information removed. The examination conclusions were manually cleaned and each endoscopic examination image was assigned the corresponding examination findings.

[0049] Step S2: Generate data based on the processed raw data based on the GPT-assisted endoscope visual instructions to obtain a multimodal instruction tracking endoscope data set, and fine-tune the constructed multimodal large language model. The specific steps include:

[0050] Step S21: Create a set of questions about the image content to enhance the ability of the large language model to describe the content of the endoscopic image.

[0051] GPT-3.5-turbo was used to create a set of questions to enhance the ability of the large language model to describe the content of endoscopic images, such as: "Describe the image concisely", "Provide a short description of the given image", "Provide a concise explanation of the picture presented", "Present a concise summary of the image photograph that is clear and crisp", etc.

[0052] Step S22: For each endoscopic image, a question is randomly selected as a language instruction, and the answer is the image description and brief conclusion after cleaning in step S21, so as to construct a single-round dialogue dataset of the image and its corresponding description text;

[0053] For each image X v , a question is randomly selected as the language instruction, and the answer is the image description and brief conclusion after cleaning in step S21. Finally, the embodiment of the present invention constructs 14,030 images and their corresponding description texts. Although the data only provides a single task instruction, namely, image captioning, it contains diverse and representative samples of biomedical concepts from the original data.

[0054] Step S23: using the single-round question-answering about the endoscopic examination images generated in step S22, using the pure language large model GPT-3.5-turbo, generate a multi-round instruction follow-up data set, and inject medical common sense into the data set;

[0055] In this embodiment, in order to make the instruction-following model consistent, the embodiment of the present invention presents and manages various instruction-following data through the provided multi-round dialogues about biomedical images. Utilizing the single-round question and answer about endoscopic images generated in S3, a multi-round instruction-following dataset was generated using the pure language model GPT, while injecting medical common sense into the dataset. Specifically, based on the image description, instructions are designed in the prompts, requiring GPT-3.5-turbo to generate multi-round questions and answers in the tone of an assistant question and answer. To achieve the best results, the embodiment of the present invention manually curates a small number of samples in the design of instruction prompts to demonstrate how to generate high-quality dialogues based on the provided titles and context to create multi-round questions and answers. Where T is the total number of rounds, q and a are questions and answers respectively.

[0056] Step S24: Generate different multi-round dialogues for endoscopic data with lesions to alleviate model bias caused by unbalanced data sets.

[0057] It should be noted that, considering the significant imbalance between normal and abnormal endoscopic images, the embodiment of the present invention generates different multi-round dialogues for abnormal data, which helps to alleviate the model bias caused by unbalanced data sets.

[0058] Finally, by adapting the concept and implementing instruction tracing techniques, 17,504 speech-image instruction tracing samples and 64,640 dialogue samples were successfully compiled.

[0059] In this embodiment, the specific steps of constructing the multimodal large language model in step S3 include:

[0060] The multimodal large language model consists of a visual encoder, a large language model, and a projection matrix;

[0061] A visual encoder using a pre-trained model of contrastive text-image pairs is used to process the input image X v , generate the corresponding visual feature Z v ;

[0062] In this embodiment, the goal of the embodiment of the present invention is to fully utilize the capabilities of pre-trained large language models and visual models, and perform fine-tuning training in GPT-assisted endoscope visual instruction data.

[0063] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions:

[0064] like Figure 3As shown, an embodiment of the present invention demonstrates a multimodal large language model framework, which is composed of a visual encoder, a large language model, and a trainable projection matrix. During model training, only some parameters of the projection matrix and the large language model are updated, and all parameters of the visual encoder are completely frozen.

[0065] The visual encoder uses the pre-trained CLIP visual encoder ViT-L / 14, which is used to process the input image X v , generate the corresponding visual feature Z v =g(X V ).

[0066] Specifically, CLIP (Contrastive Language-Image Pre-training) is a pre-training model based on contrastive text-image pairs. This model maps image and text vector representations into a joint multimodal space and can learn the matching relationship between text-image pairs. The purpose of using the visual encoder of this model is to make the generated visual feature Z v Preliminary alignment of linguistic features in a high-dimensional joint multimodal space.

[0067] The selection of the large language model base is directly related to the effect of the visual large language model. In the embodiment of the present invention, LLAMA2 is used as the language model. LLAMA2 is a high-quality large language model open sourced by Meta. The context length is extended from 2048 of LLAMA to 4096. It can understand and generate longer texts. It includes three models of 7B, 13B and 70B. It performs outstandingly in the tests of various benchmark sets. Most importantly, the model can be used for research and commercial purposes. The LLAMA2 model performs better than the open source dialogue model on most benchmarks, and is a suitable alternative to the closed source model based on the usefulness and security evaluated by humans.

[0068] In order to match these visual features with the language model, the visual features before and after the last Transformer layer in the visual encoder are transformed into the visual feature Z using a simple trainable multilayer perceptron to construct the projection matrix W. v Converted to an embedded token H aligned with the language model q , align the image to the word embedding space, denoted as H v =W·Z v ;

[0069] Generate target answer X a The probability calculation is denoted as p(X a |X v , X instruct ); At this time, X instructis a combination term. When t=1, The instruction is defined as When t>1, it is defined as or

[0070] Step S3: construct a multimodal large language model, wherein the framework of the model consists of a visual encoder, a large language model, and a trainable projection matrix; based on a progressive transfer learning method, the multimodal large language model and the existing pre-trained model are adapted to the endoscopy process, and the specific steps include:

[0071] In this embodiment, in order to better learn the medical common sense and professional knowledge in the specific endoscopy field in large-scale medical data sets, the embodiment of the present invention proposes a novel progressive transfer learning method for adapting the above-mentioned model architecture and the existing pre-trained model to the endoscopy field. The method includes two stages, both based on LoRA technology, using two different data respectively. LoRA is a technology developed by Microsoft researchers to solve the fine-tuning of large language models. Its full name is Low-Rank Adaptation of Large Language Models, which is a low-rank adaptation of large language models. It is an efficient parameter adaptation method for adapting to specific tasks by introducing a small number of trainable parameters while keeping most of the pre-trained model parameters unchanged. The design goal of LoRA is to allow large pre-trained models (such as GPT-3, BERT, etc.) to achieve efficient fine-tuning and customization in downstream tasks with lower resource consumption. It should also be noted that in order to ensure optimal performance, in the training phase, the embodiment of the present invention does not perform any training on the visual encoder of the model.

[0072] Step S31, based on the progressive transfer learning method, in the first stage, LoRA is used in combination with the public biomedical dataset from LLaVA-Med to establish a strong biomedical baseline that contains medical common sense and expertise in certain specific biomedical fields, while understanding and generating coherent multimodal dialogues;

[0073] Step S32, constructing the endoscope command tracking data, and using the same LoRA method to fine-tune the model trained in the first stage in the vertical professional field of endoscopy;

[0074] Step S33: Fuse the LoRA weights and some untrained weights to obtain the final model weights.

[0075] Specifically, in the first stage, LoRA is combined with public biomedical datasets from LLaVA-Med to establish a strong biomedical baseline that contains medical common sense and expertise in certain specific biomedical fields, while being able to understand and generate coherent multimodal conversations. In the second stage, the embodiment of the present invention further uses the endoscope instruction tracking data we constructed, and uses the same LoRA method to fine-tune the model trained in the first stage in the vertical professional field of endoscopy, aiming to improve model performance by integrating data and domain expertise for specific endoscopy tasks, and finally fuses the LoRA weights and some untrained weights to obtain the LLaVA-Endo model weights.

[0076] Step S4: Based on the obtained multimodal large language model, GPT-4 is used to manually score the orderliness and accuracy of the model's answers to obtain the final model, and an endoscope-assisted examination is performed. The specific steps include:

[0077] Step S41, randomly select several images from the validation set, and use the proposed data generation pipeline to generate detailed dialogues for the lesions;

[0078] The constructed Endoscopic Multi-Model Large Prediction Model predicts the answer based on the question and the visual input image; GPT-4 makes reference predictions based on the question and title as an upper bound for the teacher model;

[0079] After obtaining the responses from the two models, the question, visual information, and the responses generated by the two models are provided to GPT-4, which is asked to give an overall score;

[0080] Step S42: Perform manual scoring based on professional knowledge and experience, requiring high-quality and accurate evaluation of the orderliness, accuracy and relevance of the model questions and answers to obtain an overall score;

[0081] Step S43: average the scores of the two evaluation methods to obtain a final evaluation score.

[0082] Specifically, in order to evaluate the performance of our proposed model, two evaluation methods are used in the embodiment of the present invention: evaluation based on GPT-4 and manual scoring. Specifically, this experiment randomly selects 20 images from the validation set and uses the proposed data generation pipeline to generate detailed dialogues for lesions. LLaVA-Endo predicts answers based on questions and visual input images. GPT-4 makes reference predictions based on questions and titles as the upper limit of the teacher model. After obtaining the responses of the two models, the questions, visual information, and responses generated by the two models are provided to GPT-4. GPT-4 is asked to evaluate the usefulness, relevance, accuracy, and detail of the model responses, and give an overall score of 1 to 10, where a higher score indicates better overall performance. In addition, GPT-4 is also asked to provide a comprehensive explanation of the evaluation so that we can better understand the model. On the other hand, this experiment invites three researchers to perform manual scoring based on their expertise and experience, requiring high-quality and accurate evaluations of the orderliness, accuracy, and relevance of the model's questions and answers, and also giving an overall score of 1 to 10. Finally, the scores combining the two evaluation methods are averaged to obtain the final evaluation score.

[0083] In this embodiment, the proposed model is compared with the above excellent models, and the test results are shown in Table 1:

[0084] Table 1 compares the SoTA model on the endoscopy test set. The best results are shown in bold, and the second best results are shown in underline;

[0085]

[0086]

[0087] It can be proved from Table 1 that the method proposed in the embodiment of the present invention shows advantages in the field of endoscopy, can effectively capture the intention of human questions, accurately interpret medical terms, can reason based on lesion images, and effectively solves the inherent ambiguity in endoscopic medical problems.

[0088] Beneficial effects of this embodiment:

[0089] The proposed model is compared with various existing high-quality multimodal large model methods. GPT-4V and Gemini are closed-source multimodal large language models developed by OpenAI and Google DeepMind, respectively, which have demonstrated state-of-the-art performance in many benchmark tests. LLaVA is an open-source multimodal large model jointly released by researchers from the University of Wisconsin-Madison, Microsoft Research, and Columbia University. It has published multiple parameterized versions and can be used to analyze and describe images and scenes. LLaVA-Med uses the same architecture and base model as LLaVA and is trained based on large-scale biomedical datasets in the public network, enabling the model to understand and process biomedical images. mPLUG-Owl is a multimodal large model based on modular implementation, which has outstanding performance in multimodal instruction understanding, multi-round dialogue, knowledge reasoning, etc. MiniGPTv2 establishes a unified input of visual and language multimodality to complete many visual-language tasks, including image description, visual question answering, and visual basic tasks.

[0090] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents, and all should be included within the scope of protection of the present invention.

Claims

1. An auxiliary examination method based on a large language and visual assistant of a digestive endoscope, comprising the following steps: Step S1, obtaining the original data of the endoscopic examination case, and performing data cleaning and preprocessing; Step S2, generating data based on the GPT-assisted endoscope visual instructions according to the processed raw data to obtain a multimodal instruction tracking endoscope data set; Step S3, constructing a multimodal large language model, wherein the framework of the model consists of a visual encoder, a large language model, and a trainable projection matrix; based on a progressive transfer learning method, adapting the multimodal large language model and the existing pre-trained model to the field of endoscopy; Step S4: Based on the obtained multimodal large language model, GPT-4 is used to manually score the orderliness and accuracy of the model's answers to obtain the final model, and an endoscopic-assisted examination is performed.

2. According to claim 1, a large-scale language and visual assistant-assisted inspection method based on digestive endoscopy is characterized in that: The specific steps in step S1 include: Obtain hospital endoscopy examination cases and collect raw data from real hospital endoscopy diagnosis scenarios; The original data with privacy and ethical issues are cleaned, and the inspection conclusions are cleaned, and a corresponding inspection conclusion is assigned to each endoscopic inspection image.

3. According to the large-scale language and visual assistant auxiliary examination method based on digestive endoscopy in claim 1, the specific steps in step S2 include: Step S21, creating a set of questions about the image content, so as to enhance the ability of the large language model to describe the endoscope image content; Step S22: For each endoscopic image, a question is randomly selected as a language instruction, and the answer is the image description and brief conclusion after cleaning in step S21, so as to construct a single-round dialogue dataset of the image and its corresponding description text; Step S23: using the single-round question-answering about the endoscopic examination images generated in step S22, using the pure language large model GPT-3.5-turbo, generate a multi-round instruction follow-up data set, and inject medical common sense into the data set; Step S24: Generate different multi-round dialogues for endoscopic data with lesions to alleviate model bias caused by unbalanced data sets.

4. According to the large-scale language and visual assistant auxiliary inspection method based on digestive endoscopy in claim 3, the specific architecture of the multimodal large-scale language model constructed in step S3 includes: The multimodal large language model consists of a visual encoder, a large language model, and a projection matrix; A visual encoder using a pre-trained model of contrastive text-image pairs is used to process the input image X v , generate the corresponding visual feature Z v ; Among them, the visual features before and after the last Transformer layer in the visual encoder use a simple trainable multi-layer perceptron to construct the projection matrix W and transform the visual features Z v Converted to an embedded token H aligned with the language model q , align the image to the word embedding space, denoted as H v =W·Z v ; Generate target answer X a The probability calculation is denoted as p(X a ∣X v ,X instruct ); At this time, X instruct is a combination term. When t=1, The instruction is defined as When t>1, it is defined as or 5. According to the large-scale language and visual assistant auxiliary inspection method based on digestive endoscopy in claim 1, the specific steps of the progressive transfer learning in step S3 include: Step S31, based on the progressive transfer learning method, in the first stage, LoRA is used in combination with public biomedical datasets from LLaVA-Med to establish a strong biomedical baseline that contains medical common sense and expertise in certain specific biomedical fields, while understanding and generating coherent multimodal dialogues; Step S32, using the constructed endoscope instruction tracking data, and using the same LoRA method to fine-tune the model trained in the first phase in the vertical professional field of endoscopy; Step S33: Fuse the LoRA weights and some untrained weights to obtain the final model weights.

6. The auxiliary inspection method based on large-scale language and visual assistant of digestive endoscopy according to claim 1 is characterized in that: The specific steps in step S4 include: Step S41, randomly select several images from the validation set, and use the proposed data generation pipeline to generate detailed dialogues for the lesions; The constructed Endoscopic Multi-Model Large Prediction Model predicts the answer based on the question and the visual input image; GPT-4 makes reference predictions based on the question and title as an upper bound for the teacher model; After obtaining the responses from the two models, the question, visual information, and the responses generated by the two models are provided to GPT-4, which is asked to give an overall score; Step S42: Perform manual scoring based on professional knowledge and experience, requiring high-quality and accurate evaluation of the orderliness, accuracy and relevance of the model questions and answers to obtain an overall score; Step S43: average the scores of the two evaluation methods to obtain a final evaluation score.

Images