Digestive endoscopy report automatic generation method based on multi-modal large language model

Abstract

The invention discloses a digestive endoscopy report automatic generation method based on a multi-modal large language model, and the method comprises the steps: obtaining a plurality of first endoscopy image groups and corresponding first report texts, and training the multi-modal large language model to obtain a target model; in a reasoning stage, inputting a second endoscopic image group into the target model, generating an anatomical part prediction result through an anatomical part classification head, and generating an abnormal score through an abnormal probability scoring head; and determining a representative image according to the anatomical site prediction result, and generating a corresponding site description and diagnosis conclusion based on the representative image to form a second report text. By introducing an anatomical part identification and anomaly scoring mechanism into the multi-modal model, automatic generation from multiple endoscopic images to a structured report text is realized, the report generation efficiency and consistency are improved, and the method is suitable for digestive endoscopy auxiliary diagnosis scenes.

Description

Technical Field

[0001] This application relates to the field of automatic generation technology of digestive endoscopy reports based on multimodal large language models, and in particular to a method for automatic generation of digestive endoscopy reports based on multimodal large language models. Background Technology

[0002] Digestive endoscopy is a crucial tool for screening and diagnosing gastrointestinal tumors, playing a fundamental role in clinical diagnosis, follow-up management, and quality control. With the promotion of large-scale screening programs and the widespread adoption of high-throughput testing, the number of endoscopic examinations continues to grow. Whether the results can be used to generate high-quality, traceable endoscopic reports that can be used for downstream clinical decision-making directly impacts key aspects such as treatment strategy formulation, monitoring interval scheduling, and quality control auditing. Therefore, the automatic generation and standardized drafting of reports tailored to digestive endoscopy scenarios are becoming an urgent need to improve the efficiency of endoscopic diagnosis and treatment and the consistency of reports.

[0003] In related technologies, clinical practice typically involves endoscopists manually saving endoscopic images during examinations and then writing reports based on their findings after the examination. Due to the continuously increasing workload and relatively limited expert resources, report quality is prone to fluctuations, manifesting as vague descriptions and omissions of key pathological findings. This weakens the crucial link between lesion detection, standardized recording, and appropriate treatment, increasing potential clinical risks. Meanwhile, the development of multimodal large language models has driven research into medical text generation, but existing applications largely focus on generating text descriptions for pre-selected static images, primarily addressing the single-task problem of generating descriptions from given input. However, the digestive endoscopy reporting workflow is essentially a more complex dual-task scenario: on one hand, it requires selecting evidence-based keyframes from hundreds of highly redundant endoscopic images manually saved during the examination, covering key anatomical landmarks to reflect the completeness of the examination and presenting clear visual evidence of lesions; on the other hand, it requires generating a rigorously structured report text, providing separate descriptions of findings for different anatomical sites, and maintaining strict semantic consistency with the selected evidence-based image content. These differences make it difficult for existing models and methods based on a single-task paradigm to simultaneously meet the clinical needs of representative image selection and structured report generation.

[0004] Therefore, in the automatic generation of gastrointestinal endoscopy reports, the large number and high redundancy of images make it difficult to select representative keyframes, the report text needs to be structured and maintain strict semantic consistency with the evidence images, and the report quality is prone to fluctuation under high workload conditions, resulting in vague descriptions and omissions of key findings. These are problems that urgently need to be solved. Summary of the Invention

[0005] This application provides an automatic generation method for digestive endoscopy reports based on a multimodal large language model. It aims to solve the problems in the automatic generation of digestive endoscopy reports in the prior art, such as the large number and high redundancy of images leading to difficulties in selecting representative keyframes, the need for structured report text to maintain strict semantic consistency with evidence images, and the easy fluctuation of report quality under high workload conditions, with vague descriptions and omissions of key observations.

[0006] Firstly, a method for automatically generating digestive endoscopy reports based on a multimodal large language model, the method comprising: Acquire multiple first endoscopic image groups and a first report text corresponding to each first endoscopic image group. The first report text includes multiple site descriptions and a diagnostic conclusion. The first endoscopic image group is training image data collected during historical endoscopic examinations. Based on multiple sets of the first endoscopic images and multiple sets of the first report texts, the initial model is trained to obtain the target model; the initial model is a multimodal large language model. The second set of endoscopic images is input into the target model. The anatomical location prediction result is generated by the anatomical location classification head of the target model, and the abnormality score is generated by the abnormality probability scoring head of the target model. The second set of endoscopic images is the endoscopic examination image data for which the report is to be generated. Based on the predicted anatomical locations, representative images are determined; The representative images are input into the target model to generate corresponding site descriptions and diagnostic conclusions, thereby forming the second report text corresponding to the second endoscopic image group.

[0007] Optionally, in the above scheme, before training the initial model based on multiple first endoscopic image groups and the first report text corresponding to each first endoscopic image group to obtain the target model, the method further includes: Text cleaning is performed on multiple first report texts to obtain cleaned first report texts; the text cleaning includes deleting blank text, deleting incomplete text, and deleting text that does not conform to the preset report format; The multiple part descriptions in each of the first cleaned report texts are split according to anatomical parts to obtain the split part descriptions; Based on the segmented part descriptions, the corresponding anatomical part categories are determined; Generate normal or abnormal labels for each of the split parts; The diagnostic conclusions in the first report text after cleaning are subjected to terminology standardization processing to make the diagnostic conclusions conform to a preset set of medical terms.

[0008] Optionally, in the above scheme, before training the initial model based on multiple first endoscopic image groups and the first report text corresponding to each first endoscopic image group to obtain the target model, the method further includes: Image quality detection is performed on multiple first endoscopic images of the first endoscopic image group; the image quality detection includes sharpness detection, brightness detection, and glare detection. Based on preset clarity thresholds, brightness thresholds, and glare determination rules, multiple first endoscope images are screened, and first endoscope images that do not meet preset image quality conditions are removed.

[0009] Optionally, in the above scheme, the step of training the initial model based on multiple first endoscopic image groups and the first report text corresponding to each first endoscopic image group to obtain the target model includes: Acquire the first endoscopic image labeled with anatomical location as a training sample for the anatomical location; The anatomical site classification model is trained based on the training samples of the anatomical sites. Obtain the first endoscopic image corresponding to the normal label or abnormal label as an abnormal training sample; An anomaly recognition model is trained based on the aforementioned anomaly training samples.

[0010] Optionally, in the above scheme, the step of training the initial model based on multiple first endoscopic image groups and the first report text corresponding to each first endoscopic image group to obtain the target model includes: The anatomical location classification model is used to infer the first endoscopic image to obtain the predicted anatomical location label for each of the first endoscopic images. The anomaly detection model is used to infer the anomaly from the first endoscopic image to obtain the anomaly prediction result for each of the first endoscopic images. The first endoscopic image is divided into multiple anatomical site image sets based on the anatomical site prediction labels; Abnormal images are filtered from each anatomical site image set based on the abnormality prediction results.

[0011] Optionally, in the above scheme, the step of training the initial model based on multiple first endoscopic image groups and the first report text corresponding to each first endoscopic image group to obtain the target model includes: A first prompt word is generated based on the pairing result between the set of anatomical region images and the segmented region descriptions; A second prompt word is generated based on the pairing result between the abnormal image and the diagnostic conclusion; The first endoscopic image or the abnormal image and the corresponding prompt words are input into a pre-trained multimodal large language model to generate the corresponding thought chain; The first endoscopic image or the abnormal image, the corresponding first prompt word, the second prompt word, the thought chain, and the corresponding split part description or diagnostic conclusion are combined into a quadruple training data.

[0012] Optionally, in the above scheme, the step of training the initial model based on multiple first endoscopic image groups and the first report text corresponding to each first endoscopic image group to obtain the target model includes: Visual features are extracted from the first endoscope image in the first endoscope image group using a visual encoder. Add positional encoding to image features in multiple visual Transformer blocks; Visual features of multiple first endoscopic images in the same first endoscopic image group are stitched together to form a comprehensive visual feature sequence; Add a start marker Token at the beginning position and an end marker Token at the end position of the integrated visual feature sequence; The comprehensive visual feature sequence is compressed through a visual language connection layer, and the compressed visual features are projected onto the feature dimension space of the autoregressive language model. The prompt words are encoded using a text embedding layer to obtain text features, and the visual features and text features are then input into the autoregressive language model.

[0013] Optionally, in the above scheme, the step of training the initial model based on multiple first endoscopic image groups and the first report text corresponding to each first endoscopic image group to obtain the target model includes: During the instruction fine-tuning training process, a causal masking mechanism is used to calculate the cross-entropy loss for the output parts corresponding to the thought chain and the report text. In reinforcement learning training, a group relative strategy optimization algorithm is used to update the model parameters, and a medical keyword reward function and a format structure reward function are set as custom reward functions.

[0014] Optionally, in the above scheme, determining representative images based on the anatomical site prediction results includes: The anatomical site classification head and the anomaly probability scoring head are connected to the last layer output of the visual encoder; The anatomical site classification head is trained based on anatomical site label data; The anomaly probability scoring head is trained based on normal or abnormal labels; Freeze other parameters of the target model when training the anatomical site classification head and the anomaly probability scoring head; During the inference phase, multiple sets of anatomical region images are constructed based on the predicted anatomical region results. The images of each anatomical site are sorted according to the anomaly score. Images whose abnormal scores meet preset conditions are selected as representative images.

[0015] In a second aspect, a computer-readable storage medium stores a computer program that, when executed by a relevant processor, implements the method described in the first aspect.

[0016] Compared with the prior art, this application has at least the following beneficial effects: Based on further analysis and research of existing technical problems, this application recognizes that in the automatic generation of digestive endoscopy reports, the large number and high redundancy of images make it difficult to select representative keyframes; the report text needs to be structured and maintain strict semantic consistency with the evidence images; and under high workload conditions, the report quality is prone to fluctuations, resulting in vague descriptions and omissions of key findings. This application addresses these issues by acquiring multiple first endoscopic image groups and their corresponding first report texts, and then training a multimodal large language model. This enables the model to establish a mapping relationship between endoscopic images and structured medical report texts, thereby allowing for automatic analysis of second endoscopic image groups during the inference phase. This analysis is combined with the anatomical location prediction results output by the anatomical location classification head and the abnormality probability scoring head. The system generates anomaly scores, categorizes images by anatomical structure, and quantifies the degree of abnormality. Based on this, representative images are identified, and corresponding site descriptions and diagnostic conclusions are generated, thus forming a complete second report text. Since the training phase has enabled the model to learn the correspondence between image features and report structure, and the inference phase uses anatomical site recognition and anomaly scoring mechanisms to perform structured screening and information focusing on images, it can avoid the redundancy and structural chaos caused by generating text indiscriminately for all images. This achieves automatic conversion from multiple endoscopic images to structured report text, thereby solving the problems of low efficiency, large subjective differences, and difficulty in ensuring report structure consistency caused by relying on manual image review and manual report writing in existing technologies. Attached Figure Description

[0017] Figure 1 A flowchart illustrating an embodiment of the automatic generation method for digestive endoscopy reports based on a multimodal large language model provided in this application; Figure 2 A schematic diagram illustrating the steps of an automatic generation method for digestive endoscopy reports based on a multimodal large language model, provided in one embodiment of this application; Figure 3A diagram illustrating the size model collaborative processing framework of a disclosed embodiment of this application is provided. Figure 4 This is an overall model structure diagram provided for one embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0019] In one embodiment, such as Figure 1 As shown, an automatic generation method for digestive endoscopy reports based on a multimodal large language model is provided, including the following steps: Acquire multiple first endoscopic image groups and a first report text corresponding to each first endoscopic image group. The first report text includes multiple site descriptions and a diagnostic conclusion. The first endoscopic image group is training image data collected during historical endoscopic examinations. Based on multiple sets of the first endoscopic images and multiple sets of the first report texts, the initial model is trained to obtain the target model; the initial model is a multimodal large language model. The second set of endoscopic images is input into the target model. The anatomical location prediction result is generated by the anatomical location classification head of the target model, and the abnormality score is generated by the abnormality probability scoring head of the target model. The second set of endoscopic images is the endoscopic examination image data for which the report is to be generated. Based on the predicted anatomical locations, representative images are determined; The representative images are input into the target model to generate corresponding site descriptions and diagnostic conclusions, thereby forming the second report text corresponding to the second endoscopic image group.

[0020] This embodiment provides a method for automatically generating digestive endoscopy reports based on a multimodal large language model. This method is applicable to the automatic analysis of multiple endoscopic images acquired during digestive tract endoscopy examinations and generates structured medical report text based on the image content.

[0021] During the training phase, multiple first endoscopic image sets are first acquired. Each first endoscopic image set corresponds to multiple endoscopic images acquired from the same patient during the same endoscopic examination. These images may include images of different anatomical locations, such as the esophagus, gastric body, gastric antrum, or duodenum. To ensure data consistency, the images are uniformly resized and normalized to meet the size requirements of subsequent model inputs.

[0022] Each set of first endoscopic images corresponds to a first report text. The first report text includes multiple site descriptions and a diagnostic conclusion. The site descriptions detail the observations of different anatomical sites, while the diagnostic conclusion provides a medical assessment of the overall examination results. A one-to-one correspondence is established between the first set of endoscopic images and the first report text to construct supervised training samples.

[0023] Subsequently, an initial model was trained based on multiple sets of first endoscopic images and their corresponding first report texts. This initial model is a multimodal large language model capable of processing both visual and textual information simultaneously. During training, multiple images from the first endoscopic image sets were input into a visual encoder to extract visual features, which were then mapped to the feature space of the language model through a visual-language connection layer. Simultaneously, the first report text was segmented and encoded, and input together with the visual features into an autoregressive language model for joint modeling. By calculating the loss function between the generated text and the actual report text, the model parameters were updated to obtain the target model.

[0024] During the inference phase, the second set of endoscopic images is input into the target model. The target model first extracts visual features from each image using a visual encoder and outputs the corresponding anatomical location prediction result through an anatomical location classification head. Simultaneously, an anomaly score value for each image is output through an anomaly probability scoring head. Based on the anatomical location prediction results, the images are grouped according to anatomical locations, and representative images are selected for each anatomical location based on the anomaly score. The representative image can be the image with the highest anomaly score or an image that meets a preset anomaly threshold.

[0025] Finally, the representative images are input into the target model, which generates corresponding site descriptions and diagnostic conclusions, and combines them to form a second report text corresponding to the second endoscopic image group.

[0026] This embodiment combines a multimodal large language model with an anatomical site recognition mechanism and an anomaly scoring mechanism, and introduces a representative image screening strategy in the generation stage to achieve automatic generation of structured medical reports from multiple endoscopic images, thereby improving report generation efficiency and consistency of medical expression.

[0027] In this embodiment, before training the initial model based on multiple first endoscopic image groups and the first report text corresponding to each first endoscopic image group to obtain the target model, the method further includes: Text cleaning is performed on multiple first report texts to obtain cleaned first report texts; the text cleaning includes deleting blank text, deleting incomplete text, and deleting text that does not conform to the preset report format; The multiple part descriptions in each of the first cleaned report texts are split according to anatomical parts to obtain the split part descriptions; Based on the segmented part descriptions, the corresponding anatomical part categories are determined; Generate normal or abnormal labels for each of the split parts; The diagnostic conclusions in the first report text after cleaning are subjected to terminology standardization processing to make the diagnostic conclusions conform to a preset set of medical terms.

[0028] Before training the initial model, multiple first report texts undergo text cleaning. This text cleaning includes removing blank text, incomplete text, and text that does not conform to a preset report format. The preset report format may include a fixed structural template, such as a standard structure containing a part description paragraph and a diagnostic conclusion paragraph. Through this processing, the report texts used for training are ensured to have a uniform format and complete content.

[0029] After text cleaning is completed, the descriptions of multiple body parts in each cleaned first report text are split according to anatomical location. Specifically, a set of keywords for anatomical locations can be pre-established, and the corresponding descriptions can be extracted using keyword matching. For descriptions that do not explicitly identify anatomical locations, they can be completed or discarded based on context rules.

[0030] Subsequently, based on the segmented site descriptions, the corresponding anatomical site categories are determined, and a normal or abnormal label is generated for each segmented site description. These labels can be determined through rule-based matching, such as judging based on whether they contain lesion keywords.

[0031] In addition, the diagnostic conclusions in the cleaned first report text undergo terminology standardization to conform to a predefined set of medical terms. Terminology standardization can be achieved through a predefined mapping dictionary to avoid inconsistencies in labeling caused by synonyms.

[0032] This embodiment improves the consistency and structure of training data by cleaning, splitting, and standardizing the report text, thereby reducing the impact of data noise on model training and improving the model training effect.

[0033] In this embodiment, before training the initial model based on multiple first endoscopic image groups and the first report text corresponding to each first endoscopic image group to obtain the target model, the method further includes: Image quality detection is performed on multiple first endoscopic images of the first endoscopic image group; the image quality detection includes sharpness detection, brightness detection, and glare detection. Based on preset clarity thresholds, brightness thresholds, and glare determination rules, multiple first endoscope images are screened, and first endoscope images that do not meet preset image quality conditions are removed.

[0034] Before training the initial model, image quality checks are performed on the first endoscopic images from multiple sets of first endoscopic images. Image quality checks include sharpness detection, brightness detection, and glare detection. Sharpness detection is based on the degree of gradient change in the image; if the image is too blurry, it is considered an unacceptable sample. Brightness detection is based on the overall grayscale statistics of the image to avoid images that are too dark or too bright affecting model training. Glare detection determines the presence of strong reflective interference by identifying the proportion of bright areas.

[0035] Based on preset sharpness thresholds, brightness thresholds, and glare determination rules, the first endoscopic images are screened, and images that do not meet the preset image quality conditions are removed. The screened images are used for subsequent model training.

[0036] This embodiment reduces the adverse effects of low-quality images on model training by performing quality control on image samples, thereby improving the stability of model training and the reliability of generated results.

[0037] In this embodiment, the step of training the initial model based on multiple first endoscopic image groups and the first report text corresponding to each first endoscopic image group to obtain the target model includes: Acquire the first endoscopic image labeled with anatomical location as a training sample for the anatomical location; The anatomical site classification model is trained based on the training samples of the anatomical sites. Obtain the first endoscopic image corresponding to the normal label or abnormal label as an abnormal training sample; An anomaly recognition model is trained based on the aforementioned anomaly training samples.

[0038] After generating text labels and filtering image quality, a first endoscopic image with anatomical location category labels and a first endoscopic image with normal or abnormal labels can be obtained. Based on the above annotation results, an anatomical location classification model is first constructed. This model is used to identify the anatomical location category to which the endoscopic image belongs, such as the esophagus, gastric body, gastric antrum, or duodenum. This classification model can employ a convolutional neural network structure or a visual Transformer structure, and is trained through supervised learning, enabling the model to output the corresponding anatomical location category probability distribution based on image features.

[0039] Simultaneously, an anomaly detection model is constructed. This model is used to determine whether an image contains abnormal lesion features, such as erosion, ulcers, or polyps. The anomaly detection model can employ a binary or multi-class classification structure, outputting an anomaly probability score. By training the anomaly detection model using the aforementioned generated normal or abnormal labels, the model can learn lesion feature patterns in the image.

[0040] The two models described above can be trained independently or share some visual encoder parameters. In practice, the anatomical site classification model and the anomaly recognition model can be trained separately first, and then their results can be used in the subsequent automatic annotation stage.

[0041] This embodiment achieves automatic structured annotation of endoscopic images by constructing an anatomical site classification model and an anomaly recognition model, providing a structured image information foundation for multimodal model training and improving the relevance and accuracy of subsequent report generation.

[0042] In this embodiment, the step of training the initial model based on multiple first endoscopic image groups and the first report text corresponding to each first endoscopic image group to obtain the target model includes: The anatomical location classification model is used to infer the first endoscopic image to obtain the predicted anatomical location label for each of the first endoscopic images. The anomaly detection model is used to infer the anomaly from the first endoscopic image to obtain the anomaly prediction result for each of the first endoscopic images. The first endoscopic image is divided into multiple anatomical site image sets based on the anatomical site prediction labels; Abnormal images are filtered from each anatomical site image set based on the abnormality prediction results.

[0043] After the anatomical location classification model and the anomaly recognition model are trained, the anatomical location classification model is used to infer the anatomical location of the first endoscopic image to obtain the predicted anatomical location label for each first endoscopic image. Simultaneously, the anomaly recognition model is used to infer the anomaly of the first endoscopic image to obtain the anomaly prediction result or anomaly probability score for each image.

[0044] Based on the predicted anatomical location labels, the first endoscopic image is divided into multiple anatomical location image sets. Each set corresponds to an anatomical location category. Subsequently, within each anatomical location image set, abnormal images are filtered based on the anomaly prediction results. The filtering method can be to select images with anomaly probabilities higher than a preset threshold, or to select a few images with the highest anomaly probabilities.

[0045] Through the above processing, the original image data can be automatically organized into a structured image set, and abnormal images under each part can be clearly identified.

[0046] This embodiment achieves automatic structured organization of training samples through automatic annotation and image set construction mechanisms, reducing manual intervention and improving data processing efficiency and annotation consistency.

[0047] In this embodiment, the step of training the initial model based on multiple first endoscopic image groups and the first report text corresponding to each first endoscopic image group to obtain the target model includes: A first prompt word is generated based on the pairing result between the set of anatomical region images and the segmented region descriptions; A second prompt word is generated based on the pairing result between the abnormal image and the diagnostic conclusion; The first endoscopic image or the abnormal image and the corresponding prompt words are input into a pre-trained multimodal large language model to generate the corresponding thought chain; The first endoscopic image or the abnormal image, the corresponding first prompt word, the second prompt word, the thought chain, and the corresponding split part description or diagnostic conclusion are combined into a quadruple training data.

[0048] After dividing the anatomical region image sets, each image set is paired with its corresponding segmented region description, ensuring each image set has clear textual descriptive information. Simultaneously, the filtered abnormal images are paired with the diagnostic conclusions in the cleaned first report text, thus establishing a correspondence between images and diagnostic conclusions.

[0049] Based on the above pairing relationships, prompt words can be generated. These prompt words are used to guide the multimodal large language model in generating target text content. Prompt words may include anatomical location information, task instruction information, and a description of the generated target. For example, prompt text containing "Please generate a description of this location based on the image" can be constructed.

[0050] Subsequently, the first endoscopic image or abnormal image and its corresponding prompt words are input into a pre-trained multimodal large language model, which generates a thought chain text. The thought chain is the intermediate reasoning process text before the model generates the report text, used to enhance the model's reasoning ability.

[0051] Finally, the images, prompts, thought chains, and corresponding segmented part descriptions or diagnostic conclusions are combined into quadruples of training data for subsequent model training.

[0052] This embodiment constructs training data containing a chain of thought, enabling the model to learn intermediate reasoning logic during training, thereby improving the logical coherence and medical expression of the generated report.

[0053] In this embodiment, the step of training the initial model based on multiple first endoscopic image groups and the first report text corresponding to each first endoscopic image group to obtain the target model includes: Visual features are extracted from the first endoscope image in the first endoscope image group using a visual encoder. Add positional encoding to image features in multiple visual Transformer blocks; Visual features of multiple first endoscopic images in the same first endoscopic image group are stitched together to form a comprehensive visual feature sequence; Add a start marker Token at the beginning position and an end marker Token at the end position of the integrated visual feature sequence; The comprehensive visual feature sequence is compressed through a visual language connection layer, and the compressed visual features are projected onto the feature dimension space of the autoregressive language model. The text embedding layer encodes the prompt words to obtain text features, and the visual features and text features are then input into the autoregressive language model. The text embedding layer is used to segment and encode the prompt words and perform vector mapping, outputting corresponding text features. These text features and the visual features output by the visual-language connection layer are jointly input into the autoregressive language model for joint modeling.

[0054] The initial model includes a visual encoder, a visual-language connection layer, a text embedding layer, and an autoregressive language model. The visual encoder is used to extract visual features from the first endoscopic image in the first endoscopic image group. The visual encoder can employ a visual Transformer structure, dividing the image into multiple fixed-size image blocks and generating a feature vector for each image block. Within the multiple visual Transformer blocks, positional encoding is added to the image features to preserve the spatial location information of the image blocks.

[0055] For multiple first endoscopic images in the same first endoscopic image group, their visual features can be stitched together in a fixed order to form a comprehensive visual feature sequence. A start marker Token is added at the beginning position and an end marker Token is added at the end position of the comprehensive visual feature sequence to identify the boundaries of the image sequence.

[0056] The visual feature sequence is compressed and projected through a visual-language connective layer to match the feature dimension of the autoregressive language model. Subsequently, the cue words are encoded through a text embedding layer, and the visual and text features are then jointly input into the autoregressive language model for modeling.

[0057] This embodiment achieves unified modeling of multiple endoscopic images and text information through cross-modal feature compression and fusion mechanisms, thereby improving the model's comprehensive understanding of multi-view images.

[0058] In this embodiment, training an initial model based on multiple first endoscopic image groups and the first report text corresponding to each first endoscopic image group to obtain a target model includes: During the instruction fine-tuning training process, a causal masking mechanism is used to calculate the cross-entropy loss for the output parts corresponding to the thought chain and the report text. In reinforcement learning training, a group relative strategy optimization algorithm is used to update the model parameters, and a medical keyword reward function and a format structure reward function are set as custom reward functions.

[0059] This embodiment further illustrates the two-stage mechanism of model training.

[0060] In the first phase, fine-tuning training of the instructions is performed. A causal masking mechanism is used during training, ensuring that the model relies only on previous token information when generating the current token. The loss function is calculated only for the output parts corresponding to the thought chain and report text, thus avoiding unnecessary optimization interference to the prompt words.

[0061] In the second phase, reinforcement learning training is conducted. A group-relative policy optimization algorithm is used to update the model parameters. During reinforcement learning, a medical keyword reward function and a format structure reward function are introduced. The medical keyword reward function encourages the model to generate text containing key medical terms, while the format structure reward function encourages the model to generate text conforming to a preset report structure.

[0062] Through the above two-stage training mechanism, the model can not only have accurate medical semantic expression capabilities, but also generate structurally standardized report texts.

[0063] This embodiment improves the medical accuracy and format consistency of the model-generated text by combining instruction fine-tuning with reinforcement learning training strategies.

[0064] In this embodiment, determining representative images based on the anatomical site prediction results includes: The anatomical site classification head and the anomaly probability scoring head are connected to the last layer output of the visual encoder; The anatomical site classification head is trained based on anatomical site label data; The anomaly probability scoring head is trained based on normal or abnormal labels; Freeze other parameters of the target model when training the anatomical site classification head and the anomaly probability scoring head; During the inference phase, multiple sets of anatomical region images are constructed based on the predicted anatomical region results. The images of each anatomical site are sorted according to the anomaly score. Images whose abnormal scores meet preset conditions are selected as representative images.

[0065] The final output layer of the visual encoder is connected to an anatomical site classification head and an anomaly probability scoring head. The anatomical site classification head outputs the probability distribution of anatomical site categories, and the anomaly probability scoring head outputs the anomaly score.

[0066] When training the anatomical site classification head and the anomaly probability scoring head, other parameters of the target model can be frozen, and only the classification head parameters can be updated, thereby avoiding excessive perturbation to the main model structure.

[0067] During the inference phase, multiple anatomical region image sets are constructed based on the anatomical region prediction results, and each set is sorted according to anomaly scores. Images whose anomaly scores meet preset criteria are selected as representative images.

[0068] This embodiment uses a dual-classification head structure and parameter freezing strategy to implement an image optimization mechanism, thereby improving the accuracy of report generation for abnormal areas.

[0069] Figure 2 This is a flowchart illustrating the specific steps of the automatic generation method for digestive endoscopy reports based on a multimodal large language model in this embodiment, which includes the following steps: S1. Obtain all endoscopic images collected during the endoscopic examination and their corresponding report text, including site descriptions and diagnostic conclusions, with some endoscopic images having anatomical site labels; S2. A semi-automatic data cleaning, annotation, and standardization process is performed on the acquired endoscopic images and report text using a collaborative processing method involving both large and small models. This generates an inference process and constructs a four-tuple training data structure of image-prompt-thought chain-report text. The specific process of S2 is as follows: Figure 3 As shown, the specific steps include: S21. The obtained endoscopic examination report text is cleaned using a large-scale pre-trained large language model, and samples with low text quality are excluded. The description of each part is separated, and a positive anomaly label is assigned to each part description text. The diagnostic conclusions are then standardized to ensure that the diagnostic text conforms to the latest medical guidelines. S22. Perform quality screening on the acquired digestive endoscopy images to remove low-quality images such as blurry, completely black, and glare-prone images. S23. Train a small anatomical part classification model using the filtered image data labeled with anatomical parts; train a small anomaly recognition model by mapping the image part labels to positive anomaly labels in the text. S24. Using the two trained mini-models, label all acquired endoscopic images to form image sets of several images for each anatomical region. Then, pair the image sets with corresponding anatomical region text. Select the images with the highest abnormality level for each region to form another image set, and pair this set with the diagnostic conclusion text. S25. The paired image-text pairs are used to generate a thought chain (CoT) process from image to text pair by using a large-scale pre-trained multimodal large language model with input prompt words. This process mimics the doctor's perception, cognition, and thinking process. A prompt word is added to generate a report, forming a quadruplet training data of image-prompt word-thought chain-site description or image-prompt word-thought chain-diagnosis conclusion. In this embodiment, a total of 475,426 endoscopic images and 136,780 quadruplet data were finally constructed, including data from two hospitals.

[0070] S3. Using the constructed digestive endoscopy report to generate a four-tuple data, train a pre-defined digestive endoscopy report to automatically generate a multimodal large language model, with images and prompts as input and thought chains and site descriptions as output; for example... Figure 4 As shown, the specific steps include: S31. The visual encoder of the multimodal large language model processes each image through multiple visual Transformer blocks. Position encoding is added to the image within each block to obtain the image features. All image features are concatenated to obtain comprehensive features, and special tokens are added at the beginning and end of the image features. S32. The acquired image features are compressed using the visual language connection layer of the multimodal large language model and projected onto the dimensional space of the large language model. Specifically, a multilayer perceptron is used to compress the four adjacent patch features of the image into one, significantly reducing the number of visual tokens. S33. Encode the input text using the text embedding layer of a multimodal large language model and map it to a high-dimensional feature space; S34. The encoded text and visual features are fed into the large language model of the multimodal large language model, and its autoregressive mechanism is used to generate thought chain-part description or thought chain-diagnostic conclusion.

[0071] S4. For the model and output format, a strategy of one-stage multimodal data instruction fine-tuning and two-stage multimodal custom reward function reinforcement learning is adopted to train the model's report generation capability; the specific steps include: S41. The first stage involves fine-tuning the instructions for large-scale endoscopic report data. Images and prompts are used as input, and thought processes and site descriptions are used as output. During training, a causal masking mechanism is employed, preventing future text from being seen in the report text to ensure consistency between the training and inference processes. Simultaneously, loss calculation is performed only on the thought processes, site descriptions, or diagnostic conclusions, using a cross-entropy loss function for training. S42. In the second stage, the medical keyword loss function is used to optimize specific lesions and image lesion features in the model-generated report. Group relative strategy optimization (GRPO) is adopted to design a medical keyword reward function and a format structure reward function to further improve the model's reporting performance and conform to human preferences.

[0072] S5. For the large model with report generation capability, add two classification heads after its visual encoder to perform image preprocessing tasks such as endoscopic image anatomical site classification and endoscopic image abnormality probability scoring, respectively; for example... Figure 4 As shown, the specific steps include: S51. After training the multimodal large language model, it has the function of generating report text. Two multilayer perceptron (MLP) classification heads are added after the last Transformer block of the visual encoder in the trained model. S52. The two MLPs were trained using data labeled with anatomical locations and data labeled with positive abnormalities, respectively, and other parameters were frozen. Ultimately, a multimodal large language model for generating digestive endoscopy reports was obtained, integrating three functions: report text generation, anatomical location classification, and positive abnormality classification.

[0073] S6. Input the endoscopic images into the trained multimodal large language model for generating endoscopic reports, design a Clinical Visual Chain (CoCV) inference strategy for report generation, input the images into the model, automatically select representative images and generate report text; the specific steps of S6 include: S61. First, all images are processed by a visual encoder to extract features and then processed by two classification heads: anatomical location classification and positive anomaly classification, to obtain pre-labels for the images. Using methods similar to those in data processing, the input image set is constructed.

[0074] S62. The image set is then input into the multimodal large language model for digestive endoscopy to generate a site description for each area and a final diagnostic conclusion. The final report consists of representative images representing the areas with the highest degree of abnormality, and the report text comprises site descriptions and diagnostic conclusions.

[0075] S7. Utilizing the constructed digestive endoscopy report to generate a multimodal large language model and the constructed clinical visual chain reasoning strategy, reasoning is performed on a large amount of data to construct an evaluation system that combines endoscopic medical keywords and traditional indicators, supplemented by medical expert assessment; specific steps include: S71. Using the constructed digestive endoscopy report generation multimodal large language model and the constructed clinical visual chain reasoning strategy, reasoning is performed on a large amount of data to construct an endoscopy medical keyword evaluation index, and the model is evaluated by combining it with the traditional text similarity evaluation index.

[0076] S72. For some samples, medical expert evaluation is used as an auxiliary means to assess the representativeness of the images, the completeness, accuracy, and medical applicability of the report text.

[0077] Based on the same principles as the methods provided in the embodiments of this application, the embodiments of this application also provide an automatic generation system for digestive endoscopy reports based on a multimodal large language model. The specific method and steps are the same as those provided, such as... Figure 3 and 4 As shown.

[0078] For S71 in this embodiment, the automatic generation method of digestive endoscopy reports based on multimodal large language models in this embodiment is compared with the current mainstream multimodal large language models. Inference is performed on a large amount of data, and the results are evaluated on the evaluation index of endoscopic medical keywords and the traditional text similarity evaluation index. The results significantly outperform the existing multimodal large language models, demonstrating the superiority of the proposed method, as shown in Tables 1 and 2: Table 1

[0079] Table 2

[0080] In step S72 of this embodiment, the doctor evaluates the automatic generation method for digestive endoscopy reports based on a multimodal large language model, aiming to quantitatively assess the consistency between AI-generated reports and human-generated reports in a clinical setting. The clinician's evaluation results indicate that the overall quality of the generated reports is good, as shown in Table 3. Table 3

[0081] In one embodiment, a computer-readable storage medium is also provided, the computer-readable storage medium storing a computer program that, when executed by a relevant processor, implements one or more of the methods described in the above embodiments.

[0082] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A method for automatically generating digestive endoscopy reports based on a multimodal large language model, characterized in that, The method includes: Acquire multiple first endoscopic image groups and a first report text corresponding to each first endoscopic image group. The first report text includes multiple site descriptions and a diagnostic conclusion. The first endoscopic image group is training image data collected during historical endoscopic examinations. Based on multiple sets of the first endoscopic images and multiple sets of the first report texts, the initial model is trained to obtain the target model; the initial model is a multimodal large language model. The second set of endoscopic images is input into the target model. The anatomical location prediction result is generated by the anatomical location classification head of the target model, and the abnormality score is generated by the abnormality probability scoring head of the target model. The second set of endoscopic images is the endoscopic examination image data for which the report is to be generated. Based on the predicted anatomical locations, representative images are determined; The representative images are input into the target model to generate corresponding site descriptions and diagnostic conclusions, thereby forming the second report text corresponding to the second endoscopic image group.

2. The method according to claim 1, characterized in that, Before training the initial model based on multiple first endoscopic image groups and multiple first report texts to obtain the target model, the method further includes: Text cleaning is performed on multiple first report texts to obtain cleaned first report texts; the text cleaning includes deleting blank text, deleting incomplete text, and deleting text that does not conform to the preset report format; The multiple part descriptions in each of the first cleaned report texts are split according to anatomical parts to obtain the split part descriptions; Based on the segmented part descriptions, the corresponding anatomical part categories are determined; Generate normal or abnormal labels for each of the split parts; The diagnostic conclusions in the first report text after cleaning are subjected to terminology standardization processing to make the diagnostic conclusions conform to a preset set of medical terms.

3. The method according to claim 2, characterized in that, Before training the initial model based on multiple first endoscopic image groups and multiple first report texts to obtain the target model, the method further includes: Image quality detection is performed on multiple first endoscopic images of the first endoscopic image group; the image quality detection includes sharpness detection, brightness detection, and glare detection. Based on preset clarity thresholds, brightness thresholds, and glare determination rules, multiple first endoscope images are screened, and first endoscope images that do not meet preset image quality conditions are removed.

4. The method according to claim 3, characterized in that, The process of training an initial model based on multiple sets of the first endoscopic images and multiple sets of the first report texts to obtain a target model includes: Acquire the first endoscopic image labeled with anatomical location as a training sample for the anatomical location; The anatomical site classification model is trained based on the training samples of the anatomical sites. Obtain the first endoscopic image corresponding to the normal label or abnormal label as an abnormal training sample; An anomaly recognition model is trained based on the aforementioned anomaly training samples.

5. The method according to claim 4, characterized in that, The process of training an initial model based on multiple sets of the first endoscopic images and multiple sets of the first report texts to obtain a target model includes: The anatomical location classification model is used to infer the first endoscopic image to obtain the predicted anatomical location label for each of the first endoscopic images. The anomaly detection model is used to infer the anomaly from the first endoscopic image to obtain the anomaly prediction result for each of the first endoscopic images. The first endoscopic image is divided into multiple anatomical site image sets based on the anatomical site prediction labels; Abnormal images are filtered from each anatomical site image set based on the abnormality prediction results.

6. The method according to claim 5, characterized in that, The process of training an initial model based on multiple sets of the first endoscopic images and multiple sets of the first report texts to obtain a target model includes: A first prompt word is generated based on the pairing result between the set of anatomical region images and the segmented region descriptions; A second prompt word is generated based on the pairing result between the abnormal image and the diagnostic conclusion; The first endoscopic image or the abnormal image and the corresponding prompt words are input into a pre-trained multimodal large language model to generate the corresponding thought chain; The first endoscopic image or the abnormal image, the corresponding first prompt word, the second prompt word, the thought chain, and the corresponding split part description or diagnostic conclusion are combined into a quadruple training data.

7. The method according to claim 1, characterized in that, The process of training an initial model based on multiple sets of the first endoscopic images and multiple sets of the first report texts to obtain a target model includes: Visual features are extracted from the first endoscope image in the first endoscope image group using a visual encoder. Add positional encoding to image features in multiple visual Transformer blocks; Visual features of multiple first endoscopic images in the same first endoscopic image group are stitched together to form a comprehensive visual feature sequence; Add a start marker Token at the beginning position and an end marker Token at the end position of the integrated visual feature sequence; The comprehensive visual feature sequence is compressed through a visual language connection layer, and the compressed visual features are projected onto the feature dimension space of the autoregressive language model. The prompt words are encoded using a text embedding layer to obtain text features, and the visual features and text features are then input into the autoregressive language model.

8. The method according to claim 6, characterized in that, The process of training an initial model based on multiple sets of the first endoscopic images and multiple sets of the first report texts to obtain a target model includes: During the instruction fine-tuning training process, a causal masking mechanism is used to calculate the cross-entropy loss for the output parts corresponding to the thought chain and the report text. In reinforcement learning training, a group relative strategy optimization algorithm is used to update the model parameters, and a medical keyword reward function and a format structure reward function are set as custom reward functions.

9. The method according to claim 7, characterized in that, The step of determining representative images based on the anatomical location prediction results includes: The anatomical site classification head and the anomaly probability scoring head are connected to the last layer output of the visual encoder; The anatomical site classification head is trained based on anatomical site label data; The anomaly probability scoring head is trained based on normal or abnormal labels; Freeze other parameters of the target model when training the anatomical site classification head and the anomaly probability scoring head; During the inference phase, multiple sets of anatomical region images are constructed based on the predicted anatomical region results. The images of each anatomical site are sorted according to the anomaly score. Images whose abnormal scores meet preset conditions are selected as representative images.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a relevant processor, implements the method as described in any one of claims 1 to 9.

Images