Inference method and system for digestive tract endoscopic images based on multi-modal large model
By performing sparse pruning on gastrointestinal endoscopic images, the computational and perceptual redundancy problem of the gastrointestinal endoscopic diagnostic system is solved, enabling efficient and accurate diagnosis on edge devices, reducing costs and resource consumption, and supporting flexible deployment of devices from multiple vendors.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HUAZHONG UNIV OF SCI & TECH
- Filing Date
- 2026-06-05
- Publication Date
- 2026-07-10
Smart Images
Figure CN122368055A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer-aided medical image analysis, and more specifically, relates to a reasoning method and system for gastrointestinal endoscopic images based on a multimodal large model. Background Technology
[0002] Colorectal cancer is one of the leading causes of cancer-related deaths worldwide. Clinical statistics show that approximately 95% of colorectal cancers originate from polyps. If polyps are diagnosed and removed via gastrointestinal endoscopy before they become cancerous, the survival rate can exceed 90%; however, the survival rate drops sharply in advanced stages. Colorectal endoscopy is the "gold standard" for the prevention, diagnosis, and treatment of colorectal cancer, and its accurate detection and analysis of polyps is a core task of computer-aided diagnosis and treatment.
[0003] Early medical image analysis relied on expert prior knowledge and manually designed feature templates. While interpretable, its generalization ability was limited under complex anatomical variations. The introduction of Convolutional Neural Networks (CNNs) enabled end-to-end local feature extraction, improving the efficiency of automated diagnosis. However, the local receptive field of convolutional operations limited its effective perception of global anatomical structures. To address this issue, Transformer-based models implemented long-distance dependency modeling through a self-attention mechanism, further enhancing the accuracy of medical image analysis.
[0004] However, the aforementioned models based solely on visual modalities still have the following shortcomings: In real clinical diagnosis and treatment, auxiliary diagnosis not only requires the identification of image features, but also necessitates a comprehensive assessment combining clinical history, patient self-report, and treatment instructions. While existing CNN or Transformer models can accurately detect lesions, they lack cross-modal contextual understanding and deep logical reasoning capabilities, making it difficult to meet the needs of personalized report generation and interactive diagnosis and treatment.
[0005] Although the large-scale visual language model LVLM has initially achieved deep alignment between visual features and semantic space by fusing a large language model with a visual encoder, laying the foundation for general assisted diagnosis, it still faces severe technical challenges when actually deployed in gastrointestinal endoscopic diagnosis and treatment tasks: Perceptual redundancy problem: Gastrointestinal endoscopy images contain a large amount of homogeneous normal mucosa or dark background. The large number of homogeneous visual tokens generated act as explicit noise, interfering with the cross-modal interaction between text instructions and visual features, and reducing the accuracy of capturing core lesions.
[0006] Computational redundancy: The computational complexity of the Transformer architecture increases quadratically with the length of the visual token sequence. A large number of redundant visual tokens participate in attention computation, leading to significant waste of computational power. Cross-modal attention statistics show that the model relies on only a very small number of key lesion features during the deep inference stage, yet still needs to perform complete computation on all redundant visual tokens.
[0007] Therefore, it is particularly necessary to find a lightweight, plug-and-play sparse deployment scheme that is training-free and can be used to effectively eliminate the perceptual and computational redundancy caused by long sequences of visual input, while ensuring the absolute security of information on minute lesions, i.e., preventing accidental deletion due to sparsity. Summary of the Invention
[0008] To address the shortcomings and improvement needs of existing technologies, this invention provides an inference method and system for gastrointestinal endoscopic images based on a multimodal large model. The aim is to leverage existing multimodal large models by pruning gastrointestinal endoscopic images at the visual encoder and decoder stages of the large model, extracting effective information for inference. This invention overcomes existing computational bottlenecks, enabling efficient and secure deployment of a general-purpose multimodal large model on edge devices with limited computing power.
[0009] To achieve the above objectives, according to one aspect of the present invention, an inference method for gastrointestinal endoscopic images based on a multimodal large model is provided, comprising: S1: Based on the sparsity of feature distribution, redundant background removal is performed on the gastrointestinal endoscopy images input to the visual encoder in the multimodal large model. This includes calculating the local probability density, self-information, and global dependency score of the visual token in the gastrointestinal endoscopy image. The retention score vector of the visual token is calculated by weighting the self-information and global dependency score of the visual token. The gastrointestinal endoscopy image is then pruned for the first time based on the retention score vector of the visual token. S2: Input the digestive tract endoscopy images and text information after the first pruning into the decoder in the multimodal large model for decoding, and obtain the inference results of the digestive tract endoscopy images.
[0010] In summary, the above-described technical solutions conceived in this invention can achieve the following beneficial effects: (1) Low-cost deployment with no training required and plug-and-play functionality It eliminates the need for fine-tuning or retraining of the parameters of the large visual language model on the base, does not rely on massive amounts of labeled data and high computing power, significantly reduces the cost of customized adaptation for gastrointestinal endoscopy scenarios, supports flexible deployment on edge devices from different manufacturers, and overcomes the problem that existing technologies are difficult to promote due to high costs.
[0011] (2) Safely remove redundant background and improve diagnostic accuracy By progressively fusing local semantics and global dependencies, this method removes large areas of normal intestinal mucosal background while avoiding the loss of features from minute lesions, overcoming the drawback of traditional methods that are prone to diagnostic illusions. Experiments show that, with only 22% of visual tokens retained, the diagnostic accuracy reaches 97.69%, surpassing the full input baseline (96.98%) and traditional pruning methods.
[0012] (3) Optimize computing and storage efficiency to overcome edge computing power bottlenecks. The system systematically reduces the length of the visual sequence and simultaneously removes the key-value cache of pruned tokens from the decoding stage, thereby reducing resource consumption from the bottom layer. Experiments show that key-value cache usage is reduced by 77.2%, and the time taken in the pre-filling stage is reduced by about 60%. This effectively alleviates the problem of computational complexity increasing quadratically with token length in the Transformer architecture and reduces the risk of memory overflow and latency in high-concurrency scenarios for edge devices. Attached Figure Description
[0013] Figure 1 The diagram shown is a model architecture diagram of an inference method based on multimodal large model of gastrointestinal endoscopic images provided according to an embodiment of the present invention.
[0014] Figure 2 The diagram shows a cross-modal attention heatmap of gastrointestinal endoscopy images and a schematic diagram of the quantitative statistical results of text-visual attention intensity provided in an embodiment of the present invention.
[0015] Figure 3 The diagram shown is a statistical result of cross-modal attention weighting of gastrointestinal endoscopy images provided by an embodiment of the present invention.
[0016] Figure 4 The image shown is a visualization analysis diagram of token pruning effect and mask evolution based on gastrointestinal endoscopy images according to an embodiment of the present invention. Detailed Implementation
[0017] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.
[0018] In this invention, the terms "first," "second," etc. (if present) in the invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.
[0019] This invention discloses an inference method for gastrointestinal endoscopic images based on a multimodal large model. In one optional embodiment, the multimodal large model used is a Transformer architecture neural network model. Based on the base model, this invention prunes the gastrointestinal endoscopic images, removing background images that have little impact on inference while retaining key information in the images, thus reducing the computational requirements of the model. By pruning the images using the method disclosed in this invention, the inference capability for gastrointestinal endoscopic images is improved while saving computational resources. Specifically, as follows... Figure 1 As shown, it includes: S1: Based on the sparsity of feature distribution, redundant background removal is performed on the gastrointestinal endoscopy images input to the visual encoder in the multimodal large model. This includes calculating the local probability density, self-information, and global dependency score of the visual token in the gastrointestinal endoscopy image. The retention score vector of the visual token is calculated by weighting the self-information and global dependency score of the visual token. The gastrointestinal endoscopy image is then pruned for the first time based on the retention score vector of the visual token. S2: Input the digestive tract endoscopy images and text information after the first pruning into the decoder in the multimodal large model for decoding, and obtain the inference results of the digestive tract endoscopy images.
[0020] Preferably, after the gastrointestinal endoscopy image is pruned for the first time by the visual encoder, it is input to the decoder. The decoder also includes combining the text information with the gastrointestinal endoscopy image after the first pruning process to perform a second pruning process on the retained visual tokens, further retaining the visual tokens associated with reasoning. Specifically, this includes: generating a cross-modal response score based on the text tokens and the multi-head self-attention matrix, extracting the self-information corresponding to the retained visual tokens from the self-information of the visual tokens, calculating a comprehensive score vector based on the weighted sum of the self-information of the retained visual tokens and the cross-modal response scores, and performing a second pruning process on the gastrointestinal endoscopy image after the first pruning process according to the comprehensive score vector.
[0021] It should be noted that the specific method for performing the first pruning process on the gastrointestinal endoscopic images in step S1 above includes: S11: Visual token sequence output from the intermediate layer of the normalized visual encoder The normalized visual token is: ; S12: Calculate the distance matrix elements between visual tokens ,in This represents the feature distance between the i-th visual token and the j-th visual token. This represents the normalized feature vector of the i-th visual token. This represents the normalized feature vector of the j-th visual token. S13: Computational Vision Token Local probability density The expression is: ; in Visual tokens The distance between the k-th neighboring visual token in its neighborhood set. Visual tokens of A set of neighborhoods with similar features The dynamic smoothing factor refers to the global average nearest neighbor distance of all visual tokens in the set; if The more similar a visual token is to other visual tokens in its neighborhood, the better the visual token becomes. Local probability density The higher, S14: Calculate the information contribution of each visual token as an information score. ,in Calculated using the following formula: ; in The higher, the more it indicates The greater the predictability, the greater the information contribution in the reasoning process. The smaller the value, the greater the probability that it is a redundant visual token (such as a visual token with an irrelevant background). Conversely, the greater the information contribution. The larger, The higher the probability of it being a key visual token.
[0022] Furthermore, to compensate for the limitations of local metrics and prevent disruption of global semantic associations in the gut microbiome, further evaluation is performed by referencing the current layer's multi-head self-attention matrix, including: S15: Extract the mean of the multi-head self-attention matrix of the current layer. Calculate the degree to which each visual token is globally dependent, and generate a global dependency score. Visual tokens The global dependency score is calculated as follows: ; in, Visual tokens Global dependency score, This represents a vector consisting of the global dependency scores of all visual tokens in the current layer. S16: Weight and fuse the local self-information of visual tokens with the global dependency scores to obtain the retained score vector of all visual tokens in the current layer. : ; in For balance coefficient, For normalization operations, based on Keep the largest value. One visual token, prune the remaining N- A vector is used to obtain the image after the first pruning, where It is a natural number less than N.
[0023] The above pruning steps are specific methods for pruning a certain layer in a visual encoder. The selection of intermediate layers in the visual encoder is determined based on the number of layers in the visual encoder. The pruning process adopts progressive pruning with uniformly spaced layers. Preferably, through multi-layer progressive pruning, for example, in a specific pruning configuration, a uniform interval is set. The first stage of semantic self-information denoising is deployed in the visual encoder. Taking a 24-layer encoder as an example, layers 6, 12, and 18 are used to achieve progressive background feature stripping, thereby avoiding excessive foreground pruning caused by excessive pruning at once. Those skilled in the art should understand that progressive pruning is only a preferred implementation method and is not a limitation of the present invention.
[0024] It should be noted that the visual token sequence refers to the set of visual feature vectors generated after the gastrointestinal endoscopy images are input into the visual encoder of the multimodal large model and extracted through image segmentation and linear mapping.
[0025] Furthermore, the decoder in the multimodal large model also includes a second pruning of the image after the first pruning process, specifically including: S21: Extract the last text token of the current layer. Extract the multi-head self-attention matrix A of the current layer. Corresponding weight row and along multiple dimensions Perform mean aggregation to generate cross-modal response scores. ,in The number of visual tokens currently participating in the calculation is the same as the number of visual tokens retained after the first pruning. ; in express The score of the m-th visual token among the visual tokens quantifies the response score of the visual feature in the current instruction context. This represents the last text token in the multi-head self-attention matrix. Corresponding to the m-th visual token Attention weights; S22: From the original self-information Extract the subset corresponding to the currently retained visual token As a static prior constraint, it is fused with the response score of the visual token to generate a comprehensive score vector. : ; in To adjust the balance coefficient between dynamic response and static prior weights, This represents the subset of local self-information scores corresponding to the N' core visual tokens retained after the first pruning; S23: According to Keep the largest value. One visual token, prune the rest. A vector is used to obtain the image after the second pruning, where M is a vector less than 1. The natural number.
[0026] In embodiments of the present invention, the text information is prompt information, including but not limited to locating polyps or determining the nature of lesions, generating personalized diagnostic reports, and answering interactive diagnostic and treatment questions.
[0027] The second pruning strategy refers to the configuration of the first pruning and adopts progressive pruning, such as pruning based on cross-modal attention decoupling set in the decoder, such as the 4th, 8th and 16th layers of a total of 32 layers. The above is only a preferred implementation method and is not a limitation of the present invention.
[0028] Figure 2 The figure shows a heatmap of cross-modal attention in gastrointestinal endoscopy images and a schematic diagram of the quantitative statistical results of text-visual attention intensity, provided by an embodiment of the present invention. As can be seen from the figure, the normalized text-to-visual attention remains at a low level in most network layers. This is because although text prompts initially interact and align with global visual features in the shallow layers of the network, the attention weights in most regions rapidly approach zero after entering the core inference stage. This uneven distribution indicates that only a small number of key visual features are actually relied upon during the inference process.
[0029] Figure 3 The diagram illustrates the statistical results of cross-modal attention weights in gastrointestinal endoscopy images according to an embodiment of the present invention. As shown, in the shallow layers of the network, the model is in the initial stage of image-text feature alignment, and text tokens maintain broad and dispersed attention to the global visual sequence. However, when the model enters deep network inference, the distribution of cross-modal attention rapidly converges to an extremely sparse state. Text tokens tend to focus on a few core visual tokens highly relevant to the current question, while semantically decoupling occurs from most visual tokens unrelated to the context. This phenomenon provides theoretical support for the token pruning method of the present invention.
[0030] Figure 4The image shown is a visualization analysis diagram of the visual token pruning effect and mask evolution based on gastrointestinal endoscopy images according to an embodiment of the present invention.
[0031] like Figure 4 The diagram illustrates the spatial evolution of the pruning mask: the denoising module at the encoding end safely and significantly removes redundancy caused by homogeneous background regions, while the decoding end further removes semantically irrelevant interference. Throughout the entire network forward propagation process, the true core polyp lesions are always completely preserved without any accidental deletion.
[0032] Example 2: This invention discloses a system for implementing the inference method based on multimodal large model of gastrointestinal endoscopic images in Embodiment 1, comprising: Visual encoding module: used to receive gastrointestinal endoscopy images, wherein the visual encoder includes a first pruning module for performing a first pruning process on the received gastrointestinal endoscopy images; Visual decoding module: used to decode the pruned gastrointestinal endoscopy images and text information to obtain the inference results of the gastrointestinal endoscopy images.
[0033] Furthermore, the visual encoding module also includes a second pruning module, which is used to perform a second pruning process on the gastrointestinal endoscopy images after pruning by the first pruning module.
[0034] The specific process and method of pruning the image by the first pruning module and the second pruning module are the same as those in the first and second pruning processes in Example 1, and will not be described in detail here to avoid redundancy.
[0035] To better illustrate the effectiveness of this invention, specific experimental verification is provided, as shown in the table below. Table 1 presents the quantitative experimental results using HealthGPT as the baseline model. As shown in the table, this invention demonstrates high diagnostic reliability in question-answering tasks by pruning visual tokens based on existing baseline models. Experiments show that traditional similarity-based hard pruning methods are prone to significant performance degradation under extreme compression due to over-smoothing effects. However, the method proposed in this invention, even with a significant reduction in visual tokens, can still accurately locate organs and lesions in colonoscopy images and effectively suppress interference from redundant regions such as non-target intestinal segments. It not only resists the performance degradation caused by pruning but even surpasses the baseline model with full input features in some cases.
[0036] Table 1
[0037] Tables 2 and 3 present the quantitative analysis results of the computational efficiency optimization of this invention. This invention performs best in terms of key-value cache usage, reducing it by 77.2%; in terms of pre-filling stage time, FastV and this invention are comparable, both reducing it by approximately 60%. This algorithm systematically reduces the total number of visual tokens involved in inference, significantly reduces the key-value cache usage at the system's underlying level, and significantly reduces the time consumption of the multimodal context pre-filling stage, effectively mitigating the risks of memory overflow and high latency faced by edge devices (such as endoscope hosts) during high-concurrency inference.
[0038] Table 2
[0039] Table 3
[0040] In summary, the optimized inference method and system for gastrointestinal endoscopic images based on a multimodal large model disclosed in this invention does not require fine-tuning or retraining of the parameters of the large-scale visual language model, nor does it rely on massive amounts of labeled data and high computing power. This significantly reduces the cost of customized adaptation for gastrointestinal endoscopic scenarios and supports flexible deployment on edge devices from different vendors, overcoming the problem of high costs hindering the widespread adoption of existing technologies. Secondly, by progressively fusing local semantics and global dependencies, it avoids the loss of small lesion features while removing large areas of normal intestinal mucosal background, overcoming the deficiency of traditional methods that easily induce diagnostic hallucinations. Experiments show that, with only 22% of visual tokens retained, the diagnostic accuracy reaches 97.69%, surpassing the full input baseline (96.98%) and traditional pruning methods. Finally, the system systematically reduces the length of the visual sequence and simultaneously removes the key-value cache of pruned tokens from the decoding stage, reducing resource consumption from the bottom layer. Experiments show that key-value cache usage is reduced by 77.2%, and the pre-filling stage time is reduced by about 60%, effectively alleviating the problem that the computational complexity in the Transformer architecture increases with the square of the token length, and reducing the risk of memory overflow and latency in high-concurrency scenarios for edge devices.
[0041] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A reasoning method for gastrointestinal endoscopic images based on a multimodal large model, characterized in that, include: S1: Based on the sparsity of feature distribution, redundant background removal is performed on the gastrointestinal endoscopy images input to the visual encoder in the multimodal large model. This includes calculating the local probability density, self-information, and global dependency score of the visual token in the gastrointestinal endoscopy image. The retention score vector of the visual token is calculated by weighting the self-information and global dependency score of the visual token. The gastrointestinal endoscopy image is then pruned for the first time based on the retention score vector of the visual token. S2: Input the digestive tract endoscopy image and text information after the first pruning into the decoder in the multimodal large model for decoding, and obtain the inference result of the digestive tract endoscopy image. The inference result is whether the digestive tract endoscopy image includes lesions. If it includes lesions, the location of the lesions is displayed.
2. The inference method for gastrointestinal endoscopic images based on a multimodal large model according to claim 1, characterized in that, Step S2 also includes performing a second pruning process on the gastrointestinal endoscopy image after the first pruning, combined with text information, to obtain the second pruned gastrointestinal endoscopy image. The second pruned gastrointestinal endoscopy image is then input into the decoder for decoding. The second pruning includes generating a cross-modal response score based on the multi-head self-attention matrix corresponding to the text token, extracting and retaining the self-information corresponding to the visual token from the self-information of the visual token, calculating a comprehensive score vector based on the weighted sum of the retained visual token self-information and the cross-modal response score, and performing a second pruning process on the gastrointestinal endoscopy image after the first pruning process based on the comprehensive score vector.
3. The inference method for gastrointestinal endoscopic images based on a multimodal large model according to claim 1, characterized in that, The first pruning process for gastrointestinal endoscopic images includes: S11: Visual token sequence output from the intermediate layer of the normalized visual encoder The normalized visual token is: ; S12: Calculate the distance matrix elements between visual tokens ,in This represents the feature distance between the i-th visual token and the j-th visual token. This represents the normalized feature vector of the i-th visual token. This represents the normalized feature vector of the j-th visual token. S13: Computational Vision Token Local probability density The expression is: ； in Visual tokens The distance between the k-th neighboring visual token in its neighborhood set. Visual tokens of A set of neighborhoods with similar features The dynamic smoothing factor refers to the global average nearest neighbor distance of all visual tokens in the set; S14: Calculate the information contribution of each visual token as an information score. ,in Calculated using the following formula: ； in The higher the value, the greater the information contribution in the reasoning process. The smaller the value, the greater the information contribution. The larger, The higher the probability of it being a key visual token.
4. The inference method for gastrointestinal endoscopic images based on a multimodal large model according to claim 3, characterized in that, The first pruning process for gastrointestinal endoscopic images also includes: S15: Extract the mean of the multi-head self-attention matrix of the current layer. Calculate the degree to which each visual token is globally dependent, and generate a global dependency score. Visual tokens The global dependency score is calculated as follows: ； in, Visual tokens Global dependency score, This represents a vector consisting of the global dependency scores of all visual tokens in the current layer. S16: Weight and fuse the local self-information of visual tokens with the global dependency scores to obtain the retained score vector of all visual tokens in the current layer. : ； in For balance coefficient, For normalization operations, based on Obtain the image after the first pruning, where It is a natural number less than N.
5. The inference method for gastrointestinal endoscopic images based on a multimodal large model according to claim 2, characterized in that, Step S2, obtaining endoscopic images of the digestive tract after the second pruning, includes: S21: Extract the last text token of the current layer. Extract the multi-head self-attention matrix A of the current layer. Corresponding weight row and along multiple dimensions Perform mean aggregation to generate cross-modal response scores. ,in The number of visual tokens currently involved in the calculation: ； in express The score of the m-th visual token among the visual tokens quantifies the response score of the visual feature in the current instruction context. This represents the last text token in the multi-head self-attention matrix. Corresponding to the m-th visual token Attention weights; S22: From self-information Extract the subset corresponding to the currently retained visual token As a static prior constraint, it is fused with the response score of the visual token to generate a comprehensive score vector. : ； in To adjust the balance coefficient between dynamic response and static prior weights, This represents the subset of local self-information scores corresponding to the N' core visual tokens retained after the first pruning; S23: According to Keep the largest value. A visual token is used to obtain the image after the second pruning, where M is less than... The natural number.
6. The inference method for gastrointestinal endoscopic images based on a multimodal large model according to claim 1, characterized in that, The text information consists of prompts, including locating polyps or determining the nature of lesions, generating personalized diagnostic reports, and answering interactive diagnostic questions.
7. The inference method for gastrointestinal endoscopic images based on a multimodal large model according to claim 3, characterized in that, Visual token sequence refers to the set of visual feature vectors generated after the gastrointestinal endoscopy image is input into the visual encoder of a multimodal large model and extracted through image segmentation and linear mapping.
8. The inference method for gastrointestinal endoscopic images based on a multimodal large model according to claim 3, characterized in that, The selection of intermediate layers in the visual encoder is based on the number of layers in the visual encoder. The pruning process adopts progressive pruning and uniformly spaced layer pruning.
9. A system for implementing the reasoning method based on multimodal large model of gastrointestinal endoscopic images as described in any one of claims 1-8, characterized in that, include: Visual encoding module: used to receive gastrointestinal endoscopy images, wherein the visual encoder includes a first pruning module for performing a first pruning process on the received gastrointestinal endoscopy images; Visual decoding module: used to decode the pruned gastrointestinal endoscopy images and text information to obtain the inference results of the gastrointestinal endoscopy images. The inference results indicate whether the gastrointestinal endoscopy images contain lesions. If lesions are included, the location of the lesions is displayed.
10. The system according to claim 9, characterized in that, The visual encoding module also includes a second pruning module, which is used to perform a second pruning process on the gastrointestinal endoscopy images after they have been pruned by the first pruning module.