A reasoning segmentation method and system for colonoscopy images

Abstract

This invention discloses a method and system for reasoning and segmenting colonoscopy images, belonging to the field of medical image analysis technology. The colonoscopy image and corresponding query text are input into a multimodal large language model, and embedding features corresponding to lexical units related to the target information and lexical units related to the global reasoning semantics of the query text are extracted, thereby explicitly decoupling the target-level semantic information and the query-level semantic information. Based on this, through query-to-visual alignment and target-to-query alignment processes, embedding features carrying global reasoning semantics are injected into the visual space features and segmentation cue space, respectively, to establish a consistent mapping between query semantics, target activation, and pixel masks. Finally, based on the aligned target cue embedding features and query visual features, segmentation and decoding are performed to obtain a mask image corresponding to the query semantics. This invention can capture the global contextual information contained in complex reasoning queries, improving the accuracy of colonoscopy image reasoning and segmentation.

Description

Technical Field

[0001] This invention belongs to the field of medical image analysis technology, and more specifically, relates to a method and system for reasoning and segmenting colonoscopy images. Background Technology

[0002] In endoscopic screening and clinical colonoscopy image analysis, polyp segmentation is crucial for early lesion identification, assisted diagnosis, and clinical decision-making. Traditional image segmentation methods primarily rely on pixel appearance or predefined category labels to directly output masks. While this approach yields good results in standard scenarios, it cannot locate and segment objects based on complex, implicit query text descriptions that require common sense or logical reasoning. Image reasoning segmentation, on the other hand, can often perform indirect segmentation when the query text does not directly specify the target category but provides indirect descriptions that require contextual clues such as approximate location, boundaries, lesion morphology, color texture, boundary clarity, occlusion, and reflections or mucus interference. This allows the model to process and understand the query text to perform reasoning segmentation of the image, propelling visual intelligence from basic "seeing" and "detection" to more advanced "understanding" and "reasoning" based on world knowledge. This has significant application value in endoscopic screening and clinical colonoscopy image analysis. Therefore, researching a colonoscopy image reasoning segmentation method is of great importance.

[0003] With the development of multimodal large language models and cue-based segmentation methods, using natural language questions to assist in medical image segmentation has gradually become a new research direction. However, existing reasoning-aware segmentation methods typically use a single segmentation unit to simultaneously carry target-level semantic information and query-level semantic information, coupling target-level semantic information and query-level semantic information in the same representation for encoding. This fails to capture the rich global contextual information contained in complex reasoning queries, resulting in low accuracy in colonoscopy image reasoning segmentation. Summary of the Invention

[0004] In view of the above-mentioned defects or improvement needs of the prior art, the present invention provides a colonoscopy image reasoning and segmentation method and system to solve the technical problem of low accuracy of colonoscopy image reasoning and segmentation in the prior art.

[0005] To achieve the above objectives, in a first aspect, the present invention provides a colonoscopy image reasoning segmentation method, comprising: performing a reasoning segmentation operation on the colonoscopy image: Colonoscopy images and corresponding query text are input into a multimodal large language model, and the embedding features corresponding to word A are extracted from the output features of the last hidden layer of the multimodal large language model. Embedding features corresponding to word B Among them, the vocabulary used by the multimodal large language model includes a set of lexical units A for representing target information and a set of lexical units B for representing the global inference semantics of the query text; The first projection layer is used to embed the features Perform feature transformation to obtain the first feature. The second projection layer is used to embed the features. After feature transformation, a feature extraction module is used to extract features and obtain the second feature. ; The first cross-attention module is used to process the first feature. Second feature The fusion process yields the target cue embedding features. ; Feature extraction was performed on colonoscopy images to obtain image features. A second cross-attention module is used to process image features. Second feature By fusing the data, the visual features of the query can be obtained. ; Embedded features with target cues As a suggestion template, based on query visual features Segmentation and decoding are performed to segment the colonoscopy image and obtain the mask image of the target to be segmented.

[0006] More preferably, the above-mentioned use of a first cross-attention module to focus on the first feature Second feature To integrate, including: For the first feature Perform the first linear transformation to obtain the matrix. ; For the second feature Perform a second linear transformation to obtain the matrix. ; For the second feature Perform a third linear transformation to obtain the matrix. ; With matrix To query the matrix, using the matrix The key matrix is ​​the matrix. The value matrix is ​​used to calculate the target cue embedding features based on the cross-attention mechanism. .

[0007] More preferably, the above-mentioned use of a second cross-attention module to process image features Second feature To integrate, including: Image features Perform the fourth linear transformation to obtain the matrix. ; For the second feature Perform the fifth linear transformation to obtain the matrix. ; For the second feature Perform the sixth linear transformation to obtain the matrix. ; With matrix To query the matrix, using the matrix The key matrix is ​​the matrix. The value matrix is ​​calculated using a cross-attention mechanism to obtain the query visual features. .

[0008] More preferably, the above-mentioned multimodal large language model, first projection layer, second projection layer, feature extraction module, first cross-attention module, and second cross-attention module are fine-tuned in the following ways: Obtain the training set; the training samples in the training set include: colonoscopy image samples and corresponding query text; the labels of the training samples include: the corresponding real target mask image and the real answer text; the answer text includes: target information and the basis for determining the target information; Perform the above inference segmentation operation on each training sample to obtain the mask image of the target to be segmented in the corresponding training sample, and obtain the corresponding predicted answer text output by the multimodal large language model in the inference segmentation operation. By minimizing the difference loss between the mask image corresponding to each training sample and the real target mask image, as well as the difference loss between the predicted answer text and the real answer text, the above multimodal large language model, the first projection layer, the second projection layer, the feature extraction module, the first cross-attention module and the second cross-attention module are simultaneously fine-tuned.

[0009] More preferably, the decoder in the SAM series model is used to embed target cue features. As a suggestion template, based on query visual features Perform segmentation and decoding; While fine-tuning the aforementioned multimodal large language model, first projection layer, second projection layer, feature extraction module, first cross-attention module and second cross-attention module, the decoder in the SAM series model is also fine-tuned.

[0010] More preferably, the encoder in the SAM series model is used to extract features from the colonoscopy images to obtain image features. .

[0011] In a second aspect, the present invention provides a colonoscopy image reasoning and segmentation system, comprising: The semantic encoding module is used to input colonoscopy images and corresponding query text into the multimodal large language model, and extract the embedding features corresponding to word A from the output features of the last hidden layer of the multimodal large language model. Embedding features corresponding to word B The query text includes: descriptive information of the target to be segmented in the corresponding colonoscopy image; the vocabulary used by the multimodal large language model includes a lexical A for representing target information and a set of lexical B for representing the global inference semantics of the query text. Alignment module, used to apply the first projection layer to the embedded features Perform feature transformation to obtain the first feature. The second projection layer is used to embed the features. After feature transformation, a feature extraction module is used to extract features and obtain the second feature. Feature extraction was performed on colonoscopy images to obtain image features. The first cross-attention module is used to process the first feature. Second feature The fusion process yields the target cue embedding features. A second cross-attention module is used to process image features. Second feature By fusing the data, the visual features of the query can be obtained. ; The segmentation and decoding module is used to embed features with target cues. As a suggestion template, based on query visual features Segmentation and decoding are performed to segment the colonoscopy image and obtain the mask image of the target to be segmented.

[0012] Thirdly, the present invention provides a colonoscopy video inference segmentation method, comprising: segmenting each frame of the colonoscopy video using the colonoscopy image inference segmentation method provided in the first aspect of the present invention, to obtain a mask image of the target to be segmented in the corresponding frame image.

[0013] Fourthly, the present invention provides an electronic device comprising: a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the method provided in the first or second aspect of the present invention.

[0014] Fifthly, the present invention also provides a computer-readable storage medium comprising a stored computer program, wherein the computer program, when executed by a processor, controls the device in which the storage medium is located to perform the method provided in the first or second aspect of the present invention.

[0015] In a sixth aspect, the invention also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the method provided in the first or second aspect of the invention.

[0016] In summary, the above-described technical solutions conceived in this invention can achieve the following beneficial effects: 1. This invention provides a colonoscopy image reasoning segmentation method, which inputs the colonoscopy image and the corresponding query text into a multimodal large language model that includes a set of lexical units in the vocabulary to represent target information and global reasoning semantics of the query text. Embedded features corresponding to lexical units related to target information and lexical units related to global reasoning semantics of the query text are extracted respectively, thereby explicitly decoupling target-level semantic information and query-level semantic information. Building upon this foundation, considering that embedded features carrying target information cannot encode complete reasoning semantics, embedded features carrying global reasoning semantics are injected into the visual spatial features and segmentation cue space respectively to establish a consistent mapping between query semantics, target activation, and pixel masks. Specifically, embedded features carrying target information are mapped to the segmentation cue space and then aligned with embedded features carrying global reasoning semantics based on a cross-attention mechanism to enhance the consistency between the target and the global reasoning semantic context, achieving alignment from the target to the query. Simultaneously, after mapping the embedded features carrying global reasoning semantics to the visual feature dimension, they are injected into the visual spatial features through cross-attention, enabling the visual features to explicitly perceive the query semantics in spatial location, achieving alignment from the query to the visual. Finally, based on the aligned target cue embedded features and query visual features, segmentation decoding is performed to obtain a mask image of the target to be segmented corresponding to the query semantics. This invention enables query semantics to simultaneously constrain spatial feature expression and target cue expression, improving the consistency between segmentation results and query semantics, and avoiding the problem that a single segmentation term cannot simultaneously express instance activation and global reasoning semantics. It can capture the rich global contextual information contained in complex reasoning queries, significantly improving the accuracy of colonoscopy image reasoning segmentation.

[0017] 2. The colonoscopy image reasoning and segmentation method provided by this invention can not only perform accurate reasoning and segmentation when the query text provides clear target description information, but also achieve accurate reasoning and segmentation when the query text does not directly give the target category, but only provides an indirect description that requires judgment based on contextual clues such as approximate location, approximate boundary, lesion morphology, color texture, boundary clarity, occlusion, and reflection or mucus interference. It is suitable for high-quality colonoscopy image reasoning and segmentation tasks driven by problems such as target localization, boundary identification, and complex semantic reference. Attached Figure Description

[0018] Figure 1This is a flowchart of colonoscopy image reasoning and segmentation provided in an embodiment of the present invention.

[0019] Figure 2 This is a schematic diagram illustrating the process from query to visual alignment stage and target to query alignment stage provided in an embodiment of the present invention.

[0020] Figure 3 This is a schematic diagram of the structure of the colonoscopy image reasoning and segmentation system provided in an embodiment of the present invention.

[0021] Figure 4 A schematic diagram of the overall process of the colonoscopy image reasoning and segmentation system is provided for embodiments of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0023] To achieve the above objectives, in a first aspect, the present invention provides a colonoscopy image reasoning segmentation method, comprising: performing a reasoning segmentation operation on the colonoscopy image: Colonoscopy images and corresponding query text are input into a multimodal large language model, and the embedding features corresponding to word A are extracted from the output features of the last hidden layer of the multimodal large language model. Embedding features corresponding to word B The multimodal large language model incorporates a vocabulary containing lexical units A representing target information and a set of lexical units B representing the global inference semantics of the query text; the embedded features... Embedded features that carry target information; Embedded features Embedded features that carry global reasoning semantics; The first projection layer is used to embed the features Perform feature transformation to obtain the first feature. The second projection layer is used to embed the features. After feature transformation, a feature extraction module is used to extract features and obtain the second feature. ; The first cross-attention module is used to process the first feature. Second feature The fusion process yields the target cue embedding features. ; Feature extraction was performed on colonoscopy images to obtain image features. A second cross-attention module is used to process image features. Second feature By fusing the data, the visual features of the query can be obtained. ; Embedded features with target cues As a suggestion template, based on query visual features Segmentation and decoding are performed to segment the colonoscopy image and obtain the mask image of the target to be segmented.

[0024] It should be noted that the feature extraction module mentioned above can be a VIT, CNN, MLP model, etc., without limitation, but VIT is preferred. The projection layer mentioned above can be an MLP, linear projection layer, etc., without limitation, but MLP is preferred. The multimodal large language model mentioned above can be a Chat-UniVi, LLAMA, Qwen, Vicuna, etc., without limitation, but Chat-UniVi is preferred.

[0025] It should be noted that the query text mentioned above typically includes: user instructions for the corresponding colonoscopy image. These are natural language instructions used by the user to describe or indicate a specific target or region that they want to segment, containing complex logic or external knowledge. They usually include descriptive information about the target to be segmented in the corresponding colonoscopy image. This can be direct description of the target (such as target category), or it can be indirect description, requiring consideration of contextual clues such as approximate location, approximate boundary, lesion morphology, color and texture, boundary clarity, occlusion, and reflection or mucus interference. There is no limitation here; it depends on the user. For example, common query texts typically include questions about lesion assessment, lesion location, lesion boundary, lesion size and shape, lesion texture and color, lesion contextual clues, occlusion difficulty, and multiple lesions, such as "Is the visible protrusion the true target area, or is it possibly just temporary mucus / reflection artifacts?".

[0026] In one alternative implementation, the above-mentioned first cross-attention module is used to focus on the first feature. Second feature To integrate, including: For the first feature Perform the first linear transformation to obtain the matrix. ; For the second feature Perform a second linear transformation to obtain the matrix. ; For the second feature Perform a third linear transformation to obtain the matrix. ; With matrix To query the matrix, using the matrix The key matrix is ​​the matrix. The value matrix is ​​used to calculate the target cue embedding features based on the cross-attention mechanism. .

[0027] In one alternative implementation, the above-mentioned second cross-attention module is used to process image features. Second feature To integrate, including: Image features Perform the fourth linear transformation to obtain the matrix. ; For the second feature Perform the fifth linear transformation to obtain the matrix. ; For the second feature Perform the sixth linear transformation to obtain the matrix. ; With matrix To query the matrix, using the matrix The key matrix is ​​the matrix. The value matrix is ​​calculated using a cross-attention mechanism to obtain the query visual features. .

[0028] It should be noted that the various modules involved in this invention can be obtained by using existing pre-trained models, by training in stages, or by training end-to-end; no limitation is made here.

[0029] Preferably, in one optional implementation, the above-mentioned multimodal large language model, first projection layer, second projection layer, feature extraction module, first cross-attention module, and second cross-attention module are fine-tuned in the following ways: Obtain the training set; the training samples in the training set include: colonoscopy image samples and corresponding query text; the labels of the training samples include: the corresponding real target mask image and the real answer text; the answer text includes: target information and the basis for determining the target information; Perform the above inference segmentation operation on each training sample to obtain the mask image of the target to be segmented in the corresponding training sample, and obtain the corresponding predicted answer text output by the multimodal large language model in the inference segmentation operation. By minimizing the difference loss between the mask image corresponding to each training sample and the real target mask image, as well as the difference loss between the predicted answer text and the real answer text, the above multimodal large language model, the first projection layer, the second projection layer, the feature extraction module, the first cross-attention module and the second cross-attention module are simultaneously fine-tuned.

[0030] In one alternative implementation, a decoder from the SAM series models is used to embed target cue features. As a suggestion template, based on query visual features Perform segmentation and decoding; While fine-tuning the aforementioned multimodal large language model, first projection layer, second projection layer, feature extraction module, first cross-attention module and second cross-attention module, the decoder in the SAM series model is also fine-tuned.

[0031] Preferably, in one optional implementation, the encoder in the SAM series model is used to extract features from the colonoscopy images to obtain image features. .

[0032] It should be noted that the model for feature extraction from colonoscopy images is not limited to the encoder in the SAM series models, but can also be VIT, CNN, etc., without limitation here.

[0033] It should be noted that the above SAM series models can be SAM, SAM2, SAM3, etc., and there is no limitation here, but SAM2 model is preferred.

[0034] To further illustrate the colonoscopy image reasoning and segmentation method provided in the first aspect of the present invention, a specific embodiment is described in detail below: In this embodiment, the word segmenter in the multimodal large language model introduces a lexical unit A for representing target information and a set of lexical units B for representing target semantic information; lexical unit A is used to extract target-level semantics, and a set of "..." is used... "Lesson character extraction extracts query-level semantics, and cross-space alignment is achieved through two processes: query-to-visual alignment and target-to-query alignment. In this embodiment, word character A is..." "A lexical unit, at least one, is used to represent the semantics of a target-level instance; lexical unit B is..." "Word units, using 64" "Lessons are grouped together to represent problem-level global reasoning semantics."

[0035] In this embodiment, the multimodal large language model is the Chat-UniVi model; both the first and second projection layers use the MLP model; and the feature extraction module is the VIT model.

[0036] like Figure 1 As shown, the above colonoscopy image reasoning and segmentation process includes the following stages: Multimodal semantic encoding stage: Colonoscopy images and corresponding query text are input into a multimodal large language model. In this embodiment, the multimodal large language model used is the Chat-UniVi model, and LoRA is used for lightweight fine-tuning.

[0037] The hidden vector corresponding to the position of word A is extracted from the output features of the last hidden layer of the multimodal large language model and used as the embedding feature. The hidden vector corresponding to the position of word B is extracted as the embedding feature. Among them, embedded features Primarily used to represent activation information of specific target regions, corresponding to target-level semantics; embedded features It is mainly used to represent global reasoning semantics in query text, such as coarse-grained contexts like target existence, spatial location, boundary clarity, temporal state features, and interfering factors, corresponding to query-level semantics.

[0038] In the above process, the multimodal large language model generates lexical A and lexical B in the predicted response text, and extracts the embedding features of the corresponding lexical positions from the last hidden state of the multimodal large language model. and embedded features .

[0039] This embodiment explicitly decouples target-level semantics from query-level semantics, avoiding the problem that a single segmentation lexical unit cannot simultaneously express instance activation and global reasoning semantics, thereby improving the semantic expression capability driven by complex problems.

[0040] The visual alignment stage was found. This stage maps query-level semantic embeddings to visual feature dimensions and then injects them into visual spatial features through cross-attention, enabling the visual features to explicitly perceive the question's semantics in spatial location. Specifically, this involves using embedded features... After feature transformation, the data is mapped to the visual feature dimension, and then a feature extraction module is used to extract the second feature. This refers to querying semantic features. In this embodiment, the encoder in the SAM series model is used to extract features from the colonoscopy image to obtain image features. Image features As a query, the second feature As key and value inputs, the cross-attention module is used to obtain the query visual features. This process explicitly enhances the spatial regions in the image features that are relevant to the corresponding query text, thereby improving the ability to perceive the target location, boundaries, and contextual relationships.

[0041] Target-to-query alignment phase: This stage maps the target-level segmentation semantic embedding to the segmentation cue space and then cross-aligns it with the query-level semantics to enhance the consistency between target activation and global inference semantics. Specifically, a first projection layer is used to embed the features. Perform feature transformation to obtain the first feature. ; the first feature As a query, the second feature As keys and values, these are input into another cross-attention module to obtain the target cue embedding features. Through this process, the target prompt not only retains the activation information of the target region, but also further integrates the global reasoning semantics in the corresponding query text, so that the target prompt and visual semantics have a consistent query anchor point.

[0042] Segmentation and decoding stage: This embodiment embeds target hints into features. and query visual features The common input is fed into the decoder in the SAM series model. The decoder in the SAM series model is used to embed target-related cue features. As a suggestion template, based on query visual features Segmentation and decoding are performed to segment the colonoscopy image and obtain the mask image of the target to be segmented.

[0043] In this embodiment, both the visual alignment stage and the target-to-query alignment stage are implemented in the alignment module. The specific process is as follows: Figure 2 As shown in the figure. This embodiment improves image target localization, boundary delineation, and semantic consistency by using two processes: query-to-visual alignment and target-to-query alignment, so that the semantics of the question simultaneously constrain the visual spatial features and the segmentation cue space.

[0044] Meanwhile, this embodiment adopts a query-anchored dual alignment mechanism to enhance the mapping capability between multimodal semantics and pixel masks without changing the basic segmentation and decoding framework, which facilitates its application in colonoscopy image reasoning and segmentation tasks.

[0045] In this embodiment, the encoder in the SAM series model preferably has its parameters frozen, and only the multimodal large language model, the first projection layer, the second projection layer, the feature extraction module, the two cross-attention modules, and the decoder in the SAM series model are trained to reduce training costs and maintain visual feature stability. Finally, the decoder in the SAM series model outputs a target mask for each target-level segmentation lemma, supporting prediction of one or more targets under a single question.

[0046] During the training phase, a colonoscopy polyp inference dataset is constructed, and a portion of the data is taken from it as the training set. The training samples include: colonoscopy image samples and corresponding query text; the labels of the training samples include: the corresponding real target mask image and the real answer text; the answer text includes: target information and the basis for determining the target information. This embodiment unifies the colonoscopy image samples, corresponding query text, real target mask images, and real answer text into structured samples that can be directly used for inference and segmentation operations, thereby ensuring consistency between the training data organization method and the input method. In this embodiment, the target refers to polyps.

[0047] When constructing the training set, each colonoscopy image sample is required to establish a correspondence with its target region annotation (i.e., target mask image). Furthermore, query text and response text matching the target region are generated, ensuring consistency between the query text, response text, and the target region. The output is a structured JSON sample file that can be directly used for multimodal inference segmentation training and evaluation.

[0048] During training, the above inference segmentation operation is performed on each training sample to obtain the mask image of the target to be segmented in the corresponding training sample, and the corresponding predicted answer text output by the multimodal large language model in the inference segmentation operation is obtained. By minimizing the difference loss between the mask image corresponding to each training sample and the real target mask image, as well as the difference loss between the predicted answer text and the real answer text, the decoders in the above multimodal large language model, the first projection layer, the second projection layer, the feature extraction module, the two cross attention modules, and the SAM series model are simultaneously fine-tuned.

[0049] In this embodiment, pixel-level binary cross-entropy loss and Dice loss are used to measure the difference loss between the mask image corresponding to the training sample and the real target mask image. The binary cross-entropy loss is used to constrain the mask pixel classification results, and the Dice loss is used to measure the region overlap between the mask image and the real target mask image. In this embodiment, text generation cross-entropy loss is used to measure the difference loss between the predicted answer text and the real answer text.

[0050] This embodiment combines optimized text generation loss and segmentation loss, which can output both text interpretation consistent with the problem and corresponding pixel-level polyp mask, making it suitable for building interpretable clinical auxiliary analysis systems.

[0051] In a specific application example, query text related to the colonoscopy polyp scenario can be input, such as "Where is the current target lesion located?", "Which part of the boundary is the least clear?", "Which target area is occluded but still needs to be identified?". First, target-level semantics and query-level semantics related to the query text are generated. Then, query visual features and target cue embedding features are generated through query-to-visual alignment stages and target-to-query alignment stages. Finally, the corresponding mask map is output in the decoder of the SAM series model.

[0052] To verify the effectiveness of the present invention, a detailed description is provided below with reference to a specific experimental example: The colonoscopy image reasoning and segmentation method provided in this invention was tested on a colonoscopy polyp reasoning dataset. The dataset includes three subsets: training, evaluation, and testing. The testing objective was to generate target segmentation results corresponding to the semantics of a natural language question (i.e., query text corresponding to a colonoscopy image). The method was compared with GSVA, PathMR, VRS-HQ, and the method of this invention. gIoU, cIoU, and Dice were used as evaluation metrics.

[0053] Table 1 presents the validation and test results on the dataset. It can be seen that the method based on this invention performs better in terms of gIoU, cIoU, and Dice metrics.

[0054] Table 1

[0055] Secondly, the present invention provides a colonoscopy image reasoning and segmentation system, such as... Figure 3 As shown, it includes: The semantic encoding module is used to input colonoscopy images and corresponding query text into the multimodal large language model, and extract the embedding features corresponding to word A from the output features of the last hidden layer of the multimodal large language model. Embedding features corresponding to word B The query text includes: descriptive information of the target to be segmented in the corresponding colonoscopy image; the vocabulary used by the multimodal large language model includes a lexical A for representing target information and a set of lexical B for representing the global inference semantics of the query text. The alignment module specifically includes a target-to-query alignment module and a query-to-visual alignment module, used to apply the embedded features using the first projection layer. Perform feature transformation to obtain the first feature. The second projection layer is used to embed the features. After feature transformation, a feature extraction module is used to extract features and obtain the second feature. Feature extraction was performed on colonoscopy images to obtain image features. The first cross-attention module is used to process the first feature. Second feature The fusion process yields the target cue embedding features. A second cross-attention module is used to process image features. Second feature By fusing the data, the visual features of the query can be obtained. ; The segmentation and decoding module is used to embed features with target cues. As a suggestion template, based on query visual features Segmentation and decoding are performed to segment the colonoscopy image and obtain the mask image of the target to be segmented.

[0056] In one optional implementation, a schematic diagram of the overall process of the colonoscopy image reasoning and segmentation system is shown below. Figure 4 As shown.

[0057] The related technical solutions are the same as the colonoscopy image reasoning and segmentation method provided in the first aspect of this invention, and will not be described in detail here.

[0058] Thirdly, the present invention provides a colonoscopy video inference segmentation method, comprising: segmenting each frame of the colonoscopy video using the colonoscopy image inference segmentation method provided in the first aspect of the present invention, to obtain a mask image of the target to be segmented in the corresponding frame image.

[0059] The related technical solutions are the same as the colonoscopy image reasoning and segmentation method provided in the first aspect of this invention, and will not be described in detail here.

[0060] Fourthly, the present invention provides an electronic device, comprising: a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the method provided in the first or third aspect of the present invention.

[0061] The related technical solutions are the same as the colonoscopy image reasoning and segmentation method provided in the first aspect of the present invention and the colonoscopy video reasoning and segmentation method provided in the third aspect, and will not be described in detail here.

[0062] Fifthly, the present invention also provides a computer-readable storage medium comprising a stored computer program, wherein the computer program, when executed by a processor, controls the device in which the storage medium is located to perform the method provided in the first or third aspect of the present invention.

[0063] The related technical solutions are the same as the colonoscopy image reasoning and segmentation method provided in the first aspect of the present invention and the colonoscopy video reasoning and segmentation method provided in the third aspect, and will not be described in detail here.

[0064] In a sixth aspect, the invention also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the method provided in the first or third aspect of the invention.

[0065] The related technical solutions are the same as the colonoscopy image reasoning and segmentation method provided in the first aspect of the present invention and the colonoscopy video reasoning and segmentation method provided in the third aspect, and will not be described in detail here.

[0066] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for reasoning and segmenting colonoscopy images, characterized in that, include: Perform inference segmentation on colonoscopy images: Colonoscopy images and corresponding query text are input into a multimodal large language model, and the embedding features corresponding to word A are extracted from the output features of the last hidden layer of the multimodal large language model. Embedding features corresponding to word B The multimodal large language model uses a vocabulary that includes lexical units A for representing target information and a set of lexical units B for representing the global inference semantics of the query text. The embedded features are applied using a first projection layer. Perform feature transformation to obtain the first feature. The embedded features are then processed using a second projection layer. After feature transformation, a feature extraction module is used to extract features and obtain the second feature. ; The first cross-attention module is used to process the first feature. and the second feature The fusion process yields the target cue embedding features. ; Feature extraction is performed on the colonoscopy images to obtain image features. A second cross-attention module is used to process the image features. and the second feature By fusing the data, the visual features of the query can be obtained. ; Embedded features of the target prompt As a prompt template, based on the aforementioned query visual features Segmentation and decoding are performed to segment the colonoscopy image and obtain a mask image of the target to be segmented; The multimodal large language model, the first projection layer, the second projection layer, the feature extraction module, the first cross-attention module, and the second cross-attention module are fine-tuned in the following ways: Obtain a training set; the training samples in the training set include: colonoscopy image samples and corresponding query text; the labels of the training samples include: corresponding real target mask images and real answer text; the answer text includes: target information and the basis for determining the target information; Perform the inference segmentation operation on each training sample to obtain the mask image of the target to be segmented in the corresponding training sample, and obtain the corresponding predicted answer text output by the multimodal large language model in the inference segmentation operation. By minimizing the difference loss between the mask image corresponding to each training sample and the real target mask image, as well as the difference loss between the predicted answer text and the real answer text, the multimodal large language model, the first projection layer, the second projection layer, the feature extraction module, the first cross-attention module, and the second cross-attention module are simultaneously fine-tuned.

2. The colonoscopy image reasoning and segmentation method according to claim 1, characterized in that, The first cross-attention module is used to process the first feature. and the second feature To integrate, including: For the first feature Perform the first linear transformation to obtain the matrix. ; Regarding the second feature Perform a second linear transformation to obtain the matrix. ; Regarding the second feature Perform a third linear transformation to obtain the matrix. ; With the matrix To query the matrix, with the matrix The key matrix is ​​the matrix described above. The value matrix is ​​used to calculate the target cue embedding features based on the cross-attention mechanism. .

3. The colonoscopy image reasoning and segmentation method according to claim 1, characterized in that, The second cross-attention module is used to process the image features. and the second feature To integrate, including: For the image features Perform the fourth linear transformation to obtain the matrix. ; Regarding the second feature Perform the fifth linear transformation to obtain the matrix. ; Regarding the second feature Perform the sixth linear transformation to obtain the matrix. ; With the matrix To query the matrix, with the matrix The key matrix is ​​the matrix described above. The value matrix is ​​calculated using a cross-attention mechanism to obtain the query visual features. .

4. The colonoscopy image reasoning and segmentation method according to claim 1, characterized in that, The decoder in the SAM series model is used to embed the target cue features. As a prompt template, based on the aforementioned query visual features Perform segmentation and decoding; While simultaneously fine-tuning the multimodal large language model, the first projection layer, the second projection layer, the feature extraction module, the first cross-attention module, and the second cross-attention module, the decoder in the SAM series model is also fine-tuned.

5. The colonoscopy image reasoning and segmentation method according to claim 4, characterized in that, The encoder in the SAM series model is used to extract features from the colonoscopy images to obtain image features. .

6. A colonoscopy image reasoning and segmentation system, characterized in that, include: The semantic encoding module is used to input colonoscopy images and corresponding query text into the multimodal large language model, and extract the embedding features corresponding to word A from the output features of the last hidden layer of the multimodal large language model. Embedding features corresponding to word B The query text includes: descriptive information of the target to be segmented in the corresponding colonoscopy image; the vocabulary used by the multimodal large language model includes a lexical A for representing target information and a set of lexical B for representing the global inference semantics of the query text. Alignment module, used to apply a first projection layer to the embedded features Perform feature transformation to obtain the first feature. The embedded features are then processed using a second projection layer. After feature transformation, a feature extraction module is used to extract features and obtain the second feature. Feature extraction is performed on the colonoscopy images to obtain image features. The first cross-attention module is used to process the first feature. and the second feature The fusion process yields the target cue embedding features. A second cross-attention module is used to process the image features. and the second feature By fusing the data, the visual features of the query can be obtained. ; The segmentation and decoding module is used to embed features with the target cue. As a prompt template, based on the aforementioned query visual features Segmentation and decoding are performed to segment the colonoscopy image and obtain a mask image of the target to be segmented; The multimodal large language model, the first projection layer, the second projection layer, the feature extraction module, the first cross-attention module, and the second cross-attention module are fine-tuned in the following ways: Obtain a training set; the training samples in the training set include: colonoscopy image samples and corresponding query text; the labels of the training samples include: corresponding real target mask images and real answer text; the answer text includes: target information and the basis for determining the target information; Perform an inference segmentation operation on each training sample to obtain a mask image of the target to be segmented in the corresponding training sample, and obtain the corresponding predicted answer text output by the multimodal large language model in the inference segmentation operation. By minimizing the difference loss between the mask image corresponding to each training sample and the real target mask image, as well as the difference loss between the predicted answer text and the real answer text, the multimodal large language model, the first projection layer, the second projection layer, the feature extraction module, the first cross-attention module, and the second cross-attention module are simultaneously fine-tuned.

7. A method for reasoning and segmenting colonoscopy videos, characterized in that, include: Each frame of the colonoscopy video is segmented using the colonoscopy image reasoning segmentation method described in any one of claims 1-5 to obtain a mask image of the target to be segmented in the corresponding frame image.

8. An electronic device, characterized in that, include: A memory and a processor, the memory storing a computer program, the processor executing the computer program to perform the method according to any one of claims 1-5 and 7.

9. A computer program product, characterized in that, Includes a computer program / instruction that, when executed by a processor, implements the method described in any one of claims 1-5 and 7.

Images