Tumor and target region synchronous segmentation method and system based on visual language guidance

Abstract

The invention discloses a tumor and target region synchronous segmentation method and system based on visual language guidance. The method comprises the following steps: constructing a multi-modal data set fusing a medical image and a clinical text; a unified encoder is adopted to extract shared visual features, a double-branch decoder is adopted to predict a tumor region (GTV) and a clinical target region (CTV) respectively, and information sharing and task differentiation modeling are achieved; introducing a medical pre-training large language model, and respectively extracting language features related to GTV and CTV; a visual language collaborative attention module is designed, dynamic fusion and alignment of visual languages are carried out, task related semantics are guided and enhanced, and redundant information is inhibited; and outputting a precise synchronous segmentation result of the GTV and the CTV after fusing the multi-modal features. Compared with an existing scheme, the problems that the boundary of a tumor target region in a medical image is fuzzy and multi-modal information is insufficient in utilization are effectively solved, the segmentation precision and robustness are remarkably improved, and reliable support is provided for clinical radiotherapy planning.

Description

Technical Field

[0001] The present invention relates to the field of medical image processing and artificial intelligence technology, and in particular to a method and system for synchronous segmentation of tumors and target areas based on visual language guidance. Background Art

[0002] In the tumor radiotherapy pathway, accurate delineation of the tumor region (GTV) and the clinical target volume (CTV) is a core step in formulating a treatment plan. Currently, GTV and CTV mainly rely on manual annotation by doctors based on CT images and clinical experience. This process is cumbersome and highly subjective, resulting in poor consistency and low efficiency in segmentation results. In particular, CTV usually lacks clear image boundaries, and its definition relies more on the doctor's judgment of the tumor's biological behavior and clinical pathway. Therefore, it faces greater challenges in automated segmentation tasks. Although deep learning-based image segmentation methods have made some progress, most of them only use visual features for single-target modeling, which makes it difficult to accurately characterize high-order clinical semantic regions such as CTV.

[0003] It is worth noting that GTV and CTV are highly correlated in spatial distribution and semantic structure, and there is a natural synergistic relationship between the two. Existing single-target segmentation methods find it difficult to tap into their potential complementarity, resulting in poor consistency in joint segmentation results and limited clinical practicality. With the development of large-scale visual language models, cross-modal modeling capabilities have provided new ideas for complex medical semantic modeling. Although some studies have initially introduced text-guided mechanisms to assist in medical image segmentation, most methods remain at the single-target, shallow fusion stage and are unable to achieve semantic collaborative modeling and fine segmentation between GTV and CTV. Therefore, there is an urgent need for a segmentation method that integrates image and language information and has the ability to collaboratively perceive tumors and targets to improve segmentation accuracy, structural consistency, and clinical adaptability. Summary of the Invention

[0004] In view of this, the present invention proposes a method and system for simultaneous segmentation of tumors and targets based on visual language guidance. By introducing a large language model pre-trained in the medical field and a cross-modal collaborative attention mechanism, the recognition accuracy and semantic consistency of segmented targets in multimodal medical images are improved, providing high-quality segmentation results support for the precise delineation and auxiliary decision-making of radiotherapy targets.

[0005] In the first aspect, the present invention provides a method for tumor and clinical target segmentation based on visual language collaborative attention, comprising the following steps: constructing a multimodal medical dataset, pairing and annotating CT images with their corresponding clinical texts such as radiology reports and case descriptions to form cross-modal training data; encoding CT images using a unified visual encoding network to extract shared visual features; respectively inputting texts related to GTV and CTV into a large language model that has been pre-trained with medical corpus and fine-tuned by low-rank adaptation (LoRA) to extract first language embedding features and second language embedding features; respectively inputting the above-mentioned visual features and the two language embedding features into their respective visual language collaborative attention modules to generate first fusion features and second fusion features; finally, respectively sending the fusion features into a dual-branch decoder with consistent structure to output the segmentation results of GTV and CTV. The collaborative attention alignment module includes a visual-language attention module and a language gating filter module, which can adaptively regulate the information pathway according to the semantic differences of the segmentation target to achieve specific feature extraction and semantic guidance.

[0006] In a second aspect, the present invention also provides a method and system for simultaneous tumor and target segmentation based on visual language guidance, including: a multimodal data management module for constructing and managing paired image and text data; an image feature extraction module for extracting shared visual features from CT images; a language feature extraction module for extracting language embedding features related to GTV and CTV respectively; a collaborative attention alignment module for achieving modal alignment and fusion of image and language information to guide target segmentation; and a dual-branch segmentation module for generating segmentation masks for GTV and CTV respectively based on fused features. The system adopts an end-to-end architecture as a whole, and introduces a multi-task loss function during training, including segmentation loss for GTV and CTV and visual-language alignment loss based on KL divergence and contrastive learning, to enhance the model's modal integration capabilities and semantic directionality.

[0007] The beneficial effects of the present invention are as follows:

[0008] The technical solution provided by the present invention may include the following beneficial effects: a method and system for synchronous segmentation of tumors and target areas based on visual language guidance is proposed, which realizes the synchronous segmentation of GTV (tumor area) and CTV (clinical target area), and improves the segmentation accuracy through mutual guidance between the two. For the first time, the collaborative optimization of GTV and CTV tasks is introduced into the segmentation task, enabling the model to achieve precise guidance between different targets, significantly improving the segmentation performance. By combining the collaborative attention mechanism of visual features and language information, the model can fuse the semantic information in the image and text, enhance the understanding of the image context, reduce misidentification and improve the differentiation between tumors and target areas. In particular, language information not only provides semantic supplements for the target area in the image, but also helps the model better understand the spatial relationship and semantic features between the tumor and the target area, thereby improving the accuracy of segmentation. This method combines semantic perception and structural consistency expression, significantly improving the accuracy and robustness of synchronous segmentation of complex tumors and targets. By fusing image and clinical text information, it can provide a highly reliable auxiliary basis for the accurate delineation of radiotherapy targets and individualized treatment decisions, with significant clinical application value and promotion prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the specific implementation of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific implementation or the description of the prior art. Obviously, the drawings described below are an implementation of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0010] Figure 1 This is an overall flow chart of a method for simultaneous segmentation of tumors and target areas based on visual language guidance according to the first embodiment of the present invention;

[0011] Figure 2 This is a structural diagram of a system for synchronous tumor and target area segmentation based on visual language guidance according to the second embodiment of the present invention. DETAILED DESCRIPTION

[0012] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions of the invention will be clearly and completely described below with reference to the accompanying drawings. The described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments.

[0013] First embodiment:

[0014] Figure 1 This is a flowchart of a method for simultaneous segmentation of tumors and clinical targets based on visual language collaborative attention according to the first embodiment of the present invention. The method includes steps S101, S102, S103, S104, and S105.

[0015] Step S101: Construct a multimodal medical dataset integrating images and text. 3D CT image data of radiotherapy patients, including labeled gross tumor regions (GTVs) and clinical target volumes (CTVs), is collected. Each image is paired with corresponding unstructured textual information, such as radiology reports and case descriptions, to construct a complete multimodal input sample. GTV and CTV labels are reviewed and annotated by professional radiologists, and the text content is extracted and structured based on task relevance to form a multimodal paired dataset for model training.

[0016] Step S102: Shared visual feature extraction. The 3D CT image is input slice by slice into a Transformer-based visual encoder network to extract visual features with contextual modeling capabilities. This encoder performs a multi-level representation of the anatomical structures and lesion features contained in the image, outputting shared visual features that can be used for both GTV and CTV segmentation tasks.

[0017] Step S103: Extract language embedding features corresponding to GTV / CTV. Input textual information related to GTV and CTV into a large language model (PubMed-BERT) pre-trained in the medical field. This model, trained on clinical corpus, possesses strong medical semantic modeling capabilities. It is fine-tuned using LoRA low-rank adaptation to enhance its ability to guide task semantics. The first language embedding features for GTV and the second language embedding features for CTV are output, respectively.

[0018] Step S104: Vision-language multimodal semantic alignment. The shared visual features and the language embedding features of the GTV or CTV are fed into two parallel vision-language collaborative attention alignment modules. This module consists of a vision-language attention unit and a language-gated filter unit. The former implements cross-modal information fusion, while the latter adjusts the information flow path based on language semantic features, dynamically filters task-relevant features, and improves the perception of differentiated structures.

[0019] Step S105: Output high-precision segmentation results for the GTV and CTV. The two collaborative modules output first and second fused features, respectively, which are fed into a dual-branch decoder with identical structure but independent parameters. The final output is a two-dimensional segmentation mask for the GTV and CTV. This network supports end-to-end training, optimizing the segmentation loss for the GTV and CTV, as well as the visual language semantic alignment loss. This improves the model's comprehensive expression and guidance capabilities, enabling multi-structure collaborative optimization and simultaneous segmentation.

[0020] Second embodiment:

[0021] Figure 2This is a structural diagram of a tumor and target area synchronous segmentation system based on visual language guidance described in an embodiment of the present invention, which includes a multimodal data construction and management module 201, an image feature extraction module 202, a language feature generation module 203, a visual language collaborative alignment module 204 and a dual-branch decoding module 205.

[0022] Multimodal data construction and management module 201: used to construct a multimodal training dataset containing CT images, GTV / CTV annotation information and corresponding radiology text, supporting unified management, pairing and calling of original images and structured clinical text for subsequent model training.

[0023] Image feature extraction module 202: used to receive CT image input and extract shared contextual visual semantic features through a Transformer-based encoding network to provide a unified visual expression for GTV and CTV.

[0024] Language Feature Generation Module 203: This module extracts language cues corresponding to the GTV and CTV tasks and inputs them into a large language model to obtain task-specific language embedding features. This module supports the use of pre-trained medical language models such as PubMed-BERT and improves its modeling capabilities for specialized semantics through the LoRA fine-tuning mechanism.

[0025] Vision-Language Co-Alignment Module 204: This module includes two parallel vision-language co-attention submodules, one for GTV and the other for CTV segmentation. This module includes a cross-modal attention unit and a language gating unit, which align and fuse image and language features, filtering out irrelevant semantic information to improve the accuracy and robustness of segmentation features.

[0026] Dual-branch decoding module 205: This module receives the fused GTV and CTV multimodal features and generates segmentation masks for the GTV and CTV respectively through independent decoding paths. The two branches maintain consistency to share design advantages and enable structural synergy between tasks.

[0027] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them, and the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field can still modify the technical solutions described in the aforementioned embodiments or easily think of changes within the technical scope disclosed by the present invention, or make equivalent replacements for some of the technical features therein within the technical scope disclosed by the present invention; and these modifications, changes or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be based on the scope of protection of the claims.

Claims

1. A method and system for simultaneous segmentation of tumor and target area based on visual language guidance, characterized in that: The following steps are involved: Construct a multimodal medical dataset and pair and annotate CT images with corresponding clinical texts such as radiology reports and case descriptions; Inputting the CT image into a unified visual encoding network to extract shared visual features; The text information corresponding to the tumor region (GTV) and clinical target region (CTV) is input into the large language model pre-trained in the medical field to obtain the first language embedding features for the GTV and the second language embedding features for the CTV; Inputting the visual features and the first language embedding features and the second language embedding features into respective visual-language collaborative attention modules to obtain first fused features and second fused features respectively; The first fusion feature and the second fusion feature are respectively input into the corresponding dual-branch decoder, and the segmentation results of GTV and CTV are output.

2. The method according to claim 1, wherein the multimodal medical training dataset comprises: The GTV and CTV in all 3D CT images are segmented and labeled, and the corresponding unstructured text records such as radiology reports and case descriptions are paired and organized to train a segmentation model with visual-language collaborative guidance capabilities.

3. The method according to claim 1, characterized in that The language embedding features are generated by a large language model pre-trained in the medical field. The large language model is PubMed-BERT, which is trained based on the PubMed corpus. This corpus covers medical semantics and is used to encode disease descriptions, target area information, and treatment points in clinical texts. The large language model is fine-tuned using low-rank adaptation (LoRA) to enhance its representation of medical semantics related to GTV and CTV, and improve its directionality and adaptability in segmentation guidance tasks.

4. The method according to claim 1, wherein The unified visual coding network is an image encoder based on the Transformer structure, which is used to extract image semantic features with context-aware capabilities.

5. The method according to claim 1, wherein The text prompts of the GTV and CTV can be extracted separately based on different radiology report paragraphs or structured fields to achieve targeted language modeling and differentiated guidance of segmentation goals.

6. The method according to claim 1, characterized in that The visual-language collaborative attention alignment module includes a visual-language attention module and a language-gated filtering module. The visual-language attention module uses image features as queries and language embeddings as keys and values ​​to generate a fused visual-language feature vector through a cross-modal attention mechanism. The language-gated filtering module adaptively regulates the information flow path based on the semantic differences between the GTV and CTV tasks, respectively screening key features related to the tumor area or clinical target area, suppressing irrelevant information, and enabling the two decoding branches to accurately perceive and utilize differentiated semantic features, thereby improving segmentation performance and task specificity.

7. The method according to claim 1, characterized in that The first branch decoder and the second branch decoder have the same structure and are used to correspond to different segmentation target tasks respectively, and share the visual features output by the encoder to achieve collaborative optimization between tasks.

8. The method according to claim 1, characterized in that After fusion, the multimodal features are fed into the corresponding decoder via a skip connection to maintain high-resolution details and improve feature expression capabilities.

9. The method according to claim 1, characterized in that The segmentation model is trained through a jointly optimized multi-task loss function, which includes the segmentation loss of GTV and CTV, and the visual-language attention alignment loss based on KL divergence and contrastive learning, to improve the model's ability to integrate multimodal features and the semantic directionality of the segmentation task.

10. A tumor and target area synchronous segmentation system based on visual language guidance, characterized in that: include: Multimodal data management module, used to build and manage image and text datasets and establish cross-modal annotation relationships; Image feature extraction module, used to extract shared visual features of CT images; The language feature extraction module is used to extract the language embedding features corresponding to GTV and CTV from the text respectively; Collaborative attention alignment module, used to achieve modality alignment and fusion of image and language features; The dual-branch segmentation module is used to output the segmentation results of GTV and CTV respectively based on the fusion features.

Images