Three-dimensional medical image segmentation method and system based on large language model prior knowledge

Abstract

This invention discloses a 3D medical image segmentation method and system based on prior knowledge from a large language model, belonging to the field of 3D medical image computing technology. The method first constructs a medical image-text pair dataset containing 3D medical images and prior information on manually labeled organs and lesions. Then, it generates text prior knowledge by combining the prior information with a large medical language model. Text prior features and high-dimensional image features are extracted by text and image encoders, respectively. Subsequently, a multi-level semantic perception fusion strategy based on a cross-modal attention mechanism is used to fuse the dual features, generating fused semantic features. The high-dimensional image features are decoded into 3D image features matching the spatial dimensions of the original image. Finally, element-wise multiplication fusion processing is used to generate a lesion prediction mask, completing the segmentation. This invention effectively integrates prior knowledge from the medical field, solving the problems of low accuracy and poor generalization of purely data-driven segmentation methods in scenarios with small samples and small lesions.

Description

Technical Field

[0001] This invention belongs to the field of three-dimensional medical image computing technology, specifically relating to a three-dimensional medical image segmentation method and system based on prior knowledge of a large language model. Background Technology

[0002] Tumor segmentation in 3D medical images is a core component of precision tumor diagnosis and treatment. Mainstream methods rely on data-driven deep learning techniques, with typical models including 3D U-Net, VNet, Attention 3D U-Net, and TransUNet-3D. These methods train on massive amounts of labeled 3D medical images to learn the pixel / voxel features and spatial distribution patterns of tumor regions. Compared to traditional manual segmentation and threshold / edge-based machine segmentation methods, they offer improvements in segmentation efficiency and accuracy. However, in practical clinical applications, existing deep learning segmentation methods still face numerous technical bottlenecks, making it difficult to meet the demands of precision tumor diagnosis and treatment. The core issues are mainly reflected in the following aspects: First, it is highly dependent on high-quality labeled data. Three-dimensional medical images have high voxel dimensions and complex anatomical structures. Tumor segmentation and annotation require radiologists with professional medical knowledge to complete the annotation process voxel by voxel. The annotation process is time-consuming, labor-intensive, and extremely costly. In clinical practice, there are common problems such as small sample sizes and imbalanced annotation. Furthermore, data-driven deep learning models are prone to overfitting in small sample scenarios, resulting in a significant decrease in generalization ability. They are particularly ineffective in segmenting rare tumors and small tumors, and cannot adapt to the characteristics of actual clinical datasets.

[0003] Secondly, there is a lack of constraints from the integration of prior medical knowledge. Existing segmentation models are all purely data-driven learning models, learning features only from the image data itself, without effectively integrating prior medical knowledge related to tumor diagnosis and treatment, including the anatomical features, imaging features, and pathological features of tumors. This makes the models prone to learning noise features in the images, and the segmentation results often have problems such as anatomical inconsistencies, missed lesions / false positives, and blurred edge segmentation, which cannot meet the rigorous requirements of clinical diagnosis and treatment.

[0004] Third, data-driven and knowledge-driven approaches have not been effectively integrated. Natural language processing technologies, represented by large language models, have developed rapidly. Medical-specific large language models such as Med-PaLM, ChatMed, and BioGPT have acquired rich prior knowledge in the medical field through pre-training on massive amounts of text data such as medical literature, clinical guidelines, and medical records. However, existing 3D medical image tumor segmentation technologies have failed to fully leverage the knowledge advantages of large language models and have not established an effective knowledge fusion mechanism. This results in insufficient accuracy and generalization of the models in small sample and small / irregular tumor segmentation scenarios, making it difficult to meet the actual needs of precise clinical diagnosis and treatment.

[0005] In addition, existing related technologies have obvious defects. For example, the unsupervised lesion segmentation method based on large model and pseudo-label learning disclosed in patent application CN202411786169.4 does not utilize known organ and lesion information, and the steps are cumbersome, require a lot of manual intervention, and have low clinical feasibility. The multi-target medical image segmentation method based on semantic prompts disclosed in patent application CN202511245402.2 does not make full use of the text generation capability of large language models, and does not deeply integrate image features and priors. The model cannot perceive the semantic relationships and spatial distribution features between structures in medical images.

[0006] Therefore, developing a three-dimensional medical image tumor segmentation method that can effectively integrate medical prior knowledge from large language models to overcome the technical shortcomings of existing pure data-driven segmentation methods has become an urgent technical problem to be solved in this field. Summary of the Invention

[0007] The technical problem to be solved by this invention is to address the shortcomings of the prior art by providing a three-dimensional medical image segmentation method and system based on prior knowledge of a large language model. This addresses the technical problems of existing purely data-driven three-dimensional medical image segmentation methods, which lack effective integration and constraints of prior knowledge in the medical field, resulting in low accuracy and poor generalization in small sample and small / irregular lesion segmentation scenarios. The segmentation results may also contain anatomically unreasonable information, missed lesions, or false positives, which are clinically unacceptable.

[0008] The present invention adopts the following technical solution: A 3D medical image segmentation method based on prior knowledge from a large language model includes the following steps: S1. Construct a medical image-text pair dataset, which includes three-dimensional medical images and artificially annotated prior information on organs and lesions; S2. Based on the prior information about organs and lesions obtained in step S1, generate prior text knowledge through a large language model; S3. Input the prior text knowledge obtained in step S2 into the text encoder to generate prior text features; S4. Input the three-dimensional medical image obtained in step S1 into the image encoder to generate high-dimensional image features; S5. A multi-level semantic perception fusion strategy based on cross-modal attention mechanism is adopted to fuse the text prior features obtained in step S3 and the high-dimensional image features obtained in step S4 to generate fused semantic features. S6. Input the high-dimensional image features obtained in step S4 into the image decoder for decoding, and restore the three-dimensional image features that match the spatial dimension of the original three-dimensional medical image. S7. Perform element-wise multiplication fusion processing on the fused semantic features obtained in step S5 and the three-dimensional image features obtained in step S6 to generate a lesion prediction mask guided by fused semantics, thus completing the segmentation of the three-dimensional medical image.

[0009] Preferably, in step S1, the artificially labeled prior information of organs and lesions includes at least one of organ category, lesion attribution, lesion location, lesion morphology, three-dimensional size, image modality, boundary features, enhancement mode, and benign or malignant attributes of the lesion; the three-dimensional medical image is a CT medical image or an MRI medical image, and the lesion prediction mask is a binary mask used to characterize the lesion / non-lesion attributes of voxels in the three-dimensional medical image.

[0010] Preferably, in step S2, the large language model is an open-source large language model in the medical field. When generating prior knowledge of text, it also combines preset template information to perform a structured description of prior information on organs and lesions.

[0011] Preferably, in step S3, the text encoder's processing of prior text knowledge includes: word segmentation, adding special identifiers, positional encoding, linear projection, and semantic feature extraction to generate prior text features in the form of a two-dimensional matrix.

[0012] Preferably, in step S4, the image encoder's processing of the three-dimensional medical image includes: image segmentation, linear mapping, position encoding, and serialization feature extraction to generate high-dimensional image features that match the prior feature dimensions of the text.

[0013] Preferably, in step S5, the multi-level semantic perception fusion strategy based on cross-modal attention mechanism includes: performing layer normalization processing on text prior features and high-dimensional image features, calculating attention weights with text prior features as query vectors and high-dimensional image features as key vectors and value vectors, and completing cross-modal feature interaction.

[0014] Preferably, in step S6, the decoding process of the image decoder includes: deserialization processing and inverse linear projection, mapping the high-dimensional image features back to a feature space consistent with the spatial dimension of the original three-dimensional medical image.

[0015] Preferably, in step S7, after the element-wise multiplication fusion process, the method further includes global average pooling and activation function mapping steps to convert the fused features into voxel-level lesion probability values ​​and form a lesion prediction mask.

[0016] Preferably, after step S7 is completed, a loss function is calculated based on the predicted lesion mask and the actual lesion mask corresponding to the three-dimensional medical image, and the segmentation model is optimized according to the loss function; the loss function for:

[0017] in, For the loss of the dice, For the two-dimensional cross-entropy loss, The weight representing the loss from the dice roll. The weights represent the two-dimensional cross-entropy loss.

[0018] Secondly, embodiments of the present invention provide a three-dimensional medical image segmentation system based on prior knowledge of a large language model, comprising: The data module is used to construct a medical image-text pair dataset containing 3D medical images and artificially annotated prior information on organs and lesions; The prior module, connected to the data module, is used to generate textual prior knowledge based on prior information about organs and lesions through a large language model. The text module, connected to the prior module, is used to input prior text knowledge into the text encoder to generate prior text features; The image module, connected to the data module, is used to input three-dimensional medical images into the image encoder to generate high-dimensional image features; The fusion module, which connects the text module and the image module respectively, is used to generate fused semantic features by fusing prior text features and high-dimensional image features using a multi-level semantic perception fusion strategy based on a cross-modal attention mechanism. The decoding module, connected to the image module, is used to input high-dimensional image features into the image decoder and restore them to three-dimensional image features that match the spatial dimensions of the original three-dimensional medical image. The segmentation module, which is connected to the fusion module and the decoding module respectively, is used to perform element-wise multiplication fusion processing on the fused semantic features and the three-dimensional image features to generate a lesion prediction mask guided by fused semantics, thereby completing the segmentation of the three-dimensional medical image.

[0019] Thirdly, a computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the above-described three-dimensional medical image segmentation method based on prior knowledge of a large language model.

[0020] Fourthly, embodiments of the present invention provide a computer-readable storage medium including a computer program, which, when executed by a processor, implements the steps of the above-described three-dimensional medical image segmentation method based on prior knowledge of a large language model.

[0021] Fifthly, a chip includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described three-dimensional medical image segmentation method based on prior knowledge of a large language model.

[0022] In a sixth aspect, embodiments of the present invention provide an electronic device, including a computer program, which, when executed by the electronic device, implements the steps of the above-described three-dimensional medical image segmentation method based on prior knowledge of a large language model.

[0023] Compared with the prior art, the present invention has at least the following beneficial effects: A 3D medical image segmentation method based on prior knowledge from a large language model is proposed. This method introduces textual prior knowledge generated by the large language model and deeply integrates this prior knowledge with image features. This allows the segmentation process to move beyond relying solely on pixel information and instead gain rich clinical semantic guidance. The method uses manually annotated organ and lesion information as seeds, expanding them into information-rich and expressive textual priors through the large language model knowledge base, effectively compensating for the lack of data in small-sample scenarios. Finally, feature fusion and element-wise multiplication are used to generate lesion prediction masks, ensuring that the segmentation results conform to both image features and clinical logic, significantly improving the accuracy and interpretability of the segmentation.

[0024] Furthermore, by clearly defining the specific types of prior information, such as organ category, lesion attribution, location, morphology, size, modality, boundary, enhancement mode, and benign / malignant attributes, a multi-dimensional annotation system rich in clinical semantics is constructed. This refined prior information provides high-quality input material for the large language model, making its generated text priors more targeted and professional. Simultaneously, by limiting the 3D medical images to CT or MRI images and the lesion prediction mask to a binary mask, the applicable scope and output format of this invention are clarified, making the application scenarios of the technical solution clearer and providing directly usable binary segmentation results for subsequent clinical diagnosis and treatment planning.

[0025] Furthermore, the large language model is explicitly defined as an open-source model for the medical field, and pre-defined template information is introduced to provide a structured description of prior information on organs and lesions. The use of a medical-specific model ensures that the generated text prior conforms to medical terminology and clinical expression habits, resulting in higher semantic accuracy. The introduction of pre-defined templates addresses the discreteness and arbitrariness of manually labeled information, guiding the large language model to generate structured descriptions according to a unified logic and format. This guarantees the consistency and standardization of the text prior, laying a solid foundation for the subsequent stable and efficient extraction of semantic features by the text encoder.

[0026] Furthermore, through a series of operations such as word segmentation, tagging, positional encoding, and linear projection, discrete, low-dimensional natural language text is transformed into continuous, high-dimensional text prior features. This process solves the problems of sparse text data, low information density, and difficulty in direct utilization by deep learning models. In particular, the introduction of positional encoding enables the text encoder to capture long-distance dependencies in sentences, integrating scattered medical terms into contextual features with global semantics. The resulting two-dimensional matrix of text features provides structural alignment and data format convenience for subsequent cross-modal fusion with image features.

[0027] Furthermore, through image segmentation, linear mapping, positional encoding, and sequential feature extraction, the original voxel information of 3D medical images is transformed into high-dimensional image features that match the feature dimensions of the text. The image segmentation strategy reduces computational complexity, linear mapping encodes local image patches into semantic representations, and positional encoding preserves the spatial relationships of image patches, enabling the encoder to understand 3D anatomical structures. The final generated image features have the same dimension as the text features, creating the necessary conditions for subsequent efficient and accurate cross-modal fusion, and serving as a technical prerequisite for effectively guiding image features with prior textual semantics.

[0028] Furthermore, the study clarifies that cross-modal attention computation is performed using textual prior features as query vectors and image features as key-value pairs. This design enables precise questioning and focusing of textual semantics on various regions of image features, and the calculated attention weights can highlight lesion areas related to the textual description while suppressing irrelevant background noise. Through layer normalization and cross-modal feature interaction, the semantic guidance of image features by textual prior knowledge is realized, which is a key technical means to bridge the cross-modal semantic gap and achieve deep integration of knowledge-driven and data-driven approaches.

[0029] Furthermore, through operations such as deserialization and inverse linear projection, the high-dimensional image features output by the encoder are gradually restored to three-dimensional image features that are completely consistent with the spatial dimensions of the original three-dimensional medical image. This process ensures that the features ultimately used for segmentation contain both the deep semantic information extracted by the encoder and retain the spatial resolution of the original image, enabling subsequent lesion localization and boundary delineation to be accurate to the voxel level, providing spatial detail assurance for generating high-precision lesion prediction masks.

[0030] Furthermore, by aligning the fused semantic features with the 3D image features and performing element-wise multiplication, weighted filtering of the 3D image features using prior textual knowledge was achieved, enhancing the lesion region and suppressing background signals. Subsequent global average pooling and the sigmoid activation function aggregated the multi-channel fused features into single-channel, physically meaningful voxel-level lesion probability values. This series of operations concretized the abstract fused semantics into quantifiable segmentation results, ensuring the clinical readability and direct applicability of the output mask.

[0031] Furthermore, the segmentation model is trained and optimized through a weighted combination of dice loss and 2D cross-entropy loss. Dice loss focuses on optimizing the overlap between the predicted mask and the actual lesion region, and is particularly adept at handling the class imbalance problem commonly seen in medical images, where lesion regions are small and the background is large. 2D cross-entropy loss focuses on the classification accuracy of each voxel, adapting to the layered scanning characteristics of medical images by slicing the 3D mask into 2D calculations. The weighted combination of these two methods constrains the model to learn from both the region overlap and pixel accuracy dimensions, significantly improving the model's final segmentation performance and generalization ability.

[0032] It is understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0033] In summary, the method of this invention deeply integrates the medical priors of a large language model with 3D image segmentation. Through the construction of high-quality datasets, professional text prior generation, accurate encoding of bimodal features, cross-modal deep fusion, and optimization of a dedicated loss function, it solves the problems of traditional methods such as lack of medical knowledge, poor generalization with small samples, and low segmentation accuracy, thereby improving the clinical rationality and accuracy of segmentation.

[0034] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0035] Figure 1 This is a schematic flowchart of the method of the present invention; Figure 2 This is a schematic diagram of the model training architecture of the present invention; Figure 3 A schematic diagram illustrating the input of manually labeled prior attributes and template-generated text priors into the large model of this invention; Figure 4 This is a schematic diagram of the text encoder of the present invention; Figure 5 This is a schematic diagram of the image encoder of the present invention; Figure 6 This is a schematic diagram of the multi-level semantic perception fusion strategy of the present invention; Figure 7A schematic diagram of a computer device provided in an embodiment of the present invention; Figure 8 This is a block diagram of a chip provided according to an embodiment of the present invention.

[0036] Among them, 60. Computer equipment; 61. Processor; 62. Memory; 63. Computer program; 600. Electronic device; 610. Processing unit; 620. Storage unit; 6201. Random access memory unit; 6202. Cache memory unit; 6203. Read-only memory unit; 6204. Program / utility; 6205. Program module; 630. Bus; 640. Display unit; 650. Input / output interface; 660. Network adapter; 700. External device. Detailed Implementation

[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] In the description of this invention, it should be understood that the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0039] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0040] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Additionally, the character " / " in this invention generally indicates that the preceding and following objects have an "or" relationship.

[0041] It should be understood that although terms such as first, second, third, etc., may be used in the embodiments of the present invention to describe the preset range, these preset ranges should not be limited to these terms. These terms are only used to distinguish the preset ranges from one another. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0042] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0043] The accompanying drawings illustrate various structural schematic diagrams according to embodiments disclosed in this invention. These drawings are not to scale, and some details have been enlarged for clarity, and some details may have been omitted. The shapes of the various regions and layers shown in the drawings, as well as their relative sizes and positional relationships, are merely exemplary and may deviate from reality due to manufacturing tolerances or technical limitations. Furthermore, those skilled in the art can design regions / layers with different shapes, sizes, and relative positions as needed.

[0044] This invention provides a 3D medical image segmentation method based on prior knowledge from a large language model. It constructs a 3D medical image-text pair dataset, employs a large language model text prior reconstruction strategy, a text prior feature generation strategy, a medical image feature generation strategy, a multi-level semantic perception fusion strategy, and a mask generation guided by fusion semantics. This invention extracts key information from 3D medical images, uses a large language model to enhance information richness, constructs efficient text priors, and improves the generalization of the model's inference process. The multi-level semantic perception fusion strategy combines prior features generated by the medical text encoder with visual features obtained from the medical image encoder to obtain fused semantic features. This fused semantics allows for advance perception of organ and lesion information in medical images, further optimizing the alignment of medical images and text priors in the embedding space and improving segmentation accuracy during model inference.

[0045] This invention uses a large language model to summarize and output the prior attribute information of manually annotated 3D medical images as text priors that the model can input. It fully leverages the powerful generalization ability of the large language model to generate semantically rich text priors, significantly reducing the cost of manually generating text priors and improving the generalization of the model's inference process. The multi-layered semantic perception fusion strategy of this invention includes the fusion of high-dimensional image features and high-dimensional text prior features, performing feature dimension alignment. The shallow layer retains the basic semantic information of the text priors and the low-level visual information of the medical images. The middle layer involves cross-modal semantic interaction fusion, calculating the attention weights of text semantics to each region of image features, realizing the semantic guidance of text prior knowledge on image features. The deep layer performs nonlinear mapping of global features, mining the deep correlation between text semantics and image features. This not only enriches the semantic knowledge in the network learning process but also improves the accuracy of the model's inference process.

[0046] Please see Figure 1 This invention discloses a three-dimensional medical image segmentation method based on prior knowledge of a large language model, comprising the following steps: S1. Construct a medical image-text pair dataset, which includes three-dimensional medical images and prior information such as manually annotated organs and lesions; Due to the difficulty in acquiring and the high cost of annotation of 3D medical datasets, there are currently no large-scale medical image-text pair datasets. Therefore, this study first collects a large number of publicly available 3D medical image datasets. Then, the image data is manually annotated with information such as visible organs, lesion names, lesion locations, and lesion attributes to construct a large-scale medical image-text pair dataset. The specific steps for constructing this dataset include: S101. Collect and integrate different publicly available 3D medical image datasets; This invention collects and integrates five different publicly available 3D medical image segmentation datasets, totaling 1001 CT scan data, including 3D medical images spanning two main regions of the human body, and 2765 segmentation annotations. Detailed information about the datasets is shown in Table 1. Table 1: Details of 5 Medical Image Segmentation Datasets S102. Examine the targets in each 3D medical image dataset and annotate them with information such as visible organs and lesion attributes. The annotation information of the kidney and its tumor is shown in Table 2: Table 2: Labeling information for kidneys and their tumors

[0047] First, each target in each dataset is examined and manually labeled, which ensures accuracy and clarity across medical-text pairs datasets.

[0048] S2, Text Prior Knowledge Generation Extract prior information such as organs and lesions from step S1, and use the large language model to generate prior knowledge with high information richness corresponding to the template. Due to the discrete, objective, deterministic, and static nature of manually annotated text, it is difficult to use it as prior for model training. Therefore, it is necessary to leverage the powerful generalization ability of large language models to generate high-quality prior prompts based on manually annotated text and templates. In addition, considering that manual generation is costly and lacks diversity and flexibility, using large language models can generate high-quality and rich text priors.

[0049] Please see Figure 3 The specific steps for generating prior knowledge of the text are as follows: S201. Input manually annotated text and corresponding template information into the large language model. The large language model can combine this information to generate a summary description of the target medical image.

[0050] S202. Repeat step S1 multiple times to generate multiple descriptions, and randomly select one of them as the true text prior.

[0051] We use the open-source medical language model Baichuan-M3, which can analyze the labeled discrete information and combine it with its clinical prior knowledge to generate the corresponding text prior representation based on the provided template.

[0052] S3, Generated Text Features The prior knowledge generated in step S2, namely the natural language description of organs and lesions, is input into the text encoder to generate text features. Because text is low-dimensional discrete data with severe sparsity and extremely low information density, it lacks the ability to extract structured semantics, cannot capture context and long-distance dependencies, and cannot meet the requirements of model training. Therefore, a text encoder is used to transform text sequences into continuous floating-point vectors, mine the semantic associations of long-distance context, and integrate scattered medical terms (location, morphology, enhancement performance) into unified global semantic features, thereby achieving end-to-end training of the text encoder.

[0053] Text priors need to be encoded into features that the model can recognize by a text encoder. In the text encoder, each text prior is mapped to a vector space through a linear projection layer to obtain a high-dimensional vector, which is the text prior feature.

[0054] Please see Figure 4 The specific steps for obtaining text features are as follows: S301. Segment the obtained prior text into individual words; The text generated in step S2 is pre-segmented into words, usually words or sub-words.

[0055] S302. Add special words at the beginning and end of the prior sequence of text respectively; For example, special words can be added to the beginning and end of the text prior sequence to identify the start and end of the text prior.

[0056] The text encoder requires a text length of 2048, which refers to the number of words. If the text length is less than 2048, padding is used; if the text length is greater than 2048, truncation is performed.

[0057] S303. Convert the segmented words into index identifiers corresponding to the vocabulary in the text encoder. S304. Create positional encoding to provide the text decoder with information about the position of words in the sentence; Since the Transformer architecture itself does not have the ability to capture sequence order, positional encoding is added to provide the model with information about the position of words in a sentence.

[0058] S305. In the text encoder, each word is mapped to a vector space through a linear projection layer. The concatenation of one-dimensional vectors forms a two-dimensional matrix, which is the extracted text feature.

[0059] In the text encoder, each word is mapped to a vector space with a dimension of 1024 through a linear projection layer. Therefore, the dimension of the semantic features extracted from the medical report by the text encoder is 2048×1024.

[0060] S4. Generate medical image features Because the raw voxels of medical image data are only grayscale or signal values, lacking any semantic information related to lesions—such as the slightly lower density, lobulated shape, and rapid enhancement of kidney tumors—these core features are scattered values ​​in the raw voxel matrix. The model cannot directly identify which voxels belong to lesions and which are normal renal cortex / medulla. Three-dimensional medical images undergo convolution or Transformer operations in the image encoder to extract local or global features from the voxels, transforming the scattered voxel values ​​into semantic features representing edges, texture, grayscale distribution, and spatial morphology.

[0061] Please see Figure 5 The three-dimensional medical images from the medical image-text pair dataset in step S1 are input into the image encoder to generate low-dimensional image features. S401. Divide the preprocessed 3D medical image into multiple image blocks; The input image is divided into multiple patches, which are typically cuboids, such as 4×16×16 pixels.

[0062] S402, Each image patch is mapped to a higher-dimensional space through a linear layer (usually a convolutional layer); S403. Each image patch is represented with a position code, indicating the position of each image patch in the original image; In order for the model to understand the spatial information in the image, positional encoding is added to the representation of each patch, so that the model can know the position of each image patch in the original image.

[0063] S404. Serialize the image patches and input them into the image encoder to obtain image features.

[0064] These encoded patches form a sequence, which will be input into the encoder.

[0065] The input 3D medical image to the image encoder is 32×256×256. After image segmentation, the 3D medical image is divided into multiple image blocks of size 4×16×16. Thus, the 3D medical image is divided into 8×16×16 image blocks. Each image block is mapped to a dimension of 1024 through a linear layer. Then, a positional code is added to each image block to indicate its position in the original image. Finally, the image blocks are serialized and input into the image encoder. The dimension of the visual features extracted from the 3D medical image by the image encoder is 2048×1024.

[0066] S5, Multi-level Semantic Awareness Fusion Strategy Due to the inherent heterogeneity between medical text priors and 3D medical image modalities, the cross-modal semantic gap cannot be completely eliminated. Medical text is an abstract sequence of semantic symbols, while 3D medical images are concrete spatial visual tensors. The features of the two essentially belong to representations of different semantic spaces, and there are no natural rules to follow in their mapping. Inputting the dual-modal fusion features into a Transformer and mining the global deep correlations between modalities through nonlinear mapping is the core of the multi-level semantic perception fusion strategy.

[0067] Please see Figure 6 The text and image features obtained in steps S3 and S4 are fused at multiple levels to generate fused semantic features. Prior text features and high-dimensional image features are fused using a cross-attention mechanism to generate fused semantic features, which are used to guide feature map segmentation. The specific steps of the multi-level semantic awareness fusion strategy are as follows: S501. Normalize the text prior features obtained in S3 and the high-dimensional image features obtained in S4, and then pass them through a linear mapping layer to obtain the bottom-level normalized features. Given that the output dimension of the image encoder is 2048×768 and the output dimension of the text encoder is 2048×768, and the feature dimensions of the image features and text features are the same; using a layer normalization algorithm and passing through a linear mapping layer to obtain the bottom-level normalized features, the feature dimension of the medical image features is 2048×768 and the feature dimension of the text prior features is 2048×768.

[0068] S502. Introduce a cross-modal attention mechanism, using text prior features as query vectors and high-dimensional image features as key and value vectors. Calculate the attention weights of text semantics on each region of the image features, and assign weights to the image features to achieve semantic guidance of text prior knowledge on the image features, thus obtaining mid-level interactive fusion features. In the cross-attention layer, textual prior features are used as the query vector, and high-dimensional image features are used as the key and value vectors. The attention weights of textual semantics on each region of the image features are calculated, and the image features are weighted accordingly. This realizes the semantic guidance of textual prior knowledge on image features, resulting in the mid-layer interactive fusion features. At this point, the feature dimension of the output features is 2048×768. The principle of cross-attention is as follows:

[0069] in, The dimension of both the text prior and the medical image features is represented. This represents the prior semantic features of the text output by the text encoder. This represents the high-dimensional features of the three-dimensional medical image output by the image encoder. This represents the learnable weight matrix for end-to-end training in the model.

[0070] S503, the mid-layer interactive fusion features are input to the multilayer perceptron, and the global features are non-linearly mapped to mine the deep correlation between text semantics and image features. At this time, the feature dimension of the output features is 2048×768. S504. Perform batch normalization on the features after deep global semantic fusion to eliminate feature distribution differences and finally generate fused semantic features that combine textual prior semantics and medical image visual information.

[0071] S6, Medical Image Feature Decoding The low-dimensional image features obtained in step S4 are input into the image decoder to recover the three-dimensional medical image features related to the dimensionality of the three-dimensional image. S601. The high-dimensional image features obtained in step S4 are deserialized to restore discrete three-dimensional medical image mask data, and the spatial arrangement and dimensional information of the image are recovered. S602. Input the small image patch features into the inverse linear projection layer, map the high-dimensional image features back to the feature dimensions that match the original three-dimensional medical image, and finally restore the three-dimensional image features that completely correspond to the spatial dimensions (length, width, and layer thickness) of the original three-dimensional medical image.

[0072] S7. Generate a mask for fused semantic guidance. The fused semantic and image features obtained in steps S5 and S6 are multiplied to obtain the lesion prediction mask in the medical image. S701. Perform secondary verification and alignment of the spatial dimension and number of channels on the fused semantic features generated in step S5 and the three-dimensional image features recovered in step S6 to ensure that the spatial dimensions (length, width, and layer thickness) of the two are completely consistent and meet the element-by-element operation conditions. S702. Perform element-wise multiplication of the aligned fused semantic features and the 3D image features, and use the prior text knowledge in the fused semantic features to perform weighted filtering of the 3D image features, thereby strengthening the feature expression of the lesion area and suppressing the feature signals of the background and non-lesion organs. S703. Perform global average pooling on the channel dimension of the fused features after multiplication to achieve information aggregation of multi-channel features and obtain a single-channel lesion feature mapping map. S704. Input the lesion feature map into the Sigmoid activation function to map the feature values ​​to the [0,1] interval, and obtain the probability value of each voxel being a lesion region.

[0073] This invention constructs a network by designing three branch networks and a semantically guided segmentation module. The three networks are: a text encoder, an image encoder, and an image decoder. The text encoder transforms prior natural language text into high-dimensional text feature vectors that the model can recognize and compute, while preserving the semantic integrity and professionalism of the text. The image encoder extracts features, preserves spatial information, and compresses data from 3D medical images to generate high-dimensional image features. The image decoder deserializes, restores the dimensions, and reconstructs the spatial structure of the high-dimensional image features to restore features consistent with the spatial dimensions of the original 3D medical image. The semantically guided segmentation module fuses text features and high-dimensional image features to generate fused semantic features that combine semantic guidance and visual information. The fused semantic features are then weighted and interacted with the 3D image features restored by the decoder. Through multiplication operations, the lesion region features are strengthened, background interference is suppressed, and the model is guided to focus on the lesion region to generate a mask.

[0074] S8. Calculate the loss function The lesion prediction mask obtained in step S7 and the 3D medical image lesion mask in step S1 are calculated using dice loss and two-dimensional cross-entropy loss, and the losses are weighted and summed.

[0075] S801. Convert the format and unify the dimension of the lesion prediction mask obtained in step S7 and the 3D medical image real lesion mask manually annotated in step S1. S802. Calculate the dice loss based on the binarized predicted mask and the real mask to obtain the final dice loss value, and calculate the overlap error of the mask region. S803. Slice the three-dimensional mask according to the layer thickness dimension to obtain several two-dimensional mask slices. Perform binary classification cross-entropy loss calculation on each two-dimensional slice to obtain the final two-dimensional cross-entropy loss value. Calculate the classification probability error of the mask pixels. S804. Based on the task requirements of tumor segmentation in 3D medical images, set the weighting coefficients for the dice loss. Weighting coefficients of two-dimensional cross-entropy loss ,satisfy (default , (Adapting to address the issue of imbalanced lesion regions in medical segmentation) S805. The total loss of the model is obtained by weighted summing of the dice loss value and the two-dimensional cross-entropy loss value. S806. Add an L2 regularization term to the total loss value to suppress model overfitting and obtain the final loss calculation result, which is used for backpropagation and parameter update optimization of the model.

[0076] In the loss function calculation, the loss function for calculating the dice loss between the lesion prediction mask obtained in step S7 and the manually annotated 3D medical image lesion mask obtained in step S1 is defined as follows: The dice loss is a loss function derived from dice coefficients. Its core function is to quantify the overlap between the predicted result and the true label region. It is widely used in medical image segmentation tasks, especially suitable for lesion region segmentation scenarios, and can effectively optimize the segmentation accuracy under conditions of small lesions and class imbalance. This loss function is expressed as follows:

[0077] in, Indicates loss on the dice. The dice coefficients are represented by A, which represents the lesion mask region predicted by the model (corresponding to the set of voxels marked as 1 in the three-dimensional lesion prediction mask generated in step S7), and B represents the real lesion mask region annotated manually (corresponding to the set of voxels marked as 1 in the binary mask annotated by the physician in step S1). This represents the number of intersection elements between the predicted mask and the true mask, i.e., the number of correctly segmented lesion voxels. It reflects the degree of overlap between the prediction and the true mask; a larger value indicates more accurate segmentation. This represents the total number of voxels marked as 1 in the prediction mask (predicted lesion size). This represents the total number of voxels marked as 1 in the true mask (the size of the true lesion region). The coefficient 2 is used to balance the weight of the intersection term and the sum of the sizes of the two regions, avoiding coefficient deviation caused by a certain region.

[0078] In step S8, the loss function is calculated by performing a two-dimensional cross-entropy loss calculation on the lesion prediction mask obtained in step S7 and the manually annotated three-dimensional medical image lesion mask obtained in step S1. The loss function is defined as follows: Two-dimensional cross-entropy loss is an adapted form of cross-entropy loss for two-dimensional image scenarios. Its core function is to quantify single-pixel-level classification probability error. By measuring the difference between the predicted pixel probability distribution and the true pixel label, it constrains the model to optimize pixel classification accuracy. In three-dimensional medical image segmentation, the three-dimensional mask needs to be sliced ​​into two-dimensional images according to layer thickness, and the average value is then taken as the final loss, adapting to the structural characteristics of layered scanning in medical images. This loss function is expressed as follows:

[0079] Where N represents the total number of pixels in a single two-dimensional slice (including pixels from lesions, background, and normal tissue), which varies with the slice resolution. Indicates the first The true label of each pixel (binarized, corresponding to the manually labeled mask in step S1, pixels of the lesion area) Background and normal tissue pixels ), Indicates the first The predicted probability of each pixel (corresponding to the value after Sigmoid activation in step S7, representing the probability that the pixel is a lesion region), coefficients This means that the loss values ​​of all pixels are normalized (to eliminate the influence of slice size differences) to ensure that the loss of slices at different resolutions is comparable and can be averaged.

[0080] In step S805, the dice loss value and the two-dimensional cross-entropy loss value are weighted and summed to obtain the total loss value of the model. The weighted summation of the loss functions for the two losses is defined as follows:

[0081] in, It is the final total loss function. The weight representing the loss from the dice roll. The weights represent the two-dimensional cross-entropy loss.

[0082] This invention uses two loss functions to measure the segmentation accuracy of the model when calculating the loss function: the dice loss function and the two-dimensional cross-entropy loss function. It also uses weighted summation to balance the overlap error of the mask region and the classification probability error of the mask pixels, thereby improving the segmentation performance of the model in many aspects.

[0083] Please see Figure 2 This paper presents the composition of each functional module in the model training process, the direction of data flow, the collaborative interaction between modules, and the closed-loop logic of model training optimization. It intuitively reflects the entire training system from raw data input to model parameter optimization. For the first time, it integrates medical prior knowledge from large language models into the entire model training process, constructing an end-to-end training architecture of text prior generation, bimodal feature encoding, cross-modal feature fusion, segmentation mask generation, and loss calculation optimization. This realizes the semantic constraint and guidance of medical prior knowledge on image segmentation model training, solving the problems of lack of medical knowledge integration and poor generalization in small sample scenarios in traditional model training. The entire architecture uses 3D medical images and manually annotated prior information as dual inputs, lesion prediction mask generation as the core output, and loss function calculation and parameter update as the optimization closed loop. Each module is progressive, with unidirectional data flow and bidirectional feedback, ensuring the coherence and effectiveness of model training.

[0084] Data input: This is the starting point of the architecture. The input consists of integrated 3D medical images (CT / MRI) and manually labeled prior information on organs and lesions. It is the basic data source for the entire model training and provides standardized and structured training data for subsequent modules. Text Prior Generation: Connecting the data input, which is manually annotated prior information, the system generates structured and rich text prior knowledge by combining a medical domain big language model (Baichuan-M3) with preset templates. This realizes the transformation from discrete annotated information to structured medical semantic description, providing a foundation for text feature encoding. Bimodal feature encoding: This is the core feature extraction layer of the architecture, which includes a text encoder submodule and an image encoder submodule. The text encoder submodule takes text prior knowledge as input and outputs text prior features in the form of a two-dimensional matrix. The image encoder submodule takes a three-dimensional medical image as input and outputs high-dimensional image features that match the dimension of the text features. The two submodules realize the transformation of data from different modalities into a unified high-dimensional feature space. Cross-modal feature fusion: Connects bimodal feature encoding, with text prior features and high-dimensional image features as inputs. Through operations such as layer normalization, attention weight calculation, and nonlinear mapping based on cross-modal attention mechanism, it outputs fused semantic features that combine medical text semantics and image visual information. It is the core module for realizing the fusion of knowledge-driven and data-driven approaches. Image feature decoding: Connect to the image encoder sub-module. The input is high-dimensional image features. Through deserialization and inverse linear projection operations, the output is three-dimensional image features that match the spatial dimension of the original three-dimensional medical image. This restores the spatial arrangement information of the image and provides spatially adapted image features for segmentation mask generation. Segmentation mask generation: Connects cross-modal feature fusion and image feature decoding. The input is fused semantic features and 3D image features. Through operations such as element-wise multiplication fusion, global average pooling, and Sigmoid activation, the output is a lesion prediction mask guided by fused semantics, which is the core prediction output of model training. The loss calculation and model optimization module serves as the closed-loop optimization end of the architecture. The inputs are the lesion prediction mask and the manually labeled real lesion mask. It uses a weighted calculation of dice loss and two-dimensional cross-entropy loss combined with L2 regularization to output the total loss value of the model. Through backpropagation, it realizes the parameter update and optimization of each module, completes one iteration of model training, until the model converges.

[0085] The model training architecture is an iterative closed-loop training architecture. A single training cycle consists of: data input → text prior generation → dual-feature encoding → cross-modal fusion → image decoding → mask generation → loss calculation → parameter update. After parameter updates, the optimized parameters of each module are fed back to the front-end modules such as feature encoding, fusion, and decoding for the next training iteration, until the model's total loss converges to a preset threshold. At this point, the model is considered trained and can be used for practical 3D medical image segmentation inference. This closed-loop design ensures that the model can gradually improve the accuracy and generalization of lesion segmentation through continuous iterative optimization, making it particularly suitable for segmentation scenarios involving small samples and small / irregular tumors.

[0086] In another embodiment of the present invention, a three-dimensional medical image segmentation system based on prior knowledge of a large language model is provided. This system can be used to implement the above-mentioned three-dimensional medical image segmentation method based on prior knowledge of a large language model. Specifically, the three-dimensional medical image segmentation system based on prior knowledge of a large language model includes a data module, a prior module, a text module, an image module, a fusion module, a decoding module, and a segmentation module.

[0087] The data module is used to construct a medical image-text pair dataset that includes three-dimensional medical images and artificially annotated prior information on organs and lesions. The prior module, connected to the data module, is used to generate textual prior knowledge based on prior information about organs and lesions through a large language model. The text module, connected to the prior module, is used to input prior text knowledge into the text encoder to generate prior text features; The image module, connected to the data module, is used to input three-dimensional medical images into the image encoder to generate high-dimensional image features; The fusion module, which connects the text module and the image module respectively, is used to generate fused semantic features by fusing prior text features and high-dimensional image features using a multi-level semantic perception fusion strategy based on a cross-modal attention mechanism. The decoding module, connected to the image module, is used to input high-dimensional image features into the image decoder and restore them to three-dimensional image features that match the spatial dimensions of the original three-dimensional medical image. The segmentation module, which is connected to the fusion module and the decoding module respectively, is used to perform element-wise multiplication fusion processing on the fused semantic features and the three-dimensional image features to generate a lesion prediction mask guided by fused semantics, thereby completing the segmentation of the three-dimensional medical image.

[0088] This invention provides a terminal device comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, graphics processing units (GPUs), tensor processing units (TPUs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to achieve a corresponding method flow or function. The processor described in this embodiment can be used for the operation of a three-dimensional medical image segmentation method based on prior knowledge of a large language model, including: A medical image-text pair dataset is constructed, comprising 3D medical images and manually annotated prior information on organs and lesions. Based on the obtained prior information on organs and lesions, text prior knowledge is generated using a large language model. The obtained text prior knowledge is input into a text encoder to generate text prior features. The obtained 3D medical images are input into an image encoder to generate high-dimensional image features. A multi-level semantic perception fusion strategy based on a cross-modal attention mechanism is used to fuse the obtained text prior features and the obtained high-dimensional image features to generate fused semantic features. The obtained high-dimensional image features are input into an image decoder for decoding to restore 3D image features that match the spatial dimensions of the original 3D medical images. The obtained fused semantic features and the obtained 3D image features are then fused element-wise by multiplication to generate a lesion prediction mask guided by fused semantics, thus completing the segmentation of the 3D medical images.

[0089] Please see Figure 7 The terminal device is a computer device. In this embodiment, the computer device 60 includes a processor 61, a memory 62, and a computer program 63 stored in the memory 62 and executable on the processor 61. When executed by the processor 61, the computer program 63 implements the three-dimensional medical image segmentation method based on prior knowledge of a large language model as described in this embodiment. To avoid repetition, these details are not elaborated here. Alternatively, when executed by the processor 61, the computer program 63 implements the functions of each model / unit in the three-dimensional medical image segmentation system based on prior knowledge of a large language model as described in this embodiment. To avoid repetition, these details are not elaborated here.

[0090] Computer device 60 can be a desktop computer, laptop, handheld computer, cloud server, or other computing device. Computer device 60 may include, but is not limited to, a processor 61 and a memory 62. Those skilled in the art will understand that... Figure 7 This is merely an example of computer device 60 and does not constitute a limitation on computer device 60. It may include more or fewer components than shown, or combine certain components, or different components. For example, computer device may also include input / output devices, network access devices, buses, etc.

[0091] The processor 61 may be a Central Processing Unit (CPU), or other general-purpose processors, graphics processing units (GPUs), tensor processing units (TPUs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0092] The memory 62 can be an internal storage unit of the computer device 60, such as a hard disk or memory of the computer device 60. The memory 62 can also be an external storage device of the computer device 60, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on the computer device 60.

[0093] Furthermore, the memory 62 may include both internal storage units of the computer device 60 and external storage devices. The memory 62 is used to store computer programs and other programs and data required by the computer device. The memory 62 can also be used to temporarily store data that has been output or will be output.

[0094] Please see Figure 8 The terminal device is an electronic device 600, which is manifested in the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including storage unit 620 and processing unit 610), a display unit 640, etc.

[0095] The storage unit stores program code, which can be executed by the processing unit 610 to perform the steps described in the method section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 610 can perform actions such as... Figure 1 The steps are shown in the figure.

[0096] Storage unit 620 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 6201 and / or cache memory 6202, and may further include a read-only memory (ROM) 6203.

[0097] Storage unit 620 may also include a program / utility 6204 having a set (at least one) program module 6205, such program module 6205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0098] Bus 630 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the multiple bus structures.

[0099] Electronic device 600 can also communicate with one or more external devices 700 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 600, and / or with any device that enables electronic device 600 to communicate with one or more other computing devices (e.g., router, modem). This communication can be performed via input / output interface 650. Furthermore, electronic device 600 can also communicate with one or more networks (e.g., local area network, wide area network, and / or public network, such as the Internet) via network adapter 660. Network adapter 660 can communicate with other modules of electronic device 600 via bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0100] Example 4 This invention also provides a storage medium, specifically a computer-readable storage medium, which is a memory device in a terminal device for storing programs and data. It is understood that the computer-readable storage medium here can include both built-in storage media in the terminal device and extended storage media supported by the terminal device; it can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor, which can be one or more computer programs (including program code). More specific examples of the computer-readable storage medium include: an electrical connection with one or more wires, a portable disk, a hard disk, random access memory, read-only memory, erasable programmable read-only memory, optical fiber, portable compact disk read-only memory, optical storage device, magnetic storage device, or any suitable combination thereof.

[0101] Computer-readable storage media also include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium can also be any readable medium other than a readable storage medium that can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium can be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, radio frequency, etc., or any suitable combination thereof.

[0102] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0103] One or more instructions stored in a computer-readable storage medium can be loaded and executed by a processor to implement the corresponding steps of the three-dimensional medical image segmentation method based on prior knowledge of a large language model in the above embodiments; one or more instructions in the computer-readable storage medium are loaded and executed by the processor in the following steps: A medical image-text pair dataset is constructed, comprising 3D medical images and manually annotated prior information on organs and lesions. Based on the obtained prior information on organs and lesions, text prior knowledge is generated using a large language model. The obtained text prior knowledge is input into a text encoder to generate text prior features. The obtained 3D medical images are input into an image encoder to generate high-dimensional image features. A multi-level semantic perception fusion strategy based on a cross-modal attention mechanism is used to fuse the obtained text prior features and the obtained high-dimensional image features to generate fused semantic features. The obtained high-dimensional image features are input into an image decoder for decoding to restore 3D image features that match the spatial dimensions of the original 3D medical images. The obtained fused semantic features and the obtained 3D image features are then fused element-wise by multiplication to generate a lesion prediction mask guided by fused semantics, thus completing the segmentation of the 3D medical images.

[0104] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0105] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0106] Further, the generalization effect after model training is shown below: The network was trained and tested on five standard medical image tumor segmentation datasets (BTCV, MSD Liver, MSD Pancreas, LiTs 2023, KiPA 2022) to segment tumor lesions.

[0107] As shown in Table 3, the method of the present invention significantly outperforms data-driven deep learning methods such as UNETR, UNet++, nnU-Net, SwinUNETR-L, and STUNet-L in terms of performance across different organs and lesions in five datasets.

[0108] Table 3. Experimental Results of Medical Image Segmentation (5 Datasets) Note that the BTCV dataset contains 50 abdominal CT scans from patients with metastatic hepatocellular carcinoma or postoperative abdominal wall hernias. Each scan in the dataset was performed during portal vein contrast and has different volumetric and field-of-view parameters. The intra-image resolution in the dataset varies from 0.54 x 0.54 mm² to 0.98 x 0.98 mm². 2 (The slice thickness is between 2.5 mm and 5.0 mm.) The method of this invention achieved a state-of-the-art mean DSC of 88.35% on this dataset.

[0109] It is noted that on the MSD pancreas dataset (a 3D CT dataset used in medical image segmentation competitions for pancreatic and tumor segmentation, characterized by extreme label imbalance, extremely small tumor proportion, and blurred boundaries, making it one of the most challenging tasks in MSD, often used as a benchmark for algorithm robustness and small object segmentation ability), the method of this invention achieved Mean DESC=76.20%. This indicates that the method does not reach state-of-the-art performance in handling extreme label imbalance, tumor proportion <0.1%, large morphological and location variations, diverse tumor types, and high false negative rate of small tumors, which leads to the model's inference being biased towards the background. It is speculated that the amount of data sample in the dataset may be the bottleneck limiting performance improvement.

[0110] In summary, this invention presents a 3D medical image segmentation method and system based on prior knowledge from a large language model. This method deeply integrates the medical prior knowledge of the large language model with 3D medical image segmentation, effectively addressing the core technical shortcomings of traditional methods. By constructing a multi-dimensional labeled medical image-text pair dataset, a high-quality training foundation is provided for the model. The structured text prior generated using the medical large language model realizes the semantic constraint of medical knowledge on segmentation. A cross-modal attention mechanism achieves deep fusion of text and image features, ensuring the segmentation process aligns with clinical diagnostic logic. Experiments demonstrate that this invention achieves an average DSC of 84.35% on five standard medical image tumor segmentation datasets. It significantly improves generalization in small sample and small / irregular tumor segmentation scenarios, effectively reducing lesion missed detections and false positives. The anatomical rationality and accuracy of the segmentation results are greatly improved, combining clinical practicality with technological innovation.

[0111] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0112] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0113] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0114] In the embodiments provided by this invention, it should be understood that the disclosed devices / terminals and methods can be implemented in other ways. For example, the device / terminal embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0115] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0116] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0117] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random-access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0118] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0119] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0120] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0121] The above content is only for illustrating the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solution based on the technical concept proposed in this invention shall fall within the scope of protection of the claims of this invention.

Claims

1. A three-dimensional medical image segmentation method based on prior knowledge of a large language model, characterized in that, Includes the following steps: S1. Construct a medical image-text pair dataset, which includes three-dimensional medical images and artificially annotated prior information on organs and lesions; S2. Based on the prior information about organs and lesions obtained in step S1, generate prior text knowledge through a large language model; S3. Input the prior text knowledge obtained in step S2 into the text encoder to generate prior text features; S4. Input the three-dimensional medical image obtained in step S1 into the image encoder to generate high-dimensional image features; S5. A multi-level semantic perception fusion strategy based on cross-modal attention mechanism is adopted to fuse the text prior features obtained in step S3 and the high-dimensional image features obtained in step S4 to generate fused semantic features. S6. Input the high-dimensional image features obtained in step S4 into the image decoder for decoding, and restore the three-dimensional image features that match the spatial dimension of the original three-dimensional medical image. S7. Perform element-wise multiplication fusion processing on the fused semantic features obtained in step S5 and the three-dimensional image features obtained in step S6 to generate a lesion prediction mask guided by fused semantics, thus completing the segmentation of the three-dimensional medical image.

2. The three-dimensional medical image segmentation method based on prior knowledge of a large language model according to claim 1, characterized in that, In step S1, the artificially labeled prior information of organs and lesions includes at least one of organ category, lesion attribution, lesion location, lesion morphology, three-dimensional size, image modality, boundary features, enhancement mode, and benign or malignant attributes of the lesion; the three-dimensional medical image is a CT medical image or an MRI medical image, and the lesion prediction mask is a binary mask used to characterize the lesion / non-lesion attributes of voxels in the three-dimensional medical image.

3. The three-dimensional medical image segmentation method based on prior knowledge of a large language model according to claim 1, characterized in that, In step S2, the large language model is an open-source large language model in the medical field. When generating prior knowledge of text, it also combines preset template information to perform a structured description of prior information on organs and lesions.

4. The three-dimensional medical image segmentation method based on prior knowledge of a large language model according to claim 1, characterized in that, In step S3, the text encoder's processing of prior text knowledge includes: word segmentation, adding special identifiers, positional encoding, linear projection, and semantic feature extraction, generating prior text features in the form of a two-dimensional matrix.

5. The three-dimensional medical image segmentation method based on prior knowledge of a large language model according to claim 1, characterized in that, In step S4, the image encoder's processing of the three-dimensional medical image includes: image segmentation, linear mapping, position encoding, and serialization feature extraction, generating high-dimensional image features that match the prior feature dimensions of the text.

6. The three-dimensional medical image segmentation method based on prior knowledge of a large language model according to claim 1, characterized in that, In step S5, the multi-level semantic perception fusion strategy based on cross-modal attention mechanism includes: performing layer normalization processing on text prior features and high-dimensional image features, calculating attention weights with text prior features as query vectors and high-dimensional image features as key vectors and value vectors, and completing cross-modal feature interaction.

7. The three-dimensional medical image segmentation method based on prior knowledge of a large language model according to claim 1, characterized in that, In step S6, the decoding process of the image decoder includes: deserialization and inverse linear projection, mapping the high-dimensional image features back to a feature space consistent with the original three-dimensional medical image space dimension.

8. The three-dimensional medical image segmentation method based on prior knowledge of a large language model according to claim 1, characterized in that, In step S7, after the element-wise multiplication fusion process, there are also global average pooling and activation function mapping steps to convert the fused features into voxel-level lesion probability values ​​and form a lesion prediction mask.

9. The three-dimensional medical image segmentation method based on prior knowledge of a large language model according to claim 1, characterized in that, After step S7 is completed, a loss function is calculated based on the predicted lesion mask and the actual lesion mask corresponding to the three-dimensional medical image, and the segmentation model is optimized according to the loss function; the loss function for: in, For the loss of the dice, For the two-dimensional cross-entropy loss, The weight representing the loss from the dice roll. The weights represent the two-dimensional cross-entropy loss.

10. A three-dimensional medical image segmentation system based on prior knowledge of a large language model, characterized in that, include: The data module is used to construct a medical image-text pair dataset containing 3D medical images and artificially annotated prior information on organs and lesions; The prior module, connected to the data module, is used to generate textual prior knowledge based on prior information about organs and lesions through a large language model. The text module, connected to the prior module, is used to input prior text knowledge into the text encoder to generate prior text features; The image module, connected to the data module, is used to input three-dimensional medical images into the image encoder to generate high-dimensional image features; The fusion module, which connects the text module and the image module respectively, is used to generate fused semantic features by fusing prior text features and high-dimensional image features using a multi-level semantic perception fusion strategy based on a cross-modal attention mechanism. The decoding module, connected to the image module, is used to input high-dimensional image features into the image decoder and restore them to three-dimensional image features that match the spatial dimensions of the original three-dimensional medical image. The segmentation module, which is connected to the fusion module and the decoding module respectively, is used to perform element-wise multiplication fusion processing on the fused semantic features and the three-dimensional image features to generate a lesion prediction mask guided by fused semantics, thereby completing the segmentation of the three-dimensional medical image.

Images