A breast cancer medical image segmentation method fusing expert knowledge and entropy optimization

Abstract

The present application belongs to the technical field of medical image segmentation, and particularly relates to a breast cancer medical image segmentation method fusing expert knowledge and entropy optimization, steps of which comprise: obtaining text description information and breast cancer medical images of different modalities corresponding to the same sample, obtaining encoded text Tokens and encoded image Tokens based on text Tokens sequence and image Tokens sequence; constructing an information entropy optimization module to obtain an optimized image Tokens sequence; obtaining multi-modal input features through an interpolation regulation mechanism; constructing a field knowledge dynamic injection module to obtain multi-modal features with enhanced semantics; generating a high-dimensional feature vector of the breast cancer medical image by a pre-trained general model; inputting the high-dimensional feature vector into a segmentation module with dynamic constraints to obtain the final segmentation result of the breast lesion related tissue structure. The present application can improve the segmentation accuracy, semantic consistency and generalization ability of the breast lesion and related tissue structure.

Description

Technical Field

[0001] This invention belongs to the field of medical image segmentation technology, specifically relating to a breast cancer medical image segmentation method that integrates expert knowledge and entropy optimization. Background Technology

[0002] With the continuous development of artificial intelligence technology and computing infrastructure, large-scale pre-trained models have become a crucial infrastructure in the digital age. Currently, token representations generated based on large visual and multimodal models are gradually becoming the core carriers of information expression and knowledge modeling, and are regarded as a new digital production factor. In the medical field, how to enable general-purpose large models to possess professional medical semantic understanding capabilities and quickly adapt to different clinical scenarios has become an important issue in promoting the implementation of artificial intelligence industry.

[0003] In the field of medical image segmentation, although encoder-decoder structures represented by U-Net and Transformer-based visual models have made significant progress in segmentation accuracy, existing methods typically rely on large-scale labeled data for end-to-end training or full-parameter fine-tuning, resulting in high training costs and low transfer efficiency. This is especially true in the multimodal medical image segmentation scenario of breast cancer, where significant differences exist between different imaging modalities such as MRI, CT, and near-infrared spectroscopy, and tissue structures are complex. Simply relying on data-driven models makes it difficult to achieve efficient and stable cross-modal generalization.

[0004] From a systems engineering perspective, the core challenges facing the industrial application of large-scale pre-trained models are no longer limited to model structure design and training strategy optimization. The current key bottleneck lies in how to systematically manage and constrain the model's input image features, knowledge background, execution process, and output prediction through a novel mechanism, and how to inject the correct domain knowledge into the model at the right time in the inference process. This knowledge includes clinical experience, imaging standards, and individual patient information. The completeness of the input information directly determines whether the model can make professional-level judgments.

[0005] However, existing research has not yet developed a unified method to integrate the above mechanisms, and in particular, it lacks a breast cancer medical image segmentation method that can simultaneously achieve multimodal information entropy optimization, dynamic injection of domain knowledge, and dynamic constraints for segmentation prediction.

[0006] The existing technology mainly has the following problems: The lack of an information entropy-based image token optimization mechanism is a significant issue: medical images often contain a high proportion of background and noise regions, while the feature density of effective tissue structures such as breast lesions, blood vessels, and dense glands is relatively low. Existing methods struggle to retain key medical structural information while simultaneously performing entropy calculations, similarity matching, merging, or weighted compression on redundant image tokens. Furthermore, they fail to effectively coordinate the relationship between textual and image features, thus impacting the efficiency of multimodal feature fusion and segmentation accuracy.

[0007] Lack of dynamic domain knowledge injection mechanism: Existing methods mainly rely on data-driven training methods, and rarely incorporate the reading experience of radiologists, imaging features in medical literature, and individual clinical information of patients, resulting in insufficient semantic understanding ability of the model in complex breast tissue structures, such as tumors, blood vessels and dense glands.

[0008] Lack of dynamic constraint mechanism for segmentation prediction: Existing methods lack explicit constraints on the preservation of pre-trained knowledge during fine-tuning, which can easily lead to catastrophic forgetting. Furthermore, a single loss function is difficult to achieve stable convergence under multi-dimensional constraints such as cross-entropy loss, Dice loss, and distribution constraints, thus affecting the segmentation stability of breast lesions and related tissue structures.

[0009] Therefore, given that large-scale pre-trained models and token representations are gradually becoming important technological foundations for intelligent medical image analysis, it is necessary to propose a breast cancer medical image segmentation method that integrates expert knowledge and information entropy optimization. This method, through the synergistic integration of an information entropy optimization mechanism, a dynamic domain knowledge injection mechanism, and a dynamic constraint mechanism for segmentation prediction, enables the pre-trained general model to quickly adapt to multimodal breast cancer medical image segmentation tasks under a small number of sample fine-tuning conditions. This improves the model's segmentation accuracy, semantic consistency, and cross-modal generalization ability for breast lesions and related tissue structures, thus promoting the advancement of intelligent applications in the medical field. Summary of the Invention

[0010] In view of the shortcomings of the prior art, the purpose of this invention is to provide a breast cancer medical image segmentation method that integrates expert knowledge and entropy optimization, which can improve the segmentation accuracy, semantic consistency and generalization ability of breast lesions and related tissue structures.

[0011] To achieve the above objectives, this invention provides a breast cancer medical image segmentation method that integrates expert knowledge and entropy optimization, comprising the following steps: S1. Obtain the text description information and breast cancer medical images of different modalities corresponding to the same sample. Perform word embedding processing on the text description information and slice embedding processing on the breast cancer medical images to obtain the text tokens sequence and the image tokens sequence, where one sample represents a breast cancer case. S2. Input the text tokens sequence and the image tokens sequence into the corresponding encoders to obtain encoded text tokens and encoded image tokens. S3. Construct an information entropy optimization module, introduce an information entropy optimization mechanism to calculate the entropy value of the encoded image tokens, obtain the information entropy value corresponding to each encoded image token, and use a binary soft matching method to match and merge the encoded image tokens based on the similarity of the information entropy values ​​to obtain an optimized image tokens sequence. Multimodal input features are obtained by jointly adjusting the encoded text tokens and optimized image tokens sequences through an interpolation control mechanism. S4. Construct a domain knowledge dynamic injection module to generate expert knowledge representation based on multi-source medical knowledge, and achieve cross-modal semantic alignment between multimodal input features and expert knowledge through the cross-attention mechanism of graph-text interactive query to obtain semantically enhanced multimodal features; S5. Using semantically enhanced multimodal features as input, logical reasoning and effective feature extraction are performed through a pre-trained general model, and a high-dimensional feature vector of breast cancer medical images is generated by the pre-trained general model. S6. Input the high-dimensional feature vector into the segmentation module with dynamic constraints, perform cross-modal adaptation and scene adaptability optimization, and obtain the final segmentation result of breast lesion-related tissue structures.

[0012] As a preferred embodiment of the present invention, in S1, the text description information includes: patient case information for individual samples, and general medical knowledge for all samples; breast cancer medical images include MRI images, CT images, and near-infrared images; The word embedding process specifically involves: segmenting the text description information to obtain text units composed of words, characters, or sub-words, and mapping each text unit to a fixed-dimensional text token vector through the word embedding layer. Based on multiple text token vectors, a text token sequence is formed to convert discrete text into text feature representation in a continuous vector space. The segmented embedding process specifically involves: performing standardized preprocessing on breast cancer medical images of each modality, dividing the complete image into multiple non-overlapping image segments, and vectorizing each image segment through a linear projection layer to obtain image token vectors of fixed dimensions, and forming an image token sequence based on multiple image token vectors.

[0013] As a preferred embodiment of the present invention, in step S2, the text tokens sequence is input into the text encoder to obtain encoded text tokens; the image tokens sequence is input into the image encoder to obtain encoded image tokens; wherein, the encoded text tokens and the encoded image tokens are respectively used as encoded feature representations of text information and image information.

[0014] As a preferred embodiment of the present invention, the process of obtaining multimodal input features in S3 is as follows: In the information entropy optimization module, the encoded image tokens are normalized, then the ratio of the value of each dimension in each encoded image token to the sum of the vectors is calculated, and then the information entropy corresponding to each image slice is calculated using the following formula: ; ; In the formula, H represents the information entropy value of the encoded image tokens; d represents the dimension of the encoded image tokens, and i and j are the indices of the dimension; Represents the normalized probability of the i-th dimension value of the encoded image Tokens; , These represent the values ​​of the i-th and j-th dimensions of the encoded image Tokens, respectively. This represents the normalized probability function; Based on the information entropy distribution characteristics of coded image tokens, a binary soft matching method is used to compare the information entropy distribution characteristics between different coded image tokens, resulting in M ​​groups of candidate similar image tokens. The similarity is determined by the Euclidean distance of the information entropy values ​​of each candidate similar image token group. K groups of candidate similar image tokens with the highest information entropy similarity are selected from the M groups, and these K groups are then merged or weighted and compressed to obtain an optimized image token sequence, where K ≤ M. By using an interpolation adjustment mechanism, linear interpolation is used to align the number of image tokens and text tokens in the encoded text tokens and optimized image tokens sequences to obtain multimodal input features.

[0015] As a preferred embodiment of the present invention, in S4, the process of obtaining semantically enhanced multimodal features in the domain knowledge dynamic injection module is as follows: Multi-source medical knowledge includes the clinical experience of medical personnel, imaging features in medical literature, and individual clinical information of patients. Multi-source medical knowledge is fused and processed, and the fused multi-source medical knowledge is uniformly modeled and semantically integrated through a large language model to generate an expert knowledge representation corresponding to each sample. Based on the generated expert knowledge representation and multimodal input features, a cross-attention mechanism for image-text interactive queries is adopted to achieve bidirectional semantic association between multimodal input features and expert knowledge: using image tokens in the optimized image tokens sequence as query features, relevant text content in the expert knowledge representation is matched; then, using text tokens in the expert knowledge representation as query features, relevant image grayscale regions in the optimized image tokens sequence are matched. A bidirectional matching relationship between image grayscale regions and expert knowledge representation is established in the feature space, and the features of different regions in the optimized image tokens sequence are accurately bound to the corresponding expert knowledge representation, completing the cross-modal semantic alignment between multimodal input features and expert knowledge, and obtaining semantically enhanced multimodal features.

[0016] As a preferred embodiment of the present invention, the domain knowledge dynamic injection module is trained by first performing self-supervised training on a publicly available medical image dataset, and then using image data with medical image analysis reports as a training set to perform consistency training between medical images and expert knowledge on the domain knowledge dynamic injection module. During the consistency training process, a cross-modal bidirectional contrastive loss function is constructed to constrain and optimize the semantic consistency between the image tokens sequence and the text tokens represented by expert knowledge.

[0017] As a preferred embodiment of the present invention, the cross-modal bidirectional contrast loss function is obtained by weighting the image-to-text contrast loss function and the text-to-image contrast loss function, wherein the image-to-text contrast loss function is: ; In the formula, This represents the image-to-text contrast loss value for the m-th sample; Represents the cosine similarity function; This represents the image tokens corresponding to the m-th output result; , These represent the text tokens that represent the expert knowledge representations corresponding to the m-th and n-th input samples, respectively. The temperature parameter is represented; m and n represent the index variables of the samples within the batch; N represents the total number of samples in the training batch. The text-to-image contrast loss function is: ; In the formula, This represents the text-to-image contrast loss value for the m-th sample; This represents the text Tokens corresponding to the m-th output result; , These represent the image tokens in the optimized image tokens sequence corresponding to the m-th and n-th input samples, respectively; The weighted average yields the joint constraint loss function L for cross-modal bidirectional contrastive loss: ; In the formula, These are the loss weight parameters.

[0018] As a preferred embodiment of the present invention, in S5, the process of generating a high-dimensional feature vector of breast cancer medical images is as follows: the semantically enhanced multimodal features are adjusted to a standardized tensor format that matches the input dimension of the pre-trained general model, the channel dimension alignment, spatial size adaptation, and batch dimension processing of the multimodal features are completed, and standardized multimodal features that adapt to the input of the pre-trained general model are obtained. Logical reasoning and effective feature extraction are performed using the pre-trained general model to generate a high-dimensional feature vector of breast cancer medical images.

[0019] As a preferred embodiment of the present invention, the process of obtaining the final segmentation result of the breast lesion-related tissue structure in S6 is as follows: The high-dimensional feature vector output by the pre-trained general model is input into the segmentation module with dynamic constraints. The segmentation module includes a decoder and a segmentation head as the output layer. The high-dimensional feature vector is spatially resolved and its features are decoded by the decoder, and the pixel-wise class probability is output by the segmentation head to generate the initial segmentation prediction result. After training the domain knowledge dynamic injection module, an encoder, information entropy optimization module, domain knowledge dynamic injection module, and segmentation module are added to the pre-trained general model to obtain a complete prediction model. Then, the complete prediction model is fine-tuned. During the fine-tuning stage, the image encoder and text encoder are frozen, and only the decoder parameters in the segmentation module are trained. A joint loss function combining cross-entropy loss, Dice loss, and KL divergence is adopted. Dynamic parameters and a truncation mechanism are introduced to achieve a balance of multi-dimensional losses. Through the constraint of the joint loss function, the prior knowledge obtained in the pre-training stage is preserved in the fine-tuning process of the complete prediction model, which is composed of a pre-trained general model, an information entropy optimization module, a domain knowledge dynamic injection module, and a segmentation module. Dynamic constraints are also imposed on the decoder optimization process. The decoder parameters are iteratively optimized by backpropagation using a joint loss function. Based on the finely tuned complete prediction model, the final segmentation result of breast lesion-related tissue structures is obtained based on high-dimensional feature vectors.

[0020] As a preferred embodiment of the present invention, the joint loss function L1 is: ; In the formula, , These represent the cross-entropy loss of the output results of the complete prediction model before and after fine-tuning, respectively. , These represent the Dice loss of the complete prediction model before and after fine-tuning, respectively. This represents the KL divergence between the parameter distributions of the complete prediction model before and after fine-tuning; Indicates the dynamically adjusted parameters; This represents the truncation function; in, Dynamically adjusts with the number of training iterations: ; In the formula, e represents the natural constant; s represents the number of training iterations; and β is the preset scheduling coefficient.

[0021] The beneficial effects of this invention are: This invention uses an information entropy optimization module to calculate entropy values, perform binary soft matching, and merge or weighted compression processing on coded image tokens, reducing the interference of background and redundant regions on the model training and prediction process. At the same time, through a domain knowledge dynamic injection module, expert knowledge representations generated from multi-source medical knowledge are semantically aligned with image tokens in the optimized image token sequence across modalities, enabling expert knowledge to participate in the image feature enhancement process, thereby improving the model's ability to identify and segment complex tissue structures such as tumors, blood vessels, and dense glands in breast cancer medical images.

[0022] This invention, through the synergistic effect of information entropy optimization, dynamic injection of domain knowledge, and a segmentation module with dynamic constraints, enables a pre-trained general model to better adapt to breast cancer medical images under different modalities and imaging conditions; wherein, the dynamic constraints are achieved by adjusting parameters dynamically with the number of training iterations. With the truncation mechanism, prior knowledge gained in the pre-training stage is preserved during the fine-tuning of the complete prediction model, preventing the decoder parameters from deviating excessively from the pre-training state. This improves cross-modal generalization ability, adaptability to complex scenarios, and stability of segmentation results, providing reliable auxiliary basis for clinical diagnosis and treatment decisions. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating the principle of this invention; Figure 2 This is a schematic diagram of the information entropy optimization module. Detailed Implementation

[0024] The embodiments of the present invention will be further described below with reference to the accompanying drawings: Example 1: As Figure 1As shown, a breast cancer medical image segmentation method that integrates expert knowledge and entropy optimization includes the following steps: S1. Obtain the text description information and breast cancer medical images of different modalities corresponding to the same sample. Perform word embedding processing on the text description information and slice embedding processing on the breast cancer medical images to obtain the text tokens sequence and the image tokens sequence, where one sample represents a breast cancer case. S2. Input the text tokens sequence and the image tokens sequence into the corresponding encoders to obtain encoded text tokens and encoded image tokens. S3. Construct an information entropy optimization module, introduce an information entropy optimization mechanism to calculate the entropy value of the encoded image tokens, obtain the information entropy value corresponding to each encoded image token, and use a binary soft matching method to match and merge the encoded image tokens based on the similarity of the information entropy values ​​to obtain an optimized image tokens sequence. Multimodal input features are obtained by jointly adjusting the encoded text tokens and optimized image tokens sequences through an interpolation control mechanism. S4. Construct a domain knowledge dynamic injection module to generate expert knowledge representation based on multi-source medical knowledge, and achieve cross-modal semantic alignment between multimodal input features and expert knowledge through the cross-attention mechanism of graph-text interactive query to obtain semantically enhanced multimodal features; S5. Using semantically enhanced multimodal features as input, logical reasoning and effective feature extraction are performed through a pre-trained general model, and a high-dimensional feature vector of breast cancer medical images is generated by the pre-trained general model. S6. Input the high-dimensional feature vector into the segmentation module with dynamic constraints, perform cross-modal adaptation and scene adaptability optimization, and obtain the final segmentation result of breast lesion-related tissue structures.

[0025] In S1, the text description information includes: patient case information for individual samples (e.g., medical records, medical orders), general medical knowledge for all samples (e.g., authoritative medical knowledge including online resources, textbooks, and imaging reports); breast cancer medical images include MRI images, CT images, and near-infrared images. The word embedding process specifically involves: segmenting the text description information to obtain text units composed of words, characters, or sub-words, and mapping each text unit to a fixed-dimensional text token vector through the word embedding layer. Based on multiple text token vectors, a text token sequence is formed to convert discrete text into text feature representation in a continuous vector space. The segmented embedding process specifically involves: performing standardized preprocessing on breast cancer medical images of each modality, including size normalization, grayscale normalization, and noise suppression; then dividing the complete image into multiple non-overlapping image segments using an image segmentation method; and vectorizing each image segment through a linear projection layer to obtain image tokens vectors of fixed dimensions; and forming an image tokens sequence based on multiple image tokens vectors.

[0026] In S2, the text tokens sequence is input into the text encoder to obtain encoded text tokens; the image tokens sequence is input into the image encoder to obtain encoded image tokens; wherein, the encoded text tokens and the encoded image tokens serve as encoded feature representations of text information and image information, respectively.

[0027] In S3, the process of obtaining multimodal input features is as follows: In the information entropy optimization module, the encoded image tokens are normalized. Then, the ratio of the numerical value of each dimension of each encoded image token to the sum of the vectors is calculated. Finally, the information entropy corresponding to each image slice is calculated using the following formula to quantify the feature information distribution of different image tokens: ; ; In the formula, H represents the information entropy value of the encoded image tokens; d represents the dimension of the encoded image tokens, and i and j are the indices of the dimension; Represents the normalized probability of the i-th dimension value of the encoded image Tokens; , These represent the values ​​of the i-th and j-th dimensions of the encoded image Tokens, respectively. This represents the normalized probability function; generally, It will not be 0. In special cases (such as when a segment of an image consists entirely of backgrounds with no information), a very small positive constant can be added to the denominator. To avoid the denominator being 0, that is: ; Based on the information entropy distribution characteristics of coded image tokens, a binary soft matching method is used to compare the information entropy distribution characteristics between different coded image tokens, resulting in M ​​groups of candidate similar image tokens. The similarity is determined by the Euclidean distance of the information entropy values ​​of each candidate similar image token group. K groups of candidate similar image tokens with the highest information entropy similarity are selected from the M groups. These K groups are then merged or weighted and compressed to obtain an optimized image token sequence. This reduces the interference of redundant background information on the segmentation process and improves the effective feature representation capability, where K ≤ M. For example, for a conventional medium-resolution image, when M is 16, K is 4.

[0028] The process of obtaining M candidate similar image token groups using the binary soft matching method is as follows: First, all coded image tokens are divided into two disjoint token subsets A and B of equal size to construct a binary matching structure; using the information entropy value corresponding to each coded image token as the feature matching basis, the absolute difference between the information entropy values ​​of tokens in subsets A and B is calculated to quantify the similarity of information entropy values ​​of different image tokens. The smaller the absolute difference, the higher the similarity; sorted by absolute difference from smallest to largest (corresponding to similarity from highest to lowest), the M matching edges with the highest similarity are retained as candidates, and each token in subset A is constrained to participate in only one matching edge, resulting in M ​​candidate similar image token groups, each containing a pair of matched tokens.

[0029] Through an interpolation adjustment mechanism, linear interpolation is used to align the number of image tokens and text tokens in the encoded text tokens and optimized image tokens sequences, thereby obtaining multimodal input features and achieving redundant region compression and effective information preservation. The structural diagram of the information entropy optimization module when using weighted compression processing is shown below. Figure 2 As shown, the arrow in the lower right corner indicates that the information entropy optimization module finally outputs the optimized image tokens sequence.

[0030] In S4, the process of obtaining semantically enhanced multimodal features in the domain knowledge dynamic injection module is as follows: Multi-source medical knowledge includes the clinical experience of medical personnel, imaging features in medical literature, and individual clinical information of patients. Multi-source medical knowledge is fused and processed, and the fused multi-source medical knowledge is uniformly modeled and semantically integrated through a large language model (LLM) to generate an expert knowledge representation corresponding to each sample. For example, the open-source medical-specific large language model BioBERT can be used as the foundation for LLM to complete multi-source medical knowledge modeling. First, authoritative medical knowledge (online medical literature, professional textbooks, breast imaging diagnosis reports) and patient information (electronic medical records, clinical medical orders) are decomposed into standardized text fragments. These fragments are then concatenated according to a fixed Prompt template to form a unified input text, which is then fed into the LLM. The Prompt contains breast lesion feature associations and clinical diagnostic prior rules to guide the model to output medical domain-specific semantics. Inside the LLM, a multi-layer Transformer encoder completes the text semantic encoding, performs global semantic integration of scattered multi-source text information, eliminates the expression bias of texts from different sources, and outputs a fixed-dimensional hidden layer semantic feature vector as text tokens representing expert knowledge. During the training phase, the weights of the LLM foundation backbone are frozen, and only the top-level encoding layer is fine-tuned to adapt the dimensions of the image tokens.

[0031] Based on the generated expert knowledge representation and multimodal input features, to achieve a balance between accurate image-text alignment and computational performance, a cross-attention mechanism of image-text interactive query is adopted to realize bidirectional semantic association between multimodal input features and expert knowledge: using image tokens in the optimized image tokens sequence as query features, relevant text content in the expert knowledge representation is matched; then, using text tokens in the expert knowledge representation as query features, relevant image grayscale regions in the optimized image tokens sequence are matched. A bidirectional matching relationship between image grayscale regions and expert knowledge representation is established in the feature space, and the features of different regions in the optimized image tokens sequence are accurately bound to the corresponding expert knowledge representation, completing the cross-modal semantic alignment between multimodal input features and expert knowledge, and obtaining semantically enhanced multimodal features.

[0032] The cross-attention mechanism of the image-text interactive query can set up 8 attention heads to map the grayscale region of the image and the text features of the breast lesion to the same QKV space. The image and text features output by bidirectional matching adopt element-wise addition and fusion, which is different from ordinary splicing and fusion, and can retain the fine-grained correlation information between the local lesion and clinical knowledge.

[0033] The domain knowledge dynamic injection module is trained by first performing self-supervised training on a publicly available medical image dataset (e.g., the TCIA breast MRI dataset). Then, image data with medical image analysis reports is used as the training set to perform consistency training between medical images and expert knowledge on the domain knowledge dynamic injection module. During the consistency training process, a cross-modal bidirectional contrastive loss function is constructed to constrain and optimize the semantic consistency between the image token sequence and the text tokens represented by expert knowledge.

[0034] The cross-modal bidirectional contrast loss function is obtained by weighting the image-to-text contrast loss function and the text-to-image contrast loss function. The image-to-text contrast loss function is: ; In the formula, This represents the image-to-text contrast loss value for the m-th sample; Represents the cosine similarity function; This represents the image tokens corresponding to the m-th output result; , These represent the text tokens that represent the expert knowledge representations corresponding to the m-th and n-th input samples, respectively. The temperature parameter can be 0.07; m and n represent the index variables of the samples within the batch; N represents the total number of samples in the training batch. The text-to-image contrast loss function is: ; In the formula, This represents the text-to-image contrast loss value for the m-th sample; This represents the text Tokens corresponding to the m-th output result; , These represent the image tokens in the optimized image tokens sequence corresponding to the m-th and n-th input samples, respectively; The weighted average yields the joint constraint loss function L for cross-modal bidirectional contrastive loss: ; In the formula, The loss weight parameter can be set to 0.5.

[0035] In S5, the process of generating high-dimensional feature vectors for breast cancer medical images is as follows: the semantically enhanced multimodal features are adjusted to a standardized tensor format that matches the input dimension of the pre-trained general model. The channel dimension alignment, spatial size adaptation, and batch dimension processing of the multimodal features are completed to obtain standardized multimodal features that are adapted to the input of the pre-trained general model. Logical reasoning and effective feature extraction are performed using the pre-trained general model to generate high-dimensional feature vectors for breast cancer medical images.

[0036] The pre-trained general model can use any pre-trained large model (such as SAM, CLIP, and other vision-language foundational models) as its base. Its pre-training is performed through self-supervised or contrastive learning on large-scale unlabeled or weakly labeled general vision-language data, enabling the model to learn general visual feature extraction and cross-modal semantic alignment capabilities in advance. In this embodiment, PyTorch 2.10 is used as the implementation framework, and the pre-trained general model is selected as the feature inference backbone. Its image encoder uses a pre-trained ViT-Base model to initialize weights, and training is performed on a single NVIDIA H20 GPU (96GB) with an initial learning rate of 1×10⁻⁶. -5 The batch size was set to 2, the model was trained for 15,000 steps (approximately 2.5 epochs), and the weights of the pre-trained general model were kept frozen during the segmentation fine-tuning stage. The model was adapted and optimized for the breast cancer medical image segmentation task only by adding an information entropy optimization module, a domain knowledge dynamic injection module, and a segmentation module with dynamic constraints.

[0037] In S6, the process of obtaining the final segmentation results of the breast lesion-related tissue structure is as follows: The high-dimensional feature vector output by the pre-trained general model is input into the segmentation module with dynamic constraints. The segmentation module includes a decoder and a segmentation head as the output layer. The high-dimensional feature vector is spatially resolved and its features are decoded by the decoder, and the pixel-wise class probability is output by the segmentation head to generate the initial segmentation prediction result. The segmentation head, serving as the output layer, can employ a three-layer convolutional stacked structure: the 768-channel multimodal feature map output from the decoder is sequentially fed into two 3×3 convolutional transition layers and one 1×1 output convolutional layer; the first convolutional layer is configured with 256 3×3 convolutional kernels, a stride of 1, and edge padding of 1, and the convolutional output is processed by layer normalization before being connected to the GELU activation function; the second convolutional layer is configured with 128 3×3 convolutional kernels, a stride of 1, and edge padding of 1, and is also paired with layer normalization and the GELU activation function; the third layer is a 1×1 pointwise convolutional output layer with 4 convolutional kernels, corresponding to four segmentation targets: breast background, normal glandular tissue, microcalcifications, and malignant tumors. After the convolutional output is normalized by the Softmax activation function, a predicted probability map of each pixel corresponding to different breast tissue categories is obtained.

[0038] After training the domain knowledge dynamic injection module, an encoder, information entropy optimization module, domain knowledge dynamic injection module, and segmentation module are added to the pre-trained general model to obtain a complete prediction model. Then, the complete prediction model is fine-tuned. During the fine-tuning stage, the image encoder and text encoder are frozen, and only the decoder parameters in the segmentation module are trained. A joint loss function combining cross-entropy loss, Dice loss, and KL divergence is adopted. Dynamic parameters and a truncation mechanism are introduced to achieve a balance of multi-dimensional losses. Through the constraint of the joint loss function, the prior knowledge obtained in the pre-training stage is preserved in the fine-tuning process of the complete prediction model, which is composed of a pre-trained general model, an information entropy optimization module, a domain knowledge dynamic injection module, and a segmentation module. Dynamic constraints are also imposed on the decoder optimization process. The decoder parameters are iteratively optimized by backpropagation using a joint loss function. Based on the finely tuned complete prediction model, the final segmentation results of breast lesion-related tissue structures (tumors, blood vessels, and dense glands, etc.) are obtained based on high-dimensional feature vectors.

[0039] The joint loss function L1 is: ; In the formula, , These represent the cross-entropy loss of the output results of the complete prediction model before and after fine-tuning, respectively. , These represent the Dice loss of the complete prediction model before and after fine-tuning, respectively. This represents the KL divergence between the parameter distributions of the complete prediction model before and after fine-tuning; Indicates the dynamically adjusted parameters; This represents the truncation function; in, Dynamically adjusts with the number of training iterations: ; In the formula, e represents the natural constant; s represents the number of training iterations (the current number of training iterations, counting from 0); β is a preset scheduling coefficient, which can be 1×10⁻⁶. -4 .

[0040] Dynamic constraints are cutoff parameters that are dynamically adjusted during training iterations. This limits the loss ratio during the decoder optimization process to a controllable range, in order to preserve pre-trained prior knowledge and prevent overfitting.

[0041] Example 2: A breast cancer medical image segmentation device integrating expert knowledge and entropy optimization, comprising: One or more processors; Memory, used to store one or more computer programs; When one or more programs are executed by one or more processors, the one or more processors execute the method in Example 1.

[0042] Example 3: A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the method in Example 1.

[0043] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can make equivalent substitutions or modifications based on the technical solution and concept of the present invention within the scope of the technology disclosed in the present invention, and such modifications should also be considered to fall within the scope of protection of the present invention.

Claims

1. A breast cancer medical image segmentation method integrating expert knowledge and entropy optimization, characterized in that, Includes the following steps: S1. Obtain the text description information corresponding to the same sample and the breast cancer medical images of different modalities. Perform word embedding processing on the text description information and slice embedding processing on the breast cancer medical images to obtain the text tokens sequence and the image tokens sequence. One sample represents a breast cancer case. During slice embedding processing, perform standardized preprocessing on the breast cancer medical images of each modality and divide the complete image into multiple non-overlapping image slices. S2. Input the text tokens sequence and the image tokens sequence into the corresponding encoders to obtain encoded text tokens and encoded image tokens. S3. Construct an information entropy optimization module, introduce an information entropy optimization mechanism to calculate the entropy value of the encoded image tokens, obtain the information entropy value corresponding to each encoded image token, and use a binary soft matching method to match and merge the encoded image tokens based on the similarity of the information entropy values ​​to obtain an optimized image tokens sequence. Multimodal input features are obtained by jointly modulating the encoded text tokens and optimized image tokens sequences through an interpolation modulation mechanism. The process is as follows: In the information entropy optimization module, the encoded image tokens are normalized, then the ratio of the value of each dimension in each encoded image token to the sum of the vectors is calculated, and then the information entropy corresponding to each image slice is calculated using the following formula: ； ； In the formula, H represents the information entropy value of the encoded image tokens; d represents the dimension of the encoded image tokens, and i and j are the indices of the dimension; Represents the normalized probability of the i-th dimension value of the encoded image Tokens; , These represent the values ​​of the i-th and j-th dimensions of the encoded image Tokens, respectively. This represents the normalized probability function; Based on the information entropy distribution characteristics of coded image tokens, a binary soft matching method is used to compare the information entropy distribution characteristics between different coded image tokens, resulting in M ​​groups of candidate similar image tokens. The similarity is determined by the Euclidean distance of the information entropy values ​​of each candidate similar image token group. K groups of candidate similar image tokens with the highest information entropy similarity are selected from the M groups, and these K groups are then merged or weighted and compressed to obtain an optimized image token sequence, where K ≤ M. By using an interpolation adjustment mechanism, linear interpolation is used to align the number of image tokens and text tokens in the encoded text tokens and optimized image tokens sequences to obtain multimodal input features. S4. Construct a domain knowledge dynamic injection module to generate expert knowledge representation based on multi-source medical knowledge, and achieve cross-modal semantic alignment between multimodal input features and expert knowledge through the cross-attention mechanism of graph-text interactive query to obtain semantically enhanced multimodal features; S5. Using semantically enhanced multimodal features as input, logical reasoning and effective feature extraction are performed through a pre-trained general model, and a high-dimensional feature vector of breast cancer medical images is generated by the pre-trained general model. S6. Input the high-dimensional feature vector into the segmentation module with dynamic constraints, perform cross-modal adaptation and scene adaptability optimization, and obtain the final segmentation result of breast lesion-related tissue structures.

2. The breast cancer medical image segmentation method integrating expert knowledge and entropy optimization according to claim 1, characterized in that, In S1, the text description information includes: patient case information for individual samples, and general medical knowledge for all samples; breast cancer medical images include MRI images, CT images, and near-infrared images; The word embedding process specifically involves: segmenting the text description information to obtain text units composed of words, characters, or sub-words, and mapping each text unit to a fixed-dimensional text token vector through the word embedding layer. Based on multiple text token vectors, a text token sequence is formed to convert discrete text into text feature representation in a continuous vector space. The segmented embedding process specifically involves: vectorizing each image segment through a linear projection layer to obtain image token vectors of fixed dimensions, and forming an image token sequence based on multiple image token vectors.

3. The breast cancer medical image segmentation method integrating expert knowledge and entropy optimization according to claim 2, characterized in that, In step S2, the text tokens sequence is input into the text encoder to obtain encoded text tokens; the image tokens sequence is input into the image encoder to obtain encoded image tokens; wherein, the encoded text tokens and the encoded image tokens are used as encoded feature representations of text information and image information, respectively.

4. The breast cancer medical image segmentation method integrating expert knowledge and entropy optimization according to claim 1, characterized in that, In S4, the process of obtaining semantically enhanced multimodal features in the domain knowledge dynamic injection module is as follows: Multi-source medical knowledge includes the clinical experience of medical personnel, imaging features in medical literature, and individual clinical information of patients. Multi-source medical knowledge is fused and processed, and the fused multi-source medical knowledge is uniformly modeled and semantically integrated through a large language model to generate an expert knowledge representation corresponding to each sample. Based on the generated expert knowledge representation and multimodal input features, a cross-attention mechanism for image-text interactive queries is adopted to achieve bidirectional semantic association between multimodal input features and expert knowledge: using image tokens in the optimized image tokens sequence as query features, relevant text content in the expert knowledge representation is matched; then, using text tokens in the expert knowledge representation as query features, relevant image grayscale regions in the optimized image tokens sequence are matched. A bidirectional matching relationship between image grayscale regions and expert knowledge representation is established in the feature space, and the features of different regions in the optimized image tokens sequence are accurately bound to the corresponding expert knowledge representation, completing the cross-modal semantic alignment between multimodal input features and expert knowledge, and obtaining semantically enhanced multimodal features.

5. The breast cancer medical image segmentation method integrating expert knowledge and entropy optimization according to claim 4, characterized in that, The domain knowledge dynamic injection module is trained by first performing self-supervised training on a publicly available medical image dataset. Then, image data with medical image analysis reports is used as the training set to perform consistency training between medical images and expert knowledge on the domain knowledge dynamic injection module. During the consistency training process, a cross-modal bidirectional contrastive loss function is constructed to constrain and optimize the semantic consistency between the image token sequence and the text tokens represented by expert knowledge.

6. The breast cancer medical image segmentation method integrating expert knowledge and entropy optimization according to claim 5, characterized in that, The cross-modal bidirectional contrast loss function is obtained by weighting the image-to-text contrast loss function and the text-to-image contrast loss function. The image-to-text contrast loss function is: ； In the formula, This represents the image-to-text contrast loss value for the m-th sample; Represents the cosine similarity function; This represents the image tokens corresponding to the m-th output result; , These represent the text tokens that represent the expert knowledge representations corresponding to the m-th and n-th input samples, respectively. The temperature parameter is represented; m and n represent the index variables of the samples within the batch; N represents the total number of samples in the training batch. The text-to-image contrast loss function is: ； In the formula, This represents the text-to-image contrast loss value for the m-th sample; This represents the text Tokens corresponding to the m-th output result; , These represent the image tokens in the optimized image tokens sequence corresponding to the m-th and n-th input samples, respectively. The weighted average yields the joint constraint loss function L for cross-modal bidirectional contrastive loss: ； In the formula, These are the loss weight parameters.

7. The breast cancer medical image segmentation method integrating expert knowledge and entropy optimization according to claim 1, characterized in that, In S5, the process of generating high-dimensional feature vectors for breast cancer medical images is as follows: the semantically enhanced multimodal features are adjusted to a standardized tensor format that matches the input dimension of the pre-trained general model; channel dimension alignment, spatial size adaptation, and batch dimension processing of the multimodal features are completed to obtain standardized multimodal features that adapt to the input of the pre-trained general model; logical reasoning and effective feature extraction are performed using the pre-trained general model to generate high-dimensional feature vectors for breast cancer medical images.

8. The breast cancer medical image segmentation method integrating expert knowledge and entropy optimization according to claim 5, characterized in that, In step S6, the process of obtaining the final segmentation result of the breast lesion-related tissue structure is as follows: The high-dimensional feature vector output by the pre-trained general model is input into the segmentation module with dynamic constraints. The segmentation module includes a decoder and a segmentation head as the output layer. The high-dimensional feature vector is spatially resolved and its features are decoded by the decoder, and the pixel-wise class probability is output by the segmentation head to generate the initial segmentation prediction result. After training the domain knowledge dynamic injection module, an encoder, information entropy optimization module, domain knowledge dynamic injection module, and segmentation module are added to the pre-trained general model to obtain a complete prediction model. Then, the complete prediction model is fine-tuned. During the fine-tuning stage, the image encoder and text encoder are frozen, and only the decoder parameters in the segmentation module are trained. A joint loss function combining cross-entropy loss, Dice loss, and KL divergence is adopted. Dynamic parameters and a truncation mechanism are introduced to achieve a balance of multi-dimensional losses. Through the constraint of the joint loss function, the prior knowledge obtained in the pre-training stage is preserved in the fine-tuning process of the complete prediction model, which is composed of a pre-trained general model, an information entropy optimization module, a domain knowledge dynamic injection module, and a segmentation module. Dynamic constraints are also imposed on the decoder optimization process. The decoder parameters are iteratively optimized by backpropagation using a joint loss function. Based on the finely tuned complete prediction model, the final segmentation result of breast lesion-related tissue structures is obtained based on high-dimensional feature vectors.

9. The breast cancer medical image segmentation method integrating expert knowledge and entropy optimization according to claim 8, characterized in that, The joint loss function L1 is: ； In the formula, , These represent the cross-entropy loss of the output of the complete prediction model before and after fine-tuning, respectively. , These represent the Dice loss of the complete prediction model before and after fine-tuning, respectively. This represents the KL divergence between the parameter distributions of the complete prediction model before and after fine-tuning; Indicates the dynamically adjustable parameters; This represents the truncation function; in, Dynamically adjusts with the number of training iterations: ； In the formula, e represents the natural constant; s represents the number of training iterations; and β is the preset scheduling coefficient.

Images