Medical image segmentation method, electronic device and computer readable storage medium

Abstract

The present disclosure relates to a medical image segmentation method, an electronic device and a computer readable storage medium, the method comprising: determining context semantic feature information of each subword in medical text information corresponding to a medical image to be segmented and problem semantic feature information corresponding to semantic perspective direction information; performing attention aggregation on each problem semantic feature information and the context semantic feature information of all subwords to obtain corresponding semantic perspective feature information; inputting the medical image to be segmented into an image encoder to sequentially obtain feature maps output by each scale level; inputting the feature map output by the last scale level of the image encoder and the feature map obtained by modulating all semantic perspective feature information into an image decoder to obtain a target feature map output by the last scale level of the image decoder; and performing mapping processing on the target feature map by using a prediction head to generate a segmentation result of a target structure.

Description

Technical Field

[0001] This disclosure relates to the field of computer vision technology, and more specifically, to a medical image segmentation method, an electronic device, and a computer-readable storage medium. Background Technology

[0002] In modern medical imaging-assisted diagnosis, CT imaging is widely used for the examination and evaluation of skeletal system-related diseases due to its fast imaging speed and high spatial resolution. As a vital support and mobilization system of the human body, the morphology and integrity of the skeletal structure directly affect multiple clinical aspects, including fracture diagnosis, disease assessment, surgical planning, and postoperative rehabilitation. Therefore, accurate analysis of skeletal structures in CT images is of significant clinical importance. In practical clinical applications, skeletal structures typically require precise segmentation from CT images to allow physicians to visually observe bone morphology, locate lesion areas, and perform quantitative analysis. However, due to the complexity of skeletal morphology, significant individual differences, and diverse imaging manifestations influenced by factors such as disease, degeneration, or implants, manual segmentation is often time-consuming and highly dependent on the operator's professional experience, making it difficult to meet the efficiency and consistency requirements of large-scale clinical applications.

[0003] With the continuous growth of medical image data, deep learning-based automatic structural segmentation methods have been widely studied in the field of medical image analysis. Encoder-decoder models such as U-Net and nnU-Net, represented by convolutional neural networks, can automatically learn high-level semantic features from medical images through multi-layer feature extraction and multi-scale information fusion, significantly improving segmentation accuracy and efficiency in anatomical structure segmentation tasks such as bones. In recent years, with the application of the Transformer architecture in computer vision, models such as UNETR and TransUNet have further enhanced the segmentation ability of complex anatomical structures by modeling the global context of images through self-attention mechanisms. However, most existing deep learning-based medical image segmentation methods mainly rely on the image data itself for modeling, without fully considering the medical text information generated simultaneously with the images. In actual clinical procedures, imaging examinations are usually accompanied by diagnostic reports written by professional physicians. These reports contain high-level semantic descriptions of anatomical morphology, lesion location, and abnormal features, which can supplement the image content from a clinical cognitive perspective.

[0004] Although some studies have begun to attempt to incorporate textual information into medical image segmentation tasks, existing methods mostly adopt more direct fusion strategies, such as aligning visual features and textual features as a whole through cross-attention or bidirectional cross-attention mechanisms, or encoding the entire text into a single semantic vector and using it as conditional information on visual features.

[0005] While the aforementioned text utilization methods can introduce semantic priors to some extent, the high-dimensionality, complexity, and multi-granularity of text semantics mean that directly applying the overall semantic distribution to the visual feature space can easily lead to mutual interference between different semantic information, resulting in semantic aliasing. Furthermore, coarse-grained semantic alignment mechanisms struggle to clearly distinguish the guiding role of different text descriptions on specific spatial regions, potentially causing problems such as unstable gradient update directions and unclear optimization objectives. This limits the effective utilization of textual information in medical image structure segmentation tasks.

[0006] While the aforementioned methods have made some progress in medical image structure segmentation, they still have limitations in their utilization of textual information. Therefore, there is a need to provide a segmentation method based on the fusion of medical image and textual information to improve the stability and accuracy of segmentation in complex structure segmentation tasks. Summary of the Invention

[0007] One objective of this disclosure is to provide a new technical solution for medical image segmentation.

[0008] According to a first aspect of this disclosure, a medical image segmentation method is provided, comprising: Determine the contextual semantic features of each word in the medical text information corresponding to the medical image to be segmented, as well as the question semantic features corresponding to the semantic perspective direction information. The semantic perspective direction information is at least one of position, shape, boundary, signal, density, and surrounding tissue relationship. Attention is used to aggregate the semantic features of each question and the contextual semantic features of all sub-words to obtain the corresponding semantic perspective features. The medical image to be segmented is input into the image encoder, and feature maps output at each scale level are obtained sequentially. The input of the next scale level in the image encoder is the feature map output by the previous scale level in the image encoder and the feature map obtained by modulation processing of all semantic viewpoint feature information. The feature map output from the last scale level of the image encoder and the feature map obtained by modulating all semantic viewpoint feature information are input into the image decoder to obtain the target feature map output from the last scale level of the image decoder. The input of the next scale level in the image decoder is the feature map output from the previous scale level in the image decoder and the feature map output from the corresponding scale level in the image encoder. The target feature map is mapped using a prediction head to generate segmentation results for the target structure.

[0009] Optionally, determining the contextual semantic feature information of each word in the medical text information corresponding to the medical image to be segmented, as well as the question semantic feature information corresponding to the semantic perspective direction information, includes: Based on the medical text information, word segmentation is performed to obtain sub-word sequence information with a set fixed sequence length; The sub-word sequence information is input into a text encoding model to obtain the contextual semantic feature information corresponding to each sub-word; and, The medical text information and the set prompts used to guide semantic analysis are input into the set language model to obtain the question information corresponding to the semantic perspective direction information; The question information corresponding to each semantic perspective direction information is input into the text encoding model to obtain the corresponding question semantic feature information.

[0010] Optionally, the semantic feature information of each question includes a global semantic feature vector for the corresponding question, wherein, The process of attention-aggregating the semantic feature information of each question and the contextual semantic feature information of all sub-words to obtain corresponding semantic perspective feature information includes: Using the global semantic feature vector of each question as the Query, and the contextual semantic feature information of all sub-words as the Key and Value, a scaling dot product attention-based calculation is performed to obtain the corresponding semantic perspective feature information.

[0011] Optionally, the method further includes: Determine the correlation information between each pair of semantic perspective feature information; Determine the projection information of each semantic perspective feature information in the Value space; Based on the projection information of all relevant information and semantic perspective feature information in the Value space, the enhanced semantic feature information is determined; The enhanced semantic feature information is fused with the semantic perspective feature information to obtain the semantic perspective feature correction information.

[0012] Optionally, determining the correlation information between every two semantic perspective feature information includes: Determine the projection information of all semantic perspective feature information in the Query space and the projection information of all semantic perspective feature information in the Key space; Based on the projection information of all semantic perspective feature information in the Query space and the projection information of all semantic perspective feature information in the Key space, the correlation information between each pair of semantic perspective feature information is determined.

[0013] Optionally, the modulation processing of the input to the next scale level in the image encoder, which is the feature map output from the previous scale level in the image encoder and all semantic viewpoint feature information, includes: Determine the matching degree between the feature map output at each scale level and the feature information of each semantic perspective; Based on the matching degree between the feature map output at the corresponding scale level and each semantic perspective feature information, determine the weight distribution information of each semantic perspective feature information in the feature map output at the corresponding scale level. Based on the semantic viewpoint feature information, determine the channel scaling parameter and channel offset parameter in the modulation parameter space corresponding to each feature map; Based on the weight distribution information of each semantic perspective feature information in the feature map output at the corresponding scale level, and the channel scaling parameters and channel offset parameters of each semantic perspective feature information mapped to the modulation parameter space corresponding to the feature map, the feature map after modulation processing at the corresponding scale level is determined.

[0014] Optionally, the processing steps for the input at the next scale level in the image decoder include: The feature map output from the previous scale level in the image decoder is fused with the feature map output from the corresponding scale level in the image encoder to obtain a fused feature map. The fusion process is either feature concatenation or element-wise addition. The fused feature map is integrated by convolutional blocks to obtain an integrated feature map, wherein the convolutional blocks include multiple convolutions, normalization and nonlinear activation operations.

[0015] Optionally, the step of mapping the target feature map using the prediction head to generate a segmentation result of the target structure includes: Using the prediction head, class prediction is performed on each location point in the target feature map, resulting in the segmentation class prediction result for each location point in the target feature map.

[0016] According to a second aspect of this disclosure, an electronic device is provided, including a memory and a processor, the memory storing a computer program for controlling the processor to operate in order to perform the method according to any one of the first aspects of this disclosure.

[0017] According to a third aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method as described in any one of the first aspects.

[0018] The medical image segmentation method disclosed herein enables the diagnostic semantics in medical text information to participate in the image feature modeling process in a structured manner through semantic perspective decomposition, modulation processing of feature maps output at each scale level in the image encoder, and modulation processing of all semantic perspective feature information. This avoids the problem of medical text information being used in a coarse-grained manner only as an overall condition or additional input, thereby improving the practical usability of medical text information in image segmentation tasks and realizing the effective guidance of medical text information for image segmentation.

[0019] The features and advantages of the embodiments of this specification will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments of this specification and, together with their description, serve to explain the principles of these embodiments.

[0021] Figure 1 A flowchart of a medical image segmentation method according to an embodiment of the present disclosure is shown.

[0022] Figure 2 A schematic diagram of the architecture for medical image segmentation according to an embodiment of the present disclosure is shown.

[0023] Figure 3 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0024] Various exemplary embodiments of this specification will now be described in detail with reference to the accompanying drawings.

[0025] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the embodiments of this specification or their application or use.

[0026] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0027] To address the aforementioned technical issues, this disclosure provides a medical image segmentation method. By semantic perspective decomposition and modulation processing of feature maps output at each scale level in the image encoder with all semantic perspective feature information, the diagnostic semantics in medical text information can participate in the image feature modeling process in a structured manner. This avoids the problem of medical text information being used only as a whole condition or additional input in a coarse-grained manner, thereby improving the practical usability of medical text information in image segmentation tasks and realizing the effective guidance of medical text information for image segmentation.

[0028] Figure 1 A flowchart of a medical image segmentation method according to an embodiment of the present disclosure is shown. Figure 1 As shown, the method includes steps S110 to S150.

[0029] Step S110: Determine the contextual semantic features of each word in the medical text information corresponding to the medical image to be segmented, as well as the question semantic features corresponding to the semantic perspective direction information. The semantic perspective direction information is at least one of position, shape, boundary, signal, density, and surrounding tissue relationship.

[0030] This embodiment does not directly introduce medical text information as a whole semantic vector or simple conditional information into the image segmentation model. Instead, it introduces a semantic perspective decomposition mechanism on the text side, breaking down the complex, high-dimensional semantic content of the medical text information into multiple semantic perspective representations with clear diagnostic orientations. Through a question-guided approach, information of different semantic dimensions such as location, morphology, boundary, signal / density, internal structure, and surrounding tissue relationships are extracted from the medical text information. This ensures that each semantic perspective corresponds to a type of interpretable and independently modelable diagnostic criterion, thereby avoiding the mixing of multiple heterogeneous semantics into the model as a whole.

[0031] The medical image to be segmented can be a three-dimensional image or a two-dimensional image. When the medical image to be segmented is a three-dimensional CT or MRI image, H, W, and D represent the height, width, and number of slices / depth of the medical image to be segmented, respectively. When the medical image to be segmented is a two-dimensional image, Simultaneously, it acquires metadata about the medical image to be segmented, including pixel spacing, voxel spacing, orientation matrix, and origin coordinates.

[0032] Obtain the medical text information corresponding to the medical image to be segmented, and denote it as text. Medical text information can originate from manual annotation or be automatically generated by large language models from images or related data, and then reviewed and corrected by professionals. Medical text information can be at least one of the following: imaging reports, descriptions of examination findings, or structured conclusions.

[0033] To obtain a unified model input representation, the medical images to be segmented are preprocessed. Preprocessing includes at least one of registration, resampling, cropping, ROI extraction, and intensity normalization.

[0034] When a single medical image contains multiple sequences (e.g., multi-phase CT or MRI multimodal sequences) or has follow-up images, the medical image to be processed can be registered to a reference image space. Rigid or affine registration is used to determine the spatial transformation Φ, which allows the moving image to... Compared with reference image Align in the same coordinate system:

[0035] in, This indicates that a spatial transformation is applied to the moving image and interpolation resampling is performed.

[0036] Similarity metrics for registration can be mutual information (MI) or normalized mutual information (NMI), combined with a multi-resolution pyramid strategy to improve stability. For CT same-modal registration, mean squared error (MSE) can also be used as a metric.

[0037] To eliminate voxel spacing differences caused by different devices or protocols, medical images are resampled to the target resolution. The resampled medical image is recorded as... Trilinear interpolation (3D) or bilinear interpolation (2D) is used for image intensity, and nearest neighbor interpolation is used for labels or segmentation ground truth values ​​to avoid class aliasing.

[0038] Image intensity is normalized to reduce scanning protocol variations. Using Z-score as an example, the mean and standard deviation of the image to be segmented (or its foreground region) are calculated:

[0039] in, voxels The intensity value, For statistical regions (e.g., the entire image or the foreground region). For the region voxels, To prevent extremely small constants from being divided by zero, the normalized result is:

[0040] The normalized image is denoted as In CT scenarios, intensity truncation can also be applied before the Z-score, for example, truncating the HU value to... The range is adjusted to suppress the influence of extreme values ​​before the above standardization is performed: .

[0041] In some embodiments, step S110 specifically includes: segmenting medical text information to obtain sub-word sequence information of a set fixed sequence length; inputting the sub-word sequence information into a text encoding model to obtain contextual semantic feature information corresponding to each sub-word; and inputting medical text information and set prompt information for guiding semantic analysis into a set language model to obtain question information corresponding to semantic perspective direction information; and inputting the question information corresponding to each semantic perspective direction information into a text encoding model to obtain the corresponding question semantic feature information.

[0042] First, a word segmenter is used to convert the medical text information into a sequence of sub-words. ,

[0043] Among them, represents the i-th token (sub-word), and N is the length of the token sequence.

[0044] To obtain a sequence with a unified fixed sequence length, the fixed sequence length is set to L. The token sequence is truncated or padded to obtain sub-word sequence information with a length of L.

[0045] Among them, when N ≥ L, the first L tokens of the sequence are retained and the rest are truncated. When N < L, special padding tokens are added at the end of the sequence until the length is L. This processing is used to convert free text into a canonical sequence for input to the text encoding module and to minimize noise and irrelevant information.

[0046] The sub-word sequence information is input into the text encoding model for encoding. The text encoding model can be CXR-BERT, BioBERT or other equivalent models. The text encoding model and the tokenizer come from the same pre-trained model system and share the same vocabulary and tokenization rules. After being processed by the text encoding model, the context semantic feature information of each sub-word is obtained.

[0047] Among them, is a vector used to represent the global semantic information of the entire text sequence. is a vector used to represent the context semantic feature information of the i-th token. Each context semantic feature information is obtained by the text encoding model through self-attention mechanism to perform context modeling on the entire token sequence. Therefore, not only contains the semantic information of the current token, but also incorporates the context relationship with other tokens (such as etc.).

[0048] A defined language model (e.g., Large Language Model, LLM) is used to summarize common and discriminative semantic dimensions in medical text information, yielding several semantic perspective directions. The input information to the defined language model includes the medical text information and defined cue information to guide semantic analysis. The cue information guides the defined language model to identify semantic attributes related to the target structure. Guided by this cue information, the defined language model summarizes common and discriminative semantic dimensions in the medical text information, thus obtaining multiple semantic perspective directions. These semantic perspective directions include, but are not limited to, at least one of the following: location of lesions or structures, morphological features, boundary characteristics, signal or density, internal structural changes, surrounding tissue relationships, and occupancy effect. For each semantic perspective direction, a corresponding natural language question is constructed, forming a question set.

[0049] in, Used to guide the setting of the language model to focus on medical text information related to the first. Descriptions related to semantic perspective information. This indicates the amount of semantic perspective information.

[0050] To supplement implicit semantic information that may not be covered by a fixed set of questions, a learnable query question is introduced, denoted as a vector. . It does not correspond to specific natural language text, but rather consists of a set of learnable query vector parameters. First, a learnable vector is randomly initialized. ,

[0051] Where d represents the semantic feature representation dimension. This vector serves as an additional query representation, along with the question set defined above. The text is input into the text encoding model for subsequent computation. During the language model training process, As learnable parameters, they are updated via backpropagation, enabling them to automatically capture latent semantic information relevant to the current task but difficult to define explicitly through manual questions. Therefore, the final set of questions used for semantic perspective extraction can be represented as follows:

[0052] in, It is used to supplement the semantic perspective space, thereby improving the model's ability to express complex medical semantic information.

[0053] Set of problems Each question in Each text is input into the same text encoding model for encoding, resulting in the corresponding semantic feature information of the question.

[0054] in, Indicate the problem The corresponding hidden vector is used to represent the global semantic information of the problem. This represents the contextual semantic feature vector of the i-th token in the semantic feature information of the question. For the question The number of tokens.

[0055] By using multi-semantic perspective decomposition and text encoding models, multiple semantic perspective representations with clear diagnostic semantic orientations can be extracted from medical text information, thus avoiding the introduction of complex text semantics in a coarse-grained manner into the medical image segmentation process.

[0056] Step S120: Attention aggregation is performed on the semantic feature information of each question and the contextual semantic feature information of all sub-words to obtain the corresponding semantic perspective feature information.

[0057] The global semantic feature vector included in the semantic feature information of each question participates in attention aggregation, while other contextual semantic feature vectors included in the semantic feature information of each question do not participate.

[0058] In some embodiments, step S120 specifically includes: using the global semantic feature vector of each question as the Query, and the contextual semantic feature information of all sub-words as the Key and Value, performing a scaling dot product attention-based calculation to obtain the corresponding semantic perspective feature information.

[0059] With the corresponding semantic feature information of the question As a query, it uses the contextual semantic features of all subwords. Using these as Key and Value, cross-attention is computed to aggregate textual information relevant to this semantic perspective. This process extracts the most relevant semantic information from the original medical text, yielding the corresponding semantic perspective feature information.

[0060] Here, Attention represents the computation based on scaled dot-product attention.

[0061] Where Q is Query, K is Key, V is Value, and d is the dimension of semantic feature information.

[0062] In this way, the text encoding model outputs a set of semantic feature information S.

[0063] in, Indicates the first Semantic feature information, This represents supplementary semantic perspective feature information generated from learnable query questions.

[0064] For details on the generation process of the semantic feature information set S, please refer to [link / reference]. Figure 2 .

[0065] Since the semantic perspective feature information set S only retains the original discrimination ability of each perspective, in some embodiments, the semantic perspective feature information set S is further integrated with the correlation and synergy between different semantic perspectives, which effectively alleviates the semantic conflict and inconsistency problems that may occur when multiple semantic conditions are introduced in parallel, and provides stable and structured conditional semantic input for visual and semantic interaction in subsequent steps.

[0066] In this embodiment, the method further includes: determining the correlation information between every two semantic perspective feature information; determining the projection information of each semantic perspective feature information in the Value space; determining the enhanced semantic feature information based on all the correlation information and the projection information of each semantic perspective feature information in the Value space; and fusing the enhanced semantic feature information with each semantic perspective feature information to obtain the semantic perspective feature correction information.

[0067] In some embodiments, determining the correlation information between every two semantic perspective feature information includes: determining the projection information of all semantic perspective feature information in the Query space and the projection information of all semantic perspective feature information in the Key space; and determining the correlation information between every two semantic perspective feature information based on the projection information of all semantic perspective feature information in the Query space and the projection information of all semantic perspective feature information in the Key space.

[0068] To facilitate unified modeling, the semantic feature information set S is stacked into a matrix form:

[0069] Where d represents the dimension of semantic feature information (i.e., the dimension of the hidden representation output by the text encoding model). Each row corresponds to a vector of semantic feature information.

[0070] First, the input semantic perspective representation is normalized. Layer normalization (LN) is applied to matrix S to reduce the instability caused by differences in feature distributions from different perspectives, resulting in a normalized matrix. ,

[0071] Then normalize the matrix Two independent linear mapping modules are used to obtain the Query and Key representations for relational modeling.

[0072]

[0073] in, Represents the normalized matrix Projection information in the Query space, Represents the normalized matrix Projection information in the Key space. Parameters The weights are learnable linear mappings, which are optimized through backpropagation during the training of the text encoding model. This represents the intermediate feature dimension in the relation modeling stage.

[0074] Based on the above projection information, the relationship matrix R between semantic perspective feature information is calculated using matrix multiplication.

[0075] in, This represents the correlation strength between semantic perspective feature information. The i-th and j-th elements in the matrix characterize the dependency relationship between the i-th and j-th semantic perspective feature information. The relation matrix can be analyzed... Further normalization (e.g., Softmax or ) is performed to obtain a stable relation weight distribution.

[0076] After obtaining the relationship matrix between semantic perspective feature information, the semantic perspective feature information is weighted and aggregated to integrate complementary information from other semantic perspectives.

[0077] The semantic feature information matrix S is transformed using a linear mapping module to obtain a value vector representation.

[0078] in, This represents the projection of the semantic feature information matrix S into the value space, used for subsequent relation aggregation. These are learnable linear mapping parameters that are optimized through backpropagation during model training.

[0079] We use the relation matrix R of semantic feature information to perform weighted aggregation on the value vectors.

[0080] in, This represents enhanced semantic feature information that incorporates multi-perspective relational information.

[0081] The enhanced semantic feature information is fused with the feature information from each semantic perspective using residuals to obtain a set of semantic perspective feature correction information. ,

[0082]

[0083] in, Indicates the first Semantic perspective feature correction information.

[0084] Semantic perspective feature correction information set See the generation process. Figure 2 .

[0085] Step S130: Input the medical image to be segmented into the image encoder to obtain the feature maps output at each scale level in sequence. The input of the next scale level in the image encoder is the feature map obtained by modulating the feature map output of the previous scale level in the image encoder and all semantic viewpoint feature information.

[0086] The medical image to be segmented is input into the image encoder (an encoder network based on U-Net) to extract hierarchical features from the medical image, resulting in a multi-level, multi-scale visual feature representation (i.e., feature map) to characterize the texture, morphology, and spatial context information of the target structure, providing basic features for subsequent visual and semantic interaction, decoding, and segmentation.

[0087] Let the medical image be When X is a three-dimensional image, When X is a two-dimensional image, X can be extended to be input into the network in a channelized form: Where C represents the number of channels in the input image. When the input is a multimodal medical image or a multi-sequence image (such as T1, T2, FLAIR, and other MRI sequences), different modalities or sequences can be stacked as different channels. When the input is a single-modal image, C can be set to 1.

[0088] The image encoder employs a multi-level downsampling structure, progressively expanding the receptive field and extracting higher-level semantic features. The image encoder contains L scale levels, each consisting of a convolutional block and a downsampling operation. Scale hierarchy The convolutional block can consist of several consecutive convolution operations, such as two convolutions plus non-linear activation, and a normalization operation can be used to stabilize training. The process can be represented as follows:

[0089] in, This is the output feature of the previous scale level, and This represents the input medical image data. Represents convolution operation (2D or 3D convolution). Representing a non-linear activation function (e.g., ReLU, LeakyReLU), convolutional blocks can also contain normalization layers such as BatchNorm, InstanceNorm, or LayerNorm.

[0090] In the above structure, downsampling is not achieved through independent pooling layers, but is implicitly accomplished through convolutions with a stride of 2 in the convolutional block, thereby reducing spatial resolution and expanding the receptive field while extracting features.

[0091] To support fine-grained boundary restoration and spatial detail reconstruction in subsequent decoding stages, the image encoder outputs and caches the corresponding feature maps at each scale level as skip connections.

[0092] in, This represents the feature map output at the l-th scale level. Shallower features (such as...) ) typically contains richer local texture and edge information, while deeper features (such as This includes stronger global semantics and contextual information. The above multi-scale feature set... It can be used as input for visual and semantic interaction in subsequent steps, and fused with upsampled features in the decoding stage to improve the precision and robustness of the segmentation results.

[0093] The feature maps output at each scale level are modulated with all semantic perspective feature information, so that all semantic perspective feature information participates in visual representation learning in an intrusive manner, thereby generating visual feature representations guided by structured semantics.

[0094] All semantic perspective features involved in the modulation process can be the set S of the aforementioned semantic perspective features. It can also be a set of feature correction information from various semantic perspectives. ( ).

[0095] The modulation processing of the feature map output from the previous scale level in the image encoder and all semantic viewpoint feature information in the image decoder includes steps S131 to S134.

[0096] Step S131: Determine the matching degree between the feature map output at each scale level and the feature information of each semantic perspective.

[0097] Step S132: Based on the matching degree between the feature map output at the corresponding scale level and each semantic perspective feature information, determine the weight distribution information of each semantic perspective feature information in the feature map output at the corresponding scale level.

[0098] Step S133: Based on the semantic viewpoint feature information, determine the channel scaling parameter and channel offset parameter mapped to the modulation parameter space corresponding to each feature map.

[0099] Step S134: Based on the weight distribution information of each semantic perspective feature information in the feature map output at the corresponding scale level, and the channel scaling parameters and channel offset parameters of each semantic perspective feature information mapped to the modulation parameter space corresponding to the feature map, determine the feature map after modulation processing at the corresponding scale level.

[0100] First, the feature maps output at each scale level are processed. Global aggregation is performed, and Global Average Pooling (GAP) is used to obtain channel-level visual description vectors. ,

[0101] in, This represents the number of channels at the l-th scale level. This represents the average pooling operation on the spatial dimension of the feature map, and is a commonly used feature aggregation method in convolutional neural networks.

[0102] Channel-level visual description vectors are mapped using linear mapping. Projected onto a dimensional space consistent with semantic viewpoint feature information.

[0103] in, Let represent the learnable linear mapping parameter corresponding to the l-th scale level. This parameter is optimized through backpropagation during model training, and d represents the dimensional space of semantic feature information.

[0104] Correction information for each semantic perspective feature Calculate its relationship with the first The matching degree of the feature maps output at each scale level. The matching degree is calculated using a scaled dot product. ,

[0105] Where d represents the dimensional space of semantic feature information.

[0106] Correction information for all semantic perspective features and the first The matching degree of the feature maps output at the scale level is normalized to obtain the feature correction information for each semantic perspective at the 1st level. Weight distribution information in the feature maps output at scale levels ,

[0107] Weight distribution information The correction information used to characterize features from different semantic perspectives is in the first The relative importance of features in the output feature maps at the scale level.

[0108] For each semantic perspective feature correction information The mapping module generates corresponding channel modulation parameter pairs, namely channel scaling parameters and channel offset parameters.

[0109] in, , For the first Learnable mapping functions corresponding to scale levels (such as linear layers or multilayer perceptrons).

[0110]

[0111] in, The weight matrix is ​​a learnable matrix. This corresponds to the bias vector. The above parameters are optimized through backpropagation during model training to incorporate semantic viewpoint feature correction information. Mapped to the modulation parameter space corresponding to the feature map output at scale l, This represents the number of channels corresponding to the l-th scale level.

[0112] The semantic perspective correction feature information is used in the first... Weight distribution information in the feature maps output at scale levels Correction information for each semantic perspective feature Corresponding channel modulation parameters Weighted fusion is performed to obtain the semantic viewpoint feature correction information mapped to the first... Channel scaling parameters in the modulation parameter space corresponding to the feature map output at the scale level and channel offset parameters ,

[0113] The aforementioned channel modulation parameters are used for conditional modulation of the feature maps, that is, performing a channel-by-channel linear transformation on the feature map corresponding to each channel, thereby dynamically adjusting the representation of the corresponding feature map based on all semantic viewpoint feature correction information. For example, the first... Feature maps output at scale levels (or Its modulation process can be represented as follows:

[0114] The modulation process described above is executed independently at multiple scale levels of the image encoder, thereby enabling semantic viewpoint feature correction information to guide each feature map layer by layer. The final output is a set of feature maps corresponding to each scale level after conditional modulation.

[0115] in, This represents the modulated feature map corresponding to the l-th scale level.

[0116] This embodiment does not introduce medical text information as a holistic feature or a single conditional signal into the image network. Instead, it uses semantic perspectives as the basic modeling unit, mapping the corrected multi-semantic perspective representations to conditional parameters used to modulate the response of image feature channels. During image encoding, each layer of visual features corresponds to a set of modulation parameters jointly determined by multiple semantic perspectives, ensuring that visual features at different levels are constrained by semantic conditions during the generation stage. In this mechanism, each semantic perspective does not act independently on visual features, but rather participates in the generation of modulation parameters through structured semantic representations formed by modeling relationships between perspectives, thereby jointly controlling the distribution of visual features at the channel level. This design allows semantic conditions to continuously influence the formation process of image features at multiple scales and multiple abstract levels, constituting a layer-by-layer embedded modulation method of semantic conditions in the visual backbone network.

[0117] according to Figure 2 As shown, the image encoder comprises four scale layers connected sequentially from top to bottom. The image decoder comprises four scale layers connected sequentially from bottom to top.

[0118] See Figure 2The input to the next scale level in the image encoder is the feature map output from the previous scale level in the image encoder and the feature map obtained by modulating all semantic viewpoint feature information.

[0119] Step S140: Input the feature map output by the last scale level of the image encoder and the feature map obtained by modulation processing of all semantic viewpoint feature information into the image decoder to obtain the target feature map output by the last scale level of the image decoder. The input of the next scale level in the image decoder is the feature map output by the previous scale level in the image decoder and the feature map output by the corresponding scale level in the image encoder.

[0120] The image decoder employs a bottom-up, stepwise upsampling structure to restore the spatial resolution of low-resolution, high-semantic-level features. The second feature map is output from the last scale level of the image encoder. Initially, an upsampling operation is performed to obtain a feature map at one scale level output from the image decoder.

[0121] in, This is either a transposed convolution or an interpolation upsampling combined convolution.

[0122] To fully utilize the spatial detail information preserved by the image encoder, the upsampled features... The second feature map corresponding to the scale level in the image encoder The fusion process is performed. Fusion can be achieved through feature concatenation or element-wise addition.

[0123] Features after fusion By integrating features through convolutional blocks, a second integrated feature map is obtained.

[0124] The convolutional block includes multiple convolutions, normalization, and nonlinear activation operations.

[0125] The above upsampling and fusion process is repeated at each scale level until the spatial resolution is restored to be consistent with the original medical image or the preset resolution.

[0126] Step S150: The prediction head is used to map the target feature map to generate the segmentation result of the target structure.

[0127] This step is used to map the high-dimensional feature representation to the category space. In some embodiments, step S150 specifically includes: using the prediction head to predict the category of each location point in the target feature map, thereby obtaining the segmentation category prediction result for each location point in the target feature map.

[0128] The prediction head can use 1×1×1 convolutional layers or equivalent linear mappings to map the target feature map to the class space.

[0129] in, Represents the target feature map. This represents the prediction mapping function, used to map the target feature map to a category prediction.

[0130] The predicted output P represents the segmentation category prediction result for each location point in the target feature map.

[0131] in, H,W,(D) represents the number of segmentation categories, and H,W,(D) represents the spatial resolution of the output segmentation map.

[0132] When the segmentation task is binary classification =2, and the probability map of the target structure is obtained through the Sigmoid function. When performing multi-class segmentation, the Softmax function is used to obtain the probability distribution of each class.

[0133] The segmentation category prediction results can be a probability map, a segmentation mask, or a combination thereof.

[0134] In this embodiment, the dataset used is the publicly available UltraBones100k skeletal ultrasound image dataset. This dataset is mainly used for automatic bone surface segmentation in skeletal ultrasound images and contains approximately 100,000 lower limb skeletal ultrasound images and their corresponding bone surface segmentation annotations. The skeletal regions involved mainly include the tibia, fibula, and related bone structures of the foot. The images were acquired by ultrasound equipment under different probe angles, scanning positions, and imaging conditions, and have strong noise, echo attenuation, and artifact interference characteristics.

[0135] Since the original UltraBones100k dataset only contains images and bone surface annotations, without corresponding medical text descriptions, this embodiment, referencing the construction approach of the TextBraTS dataset, uses a templated text generation method to construct corresponding medical report text for each ultrasound image sample. Based on bone surface segmentation labels and ultrasound image statistical features, a structured text description is generated, including the anatomical location of the bone, the continuity of bone surface echoes, contour morphology, and the echo characteristics of surrounding soft tissues, to simulate the textual description of bone structures in clinical ultrasound examination reports.

[0136] In the initial data processing stage, ultrasound images were uniformly normalized in size, with the input resolution set to 256×256. Image intensity was normalized using the Z-score normalization method to reduce intensity differences under different scanning conditions. After word segmentation, the maximum length of text data was set to 80, and padding and truncation strategies were used to meet the model input requirements.

[0137] In the text semantic modeling stage, based on the characteristics of the skeletal ultrasound segmentation task, a set of multi-semantic perspective questions is constructed, including the spatial location of bones in the image, the continuity of bone surface echoes, bone contour morphology features, and imaging artifacts. Simultaneously, a learnable query perspective is introduced to supplement implicit semantic information not covered by the fixed questions. The text encoding module uses BioBERT as the base language model, and the output semantic perspective feature dimension is set to 256.

[0138] In the image encoder, a U-Net-based convolutional coding structure is used to extract multi-scale visual features. The encoder consists of multiple layers of two-dimensional convolutional modules. Each convolutional module includes two 3×3 convolution operations and a ReLU activation function, and downsampling is achieved using a stride of 2. As the network depth increases, the spatial resolution of the feature map is halved layer by layer, while the number of channels doubles layer by layer. To enhance the stability of deep networks, residual connection structures are introduced into the encoder to alleviate the gradient vanishing problem. The number of output channels for each layer of the encoder is set to 32, 64, 128, and 256, respectively.

[0139] In the visual-semantic interaction stage, a semantic conditional modulation mechanism is introduced into the visual features output by each layer in the encoder. Specifically, each semantic viewpoint generates a set of channel modulation parameters, and the parameters are weighted and fused according to the matching relationship between the semantic viewpoint and the visual features of the current layer. Finally, the visual features are modulated layer by layer at the channel level through Feature Linear Modulation (FiLM), so that the textual semantic information participates in the image feature modeling process in an intrusive manner.

[0140] The decoder structure maintains a symmetrical design with the encoder, performing progressive upsampling through transposed convolutions and fusing high-resolution features using skip connections. After multi-scale feature fusion, a 1×1 convolution is used as the prediction head to output a bone surface segmentation probability map.

[0141] During model training, the Adam optimizer was selected, with an initial learning rate of 0.001, a batch size of 64, and 50 training epochs. A cosine annealing learning rate scheduling strategy was introduced to improve model convergence performance. The final model output is the bone surface segmentation result, used to evaluate the effectiveness of ultrasound bone segmentation guided by textual semantics.

[0142] In this embodiment, to effectively supervise the segmentation results of the model, a segmentation loss function combining region overlap and pixel-level classification constraints is adopted during the training phase. In one implementation, the segmentation loss is composed of a combination of Dice loss and binary cross-entropy loss, used to simultaneously constrain the overall segmentation consistency of the target structure and the local pixel classification accuracy.

[0143] Furthermore, to fully utilize the semantic information contained in features at different scales during the decoding process and to improve the segmentation stability of the model on multi-scale structures, a multi-scale deep supervision strategy is introduced into the decoding and prediction modules. Specifically, auxiliary segmentation prediction branches are set up at multiple intermediate levels of the decoder, and the prediction results output by each level and the segmentation labels at the corresponding scales are subject to supervision and constraints. The loss function used in the final training process is composed of a weighted average of the segmentation losses at each scale, thereby guiding the network to gradually learn the spatial distribution characteristics of the target structure at different stages of feature reconstruction.

[0144] By introducing the aforementioned segmentation loss and multi-scale deep supervision mechanism, the stability and convergence of the model training process can be improved without changing the overall network structure.

[0145] This disclosure also provides an electronic device for implementing any of the above method embodiments. Figure 3 A structural block diagram of an electronic device according to some embodiments is shown. This electronic device may be a PC, workstation, laptop, server, etc., and is not limited thereto.

[0146] like Figure 3 As shown, the electronic device includes a processor 310 and a memory 320 for storing executable instructions of the processor 310. The processor 310 is configured to implement a medical image segmentation method according to any embodiment of the present disclosure when executing the instructions stored in the memory 320.

[0147] The processor 310 is used to execute computer instructions, which can be written using instruction sets of architectures such as x86, Arm, RISC, MIPS, and SSE. The memory 320 includes, for example, ROM (Read-Only Memory), RAM (Random Access Memory), and non-volatile memory such as hard disks, etc., and is not limited here.

[0148] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. For the apparatus embodiments, relevant parts can be referred to the descriptions in the method embodiments.

[0149] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0150] Embodiments of this specification may be systems, methods, and / or computer program products. A computer program product may include a computer-readable storage medium having computer instructions stored thereon for causing a processor to implement various aspects of the embodiments of this specification.

[0151] Computer-readable storage media can be tangible devices capable of holding and storing computer instructions for use by computer instruction execution devices. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing computer instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0152] The computer instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network layer, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network layer may include copper cables, fiber optic cables, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network layer adapter card or network layer interface in each computing / processing device receives computer instructions from the network layer and forwards those instructions for storage on computer-readable storage media within the respective computing / processing device.

[0153] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this specification. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of computer instructions, which contains one or more executable computer instructions for implementing a specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent.

[0154] Various embodiments of this specification have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A medical image segmentation method, characterized in that, include: Determine the contextual semantic features of each word in the medical text information corresponding to the medical image to be segmented, as well as the question semantic features corresponding to the semantic perspective direction information. The semantic perspective direction information is at least one of position, shape, boundary, signal, density, and surrounding tissue relationship. Attention is used to aggregate the semantic features of each question and the contextual semantic features of all sub-words to obtain the corresponding semantic perspective features. The medical image to be segmented is input into the image encoder, and feature maps output at each scale level are obtained sequentially. The input of the next scale level in the image encoder is the feature map output by the previous scale level in the image encoder and the feature map obtained by modulation processing of all semantic viewpoint feature information. The feature map output from the last scale level of the image encoder and the feature map obtained by modulating all semantic viewpoint feature information are input into the image decoder to obtain the target feature map output from the last scale level of the image decoder. The input of the next scale level in the image decoder is the feature map output from the previous scale level in the image decoder and the feature map output from the corresponding scale level in the image encoder. The target feature map is mapped using a prediction head to generate segmentation results for the target structure.

2. The method according to claim 1, characterized in that, The determination of the contextual semantic feature information of each word in the medical text information corresponding to the medical image to be segmented, as well as the question semantic feature information corresponding to the semantic perspective direction information, includes: Based on the medical text information, word segmentation is performed to obtain sub-word sequence information with a set fixed sequence length; The sub-word sequence information is input into a text encoding model to obtain the contextual semantic feature information corresponding to each sub-word; and, The medical text information and the set prompts used to guide semantic analysis are input into the set language model to obtain the question information corresponding to the semantic perspective direction information; The question information corresponding to each semantic perspective direction information is input into the text encoding model to obtain the corresponding question semantic feature information.

3. The method according to claim 1, characterized in that, The semantic feature information of each question includes the global semantic feature vector of the corresponding question, wherein, The process of attention-aggregating the semantic feature information of each question and the contextual semantic feature information of all sub-words to obtain corresponding semantic perspective feature information includes: Using the global semantic feature vector of each question as the Query, and the contextual semantic feature information of all sub-words as the Key and Value, a scaling dot product attention-based calculation is performed to obtain the corresponding semantic perspective feature information.

4. The method according to claim 1, characterized in that, The method further includes: Determine the correlation information between each pair of semantic perspective feature information; Determine the projection information of each semantic perspective feature information in the Value space; Based on the projection information of all relevant information and semantic perspective feature information in the Value space, the enhanced semantic feature information is determined; The enhanced semantic feature information is fused with the semantic perspective feature information to obtain the semantic perspective feature correction information.

5. The method according to claim 4, characterized in that, The determination of the correlation information between every two semantic perspective feature information includes: Determine the projection information of all semantic perspective feature information in the Query space and the projection information of all semantic perspective feature information in the Key space; Based on the projection information of all semantic perspective feature information in the Query space and the projection information of all semantic perspective feature information in the Key space, the correlation information between each pair of semantic perspective feature information is determined.

6. The method according to claim 1, characterized in that, The input to the next scale level of the image encoder is the modulation processing of the feature map output from the previous scale level of the image encoder and all semantic viewpoint feature information, including: Determine the matching degree between the feature map output at each scale level and the feature information of each semantic perspective; Based on the matching degree between the feature map output at the corresponding scale level and each semantic perspective feature information, determine the weight distribution information of each semantic perspective feature information in the feature map output at the corresponding scale level. Based on the semantic viewpoint feature information, determine the channel scaling parameter and channel offset parameter in the modulation parameter space corresponding to each feature map; Based on the weight distribution information of each semantic perspective feature information in the feature map output at the corresponding scale level, and the channel scaling parameters and channel offset parameters of each semantic perspective feature information mapped to the modulation parameter space corresponding to the feature map, the feature map after modulation processing at the corresponding scale level is determined.

7. The method according to claim 1, characterized in that, The processing steps for the input at the next scale level in the image decoder include: The feature map output from the previous scale level in the image decoder is fused with the feature map output from the corresponding scale level in the image encoder to obtain a fused feature map. The fusion process is either feature concatenation or element-wise addition. The fused feature map is integrated by convolutional blocks to obtain an integrated feature map, wherein the convolutional blocks include multiple convolutions, normalization and nonlinear activation operations.

8. The method according to claim 1, characterized in that, The step of mapping the target feature map using a prediction head to generate a segmentation result of the target structure includes: Using the prediction head, class prediction is performed on each location point in the target feature map, resulting in the segmentation class prediction result for each location point in the target feature map.

9. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program for controlling the processor to operate in order to perform the medical image segmentation method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the medical image segmentation method as described in any one of claims 1 to 8.

Images