A multi-modal image data segmentation method and related products

Abstract

The application discloses a multi-modal image data segmentation method and related products, which comprises the following steps: acquiring multi-modal image data; using a segmentation model to perform segmentation processing on the multi-modal image data to obtain a segmentation result of the multi-modal image data output by the segmentation model; the segmentation model at least comprises a multi-level encoding layer and a multi-level decoding layer; the multi-level encoding layer is used for performing cross-modal and cross-window feature enhancement processing on the multi-modal image data; and the multi-level decoding layer is used for performing semantic complementary fusion processing on the enhanced features of the multi-modal image data. The application realizes high-precision and high-stability multi-modal medical image automatic segmentation without significantly increasing the calculation complexity, and overcomes the defects of the prior art, such as insufficient cross-modal alignment, weak semantic consistency and limited fine-grained structure reconstruction capability.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a multimodal image data segmentation method and related products. Background Technology

[0002] Medical image segmentation is a crucial foundational step in medical image analysis, disease-aided diagnosis, treatment planning, and postoperative evaluation, playing a vital role in organ structure measurement, functional analysis, and lesion localization. With the development of medical imaging equipment, multiple imaging modalities, such as computed tomography (CT), magnetic resonance imaging (MRI), and positron emission tomography (PET), are widely used in clinical practice for combined evaluation. Different imaging modalities differ significantly in their imaging mechanisms and information focuses. For example, CT excels in imaging high-density tissues, bone structures, and calcified areas; MRI is superior in soft tissue contrast and tissue morphology; and PET provides tissue function and metabolic information, supplementing structural imaging. Effectively integrating multimodal medical image information to achieve accurate and stable automatic segmentation has become an important research direction in the field of medical image processing.

[0003] Most existing multimodal medical image segmentation methods employ simple fusion strategies such as feature stitching and element-wise addition, directly merging features from different modalities during the encoding or decoding stages. These methods typically assume a high degree of consistency between different modalities in spatial location and semantic representation. However, in actual clinical data, due to factors such as scanning protocols, imaging resolution, patient positioning differences, and physiological movements, even after preprocessing and registration, local spatial misalignment and structural inconsistencies still exist between images of different modalities. Simple fusion strategies struggle to effectively model the structural correspondences between modalities, easily introducing redundant information or amplifying modality-specific noise, thus affecting the stability and accuracy of the segmentation results.

[0004] In encoder-decoder network architectures, the encoder focuses on high-level semantic abstraction, while the decoder needs to progressively restore spatial resolution and reconstruct fine structures. Existing methods typically employ the skip connection mechanism found in unimodal networks, directly passing low-level encoded features to the decoder. However, in multimodal scenarios, low-level features contain a large amount of modality-related texture differences and noise information. Directly introducing these features into the decoding stage can easily disrupt the cross-modal semantic consistency established in the encoding stage, leading to problems such as blurred target structure boundaries and breaks in fine structures. This impact is particularly pronounced in regions with small structures or complex morphologies.

[0005] In addition, some methods attempt to model intermodal relationships through global attention mechanisms, but the computational complexity is high, making it difficult to balance efficiency and local structure modeling capabilities in 3D medical image processing, thus limiting their clinical application.

[0006] In summary, existing multimodal medical image segmentation techniques still have shortcomings in cross-modal structural alignment, semantic consistency maintenance during the encoding-decoding stage, and fine structural reconstruction. Summary of the Invention

[0007] To address the aforementioned issues, this application provides a multimodal image data segmentation method and related products. The aim is to achieve a segmentation method that combines consistent modeling of multimodal feature space with semantic complementarity while ensuring computational efficiency, thereby improving segmentation accuracy and robustness.

[0008] The embodiments of this application disclose the following technical solutions: The first aspect of this application provides a multimodal image data segmentation method, the method comprising: Acquire multimodal image data; The multimodal image data is segmented using a segmentation model to obtain the segmentation result of the multimodal image data output by the segmentation model; the segmentation model includes at least a multi-level encoding layer and a multi-level decoding layer; the multi-level encoding layer is used to perform cross-modal and cross-window feature enhancement processing on the multimodal image data; the multi-level decoding layer is used to perform semantic complementary fusion processing on the enhanced features of the multimodal image data.

[0009] Optionally, the multimodal image data includes multiple modal images; the step of segmenting the multimodal image data using a segmentation model to obtain the segmentation result of the multimodal image data output by the segmentation model specifically includes: The multi-level coding layer is used to encode the multimodal image data to obtain enhanced data for each modality image; the enhanced data includes enhanced features for each level. The enhanced data of each modal image is interactively fused using the multi-level decoding layer to obtain a fusion result for each modal image; the fusion result includes the fusion features of each level; Based on the fusion result of the multimodal image data and each modal image, the segmentation result of the multimodal image data is obtained.

[0010] Optionally, the segmentation model includes a four-level coding layer. The step of encoding the multimodal image data using the multi-level coding layer to obtain enhanced data for each modality image specifically includes: The first-level coding layer is used to enhance the features of each modality image to obtain the first enhanced feature of each modality image; The second-level coding layer is used to enhance the first enhancement feature of each modality image to obtain the second enhancement feature of each modality image; The second enhancement feature of each modality image is enhanced using the third-level coding layer to obtain the third enhancement feature of each modality image; The third enhancement feature of each modality image is enhanced using the fourth-level coding layer to obtain the fourth enhancement feature of each modality image; The first enhancement feature, the second enhancement feature, the third enhancement feature, and the fourth enhancement feature are used as enhancement data for the corresponding modality image.

[0011] Optionally, the step of using the first-level coding layer to perform feature enhancement on each modality image to obtain the first enhanced feature of each modality image specifically includes: Feature extraction is performed on each modality image to obtain the encoded features of each modality image; The encoded features of each modality image are divided into multiple local three-dimensional windows. Within the same local three-dimensional window, a correspondence is established between the encoded features of multiple modality images to obtain the attention enhancement features of each modality image. Within the same local 3D window, intra-window enhancement processing is performed on the corresponding modal image based on attention enhancement features and coding features. Between adjacent local 3D windows, cross-window enhancement processing is performed on the corresponding modal image based on attention enhancement features and coding features to obtain the first enhancement feature of each modal image.

[0012] Optionally, the segmentation model includes a three-level decoding layer. The step of interactively fusing the enhanced data of each modality image using the multi-level decoding layer to obtain the fusion result of each modality image specifically includes: Perform the following on the augmented data of any modality image: The fourth stitching feature and the third enhancement feature of the modal image are input into the third-level decoding layer to obtain the first fusion feature of the modal image; the fourth stitching feature is obtained by stitching the fourth enhancement feature of the modal image and the fourth enhancement feature of any modal image; the arbitrary modal image is any modal image other than the modal image in the multimodal image data; The first fusion feature and the second enhancement feature of the modal image are input into the second-level decoding layer to obtain the second fusion feature of the modal image; The second fusion feature and the third enhancement feature of the modal image are input into the first-level decoding layer to obtain the third fusion feature of each modal image; The first fusion feature, the second fusion feature, and the third fusion feature are used as the fusion result of the modal image.

[0013] Optionally, the step of inputting the fourth stitching feature and the third enhancement feature of the modal image into the third-level decoding layer to obtain the first fusion feature of the modal image specifically includes: The fourth enhancement feature of the modal image is concatenated with the fourth enhancement feature of any modal image to obtain the fourth concatenated feature of the modal image; The third enhancement feature of the modal image is semantically mapped to obtain the mapped enhancement feature of the modal image; Based on the fourth stitching feature of the modal image, the spatial offset of the modal image is obtained; Based on the spatial offset and mapping enhancement features of the modal image, the third enhancement feature of the arbitrary modal image is resampled to obtain the sampled feature; The sampling features and the mapping enhancement features of the modal image are divided into multiple local three-dimensional windows respectively. Fine-grained feature modeling is performed within the same local three-dimensional window. Information transmission paths are established between adjacent local three-dimensional windows to propagate local features across regions, thereby obtaining the cross-modal enhancement features of the modal image. The cross-modal enhancement features and mapping enhancement features of the modal image are fused to obtain the first fused feature of the modal image.

[0014] A second aspect of this application provides a multimodal image data segmentation apparatus, the multimodal image data segmentation apparatus comprising: The acquisition module is used to acquire multimodal image data; A segmentation module is used to segment the multimodal image data using a segmentation model to obtain the segmentation result of the multimodal image data output by the segmentation model; the segmentation model includes at least a multi-level encoding layer and a multi-level decoding layer; the multi-level encoding layer is used to perform cross-modal and cross-window feature enhancement processing on the multimodal image data; the multi-level decoding layer is used to perform semantic complementary fusion processing on the enhanced features of the multimodal image data.

[0015] A third aspect of this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the multimodal image data segmentation method provided in the first aspect.

[0016] The fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multimodal image data segmentation method provided in the first aspect.

[0017] The fifth aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the multimodal image data segmentation method provided in the first aspect.

[0018] Compared with the prior art, this application has the following beneficial effects: This application includes acquiring multimodal image data; segmenting the multimodal image data using a segmentation model to obtain a segmentation result of the multimodal image data output by the segmentation model; the segmentation model includes at least a multi-level encoding layer and a multi-level decoding layer; the multi-level encoding layer is used to perform cross-modal and cross-window feature enhancement processing on the multimodal image data; the multi-level decoding layer is used to perform semantic complementary fusion processing on the enhanced features of the multimodal image data.

[0019] This application effectively alleviates the problems of local spatial misalignment and inconsistent structural representation caused by factors such as scanning protocols, resolution differences, body position changes, or physiological movements in different imaging modalities (such as CT, MRI, and PET) by performing cross-modal and cross-window feature enhancement processing on multimodal image data in multi-level coding layers. It can dynamically mine the structural correspondences between modalities, suppressing modality-specific noise while strengthening common semantic features, thereby improving the consistency and robustness of feature representation in the coding stage. Based on this, multi-level decoding layers perform semantic complementary fusion processing on the enhanced multimodal features, avoiding the texture differences and redundant information introduced by directly transmitting low-level skip connections in traditional methods. This allows the decoding process to more accurately integrate the advantages of each modality at different semantic levels. Especially for small, blurred-boundary, or anatomically complex regions, it significantly improves the completeness and boundary clarity of the segmentation results. In short, this application achieves high-precision and high-stability automatic segmentation of multimodal medical images without significantly increasing computational complexity, overcoming the shortcomings of existing methods such as insufficient cross-modal alignment, weak semantic consistency, and limited fine-grained structural reconstruction capabilities. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A flowchart illustrating a multimodal image data segmentation method provided in this application embodiment; Figure 2 A structural diagram of the first-level coding layer provided in the embodiments of this application; Figure 3A structural diagram of bidirectional cross-modal spatial attention provided in an embodiment of this application; Figure 4 The structural diagram of the third-level decoding layer provided in the embodiments of this application; Figure 5 A schematic diagram illustrating a multimodal image data segmentation method provided in an embodiment of this application; Figure 6 This is a structural diagram of a multimodal image data segmentation device provided in an embodiment of this application. Detailed Implementation

[0022] As described earlier, medical image segmentation is a crucial foundation for disease-aided diagnosis, treatment planning, and postoperative assessment, and is essential for organ measurement, functional analysis, and lesion localization. Clinically, multiple imaging modalities such as CT, MRI, and PET are often used in combination, each with its own advantages in imaging mechanisms and information emphasis: CT excels at high-density tissue and bone structure imaging, MRI performs well in soft tissue contrast, and PET provides functional metabolic information. However, existing segmentation methods often employ simple fusion strategies such as feature stitching or element-wise addition, implicitly assuming high spatial and semantic consistency among modalities. In reality, influenced by factors such as scanning protocols, resolution differences, body position changes, and physiological movements, even after registration preprocessing, local misalignments and structural inconsistencies still exist between modalities. Simple fusion easily introduces redundancy or amplifies modality-specific noise, impairing segmentation accuracy. Furthermore, in the encoder-decoder architecture, directly using the skip connection mechanism of a single modality will bring low-level modality-related textures and noise into the decoding stage, disrupting cross-modal semantic consistency and leading to blurred boundaries or loss of fine structures. While some methods introduce global attention mechanisms to model modal relationships, they have high computational overhead and are difficult to balance efficiency in 3D medical image modeling with local detail modeling.

[0023] In view of the above problems, this application provides a multimodal image data segmentation generation method and related products. The method includes: acquiring multimodal image data; performing segmentation processing on the multimodal image data using a segmentation model to obtain the segmentation result of the multimodal image data output by the segmentation model; the segmentation model includes at least a multi-level encoding layer and a multi-level decoding layer; the multi-level encoding layer is used to perform cross-modal and cross-window feature enhancement processing on the multimodal image data; the multi-level decoding layer is used to perform semantic complementary fusion processing on the enhanced features of the multimodal image data.

[0024] This application introduces cross-modal and cross-window feature enhancement mechanisms in multi-level coding layers to dynamically model the structural correspondences between modalities. This effectively suppresses modality-specific noise while strengthening common semantic features, significantly improving the consistency and robustness of feature representations during the coding stage. Based on this, multi-level decoding layers perform semantic complementary fusion of the enhanced multimodal features, abandoning the traditional method of directly passing low-level skip connections and avoiding the introduction of modality-related texture differences and redundant information. This allows for more accurate integration of the advantages of each modality at different semantic levels. Especially for regions with small volume, blurred boundaries, or complex anatomical structures, the completeness and boundary clarity of the segmentation results are significantly improved. The overall scheme effectively overcomes the key shortcomings of existing multimodal segmentation methods, such as insufficient cross-modal alignment, weak semantic consistency, and limited fine-grained structure reconstruction capabilities, without significantly increasing computational complexity, achieving high-precision and high-stability automatic medical image segmentation.

[0025] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0026] Figure 1 A flowchart of a multimodal image data segmentation method provided in this application embodiment is shown below. Figure 1 As shown, a multimodal image data segmentation method includes: S101: Acquire multimodal image data.

[0027] This application does not limit the method of acquiring multimodal image data. It can be acquired directly through clinical medical imaging equipment (such as CT, MRI, PET, ultrasound, SPECT, etc.), or retrieved from hospital information systems (HIS), picture archiving and communication systems (PACS), or public / private medical image databases. The acquired multimodal image data can come from different imaging examinations performed on the same patient within a similar time window.

[0028] Furthermore, this application does not limit the specific types and combinations of multimodal image data. Multimodal image data may include multiple modal images, typical examples including but not limited to: computed tomography (CT) images, magnetic resonance imaging (MRI) images (such as T1-weighted, T2-weighted, FLAIR, DWI sequences, etc.), positron emission tomography (PET) images, as well as functional imaging (such as fMRI, ASL), ultrasound images, digital subtraction angiography (DSA), etc. Different modalities can be combined in pairs (such as CT+PET, T1-MRI+T2-MRI) or in multimodal combinations (such as CT+MRI+PET) to fully utilize the complementary information of each imaging technology in terms of tissue contrast, functional metabolism, etc., providing a more comprehensive and richer input basis for subsequent high-precision segmentation.

[0029] To eliminate the inherent differences in spatial resolution, voxel scale, and intensity distribution among different imaging modalities, a series of standardization preprocessing operations can be performed on the raw multimodal image data, including but not limited to: Intensity normalization (such as Z-score normalization, histogram matching, or percentile-based linear / nonlinear scaling) can be used to mitigate the problem of inconsistent gray-level distribution between modes. Spatial registration (rigid, affine, or non-rigid registration) aligns modal images to a unified anatomical coordinate system, ensuring that corresponding structures accurately coincide in space; Uniform size and voxel scale (e.g., resampling to the same spatial resolution and image dimension) ensures that each modal input has a consistent grid structure, facilitating subsequent parallel processing and feature fusion of the network.

[0030] After the above preprocessing, the obtained multimodal image data exhibits higher consistency in spatial geometry and intensity representation, effectively enhancing the model's ability to learn cross-modal semantic associations. Let the m-th modal image after preprocessing be... for: ; Where H, W, and D represent the unified height, width, and depth (number of slices), respectively, and M represents the number of modalities; this representation is applicable to two-dimensional slice sequences or three-dimensional volume data input and is compatible with the tensor organization methods of mainstream deep learning frameworks.

[0031] S102: The multimodal image data is segmented using a segmentation model to obtain the segmentation result of the multimodal image data output by the segmentation model.

[0032] This application does not limit the structure of the segmentation model used, but the segmentation model shall include at least multiple levels of coding layers and multiple levels of decoding layers. Wherein: The multi-level coding layer is used to perform cross-modal and cross-window feature enhancement processing on the multimodal image data, specifically including the following steps: Each modality image is mapped to a unified high-dimensional feature space through shared or modality-specific embedding modules; A window partitioning strategy is adopted to divide the feature map into multiple local sub-windows of fixed size, and Intra-Window Spatial Refinement (IWSR) is performed within each window to enhance local structural details; At the same time, Cross-Window Spatial Refinement (CWSR) is introduced to improve structural continuity and global consistency through information exchange between windows; We further designed a bidirectional inter-modality spatial attention (BISA) mechanism to explicitly model the correspondence between different modalities in the spatial dimension, realize the dynamic alignment and complementarity of features between modalities, thereby enhancing the consistency of multimodal structures and improving global context awareness.

[0033] The multi-level decoding layer is used to perform semantic complementary fusion processing on the enhanced features of the multimodal image data, specifically including: The enhanced features of each modality are subjected to unified semantic space mapping and normalization, and projected onto the same semantic subspace to eliminate semantic offset between modalities; By introducing a Position Offset Modeling Network and a Learnable Spatial Transformation Network, residual local spatial misalignment can be deformably aligned, further mitigating structural deviations caused by registration errors or physiological movements. The aligned features are divided into local windows, and a cross-modal attention mechanism is built within each window to achieve semantic mutual guidance and feature compensation between modalities (e.g., using the soft tissue contrast of MRI to compensate for low contrast areas of CT). At the same time, cross-window links are established between adjacent windows to enhance the structural continuity and boundary consistency between local areas; The enhanced fusion features mentioned above are adaptively fused with the corresponding hierarchical features of the original dominant modality (such as the clinical priority modality) through residual connections and feedforward networks (FFNs), which effectively preserves key anatomical details and achieves high-fidelity and high-robust fine structural reconstruction.

[0034] Through the aforementioned encoding-decoding collaborative optimization mechanism, this application achieves accurate segmentation of complex, minute, or blurred-boundary structures in multimodal medical images without significantly increasing computational burden, which is significantly better than existing segmentation methods that rely on simple feature splicing or static skip connections.

[0035] This application does not limit the training method of the segmentation model, for example: A composite loss function combining multi-class Dice loss and binary cross-entropy (BCE) loss is employed to independently and collaboratively optimize each structural class in the segmentation model. Dice loss directly measures the similarity between the predicted region and the ground truth label in spatial overlap, exhibiting inherent robustness to class imbalance, particularly beneficial for improving the segmentation accuracy of small, sparse structures. BCE loss, on the other hand, enhances the model's ability to discriminate boundary details and local semantics through pixel-wise probabilistic supervision. The weighted fusion of these two loss functions effectively balances global region consistency with local pixel-level accuracy, guiding the network to learn feature representations of structures at different scales more evenly during backpropagation.

[0036] The weighted combination of Dice loss and binary cross-entropy loss is as follows: ; ; ; Where C represents the total number of categories in the segmentation task; This represents the actual label, and its value is either 0 or 1. The predicted probability output by the model is typically activated by the sigmoid function; P represents the predicted region; G represents the true label region; to enhance numerical stability, a smoothing term is introduced in the actual calculation. To avoid division by zero errors.

[0037] By driving model training with this composite loss function, not only is the optimization bias caused by the large volume difference mitigated, but the multimodal features are also encouraged to achieve closer spatial alignment and semantic complementarity in the encoding and decoding stages, thereby improving the overall model's segmentation performance and generalization ability for complex anatomical structures.

[0038] This application is widely applicable to various medical imaging modalities such as CT, MRI, and PET. In the encoding stage, by modeling the spatial correspondence between different modalities within and between local windows, shared anatomical structural information in multimodal images is explicitly captured, effectively improving the cross-modal consistency and robustness of feature representation. In the decoding stage, through intermodal semantic cross-referencing, local alignment, and context enhancement, complementary fusion of advantageous features from each modality is achieved, accurately reconstructing anatomical structures with blurred boundaries, small volumes, or complex shapes. The overall framework, through co-optimization of encoding and decoding, significantly improves the accuracy, detail fidelity, and clinical applicability stability of multimodal medical image segmentation without significantly increasing computational overhead.

[0039] The above describes the main technical solution of this application. Further implementations of the main technical solution are now introduced. Details are as follows: Regarding S102, which uses a segmentation model to segment the multimodal image data to obtain the segmentation result of the multimodal image data output by the segmentation model, this application provides an optional embodiment: The multi-level coding layers are used to encode the multimodal image data to obtain enhanced data for each modality. The enhanced data includes enhancement features for each level.

[0040] The enhanced data of each modal image are interactively fused using the multi-level decoding layers to obtain a fusion result for each modal image. The fusion result includes the fusion features of each level.

[0041] Based on the fusion result of the multimodal image data and each modal image, the segmentation result of the multimodal image data is obtained.

[0042] To address the issue of encoding the multimodal image data using the multi-level coding layers to obtain enhanced data for each modality image, this application provides an optional embodiment. This optional embodiment primarily targets application scenarios where the segmentation model includes a four-level coding layer: The first-level coding layer is used to enhance the features of each modality image to obtain the first enhanced feature of each modality image.

[0043] The second-level coding layer is used to enhance the first enhancement feature of each modality image to obtain the second enhancement feature of each modality image.

[0044] The second enhancement feature of each modality image is enhanced using the third-level coding layer to obtain the third enhancement feature of each modality image.

[0045] The third enhancement feature of each modality image is enhanced using the fourth-level coding layer to obtain the fourth enhancement feature of each modality image.

[0046] The first enhancement feature, the second enhancement feature, the third enhancement feature, and the fourth enhancement feature are used as enhancement data for the corresponding modality image.

[0047] This application does not limit the level of coding layers, as long as each coding layer can perform feature enhancement on each modal image. To clearly illustrate the technical solution of this application, this application takes the first-level coding layer as an example. Specifically, for using the first-level coding layer to perform feature enhancement on each modal image to obtain the first enhanced feature of each modal image, this application provides an optional embodiment: Feature extraction is performed on each modality image to obtain the encoded features of each modality image.

[0048] For example, each modality image is input into an encoder network with a shared structure or independent parameters. High-dimensional features are extracted through convolution and normalization operations to obtain the encoded features of each modality image. For example, the encoded features of the m-th modality image are represented as follows: ; The encoded features of each modality image are divided into multiple local 3D windows. Within the same local 3D window, a correspondence is established between the encoded features of multiple modality images to obtain the attention enhancement features of each modality image.

[0049] For example, encoding features Divided into multiple local 3D windows: ; Intra-Window Spatial Refinement (CWSR) is introduced within each window to enhance the expression of local structure; at the same time, Intra-Window Spatial Refinement (IWSR) is introduced between adjacent windows to compensate for the spatial continuity at window boundaries.

[0050] Within the same local 3D window, intra-window enhancement processing is performed on the corresponding modal image based on attention enhancement features and coding features. Between adjacent local 3D windows, cross-window enhancement processing is performed on the corresponding modal image based on attention enhancement features and coding features to obtain the first enhancement feature of each modal image.

[0051] For example, within the corresponding window, a bidirectional cross-modal spatial attention mechanism is introduced for different modal features. and modality For example, the attention calculation process is as follows: ; By using bidirectional modeling, an explicit correspondence is established between different modalities at the spatial structure level, resulting in the first enhanced feature of each modal image.

[0052] This application also provides a specific application embodiment. Figure 2 The structural diagram of the first-level coding layer provided in the embodiments of this application is as follows: Figure 2 As shown: X-ray of CT modal images CT and MRI modal images X MR After each feature is independently encoded, the Win_merge module performs preliminary window-level feature alignment and fusion (possibly using channel concatenation or weighted fusion) to generate intermediate feature X. mi .

[0053] Introduce in-window space enhancement within each window, specifically for X. CT and X MR Perform BISA (Bidirectional Inter-Modality Spatial Attention) operations separately to achieve explicit spatial interaction between modalities; restore spatial resolution through upsampling (e.g., upsampling to the original scale or matching the decoder level) and output enhanced features E'. CT With E' MR , representing the CT and MR feature representations after cross-modal collaborative optimization, respectively.

[0054] Introduce cross-window spatial enhancements between adjacent windows, specifically: E' CT With X CT Perform feature splicing ( Figure 2 The “⊕” symbol in the middle is used to fuse features from different paths or modalities, and E' MR With X MR Feature stitching is performed, and BISA (which can be understood as an in-window self-attention or intramodal enhancement variant) is applied to process the stitched features, focusing on structural details and texture consistency within the local window; the first enhancement feature F of the CT modality image is output. CT The first enhancement feature F of the MRI modality image MR This serves as input to subsequent segmentation heads (such as those used to generate the final segmentation map).

[0055] This application does not specify BISA. Figure 3 A structural diagram of bidirectional cross-modal spatial attention provided in the embodiments of this application is shown below. Figure 3 As shown, the CT feature branch (upper part): Q CT K CT V CT: These represent the query, key, and value features of the CT modality, respectively, and are usually obtained by linear projection of the input features.

[0056] MRI characteristic branch (lower part): Q MR K MR V MR Similarly, this represents the three sets of attention features corresponding to the MRI modality. This embodiment employs standard Transformer-style attention decomposition, but modal decoupling design is implemented specifically for the characteristics of medical images.

[0057] Q CT and K CT Perform dot product operation ( Figure 3 middle" After calculating the intermodal spatial similarity score (using the symbol), the results are sequentially input into normalization (Softmax) and Dropout (to enhance generalization ability and prevent overfitting) to obtain the first enhancement result. The first enhancement result is then compared with V... CT Perform a dot product operation to obtain the first dot product result.

[0058] Q CT and K MR After performing the dot product operation, the results are sequentially input into normalization (Softmax) and Dropout to obtain the second augmented result. This second augmented result is then compared with V... MR Perform a dot product operation to obtain the second dot product result; then set K... CT and Q MR After performing the dot product operation, the results are sequentially input into normalization (Softmax) and Dropout to obtain the third augmentation result. This third augmentation result is then compared with V... CT Perform a dot product operation to obtain the third dot product result; then convert Q... MR and K MR After performing the dot product operation, the results are sequentially input into the normalization (Softmax) and Dropout processes to obtain the fourth augmentation result. This fourth augmentation result is then compared with V... MR Perform a dot product operation to obtain the fourth dot product result.

[0059] Concatenate the first dot product result and the second dot product result (Concat, Figure 3 The input (with the "⊕" symbol) is then fed into a convolutional layer (Conv, used for non-linear mapping and channel recalibration to further improve feature discriminativeness), and the final fused feature A is output. CT←MR The third and fourth dot product results are concatenated and then fed into a convolutional layer (Conv) to output the final fused feature A. MR←CT .

[0060] BISA achieves cross-modal semantic matching at the pixel / window level, which is superior to simple splicing or average fusion; the attention weights adapt to the content and give higher response to key areas such as lesions and edges; it requires only a few additional parameters (linear projection + small convolution) and is easy to embed into mainstream segmentation networks (such as U-Net++, Swin-Unet).

[0061] This embodiment ensures macroscopic structural alignment (such as positional consistency) between multimodal images and enhances the fidelity of details in local areas (such as tumor boundaries and small lesions). It is particularly suitable for solving misalignment and blurring problems caused by imaging differences, providing a robust feature foundation for high-precision multimodal image segmentation.

[0062] To address the interactive fusion processing of the enhanced data of each modal image using the multi-level decoding layers to obtain the fusion result of each modal image, this application provides an optional embodiment. This optional embodiment mainly targets application scenarios where the segmentation model includes a three-level decoding layer: Perform the following on the augmented data of any modality image: The fourth stitching feature and the third enhancement feature of the modal image are input into the third-level decoding layer to obtain the first fusion feature of the modal image; the fourth stitching feature is obtained by stitching the fourth enhancement feature of the modal image and the fourth enhancement feature of any modal image; the arbitrary modal image is any modal image other than the modal image in the multimodal image data.

[0063] The first fusion feature and the second enhancement feature of the modal image are input into the second-level decoding layer to obtain the second fusion feature of the modal image.

[0064] The second fusion feature and the third enhancement feature of the modal image are input into the first-level decoding layer to obtain the third fusion feature of each modal image.

[0065] The first fusion feature, the second fusion feature, and the third fusion feature are used as the fusion result of the modal image.

[0066] This application does not limit the level of the decoding layer, as long as each level of the coding layer can perform feature fusion on each modal image. To clearly illustrate the technical solution of this application, this application takes the third-level decoding layer as an example. Specifically, for inputting the fourth stitching feature and the third enhancement feature of the modal image into the third-level decoding layer to obtain the first fused feature of the modal image, this application provides an optional embodiment: The fourth enhancement feature of the modal image is concatenated with the fourth enhancement feature of any modal image to obtain the fourth concatenated feature of the modal image.

[0067] This application does not limit the specific method of splicing. Channel-wise concatenation, spatial stacking, weighted fusion, or adaptive splicing based on attention mechanism can be adopted as long as the multimodal context information can be effectively integrated and the discriminative features of each modality can be preserved.

[0068] The third enhancement feature of the modal image is semantically mapped to obtain the mapped enhancement feature of the modal image.

[0069] For example, the third enhancement feature of the m-th modality image output from the encoding stage is projected onto a unified semantic space through linear mapping and normalization operations to reduce the impact of modality differences on feature fusion in the decoding stage, thus obtaining the mapped enhancement feature of the modality image. ,as follows: ; in, Representation layer normalization, These are learnable parameters that are updated via backpropagation during model training.

[0070] Based on the fourth stitching feature of the modal image, the spatial offset of the modal image is obtained.

[0071] For example, a position offset modeling network is introduced to predict the spatial offset of modal images. : ; Based on the spatial offset and mapping enhancement features of the modal image, the third enhancement feature of the arbitrary modal image is resampled to obtain the sampled feature.

[0072] For example, the third enhancement feature of the arbitrary modality image is resampled based on the spatial offset and mapping enhancement features of the modality image using a Spatial Transformation Network (STN) to obtain the sampled features. : ; The sampling features and the mapping enhancement features of the modal image are divided into multiple local three-dimensional windows. Fine-grained feature modeling is performed within the same local three-dimensional window. Information transmission paths are established between adjacent local three-dimensional windows to propagate local features across regions, thereby obtaining the cross-modal enhancement features of the modal image.

[0073] For example, at a local scale, dividing a 3D feature map into several fixed-size, non-overlapping local 3D windows and performing fine-grained feature modeling within each local 3D window effectively limits the scope of attention computation, thereby reducing computational complexity. Simultaneously, it enhances the model's ability to perceive local structures, particularly helpful in depicting details such as myocardial boundaries, thin-layer tissues, and small-volume anatomical structures. However, relying solely on modeling within a window can lead to spatial information fragmentation at window boundaries, affecting the consistent representation of the global structure. Therefore, information transmission paths are established between adjacent local 3D windows, enabling local features to propagate across regions, thus restoring the continuity of the overall structure. This design effectively alleviates the spatial discontinuity problem caused by window partitioning.

[0074] The cross-modal enhancement features and mapping enhancement features of the modal image are fused to obtain the first fused feature of the modal image.

[0075] To achieve effective fusion of multimodal features, a cross-modal attention mechanism is introduced between different modal features at corresponding spatial locations. By constructing explicit feature associations, the model can interactively model multimodal information under unified spatial constraints, thereby improving cross-modal structural consistency. Simultaneously, this mechanism can adaptively adjust the contribution weights of different modal features, enabling the model to dynamically utilize complementary information from CT and MRI in different regions. Specifically: The cross-modal enhancement features and the dominant modality features are fused through residual connections and a feedforward network to obtain the first fused feature of the modality image. : ; Figure 4 The structural diagram of the third-level decoding layer provided in the embodiments of this application is as follows: Figure 4 As shown, CT features (CT enhancement features) and MRI features (MRI enhancement features) are presented. CT features are processed using normalization (such as LayerNorm or InstanceNorm) to eliminate differences in intensity distribution between modalities. The normalized CT features are then used as reference features and added element-wise to the original MRI features. Figure 4 The “⊕” symbol in the middle generates a mapping-enhanced feature offset.

[0076] The offset is taken as input and fed into STN (Spatial Transformer Network); STN outputs spatial offset P (such as an affine transformation matrix) and applies it to CT or MRI features to achieve differentiable spatial registration; Window partitioning is performed on the reference features and MRI features respectively, dividing the global features into local non-overlapping windows (such as the Swing Transformer style); cross-window attention is performed within the window to establish information transmission paths between adjacent local 3D windows to propagate local features across regions, thereby obtaining cross-modal enhancement features of the modal image.

[0077] The cross-modal enhancement features of the modal image are input into the FFN (Feed-Forward Network, usually two layers of MLP + GELU activation) to further extract higher-order nonlinear modes; before the FFN output, it is fused with the mapping enhancement features to form residual connections; after Dropout regularization, the first fused features of the modal image are output.

[0078] It significantly improves the segmentation accuracy and generalization ability of multimodal medical images in scenarios with complex anatomical structures and large contrast differences.

[0079] This application also provides a specific application embodiment: Figure 5 This is a schematic diagram of a multimodal image data segmentation method provided in an embodiment of this application, as shown below. Figure 5 As shown, CT modal images and MRI modal images are sequentially input into the first coding layer (size W / 4×H / 4×D / 4×C), the second coding layer (size W / 8×H / 8×D / 8×2×C), the third coding layer (size W / 16×H / 16×D / 16×4×C), and the fourth coding layer (size W / 32×H / 32×D / 32×8×C) (as shown). Figure 5 As shown, each coding layer includes two Transformer Blocks (upsampling and downsampling) and two CM-CWSAs (i.e., Figure 2 (See Cross-Window Spatial Refinement and Intra-Window Spatial Refinement).

[0080] The enhanced features output from each coding layer are input into a multi-level decoder (e.g., Figure 5 The MMCF shown obtains the fusion result for each modality, and then the fusion result of each modality image is sequentially input into multiple Transformer Blocks to obtain the final fusion feature of each modality image; such as Figure 5As shown, the CT modal image and the MRI modal image are stitched together and then input into the Transformer Block to obtain the stitched image. The final fusion features of the stitched image and the CT modal image are then input into the Transformer Block to obtain the segmentation result of the CT modal image.

[0081] During the coding phase, this application proposes a cross-modal cross-window spatial attention mechanism (CM-CWSA) to explicitly establish structural correspondences between different modalities at a local window scale. This mechanism, through bidirectional query-key interaction, enables heterogeneous features such as CT and MRI to mutually perceive and dynamically modulate each other, suppressing modality-specific noise while enhancing the expression of shared anatomical structures. Simultaneously, a cross-window connectivity strategy is introduced to maintain spatial continuity and long-range dependency modeling capabilities while keeping computationally low, achieving efficient and accurate structural consistency modeling.

[0082] During the decoding stage, this application integrates deformable spatial alignment and unified semantic space mapping mechanisms to ensure that high-level features from different modalities are highly aligned in both spatial location and semantic distribution dimensions. Building upon this, it combines cross-modal attention within local windows with multi-scale skip connections to achieve semantic complementarity and hierarchical backhaul of detailed information, significantly alleviating typical problems such as boundary blurring and small structural breaks, and improving the anatomical rationality and clinical usability of the segmentation results.

[0083] At the overall optimization level, this application further designs a composite loss function that takes into account both regional overlap and pixel-level discrimination capability, and applies a stable and adaptive supervision signal to anatomical structures with significant differences in scale and volume (such as small lesions and slender blood vessels), thereby enhancing the robustness and generalization ability of model training.

[0084] In summary, this application forms a closed-loop synergy from three dimensions: structural alignment, semantic consistency preservation, and fine reconstruction. In principle, it breaks through the bottlenecks of existing multimodal segmentation methods in terms of insufficient cross-modal registration, coarse fusion mechanism, and weak detail recovery ability, and provides a systematic solution for high-precision and robust intelligent analysis of multimodal medical images.

[0085] Figure 6 A structural diagram of a multimodal image data segmentation device provided in an embodiment of this application is shown below. Figure 6 As shown, based on the multimodal image data segmentation method provided in the preceding embodiments, this application also provides a multimodal image data segmentation apparatus, including: The acquisition module is used to acquire multimodal image data; A segmentation module is used to segment the multimodal image data using a segmentation model to obtain the segmentation result of the multimodal image data output by the segmentation model; the segmentation model includes at least a multi-level encoding layer and a multi-level decoding layer; the multi-level encoding layer is used to perform cross-modal and cross-window feature enhancement processing on the multimodal image data; the multi-level decoding layer is used to perform semantic complementary fusion processing on the enhanced features of the multimodal image data.

[0086] As an optional embodiment, the multimodal image data includes multiple modal images; the segmentation module specifically includes: The encoding unit is used to encode the multimodal image data using the multi-level encoding layers to obtain enhanced data for each modality image; the enhanced data includes enhanced features for each level; The decoding unit is used to perform interactive fusion processing on the enhanced data of each modal image using the multi-level decoding layers to obtain the fusion result of each modal image; the fusion result includes the fusion features of each level; The segmentation unit is used to obtain the segmentation result of the multimodal image data based on the fusion result of the multimodal image data and each modal image.

[0087] As an optional embodiment, the segmentation model includes a four-level coding layer, and the coding unit specifically includes: The first-level coding layer is used to enhance the features of each modality image to obtain the first enhanced feature of each modality image; The second-level coding layer is used to enhance the first enhancement feature of each modality image to obtain the second enhancement feature of each modality image; The second enhancement feature of each modality image is enhanced using the third-level coding layer to obtain the third enhancement feature of each modality image; The third enhancement feature of each modality image is enhanced using the fourth-level coding layer to obtain the fourth enhancement feature of each modality image; The first enhancement feature, the second enhancement feature, the third enhancement feature, and the fourth enhancement feature are used as enhancement data for the corresponding modality image.

[0088] As an optional embodiment, feature enhancement is performed on each modality image using the first-level coding layer to obtain the first enhanced feature of each modality image, specifically including: Feature extraction is performed on each modality image to obtain the encoded features of each modality image; The encoded features of each modality image are divided into multiple local three-dimensional windows. Within the same local three-dimensional window, a correspondence is established between the encoded features of multiple modality images to obtain the attention enhancement features of each modality image. Within the same local 3D window, intra-window enhancement processing is performed on the corresponding modal image based on attention enhancement features and coding features. Between adjacent local 3D windows, cross-window enhancement processing is performed on the corresponding modal image based on attention enhancement features and coding features to obtain the first enhancement feature of each modal image.

[0089] As an optional embodiment, the segmentation model includes a three-level decoding layer and segmentation units, specifically including: Perform the following on the augmented data of any modality image: The fourth stitching feature and the third enhancement feature of the modal image are input into the third-level decoding layer to obtain the first fusion feature of the modal image; the fourth stitching feature is obtained by stitching the fourth enhancement feature of the modal image and the fourth enhancement feature of any modal image; the arbitrary modal image is any modal image other than the modal image in the multimodal image data; The first fusion feature and the second enhancement feature of the modal image are input into the second-level decoding layer to obtain the second fusion feature of the modal image; The second fusion feature and the third enhancement feature of the modal image are input into the first-level decoding layer to obtain the third fusion feature of each modal image; The first fusion feature, the second fusion feature, and the third fusion feature are used as the fusion result of the modal image.

[0090] As an optional embodiment, the fourth stitching feature and the third enhancement feature of the modal image are input into the third-level decoding layer to obtain the first fusion feature of the modal image, specifically including: The fourth enhancement feature of the modal image is concatenated with the fourth enhancement feature of any modal image to obtain the fourth concatenated feature of the modal image; The third enhancement feature of the modal image is semantically mapped to obtain the mapped enhancement feature of the modal image; Based on the fourth stitching feature of the modal image, the spatial offset of the modal image is obtained; Based on the spatial offset and mapping enhancement features of the modal image, the third enhancement feature of the arbitrary modal image is resampled to obtain the sampled feature; The sampling features and the mapping enhancement features of the modal image are divided into multiple local three-dimensional windows respectively. Fine-grained feature modeling is performed within the same local three-dimensional window. Information transmission paths are established between adjacent local three-dimensional windows to propagate local features across regions, thereby obtaining the cross-modal enhancement features of the modal image. The cross-modal enhancement features and mapping enhancement features of the modal image are fused to obtain the first fused feature of the modal image.

[0091] This application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a multimodal image data segmentation method.

[0092] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements a multimodal image data segmentation method.

[0093] This application provides a computer program product, including a computer program that, when executed by a processor, implements a multimodal image data segmentation method.

[0094] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the device and equipment embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments. The device and equipment embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components indicated as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the solution in this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0095] The above description is merely one specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method of multi-modal image data segmentation, characterized in that, The method includes: Acquire multimodal image data; The multimodal image data is segmented using a segmentation model to obtain the segmentation result of the multimodal image data output by the segmentation model; the segmentation model includes at least a multi-level encoding layer and a multi-level decoding layer; the multi-level encoding layer is used to perform cross-modal and cross-window feature enhancement processing on the multimodal image data; the multi-level decoding layer is used to perform semantic complementary fusion processing on the enhanced features of the multimodal image data.

2. The multi-modal image data segmentation method of claim 1, wherein, The multimodal image data includes multiple modal images; the segmentation process of the multimodal image data using a segmentation model to obtain the segmentation result of the multimodal image data output by the segmentation model specifically includes: The multi-level coding layer is used to encode the multimodal image data to obtain enhanced data for each modality image; the enhanced data includes enhanced features for each level. The enhanced data of each modal image is interactively fused using the multi-level decoding layer to obtain a fusion result for each modal image; the fusion result includes the fusion features of each level; Based on the fusion result of the multimodal image data and each modal image, the segmentation result of the multimodal image data is obtained.

3. The multimodal image data segmentation method according to claim 2, characterized in that, The segmentation model includes a four-level coding layer. The process of encoding the multimodal image data using these multi-level coding layers to obtain enhanced data for each modality image specifically includes: The first-level coding layer is used to enhance the features of each modality image to obtain the first enhanced feature of each modality image; The second-level coding layer is used to enhance the first enhancement feature of each modality image to obtain the second enhancement feature of each modality image; The second enhancement feature of each modality image is enhanced using the third-level coding layer to obtain the third enhancement feature of each modality image; The third enhancement feature of each modality image is enhanced using the fourth-level coding layer to obtain the fourth enhancement feature of each modality image; The first enhancement feature, the second enhancement feature, the third enhancement feature, and the fourth enhancement feature are used as enhancement data for the corresponding modality image.

4. The multimodal image data segmentation method according to claim 3, characterized in that, The first-level coding layer is used to perform feature enhancement on each modality image to obtain the first enhanced feature of each modality image, specifically including: Feature extraction is performed on each modality image to obtain the encoded features of each modality image; The encoded features of each modality image are divided into multiple local three-dimensional windows. Within the same local three-dimensional window, a correspondence is established between the encoded features of multiple modality images to obtain the attention enhancement features of each modality image. Within the same local 3D window, intra-window enhancement processing is performed on the corresponding modal image based on attention enhancement features and coding features. Between adjacent local 3D windows, cross-window enhancement processing is performed on the corresponding modal image based on attention enhancement features and coding features to obtain the first enhancement feature of each modal image.

5. The multimodal image data segmentation method according to claim 2, characterized in that, The segmentation model includes a three-level decoding layer. The process of interactively fusing the enhanced data of each modal image using the multi-level decoding layer to obtain the fusion result of each modal image specifically includes: Perform the following on the augmented data of any modality image: The fourth stitching feature and the third enhancement feature of the modal image are input into the third-level decoding layer to obtain the first fusion feature of the modal image; the fourth stitching feature is obtained by stitching the fourth enhancement feature of the modal image and the fourth enhancement feature of any modal image; the arbitrary modal image is any modal image other than the modal image in the multimodal image data; The first fusion feature and the second enhancement feature of the modal image are input into the second-level decoding layer to obtain the second fusion feature of the modal image; The second fusion feature and the third enhancement feature of the modal image are input into the first-level decoding layer to obtain the third fusion feature of each modal image; The first fusion feature, the second fusion feature, and the third fusion feature are used as the fusion result of the modal image.

6. The multimodal image data segmentation method according to claim 5, characterized in that, The step of inputting the fourth stitching feature and the third enhancement feature of the modal image into the third-level decoding layer to obtain the first fusion feature of the modal image specifically includes: The fourth enhancement feature of the modal image is concatenated with the fourth enhancement feature of any modal image to obtain the fourth concatenated feature of the modal image; The third enhancement feature of the modal image is semantically mapped to obtain the mapped enhancement feature of the modal image; Based on the fourth stitching feature of the modal image, the spatial offset of the modal image is obtained; Based on the spatial offset and mapping enhancement features of the modal image, the third enhancement feature of the arbitrary modal image is resampled to obtain the sampled feature; The sampling features and the mapping enhancement features of the modal image are divided into multiple local three-dimensional windows respectively. Fine-grained feature modeling is performed within the same local three-dimensional window. Information transmission paths are established between adjacent local three-dimensional windows to propagate local features across regions, thereby obtaining the cross-modal enhancement features of the modal image. The cross-modal enhancement features and mapping enhancement features of the modal image are fused to obtain the first fused feature of the modal image.

7. A multimodal image data segmentation device, characterized in that, The multimodal image data segmentation device includes: The acquisition module is used to acquire multimodal image data; A segmentation module is used to segment the multimodal image data using a segmentation model to obtain the segmentation result of the multimodal image data output by the segmentation model; the segmentation model includes at least a multi-level encoding layer and a multi-level decoding layer; the multi-level encoding layer is used to perform cross-modal and cross-window feature enhancement processing on the multimodal image data; the multi-level decoding layer is used to perform semantic complementary fusion processing on the enhanced features of the multimodal image data.

8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the multimodal image data segmentation method according to any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the multimodal image data segmentation method according to any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the multimodal image data segmentation method according to any one of claims 1-6.

Images