Multi-modal liver tumor segmentation method and device, electronic equipment and storage medium

Abstract

The invention provides a multi-mode liver tumor segmentation method and device, electronic equipment and a storage medium, which are applied to the technical field of image processing, and the method comprises the following steps: obtaining an original liver tumor image; performing image synthesis on the modal prompts associated with the multiple modals and the liver background images of the multiple modals through a pre-trained multi-modal image synthesis sub-model to obtain a target synthesized multi-modal image; through a pre-trained multi-modal image segmentation sub-model, performing image segmentation on the modal prompts associated with the plurality of modalities and the target synthesized multi-modal image to obtain a target liver tumor segmentation image; according to the invention, through the multi-modal image synthesis sub-model and the multi-modal image segmentation sub-model, especially through the modal prompt, the modal driving model and the context perception fusion module in the models, learning of correlation characteristics between different modals is realized; therefore, the target liver tumor segmentation image is obtained from the original liver tumor images in multiple modes, and the liver tumor image segmentation precision is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a multimodal liver tumor segmentation method, device, electronic device and storage medium. Background Art

[0002] Among related technologies, multimodal liver tumor segmentation has demonstrated significant application value in liver lesion screening and clinical decision support by integrating complementary soft tissue features and multidimensional imaging information. Conventional computed tomography scan phases include the non-enhanced phase, arterial phase, portal venous phase, and delayed phase, each of which can present specific anatomical structures and pathological features. In practical applications, since a single imaging modality cannot fully capture the complex tissue information of an organ, it is necessary to combine multiple phases to reveal pathological changes.

[0003] Proposed liver tumor segmentation techniques can be summarized as follows: 1) threshold-based and spatially constrained segmentation methods; 2) traditional algorithms that combine local features with learners; and 3) deep neural network architectures. Deep learning, particularly convolutional neural networks, is a data-driven approach that can automatically learn feature representations through end-to-end optimization. However, its effectiveness is heavily dependent on large-scale labeled data.

[0004] However, the above existing methods are generally heavily dependent on large amounts of labeled training data. Due to factors such as limited scanning time, image corruption caused by motion or artifacts, and the use of different imaging protocols, it can be challenging to obtain complete multimodal images for each patient. In addition, manual delineation is time-consuming and labor-intensive, and there is a large variability between and within raters, resulting in extremely high label acquisition costs for multimodal images. In addition, most current deep learning-based multimodal segmentation methods assume that the source images being segmented are strictly aligned at the pixel level. Only when this assumption holds true can the multimodal segmentation algorithm produce the expected results. For these reasons, current multimodal segmentation algorithms are difficult to apply to liver tumor segmentation images, resulting in low segmentation accuracy. Summary of the Invention

[0005] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application proposes a multimodal liver tumor segmentation method, device, electronic device and storage medium, aiming to improve the accuracy of liver tumor image segmentation.

[0006] In a first aspect, an embodiment of the present application provides a multimodal liver tumor segmentation method, the method comprising: Acquire an original liver tumor image set, where the original liver tumor image set includes original liver tumor images of multiple modalities; performing tumor removal on each of the original liver tumor images to obtain a liver background image of each modality; A pre-trained multimodal image synthesis sub-model is used to synthesize the first modal prompts associated with multiple modalities and the liver background images of multiple modalities to obtain a target synthesized multimodal image; wherein the multimodal image synthesis sub-model includes a feature encoder, a diffusion encoder, a first modal driving module, a diffusion decoder, and a feature decoder that are cascaded in sequence, and the first modal prompt acts on the first modal driving module; Through a pre-trained multimodal image segmentation sub-model, image segmentation is performed on the second modal prompts associated with multiple modalities and the target synthetic multimodal image to obtain a target liver tumor segmentation image; wherein, the multimodal image segmentation sub-model includes a segmentation encoder, a second modal driving module, a segmentation decoder and a context-aware fusion module cascaded in sequence, and the second modal prompt acts on the second modal driving module.

[0007] According to some embodiments of the present application, the pre-trained multimodal image synthesis sub-model is used to synthesize the first modal prompt associated with multiple modalities and the liver background image of multiple modalities to obtain a target synthesized multimodal image, including: Performing mapping processing on the liver background image by the feature encoder to obtain an image potential representation; Performing a diffusion operation on the first modality prompts associated with multiple modalities and the image latent representation through the diffusion encoder, the first modality driving module, and the diffusion decoder to obtain a plurality of first latent features; The feature decoder performs image synthesis on the plurality of first potential features to obtain the target synthesized multimodal image.

[0008] According to some embodiments of the present application, the diffusion encoder, the first modality driving module, and the diffusion decoder perform a diffusion operation on the first modality prompts associated with multiple modalities and the image potential representation to obtain multiple first potential features, including: adding noise to the latent representation image by the diffusion encoder to obtain a plurality of second latent features; Performing modality driving on the first modality prompts associated with the multiple modalities and the multiple second latent features by the first modality driving module to obtain multiple third latent features; The diffusion decoder performs a time-inverse denoising process on each of the third latent features to obtain a plurality of denoised first latent features.

[0009] According to some embodiments of the present application, performing image segmentation on the second modality prompts associated with multiple modalities and the target synthetic multimodal image using a pre-trained multimodal image segmentation sub-model to obtain a target liver tumor segmentation image includes: encoding the target synthesized multimodal image of multiple modalities by the segmentation encoder to obtain liver tumor image coding features; Performing modality driving on the second modality prompt associated with the multiple modalities and the liver tumor image coding feature through the second modality driving module to obtain a deep liver tumor image coding feature; Decoding the deep liver tumor image encoding features by the segmentation decoder to obtain deep liver tumor image decoding features; The context-aware fusion module performs context-aware fusion on the deep liver tumor image decoding features to obtain a target liver tumor segmentation image.

[0010] According to some embodiments of the present application, the multimodal image synthesis sub-model and the multimodal image segmentation sub-model constitute a liver tumor segmentation model, the context-aware fusion module includes an average adaptive fusion module, and the liver tumor segmentation model is trained by the following steps: weighting the deep liver tumor image decoding features using the average adaptive fusion module to obtain attention weights of multiple modalities; Multiplying the attention weight and the deep liver tumor image decoding feature voxel by voxel to obtain a first result, and performing weighted sum processing on the first result to obtain an adaptive fusion feature map; extracting information from the original liver tumor images of the multiple modalities to obtain tumor annotations; performing loss calculation based on the tumor annotation, the target liver tumor segmentation image, and the adaptive fusion feature map to obtain a loss value; The liver tumor segmentation model is optimized according to the loss value to obtain the liver tumor segmentation model.

[0011] According to some embodiments of the present application, before performing tumor removal on each of the original liver tumor images to obtain a liver background image of each modality, the method further includes updating the original liver tumor image, wherein updating the original liver tumor image includes: performing segmentation processing on each of the original liver tumor images to obtain a tumor region image; performing a connected region analysis on the tumor region image to obtain one or more tumor masses in the tumor region image; Calculating the number of voxels of the tumor mass to obtain the number of voxels of the tumor mass; The image resolution of each original liver tumor image is obtained. When the number of voxels is less than a first preset threshold or the product of the number of voxels and the image resolution is less than a second preset threshold, the tumor mass in the original liver tumor image is ignored, and the original liver tumor image is updated to obtain an updated original liver tumor image.

[0012] According to some embodiments of the present application, performing tumor removal on each of the original liver tumor images to obtain a liver background image of each modality includes: Obtaining a binary mask of each of the original liver tumor images; stacking each of the original liver tumor images and the binary mask to obtain an input liver tumor image; The input liver tumor image is convolved using a preset restoration network to obtain a liver background image of each modality.

[0013] In a second aspect, an embodiment of the present application provides a multimodal liver tumor segmentation apparatus, comprising: an original liver tumor image set acquisition module, configured to acquire an original liver tumor image set, wherein the original liver tumor image set includes original liver tumor images of multiple modalities; a liver background image generation module, configured to perform tumor removal on each of the original liver tumor images to obtain a liver background image of each modality; a synthesis module, configured to perform image synthesis on the first modal prompts associated with the multiple modalities and the liver background images of the multiple modalities using a pre-trained multimodal image synthesis sub-model to obtain a target synthesized multimodal image; wherein the multimodal image synthesis model includes a feature encoder, a diffusion encoder, a first modal driving module, a diffusion decoder, and a feature decoder cascaded in sequence, and the first modal prompt acts on the first modal driving module; A segmentation module is used to perform image segmentation on the second modality prompts associated with the multiple modalities and the target synthetic multimodal image through a pre-trained multimodal image segmentation sub-model to obtain a target liver tumor segmentation image; wherein the multimodal image segmentation model includes a segmentation encoder, a second modality driving module, a segmentation decoder and a context-aware fusion module cascaded in sequence, and the second modality prompt acts on the second modality driving module.

[0014] In a third aspect, an embodiment of the present application provides an electronic device comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the multimodal liver tumor segmentation method of the first aspect when executing the computer program.

[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the multimodal liver tumor segmentation method as described in the first aspect above.

[0016] According to the technical solution of the embodiment of the present application, there are at least the following beneficial effects: the embodiment of the present application proposes a multimodal liver tumor segmentation method, device, electronic device and storage medium, which are applied to the field of image processing technology, the method comprising: first acquiring an original liver tumor image, the original liver tumor image set including original liver tumor images of multiple modalities; then, using a pre-trained multimodal image synthesis sub-model, performing image synthesis on modal prompts associated with multiple modalities and liver background images of multiple modalities to obtain a target synthesized multimodal image; wherein the multimodal image synthesis model includes a diffusion encoder, a first modal driving module and a diffusion decoder cascaded in sequence, and the modal prompt acts on the first modal driving module; then, using a pre-trained multimodal image segmentation sub-model, performing image segmentation on the modal prompts associated with multiple modalities and the target synthesized multimodal image to obtain a target liver tumor segmentation image; wherein the multimodal image segmentation model includes a segmentation encoder, a second modal driving module, a segmentation decoder and a context-aware fusion module cascaded in sequence, and the modal prompt acts on the second modal driving module. Therefore, the embodiments of the present application can achieve learning of correlation features between different modalities through a multimodal image synthesis sub-model and a multimodal image segmentation sub-model, especially through modal prompts, a modality-driven model in the model, and a context-aware fusion module, thereby achieving the target liver tumor segmentation image from the original liver tumor images of multiple modalities, thereby improving the accuracy of liver tumor image segmentation.

[0017] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become obvious from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings are used to provide a further understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0019] Figure 1 This is a flowchart of a multimodal liver tumor segmentation method provided by one embodiment of the present application; Figure 2 is a flowchart of a multimodal liver tumor segmentation method provided by another embodiment of the present application; Figure 3 is a flowchart of a multimodal liver tumor segmentation method provided by another embodiment of the present application; Figure 4is a flowchart of a multimodal liver tumor segmentation method provided by another embodiment of the present application; Figure 5 is a flowchart of a multimodal liver tumor segmentation method provided by another embodiment of the present application; Figure 6 is a flowchart of a multimodal liver tumor segmentation method provided by another embodiment of the present application; Figure 7 is a flowchart of a multimodal liver tumor segmentation method provided by another embodiment of the present application; Figure 8 This is a flowchart of a multimodal liver tumor segmentation method provided by an overall embodiment of the present application; Figure 9 1 is a schematic structural diagram of a multimodal liver tumor segmentation device provided in one embodiment of the present application; Figure 10 Schematic diagram of an electronic device for performing a multimodal liver tumor segmentation method provided by an embodiment of the present application. DETAILED DESCRIPTION

[0020] The following describes in detail embodiments of the present application. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application and are not to be construed as limiting the present application.

[0021] In the description of this application, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, they cannot be understood as limitations on this application.

[0022] In the description of this application, "several" means one or more, "many" means more than two, "greater than," "less than," and "exceed" are understood to exclude the number itself, while "above," "below," and "within" are understood to include the number itself. The terms "first" and "second" are used solely to distinguish technical features and are not to be construed as indicating or implying relative importance, or as implicitly specifying the number or order of the technical features indicated.

[0023] In the description of this application, unless otherwise clearly defined, terms such as setting, installing, and connecting should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in this application based on the specific content of the technical solution.

[0024] In some cases, multimodal liver tumor segmentation has demonstrated significant application value in liver lesion screening and clinical decision support by integrating complementary soft tissue features and multidimensional imaging information. Conventional computed tomography scan phases include the non-enhanced phase, arterial phase, portal venous phase, and delayed phase, each of which can present specific anatomical structures and pathological features. In practical applications, since a single imaging modality cannot fully capture the complex tissue information of an organ, it is necessary to combine multiple phases to reveal pathological changes.

[0025] Proposed liver tumor segmentation techniques can be summarized as follows: 1) threshold-based and spatially constrained segmentation methods; 2) traditional algorithms that combine local features with learners; and 3) deep neural network architectures. Deep learning, particularly convolutional neural networks, is a data-driven approach that can automatically learn feature representations through end-to-end optimization. However, its effectiveness is heavily dependent on large-scale labeled data.

[0026] However, the above existing methods are generally heavily dependent on large amounts of labeled training data. Due to factors such as limited scanning time, image corruption caused by motion or artifacts, and the use of different imaging protocols, it can be challenging to obtain complete multimodal images for each patient. In addition, manual delineation is time-consuming and labor-intensive, and there is a large variability between and within raters, resulting in extremely high label acquisition costs for multimodal images. In addition, most current deep learning-based multimodal segmentation methods assume that the source images being segmented are strictly aligned at the pixel level. Only when this assumption holds true can the multimodal segmentation algorithm produce the expected results. For these reasons, current multimodal segmentation algorithms are difficult to apply to liver tumor segmentation images, resulting in low segmentation accuracy.

[0027] Based on the above situation, the embodiments of the present application propose a multimodal liver tumor segmentation method, device, electronic device and storage medium, aiming to improve the accuracy of liver tumor image segmentation.

[0028] The following further describes various embodiments of the multimodal liver tumor segmentation method of the present application in conjunction with the accompanying drawings.

[0029] like Figure 1 As shown, Figure 1 4 is a flowchart of a multimodal liver tumor segmentation method provided by an embodiment of the present application; the multimodal liver tumor segmentation method may include but is not limited to step S110, step S120, step S130 and step S140.

[0030] Step S110: acquiring an original liver tumor image set, where the original liver tumor image set includes original liver tumor images of multiple modalities; Step S120: performing tumor removal on each original liver tumor image to obtain a liver background image of each modality; Step S130: synthesizing the first modal prompts associated with multiple modalities and the liver background images of multiple modalities using a pre-trained multimodal image synthesis sub-model to obtain a target synthesized multimodal image; wherein the multimodal image synthesis sub-model includes a feature encoder, a diffusion encoder, a first modal driving module, a diffusion decoder, and a feature decoder that are cascaded in sequence, and the first modal prompt acts on the first modal driving module; Step S140: Perform image segmentation on the second modality prompts associated with multiple modalities and the target synthetic multimodal image through a pre-trained multimodal image segmentation sub-model to obtain a target liver tumor segmentation image; wherein the multimodal image segmentation sub-model includes a segmentation encoder, a second modality driving module, a segmentation decoder and a context-aware fusion module cascaded in sequence, and the second modality prompt acts on the second modality driving module.

[0031] In one embodiment, the original liver tumor image is first acquired, and the original liver tumor image set includes original liver tumor images of multiple modalities; then, a pre-trained multimodal image synthesis sub-model is used to synthesize the modal prompts associated with multiple modalities and the liver background images of multiple modalities to obtain a target synthesized multimodal image; then, a pre-trained multimodal image segmentation sub-model is used to segment the modal prompts associated with multiple modalities and the target synthesized multimodal image to obtain a target liver tumor segmentation image. Therefore, the embodiment of the present application can learn the correlation features between different modalities through the multimodal image synthesis sub-model and the multimodal image segmentation sub-model, especially through the modal prompts, the modal driving model in the model and the context-aware fusion module, so as to obtain the target liver tumor segmentation image from the original liver tumor images of multiple modalities, thereby improving the accuracy of liver tumor image segmentation.

[0032] It can be understood that the multimodal image synthesis sub-model includes a feature encoder, a diffusion encoder, a first modality driving module, a diffusion decoder and a feature decoder that are cascaded in sequence, and the first modality prompt acts on the first modality driving module.

[0033] It can be understood that the multimodal image segmentation sub-model includes a segmentation encoder, a second modality driving module, a segmentation decoder and a context-aware fusion module that are cascaded in sequence, and the second modality prompt acts on the second modality driving module.

[0034] In addition, if Figure 2 As shown, Figure 2This is a flowchart of a multimodal liver tumor segmentation method provided by another embodiment of the present application; regarding the pre-trained multimodal image synthesis sub-model in the above-mentioned step S130, the first modal prompts associated with multiple modalities and the liver background images of multiple modalities are synthesized to obtain a target synthesized multimodal image, which may include but is not limited to steps S210, S220 and S230.

[0035] Step S210: mapping the liver background image through a feature encoder to obtain a potential image representation; Step S220: performing a diffusion operation on the first modality prompts and image latent representations associated with multiple modalities through a diffusion encoder, a first modality driving module, and a diffusion decoder to obtain multiple first latent features; Step S230: Perform image synthesis on the plurality of first latent features through a feature decoder to obtain a target synthesized multimodal image.

[0036] In one embodiment, after the liver background image enters the pre-trained multimodal image synthesis sub-model, the liver background image is first mapped by a feature encoder to obtain an image latent representation; then, in a low-dimensional latent space, a diffusion operation is performed on the first modal cues and image latent representations associated with multiple modalities by a diffusion encoder, a first modal driving module, and a diffusion decoder to obtain multiple first latent features; finally, the multiple first latent features are synthesized into images by a feature decoder to obtain a target synthetic multimodal image.

[0037] It can be understood that after the liver background image is mapped by the feature encoder, the liver background image will enter the low-dimensional latent space, thereby obtaining the image potential representation.

[0038] It is understandable that the target synthetic multimodal image can also be mapped back to the digital space through the feature decoder, where the digital space refers to the liver background image data before inputting the feature encoder and the target synthetic multimodal image after the feature decoder outputs.

[0039] For example, assuming that the data space is x and the low-dimensional latent space is z, the image data will be mapped to the low-dimensional latent space z through the feature encoder, and then the diffusion operation will be performed in the low-dimensional latent space z. Finally, the image data after the diffusion operation will be synthesized and mapped back to the data space x through the feature decoder.

[0040] In addition, if Figure 3 As shown, Figure 3 This is a flowchart of a multimodal liver tumor segmentation method provided by another embodiment of the present application; regarding the above-mentioned step S220, it may include but is not limited to step S310, step S320 and step S330.

[0041] Step S310: adding noise to the latent representation image through a diffusion encoder to obtain a plurality of second latent features; Step S320: Perform modal driving on the first modal prompts associated with the multiple modalities and the multiple second latent features through the first modal driving module to obtain multiple third latent features; Step S330: Perform inverse time denoising processing on each third latent feature through a diffusion decoder to obtain a plurality of denoised first latent features.

[0042] In one embodiment, in a low-dimensional latent space, first, a diffusion encoder is used to add noise to the latent representation image in each time step tϵ{1,…,T} to obtain a plurality of second latent features; then, a first modality driving module is used to modally drive the first modality cues associated with the plurality of modalities and the plurality of second latent features to obtain a plurality of third latent features; finally, a diffusion encoder is used to gradually denoise each third latent feature in reverse time step from T to 1 to obtain a plurality of denoised first latent features.

[0043] It can be understood that multimodal images have different image properties, intensity distributions, and noise patterns. By incorporating modality-specific prior information into the first modality driving module through modal cues, the ability to characterize multimodal images can be enhanced.

[0044] It can be understood that the first modal driving module performs modal driving through the first modal prompt and multiple second potential features, and can obtain an updated modal prior and multiple third potential features that integrate modal information, thereby realizing adaptive fusion of features; among which, this fusion process can enable the first modal driving module to dynamically adjust the influence weight of the modal prior on the feature map, thereby enhancing the first modal driving module's perception ability of modal-specific features.

[0045] In addition, if Figure 4 As shown, Figure 4 This is a flowchart of a multimodal liver tumor segmentation method provided by another embodiment of the present application; regarding the above-mentioned step S140, using the pre-trained multimodal image segmentation sub-model, image segmentation is performed on the second modal prompts associated with multiple modalities and the target synthetic multimodal image to obtain a target liver tumor segmentation image, which may include but is not limited to steps S410, step S420, step S430 and step S440.

[0046] Step S410: encoding the target synthesized multimodal image of multiple modalities by a segmentation encoder to obtain a coding feature of the liver tumor image; Step S420: Modally drive the second modality prompts associated with the multiple modalities and the liver tumor image coding features through the second modality driving module to obtain the deep liver tumor image coding features; Step S430: Decoding the deep liver tumor image encoding features by a segmentation decoder to obtain deep liver tumor image decoding features; Step S440: Using a context-aware fusion module, context-aware fusion is performed on the deep liver tumor image decoding features to obtain a target liver tumor segmentation image.

[0047] It is understandable that multimodal images have different image properties, intensity distributions, and noise patterns. By incorporating modality-specific prior information into the second modality driving module through modal cues, the ability to characterize multimodal images can be enhanced.

[0048] It can be understood that the second modality driving module is modally driven through the second modality prompt and the liver tumor image coding features, and can obtain the updated modality prior and the deep liver tumor image coding features that integrate the modality information, thereby realizing adaptive fusion of features; among them, this fusion process can enable the first modality driving module to dynamically adjust the influence weight of the modality prior on the feature map, thereby enhancing the first modality driving module's perception ability of modality-specific features.

[0049] It can be understood that by fusing the deep liver tumor image decoding features through the context-aware fusion module and adjusting the weight ratio under each modality, the accuracy of the target liver tumor segmentation image can be improved.

[0050] In addition, if Figure 5 As shown, Figure 5 This is a flowchart of a multimodal liver tumor segmentation method provided by another embodiment of the present application; the multimodal image synthesis submodel and the multimodal image segmentation submodel constitute a liver tumor segmentation model, the context-aware fusion module includes an average adaptive fusion module, and the liver tumor segmentation model is obtained by training through the following steps, which may include but are not limited to step S510, step S520, step S530, step S540 and step S550.

[0051] Step S510: weighting the deep liver tumor image decoding features through the average adaptive fusion module to obtain attention weights of multiple modalities; Step S520: performing voxel-by-voxel multiplication on the attention weight and the deep liver tumor image decoding feature to obtain a first result, and performing weighted summation processing on the first result to obtain an adaptive fusion feature map; Step S530: extract information from the original liver tumor images of multiple modalities to obtain tumor annotations; Step S540: performing loss calculation based on the tumor annotation, the target liver tumor segmentation image, and the adaptive fusion feature map to obtain a loss value; Step S550: Optimize the liver tumor segmentation model according to the loss value to obtain a liver tumor segmentation model.

[0052] It can be understood that by decoding the deep liver tumor image encoding features through the segmentation decoder, the deep liver tumor image decoding features of multiple modal features can be obtained; in order to effectively aggregate image features from different modalities, the deep liver tumor image decoding features of multiple modal features are averaged through the Average-Adaptive Fusion (AaF) module to generate an average fusion feature map; then, the average fusion feature map is weighted by the convolutional layer to obtain the initial attention weights of multiple modalities, and the initial attention weights are normalized to obtain the attention weights of multiple modalities.

[0053] It is understandable that the initial attention weights of multiple modalities can be calculated by the following formula, ,in, Represents the S-type function, which is used to map features to the range [0, 1]. represents the learning parameters of two cascaded convolutional layers, represents the initial attention weight, represents the convolutional layer, represents the average fusion feature map, represents the deep liver tumor image decoding features, and i represents four modalities.

[0054] It is understandable that the attention weights of multiple modalities can be calculated using the following formula: ,in, represents the attention weight, and i represents the four modalities.

[0055] It can be understood that the loss function is obtained by the following formula, ,in, represents the loss function, k and The factor representing the weight ratio of the loss function, y represents the tumor label, represents the target liver tumor segmentation image, represents the average fusion feature map, Represents the adaptive fusion feature map.

[0056] In addition, if Figure 6 As shown, Figure 6This is a flowchart of a multimodal liver tumor segmentation method provided by another embodiment of the present application; before the above step S120, the multimodal liver tumor segmentation method may also include but is not limited to step S610, step S620, step S630 and step S640.

[0057] Step S610: Segment each original liver tumor image to obtain a tumor region image; Step S620: performing connected region analysis on the tumor region image to obtain one or more tumor masses in the tumor region image; Step S630: Calculate the number of voxels in the tumor mass to obtain the number of voxels in the tumor mass; Step S640: Obtain the image resolution of each original liver tumor image. When the number of voxels is less than a first preset threshold or the product of the number of voxels and the image resolution is less than a second preset threshold, ignore the tumor block in the original liver tumor image, and update the original liver tumor image to obtain an updated original liver tumor image.

[0058] In one embodiment, each original liver tumor image is segmented to obtain a tumor region image. Next, a connected region analysis is performed on the tumor region image to obtain one or more tumor blocks in the tumor region image. Then, each individual tumor block is traversed and the number of voxels is calculated to obtain the number of voxels in the tumor block. Finally, the image resolution of each original liver tumor image is obtained. When the number of voxels is less than a first preset threshold or the product of the number of voxels and the image resolution is less than a second preset threshold, the tumor block in the original liver tumor image is ignored and the original liver tumor image is updated to obtain an updated original liver tumor image.

[0059] It is understandable that by segmenting each original liver tumor image, images of the liver and tumor regions can be obtained. Connected region analysis is performed on the liver and tumor region images to find all connected regions. If multiple connected regions exist, only the largest region (i.e., the main liver region) is retained.

[0060] In addition, if Figure 7 As shown, Figure 7 This is a flowchart of a multimodal liver tumor segmentation method provided by another embodiment of the present application; regarding the above-mentioned step S120, it may include but is not limited to step S710, step S720 and step S730.

[0061] Step S710: obtaining a binary mask of each original liver tumor image; Step S720: stacking each original liver tumor image and the binary mask to obtain an input liver tumor image; Step S730: Perform convolution processing on the input liver tumor image through a preset restoration network to obtain a liver background image of each modality.

[0062] It is understandable that the above-mentioned repair network may be a feedforward repair network, which may be selected according to actual conditions and is not specifically limited in the embodiments of the present application.

[0063] Based on the multimodal liver tumor segmentation methods of the above-mentioned embodiments, the overall embodiments of the multimodal liver tumor segmentation method of the present application are respectively proposed below.

[0064] like Figure 8 As shown, Figure 8 This is a flowchart of a multimodal liver tumor segmentation method provided by an overall embodiment of the present application; 1.1 Multimodal Image Processing 1.1.1 Image Registration and Preprocessing First, for the original liver tumor image set, we truncated the intensity value of each volume to the range of [-21, 189] and then standardized it to have zero mean and unit standard deviation. The multimodal images were registered using a traditional registration method based on differential homogeneous deformation. Specifically, the portal venous phase image was used as the target image for registration, and the other three phase images were used as images to be registered. The degrees of freedom were set to 6, and the cost function of the registration was set to boundary-based registration. After two iterative registrations, the images of the four modalities, especially in the liver tumor area, showed good registration results. Finally, since tiny tumors are difficult to identify even with the naked eye, we excluded tumor areas with less than 8 voxels or a physical volume size less than 1. (number of voxels × image resolution) masks.

[0065] The specific steps are as follows: (1) Data loading: Read the original liver tumor images of multiple modalities one by one and extract information such as pixel spacing (resolution); (2) Separation of organs and tumor regions: generating regions containing both liver and tumor and regions containing only tumor; (3) Ensure the integrity of the liver region: Perform a connected region analysis on the liver and tumor regions to find all connected regions. If there are multiple connected regions, only the largest region (i.e., the main liver region) is retained. (4) Screening tumor areas: Perform connected region analysis on the tumor area to find all separate tumor blocks. Traverse each tumor block and calculate the number of voxels. If the number of voxels is greater than 8 or the number of voxels × image resolution is greater than 1, , then the corresponding pixel value in the liver and tumor area is updated to 2 (i.e., the tumor area), otherwise the tumor block is ignored.

[0066] (5) Save the processed data: After processing is completed, save the updated data containing the liver and tumor areas as a new label file.

[0067] 1.1.2 Liver background image generation First, a binary mask of each original liver tumor image is obtained. Then, each original liver tumor image and the binary mask are stacked to obtain an input liver tumor image. Then, the input liver tumor image is convolved with a preset restoration network to obtain a liver background image for each modality.

[0068] 1.2 Multimodal Image Segmentation Based on Online Synthesis 1.2.1 Synthetic Model The data space refers to the image data before it is input into the feature encoder and the image data after it is output. Image data processing typically occurs in the data space, but after passing through the feature encoder, it is compressed into a low-dimensional latent space. After processing, it is reconstructed back to its original size.

[0069] Assuming the data space is X and the low-dimensional latent space is Z, we simulate a diffusion process in the latent space of a pretrained feature encoder to synthesize the foreground of interest. First, we construct an autoencoder architecture based on a variational feature encoder, implementing a feature encoder (mapping from digital space to low-dimensional latent space) and a feature decoder (mapping from low-dimensional latent space to digital space). Next, we perform a diffusion operation in the low-dimensional latent space Z, which consists of a forward and a backward process. The forward process uses the diffusion encoder to gradually add noise to the latent representation image, generating a series of increasingly noisy second latent features z1, z2, ..., zT. Then, the Modality Prompting-Driven (MPD) module modally drives the first modality cues and multiple second latent features associated with multiple modalities to generate multiple third latent features. The diffusion decoder then performs a reverse-temporal denoising process on each third latent feature, generating multiple denoised first latent features. Finally, the feature decoder uses these denoised first latent features to generate the target synthetic multimodal image.

[0070] 1.2.2 Modal Prompt Driven Strategy Each modality is represented by one or more tokens of length l, which capture modality-specific information such as intensity distribution, texture features, and noise patterns. For a modality-specific image, we select the corresponding modality prior token from the prior pool. In the backbone networks of the multimodal image synthesis and multimodal image segmentation sub-models, the encoded feature maps are then adaptively fused with the task context prior tokens and image features using the prior fusion context-aware fusion module (feature fusion based on a bidirectional cross-attention mechanism).

[0071] 1.3 Context-aware fusion module The specific workflow of the Context-Aware Fusion (CAF) module is as follows: Acquisition of input features: After passing through the multimodal image segmentation sub-model, a set of liver tumor image encoding features is output. These intermediate features are input into the decoder in the multimodal image segmentation sub-model to first obtain a set of deep liver tumor image decoding features.

[0072] Acquisition of attention map: The context-aware fusion module calculates the attention weights between multimodal features and obtains the corresponding attention map.

[0073] Feature adaptive fusion: The segmentation features are fused through an adaptive fusion network, in which the weight ratio of the arterial phase and the portal venous phase is increased so that the model pays more attention to the characteristics of these two modalities.

[0074] Output update: The context-aware fusion module outputs the updated features, calculates the loss function together with the output result graph and the true annotation, and updates the model parameters.

[0075] Based on the multimodal liver tumor segmentation methods of the above-mentioned embodiments, various embodiments of the multimodal liver tumor segmentation device of the present application are respectively proposed below.

[0076] like Figure 9 As shown, Figure 9 1 is a schematic structural diagram of a multimodal liver tumor segmentation device provided in one embodiment of the present application; In one embodiment, a multimodal liver tumor segmentation apparatus comprises: an original liver tumor image set acquisition module, used to acquire an original liver tumor image set, wherein the original liver tumor image set includes original liver tumor images of multiple modalities; A liver background image generation module is used to perform tumor removal on each original liver tumor image to obtain a liver background image of each modality; a synthesis module configured to synthesize the first modality cues associated with multiple modalities and the liver background images of multiple modalities using a pre-trained multimodal image synthesis sub-model to obtain a target synthesized multimodal image; wherein the multimodal image synthesis model includes a feature encoder, a diffusion encoder, a first modality driving module, a diffusion decoder, and a feature decoder in cascade order, and the first modality cues act on the first modality driving module; A segmentation module is used to perform image segmentation on a target synthetic multimodal image of a plurality of modalities associated with a second modality prompt and a pre-trained multimodal image segmentation sub-model to obtain a target liver tumor segmentation image; wherein the multimodal image segmentation model includes a segmentation encoder, a second modality driving module, a segmentation decoder and a context-aware fusion module cascaded in sequence, and the second modality prompt acts on the second modality driving module.

[0077] The contents of the above-mentioned multimodal liver tumor segmentation method embodiment are all applicable to the multimodal liver tumor segmentation device embodiment. The functions specifically implemented by this multimodal liver tumor segmentation device embodiment are the same as those of the above-mentioned multimodal liver tumor segmentation method embodiment, and the beneficial effects achieved are also the same as those achieved by the above-mentioned multimodal liver tumor segmentation method embodiment.

[0078] Based on the multimodal liver tumor segmentation methods of the above-mentioned embodiments, various embodiments of the electronic device and computer-readable storage medium of the present application are respectively proposed below.

[0079] like Figure 10 As shown, Figure 10 Schematic diagram of an electronic device for performing a multimodal liver tumor segmentation method provided by an embodiment of the present application. The electronic device 700 implemented in the present application includes: a processor 710, a memory 720, and a computer program stored in the memory 720 and executable on the processor 710, wherein: Figure 10 In the figure, a processor 710 and a memory 720 are taken as an example.

[0080] The processor 710 and the memory 720 may be connected via a bus or other means. Figure 10 The bus connection is taken as an example.

[0081] The memory 720 is a non-transitory computer-readable storage medium that can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory 720 may include a high-speed random access memory and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 720 may optionally include a memory 720 remotely located relative to the processor 710, and these remote memories 720 may be connected to the electronic device 700 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0082] Those skilled in the art will understand that Figure 10 The device structure shown in the figure does not constitute a limitation on the electronic device 700, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0083] exist Figure 10 In the illustrated electronic device 700, processor 710 can be used to invoke a control program stored in memory 720 to implement the aforementioned multimodal liver tumor segmentation method. Specifically, the non-transitory software program and instructions required to implement the aforementioned multimodal liver tumor segmentation method are stored in memory 720 and, when executed by processor 710, perform the aforementioned multimodal liver tumor segmentation method.

[0084] It is worth noting that since the electronic device 700 of the embodiment of the present application can execute the multimodal liver tumor segmentation method of any of the above embodiments, the specific implementation and technical effects of the electronic device 700 of the embodiment of the present application can refer to the specific implementation and technical effects of the multimodal liver tumor segmentation method of any of the above embodiments.

[0085] In addition, an embodiment of the present application further provides a computer-readable storage medium, which stores computer-executable instructions for executing the above-described multimodal liver tumor segmentation method. Figures 1 to 7 The method steps in .

[0086] It is worth noting that since the computer-readable storage medium of the embodiment of the present application can execute the multimodal liver tumor segmentation method of any of the above embodiments, the specific implementation and technical effects of the computer-readable storage medium of the embodiment of the present application can refer to the specific implementation and technical effects of the multimodal liver tumor segmentation method of any of the above embodiments.

[0087] Those skilled in the art will appreciate that all or some of the steps and systems disclosed above can be implemented as software, firmware, hardware, or any suitable combination thereof. Some or all of the physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on computer-readable media, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is well known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disks (DVDs) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. Furthermore, as is well known to those skilled in the art, communication media typically includes computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0088] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0089] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0090] It should also be understood that the various implementation methods provided in the embodiments of the present application can be combined arbitrarily to achieve different technical effects.

[0091] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the above implementation mode. Technical personnel familiar with the art can also make various equivalent modifications or substitutions under the shared conditions that do not violate the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A multimodal liver tumor segmentation method, characterized in that: The method comprises: Acquire an original liver tumor image set, where the original liver tumor image set includes original liver tumor images of multiple modalities; performing tumor removal on each of the original liver tumor images to obtain a liver background image of each modality; A pre-trained multimodal image synthesis sub-model is used to synthesize the first modal prompts associated with multiple modalities and the liver background images of multiple modalities to obtain a target synthesized multimodal image; wherein the multimodal image synthesis sub-model includes a feature encoder, a diffusion encoder, a first modal driving module, a diffusion decoder, and a feature decoder that are cascaded in sequence, and the first modal prompt acts on the first modal driving module; Through a pre-trained multimodal image segmentation sub-model, image segmentation is performed on the second modal prompts associated with multiple modalities and the target synthetic multimodal image to obtain a target liver tumor segmentation image; wherein, the multimodal image segmentation sub-model includes a segmentation encoder, a second modal driving module, a segmentation decoder and a context-aware fusion module cascaded in sequence, and the second modal prompt acts on the second modal driving module.

2. The method according to claim 1, characterized in that The method comprises synthesizing the first modal prompts associated with multiple modalities and the liver background images of multiple modalities through the pre-trained multimodal image synthesis sub-model to obtain a target synthesized multimodal image, including: Performing mapping processing on the liver background image by the feature encoder to obtain an image potential representation; Performing a diffusion operation on the first modality prompts associated with multiple modalities and the image latent representation through the diffusion encoder, the first modality driving module, and the diffusion decoder to obtain a plurality of first latent features; The feature decoder performs image synthesis on the plurality of first potential features to obtain the target synthesized multimodal image.

3. The method according to claim 2, characterized in that The diffusion encoder, the first modality driving module, and the diffusion decoder perform a diffusion operation on the first modality prompts associated with multiple modalities and the image potential representation to obtain multiple first potential features, including: adding noise to the latent representation image by the diffusion encoder to obtain a plurality of second latent features; Performing modality driving on the first modality prompts associated with the multiple modalities and the multiple second latent features by the first modality driving module to obtain multiple third latent features; The diffusion decoder performs a time-inverse denoising process on each of the third latent features to obtain a plurality of denoised first latent features.

4. The method according to claim 1, wherein The method includes performing image segmentation on the second modality prompts associated with multiple modalities and the target synthesized multimodal image through the pre-trained multimodal image segmentation sub-model to obtain a target liver tumor segmentation image, including: encoding the target synthesized multimodal image of multiple modalities by the segmentation encoder to obtain liver tumor image coding features; Performing modality driving on the second modality prompt associated with the multiple modalities and the liver tumor image coding feature through the second modality driving module to obtain a deep liver tumor image coding feature; Decoding the deep liver tumor image encoding features by the segmentation decoder to obtain deep liver tumor image decoding features; The context-aware fusion module performs context-aware fusion on the deep liver tumor image decoding features to obtain a target liver tumor segmentation image.

5. The method according to any one of claims 1 to 4, characterized in that The multimodal image synthesis sub-model and the multimodal image segmentation sub-model constitute a liver tumor segmentation model. The context-aware fusion module includes an average adaptive fusion module. The liver tumor segmentation model is trained by the following steps: weighting the deep liver tumor image decoding features using the average adaptive fusion module to obtain attention weights of multiple modalities; Multiplying the attention weight and the deep liver tumor image decoding feature voxel by voxel to obtain a first result, and performing weighted sum processing on the first result to obtain an adaptive fusion feature map; extracting information from the original liver tumor images of the multiple modalities to obtain tumor annotations; performing loss calculation based on the tumor annotation, the target liver tumor segmentation image, and the adaptive fusion feature map to obtain a loss value; The liver tumor segmentation model is optimized according to the loss value to obtain the liver tumor segmentation model.

6. The method according to any one of claims 1 to 4, characterized in that Before performing tumor removal on each of the original liver tumor images to obtain a liver background image of each modality, the method further includes updating the original liver tumor image, wherein updating the original liver tumor image includes: performing segmentation processing on each of the original liver tumor images to obtain a tumor region image; performing a connected region analysis on the tumor region image to obtain one or more tumor masses in the tumor region image; Calculating the number of voxels of the tumor mass to obtain the number of voxels of the tumor mass; The image resolution of each original liver tumor image is obtained. When the number of voxels is less than a first preset threshold or the product of the number of voxels and the image resolution is less than a second preset threshold, the tumor mass in the original liver tumor image is ignored, and the original liver tumor image is updated to obtain an updated original liver tumor image.

7. The method according to any one of claims 1 to 4, characterized in that The step of performing tumor removal on each of the original liver tumor images to obtain a liver background image of each modality includes: Obtaining a binary mask of each of the original liver tumor images; stacking each of the original liver tumor images and the binary mask to obtain an input liver tumor image; The input liver tumor image is convolved using a preset restoration network to obtain a liver background image of each modality.

8. A multimodal liver tumor segmentation device, characterized in that: include: an original liver tumor image set acquisition module, configured to acquire an original liver tumor image set, wherein the original liver tumor image set includes original liver tumor images of multiple modalities; a liver background image generation module, configured to perform tumor removal on each of the original liver tumor images to obtain a liver background image of each modality; a synthesis module, configured to perform image synthesis on the first modal prompts associated with the multiple modalities and the liver background images of the multiple modalities using a pre-trained multimodal image synthesis sub-model to obtain a target synthesized multimodal image; wherein the multimodal image synthesis model includes a feature encoder, a diffusion encoder, a first modal driving module, a diffusion decoder, and a feature decoder cascaded in sequence, and the first modal prompt acts on the first modal driving module; A segmentation module is used to perform image segmentation on the second modality prompts associated with the multiple modalities and the target synthetic multimodal image through a pre-trained multimodal image segmentation sub-model to obtain a target liver tumor segmentation image; wherein the multimodal image segmentation model includes a segmentation encoder, a second modality driving module, a segmentation decoder and a context-aware fusion module cascaded in sequence, and the second modality prompt acts on the second modality driving module.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the multimodal liver tumor segmentation method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: Computer-executable instructions are stored, and the computer-executable instructions are used to execute the multimodal liver tumor segmentation method according to any one of claims 1 to 7.

Images