A multi-modal image segmentation method, device, equipment and storage medium

Abstract

The application is a multi-modal image segmentation method, device, equipment and storage medium, and particularly relates to the technical field of image detection. The method comprises the following steps: acquiring target image data of at least two modalities; for the target image data of each modality, sequentially processing through each layer down-sampling module in a target detection model to obtain image features of each layer; the target detection model further comprises each layer up-sampling module corresponding to each layer down-sampling module; fusing the image features of each layer according to the layer level to obtain fused features of each layer; based on the fused features of the corresponding layer level of each layer up-sampling module, sequentially processing the bottom fused features in the fused features of each layer through each layer up-sampling module to obtain a target segmentation result. Based on the above scheme, the image segmentation accuracy of the target detection model is improved.

Description

Technical Field

[0001] This application relates to the field of image detection, specifically to a multimodal image segmentation method, apparatus, device, and storage medium. Background Technology

[0002] Magnetic resonance imaging (MRI) is an imaging technique that reconstructs images from signals generated by the resonance of atomic nuclei within a magnetic field. Deep learning is a type of machine learning that aims to learn the inherent patterns and hierarchical representations of sample data in order to identify target data.

[0003] Traditional methods rely on experts to observe magnetic resonance images (MRI) to determine the presence of lesions. In recent years, deep learning has been increasingly applied to MRI image segmentation. This method requires manual annotation of lesion information in the MRI image to train the target detection model and obtain the segmentation results.

[0004] However, the above methods require a large number of samples and have low recognition accuracy when performing magnetic resonance image segmentation tasks. Summary of the Invention

[0005] This application provides a multimodal image segmentation method, apparatus, device, and storage medium, which achieves high recognition accuracy during image segmentation. The technical solution is as follows.

[0006] On the one hand, a multimodal image segmentation method is provided, the method comprising:

[0007] Acquire target image data in at least two modalities;

[0008] For each modality of target image data, it is processed sequentially through the downsampling modules of each layer in the target detection model to obtain the image features of each layer; the target detection model also includes upsampling modules corresponding to the downsampling modules of each layer respectively.

[0009] The image features of each layer are fused according to the layer to obtain the fused features of each layer;

[0010] Based on the fusion features of the corresponding layers of the upsampling modules, the bottom-level fusion features in each layer are processed sequentially through the upsampling modules to obtain the target segmentation result.

[0011] On another front, a multimodal image segmentation method is provided, the method comprising:

[0012] Acquire sample image data of at least two modalities; the sample image data contains data annotations;

[0013] For each modality of sample image data, the data is processed sequentially through the downsampling modules of each layer in the target detection model to obtain the sample image features of each layer; the target detection model also includes upsampling modules corresponding to the downsampling modules of each layer respectively;

[0014] The features of the sample images from each layer are fused according to the layer to obtain the fused features of the samples from each layer.

[0015] Based on the sample fusion features of the corresponding layers of each upsampling module, the bottom sample fusion features in each layer are processed sequentially through each layer upsampling module to obtain the sample output results.

[0016] Based on the sample output results and data annotations, the target detection model is trained to obtain a trained target detection model; the trained target detection model is used to process target image data to obtain target segmentation results.

[0017] In another aspect, a multimodal image segmentation apparatus is provided, the apparatus comprising:

[0018] The first data acquisition module is used to acquire target image data in at least two modalities;

[0019] The first downsampling module is used to process the target image data of each modality sequentially through the downsampling modules of each layer in the target detection model to obtain the image features of each layer; the target detection model also includes upsampling modules corresponding to the downsampling modules of each layer respectively.

[0020] The first feature fusion module is used to fuse the image features of each layer according to the layer to obtain the fused features of each layer;

[0021] The first upsampling module is used to process the bottom-level fusion features in each layer of the fusion features sequentially through the upsampling modules based on the fusion features of the corresponding layers of each upsampling module, so as to obtain the target segmentation result.

[0022] In one possible implementation, the upsampling modules at each layer include a bottom layer upsampling module, a top layer upsampling module, and an intermediate layer upsampling module;

[0023] The step of processing the bottom-level fusion features from the fusion features of each layer upsampling module sequentially through the upsampling modules to obtain the target segmentation result includes:

[0024] The bottom-level upsampling module performs upsampling processing on the bottom-level fusion features to obtain the upsampling result output by the bottom-level upsampling module;

[0025] For each intermediate layer upsampling module, the upsampling result output by the lower layer upsampling module of the intermediate layer upsampling module is fused with the fusion feature of the corresponding layer of the intermediate layer upsampling module to obtain intermediate fusion features. The intermediate fusion features are then upsampled by the intermediate layer upsampling module to obtain the upsampling result output by the intermediate layer upsampling module.

[0026] The upsampling result output by the lower-level upsampling module of the top-level upsampling module is fused with the fusion feature of the corresponding level of the top-level upsampling module, and the fused feature is processed by the top-level upsampling module to obtain the target segmentation result.

[0027] In one possible implementation, each downsampling module has at least two downsampling branches;

[0028] The target image data for each modality is processed sequentially through the downsampling modules of each layer in the target detection model to obtain the image features of each layer, including:

[0029] For each modality of target image data, the data is processed sequentially through the downsampling branch of the downsampling module in each layer of the target detection model to obtain the image features of each layer of the corresponding modality.

[0030] In one possible implementation, the downsampling module includes a first convolution module, a second convolution module, a third convolution module, a first normalization module, a second normalization module, a third normalization module, and an activation module;

[0031] The first convolution module, the first normalization module, the third convolution module, and the second normalization module are connected in sequence to process the input image data to obtain the first result;

[0032] The second convolution module and the third normalization module are connected in sequence to process the input image data to obtain the second result;

[0033] The activation module is used to process the fusion result of the first result and the second result to obtain the image features of the corresponding level of the downsampling module.

[0034] In one possible implementation, in each downsampling module, the first convolutional module and the second convolutional module in the downsampling branches of different modalities are the same.

[0035] In another aspect, a multimodal image segmentation apparatus is provided, the apparatus comprising:

[0036] The second data acquisition module is used to acquire sample image data of at least two modalities; the sample image data contains data annotations.

[0037] The second downsampling module is used to process the sample image data of each modality sequentially through the downsampling modules of each layer in the target detection model to obtain the sample image features of each layer; the target detection model also includes upsampling modules corresponding to the downsampling modules of each layer respectively;

[0038] The second feature fusion module is used to fuse the features of the sample images of each layer according to the layer to obtain the fused features of the sample images of each layer.

[0039] The second upsampling module is used to process the bottom sample fusion features in each layer of the sample fusion features based on the sample fusion features of the corresponding layer of each upsampling module, and then process them through each layer of the upsampling module to obtain the sample output results.

[0040] The training module is used to train the target detection model based on the sample output results and data annotations to obtain the trained target detection model; the trained target detection model is used to process the target image data to obtain the target segmentation result.

[0041] In one possible implementation, the upsampling modules at each layer include a bottom layer upsampling module, a top layer upsampling module, and an intermediate layer upsampling module;

[0042] The step of processing the bottom-level fusion features from the fusion features of each layer upsampling module sequentially through the upsampling modules to obtain the target segmentation result includes:

[0043] The bottom-level upsampling module performs upsampling processing on the bottom-level fusion features to obtain the upsampling result output by the bottom-level upsampling module;

[0044] For each intermediate layer upsampling module, the upsampling result output by the lower layer upsampling module of the intermediate layer upsampling module is fused with the fusion feature of the corresponding layer of the intermediate layer upsampling module to obtain intermediate fusion features. The intermediate fusion features are then upsampled by the intermediate layer upsampling module to obtain the upsampling result output by the intermediate layer upsampling module.

[0045] The upsampling result output by the lower-level upsampling module of the top-level upsampling module is fused with the fusion feature of the corresponding level of the top-level upsampling module, and the fused feature is processed by the top-level upsampling module to obtain the target segmentation result.

[0046] In one possible implementation, each downsampling module has at least two downsampling branches;

[0047] The target image data for each modality is processed sequentially through the downsampling modules of each layer in the target detection model to obtain the image features of each layer, including:

[0048] For each modality of target image data, the data is processed sequentially through the downsampling branch of the downsampling module in each layer of the target detection model to obtain the image features of each layer of the corresponding modality.

[0049] In one possible implementation, the downsampling module includes a first convolution module, a second convolution module, a third convolution module, a first normalization module, a second normalization module, a third normalization module, and an activation module;

[0050] The first convolution module, the first normalization module, the third convolution module, and the second normalization module are connected in sequence to process the input image data to obtain the first result;

[0051] The second convolution module and the third normalization module are connected in sequence to process the input image data to obtain the second result;

[0052] The activation module is used to process the fusion result of the first result and the second result to obtain the image features of the corresponding level of the downsampling module.

[0053] In one possible implementation, in each downsampling module, the first convolutional module and the second convolutional module in the downsampling branches of different modalities are the same.

[0054] In another aspect, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one instruction, the at least one instruction being loaded and executed by the processor to implement the above-described multimodal image segmentation method.

[0055] In another aspect, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement the above-described multimodal image segmentation method.

[0056] In another aspect, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform the aforementioned multimodal image segmentation method.

[0057] The technical solution provided in this application may include the following beneficial effects:

[0058] This application first acquires target image data in at least two modalities; then, for each modal of target image data, it sequentially processes the target image data through the downsampling modules of each layer in the target detection model to obtain image features at each layer; the target detection model also includes upsampling modules corresponding to each downsampling module; then, the image features at each layer are fused hierarchically to obtain fused features at each layer; finally, based on the fused features corresponding to the layers of each upsampling module, the lowest-level fused features in each layer are sequentially processed through the upsampling modules to obtain the target segmentation result. This scheme, by fusing image features at each layer hierarchically to obtain fused features at each layer, and then obtaining the target segmentation result based on the fused features and the features obtained by upsampling through the upsampling modules, compensates for the information lost during downsampling to a certain extent, thereby improving the image segmentation accuracy of the target detection model. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0060] Figure 1 This is a schematic diagram illustrating the structure of a multimodal image segmentation system according to an exemplary embodiment.

[0061] Figure 2 This is a flowchart illustrating a multimodal image segmentation method according to an exemplary embodiment.

[0062] Figure 3 This is a flowchart illustrating a multimodal image segmentation method according to an exemplary embodiment.

[0063] Figure 4 This is a flowchart illustrating a multimodal image segmentation method according to an exemplary embodiment.

[0064] Figure 5 A schematic diagram of the target detection model involved in an embodiment of this application is shown.

[0065] Figure 6 A schematic diagram of the downsampling module involved in an embodiment of this application is shown.

[0066] Figure 7 An example diagram of feature fusion involved in an embodiment of this application is shown.

[0067] Figure 8 This is a structural block diagram of a multimodal image segmentation apparatus according to an exemplary embodiment.

[0068] Figure 9 This is a structural block diagram of a multimodal image segmentation apparatus according to an exemplary embodiment.

[0069] Figure 10 This is a structural block diagram of a computer device according to an exemplary embodiment. Detailed Implementation

[0070] The technical solutions of this application will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0071] It should be understood that the term "instruction" mentioned in the embodiments of this application can be a direct instruction, an indirect instruction, or an indication of a relationship. For example, A instructing B can mean that A directly instructs B, such as B being able to obtain information through A; it can also mean that A indirectly instructs B, such as A instructing C, so B can obtain information through C; or it can mean that there is a relationship between A and B.

[0072] In the description of the embodiments of this application, the term "correspondence" may indicate that there is a direct or indirect correspondence between two things, or that there is an association between two things, or that there is a relationship of instruction and being instructed, configuration and being configured, etc.

[0073] In the embodiments of this application, "predefined" can be achieved by pre-storing corresponding codes, tables or other means that can be used to indicate relevant information in the device (e.g., including terminal devices and network devices). This application does not limit the specific implementation method.

[0074] Figure 1 This is a schematic diagram illustrating the structure of a multimodal image segmentation system according to an exemplary embodiment. The image segmentation system includes a server 110 and a terminal device 120. The terminal device 120 may include a data processing device and a data storage module.

[0075] Optionally, the terminal device 120 can communicate with the server 110 through a transmission network (such as a wireless communication network). The terminal device 120 can upload various data (such as image data) stored in the data storage module to the server 110 through the wireless communication network so that the server 110 can process the acquired image data, for example, by training a convolutional neural network model for image segmentation and other applications using the uploaded image data.

[0076] Optionally, the terminal device 120 may also include command input components, such as a mouse, keyboard, touch screen, etc. Figure 1 (Not shown) After receiving a specified command from the user, the command input component can input corresponding data on the terminal device. For example, when image segmentation software is installed on the terminal device, the user can input corresponding commands to the terminal device through the command input component to control the image segmentation software to output the corresponding image segmentation results.

[0077] Optionally, the terminal device can upload the image file to the server 110 so that the server 110 can train convolutional neural network models for applications such as image segmentation.

[0078] Optionally, the terminal device also includes a data processing device, which can segment the image file using a convolutional neural network model sent by the server when the terminal device 120 opens the image file.

[0079] Optionally, the server 110 can obtain image files uploaded by various terminal devices and annotate the image files manually or otherwise, so as to train the convolutional neural network model using the uploaded image files and annotation information. After training is completed, the trained convolutional neural network model can be transmitted to the terminal device so that the terminal device can perform image segmentation on the image files.

[0080] Optionally, after the terminal device receives and opens the image file, it can upload the image file to the server 110 so that the pre-trained convolutional neural network model in the server 110 can segment the image file, obtain the segmentation result, and return it to the terminal device, thereby realizing online segmentation of the image file.

[0081] Optionally, the aforementioned server can be a server cluster or a distributed system consisting of multiple physical servers, or it can be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms and other technology computing services.

[0082] Optionally, the system may also include a management device for managing the system (such as managing the connection status between each module and the server), and the management device is connected to the server via a communication network. Optionally, the communication network may be a wired network or a wireless network.

[0083] Optionally, the aforementioned wireless or wired networks use standard communication technologies and / or protocols. The network is typically the Internet, but can also be any other network, including but not limited to any combination of local area networks (LANs), metropolitan area networks (MANs), wide area networks (WANs), mobile, wired or wireless networks, private networks, or virtual private networks (VPNs). In some embodiments, technologies and / or formats including Hypertext Markup Language (HTML), Extensible Markup Language (XML), etc., are used to represent data exchanged over the network. Furthermore, conventional encryption technologies such as Secure Sockets Layer (SSL), Transport Layer Security (TLS), VPNs, and Internet Protocol (IP) security can be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can be used to replace or supplement the aforementioned data communication technologies.

[0084] Figure 2 This is a flowchart illustrating a multimodal image segmentation method according to an exemplary embodiment. The method is performed by a computer device, which may be, for example... Figure 1 One of the terminal devices and the server shown. Figure 2 As shown, the image segmentation method may include the following steps:

[0085] Step 201: Obtain target image data in at least two modalities.

[0086] Magnetic resonance imaging (MRI) is an imaging technique that reconstructs images by utilizing signals generated by the resonance of atomic nuclei within a magnetic field. Deep learning is a method of machine learning that learns feature representations of input data through multiple layers of nonlinear transformations, thereby automating various tasks.

[0087] In this embodiment of the application, since magnetic resonance images have multimodal characteristics, the data distribution differences of magnetic resonance images of different modalities will affect the accuracy of segmentation. In this embodiment of the application, magnetic resonance images of different modalities (that is, target image data of at least two modalities) can be obtained first, and then a suitable method can be selected to perform feature fusion on magnetic resonance images of different modalities to improve the accuracy of segmentation.

[0088] Step 202: For each modality of target image data, the data is processed sequentially through the downsampling modules of each layer in the target detection model to obtain the image features of each layer.

[0089] The target detection model also includes upsampling modules corresponding to the downsampling modules at each layer.

[0090] Downsampling, or image downsampling, aims to generate thumbnails of images that meet the required size. It involves downsampling the target image data for each modality individually at the target layer level, thereby obtaining the image features for the corresponding modality and layer.

[0091] Step 203: The image features of each layer are fused according to the layer to obtain the fused features of each layer.

[0092] Feature fusion refers to combining features from different layers or branches. It involves fusing image features from various modalities at each layer to obtain fused features for each layer.

[0093] Step 204: Based on the fusion features of the corresponding layers of each upsampling module, the bottom-level fusion features of each layer are processed sequentially through the upsampling modules to obtain the target segmentation result.

[0094] Upsampling is the process of enlarging an image. The main purpose of upsampling is to generate an enlarged version of the image that meets the required size.

[0095] Optionally, only the underlying fusion features can be processed sequentially through the upsampling modules of each layer.

[0096] Optionally, the bottom-level fusion features are upsampled by the upsampling module of the layer above the bottom (the penultimate layer), and then fused with the corresponding fusion features of the penultimate layer to obtain the penultimate intermediate fusion features. The penultimate intermediate fusion features are then processed sequentially by the upsampling modules of each subsequent layer (each layer above the penultimate layer).

[0097] Optionally, a target layer can be set. In each target layer, the intermediate fusion features of the layer preceding the target layer are fused with the fusion features corresponding to the target layer before upsampling. The target layer can be a continuous layer, a layer spaced a certain number of times apart, or a random layer.

[0098] In summary, this application first acquires target image data in at least two modalities; then, for each modal of target image data, it processes the image data sequentially through the downsampling modules of each layer in the target detection model to obtain image features at each layer; the target detection model also includes upsampling modules corresponding to each downsampling module; then, the image features at each layer are fused hierarchically to obtain fused features at each layer; finally, based on the fused features corresponding to the layers of each upsampling module, the lowest-level fused features in each layer are processed sequentially through the upsampling modules to obtain the target segmentation result. This scheme, by fusing image features at each layer hierarchically to obtain fused features at each layer, and then obtaining the target segmentation result based on the fused features and the features obtained by upsampling through the upsampling modules, compensates for the information lost during downsampling to a certain extent, thereby improving the image segmentation accuracy of the target detection model.

[0099] Figure 3 This is a flowchart illustrating a multimodal image segmentation method according to an exemplary embodiment. The method is performed by a computer device, which may be, for example... Figure 1 One of the terminal devices and the server shown. Figure 3 As shown, the image segmentation method may include the following steps:

[0100] Step 301: Obtain sample image data for at least two modalities.

[0101] The sample image data includes data annotations.

[0102] Magnetic resonance imaging (MRI) is an imaging technique that reconstructs images by utilizing signals generated by the resonance of atomic nuclei within a magnetic field. Deep learning is a method of machine learning that learns feature representations of input data through multiple layers of nonlinear transformations, thereby automating various tasks.

[0103] In this embodiment of the application, since magnetic resonance images have multimodal characteristics, the data distribution differences of magnetic resonance images of different modalities will affect the accuracy of segmentation. In this embodiment of the application, magnetic resonance images of different modalities (that is, target image data of at least two modalities) can be obtained first, and then a suitable method can be selected to perform feature fusion on magnetic resonance images of different modalities to improve the accuracy of segmentation.

[0104] Optionally, the data annotation can be done manually by experts or by experts using annotation tools. This data annotation can mark the target objects in the sample image and compare them with the recognition results of the target detection model, so that the recognition results of the target detection model are closer to the data annotation.

[0105] Optionally, a binary detection label or a detection map with the highest confidence level within the lesion can be generated in the post-processing step.

[0106] Optionally, the Getboundingbox algorithm (aimed at obtaining the bounding rectangle of the ROI in the medical image) in the simpleitk package (a medical image processing tool) can be used to extract the bounding box of the data annotation (the bounding rectangle of the ROI in the medical image), and the ROI region can be determined accordingly to crop the sequence and lesion labels. Here, ROI is the region of interest.

[0107] Optionally, the sample image data can be divided into training, validation, and test sets, for example, in a 6:2:2 ratio. A minibatch contains two negative sample image data and two positive sample image data.

[0108] Step 302: For the sample image data of each modality, the sample image data is processed sequentially through the downsampling modules of each layer in the target detection model to obtain the sample image features of each layer.

[0109] The target detection model also includes upsampling modules corresponding to the downsampling modules at each layer.

[0110] Downsampling, or image downsampling, aims to generate thumbnails of images that meet the required size. It involves downsampling the target image data for each modality individually at the target layer level, thereby obtaining the image features for the corresponding modality and layer.

[0111] Step 303: The features of the sample images of each layer are fused according to the layer to obtain the fused features of each layer.

[0112] Feature fusion refers to combining features from different layers or branches. It involves fusing image features from various modalities at each layer to obtain fused features for each layer.

[0113] Step 304: Based on the sample fusion features of the corresponding layer of each upsampling module, the bottom sample fusion features in each layer are processed sequentially through the upsampling module to obtain the sample output results.

[0114] Upsampling is the process of enlarging an image. The main purpose of upsampling is to generate an enlarged version of the image that meets the required size.

[0115] Optionally, only the underlying fusion features can be processed sequentially through the upsampling modules of each layer.

[0116] Optionally, the bottom-level fusion features are upsampled by the upsampling module of the layer above the bottom (the penultimate layer), and then fused with the corresponding fusion features of the penultimate layer to obtain the penultimate intermediate fusion features. The penultimate intermediate fusion features are then processed sequentially by the upsampling modules of each subsequent layer (each layer above the penultimate layer).

[0117] Optionally, a target layer can be set. In each target layer, the intermediate fusion features of the layer preceding the target layer are fused with the fusion features corresponding to the target layer before upsampling. The target layer can be a continuous layer, a layer spaced a certain number of times apart, or a random layer.

[0118] Optionally, deep supervision can be set in the target layer during the upsampling process, such as setting three layers of deep supervision. Deep supervision is a technique that adds an auxiliary classifier as a branch to some intermediate hidden layers of a deep neural network to supervise the backbone network, and is used to solve problems such as vanishing gradients and slow convergence speed during deep neural network training.

[0119] Step 305: Based on the sample output and data annotation, train the target detection model to obtain the trained target detection model.

[0120] The trained object detection model is used to process object image data to obtain object segmentation results.

[0121] Because the dataset contains some negative cases (negative sample image data), the lesion labels (data annotations) for these negative cases are all 0. In the later stages of network training, the predicted labels for these images are all 0, resulting in a loss of 0, and they no longer contribute to the overall network learning. Optionally, to fully utilize this data, a memory-based learning strategy is designed: first, the network input is adjusted to all negative cases, and the network is pre-trained using focaloss (γ=2) until convergence; then, the network input is adjusted to include both positive and negative cases, and two dataloaders (the data loading mechanism of the PyTorch neural network framework) are constructed to load positive and negative cases respectively, ensuring that each minibatch input to the network contains positive samples. This prevents gradient disorder caused by random minibatches containing all negative cases in the later stages of training, where the loss is 0. The model trained in the previous stage is then loaded. At this point, the model has higher sensitivity to positive samples, which helps improve the lesion detection rate. The model is trained using focaloss until convergence, and the optimal model is saved. Here, focaloss is a loss function, and γ is a parameter of focaloss.

[0122] In summary, this application first acquires target image data in at least two modalities; then, for each modal of target image data, it processes the image data sequentially through the downsampling modules of each layer in the target detection model to obtain image features at each layer; the target detection model also includes upsampling modules corresponding to each downsampling module; then, the image features at each layer are fused hierarchically to obtain fused features at each layer; finally, based on the fused features corresponding to the layers of each upsampling module, the lowest-level fused features in each layer are processed sequentially through the upsampling modules to obtain the target segmentation result. This scheme, by fusing image features at each layer hierarchically to obtain fused features at each layer, and then obtaining the target segmentation result based on the fused features and the features obtained by upsampling through the upsampling modules, compensates for the information lost during downsampling to a certain extent, thereby improving the image segmentation accuracy of the target detection model.

[0123] Figure 4 This is a flowchart illustrating a multimodal image segmentation method according to an exemplary embodiment. The method is performed by a computer device, which may be, for example... Figure 1 One of the terminal devices and the server shown. Figure 4 As shown, the image segmentation method may include the following steps:

[0124] Step 401: Obtain target image data in at least two modalities.

[0125] Magnetic resonance imaging (MRI) is an imaging technique that reconstructs images from signals generated by the resonance of atomic nuclei within a magnetic field. Deep learning is a method of machine learning that learns feature representations of input data through multi-layer nonlinear transformations, thereby automating various tasks. By segmenting MRI images using deep learning, lesions in target organs can be identified from the images, such as clinically significant prostate cancer lesions.

[0126] In this embodiment of the application, since magnetic resonance images have multimodal characteristics, the data distribution differences of magnetic resonance images of different modalities will affect the accuracy of segmentation. In this embodiment of the application, magnetic resonance images of different modalities (that is, target image data of at least two modalities) can be obtained first, and then a suitable method can be selected to perform feature fusion on magnetic resonance images of different modalities to improve the accuracy of segmentation.

[0127] To segment different modalities of magnetic resonance images using deep learning, it is first necessary to acquire target image data in at least two modalities. For example, target image data in the T2W (T2-weighted image, where T2 is a physical quantity used to measure electromagnetic waves) mode, ADC (Apparent Diffusion Coefficient) mode, and DWI (Diffusion Weighted Imaging) mode of cross-sections of the target organ (e.g., the prostate) can be acquired separately.

[0128] Optionally, the target image data can be preprocessed. For example, the target image data can first be rigidly registered using the registration tool Elastix, with the T2W mode target image data as the fixed image and the ADC mode and DWI mode target image data as floating images, ensuring spatial alignment of the target image data for each mode, and then resampling can be performed. Resampling refers to resampling the digital image composed of discrete data after sampling according to the required pixel positions or pixel spacing to construct a new image after geometric transformation.

[0129] Furthermore, data augmentation is performed on the rigidly registered target image data. The purpose of data augmentation is to increase the sample size. The target detection model trained on the target image data augmented through various random transformations has stronger robustness and can achieve better segmentation results. Optionally, data augmentation is performed through rotation, normalization, random scaling, and random affine transformations, and the augmented target image data is read in through the dataloader (PyTorch's data loading mechanism) in PyTorch (a neural network framework).

[0130] Step 402: For each modality of target image data, the downsampling modules of each layer in the target detection model are processed sequentially to obtain the image features of each layer.

[0131] The target detection model also includes upsampling modules corresponding to the downsampling modules at each layer. Figure 5 A schematic diagram of the target detection model involved in an embodiment of this application is shown. Here, fusion represents the process of fusing.

[0132] Optionally, each downsampling module has at least two downsampling branches.

[0133] For example, using dynUnet as the baseline network for the object detection model, in order to handle multimodal image sequences with different data distributions (target image data of at least two modalities) and improve the performance ceiling, it is extended to three encoders (downsampling branches) to extract modality-independent features respectively.

[0134] Furthermore, for each modality of target image data, the data is processed sequentially through the downsampling branch of the corresponding modality in each downsampling module of the target detection model to obtain the image features of each layer of the corresponding modality.

[0135] Optionally, the downsampling module includes a first convolution module, a second convolution module, a third convolution module, a first normalization module, a second normalization module, a third normalization module, and an activation module;

[0136] The first convolution module, the first normalization module, the third convolution module, and the second normalization module are connected in sequence to process the input image data to obtain the first result;

[0137] The second convolution module and the third normalization module are connected in sequence to process the input image data to obtain the second result;

[0138] The activation module is used to process the fusion result of the first result and the second result to obtain the image features of the corresponding level of the downsampling module.

[0139] Optionally, in each downsampling module, the first convolutional module and the second convolutional module are identical in the downsampling branches of different modalities. Since the target image data of each modality has different representations, but the relative position of the lesion at the target organ is fixed, and in some cases the lesion region in the target image data of some modalities is not obvious, hard parameter sharing of downsampling convolution can be implemented for each downsampling branch, setting the parameters of the corresponding target modules in each downsampling branch to the same parameters. That is, there are corresponding modules with the same parameters in each downsampling branch, while the parameters of other modules may not be the same. During the downsampling of target image data of each modality, the same rules are followed to preserve the relative position of the lesion in the target image data, including implicit modal mutual supervision.

[0140] Figure 6 A schematic diagram of the structure of the downsampling module involved in the embodiments of this application is shown (corresponding to...). Figure 5 Part ① of the code. Here, encoder1, encoder2, and encoder3 are each a downsampling branch (encoder), F i The image features of the target layer are downsampled and then transformed into image features F of the next layer below the target layer. i+1 The size is reduced to half of its original size. Three downsampling branches (encoder1, encoder2, and encoder3) extract modality-independent features. Although the features of different modalities differ, after registration, the image has a consistent spatial structure. Hard parameter sharing between downsampling convolutions with a stride of 2 in the same layer allows for mutual supervision between different modalities, forcing different downsampling branches to learn the same spatial structure information. Modules sharing parameters learn spatial structure information, while other modules are modally independent, learning different modal feature representations. It should be noted that in the figure, `norm` represents normalization, `leakyReLU` is an activation function, the first convolutional module is a 3×3×3 stride2 module, and the second convolutional module is a 1×1×1 stride2 module.

[0141] Step 403: The image features of each layer are fused according to the layer to obtain the fused features of each layer.

[0142] To address the information sharing problem caused by differences in data distribution among target image data of different modalities, attention maps can be used to enhance the correlation of multimodal target image data. This can be achieved at the jump connections in the target detection model (corresponding to...). Figure 5 (Part ②) fuses image features of the same layer.

[0143] Figure 7 An example diagram of feature fusion involved in an embodiment of this application is shown. For example... Figure 7As shown, 3D spatial attention maps are calculated for image features from three modalities. The three spatial attention maps are then element-wise mixed to obtain a hybrid activation response map. This forces the network to increase attention to the same spatial location across different modalities. While focusing on common high-response regions across modalities, the attention to high-response regions of modal-independent features is appropriately adjusted, further enabling the network to learn the same spatial information and accurately locate lesion voxels. Wherein, F... t2w F adc and F dwi These represent image features from three different modalities, with Spital attention representing spatial attention, and A... s(t2w) A s(adc) And A s(dwi) These represent attention maps for three modalities, respectively. Fuse represents the mixture, and A... s(fuse) This represents a fused spatial attention map, where pre-act means pre-activation, element-wise add means element-wise fusion, matrix multiplication is the Hadamard product, concatenate means feature union, pixel normalization is the instance norm, 3D spatial attention map is the 3D spatial attention map, and element-wise max means taking the maximum value of each element.

[0144] Spatial attention map is a component of CBAM (Convolutional Block Attention Module, Hybrid Attention Mechanism), Hybrid Activation Response Map A s(m) The value is obtained according to the following formula:

[0145] A s(m) = sigmoid(conv(AvgPool(F m MaxPool(F) m ))), m∈ [t2w,adc,dwi]

[0146] Among them, F mLet m be the modality, sigmoid be the activation function, conv be the convolution, AvgPool be the average pooling, and MaxPool be the max pooling. The encoder obtains image features for a specific layer. For a single modality, the spatial attention map is obtained through global average pooling and global max pooling, concatenated, and then calculated using a 1x1 convolution to obtain the probability map. Finally, a sigmoid function is used to map it to the range [0,1]. The spatial attention map focuses on the spatial structure information of the image. After obtaining the spatial attention maps for the three modalities, the hybrid activation map A is calculated according to the following formula. s(fuse) :

[0147] A s(fuse) = ∏(1+A s(m) ), m∈[t2w,adc,dwi]

[0148] Step 404: The bottom-level upsampling module performs upsampling processing on the bottom-level fusion feature to obtain the upsampling result output by the bottom-level upsampling module.

[0149] Optionally, the upsampling modules at each layer include a bottom layer upsampling module, a top layer upsampling module, and an intermediate layer upsampling module.

[0150] After obtaining the fusion features of each layer, the upsampling process of this application is to first perform upsampling, then fuse the upsampling result with the fusion features of the corresponding layer to obtain the intermediate fusion features of the corresponding layer, and then perform upsampling on the intermediate fusion features of the corresponding layer. Since the bottom upsampling module directly upsamples the bottom fusion features, while the top upsampling module needs to obtain the target segmentation result, the upsampling modules of each layer are divided into bottom upsampling module, top upsampling module and intermediate layer upsampling module.

[0151] Step 405: For each intermediate layer upsampling module, the upsampling result output by the lower layer upsampling module of the intermediate layer upsampling module is fused with the fusion feature of the corresponding layer of the intermediate layer upsampling module to obtain an intermediate fusion feature. The intermediate fusion feature is then upsampled through the intermediate layer upsampling module to obtain the upsampling result output by the intermediate layer upsampling module.

[0152] The steps performed by each intermediate layer upsampling module are similar: the upsampling result of the next layer of the intermediate layer upsampling module is fused with the fusion feature of the corresponding layer of the intermediate layer upsampling module before upsampling.

[0153] Step 406: The upsampling result output by the lower-level upsampling module of the top-level upsampling module is fused with the fusion feature of the corresponding level of the top-level upsampling module, and the fused feature is processed by the top-level upsampling module to obtain the target segmentation result.

[0154] The top-level upsampling module not only upsamples the upsampling results of its next layer and the fusion results of the corresponding layer's fusion features (fused features), but also performs further processing to obtain the target segmentation results. Therefore, the steps performed by the top-level upsampling module are different from those of the intermediate layer upsampling module.

[0155] It should be noted that, Figure 5 The deep supervision in the context refers to Figure 3 The deep supervision component added during the training of the object detection model in the embodiment... Figure 3 The other structures of the target detection model in the embodiment are all the same as those in the example. Figure 4 The target detection model in the embodiment has the same corresponding structure.

[0156] In summary, this application first acquires target image data in at least two modalities; then, for each modal of target image data, it processes the image data sequentially through the downsampling modules of each layer in the target detection model to obtain image features at each layer; the target detection model also includes upsampling modules corresponding to each downsampling module; then, the image features at each layer are fused hierarchically to obtain fused features at each layer; finally, based on the fused features corresponding to the layers of each upsampling module, the lowest-level fused features in each layer are processed sequentially through the upsampling modules to obtain the target segmentation result. This scheme, by fusing image features at each layer hierarchically to obtain fused features at each layer, and then obtaining the target segmentation result based on the fused features and the features obtained by upsampling through the upsampling modules, compensates for the information lost during downsampling to a certain extent, thereby improving the image segmentation accuracy of the target detection model.

[0157] Figure 8 This is a structural block diagram illustrating a multimodal image segmentation apparatus according to an exemplary embodiment. The image segmentation apparatus includes:

[0158] The first data acquisition module 801 is used to acquire target image data in at least two modalities;

[0159] The first downsampling module 802 is used to process the target image data of each modality sequentially through the downsampling modules of each layer in the target detection model to obtain the image features of each layer; the target detection model also includes upsampling modules corresponding to each downsampling module.

[0160] The first feature fusion module 803 is used to fuse the image features of each layer according to the layer to obtain the fused features of each layer;

[0161] The first upsampling module 804 is used to process the bottom-level fusion features of each layer's fusion features sequentially through the upsampling modules based on the fusion features of the corresponding layers of each upsampling module, so as to obtain the target segmentation result.

[0162] In one possible implementation, the upsampling modules at each layer include a bottom layer upsampling module, a top layer upsampling module, and an intermediate layer upsampling module;

[0163] Based on the fusion features of the corresponding layers of the upsampling modules, the lowest-level fusion features in each layer are processed sequentially through the upsampling modules to obtain the target segmentation result, including:

[0164] The underlying upsampling module performs upsampling processing on the underlying fused features to obtain the upsampling result output by the underlying upsampling module;

[0165] For each intermediate layer upsampling module, the upsampling result output by the lower layer upsampling module of the intermediate layer upsampling module is fused with the fusion feature of the corresponding layer of the intermediate layer upsampling module to obtain intermediate fusion feature. Then, the intermediate fusion feature is upsampled through the intermediate layer upsampling module to obtain the upsampling result output by the intermediate layer upsampling module.

[0166] The upsampling result output by the lower-level upsampling module of the top-level upsampling module is fused with the fusion feature of the corresponding level of the top-level upsampling module, and the fused feature is processed by the top-level upsampling module to obtain the target segmentation result.

[0167] In one possible implementation, each downsampling module has at least two downsampling branches;

[0168] For each modality of target image data, the data is processed sequentially through the downsampling modules of each layer in the target detection model to obtain the image features of each layer, including:

[0169] For each modality of target image data, the data is processed sequentially through the downsampling branch of the downsampling module in each layer of the target detection model to obtain the image features of each layer of the corresponding modality.

[0170] In one possible implementation, the downsampling module includes a first convolution module, a second convolution module, a third convolution module, a first normalization module, a second normalization module, a third normalization module, and an activation module;

[0171] The first convolution module, the first normalization module, the third convolution module, and the second normalization module are connected in sequence to process the input image data to obtain the first result;

[0172] The second convolution module and the third normalization module are connected in sequence to process the input image data to obtain the second result;

[0173] The activation module is used to process the fusion result of the first result and the second result to obtain the image features of the corresponding level of the downsampling module.

[0174] In one possible implementation, in each downsampling module, the first convolutional module and the second convolutional module in the downsampling branches of different modalities are the same.

[0175] In summary, this application first acquires target image data in at least two modalities; then, for each modal of target image data, it processes the image data sequentially through the downsampling modules of each layer in the target detection model to obtain image features at each layer; the target detection model also includes upsampling modules corresponding to each downsampling module; then, the image features at each layer are fused hierarchically to obtain fused features at each layer; finally, based on the fused features corresponding to the layers of each upsampling module, the lowest-level fused features in each layer are processed sequentially through the upsampling modules to obtain the target segmentation result. This scheme, by fusing image features at each layer hierarchically to obtain fused features at each layer, and then obtaining the target segmentation result based on the fused features and the features obtained by upsampling through the upsampling modules, compensates for the information lost during downsampling to a certain extent, thereby improving the image segmentation accuracy of the target detection model.

[0176] Figure 9 This is a structural block diagram illustrating a multimodal image segmentation apparatus according to an exemplary embodiment. The image segmentation apparatus includes:

[0177] The second data acquisition module 901 is used to acquire sample image data of at least two modalities; the sample image data contains data annotations.

[0178] The second downsampling module 902 is used to process the sample image data of each modality sequentially through the downsampling modules of each layer in the target detection model to obtain the sample image features of each layer; the target detection model also includes upsampling modules corresponding to each downsampling module.

[0179] The second feature fusion module 903 is used to fuse the features of each layer of sample images according to the layer to obtain the fused features of each layer of samples.

[0180] The second upsampling module 904 is used to process the bottom sample fusion features in each layer of the sample fusion features based on the sample fusion features of the corresponding layer of each upsampling module, and obtain the sample output results by passing them through the upsampling modules of each layer in sequence.

[0181] The training module 905 is used to train the target detection model based on the sample output and data annotation to obtain the trained target detection model; the trained target detection model is used to process the target image data to obtain the target segmentation result.

[0182] In one possible implementation, the upsampling modules at each layer include a bottom layer upsampling module, a top layer upsampling module, and an intermediate layer upsampling module;

[0183] Based on the fusion features of the corresponding layers of the upsampling modules, the lowest-level fusion features in each layer are processed sequentially through the upsampling modules to obtain the target segmentation result, including:

[0184] The underlying upsampling module performs upsampling processing on the underlying fused features to obtain the upsampling result output by the underlying upsampling module;

[0185] For each intermediate layer upsampling module, the upsampling result output by the lower layer upsampling module of the intermediate layer upsampling module is fused with the fusion feature of the corresponding layer of the intermediate layer upsampling module to obtain intermediate fusion feature. Then, the intermediate fusion feature is upsampled through the intermediate layer upsampling module to obtain the upsampling result output by the intermediate layer upsampling module.

[0186] The upsampling result output by the lower-level upsampling module of the top-level upsampling module is fused with the fusion feature of the corresponding level of the top-level upsampling module, and the fused feature is processed by the top-level upsampling module to obtain the target segmentation result.

[0187] In one possible implementation, each downsampling module has at least two downsampling branches;

[0188] For each modality of target image data, the data is processed sequentially through the downsampling modules of each layer in the target detection model to obtain the image features of each layer, including:

[0189] For each modality of target image data, the data is processed sequentially through the downsampling branch of the downsampling module in each layer of the target detection model to obtain the image features of each layer of the corresponding modality.

[0190] In one possible implementation, the downsampling module includes a first convolution module, a second convolution module, a third convolution module, a first normalization module, a second normalization module, a third normalization module, and an activation module;

[0191] The first convolution module, the first normalization module, the third convolution module, and the second normalization module are connected in sequence to process the input image data to obtain the first result;

[0192] The second convolution module and the third normalization module are connected in sequence to process the input image data to obtain the second result;

[0193] The activation module is used to process the fusion result of the first result and the second result to obtain the image features of the corresponding level of the downsampling module.

[0194] In one possible implementation, in each downsampling module, the first convolutional module and the second convolutional module in the downsampling branches of different modalities are the same.

[0195] In summary, this application first acquires target image data in at least two modalities; then, for each modal of target image data, it processes the image data sequentially through the downsampling modules of each layer in the target detection model to obtain image features at each layer; the target detection model also includes upsampling modules corresponding to each downsampling module; then, the image features at each layer are fused hierarchically to obtain fused features at each layer; finally, based on the fused features corresponding to the layers of each upsampling module, the lowest-level fused features in each layer are processed sequentially through the upsampling modules to obtain the target segmentation result. This scheme, by fusing image features at each layer hierarchically to obtain fused features at each layer, and then obtaining the target segmentation result based on the fused features and the features obtained by upsampling through the upsampling modules, compensates for the information lost during downsampling to a certain extent, thereby improving the image segmentation accuracy of the target detection model.

[0196] Figure 10A structural block diagram of a computer device 1000 illustrated in an exemplary embodiment of this application is shown. This computer device can be implemented as a server as described above in this application. The computer device 1000 includes a Central Processing Unit (CPU) 1001, a system memory 1004 including Random Access Memory (RAM) 1002 and Read-Only Memory (ROM) 1003, and a system bus 1005 connecting the system memory 1004 and the CPU 1001. The computer device 1000 also includes a mass storage device 1006 for storing an operating system 1009, application programs 1010, and other program modules 1011.

[0197] The mass storage device 1006 is connected to the central processing unit 1001 via a mass storage controller (not shown) connected to the system bus 1005. The mass storage device 1006 and its associated computer-readable media provide non-volatile storage for the computer device 1000. That is, the mass storage device 1006 may include computer-readable media (not shown) such as a hard disk or a compact disc read-only memory (CD-ROM) drive.

[0198] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state storage technologies, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage media are not limited to the above-mentioned types. The system memory 1004 and mass storage device 1006 described above can be collectively referred to as memory.

[0199] According to various embodiments of this disclosure, the computer device 1000 can also be connected to a remote computer on a network, such as the Internet. That is, the computer device 1000 can be connected to the network 1008 via a network interface unit 1007 connected to the system bus 1005, or the network interface unit 1007 can be used to connect to other types of networks or remote computer systems (not shown).

[0200] The memory also includes at least one computer program stored in the memory, and the central processing unit 1001 executes the at least one computer program to implement all or part of the steps in the methods shown in the above embodiments.

[0201] In one exemplary embodiment, a computer-readable storage medium is also provided for storing at least one computer program, which is loaded and executed by a processor to implement all or part of the steps in the above-described method. For example, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, or optical data storage device, etc.

[0202] In one exemplary embodiment, a computer program product or computer program is also provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned actions. Figure 2 , Figure 3 or Figure 4 All or part of the steps of the method shown in any embodiment.

[0203] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0204] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A multi-modal image segmentation method, characterized in that, The method comprises: acquiring target image data of at least two modalities; for the target image data of each modality, sequentially processing through each layer down-sampling module in a target detection model to obtain image features of each layer; the target detection model further comprises each layer up-sampling module corresponding to each layer down-sampling module; wherein each layer down-sampling module comprises down-sampling branches of different modalities; each down-sampling branch is subjected to down-sampling convolution hard parameter sharing, so that corresponding modules in each down-sampling branch have the same parameters; fusing image features of each layer according to the layer level to obtain fused features of each layer; based on the fused features of the corresponding layer level of each layer up-sampling module, sequentially processing bottom fused features in the fused features of each layer through the each layer up-sampling module to obtain a target segmentation result; the step of fusing image features of each layer according to the layer level to obtain fused features of each layer comprises: respectively calculating spatial attention maps for each image feature of the same layer, mixing the spatial attention maps corresponding to each image feature of the same layer to obtain a mixed activation map; performing matrix multiplication on each image feature of the same layer and the mixed activation map, then sequentially performing element-by-element mixing and pre-activation to obtain a pre-activation result, and performing feature combination on the pre-activation results corresponding to each image feature of the same layer to obtain fused features.

2. The method of claim 1, wherein, each layer up-sampling module comprises a bottom layer up-sampling module, a top layer up-sampling module and an intermediate layer up-sampling module; the step of sequentially processing bottom fused features in the fused features of each layer through the each layer up-sampling module to obtain a target segmentation result based on the fused features of the corresponding layer level of each layer up-sampling module comprises: the bottom layer up-sampling module performs up-sampling processing on the bottom fused features to obtain an up-sampling result output by the bottom layer up-sampling module; for each intermediate layer up-sampling module, fusing the up-sampling result output by the lower layer up-sampling module of the intermediate layer up-sampling module and the fused features of the corresponding layer level of the intermediate layer up-sampling module to obtain intermediate fused features, and performing up-sampling processing on the intermediate fused features by the intermediate layer up-sampling module to obtain an up-sampling result output by the intermediate layer up-sampling module; fusing the up-sampling result output by the lower layer up-sampling module of the top layer up-sampling module and the fused features of the corresponding layer level of the top layer up-sampling module, and processing the fused features by the top layer up-sampling module to obtain the target segmentation result.

3. The method of claim 2, wherein, each layer down-sampling module has at least two down-sampling branches; the step of sequentially processing each layer down-sampling module in a target detection model to obtain image features of each layer for target image data of each modality comprises: sequentially processing the down-sampling branch corresponding to each modality in each layer down-sampling module in a target detection model to obtain image features of each layer for target image data of each modality.

4. The method of claim 3, wherein, the down-sampling module comprises a first convolution module, a second convolution module, a third convolution module, a first normalization module, a second normalization module, a third normalization module and an activation module; The first convolution module, the first normalization module, the third convolution module and the second normalization module are sequentially connected to process the input image data to obtain a first result. The second convolution module and the third normalization module are sequentially connected to process the input image data to obtain a second result. The activation module is configured to process a fusion result of the first result and the second result to obtain an image feature of a corresponding level of the down-sampling module.

5. The method of claim 4, wherein, In each layer of the down-sampling module, the first convolution module in the down-sampling branch of different modalities is the same, and the second convolution module is the same.

6. A multi-modal image segmentation method, characterized in that, The method comprises: obtaining sample image data of at least two modalities; the sample image data contains data annotations; for each type of sample image data, sequentially processing through each layer of the down-sampling module in the target detection model to obtain sample image features of each layer; the target detection model further comprises each layer of the up-sampling module corresponding to each layer of the down-sampling module; each layer of the down-sampling module comprises down-sampling branches of different modalities; each down-sampling branch is subjected to down-sampling convolution hard parameter sharing, so that there are corresponding modules with the same parameters in each down-sampling branch; fusing sample image features of each layer according to the level to obtain sample fusion features of each layer; wherein, spatial attention maps are respectively calculated for each sample image feature of the same layer, and the spatial attention maps corresponding to each sample image feature of the same layer are mixed to obtain a mixed activation map; each sample image feature of the same layer is multiplied by the mixed activation map, and then sequentially subjected to element-by-element mixing and pre-activation to obtain a pre-activation result, and the pre-activation results corresponding to each sample image feature of the same layer are combined to obtain a sample fusion feature; based on the sample fusion features of the corresponding levels of each layer of the up-sampling module, processing the bottom sample fusion features in the sample fusion features of each layer through the up-sampling module to obtain a sample output result; training the target detection model according to the sample output result and the data annotations to obtain a trained target detection model; the trained target detection model is used to process target image data to obtain a target segmentation result.

7. A multi-modal image segmentation apparatus, characterized by, The device comprises: a first data acquisition module configured to obtain target image data of at least two modalities; a first down-sampling module configured to sequentially process each layer of the down-sampling module in the target detection model for each type of target image data to obtain image features of each layer; the target detection model further comprises each layer of the up-sampling module corresponding to each layer of the down-sampling module; each layer of the down-sampling module comprises down-sampling branches of different modalities; each down-sampling branch is subjected to down-sampling convolution hard parameter sharing, so that there are corresponding modules with the same parameters in each down-sampling branch; a first feature fusion module configured to fuse image features of each layer according to the level to obtain fusion features of each layer; The first up-sampling module is configured to sequentially process bottom-layer fusion features in the layer fusion features through the layer up-sampling modules based on sample fusion features at corresponding levels of the layer up-sampling modules, to obtain a target segmentation result. The first feature fusion module is further configured to: respectively calculate spatial attention maps for the image features at the same layer, mix the spatial attention maps corresponding to the image features at the same layer to obtain a mixed activation map, and perform matrix multiplication on the image features at the same layer and the mixed activation map, followed by element-by-element mixing and pre-activation to obtain a pre-activation result, and perform feature combination on the pre-activation results corresponding to the image features at the same layer to obtain the fusion features.

8. A multi-modal image segmentation apparatus, characterized by, The device comprises: The second data acquisition module is configured to acquire sample image data of at least two modalities, and the sample image data comprises data annotations; The second down-sampling module is configured to sequentially process sample image features of each modality through the layer down-sampling modules in the target detection model to obtain the layer sample image features; the target detection model further comprises layer up-sampling modules corresponding to the layer down-sampling modules; each layer down-sampling module comprises down-sampling branches of different modalities; the down-sampling convolution hard parameters of each down-sampling branch are shared, so that corresponding modules with the same parameters exist in each down-sampling branch; The second feature fusion module is configured to fuse the layer sample image features according to the levels to obtain layer sample fusion features; the spatial attention maps are respectively calculated for the sample image features at the same layer, the spatial attention maps corresponding to the sample image features at the same layer are mixed to obtain a mixed activation map, and the matrix multiplication is performed on the sample image features at the same layer and the mixed activation map, followed by element-by-element mixing and pre-activation to obtain a pre-activation result, and the feature combination is performed on the pre-activation results corresponding to the sample image features at the same layer to obtain the sample fusion features; The second up-sampling module is configured to sequentially process bottom-layer sample fusion features in the layer sample fusion features through the layer up-sampling modules based on sample fusion features at corresponding levels of the layer up-sampling modules, to obtain a sample output result; The training module is configured to train the target detection model according to the sample output result and the data annotations, to obtain a trained target detection model; the trained target detection model is configured to process target image data to obtain a target segmentation result.

9. A computer device, comprising: The computer device comprises a processor and a memory, and the memory stores at least one instruction, which is loaded and executed by the processor to implement the multi-modal image segmentation method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, which is loaded and executed by the processor to implement the multi-modal image segmentation method according to any one of claims 1 to 6.

Images