Focus segmentation method and device based on multi-modal feature fusion, equipment and medium

Abstract

The invention discloses a focus segmentation method and device based on multi-modal feature fusion, equipment and a medium. The method comprises the following steps: preprocessing to-be-processed images of a target part to which a focus belongs in at least two modes to obtain to-be-used images in each mode; inputting the to-be-used image in each mode into a target lesion segmentation model for lesion region segmentation processing to obtain a target segmented image marked with a target lesion; wherein the target focus segmentation model comprises an encoding part and a decoding part, the encoding part comprises an image segmentation module, a plurality of feature extraction modules and a downsampling module, and the decoding part comprises a feature fusion module and an upsampling module. According to the method, the focus segmentation precision and efficiency are improved, and the accuracy and reliability of the target segmentation image are ensured.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a multi-modal feature fusion lesion segmentation method, device, equipment and medium. Background Art

[0002] Magnetic resonance imaging (MRI) is a non-invasive imaging technique used for imaging of the brain, spine, joints, etc. Multimodal MRI techniques, such as T1-weighted imaging (T1WI), T2-weighted imaging (T2WI), or fluid attenuated inversion recovery (FLAIR) sequences, can provide rich tissue contrast information.

[0003] At present, the traditional lesion segmentation method mainly relies on manual operation, that is, the staff performs lesion segmentation on the MRI image. However, when using the above method to perform lesion segmentation, it will be affected by the accumulated experience of the staff, so the segmentation result has a certain subjectivity. In addition, when segmenting complex lesions or low-contrast MRI images, the segmentation accuracy of the above method is not high, and the problem of inaccurate segmentation results is prone to occur. The above method also cannot make full use of the tissue contrast information provided by the multimodal MRI image, resulting in unsatisfactory segmentation results. In addition, the segmentation process is performed manually, and its segmentation efficiency is also low. Summary of the invention

[0004] The present invention provides a multi-modal feature fusion lesion segmentation method, device, equipment and medium, which improves the precision and efficiency of lesion segmentation and ensures the accuracy and reliability of target segmentation images.

[0005] According to one aspect of the present invention, a method for lesion segmentation using multimodal feature fusion is provided, the device comprising:

[0006] Preprocessing the images to be processed of the target part to which the lesion belongs in at least two modalities to obtain the image to be used in each modality;

[0007] The image to be used in each modality is input into the target lesion segmentation model to perform lesion area segmentation processing, so as to obtain a target segmentation image with the target lesion marked;

[0008] Among them, the target lesion segmentation model includes an encoding part and a decoding part. The encoding part includes an image segmentation module, multiple feature extraction modules, and a downsampling module. The decoding part includes a feature fusion module and an upsampling module. The image segmentation module is used to divide the image to be used into multiple images to be fused. Multiple feature extraction modules are used to extract and fuse features of the images to be fused under each modality. The feature fusion module is used to fuse the features extracted by each feature extraction module. The downsampling module is used to downsample the image to be used or the image to be fused. The upsampling module is used to upsample the image after fusion.

[0009] According to another aspect of the present invention, a multimodal feature fusion lesion segmentation device is provided, the device comprising:

[0010] An image preprocessing module, used to obtain an image to be used in each modality by preprocessing the images to be processed of the target part to which the lesion belongs in at least two modalities;

[0011] An image segmentation module is used to input the image to be used in each modality into the target lesion segmentation model to perform lesion area segmentation processing, and obtain a target segmentation image with the target lesion marked;

[0012] Among them, the target lesion segmentation model includes an encoding part and a decoding part. The encoding part includes an image segmentation module, multiple feature extraction modules, and a downsampling module. The decoding part includes a feature fusion module and an upsampling module. The image segmentation module is used to divide the image to be used into multiple images to be fused. Multiple feature extraction modules are used to extract and fuse features of the images to be fused under each modality. The feature fusion module is used to fuse the features extracted by each feature extraction module. The downsampling module is used to downsample the image to be used or the image to be fused. The upsampling module is used to upsample the image after fusion.

[0013] According to another aspect of the present invention, there is provided an electronic device, the electronic device comprising:

[0014] at least one processor; and

[0015] a memory communicatively connected to at least one processor; wherein,

[0016] The memory stores a computer program that can be executed by at least one processor, and the computer program is executed by at least one processor so that the at least one processor can execute the lesion segmentation method with multimodal feature fusion according to any embodiment of the present invention.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, which stores computer instructions, and the computer instructions are used to enable a processor to implement the lesion segmentation method with multimodal feature fusion of any embodiment of the present invention when executed.

[0018] According to another aspect of the present invention, there is provided a computer program product, comprising a computer program, characterized in that when the computer program is executed by a processor, the computer program implements the lesion segmentation method using multimodal feature fusion as described in any embodiment of the present invention.

[0019] The technical solution of the embodiment of the present invention obtains the image to be used in each mode by preprocessing the image to be processed of the target part to which the lesion belongs in at least two modes. By obtaining the image to be used in at least two modes, the image information corresponding to different modes can be fully utilized in the subsequent lesion segmentation to improve the accuracy and reliability of the segmentation process. The image to be used in each mode is input into the target lesion segmentation model for lesion area segmentation processing to obtain a target segmented image with the target lesion marked. Among them, the target lesion segmentation model includes an encoding part and a decoding part, the encoding part includes an image segmentation module, multiple feature extraction modules, and a downsampling module, and the decoding part includes a feature fusion module and an upsampling module. Through each module in the target lesion segmentation model, accurate segmentation of the lesion in the image to be used can be achieved, and the accuracy of the segmentation is improved. The present invention solves the problems of subjectivity, low segmentation accuracy and low segmentation efficiency in the segmentation results in the prior art. The accuracy and efficiency of lesion segmentation are improved, and the accuracy and reliability of the target segmented image are guaranteed.

[0020] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 It is a flow chart of a lesion segmentation method of multimodal feature fusion provided by an embodiment of the present invention;

[0023] Figure 2 is an example diagram of a target lesion segmentation model provided by an embodiment of the present invention;

[0024] Figure 3is a flowchart of an image encoding process to be used provided by an embodiment of the present invention;

[0025] Figure 4 is a flowchart of a decoding process of an image to be used provided by an embodiment of the present invention;

[0026] Figure 5 It is a structural schematic diagram of a lesion segmentation device with multimodal feature fusion provided by an embodiment of the present invention;

[0027] Figure 6 It is a structural schematic diagram of an electronic device for implementing the multimodal feature fusion lesion segmentation method of an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] Embodiment 1

[0031] Figure 1 This is a flowchart of a multimodal feature fusion lesion segmentation method provided in Example 1 of the present invention. This embodiment can be applied to image segmentation of multimodal nuclear magnetic resonance images to obtain a target segmentation image that accurately marks the target lesion. The method can be executed by a multimodal feature fusion lesion segmentation device, which can be implemented in the form of hardware and / or software. The multimodal feature fusion lesion segmentation device can be configured in electronic devices such as mobile phones, computers or servers. Figure 1 As shown, the method includes:

[0032] S110 , obtaining an image to be used in each modality by preprocessing the images to be processed of the target part to which the lesion belongs in at least two modalities.

[0033] Among them, the target part to which the lesion belongs can be the part where the lesion is located. For example, the target part to which the lesion belongs can be the prostate part. Before performing lesion segmentation, different imaging techniques in magnetic resonance imaging technology can be used to collect images of the target part to which the lesion belongs at different angles or shapes, and obtain images to be processed under at least two modes. The image to be processed is the original magnetic resonance image of the target part to which the lesion belongs under the current mode. For example, different imaging techniques can include imaging techniques such as T1-weighted imaging technology, T2-weighted imaging technology, and fluid-attenuated inversion recovery imaging technology. Correspondingly, the images to be processed under at least two modes can be at least two images such as T1-weighted images, T2-weighted images, and fluid-attenuated inversion recovery images.

[0034] Since the images to be processed in different modalities can reflect different information of the target part to which the lesion belongs, by comprehensively analyzing the images to be processed in at least two modalities, the nature, size, location and other information of the lesion can be accurately determined, thereby achieving accurate segmentation of the lesion. The image to be used can be an image obtained by normalizing the images to be processed in different modalities. After the images to be processed in each modality are normalized, the image to be used in that modality is obtained.

[0035] Specifically, in order to improve the accuracy of subsequent image segmentation, different imaging technologies are used to collect images of the target part to which the lesion belongs, and obtain images to be processed under at least two modalities. Since the images to be processed under different modalities are obtained based on different imaging technologies, the image sizes of the images to be processed may be different. Then, the images to be processed of different modalities can be resampled to obtain images to be processed of the same image size under each modality. Since the positions of the target part to which the lesion belongs in the images to be processed of the same image size under different modalities may be different, the images to be processed under at least two modalities after resampling can be image registered to obtain the registered images to be processed. And according to the pixel values ​​of the target part to which the lesion belongs in each registered image to be processed, the mean and standard deviation of the target part to which the lesion belongs in each image to be processed are calculated to realize the normalization of the registered images to be processed, and obtain the images to be used under each modality.

[0036] In an embodiment of the present invention, the image to be used in each modality may be determined by performing registration of the images to be processed according to target parts in at least two modalities to obtain the image to be used in each modality.

[0037] The registration of the images to be processed may be understood as performing a rigid registration process on the target part to which the lesion belongs in the images to be processed in at least two modalities.

[0038] Specifically, for the images to be processed under at least two modalities, the image size of the images to be processed under each modality is determined. According to the image size of the images to be processed under each modality, the images to be processed are resampled to unify the image sizes of the images to be processed under different modalities, and the resampled images to be processed are obtained. Optionally, the resampling method can be at least one of the nearest neighbor interpolation method, the bilinear interpolation method and the cubic convolution interpolation method. Since the images to be processed under different modalities are collected based on different imaging technologies, there may be movement of the target part during the collection process. Therefore, the target parts under different modalities after resampling can be registered for the images to be processed to obtain the images to be processed under each modality after registration. Optionally, the image registration method can be to rotate and translate the pixel points of the target part in the image to be processed. For the images to be processed under at least two modalities after registration, according to the pixel value of the target part to which the lesion belongs in the image to be processed of the current modality after registration, the pixel mean and pixel standard deviation corresponding to the target part of the current modality after registration are determined. The pixel value of each pixel in the image to be processed under the current modality after registration is subtracted from the pixel mean corresponding to the target part, and divided by the pixel standard deviation to obtain the normalized image to be processed, that is, the image to be used under the current modality.

[0039] Exemplarily, the images to be processed of at least two modalities are T2-weighted images and diffusion-weighted (DWI) images as an example for explanation. If the image size of the T2-weighted image is 128*128 and the image size of the DWI image is 64*64, the T2-weighted image and the DWI image can be resampled by linear interpolation to obtain a T2-weighted image with an image size of 128*128 and a DWI image with an image size of 128*128. By means of rigid registration, the target parts in the T2-weighted image and the DWI image with an image size of 128*128 are rotated and translated to achieve the overlap of the target parts in the two images, and the registered T2-weighted image and the registered DWI image are obtained. For the registered T2-weighted image, the pixel mean and pixel standard deviation corresponding to the target part are determined according to the pixel value of each pixel point of the target part in the registered T2-weighted image. The following normalization formula is used to normalize the registered T2-weighted image to obtain a normalized T2-weighted image, i.e., the image to be used in the current modality. Correspondingly, for the registered DWI image, the image normalization processing method is similar and will not be repeated here. The normalization formula is expressed as follows:

[0040]

[0041] Among them, x' is the pixel value of the pixel point of the target part in the image to be used, x is the pixel value of the pixel point of the target part in the image to be processed after registration, and x mean is the pixel mean corresponding to the target part, x std is the pixel standard deviation corresponding to the target part.

[0042] S120, inputting the image to be used in each modality into the target lesion segmentation model to perform lesion region segmentation processing, and obtaining a target segmentation image with the target lesion marked.

[0043] Among them, the target lesion segmentation model can be understood as a model for lesion segmentation of the image to be used. The target lesion segmentation model includes an encoding part and a decoding part. The encoding part includes an image segmentation module, multiple feature extraction modules, and a downsampling module. The decoding part includes a feature fusion module and an upsampling module. The image segmentation module is used to divide the image to be used into multiple images to be fused. Multiple feature extraction modules are used to extract and fuse features of the images to be fused under each modality. The feature fusion module is used to fuse the features extracted by each feature extraction module. The downsampling module is used to downsample the image to be used or the image to be fused. The upsampling module is used to upsample the fused image.

[0044] The image to be fused is a partial image obtained after image segmentation of the image to be used. For example, if the image to be used is divided into 16 image blocks, each image block is an image to be fused. Image downsampling processing, that is, image downsampling processing, is used to reduce the image resolution of the image to be used or the image to be fused. Image downsampling processing can help extract the main features of the image and reduce redundant information of the image. Image upsampling processing is used to increase the image resolution of the image after fusion. Image upsampling processing can increase the details and clarity of the image, thereby improving the image quality. The target lesion can be understood as the lesion site in the target site predicted by the model. The target segmented image is an image containing the target lesion.

[0045] Among them, the encoding part in the target lesion segmentation model includes at least two processing branches, and at least two processing branches are used to encode the images to be used in the corresponding modality. The at least two processing branches are described from the input and output perspectives including an image segmentation module, a feature extraction module, a downsampling module, a feature extraction module, a downsampling module, and a feature extraction module.

[0046] Specifically, for images to be used in at least two modalities, the image to be used in the current modality is input into the current processing branch of the coding part in the target lesion segmentation model. The image to be used in another modality is input into another processing branch of the coding part in the target lesion segmentation model. It should be noted that if there are images to be used in three or more modalities, three or more processing branches are required. Therefore, the coding part of the target lesion segmentation model includes at least two processing branches.

[0047] Since each processing branch of the target lesion segmentation model processes the image to be used in each modality in a similar manner, two processing branches are used as an example for explanation here.

[0048] For the encoding part of the target lesion segmentation module, the image to be used in the current modality is input into the current processing branch of the encoding part in the target lesion segmentation model. The image segmentation module in the current processing branch performs segmentation processing on the image to be used in the current modality to obtain multiple images to be fused in the current modality. The image to be used in another modality is input into another processing branch of the encoding part in the target lesion segmentation model. The image segmentation module in another branch performs image segmentation processing on the image to be used in another modality to obtain multiple images to be fused in another modality.

[0049] The feature extraction module in the current processing branch performs feature extraction processing on multiple images to be fused under the current modality to obtain a feature image corresponding to the current modality. The feature extraction module in another processing branch performs feature extraction processing on multiple images to be fused under another modality to obtain a feature image corresponding to the other modality. The feature extraction module in the current processing branch performs feature fusion processing on the feature image corresponding to the current modality and the feature image corresponding to the other modality to obtain a fused feature image. The feature extraction module in another processing branch performs fusion processing on the feature image corresponding to the other modality and the feature image corresponding to the current modality to obtain a fused feature image. The downsampling module in the current processing branch performs downsampling processing on the fused feature image corresponding to the current processing branch to obtain a downsampled image. The downsampling module in another processing branch performs downsampling processing on the fused feature image corresponding to the other processing branch to obtain a downsampled image. Thereafter, the downsampled image is sequentially subjected to feature extraction processing, downsampling processing, and feature extraction processing to obtain a feature image to be fused corresponding to the current processing branch and a feature image to be fused corresponding to the other processing branch. The feature images to be fused corresponding to the two processing branches are subjected to feature fusion processing through the feature extraction module to obtain the multimodal fusion features output by the encoding part in the target lesion segmentation model. It should be noted that each feature extraction module and each downsampling module in the encoding part of the target lesion segmentation model process the input in a similar manner.

[0050] For the decoding part of the target lesion segmentation model, deep feature extraction is performed on the multi-modal fusion features output by the encoding part to obtain the first decoded image features. The first decoded image features and the image features output by the third feature extraction module of the encoding part are spliced ​​through the feature fusion module to obtain spliced ​​image features. Feature extraction is performed on the spliced ​​image features to obtain the first decoded feature image. The first decoded feature image is upsampled to obtain the upsampled image features. After that, the upsampled image features are spliced ​​and feature extracted with the image features output by the second feature extraction module of the encoding part to obtain the second decoded feature image. The second decoded feature image is upsampled to obtain the second upsampled image features. The second upsampled image features are spliced ​​and feature extracted with the image features output by the first feature extraction module of the encoding part to obtain the third decoded feature image.

[0051] According to the feature fusion module, the first decoded feature image, the second decoded feature image and the third decoded feature image are subjected to feature splicing processing to obtain the target segmentation result. Among them, the target segmentation result can be the probability value of each pixel in the image to be used being a lesion pixel. According to the preset segmentation threshold and the target segmentation result, multiple pixels belonging to the lesion pixels in the image to be used are determined. For example, the preset segmentation threshold can be 0.5, and the pixels with probability values ​​higher than 0.5 are determined as the pixels of the lesion. The pixels with probability values ​​lower than 0.5 are determined as non-lesion pixels. Connected domain analysis is used to remove erroneous pixels to obtain the pixels of the target lesion. According to the pixels of multiple target lesions, a target segmentation image with the target lesion marked is determined.

[0052] For example, see Figure 2 , Figure 2 This is an example of a target lesion segmentation model. Figure 2 In the figure, patchembedding is the image segmentation module, the feature extraction module is composed of 4 Swin Transformer Blocks, Swin Transformer Block is the feature extraction unit, DownSample is the downsampling module, and upSample is the upsampling module. Figure 2 The circle C in the figure is the feature fusion module.

[0053] Figure 2The left part is the encoding part. The specific processing method is: input the image to be used in the current modality with an image size of W×H into the current processing branch of the encoding part in the target lesion segmentation model. Among them, W is the width value of the image, and H is the height value of the image. The image segmentation module patch embedding in the current processing branch performs segmentation processing on the image to be used in the current modality to obtain 16 images to be fused in the current modality. At this time, the image size of the image to be fused is The image to be used in another modality is input into another processing branch of the encoding part in the target lesion segmentation model. The image segmentation module patch embedding in another branch performs image segmentation processing on the image to be used in another modality to obtain 16 images to be fused in another modality.

[0054] The feature extraction unit Swin TransformerBlock in the feature extraction module in the current processing branch performs feature extraction processing on the image to be fused in the current modality to obtain the feature image of the current modality. The feature extraction unit Swin Transformer Block in another processing branch performs feature extraction processing on the image to be fused in another modality to obtain the feature image of the other modality. Afterwards, the feature extraction units SwinTransformer Block in parallel in the two processing branches perform feature extraction processing and feature fusion processing on the feature image of the current modality and the feature image of the other modality to obtain the fused feature image under the current modality and the fused feature image under the other modality. The downsampling module DownSample of the current processing branch is used to downsample the fused feature image under the current processing branch to obtain the downsampled image. The downsampling module DownSample of another processing branch is used to downsample the fused feature image under the other processing branch to obtain the downsampled image. Afterwards, the downsampled image is processed by the feature extraction module, the downsampling module DownSample and the feature extraction module in the two processing branches in turn to obtain the feature image to be fused in the current processing branch and the feature image to be fused under the other processing branch. The feature images to be fused under the two processing branches are fused through the feature extraction unit Swin Transformer Block to obtain the multi-modal fusion features output by the encoding part.

[0055] Figure 2The right part in the middle is the decoding part. The specific processing method is: deep feature extraction is performed on the multi-modal fusion features output by the encoding part through the atrous spatial pyramid pooling layer (ASPP) to obtain the first decoded image feature. The first decoded image feature and the image feature output by the third feature extraction module of the encoding part are spliced ​​through the feature fusion module to obtain the spliced ​​image feature. The spliced ​​image feature is input into the feature extraction unit Swin Transformer Block to obtain the first decoded feature image. The first decoded feature image is upsampled through the upsampling module upSample to obtain the upsampled image feature. The upsampled image feature is spliced ​​with the image feature output by the second feature extraction module of the encoding part through the feature fusion module, and the feature extraction is performed through the feature extraction unit Swin Transformer Block to obtain the second decoded feature image. The second decoded feature image is upsampled through the upsampling module upSample to obtain the second upsampled image feature. The second upsampled image features are spliced ​​and feature extracted with the image features output by the first feature extraction module of the encoding part through the feature fusion module and the feature extraction unit Swin Transformer Block to obtain the third decoded feature image. The first decoded feature image, the second decoded feature image and the third decoded feature image are subjected to feature fusion processing according to the feature fusion module to obtain the target segmentation image.

[0056] Optionally, the method further includes: acquiring training samples to train the lesion segmentation model to be trained, so as to obtain a target lesion segmentation model;

[0057] The loss functions in the lesion segmentation model to be trained and the target lesion segmentation model are:

[0058]

[0059] Where n represents the number of training samples, y i represents the true value of the training sample, y i is 0 or 1, is the model prediction value, Y represents the true segmentation result corresponding to the training sample, Y is 0 or 1, represents the segmentation result predicted by the model,

[0060] It should be noted that the above loss function is a combination of the Binary Cross Entropy (BCE) loss function and the Dice (Dice Loss) loss function. The BCE loss function is used to compare the difference between the predicted value and the true value of the lesion segmentation model to be trained. The Dice loss function is used to measure the similarity between the true segmentation result and the actual segmentation result. In the above loss function, is the BCE loss function. is the Dice loss function.

[0061] Wherein, the training sample includes sample images of at least two modalities and actual segmentation results corresponding to the sample images. The model parameters of the lesion segmentation model to be trained are default values. The model parameters in the lesion segmentation model to be trained are corrected by the training samples to obtain the target lesion segmentation model. The true value is predetermined and is used to characterize whether the sample image in the training sample contains a lesion. Optionally, if the true value is 1, the sample image contains a lesion. If the true value is 0, the training sample does not contain a lesion. The model prediction value is output by the lesion segmentation model to be trained, which is used to characterize the probability value of the presence of a lesion in the sample image.

[0062] The true segmentation result is a predetermined result of whether the training sample contains a lesion. Optionally, if the sample image in the training sample contains a lesion, the true segmentation result is 1. If the sample image in the training sample does not contain a lesion, the true segmentation result is 0. The segmentation result predicted by the model is the probability value of each pixel in the sample image being a lesion pixel output by the lesion segmentation model to be trained. It should be noted that y in the BCE loss function i Corresponds to Y in the Dice loss function. Corresponding to the Dice loss function

[0063] Specifically, before training the lesion segmentation model to be trained, multiple training samples can be obtained to train the model based on the training samples. In order to improve the accuracy of the model, as many and rich training samples as possible can be obtained. It should be noted that the sample images in these training samples are images obtained after preprocessing such as image registration, normalization, and noise removal. The training samples are input into the lesion segmentation model to be trained to obtain the segmentation results predicted by the model.

[0064] The segmentation results predicted by the model and the actual segmentation results are processed according to the loss function, and the model parameters in the lesion segmentation model to be trained are modified based on the loss value to obtain the target lesion segmentation model.

[0065] When using the loss value to correct the model parameters in the lesion segmentation model to be trained, the convergence of the loss function can be used as a training goal, such as whether the training error is less than the preset error, or whether the error change tends to be stable, or whether the current number of iterations is equal to the preset number. If the detection reaches the convergence condition, such as the training error of the loss function is less than the preset error, or the error change trend tends to be stable, it indicates that the training of the lesion segmentation model to be trained is completed, and the iterative training can be stopped at this time. If it is detected that the convergence condition is not met at present, other training samples can be further obtained to continue training the lesion segmentation model to be trained until the training error of the loss function is within the preset range. When the training error of the loss function reaches convergence, the trained lesion segmentation model to be trained can be used as the target lesion segmentation model.

[0066] The technical solution of this embodiment obtains the image to be used in each mode by preprocessing the image to be processed of the target part to which the lesion belongs under at least two modes. By obtaining the image to be used under at least two modes, the image information corresponding to different modes can be fully utilized in the subsequent lesion segmentation to improve the accuracy and reliability of the segmentation process. The image to be used under each mode is input into the target lesion segmentation model for lesion area segmentation processing to obtain a target segmented image with the target lesion marked. Among them, the target lesion segmentation model includes an encoding part and a decoding part, the encoding part includes an image segmentation module, multiple feature extraction modules, and a downsampling module, and the decoding part includes a feature fusion module and an upsampling module. Through each module in the target lesion segmentation model, accurate segmentation of the lesion in the image to be used can be achieved, and the accuracy of the segmentation is improved. The present invention solves the problems of subjectivity, low segmentation accuracy and low segmentation efficiency in the segmentation results in the prior art. The accuracy and efficiency of lesion segmentation are improved, and the accuracy and reliability of the target segmented image are guaranteed.

[0067] Embodiment 2

[0068] Figure 3 This is a flowchart of a processing of image coding provided by the second embodiment of the present invention. This embodiment is based on the above embodiment and refines the step of "processing the image coding based on the coding part in the target lesion segmentation model". The specific implementation method can refer to the technical solution of this embodiment. Among them, the technical terms that are the same or corresponding to the above embodiment are not repeated here. Figure 3 As shown, the method includes:

[0069] S210 , for at least two processing branches, perform image segmentation processing based on the image segmentation modules in the processing branches to obtain a plurality of images to be fused.

[0070] The image segmentation module can be understood as a module for performing image segmentation processing on the image to be used. For example, Figure 2 The patch embedding in represents the image segmentation module. Figure 2 The image segmentation module patchembedding in can cut the image to be used into 16 non-overlapping images to be fused. The image to be fused can be understood as a partial image of the image to be used.

[0071] Specifically, since the model can process images to be used in at least two modalities, the encoding part of the target lesion segmentation model includes at least two processing branches. Each processing branch is used to process the image to be used in the corresponding modality. The image to be used is segmented into multiple non-overlapping images to be fused by the image segmentation module in the processing branch.

[0072] For example, see Figure 2 , take the images to be used in two modalities as examples for explanation. Correspondingly. The processing branches in the target lesion segmentation model are two-way. The image size of the image to be used is [H, W]. The image to be used in the current modality is input into the current processing branch, and the image to be used in the current modality is segmented by using the image segmentation module patch embedding to obtain 16 non-overlapping images to be fused. The size of each image to be fused is [H / 4, W / 4].

[0073] S220, extracting features of multiple images to be fused based on a feature extraction module and fusing them to obtain a first fused image;

[0074] Among them, the feature extraction module in at least two processing branches includes a first extraction unit and a second extraction unit, the first extraction unit is used to process images under one modality, and the second extraction unit is used to process images under another modality. The first extraction unit and the second extraction unit both include a first subunit and a second subunit, and the second subunit is used to fuse the output result of the first subunit and the output result of another first subunit and extract features. The fused first fused image can be a feature image obtained by fusing the image features corresponding to the images to be fused of at least two branches. The output result of the first subunit can be understood as the image feature information corresponding to the image to be fused obtained after feature extraction of the image to be fused.

[0075] Specifically, since the processing methods of the images to be fused under multiple modalities are similar, the processing of the images to be fused under two modalities is used as an example for explanation. According to the first subunit of the first extraction unit of the current processing branch in the feature extraction module, feature extraction processing is performed on multiple images to be fused under the current modality to obtain feature information corresponding to the images to be fused under the current modality, that is, the output result corresponding to the first subunit. According to the first subunit of the second extraction unit of another processing branch in the feature extraction module, feature extraction processing is performed on multiple images to be fused under another modality to obtain feature information corresponding to the images to be fused under another modality, that is, the output result of another first subunit.

[0076] According to the second subunit of the first extraction unit of the current processing branch in the feature extraction module, the output result of the first subunit and the output result of another first subunit are fused and feature extracted to obtain a fused first fused image corresponding to the current processing branch. According to the second subunit of the second extraction unit of another processing branch in the feature extraction module, the output result of the first subunit and the output result of another first subunit are fused and feature extracted to obtain a fused first fused image corresponding to another processing branch.

[0077] For example, in combination with the above examples, Figure 2 The feature extraction module includes four Swin Transformer Blocks. The left processing branch is the current processing branch, and the right processing branch is another processing branch. The two Swin Transformer Blocks in the current processing branch constitute the first extraction unit. And, in the order of input and output, the upper Swin Transformer Block is the first subunit, and the lower Swin Transformer Block is the second subunit. The two Swin Transformer Blocks in the other processing branch constitute the second extraction unit. And in the order of input and output, from top to bottom, they are the first subunit and the second subunit of the second extraction unit.

[0078] The image to be fused in the current modality is input into the first subunit Swin Transformer Block of the first extraction unit, and feature extraction processing is performed to obtain image feature information of the image to be fused in the current modality. The image to be fused in another modality is input into the first subunit Swin Transformer Block of the second extraction unit, and feature extraction processing is performed to obtain image feature information corresponding to the image to be fused in another modality.

[0079] According to the second subunit Swin Transformer Block of the first extraction unit under the current processing branch, the image feature information of the image to be fused under the current modality and the image feature information corresponding to the image to be fused under another modality are subjected to feature fusion processing to obtain the first fused image corresponding to the current processing branch. According to the second subunit Swin Transformer Block of the second extraction unit under another processing branch, the image feature information of the image to be fused under another modality and the image feature information of the image to be fused under the current modality are subjected to feature fusion processing to obtain the first fused image corresponding to the other processing branch. The image feature size corresponding to the first fused image is [H / 4, W / 4, C], where C represents the number of image features.

[0080] S230 . Downsample the first fused image based on a downsampling module to obtain a first downsampled image.

[0081] The downsampling module is used to downsample the first fused image to facilitate subsequent extraction of main features of the image and reduce redundant information in the image. The first downsampled image is a feature image obtained after downsampling the first fused image.

[0082] Specifically, the processing of the to-be-fused images under two modalities is taken as an example for explanation. The first fused image output by the feature extraction module of the current processing branch is downsampled according to the downsampling module in the current processing branch to obtain the first downsampled image corresponding to the current processing branch. It should be noted that, for at least two processing branches, the downsampling processing of the first fused image by the downsampling module in each processing branch is similar, which will not be repeated here.

[0083] Exemplarily, in combination with the above example, DownSample represents a downsampling module. Figure 2 The downsampling module DownSample of the current processing branch processes the first fused image of the current processing branch to obtain the first downsampled image corresponding to the current processing branch. The downsampling module DownSample of another processing branch processes the first fused image of another processing branch to obtain the first downsampled image corresponding to the other processing branch.

[0084] S240: Extract features of the first downsampled image based on the feature extraction module and perform fusion processing to obtain a second fused image, and downsample the second fused image to obtain a second downsampled image.

[0085] The second fused image can be understood as a feature image obtained by fusing the image features of the first downsampled images of at least two branches. The second downsampled image can be a feature image obtained by downsampling the second fused image.

[0086] Specifically, two processing branches are used as an example for explanation. According to the feature extraction module, feature extraction processing is performed on the first downsampled image in the current processing branch to obtain image feature information corresponding to the first downsampled image under the current processing branch. According to the feature extraction module, feature extraction processing is performed on the first downsampled image in another processing branch to obtain image feature information corresponding to the first downsampled image under the other processing branch. The feature extraction module is used to fuse the image feature information corresponding to the first downsampled image under the current processing branch and the image feature information corresponding to the first downsampled image under the other processing branch to obtain the second fused image under each processing branch. The downsampling module of the current processing branch is used to downsample the second fused image under the current processing branch to obtain the second downsampled image. The downsampling module of another processing branch is used to downsample the second fused image under another processing branch to obtain the second downsampled image.

[0087] Exemplarily, in combination with the above examples, according to Figure 2 In the process, the first downsampled image is subjected to feature extraction processing by the feature extraction unit Swin Transformer Block in the feature extraction module under the current processing branch to obtain the image feature information corresponding to the first downsampled image under the current processing branch. The first downsampled image is subjected to feature extraction processing by the feature extraction unit Swin Transformer Block in the feature extraction module under another processing branch to obtain the image feature information corresponding to the first downsampled image under another processing branch. The image feature information corresponding to the first downsampled image under the current processing branch and the image feature information corresponding to the first downsampled image under another processing branch are subjected to feature fusion processing by the next feature extraction unit Swin Transformer Block of the feature extraction module under the current processing branch to obtain the second fused image under the current processing branch. Correspondingly, the second fused image under another processing branch is obtained by the next feature extraction unit Swin Transformer Block of the feature extraction module under another processing branch. At this time, the feature size of the image feature corresponding to the second fused image is [H / 8, W / 8, 2C]. The second fused image is subjected to downsampling processing by the downsampling module DownSample of the two processing branches to obtain the second downsampled image.

[0088] S250: Extract features of the second downsampled image based on the feature extraction module and perform fusion processing to obtain a third fused image.

[0089] The third fused image may be understood as a feature image obtained by fusing image features corresponding to the second down-sampled images under at least two branches.

[0090] Specifically, feature extraction processing is performed on the second downsampled image in the current processing branch according to the feature extraction module to obtain image feature information corresponding to the second downsampled image in the current processing branch. Feature extraction processing is performed on the second downsampled image in another processing branch according to the feature extraction module to obtain image feature information corresponding to the second downsampled image in another processing branch. The feature extraction module is used to fuse the image feature information corresponding to the second downsampled image in the current processing branch and the image feature information corresponding to the second downsampled image in another processing branch to obtain a third fused image in each processing branch.

[0091] Exemplarily, in combination with the above example. The feature extraction unit Swin Transformer Block in the feature extraction module under the current processing branch performs feature extraction processing on the second downsampled image to obtain image feature information corresponding to the second downsampled image under the current processing branch. The feature extraction unit Swin Transformer Block in the feature extraction module under another processing branch performs feature extraction processing on the second downsampled image to obtain image feature information corresponding to the second downsampled image under another processing branch. The next feature extraction unit Swin Transformer Block of the feature extraction module under the current processing branch performs feature fusion processing on the image feature information corresponding to the second downsampled image under the current processing branch and the image feature information corresponding to the second downsampled image under another processing branch to obtain a third fused image under the current processing branch. Correspondingly, the next feature extraction unit Swin Transformer Block of the feature extraction module under another processing branch obtains a third fused image under another processing branch. At this time, the feature size of the image feature corresponding to the third fused image is [H / 16, W / 16, 4C].

[0092] S260 , performing feature splicing on the third fused image under multiple modalities to obtain multimodal fusion features after multimodal encoding of the image to be used.

[0093] Among them, the multimodal fusion feature can be a feature obtained by feature stitching of image features of the corresponding third fused image under at least two modalities.

[0094] Specifically, feature splicing processing is performed on image features corresponding to the third fused images of multiple modalities under at least two processing branches to obtain multimodal fusion features after multimodal encoding processing is performed on the image to be used.

[0095] Exemplarily, in combination with the above example, the image features corresponding to the third fused image under the two processing branches are subjected to feature splicing processing, and the spliced ​​image features are subjected to feature extraction by the feature extraction unit Swin Transformer Block to obtain multimodal fusion features. At this time, the feature size of the multimodal fusion feature is [H / 16, W / 16, 8C]. Based on this, the encoding processing of the image to be used is completed.

[0096] The technical solution of this embodiment is that for at least two processing branches, the image to be used is segmented and processed based on the image segmentation module in the processing branch to obtain multiple images to be fused. Compared with directly processing the image to be used, the efficiency of image processing can be improved, and it is convenient to perform fine analysis of the image. The feature extraction module extracts and fuses the features of multiple images to be fused to obtain a first fused image after fusion. The image information in the images to be fused can be fully integrated and analyzed through feature extraction and feature fusion. The first fused image is downsampled by the downsampling module to reduce redundant information in the first fused image to obtain a first downsampled image. The first downsampled image is extracted and fused based on the feature extraction module to obtain a second fused image, and the second fused image is downsampled to obtain a second downsampled image. The second downsampled image is extracted and fused based on the feature extraction module to obtain a third fused image. By performing feature extraction, feature fusion and downsampling on the first downsampled image in sequence, the image information of the first downsampled image under multimodality can be fully integrated and analyzed. The third fused image under multiple modalities is feature spliced ​​to obtain the multimodal fusion features after multimodal encoding of the image to be used, and subsequent decoding processing is performed based on the multimodal fusion features to obtain the target segmented image. The present invention solves the problem that the image information of the multimodal image cannot be fully utilized in the prior art, and realizes the full integration and analysis of the image to be used under multimodality, so as to facilitate the subsequent decoding and analysis to obtain an accurate and reliable target segmented image.

[0097] Embodiment 3

[0098] Figure 4 This is a flowchart of a decoding process for an image to be used provided by the third embodiment of the present invention. This embodiment is based on the above embodiment and refines the step of "decoding the multimodal fusion features based on the decoding part in the target lesion segmentation model". The specific implementation method can refer to the technical solution of this embodiment. Among them, the technical terms that are the same or corresponding to the above embodiment are not repeated here. Figure 4 As shown, the method includes:

[0099] S310: Process the multimodal fusion features based on an adaptive linear mean pooling layer to obtain a first decoding feature.

[0100] Among them, the decoding part in the target lesion segmentation model includes a feature extraction unit and an upsampling module. The adaptive linear mean pooling layer is used to further extract the multimodal fusion features. The first decoding feature can be understood as the feature obtained after feature extraction of the multimodal fusion features. The feature extraction unit in the decoding part of the target lesion segmentation model is used to extract features from the spliced ​​features. The upsampling module is used to upsample the fused image. The image upsampling process can increase the details and clarity of the image, thereby improving the image quality.

[0101] Specifically, the multimodal fusion features output by the encoding part are subjected to feature extraction processing through the adaptive linear mean pooling layer of the decoding part in the target lesion segmentation model to obtain the first decoding feature.

[0102] Exemplarily, in combination with the above example, the adaptive linear mean pooling layer is taken as ASPP as an example for explanation. The multimodal fusion feature output by the feature extraction unit Swin Transformer Block is subjected to feature extraction processing by the ASPP module to obtain a first decoding feature.

[0103] S320: splice the first decoding feature with the third fused feature of the third fused image, and input the concatenated features into a feature extraction unit to obtain a first decoded image.

[0104] The third fusion feature can be understood as an image feature corresponding to the third fusion image. The first decoded image can be a feature image obtained by splicing and decoding the first decoded feature and the third fusion feature.

[0105] Specifically, the first decoding feature and the third fused feature corresponding to the third fused image output by the encoding part feature extraction module are subjected to feature splicing or feature cascade processing to obtain a spliced ​​feature, and the spliced ​​feature is output to the feature extraction unit to obtain a first decoded image.

[0106] Exemplarily, in combination with the above example, Swin Transformer Block represents a feature extraction unit. The feature size of the first decoded feature is [H / 16, W / 16, 8C], and the feature size corresponding to the third fused feature is [H / 16, W / 16, 4C]. The first decoded feature and the third fused feature are subjected to feature cascade processing to obtain a spliced ​​feature. The size of the spliced ​​feature is [H / 16, W / 16, 12C]. The spliced ​​feature is input into the feature extraction unit Swin Transformer Block to obtain a first decoded image. Among them, the feature size of the image feature corresponding to the first decoded image is [H / 16, W / 16, 4C].

[0107] S330, splicing the first decoded image after upsampling processing with the second fused features of the second fused image, and inputting the spliced ​​features into a feature extraction unit to obtain a second decoded image.

[0108] The second decoded image may be a feature image obtained by splicing and decoding the image features of the first decoded image and the second fusion features.

[0109] Specifically, the first decoded image is upsampled by the upsampling module of the decoding part to obtain an upsampled image corresponding to the first decoded image. The image features of the upsampled image corresponding to the first decoded image are spliced ​​with the second fused features of the second fused image to obtain spliced ​​features. The spliced ​​features are input into the feature extraction unit to obtain the second decoded image.

[0110] Exemplary, combined with the above example. upSample represents an upsampling module. The first decoded image is upsampled by the upsampling module upSample of the decoding part to obtain an upsampled image corresponding to the first decoded image. The feature size of the image feature corresponding to the upsampled image is [H / 8, W / 8, 4C]. The image feature corresponding to the upsampled image is spliced ​​with the second fusion feature to obtain the spliced ​​feature. At this time, the size of the spliced ​​feature is [H / 8, W / 8, 6C]. The spliced ​​feature is processed by the feature extraction unit Swin Transformer Block to obtain the second decoded image. Among them, the feature size of the image feature of the second decoded image is [H / 8, W / 8, 2C].

[0111] S340, splicing the second decoded image after upsampling with the first fused features of the first fused image, and inputting the spliced ​​features into a feature extraction unit to obtain a third decoded image.

[0112] The third decoded image may be a feature image obtained by concatenating and extracting the image features of the second decoded image and the first fusion features.

[0113] Specifically, the second decoded image is upsampled by the upsampling module of the decoding part to obtain an upsampled image corresponding to the second decoded image. The image features of the upsampled image corresponding to the second decoded image are spliced ​​with the first fused features of the first fused image to obtain spliced ​​features. The spliced ​​features are input into the feature extraction unit to obtain a third decoded image.

[0114] Exemplarily, in combination with the above example. The second decoded image is upsampled by the upsampling module upSample of the decoding part to obtain an upsampled image corresponding to the second decoded image. The feature size of the image feature corresponding to the upsampled image is [H / 4, W / 4, 2C]. The image feature corresponding to the upsampled image is spliced ​​with the first fusion feature to obtain the spliced ​​feature. At this time, the size of the spliced ​​feature is [H / 4, W / 4, 3C]. The spliced ​​feature is processed by the feature extraction unit Swin Transformer Block to obtain a third decoded image. Among them, the feature size of the image feature of the third decoded image is [H / 4, W / 4, C].

[0115] S350: Determine a target segmented image after decoding based on the third decoded image, the second decoded image, and the first decoded image.

[0116] The target segmentation image may be an image marked with a target lesion.

[0117] Specifically, feature splicing is performed on the image features corresponding to the third decoded image, the image features corresponding to the second decoded image, and the image features corresponding to the first decoded image, and all the spliced ​​features are input into the segmentation layer of the target segmentation model to obtain the probability value of each pixel in the image being a lesion pixel. According to the preset lesion threshold and the probability value of each pixel, multiple pixels belonging to the lesion are determined. The connected domain analysis method is used to remove erroneous pixels from the multiple pixels to obtain the target segmented image.

[0118] In an embodiment of the present invention, the decoding part also includes a PixelShuffle module, and the method for determining the target segmented image after decoding processing can be: based on the PixelShuffle module, the first decoded image and the second decoded image are processed respectively to obtain a first image to be applied corresponding to the first decoded image, and a second image to be applied corresponding to the second decoded image; based on the first image to be applied, the second image to be applied and the third decoded image, a target fusion image is determined; based on the target segmentation layer, the target lesion in the target fusion image is segmented to obtain a target segmented image.

[0119] Among them, the PixelShuffle module is used to upsample the low-resolution feature image to obtain a high-resolution feature image. The first image to be applied can be a feature image obtained by upsampling the first decoded image. The image resolution of the first image to be applied is higher than the image resolution of the first decoded image. The second image to be applied can be a feature image obtained by upsampling the second decoded image. The image resolution of the second image to be applied is higher than the image resolution of the second decoded image. The target fusion image can be a feature image obtained by cascading or fusing the image features in the first image to be applied, the second image to be applied, and the third decoded image. The target segmentation layer is used to segment the target lesion in the target fusion image.

[0120] Specifically, the first decoded image is upsampled by the PixelShuffle module to obtain a first image to be applied corresponding to the first decoded image. The second decoded image is upsampled by the PixelShuffle module to obtain a second image to be applied corresponding to the second decoded image. Feature cascading or feature fusion processing is performed on the image features corresponding to the first image to be applied, the second image to be applied, and the third decoded image to obtain a target fused image. The target fused image is input into the target segmentation layer, and the target lesion is determined by determining the probability value of each pixel point in the target fusion image belonging to the target lesion pixel point. The target lesion in the target fusion image is segmented to obtain a target segmented image.

[0121] Exemplarily, in combination with the above example, the first decoded image is processed by the PixelShuffle module to obtain the first image to be applied corresponding to the first decoded image. The image feature size of the first image to be applied is [H / 4, W / 4, C / 4]. The second decoded image is processed by the PixelShuffle module to obtain the second image to be applied corresponding to the second decoded image. The feature size of the image feature corresponding to the second image to be applied is [H / 4, W / 4, C / 2]. The image features corresponding to the first image to be applied, the second image to be applied and the third decoded image are subjected to feature cascading or feature fusion processing to obtain a target fused image. The target fused image is input into the target segmentation layer Segent Head, and the target lesion is determined by determining the probability value of each pixel point in the target fusion image belonging to the target lesion pixel point. The target lesion in the target fusion image is segmented to obtain a target segmented image.

[0122] The technical solution of this embodiment processes the multimodal fusion feature through an adaptive linear mean pooling layer to obtain a first decoding feature. After the first decoding feature is spliced ​​with the third fusion feature of the third fusion image, it is input into the feature extraction unit to obtain a first decoded image. After the first decoded image is upsampled and processed, it is spliced ​​with the second fusion feature of the second fusion image, and the spliced ​​features are input into the feature extraction unit to obtain a second decoded image. After the second decoded image is upsampled and processed, it is spliced ​​with the first fusion feature of the first fusion image, and the spliced ​​features are input into the feature extraction unit to obtain a third decoded image. Based on the third decoded image, the second decoded image and the first decoded image, the target segmentation image after decoding is determined. Through multiple feature splicing and feature decoding, an accurate and reliable target segmentation image is obtained. The problem of subjectivity, low segmentation accuracy and low segmentation efficiency in the segmentation results in the prior art is solved. The robustness of the target segmentation model is enhanced, the accuracy and efficiency of lesion segmentation are improved, and the accuracy and reliability of the target segmentation image are guaranteed.

[0123] Embodiment 4

[0124] Figure 5 Schematic diagram of the structure of a multi-modal feature fusion lesion segmentation device provided in the fourth embodiment of the present invention. Figure 5 As shown, the device includes: an image preprocessing module 410 and an image segmentation module 420.

[0125] The image preprocessing module 410 is used to obtain the image to be used in each modality by preprocessing the image to be processed of the target part to which the lesion belongs in at least two modalities; the image segmentation module 420 is used to input the image to be used in each modality into the target lesion segmentation model to perform lesion area segmentation processing to obtain a target segmented image with the target lesion marked; wherein the target lesion segmentation model includes an encoding part and a decoding part, the encoding part includes an image segmentation module, multiple feature extraction modules, and a downsampling module, the decoding part includes a feature fusion module and an upsampling module, the image segmentation module is used to divide the image to be used into multiple images to be fused, the multiple feature extraction modules are used to extract and fuse features of the images to be fused in each modality, the feature fusion module is used to fuse the features extracted by each feature extraction module, the downsampling module is used to downsample the image to be used or the image to be fused, and the upsampling module is used to upsample the fused image.

[0126] The technical solution of this embodiment obtains the image to be used in each mode by preprocessing the image to be processed of the target part to which the lesion belongs under at least two modes. By obtaining the image to be used under at least two modes, the image information corresponding to different modes can be fully utilized in the subsequent lesion segmentation to improve the accuracy and reliability of the segmentation process. The image to be used under each mode is input into the target lesion segmentation model for lesion area segmentation processing to obtain a target segmented image with the target lesion marked. Among them, the target lesion segmentation model includes an encoding part and a decoding part, the encoding part includes an image segmentation module, multiple feature extraction modules, and a downsampling module, and the decoding part includes a feature fusion module and an upsampling module. Through each module in the target lesion segmentation model, accurate segmentation of the lesion in the image to be used can be achieved, and the accuracy of the segmentation is improved. The present invention solves the problems of subjectivity, low segmentation accuracy and low segmentation efficiency in the segmentation results in the prior art. The accuracy and efficiency of lesion segmentation are improved, and the accuracy and reliability of the target segmented image are guaranteed.

[0127] On the basis of the above embodiment, optionally, an image preprocessing module is used to perform registration of the image to be processed according to the target parts under at least two modalities to obtain the image to be used under each modality.

[0128] Optionally, the encoding part in the target lesion segmentation model includes at least two processing branches, and the at least two processing branches are used to encode the images to be used in the corresponding modality. The at least two processing branches are described from the input and output perspectives including an image segmentation module, a feature extraction module, a downsampling module, a feature extraction module, a downsampling module, and a feature extraction module.

[0129] Optionally, in the image segmentation module, the image encoding processing is performed based on the encoding part in the target lesion segmentation model, including: a to-be-fused image determination unit, used for, for at least two processing branches, performing the to-be-fused image segmentation processing based on the image segmentation module in the processing branch, to obtain multiple to-be-fused images; a first fused image determination unit, used for extracting and fusing features of multiple to-be-fused images based on the feature extraction module, to obtain a fused first fused image; the feature extraction modules in at least two processing branches include a first extraction unit and a second extraction unit, the first extraction unit is used for image processing in one modality, and the second extraction unit is used for image processing in another modality, the first extraction unit and the second extraction unit both include a first subunit and a second subunit, and the second subunit The unit is used to fuse the output result of the first subunit and the output result of another first subunit and extract features; the first down-sampled image determination unit is used to down-sample the first fused image based on the down-sampling module to obtain the first down-sampled image; the second down-sampled image determination unit is used to extract features of the first down-sampled image and fuse it based on the feature extraction module to obtain the second fused image, and down-sample the second fused image to obtain the second down-sampled image; the third fused image determination unit is used to extract features of the second down-sampled image and fuse it based on the feature extraction module to obtain the third fused image; the multimodal fusion feature determination unit is used to perform feature splicing on the third fused image under multiple modalities to obtain the multimodal fusion feature after multimodal encoding of the image to be used.

[0130] Optionally, the decoding part in the target lesion segmentation model includes a feature extraction unit and an upsampling module. In the image segmentation module, the multimodal fusion feature is decoded based on the decoding part in the target lesion segmentation model, including: a first decoding feature determination unit, which is used to process the multimodal fusion feature based on an adaptive linear mean pooling layer to obtain a first decoding feature; a first decoding image determination unit, which is used to splice the first decoding feature with the third fusion feature of the third fusion image, and input it to the feature extraction unit to obtain a first decoded image; a second decoding image determination unit, which is used to splice the first decoded image with the second fusion feature of the second fusion image after upsampling, and input the spliced ​​feature to the feature extraction unit to obtain a second decoded image; a third decoding image determination unit, which is used to splice the second decoded image with the first fusion feature of the first fusion image after upsampling, and input the spliced ​​feature to the feature extraction unit to obtain a third decoded image; a target segmentation image determination unit, which is used to determine the target segmentation image after decoding based on the third decoded image, the second decoded image and the first decoded image.

[0131] Optionally, the decoding part also includes a PixelShuffle module and a target segmentation image determination unit, which is used to process the first decoded image and the second decoded image respectively based on the PixelShuffle module to obtain a first image to be applied corresponding to the first decoded image, and a second image to be applied corresponding to the second decoded image; determine a target fusion image based on the first image to be applied, the second image to be applied and the third decoded image; and segment and process the target lesion in the target fusion image based on the target segmentation layer to obtain a target segmentation image.

[0132] Optionally, the device further includes: a model training module, which is used to: obtain training samples to train the lesion segmentation model to be trained to obtain a target lesion segmentation model; the loss function in the lesion segmentation model to be trained and the target lesion segmentation model is:

[0133]

[0134] Where n represents the number of training samples, y i represents the true value of the training sample, y i is 0 or 1, is the model prediction value, Y represents the true segmentation result corresponding to the training sample, Y is 0 or 1, represents the segmentation result predicted by the model,

[0135] The multimodal feature fusion lesion segmentation device provided in the embodiment of the present invention can execute the multimodal feature fusion lesion segmentation method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0136] Embodiment 5

[0137] Figure 6 1 is a schematic diagram of the structure of an electronic device provided in Embodiment 5 of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0138] like Figure 6As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0139] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0140] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 executes the various methods and processes described above, such as a lesion segmentation method of multimodal feature fusion.

[0141] In some embodiments, the lesion segmentation method of multimodal feature fusion can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the lesion segmentation method of multimodal feature fusion described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the lesion segmentation method of multimodal feature fusion by any other appropriate means (e.g., by means of firmware).

[0142] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0143] The computer program for implementing the lesion segmentation method of multimodal feature fusion of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the computer program makes the function / operation specified in the flow chart and / or the block diagram be implemented when executed by the processor. The computer program can be executed completely on the machine, partially on the machine, as an independent software package, partially on the machine and partially on the remote machine or completely on the remote machine or server.

[0144] Embodiment 6

[0145] Embodiment 6 of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to execute a lesion segmentation method with multimodal feature fusion, the method comprising:

[0146] By preprocessing the images to be processed of the target part to which the lesion belongs in at least two modalities, the images to be used in each modality are obtained; the images to be used in each modality are input into the target lesion segmentation model for lesion area segmentation processing, and a target segmentation image with the target lesion marked is obtained; wherein the target lesion segmentation model includes an encoding part and a decoding part, the encoding part includes an image segmentation module, multiple feature extraction modules, and a downsampling module, the decoding part includes a feature fusion module and an upsampling module, the image segmentation module is used to divide the image to be used into multiple images to be fused, the multiple feature extraction modules are used to extract and fuse features of the images to be fused in each modality, the feature fusion module is used to fuse the features extracted by each feature extraction module, the downsampling module is used to downsample the image to be used or the image to be fused, and the upsampling module is used to upsample the fused image.

[0147] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0148] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0149] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0150] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0151] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0152] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A lesion segmentation method using multimodal feature fusion, characterized in that: include: Preprocessing the images to be processed of the target part to which the lesion belongs in at least two modalities to obtain the image to be used in each modality; Inputting the to-be-used image in each modality into a target lesion segmentation model to perform lesion region segmentation processing, and obtaining a target segmentation image with the target lesion marked; Among them, the target lesion segmentation model includes an encoding part and a decoding part, the encoding part includes an image segmentation module, multiple feature extraction modules, and a downsampling module, the decoding part includes a feature fusion module and an upsampling module, the image segmentation module is used to divide the image to be used into multiple images to be fused, the multiple feature extraction modules are used to extract and fuse features of the images to be fused under each modality, the feature fusion module is used to fuse the features extracted by each feature extraction module, the downsampling module is used to downsample the image to be used or the image to be fused, and the upsampling module is used to upsample the fused image.

2. The method according to claim 1, characterized in that The method of preprocessing the images to be processed in at least two modes of the target part to which the lesion belongs to, to obtain the images to be used in each mode, includes: The images to be processed are registered according to the target parts in the at least two modalities to obtain the images to be used in each modality.

3. The method according to claim 1, characterized in that The encoding part in the target lesion segmentation model includes at least two processing branches, and the at least two processing branches are used to encode the image to be used in the corresponding modality. The at least two processing branches are described from the input and output perspectives and include an image segmentation module, the feature extraction module, a downsampling module, a feature extraction module, a downsampling module, and a feature extraction module.

4. The method according to claim 3, characterized in that The encoding process of the image to be used based on the encoding part in the target lesion segmentation model comprises: For the at least two processing branches, segmenting the image to be used based on the image segmentation module in the processing branch to obtain a plurality of images to be fused; Extracting features of the multiple images to be fused based on the feature extraction module and fusing them to obtain a fused first fused image; the feature extraction module in the at least two processing branches includes a first extraction unit and a second extraction unit, the first extraction unit is used to process an image under one modality, and the second extraction unit is used to process an image under another modality, the first extraction unit and the second extraction unit both include a first subunit and a second subunit, and the second subunit is used to fuse and extract features from an output result of the first subunit and an output result of another first subunit; Downsampling the first fused image based on the downsampling module to obtain a first downsampled image; Extracting features of the first downsampled image based on the feature extraction module and fusing the features to obtain a second fused image, and downsampling the second fused image to obtain a second downsampled image; Extracting features of the second downsampled image based on the feature extraction module and fusing the features to obtain a third fused image; Feature splicing is performed on the third fused image under multiple modalities to obtain multimodal fusion features after multimodal encoding of the image to be used.

5. The method according to claim 4, characterized in that The decoding part in the target lesion segmentation model includes a feature extraction unit and an upsampling module, and the multimodal fusion feature is decoded based on the decoding part in the target lesion segmentation model, including: Processing the multimodal fusion feature based on an adaptive linear mean pooling layer to obtain a first decoding feature; After splicing the first decoding feature and the third fused feature of the third fused image, the splicing is input into a feature extraction unit to obtain a first decoded image; splicing the first decoded image after upsampling with the second fused features of the second fused image, and inputting the spliced ​​features into the feature extraction unit to obtain a second decoded image; splicing the second decoded image after upsampling with the first fused feature of the first fused image, and inputting the spliced ​​feature into the feature extraction unit to obtain a third decoded image; A target segmented image after decoding is determined based on the third decoded image, the second decoded image, and the first decoded image.

6. The method according to claim 5, characterized in that The decoding part further includes a PixelShuffle module, and the determining of a target segmented image after decoding based on the third decoded image, the second decoded image, and the first decoded image includes: Based on the PixelShuffle module, the first decoded image and the second decoded image are processed respectively to obtain a first image to be applied corresponding to the first decoded image and a second image to be applied corresponding to the second decoded image; Determine a target fused image based on the first image to be applied, the second image to be applied, and the third decoded image; The target lesion in the target fusion image is segmented based on the target segmentation layer to obtain the target segmentation image.

7. The method according to claim 1, characterized in that The method further comprises: Acquire training samples to train the lesion segmentation model to be trained, so as to obtain the target lesion segmentation model; The loss functions in the lesion segmentation model to be trained and the target lesion segmentation model are: Where n represents the number of training samples, y i represents the true value of the training sample, y i is 0 or 1, is the model prediction value, Y represents the true segmentation result corresponding to the training sample, Y is 0 or 1, represents the segmentation result predicted by the model, 8. A multimodal feature fusion lesion segmentation device, characterized in that: include: An image preprocessing module, used to obtain an image to be used in each modality by preprocessing the images to be processed of the target part to which the lesion belongs in at least two modalities; An image segmentation module, used for inputting the to-be-used image in each modality into a target lesion segmentation model to perform lesion region segmentation processing, and obtaining a target segmentation image with the target lesion marked; Among them, the target lesion segmentation model includes an encoding part and a decoding part, the encoding part includes an image segmentation module, multiple feature extraction modules, and a downsampling module, the decoding part includes a feature fusion module and an upsampling module, the image segmentation module is used to divide the image to be used into multiple images to be fused, the multiple feature extraction modules are used to extract and fuse features of the images to be fused under each modality, the feature fusion module is used to fuse the features extracted by each feature extraction module, the downsampling module is used to downsample the image to be used or the image to be fused, and the upsampling module is used to upsample the fused image.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the lesion segmentation method of multimodal feature fusion according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the multimodal feature fusion lesion segmentation method according to any one of claims 1 to 7 when executed.

Images