A multi-modal image segmentation method based on self-supervised learning

Abstract

The application provides a kind of multi-modal image segmentation method based on self-supervised learning, first, after the processing of data set, it is segmented using the multi-encoding path-single decoding path network suitable for different modal inputs.Second, in order to encourage the network to learn the important semantic features of different modalities, a feature fusion strategy applied to the bottleneck layer and a skip-connection attention mechanism are designed.Finally, the problem of small data set size is alleviated using self-supervised learning.The application uses self-supervised learning strategy to extract and combine rich information from multi-modal image data, enabling accurate image segmentation and testing on soft tissue sarcoma data set.The experimental results show that, compared with the traditional method, the method of the application can more accurately locate and segment, providing strong support for medical image analysis.This method is not only suitable for soft tissue sarcoma multi-modal MRI image segmentation, but also can be widely used in other multi-modal image segmentation tasks.

Description

Technical Field

[0001] This invention belongs to the field of image processing and proposes a multimodal image segmentation method based on self-supervised learning. Background Technology

[0002] Traditional image segmentation methods primarily rely on hand-designed feature extraction and machine learning algorithms, such as K-means clustering and support vector machines. However, these methods have limited performance in complex scenes and pose significant challenges for multimodal image segmentation. For example, the multimodal image segmentation model proposed in "Soft Tissue Sarcoma Co-Segmentation in Combined MRI and PET / CT Data" fails to extract and fuse shared and unique information between different modalities, does not consider how the model can utilize existing modalities to generate features for missing modalities, and does not employ techniques such as self-supervised learning to alleviate the problem of small datasets in multimodal image processing. Consequently, it may perform poorly in multimodal image segmentation. With the development of deep learning technology, convolutional neural networks (CNNs) have made significant progress in image segmentation tasks. However, traditional supervised learning methods typically require large amounts of labeled data, and the labeling cost for multimodal image data is high, limiting their application scope. In the field of image segmentation, many methods have been proposed to handle different types of image data:

[0003] Multimodal fusion methods combine image information from different sensors or modalities to improve the accuracy of image segmentation. For example, fusing infrared and visible light images utilizes their complementary information to improve target detection and segmentation results.

[0004] Graph-based segmentation methods: Image segmentation can be modeled as a graph theory problem, where the pixels of an image are considered as nodes in a graph, and the similarity between pixels determines the weight of the edges. Then, graph cut algorithms (such as GraphCut, Random Walks, etc.) are used to segment the image, dividing the pixels into different regions.

[0005] Deep learning-based methods: In recent years, deep learning technology has made significant progress in the field of image segmentation. Convolutional Neural Networks (CNNs) are widely used for image segmentation tasks. By training a CNN network, pixel-level semantic segmentation of images can be achieved.

[0006] Multi-task learning methods: These methods improve image segmentation performance by simultaneously processing multiple related tasks within a single network, enabling shared feature learning. For example, image semantic analysis can be performed concurrently with segmentation.

[0007] Multimodal MRI images are images that integrate different imaging modalities and parameters, providing rich information about biological tissues. However, this complexity also presents challenges for image segmentation, such as information fusion between different modalities, image noise, and structural variations. To address these challenges, innovative methods need to be developed to improve the accuracy and robustness of multimodal image segmentation. Summary of the Invention

[0008] To address the problems of existing technologies, this invention provides a multimodal image segmentation method based on self-supervised learning. This method utilizes a self-supervised learning strategy to extract and combine the rich information contained in each modality of the multimodal image data, achieving accurate image segmentation. This method can fully exploit the unique information between different modalities and within a single modality, thereby improving the accuracy and robustness of segmentation. This invention proposes a semantic segmentation network with a multi-encoding path and a single decoding path, a fusion strategy, a skip connection attention mechanism, and an auxiliary encoding path, effectively utilizing the complementary information in the multimodal data and exploring the impact of different modal input combinations on the segmented regions.

[0009] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0010] A multimodal image segmentation method based on self-supervised learning is proposed. The method first involves data acquisition, which requires acquiring multimodal images. These images undergo a series of preprocessing steps to obtain a multimodal image dataset. For example, segmenting multimodal medical images requires acquiring multimodal medical images from hospital imaging systems, resulting in multimodal medical images of multiple patients. Each patient's multimodal image undergoes preprocessing operations such as windowing, resampling, data labeling, and cropping to ultimately obtain the multimodal medical image dataset. Segmenting satellite remote sensing multimodal images requires acquiring multimodal satellite remote sensing images from terminal equipment or ground receiving stations. For multimodal natural image segmentation, different sensors or cameras can be used to capture images of the same scene under the same conditions, forming a multimodal image dataset of natural images. Then, based on the potential correlations existing in the multimodal image data, multiple encoders are used to extract features from different modalities. Furthermore, to encourage the model to learn important semantic features of different modalities, this invention designs a feature fusion strategy applied to the bottleneck layer and a skip connection attention mechanism to incentivize the model to select more important feature information for image segmentation. Finally, a self-supervised learning strategy is used in conjunction with an image reconstruction task to pre-train the network, mitigating the potential problem of insufficient model training due to the limited amount of data collected in the dataset. Specifically, the following steps are included:

[0011] The first step of this invention is to design as follows: Figure 1The image segmentation model shown;

[0012] The image segmentation model described above uses images of the same object in different modalities as input, denoted as X. i , where i = {1, 2, 3}. X i The latent feature representations of each modality are obtained through mutually independent encoding paths, denoted as f. i Where i = {1, 2, 3}. Latent feature representations from different modalities are processed by a fusion module to obtain complementary feature information from different modules. The fused feature information is then upsampled and passed to the decoder for decoding. Feature maps at various levels along different encoding paths are passed to the corresponding levels of the encoder via a skip connection attention mechanism for skip connections. The encoder ultimately outputs the segmentation region prediction result.

[0013] Meanwhile, this invention provides X with... i Adding noise yields noise X′ i , where i = {1, 2, 3}. X′ i After extracting features through an auxiliary coding path that is structurally identical to multiple coding ends and shares parameters, the latent feature representation of each modal noise input is obtained, denoted as f′. i , where i = {1, 2, 3}. f i and f′ i It is used in the self-supervised training process of the model.

[0014] The second step is to design a U-shaped network model.

[0015] like Figure 2 The diagram shows the detailed structure of the model. The model has a U-shaped structure and accepts images of the same object in different modalities as input. Each modal input undergoes feature extraction through an independent encoding path. Each modal input x... i The corresponding modal feature f i Before entering the decoder, the image undergoes bottleneck-level feature fusion through a Fusion and Skip Attention fusion module, ultimately outputting a target region segmentation prediction map at the decoder. The model employs Fusion and Skip Attention at different levels of feature fusion to highlight regions relevant to the segmented content in different modalities.

[0016] Furthermore, the encoding and decoding paths in the second step are as follows: each encoding path has four layers, each consisting of two consecutive residual convolution operations and one 2*2 max downsampling operation. The convolution kernel size is 3*3, and the activation function after convolution is the ReLU function. The number of convolution kernels starts at 64 and increases exponentially with each layer, with the last layer having 1024 kernels. The decoding end also has four layers. The input of each layer consists of the feature map obtained by upsampling from the previous layer and the feature map of the corresponding layer in the encoding end, which are then connected by skip connection attention. After two consecutive convolution operations, a 2*2 upsampling operation is performed. The convolution kernel size, number of kernels, and activation function are the same as the corresponding layer in the encoding end.

[0017] Furthermore, the Fusion and Skip Attention fusion modules of this invention will be described in detail below:

[0018] 1) Fusion: The correlation and strength between different modalities vary, so this invention uses both channel attention and spatial attention in the Fusion module to fuse features from different modalities. Features f from different modalities i Before fusion, they are concatenated into F. Along the channel attention path, this is considered as F = [F1, F2, ..., F...]. C ], where F i ∈R H*W ,i∈{1,2,...C}. Then, a convolutional compression F with a kernel size of H×W is applied. i Obtain the channel dependency g∈R C×1×1 Then, two fully connected layers are used to encode the channel dependencies g, and channel attention is obtained after passing through an activation function. With F M After multiplication, we obtain the channel attention-weighted feature F. c The entire process is given by formulas 1 to 2.

[0019]

[0020]

[0021] Where δ(·) and σ(·) represent the ReLU and Sigmoid activation functions, respectively. W represents the convolution operation. cq ∈R 1 ×H×W For convolution kernel weights, The weights are the weights of the two fully connected layers.

[0022] In the spatial attention path, it can be viewed as F = [F 1,1,F 1,2 ,…,F i,j ,F H,W ], where F i,j ∈R C×1×1 This represents the features across all channels at position (i,j). First, a 1×1 convolution is used to compress each position to obtain the spatial dependency q∈R. 1×C×W Then, a 3x3 convolution is used for feature calibration, which is then activated by the Sigmoid function and multiplied by F to obtain the spatial attention-weighted feature F. s The entire process is given by public parameter 3. Where W... 1*1 and W 3*3 δ(·) and σ(·) represent the 1*1 convolution kernel and 3*3 convolution weights, respectively, and represent the ReLU and Sigmoid activation functions, respectively. This represents the convolution operation.

[0023]

[0024] Channel weighted feature map F c and spatially weighted special graph F s The features are added together and fused to obtain the attention-weighted fused feature representation F of the input feature map. f As shown in Equation 4, this module encourages the model to learn more useful representations. It is also applicable to various multimodal fusion problems and can still fuse the remaining modes even when modes are missing.

[0025] F f =F s +F c (4)

[0026] 2) Skip Attention: The Skip Attention module is used before skip connections are made at each layer of the encoding and decoding paths. It utilizes modal features from the corresponding layers of each decoding path. The feature map x∈R is obtained by performing element-wise summation, finding the maximum value of each element, and calculating the mean of each element. 3C×H×W The relevant operations are shown in Formula 5.

[0027]

[0028] After x passes through the residual convolution module, the number of channels decreases. The output of the residual convolution block is compared with the gated attention g obtained using the features g from the previous layer in the decoding path. att Recalibrate, then use the resulting data and the upsampled feature map of g to perform a skip connection, and gate the attention g. att The calculation formula is shown in Formula 6.

[0029]

[0030] Among them, W 1*1 It refers to C x The convolution kernel weights of each 1x1 convolution, where W 3*3 The weights represent the kernel weights of a 3x3 convolution, and δ(·) and σ(·) represent the ReLU and Sigmoid activation functions, respectively. Resample() represents the convolution operation, and Resample() represents the upsampling operation.

[0031] After being weighted by Skip Attention, low-level features from multiple modalities are filtered out to extract more valuable information, which is then used for skip connections at the decoding end.

[0032] Thirdly, this invention aims to enable the model designed in the second step to learn complementary information from different modalities. Furthermore, in cases where certain modalities are missing, the model should be able to utilize other modalities to obtain the latent semantic features of the missing modalities. Therefore, leveraging the idea of ​​self-supervised learning, a modality latent feature restoration task is designed to be performed simultaneously with the image segmentation task. An auxiliary encoding path with the same structure as the encoding path and shared weight parameters is set up. Specifically:

[0033] The input x of the auxiliary encoding path i By masking the input x of the encoded path i Obtained. x i The feature representation f' is obtained through the auxiliary encoding path. i f′ i and f i This can be viewed as the latent feature representations of the modalities extracted by the network in the cases of modality absence and modality integrity, respectively. Therefore, cosine similarity is used to evaluate f′. i and f i Similarity is used as part of the loss function to encourage the network to learn more useful latent feature representations. The similarity loss function is shown in Equation 7.

[0034]

[0035] in, and These represent the feature representation sets obtained from the encoding path and the auxiliary encoding path, respectively.

[0036] As shown in Equation 8, the final loss function of the model is the sum of the similarity loss and the segmentation loss. Where y pre and y gt These refer to the network's prediction and the true mask when the inputs are X and X′, respectively.

[0037] LOSS = L Similarity +L seg (ypre ,y gt (8)

[0038] In summary, this invention proposes a multimodal image segmentation method based on self-supervised learning. The method first collects data from the images to be segmented, obtaining a multimodal image dataset. Then, based on the potential correlations existing in the multimodal image data, multiple encoders are used to extract features from different modalities. Simultaneously, a feature fusion strategy applied to the bottleneck layer and a skip connection attention mechanism are designed and used to encourage the model to learn important semantic features of different modalities for image segmentation. Finally, the model is pre-trained using a self-supervised learning strategy and an image reconstruction task to alleviate the problem of insufficient model training caused by the small size of the multimodal image dataset.

[0039] The beneficial effects of this invention are as follows:

[0040] This system overcomes the limitations of traditional methods in multimodal image segmentation tasks, maintaining good segmentation performance even with small training datasets or when multimodal images contain missing modalities. This method leverages latent correlations within multimodal images, employing fusion strategies and attention mechanisms to apply important feature information from different modalities and key intermodal correlations to multimodal image segmentation, resulting in more accurate segmentation results. This method not only has broad application prospects in multimodal medical image segmentation but can also be extended and applied to other fields. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the overall structure of the model.

[0042] Figure 2 This is a detailed structural diagram of the model.

[0043] Figure 3 This is the original MRI image.

[0044] Figure 4 This is a sagittal resampled image.

[0045] Figure 5 This is a coronal resampled image.

[0046] Figure 6 This is a normalized image from a sagittal scan.

[0047] Figure 7 This is a normalized image from a coronal scan.

[0048] Figure 8 Image of a soft tissue sarcoma region.

[0049] Figure 9 This is a diagram illustrating the cutting operation. Detailed Implementation

[0050] The present invention will be further described below with reference to specific embodiments and accompanying drawings.

[0051] This invention utilizes a self-supervised learning method to learn feature representations of the latent correlations between different modalities of image data by leveraging the latent correlations present in multimodal image data. These representations are then input into a segmentation network to achieve accurate segmentation of the multimodal images.

[0052] This invention designs a multimodal image semantic segmentation method based on self-supervised learning, which has been validated on a multimodal MRI image dataset of soft tissue sarcoma, but is not limited to the segmentation of multimodal MRI images of soft tissue sarcoma. The following is a specific implementation method on the multimodal MRI image dataset of soft tissue sarcoma.

[0053] We collected coronal T1WI, T2WI, and STIR modal MRI images from 20 patients with soft tissue sarcomas in the thigh and sagittal T1WI, T2WI, and STIR modal images from 30 patients with soft tissue sarcomas in the thigh from the hospital. After a series of preprocessing operations, multiple clinicians jointly labeled the soft tissue sarcoma regions. Then, we cropped and edited MRI images of the same patient at the same shooting angle (COR or AX) to obtain a multimodal image dataset of soft tissue sarcomas in the thigh at COR and AX angles, as shown below. Figure 3 As shown.

[0054] (1) Resampling: Because the imaging equipment used by patients and the parameter settings during imaging are different, there are inconsistencies in the pixel size and granularity of different scanning planes of different modalities of soft tissue sarcoma patients collected in this invention. Such inconsistencies between different modal data and between different scanning planes of the same modality are not conducive to the training of convolutional models. Therefore, a linear interpolation algorithm is first used to resample the collected data so that the pixel size and scanning plane spacing of the coronal and sagittal MRI images in each scanning plane are 1 mm. This solves the inconsistency problem and increases the number of scanning planes of different MRI modal sequences. Figure 4 and Figure 5 The images of a patient in the dataset before and after resampling are shown in the sagittal (original slice thickness 7mm) and coronal (original slice thickness 9mm) scan planes.

[0055] (2) Modal Adaptive Max-Min Normalization: MRI images are significantly affected by imaging parameters, and the signal intensity relationship between the tumor region and the surrounding normal region varies across different modalities. Therefore, MRI images are not suitable for normalization using methods employed in CT images to adjust the contrast between the tumor or lesion region and other normal regions. To address this issue and to highlight the tumor region and enhance the contrast between the tumor region and the surrounding normal region, this invention designs a modal adaptive max-min normalization algorithm to normalize MRI images of different patients across various modalities. For each modality of each patient, this invention first finds the maximum signal intensity S in that modal sequence for that patient. max and the minimum signal strength S min Then, all images in the patient's modality sequence were normalized according to Formula 9, so that the range of each pixel point in each scan plane in the modality sequence was limited to [0,1], where S and S′ represent the original signal intensity and the normalized signal intensity at each position of each scan plane, respectively. At the same time, in order to eliminate the influence of extreme values, large signal intensities less than zero and those with a frequency below 0.05 were excluded before the selection of the maximum and minimum signal intensities.

[0056]

[0057] Images of a patient in the dataset, in sagittal and coronal modalities, on a specific scanning plane, before and after the above normalization operation are shown below. Figure 6 and Figure 7 As shown, after the maximum-minimum normalization operation, the overall brightness of the patient's images in each modality on the same scanning plane is improved. The brightness of the normal area surrounding the tumor area is improved in the T1 modality, and the brightness of the tumor area is significantly improved in the T2WI and STIR modalities. At the same time, the contrast between the tumor area and other areas in the T2WI and STIR modalities is also enhanced.

[0058] (3) MRI Image Marking of Soft Tissue Sarcoma: Multiple clinicians comprehensively assessed the sarcoma region based on the patient's various modal image data, and marked it using the smallest rectangle that could completely cover the sarcoma region. The marked bounding box was saved as a mask image, and the same bounding box was used for different modal images of the same patient across different scan planes. The following is an example of a patient's coronal and sagittal scan plane images and their marked mask images: Figure 8 As shown.

[0059] (4) Tumor Region Cropping and Slicing: The default size of each MRI scan plane is 512×512. If each modality of each scan plane is input into the model at the default size, more computational resources are required. Appropriate input image size is beneficial for training the convolutional model. Therefore, this invention statistically analyzed the height and width of all sarcoma region bounding boxes in the dataset. The width and height of most bounding boxes fall within the range of [50, 150]. Therefore, this invention chooses to randomly crop each modality image of each scan plane into a 256×256 image around the sarcoma bounding box during training. The cropped image completely surrounds the region represented by the sarcoma bounding box and does not exceed the original image area. Considering that not every scan plane contains sarcoma images, this invention only takes the modality images corresponding to 64 scan planes for each patient. Finally, during training, each patient's image is cropped to 64×256×256 and input into the model. Figure 9 The results show several possible outcomes of randomly cropping images from a certain sagittal (AX) scan of a patient within the dataset.

[0060] Based on the solution of this invention, the experimental analysis is as follows:

[0061] (1) Experimental data processing and segmentation

[0062] The two datasets were randomly divided into 80% training set and 20% test set. The network of this invention was implemented using PyTorch 1.10 and trained for 100 epochs on an NVIDIA GeForce GTX 3090. The network was optimized using the Adam optimizer with a learning rate of 1e-5. If the validation loss did not improve within 10 epochs, training was terminated early to avoid overfitting.

[0063] (2) Evaluation indicators

[0064] After training the model, it is necessary to test and evaluate the sarcoma segmentation model. In the segmentation data display module, to evaluate the proposed method, this invention uses the following metrics to assess the model's performance: TP: true positive, FP: false positive, TN: true negative, FN: false negative.

[0065] Accuracy (Acc): Determines the performance when correctly classifying classes in the test set, as shown in Equation 10.

[0066]

[0067] Dice Score: Used to evaluate the overlap between the prediction result and the ground truth. The Dice ranges from 0 to 1. The better the prediction result, the larger the Dice value, as shown in Formula 11.

[0068]

[0069] Sensitivity (SENS): It measures the proportion of actual positives that are correctly identified, as shown in Formula 12.

[0070]

[0071] Specificity (SPE): The ability to correctly exclude irrelevant samples, that is, to accurately identify negative samples, which are samples that do not actually contain the disease or do not belong to the classification. It shows the proportion of correctly identified negative samples out of the total number of negative samples. Higher specificity indicates a higher probability of diagnosis. It is illustrated in Equation 13.

[0072]

[0073] The Hausdorff Distance (HF95) is the distance between the predicted boundary and the ground truth value, and it represents the largest segmentation error. The better the prediction, the smaller the HF95 value. Here, d(x,y) represents the Euclidean distance between x and y, as shown in Equation 14.

[0074] HF95=maxd(y,y'),d(y',y) (14)

[0075] (3) Analysis of soft tissue sarcoma results

[0076] In the experiments, ablation experiments were first used to demonstrate the importance of each component of the network, and also to prove the effectiveness and usability of the dataset of this invention. Secondly, the performance of the network of this invention under full-modal input was compared with that of other classic networks. Then, the impact of different modal input combinations on the network segmentation performance was investigated on two datasets, demonstrating that the network can perform well even with missing modalities, and also proving that the network of this invention can mine heterogeneous information between different modalities.

[0077] Table 1 shows the ablation experimental results on the COR dataset.

[0078]

[0079] 1) Ablation Experiments: To demonstrate the effectiveness of the dataset and understand the importance of each component in the network of this invention, ablation experiments were first conducted on the COR dataset. The results are shown in Table 1. Using the original network without the Fusion module, SkipAttention module, and self-supervised learning strategy as the baseline model, it can be seen that with the addition of the Fusion, SkipAttention, and self-supervised learning strategies, the network's performance on both datasets improved. In particular, after adding the self-supervised learning strategy, the network achieved improvements of 17.96% in Dice coefficient and 21.79% in sensitivity compared to the baseline model. This indicates that the Fusion and SkipAttention modules in the network can encourage the network to learn key information in different modalities. It also shows that the self-supervised learning strategy can improve the encoder's feature extraction ability on small datasets, alleviating the problem of insufficient data to some extent.

[0080] 2) Comparison with other methods across all modalities: To compare the performance of different methods, this invention was compared with UNet, UNet-based multi-encoder UNet, ResUNet-based multi-encoder ResUNet, Attention UNet-based multi-encoder Attention UNet, and the proposed method on two datasets. The results are shown in Tables 2 and 3. On the AX dataset, the proposed network improved the Dice coefficient and SENS by 7.41% and 8.09% respectively compared to other best networks. On the COR dataset, the proposed network improved the Dice coefficient and sensitivity by 16.63% and 19.75% respectively compared to other best networks, indicating that the proposed network's segmentation performance on both the AX and COR datasets is significantly better than other networks, and the proposed network also has a lower false negative rate. In terms of accuracy, specificity, and Hough distance, the proposed network's performance on both datasets is comparable to other networks. Furthermore, this invention analyzes that the proposed network's performance on COR is better than its performance on AX, which may be due to the following three reasons:

[0081] The AX dataset has strong correlations between its various modalities, resulting in less heterogeneous information between them. Using a single encoder structure can better handle data with strong modal correlations.

[0082] The COR dataset has a smaller data volume compared to the AX dataset, making the use of a self-supervised learning strategy more effective in improving network performance.

[0083] The AX and COR datasets contain scanned images of soft tissue sarcoma regions from different perspectives. Different sarcomas have different diameters in the AX and COR directions, resulting in different numbers of scanned images of the same sarcoma region in the COR and AX directions. This difference in number affects the training of the network.

[0084] Table 2 shows the comparison results with other models under full-modal input conditions on the AX dataset.

[0085]

[0086] Table 3 shows the comparison results with other models under full-modal input conditions on the COR dataset.

[0087]

[0088] Note ①: "S" in parentheses indicates the use of a single encoding path, and "M" indicates the use of a multi-encoding path.

[0089] Note ②: Bold results indicate the best results among the relevant indicators, and underlined results indicate the second best results.

[0090] 3) Performance under Modality Missing Conditions: Considering that patient image data often contains missing modalities due to various reasons in practical applications, this invention also uses different combinations of patient modality data as input to explore the performance of the proposed method under modality missing conditions. The results are shown in Tables 4 and 5. Overall, the performance of most aspects of the network in this invention improves with the increase of the number of input modalities. This indicates that there is indeed heterogeneous information between different modalities in the dataset of this invention, and the network of this invention can attempt to obtain semantic feature representations of missing modalities through existing modality inputs and use them for sarcoma region segmentation. In the bimodal input combination experiment, different bimodal input combinations result in different network performance, which also proves that there are different potential semantic feature representations in different modalities. However, the correlation between potential semantic feature representations between different modalities is not consistent with the previous conclusions of Pearson coefficients, etc.

[0091] Pearso and other coefficients indicate that the modal correlation of {T1WI,STIR} is relatively strong in the AX dataset, and the modal correlation of {T1WI,T2WI} is also strong. Therefore, the heterogeneous information contained in the T1WI,STIR input combination should be more. However, in the dual-modal input combination of the AX dataset, the performance of networks trained with the {T1WI,STIR} input combination is lower than that of networks trained with the {T1WI,T2WI} input combination in most cases.

[0092] Pearson coefficients indicate that the modal correlation of the COR dataset {T1WI,STIR} is relatively weak, while the modal correlation of {T1WI,T2WI} is strong. Therefore, the heterogeneous information contained in the input combination of T1WI and STIR should be greater. Experiments also show that the network trained with the input combination of {T1WI,STIR} has higher performance in all metrics than the network trained with the input combination of {T1WI,T2WI}.

[0093] The reason why this phenomenon occurs is twofold: firstly, Pearson coefficients can only measure linear and simple nonlinear relationships; secondly, while there are indeed correlations between T1WI, T2WI, and STIR modes, these correlations are not simple linear or nonlinear relationships, but rather complex superlinear relationships. Therefore, the conclusions drawn from using Pearson coefficients to measure these correlations are not entirely accurate. However, this invention can learn and utilize these correlations between modes through its network.

[0094] Table 4. Model performance on the AX dataset under different modal input combinations.

[0095]

[0096] Table 5. Model performance on the COR dataset under different modal input combinations.

[0097]

[0098] Note ①: Each row represents an input combination. T1 and T2 represent T1WI and T2WI respectively. · indicates that the modal data of the corresponding column is in the input combination, and ο indicates that the modal data of the corresponding column is not in the input combination.

[0099] Note ②: Bold results indicate the best results among the relevant indicators, and underlined results indicate the second best results.

[0100] The above-described embodiments are merely illustrative of the implementation methods of the present invention, but should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the protection scope of the present invention.

Claims

1. A multimodal image segmentation method based on self-supervised learning, characterized in that, The multimodal image segmentation method first acquires multimodal images and preprocesses them to obtain a multimodal image dataset; then, the multimodal image dataset is input into an image segmentation model for image segmentation; finally, a self-supervised learning strategy is used in conjunction with an image reconstruction task to pre-train the model. Design an image segmentation model: The image segmentation model uses images of the same object in different modalities as input, denoted as... ,in ; The latent feature representations of each modality are obtained through mutually independent encoding paths, denoted as... ,in Latent feature representations from different modalities The Fusion module acquires complementary feature information from different modules. The fused feature information is then upsampled and passed to the decoder for decoding. The feature maps at each level on different encoding paths are passed to the corresponding level of the decoder for skip connection after passing through the Skip Attention module. The decoder finally outputs the segmentation region prediction result. Fusion and Skip Attention are: 1) Fusion: The Fusion module simultaneously uses channel attention and spatial attention to fuse features from different modalities; features from different modalities It was pieced together before being merged. On the channel attention path, it is regarded as ,in Next, use a convolution kernel size of... Convolutional compression Obtain channel dependencies Then, two fully connected layers are used to handle channel dependencies. Encode the channel and then pass it through an activation function to obtain channel attention. ;and The channel attention-weighted features are obtained after multiplication. The entire process is given by formulas 1 to 2; (1) (2) in, and These represent the ReLU and Sigmoid activation functions, respectively. Represents the convolution operation. For convolution kernel weights, The weights of the two fully connected layers; In the spatial attention path, it is regarded as ,in Representative at Features on all channels at the location; first use a size of The convolution squeezes each location to obtain the spatial dependency. Then use a convolution kernel size of Feature calibration is performed using convolutions, followed by activation via the Sigmoid function and then... Multiplying them together yields the spatial attention-weighted features. The entire process is given by formula 3; where and Represent convolution kernel Convolution weights and These represent the ReLU and Sigmoid activation functions, respectively. Represents the convolution operation; (3) Channel weighted feature map and spatially weighted special maps The features are added together and fused to obtain the attention-weighted fused feature representation of the input feature map. As shown in Equation 4, this module encourages the model to learn more useful representations. It is also applicable to various multimodal fusion problems and can still fuse the remaining modes even when modes are missing. (4) 2) Skip Attention: The Skip Attention module is used before skip connections are made at each layer of the encoding and decoding paths; it handles modal features from the corresponding layers of each decoding path. The feature map is obtained by performing element-wise summation, finding the maximum value of each element, and calculating the mean of each element. The relevant operations are shown in Formula 5; (5) The number of channels is reduced after passing through the residual convolution module. The output of the residual convolution block is similar to that of the layer using features from the previous layer in the decoding path. Obtained gating attention Perform recalibration, using the results and Skip connections are made to upsampled feature maps, and gated attention is applied. The calculation formula is shown in Formula 6; (6) in, It means indivual The convolution kernel weights, where represent The kernel weights of convolution. and These represent the ReLU and Sigmoid activation functions, respectively. Represents the convolution operation. This represents an upsampling operation; After being weighted by Skip Attention, low-level features from multiple modalities are filtered out to extract more valuable information, which is then used for skip connections at the decoding end.

2. The multimodal image segmentation method based on self-supervised learning according to claim 1, characterized in that, To enable the designed image segmentation model to learn complementary information from different modalities, and to obtain the latent semantic features of the missing modalities using other modalities when some modalities are missing, a modality latent feature restoration task is designed to be performed simultaneously with the image segmentation task using a self-supervised learning method. An auxiliary encoding path with the same structure and shared weight parameters as the encoding path is set up; specifically as follows: The input of the auxiliary encoding path By masking the input of the encoded path get; Feature representations are obtained through auxiliary encoding paths. ; and It can be viewed as the latent feature representation of the modality extracted by the image segmentation model in the cases of modality missing and modality complete, respectively. Therefore, cosine similarity is used to evaluate it. and Similarity is used as part of the loss function to encourage the model to learn more useful latent feature representations. The similarity loss function is shown in Equation 7. (7) in, and These represent the feature representation sets obtained from the encoding path and the auxiliary encoding path, respectively; As shown in Equation 8, the final loss function of the model is the sum of the similarity loss and the segmentation loss; where and These refer to the input being and At that time, the model's prediction results and the true mask; (8)。 3. The multimodal image segmentation method based on self-supervised learning according to claim 2, characterized in that, The encoding and decoding paths are as follows: each encoding path has 4 layers, and each layer consists of two consecutive residual convolution operations and one... The maximum downsampling operation consists of a convolution kernel size of [value missing]. The activation function after convolution is ReLU. The number of convolution kernels starts at 64 and increases exponentially with each layer, reaching 1024 kernels in the final layer. The decoding layer also has four layers. Each layer's input consists of a skip connection between the feature map upsampled from the previous layer and the feature map from the corresponding layer of the encoding path, processed by Skipconnection Attention. This is followed by two consecutive convolution operations. The upsampling operation and convolution operation use the same kernel size, number of kernels, and activation function as the corresponding layer in the encoding path.

Images