Multi-modal medical image-oriented cross-domain anomaly detection method

Abstract

The invention discloses a multi-modal medical image-oriented cross-domain anomaly detection method. The method comprises the steps of obtaining a multi-modal medical image data set; performing medical modal cross-domain migration on the pre-trained encoder model based on a comparative learning method; extracting multi-scale features of the normal image sample by using a migration encoder; performing compression fusion on different scale features, and inputting the fused features into a decoder model for reconstruction; a training process loss function and a test process anomaly score are calculated based on the feature reconstruction error. According to the invention, auxiliary detection of lesions in various modes such as CT, MRI, X-ray and the like in clinical imaging diagnosis can be realized.

Description

Technical Field

[0001] This invention belongs to the field of medical image analysis technology, specifically relating to a cross-domain anomaly detection method for multimodal medical images. Background Technology

[0002] Medical imaging examinations, such as computed tomography (CT), magnetic resonance imaging (MRI), and X-ray imaging, can visually display the internal structure of the human body and its lesions from different angles, providing diversified diagnostic information for disease screening and becoming an important auxiliary tool in clinical diagnosis. However, traditional manual diagnostic methods often suffer from inefficiency and strong subjectivity when faced with large amounts of complex medical imaging data. Currently, computer-aided diagnostic technology has shown great potential in improving diagnostic efficiency and identifying early lesions. Compared with supervised learning methods that rely on high-quality labeled data, unsupervised anomaly detection technology has shown significant advantages in terms of cost and robustness. It establishes an anomaly scoring model by learning the distribution characteristics of normal samples, and then identifies abnormal patterns in samples to achieve disease detection, providing a reliable auxiliary tool for clinical diagnosis.

[0003] Currently, the most mainstream unsupervised anomaly detection methods utilize large-scale datasets, such as ImageNet-based pre-trained models, to provide information-rich and highly discriminative image representations, thereby acquiring anomaly information from multi-scale image features. However, anomaly detection methods based on pre-trained models still face some key challenges in the field of medical imaging. On the one hand, due to the semantic gap between natural images and different medical images, the cross-domain transfer capability of frozen encoders is limited, but fine-tuning optimization based on medical images often leads to the degradation of encoder feature diversity, i.e., pattern collapse. On the other hand, the diversity of normal patterns caused by differences in tissue anatomy is easily misjudged as lesion abnormalities during the reconstruction process, while weakening the model's sensitivity to weak abnormal patterns such as early lesions, making it difficult to effectively capture subtle differences between normal and abnormal features. These challenges often make it difficult for models to accurately detect true pathological abnormalities in images, limiting their lesion detection capabilities.

[0004] Therefore, there is an urgent need for a robust and unsupervised cross-domain anomaly detection method to address the detection accuracy issues caused by the weak inter-domain adaptability and insufficient generalization ability of traditional models, thereby enhancing the application value of the method in clinical diagnosis and treatment. Summary of the Invention

[0005] The purpose of this invention is to provide a cross-domain anomaly detection method for multimodal medical imaging, which solves the problems of insufficient robustness and low detection accuracy in existing methods, and realizes auxiliary detection of lesions in multiple modalities such as CT, MRI, and X-ray in clinical imaging diagnosis.

[0006] To achieve the above objectives, the technical solution adopted in this invention is: a cross-domain anomaly detection method for multimodal medical images, comprising: acquiring a multimodal medical image dataset; performing cross-domain transfer of a pre-trained encoder model based on a contrastive learning method; extracting multi-scale features of normal image samples using a transfer encoder; inputting the compressed and fused features of different scales into a decoder model for reconstruction; and calculating the training loss function and the test anomaly score based on the feature reconstruction error.

[0007] The technical solution of the present invention also has the following characteristics: A cross-domain anomaly detection method for multimodal medical images: Step 1, collect medical data from multiple imaging modalities, perform format conversion, preprocessing, and image enhancement operations, and divide the data into training and testing sets according to the data labels; Step 2: Construct a ResNet-50-2 encoder network pre-trained on the large-scale ImageNet dataset; Step 3: Fine-tune and optimize the pre-trained ResNet-50-2 encoder network based on the BYOL self-supervised contrastive learning method to adapt to the data distribution in the medical domain; Step 4: Freeze the encoder parameters after BYOL fine-tuning and training, and input preprocessed normal image samples to extract multi-scale features; Step 5: Generate compact features from the highest-level features extracted by the encoder through a trainable single-class bottleneck embedding module; Step 6: Input the compact features output by the single-class bottleneck embedding module into the de-ResNet-50-2 decoder, and reconstruct the features step by step through spatial upsampling operations to ensure that they match the resolution and number of channels of the feature maps output by each encoder. Step 7: Compare the output features of the decoder and encoder, and calculate the joint loss function for model training; Step 8: Calculate the weighted cosine similarity between the output features of each layer of the encoder-decoder to form a multi-scale difference score, which is then upsampled and merged to obtain a pixel-by-pixel anomaly image.

[0008] Step 1, specifically: Step 1.1: Collect medical image datasets containing multiple modalities including X-ray, MRI, and OCT; Step 1.2: Perform data cleaning to remove noisy data, and uniformly transform the dimensions of each modality data to adapt to the model input requirements; Step 1.3: Preprocess the data, including normalization to unify the numerical range to [0,1], and perform various transformations such as flipping and cropping to augment the data; Step 1.4: Based on the corresponding labels of the data, select the data labeled as normal as the training dataset; select the data labeled as abnormal or specific lesions, and merge the normal data that was not selected as the training dataset to form a test dataset.

[0009] In step 2, the pre-trained ResNet-50-2 encoder network contains three encoder layers, each with an output size equal to the original size. Figure 1 Feature maps of sizes 4, 1 / 8, and 1 / 16, with 256 / 512 / 1024 channels respectively.

[0010] Step 3 specifically involves: Step 3.1: Construct the online network, which includes a pre-trained encoder, a projector, and a predictor. The projector consists of a two-layer MLP network with a hidden layer dimension of 2048 and an output layer dimension of 256. The predictor also consists of a two-layer MLP network with the same structure as the projector and is connected after the projector in the online network. Step 3.2: Construct the target network, which consists of a pre-trained encoder and a projector, with the same structure as the online network. Step 3.3, the loss function used for training the BYOL framework is based on the normalized mean square error between the output of the online network predictor and the output of the target network projector; Step 3.4, Set the target attenuation rate The parameter is 0.996. After each training epoch, a slow update of the target network is performed using EMA. The expression for the parameter update is:

[0011] in, θ and ξ These represent the parameters of the online network and the target network, respectively.

[0012] Step 5 specifically involves: Step 5.1: Construct a multi-scale feature fusion block, which combines features from the three encoder layers at different scales. The layers are fused using 2 / 1 / 0 3×3 convolutional layers of uniform size and through channel splicing operations. Step 5.2: Construct a single-class embedding block, which contains a residual connection block, consisting of 1×1, 3×3, and 1×1 convolutional layers and residual connections, to perform depth reduction on the fused features output by the multi-scale feature fusion module; Step 5.3: Merge the multi-scale feature fusion block and the single-class embedding block into an OCBE module. Compress the multi-scale features output by the encoder through the OCBE module and use them as the input of the decoder.

[0013] In step 6, the de-ResNet-50-2 decoder network contains three decoder layers, each with an output size equal to the original size. Figure 1 Feature maps of sizes 4, 1 / 8, and 1 / 16, with 256 / 512 / 1024 channels respectively.

[0014] Step 7 specifically includes: Step 7.1: Construct the tolerance attention module, which includes two sub-modules: channel tolerance attention and spatial tolerance attention. The tolerance attention module extracts channel features through global average pooling and max pooling, respectively. Then, it uses 1x1 convolution to process the pooling results for dimensionality reduction and dimensionality increase. The features extracted by the two pooling operations are added and fused together, and the output channel weights are activated by the Sigmoid function. The spatial tolerance attention module generates spatial feature maps by calculating the average and maximum values ​​in the channel dimension. The features are then concatenated and spatial weights are generated by 7x7 convolution. Step 7.2, the feature reconstruction loss is calculated by taking the cosine similarity between the features generated by the encoder and decoder, and then weighting it using tolerant attention. The expression is:

[0015] in, N For the number of feature layers, and They correspond to the first i Feature vectors of layer encoder and decoder and To tolerate the channel and spatial tolerance weights generated by the attention module, Represents the L2 norm; Step 7.3: Construct the tolerant attention regularization loss, which is calculated using the two weights output by the tolerant attention module. The expression is as follows:

[0016] Step 7.4: Construct a joint loss function, which is equal to the sum of the feature reconstruction loss and the tolerance attention regularization loss.

[0017] Step 8 specifically includes: Step 8.1: Construct an anomaly score calculation method based on cosine similarity. First, upsample the feature images at different scales to restore them to their original size. Then, calculate the anomaly score pixel by pixel. The specific calculation expression is as follows:

[0018]

[0019] in, Represents pixels i Abnormal scores, Encoder features With decoder features dot product, They are vectors The L2 norm; Step 8.2: Return the anomaly score pixel by pixel to form a complete anomaly image, and perform Gaussian filtering on the anomaly image with a standard deviation of 4 to reduce noise and smooth edges of the anomaly image and enhance the continuity of the anomaly region.

[0020] The beneficial effects of this invention are: (1) The present invention provides a cross-domain anomaly detection method for multimodal medical images. Unlike the existing technology that relies on the general representation capabilities provided by the ImageNet pre-trained model to model images, the present invention designs the BYOL contrastive learning cross-domain transfer framework, uses the interactive contrastive learning strategy to construct an online network + target network dual-path model, applies robust constraints to the fine-tuning process of the pre-trained model, can perform medical domain adaptive optimization of the model, and effectively avoids the mode collapse problem.

[0021] (2) The present invention provides a cross-domain anomaly detection method for multimodal medical images. Unlike existing technologies that use simple cosine similarity calculations on multi-scale features output by the encoder and decoder as a means of reconstructing error, this invention designs a reconstructing error calculation strategy that tolerates attention module weighting. This strategy performs region activation and weighted filtering in both spatial and channel dimensions, and applies selective smoothing constraints to the reconstruction process. By integrating tolerant attention into the loss function and anomaly score calculation process, this invention significantly enhances the model's ability to distinguish real anomalies and guides the reconstruction results to focus on lesion anomalies, thereby improving the accuracy and robustness of the model's lesion detection capabilities. Attached Figure Description

[0022] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a flowchart illustrating the overall process of a cross-domain anomaly detection method for multimodal medical images, as provided in an embodiment of the present invention. Figure 2 This is a structural diagram of a multi-level encoding-decoding anomaly detection framework provided in an embodiment of the present invention; Figure 3 This is a structural diagram of the BYOL encoder self-supervised fine-tuning module provided in an embodiment of the present invention; Figure 4 This is a structural diagram of the tolerance attention module provided in an embodiment of the present invention. Detailed Implementation

[0023] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0024] Example 1 like Figure 1 As shown, the present invention provides a cross-domain anomaly detection method for multimodal medical images, which is implemented according to the following steps: Step 1: Collect and organize medical image data of various modalities, perform preprocessing on the data including image enhancement, standardization and format conversion, and divide the data into training set and test set according to image labels; Step 2: Construct a ResNet-50-2 encoder network pre-trained on the ImageNet large-scale dataset as the cornerstone model for image feature extraction.

[0025] Figure 2 This is a diagram of the overall architecture of the encoder network in this embodiment of the invention, including convolutional layers, pooling layers, and bottleneck layers. The bottleneck block, composed of three bottleneck layers, is the main feature extraction module, consisting of residual connection structures, and outputs layers with dimensions equal to the original... Figure 1 Feature maps of sizes 4, 1 / 8, and 1 / 16, with 256 / 512 / 1024 channels; Step 3: Based on the Bootstrap Your Own Latent (BYOL) self-supervised contrastive learning method, fine-tune and optimize the pre-trained ResNet-50-2 encoder network to adapt to the data distribution in the medical domain. Figure 3 This is a schematic diagram of the BYOL encoder self-supervised fine-tuning module in an embodiment of the present invention; Step 4: Freeze the encoder parameters after BYOL fine-tuning and training, and input preprocessed normal image samples to extract multi-scale features; Step 5: The highest-level features extracted by the encoder are used to generate compact features through a trainable single-class bottleneck embedding (OCBE) module to preserve the core information of normal samples and compress the multi-scale pattern into a low-dimensional space. Figure 4 This is a schematic diagram of the OCBE module structure in an embodiment of the present invention; Step 6: Input the compact features output by the OCBE module into the de-ResNet-50-2 decoder as the inverse model of the encoder. This decoder uses transposed convolution as its basic structure and consists of three corresponding decoder layers. The decoder spatially upsamples the OCBE output features, reconstructs multi-scale features that are progressively aligned with each level of the encoder, and ensures that the resolution and number of channels of the feature map correspond to each other. Step 7: Compare the output features of the decoder and encoder, and calculate the joint loss function for model training. The joint loss function consists of feature reconstruction loss and tolerance attention (TA) regularization constraint loss. It aims to encourage the student decoder to accurately reconstruct the multi-scale deep features extracted by the teacher encoder by constraining the directional consistency between the encoder and decoder features, thereby learning the feature distribution of normal samples. Figure 4 A structural diagram of the tolerance attention module provided in an embodiment of the present invention; Step 8: Calculate the weighted cosine similarity between the output features of each layer of the encoder-decoder to form a multi-scale difference score, and obtain a pixel-by-pixel anomaly image after upsampling and merging.

[0026] Therefore, the cross-domain anomaly detection method for multimodal medical images proposed in this invention differs from existing technologies that rely on the general representation capabilities provided by ImageNet pre-trained models to model images. This invention designs a BYOL contrastive learning cross-domain transfer framework, which uses an interactive contrastive learning strategy to construct an online network + target network dual-path model. It applies robust constraints to the fine-tuning process of the pre-trained model, enabling medical domain adaptive optimization of the model while effectively avoiding mode collapse problems.

[0027] Furthermore, this invention provides a cross-domain anomaly detection method for multimodal medical images. Unlike existing technologies that rely on simple cosine similarity calculations on multi-scale features output by the encoder and decoder to calculate reconstruction errors, this invention designs a tolerant attention-based weighted reconstruction error calculation strategy. This strategy performs region activation and weighted filtering in both spatial and channel dimensions, and applies selective smoothing constraints to the reconstruction process. By integrating tolerant attention into the loss function and anomaly score calculation process, this invention significantly enhances the model's ability to distinguish real anomalies and guides the reconstruction results to focus on lesion anomalies, thereby improving the accuracy and robustness of the model's lesion detection capabilities.

[0028] Example 2 Unlike Example 1, in Example 2, a cross-domain anomaly detection method for multimodal medical images according to the present invention, step 1 specifically includes: Step 1.1: Collect medical image datasets of various modalities, including the RSNA dataset (X-ray modality), the BR35H and Brain Tumor datasets (MRI modality), the OCT2017 dataset (Optical Coherence Tomography (OCT) modality), and the APTOS dataset (RGB color images) and the ISIC dataset (dermatology). Step 1.2: Perform data cleaning, transform all data dimensions into a four-dimensional format of data quantity / 3 channels / image height / image width, normalize the data to the [0,1] interval, and perform a series of enhancement operations on the data, including random rotation between -5° and +5°, brightness change within a range of ±10%, random cropping of the area to 90%-100% of the original image with an aspect ratio range of 0.95-1.05, and flipping with a probability of 0.1. Then convert the data to the standard input format NPY. Step 1.3: Based on the data labels, a portion of the samples labeled as normal are selected as the training set for model training. For example, the OCT2017 dataset contains 26,565 normal images, and 26,315 of them are selected as the training dataset; the APTOS dataset contains 1,805 normal images, and 1,000 of them are selected as the training dataset. The mixed data of normal and abnormal samples is divided into a test set to test the model performance. For example, for the OCT2017 dataset, the remaining 250 normal images are selected and combined with 250 choroidal neovascularization (CNV) samples, 250 diabetic macular edema (DME) samples, and drusen samples to form the test data; for the APTOS dataset, the remaining 805 normal images and 1,857 diabetic retinopathy images are combined to form the test dataset.

[0029] Example 3 Unlike Example 2, in Example 3, a cross-domain anomaly detection method for multimodal medical images according to the present invention, step 3 specifically includes: Step 3.1: Construct a two-branch BYOL framework, which mainly includes two network structures: an online network and a target network; Step 3.2: Construct the online network, which includes the pre-trained encoder described in Step 2, as well as a projector and a predictor. The projector consists of a two-layer multilayer perceptron (MLP) network with hidden layers (2048 dimensions) and an output layer (256 dimensions). Its function is to map the features of the encoder to a low-dimensional latent space, enhancing the abstractness and discriminativeness of the features. The predictor also consists of two MLP layers with the same structure as the projector, located after the projector in the online network. Its core function is to further constrain the online network to learn consistent feature representations by predicting the output of the target network projector. The network weights of the online network encoder, projector, and predictor are expressed as follows: f θ 、g θ and q θ ; Step 3.3: Construct the target network, which consists of a pre-trained encoder and a projector. The network weights of the encoder and projector are expressed as follows: f ξ and g ξ ; Step 3.4: Construct a loss function for training the BYOL framework. This loss is based on the normalized mean square error between the output of the online network predictor and the output of the target network projector, and is expressed as:

[0030] in, It is the predicted output of the online network. It is the projection output of the target network. It is a vector dot product. Represents the L2 norm; Step 3.5: Train BYOL's dual-branch network, inputting two random augmented views of the same image. v and v′ The online network and the target network output respectively and The loss function is calculated and the parameters of the online network are updated using backpropagation, while a given target decay rate is applied. In this embodiment, it is set to 0.996. After each training epoch, the target network is slowly updated using EMA, as shown below:

[0031] in, θ and ξ These represent the parameters of the online network and the target network, respectively.

[0032] Example 4 Unlike Example 3, in Example 4, a cross-domain anomaly detection method for multimodal medical images according to the present invention, step 5 specifically includes: Step 5.1: Construct a multi-scale feature fusion (MFF) block, which combines the output features of the three encoder layers described in Step 2. Different numbers of 3×3 convolutional layers are input to achieve feature map size matching, and then fused through channel concatenation. Step 5.2: Construct a single-class embedding (OCE) block, based on a standard ResNet block (1×1, 3×3, 1×1 convolutional layers and residual connections), and perform deep dimensionality reduction on the fused features output by the MFF module through trainable convolutional layers; Step 5.3: Merge the MFF and OCE blocks into an OCBE module. Compress the multi-scale features described in Step 4 through the OCBE module and use them as input to the decoder network.

[0033] Example 5 Unlike Example 4, in Example 5, a cross-domain anomaly detection method for multimodal medical images according to the present invention, step 7 specifically includes: Step 7.1: Construct the Tolerant Attention (TA) module. By calculating the weight tensors of the decoder output features in the spatial and channel dimensions, the model focuses on anomalous patterns in the features. The TA module includes a Channel Tolerant Attention (CTA) module and a Spatial Tolerant Attention (STA) sub-module. CTA extracts channel features and generates channel weights through global average pooling and max pooling. cw The pooling results are processed using 1x1 convolutions for dimensionality reduction and expansion, and then the output is activated using a sigmoid function. STA generates spatial feature maps by averaging and maximizing the channel dimensions, and after concatenation, spatial weights are generated using 7x7 convolutions. sw The TA module outputs respectively. cw and sw The two are then multiplied and combined into a weight tensor. tw This is used for the weighted calculation of subsequent feature maps and reconstruction errors; Step 7.2: Construct the feature reconstruction loss, which mainly consists of the cosine similarity between the features generated by the encoder and decoder after TA weighting, expressed as:

[0034] Where N is the number of feature layers, and They correspond to the first i Feature vectors of layer encoder and decoder and The channel and space tolerance weights generated for the TA module Represents the L2 norm; Step 7.3: Construct the TA regularization loss, which suppresses high reconstruction errors caused by spurious anomaly regions based on attention weights. Its expression is:

[0035] Step 7.4: Construct the joint loss function, which includes feature reconstruction loss and TA regularization loss, expressed as: .

[0036] Example 6 Unlike Example 5, in Example 6, a cross-domain anomaly detection method for multimodal medical images according to the present invention, step 8 specifically includes: Step 8.1: Construct an anomaly score calculation method based on cosine similarity. First, upsample the feature images at different scales to restore them to their original size. Then, calculate the anomaly score pixel by pixel. The specific calculation expression is as follows:

[0037]

[0038] in, Represents pixels i Abnormal scores, Encoder features With decoder features dot product, They are vectors The L2 norm; Step 8.2: Return the anomaly score pixel by pixel to form a complete anomaly image, and perform Gaussian filtering with a standard deviation of 4 on the anomaly image to reduce noise and smooth the edges, thereby enhancing the continuity of the anomaly region.

[0039] To verify the effectiveness of the proposed cross-domain anomaly detection method for multimodal medical images, the data collected in the above steps were processed, modeled, trained, and tested. Several sets of comparative experimental results with other existing models were obtained, using five metrics: area under the curve (AUC), accuracy, specificity, recall, and F1 score. Experimental results on some datasets are shown in the table below. The experimental results show that the method of this invention is significantly better than other control methods in all five core indicators, which fully verifies the accuracy and robustness of the proposed cross-domain anomaly detection method for multimodal medical images in cross-modal medical image datasets.

[0040] ISIC

[0041] OCT

[0042] Br35H

[0043] The foregoing description illustrates and describes several preferred embodiments of the invention. However, as previously stated, it should be understood that the invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the inventive concept described herein through the foregoing teachings or techniques or knowledge in related fields. Any modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the invention should be within the protection scope of the appended claims.

Claims

1. A method for cross-domain anomaly detection in multimodal medical imaging, characterized in that, include: Acquire a multimodal medical image dataset; perform cross-domain transfer of medical modalities on a pre-trained encoder model based on a contrastive learning method; A transfer encoder was used to extract multi-scale features from normal image samples. Features at different scales are compressed and fused before being input into the decoder model for reconstruction. The training loss function and the test anomaly score are calculated based on the feature reconstruction error.

2. The cross-domain anomaly detection method for multimodal medical images according to claim 1 is implemented specifically according to the following steps: Step 1: Collect medical data from multiple imaging modalities, perform format conversion, preprocessing, and image enhancement operations, and divide the data into training and testing sets according to the data labels; Step 2: Construct a ResNet-50-2 encoder network pre-trained on the large-scale ImageNet dataset; Step 3: Fine-tune and optimize the pre-trained ResNet-50-2 encoder network based on the BYOL self-supervised contrastive learning method to adapt to the data distribution in the medical domain; Step 4: Freeze the encoder parameters after BYOL fine-tuning and training, and input preprocessed normal image samples to extract multi-scale features; Step 5: Generate compact features from the highest-level features extracted by the encoder through a trainable single-class bottleneck embedding module; Step 6: Input the compact features output by the single-class bottleneck embedding module into the de-ResNet-50-2 decoder, and reconstruct the features step by step through spatial upsampling operations to ensure that they match the resolution and number of channels of the feature maps output by each encoder. Step 7: Compare the output features of the decoder and encoder, and calculate the joint loss function for model training; Step 8: Calculate the weighted cosine similarity between the output features of each layer of the encoder-decoder to form a multi-scale difference score, which is then upsampled and merged to obtain a pixel-by-pixel anomaly image.

3. The cross-domain anomaly detection method for multimodal medical images according to claim 2, characterized in that, Step 1 specifically includes: Step 1.1: Collect medical image datasets containing multiple modalities including X-ray, MRI, and OCT; Step 1.2: Perform data cleaning to remove noisy data, and uniformly transform the dimensions of each modality data to adapt to the model input requirements; Step 1.3: Preprocess the data, including normalization to unify the numerical range to [0,1], and perform various transformations such as flipping and cropping to augment the data; Step 1.4: Select the data with normal labels as the training dataset based on the corresponding labels of the data. Data labeled as abnormal or specific lesions were selected, and normal data that were not selected for the training set were merged together to form the test dataset.

4. The cross-domain anomaly detection method for multimodal medical images according to claim 3, characterized in that, In step 2, the pre-trained ResNet-50-2 encoder network contains three encoder layers, which output feature maps with sizes of 1 / 4, 1 / 8, and 1 / 16 of the original image, and with 256, 512, and 1024 channels, respectively.

5. The cross-domain anomaly detection method for multimodal medical images according to claim 4, characterized in that, Step 3 specifically involves: Step 3.1: Construct an online network, which includes a pre-trained encoder, a projector, and a predictor; the projector consists of a two-layer MLP network with a hidden layer dimension of 2048 and an output layer dimension of 256; the predictor also consists of a two-layer MLP network with the same structure as the projector and is connected after the projector in the online network. Step 3.2: Construct the target network, which consists of a pre-trained encoder and a projector, with the same structure as the online network. Step 3.3, the loss function used for training the BYOL framework is based on the normalized mean square error between the output of the online network predictor and the output of the target network projector; Step 3.4, Set the target attenuation rate The parameter is 0.

996. After each training epoch, a slow update of the target network is performed using EMA. The expression for the parameter update is: in, θ and ξ These represent the parameters of the online network and the target network, respectively.

6. The cross-domain anomaly detection method for multimodal medical images according to claim 5, characterized in that, Step 5 specifically involves: Step 5.1: Construct a multi-scale feature fusion block, which combines features from the three encoder layers at different scales. The layers are fused using 2 / 1 / 0 3×3 convolutional layers of uniform size and through channel splicing operations. Step 5.2: Construct a single-class embedding block, which contains a residual connection block, consisting of 1×1, 3×3, and 1×1 convolutional layers and residual connections, to perform depth reduction on the fused features output by the multi-scale feature fusion module; Step 5.3: Merge the multi-scale feature fusion block and the single-class embedding block into an OCBE module. Compress the multi-scale features output by the encoder through the OCBE module and use them as the input of the decoder.

7. The cross-domain anomaly detection method for multimodal medical images according to claim 6, characterized in that, In step 6, the de-ResNet-50-2 decoder network contains three decoder layers, which output feature maps with sizes of 1 / 4, 1 / 8, and 1 / 16 of the original image, and with 256, 512, and 1024 channels, respectively.

8. The cross-domain anomaly detection method for multimodal medical images according to claim 7, characterized in that, Step 7 specifically involves: Step 7.1: Construct the tolerance attention module, which includes two sub-modules: channel tolerance attention and spatial tolerance attention. The tolerance attention module extracts channel features through global average pooling and max pooling, respectively. Then, it uses 1x1 convolution to process the pooling results for dimensionality reduction and dimensionality increase. The features extracted by the two pooling operations are added and fused together, and the output channel weights are activated by the Sigmoid function. The spatial tolerance attention module generates spatial feature maps by calculating the average and maximum values ​​in the channel dimension. The features are then concatenated and spatial weights are generated by 7x7 convolution. Step 7.2, the feature reconstruction loss is calculated by taking the cosine similarity between the features generated by the encoder and decoder, and then weighting it using tolerant attention. The expression is: in, N For the number of feature layers, and They correspond to the first i Feature vectors of layer encoder and decoder and To tolerate the channel and spatial tolerance weights generated by the attention module, Represents the L2 norm; Step 7.3: Construct the tolerant attention regularization loss, which is calculated using the two weights output by the tolerant attention module. The expression is as follows: Step 7.4: Construct a joint loss function, which is equal to the sum of the feature reconstruction loss and the tolerance attention regularization loss.

9. The cross-domain anomaly detection method for multimodal medical images according to claim 8, characterized in that, Step 8 specifically involves: Step 8.1: Construct an anomaly score calculation method based on cosine similarity. First, upsample the feature images at different scales to restore them to their original size. Then, calculate the anomaly score pixel by pixel. The specific calculation expression is as follows: in, Represents pixels i Abnormal scores, Encoder features With decoder features dot product, They are vectors The L2 norm; Step 8.2: Return the anomaly score pixel by pixel to form a complete anomaly image, and perform Gaussian filtering on the anomaly image with a standard deviation of 4 to reduce noise and smooth edges of the anomaly image and enhance the continuity of the anomaly region.

Images