Collaborative data model adaptation-based adaptive medical image segmentation method during test

Abstract

The invention discloses a collaborative data model adaptation-based adaptive medical image segmentation method during testing, and aims to solve the problem that the image segmentation performance of an existing segmentation method needs to be improved. According to the technical scheme, a collaborative data model adaptation-based self-adaptive medical image segmentation system during testing is firstly constructed, a prompt update model is arranged in a prompt update module of the segmentation system, an image segmentation module is composed of image segmentation models, and batch normalization layers are arranged in the two models; a source domain model and a medical image are adopted to test a segmentation system, a batch normalization layer is used as a bidirectional bridge to realize collaborative self-adaption of data adaption and model adaption, and low-level distribution alignment and high-level semantic feature adaption are realized in a Fourier space; and segmenting the medical image by using the adapted segmentation system. By adopting the method, error accumulation and disastrous forgetting can be avoided, data adaptation and model adaptation are coordinated, the image segmentation effect can be improved, and the Dice coefficient is improved.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and medical image processing, and specifically to a test-time adaptive medical image segmentation method based on collaborative data model adaptation. Background Technology

[0002] Medical image segmentation aims to automatically locate and identify lesion regions, improving diagnostic accuracy and efficiency while reducing human error. With the successful application of deep learning technology, the accuracy of medical image segmentation has significantly improved. However, deep networks are highly sensitive to domain shift (the phenomenon where the model's distribution differs between the training data (source domain) and the actual application data (target domain), which is extremely common in medical image scenarios. Medical images are susceptible to factors such as scanner type, imaging parameters, and fluctuations in signal-to-noise ratio over time, leading to significant domain shifts and consequently significantly reducing the accuracy of segmentation models in practical applications. This has spurred research into unsupervised domain adaptation (UDA) methods for medical image segmentation. This method uses source and target domain data for model training. The source domain represents a region different from the sample to be segmented and generally has rich labels as supervision information; the target domain represents the region where the sample to be segmented resides and often lacks label information. The goal of unsupervised domain adaptation is to reduce the impact of domain shift and maintain model segmentation performance in the target domain. However, in clinical practice, the timeliness of diagnosis and treatment often requires immediate predictions on a single test sample or a small batch of data, making it impossible to wait for a large number of test images from the same target domain, thus limiting the practicality of UDA medical image segmentation methods. Furthermore, due to privacy concerns regarding medical data, access to source domain training data is often not feasible during the testing phase.

[0003] Therefore, Test-Time Adaptation (TTA) medical image segmentation becomes a more suitable solution in such scenarios. Test-time adaptation refers to the model dynamically adjusting its parameters or strategies based on the input data during testing to address domain shifts between training and test data. Its core feature is real-time self-optimization during model usage, rather than traditional pre-training or fine-tuning. TTA relies solely on streaming unlabeled target domain data, enabling the model to continuously adapt to the ever-changing data distribution in the clinical environment.

[0004] A common implementation of test-time adaptation in medical image segmentation is the model-adaptive segmentation method, which continuously updates the weights of the source model using continuously arriving target domain data, thereby adapting the model to the new data. However, due to the lack of supervised information, most TTA methods employ a self-supervised strategy. The paper "Dequan Wang, et al.Tent:Fullytest-time adaptation by entropy minimization[J].arXiv preprint arXiv:2006.10726,2020.Dequan Wang et al.'s paper:Tent:A Fully Test-Time Adaptive Method Based on Entropy Minimization" (TENT) aims to achieve adaptation during the testing phase by minimizing the entropy value predicted by the model. To minimize entropy, TENT normalizes and transforms the inference process of the target data by estimating statistics and optimizing affine parameters, updating them in batches. The paper "Qin Wang, et al. Continual test-time domain adaptation [C]. Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2022:7201-7211" by Qin Wang et al., improves pseudo-label quality within a self-training framework through a weighted average teacher model and augmented average prediction, and proposes randomly restoring some neurons in the network to the pre-trained source model state. However, such methods are susceptible to noisy supervision, and errors may accumulate over time. Furthermore, the updated model is prone to catastrophic forgetting during continuous exposure to new domain data, leading to a significant decline in prediction performance. To alleviate these problems, data adaptation segmentation methods have emerged in recent years.

[0005] Unlike model-adaptive segmentation methods, data-adaptive segmentation methods in test-time adaptation do not update model weights. Instead, they transform the distribution of target domain data to make it closer to the source domain. The paper "Ziyang Chen, et al. Each test image deserves a specific prompt: Continual test-time adaptation for 2d medical image segmentation [C]. Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2024:11184-11193. Ziyang Chen et al.'s paper: Each test image deserves a specific prompt: Continual test-time adaptation for 2D medical image segmentation" (VPTTA) avoids error accumulation and catastrophic forgetting by freezing the parameters of the source domain segmentation model and learning specific visual cues for each test image to achieve adaptation. This cueing mechanism can adapt each test image to the frozen source domain segmentation model, balancing adaptability and knowledge retention. Because the model remains frozen, this type of method avoids problems such as error accumulation and catastrophic forgetting. However, due to the inability to access the source domain data, the transformed target data still struggles to align with the source domain segmentation model in the semantic feature space: even if they appear visually similar, the source domain segmentation model may interpret them in drastically different ways, resulting in suboptimal performance.

[0006] Test-time adaptive medical image segmentation adjusts the model or data during testing to adapt to new data from different hospitals, devices, or imaging protocols, without requiring training samples or annotations of the target domain. Compared to traditional medical image methods that rely on adaptation during the training phase, test-time adaptive segmentation offers greater flexibility and generalization capabilities, making it particularly suitable for scenarios where target domain data is difficult to obtain in clinical settings. The rise of this method is primarily driven by the need for cross-institutional deployment, privacy constraints, and model robustness requirements, and it is becoming a promising research direction in intelligent medical image analysis.

[0007] Therefore, how to improve the performance of adaptive medical image segmentation during testing remains a technical issue of great concern to those skilled in the art. Summary of the Invention

[0008] The technical problem this invention aims to solve is that existing test-time adaptive medical image segmentation methods suffer from various issues, including error accumulation and catastrophic forgetting in model-adaptive segmentation, difficulty in aligning data and models in feature space, difficulty in combining the advantages of data adaptation and model adaptation, and the need to improve image segmentation performance. This invention proposes a test-time adaptive medical image segmentation method based on collaborative data-model adaptation. It utilizes a batch normalization (BN) layer as a bidirectional bridge to achieve a collaborative adaptive process between data adaptation and model adaptation. In data adaptation, domain-aware cues are used to modify the BN layer statistics of the test data, thereby achieving low-level distribution alignment in Fourier space. In model adaptation, a strong-weak enhancement consistency and entropy minimization strategy are combined to dynamically optimize the affine parameters of the BN layer, achieving adaptation of high-level semantic features.

[0009] To address the aforementioned technical problems, the technical solution of this invention is to construct a test-time adaptive medical image segmentation system based on collaborative data model adaptation. This system comprises a Fourier transform module, a memory module, a hint update module, an inverse Fourier transform module, a strong enhancement module, a weak enhancement module, an image segmentation module, a consistency loss function calculation module, an entropy calculation module, and a summation module. The test-time adaptive medical image segmentation system based on collaborative data model adaptation is tested using a user-provided source domain model and user-input medical images. During testing, data adaptation and model adaptation are performed, and then the adapted test-time adaptive medical image segmentation system based on collaborative data model adaptation is used to process and segment the medical images.

[0010] This invention includes the following steps:

[0011] The first step is to construct a test-time adaptive medical image segmentation system based on a collaborative data model. For example... Figure 1 As shown, the test-time adaptive medical image segmentation system based on collaborative data model adaptation consists of a Fourier transform module, a memory module, a hint update module, an inverse Fourier transform module, a strong enhancement module, a weak enhancement module, an image segmentation module, a consistency loss function calculation module, an entropy calculation module, and a summation module.

[0012] The Fourier transform module is connected to the memory bank module, the prompt update module, and the inverse Fourier transform module. During testing, the Fourier transform module receives an image input at the current time, performs a Fast Fourier Transform on the image, and obtains the amplitude and phase. The low-frequency component of the amplitude is sent to the memory bank module, and the amplitude and phase are sent to the prompt update module and the inverse Fourier transform module.

[0013] The memory module is connected to the Fourier transform module and the prompt update module. During testing, the memory module receives low-frequency components from the Fourier transform module, calculates the similarity between the low-frequency components and the existing low-frequency components in the memory, selects the most similar low-frequency components, and obtains an initialized prompt by weighted averaging the prompts corresponding to the most similar low-frequency components. The initialized prompt is then sent to the prompt update module. The module receives the updated prompt from the prompt update module, adds the low-frequency components and the updated prompt as low-frequency component-prompt pairs to the memory, and pops the oldest low-frequency component-prompt pair from the memory, maintaining a fixed number of low-frequency component-prompt pairs in the memory.

[0014] The prompt update module is connected to the Fourier transform module, the memory module, and the inverse Fourier transform module. The prompt update module contains a segmentation model (let's call it the prompt update model), whose parameters are taken from the source domain segmentation model. During testing, the prompt update module receives the initialized prompt from the memory module and the amplitude and phase from the Fourier transform module. The prompt update module pads the initialized prompt with 1s around its edges until it matches the amplitude, and then performs an inverse fast Fourier transform on the initialized prompt along with the phase to obtain the initialized image. This initialized image is then input into the prompt update model within the prompt update module to obtain a loss function based on a batch normalization layer. The loss obtained from this batch normalization layer loss function is backpropagated to the prompt update module to update the prompt, resulting in the updated prompt. The updated prompt is then sent to the inverse Fourier transform module and the memory module. The update model uses the same ResUNet architecture as the source domain segmentation model. This architecture combines ResNet (see the paper "Kaiming He, et al. Deep Residual Learning for Image Recognition[J]. Proceedings of the IEEE conference on computer vision and pattern recognition. 2016:770-778" by Kaiming He et al., "Network Architecture in Deep Residual Learning for Image Recognition") with U-Net (see the paper "Olaf Ronneberger, et al. U-net: Convolutional networks for biomedical image segmentation[C]. Medical image computing and computer-assisted intervention–MICCAI 2015:18th international conference, Munich, Germany, October 5-9, 2015, procedures, part III 18. Springer international publishing, 2015:234-241" by Olaf Ronneberger et al., "Network Architecture in Deep Residual Learning for Image Recognition" by Kaiming He ... The paper by Ronneberger et al.: U-net: A natural evolution of convolutional networks for biomedical image segmentation.ResNet networks are constructed by stacking a series of residual blocks, each consisting of convolutional layers and normalization layers. Depending on the network depth, variants of ResNet (such as ResNet34, ResNet50, ResNet101, etc.) are commonly used in specific applications. ResNet networks have small parameters, short training time, and good feature extraction capabilities. In ResNet, each convolutional layer, except for the output layer, is followed by a normalization layer. U-Net is a convolutional neural network specifically designed for biomedical image segmentation. Its core feature is its unique U-shaped symmetrical structure, comprising an encoder (downsampling) and a decoder (upsampling), fusing features from different levels through skip connections. This allows for the extraction of high-level semantic information while preserving spatial details. This network performs exceptionally well with limited labeled data, making it particularly suitable for medical imaging tasks such as cell segmentation and tumor detection. Its high accuracy and elegant structure have made it a classic model in image segmentation. The original U-Net does not have a normalization layer. ResUNet is an improved segmentation network that combines U-Net and ResNet. By introducing residual modules into the traditional U-Net architecture, it effectively solves the vanishing gradient problem in deep networks, improving feature reuse capabilities and training stability. This network retains the encoder-decoder structure and skip connections of U-Net, while adding cross-layer short-circuit connections in each convolutional block, enabling the network to learn more complex feature representations. It performs exceptionally well in tasks requiring fine boundary prediction, such as medical image segmentation and remote sensing image analysis, and is particularly suitable for segmentation scenarios with small sample sizes and high accuracy. In ResUNet, in addition to the output layer and skip connections, each convolutional layer is followed by a normalization layer.

[0015] The Inverse Fourier Transform (IFT) module is connected to the Fourier Transform (FT) module, the cue update module, the strong enhancement module, the weak enhancement module, and the image segmentation module. During testing, the IFT module receives the updated cue from the cue update module and the amplitude and phase from the Fourier Transform (FT) module. It pads the edges of the updated cue with 1s until it matches the amplitude value. Then, it performs an IFT with the phase value after multiplying the updated cue by the amplitude, obtaining the data-adapted image. This data-adapted image is then sent to the strong enhancement module, the weak enhancement module, and the image segmentation module, respectively.

[0016] The enhanced image enhancement module is connected to the inverse Fourier transform module and the image segmentation module. During testing, the enhanced image enhancement module receives the adapted image from the inverse Fourier transform module, performs enhanced image enhancement on the adapted image to obtain a highly enhanced image, and then sends the highly enhanced image to the image segmentation module.

[0017] The weak enhancement module is connected to the inverse Fourier transform module and the image segmentation module. During testing, the weak enhancement module receives the adapted image from the inverse Fourier transform module and performs different weak enhancement processes on the adapted image, namely, horizontal flipping, vertical flipping, rotation by 90 degrees, rotation by 180 degrees, and rotation by 270 degrees, respectively, to obtain five weakly enhanced images. These five weakly enhanced images are designated as the weakly enhanced image set and sent to the image segmentation module.

[0018] The image segmentation module is connected to the inverse Fourier transform module, strong enhancement module, weak enhancement module, consistency loss function calculation module, entropy calculation module, and summation module. During testing, the image segmentation module receives the strongly enhanced image from the strong enhancement module, segments it to obtain a probability map of the strongly enhanced image, and sends this probability map to the consistency loss function calculation module. The image segmentation module receives the weakly enhanced image set from the weak enhancement module, segments each weakly enhanced image in the set, averages the segmentation results to obtain the average probability map of the weakly enhanced image set, and sends this average probability map to the consistency loss function calculation module. The image segmentation module receives the data-adapted image from the inverse Fourier transform module, segments it to obtain the image's probability map, and sends this probability map to the entropy calculation module. The image segmentation module consists of an image segmentation model, which, like the cue update model, uses the same ResUNet neural network architecture as the source domain model, and its initial parameters are also the same as those of the source domain segmentation model. After the convolutional layer of the image segmentation model, there is a batch normalization layer. The batch normalization layer needs to be adapted by the model to ensure accurate segmentation. After the image segmentation model is adapted, the image segmentation module segments the image after it is adapted by the data received from the inverse Fourier transform module and outputs the final segmentation result.

[0019] The consistency loss function calculation module is connected to the image segmentation module and the summation module. During testing, the consistency loss function calculation module receives the probability map of the strongly enhanced image and the average probability map of the weakly enhanced image set from the image segmentation module. It calculates the consistency loss value based on the probability map of the strongly enhanced image and the average probability map of the weakly enhanced image set, and then sends the consistency loss value to the summation module.

[0020] The entropy calculation module is connected to the image segmentation module and the summation module. During testing, the entropy calculation module receives the probability map of the image from the image segmentation module, calculates the entropy loss value based on the probability map, and sends the entropy loss value to the summation module.

[0021] The summation module is connected to the consistency loss function calculation module, the entropy calculation module, and the image segmentation module. During testing, the summation module receives consistency loss values ​​from the consistency loss function calculation module and entropy loss values ​​from the entropy calculation module. It calculates the weighted sum of the consistency loss value and the entropy loss value to obtain the final loss value, and then backpropagates the final loss value to the image segmentation model of the image segmentation module for model adaptation.

[0022] In the test-time adaptive medical image segmentation system based on collaborative data model adaptation, all modules operate during testing. For each image to be segmented, data adaptation is performed, and simultaneously, the model adapts to the image. The adapted system then segments the image to obtain the segmentation result.

[0023] The second step involves the test-time adaptive medical image segmentation system based on collaborative data model adaptation, constructed in the first step, receiving the user-input image to be segmented. This user-input image is then used to test the system. During testing, the image to be segmented undergoes data adaptation, and the adapted image is then used to adapt the image segmentation model within the system. The method is as follows:

[0024] 2.1 Initialize the model parameters in the test-time adaptive medical image segmentation system based on collaborative data model adaptation. The method is to load the model parameters of the user-provided source domain model (generally, the user will train the source domain model on labeled source domain data (fundus images of a certain institution) using deep learning methods) into the prompt update model in the prompt update module and the image segmentation model in the image segmentation module.

[0025] 2.2 The Adam optimizer was selected as the optimizer for both the cue update model and the image segmentation model. The learning rate for the cue update model was initialized to 0.05, and the learning rate for the image segmentation model was initialized to 1×10⁻⁶. -5 ;

[0026] 2.3 Construct the memory bank in the memory bank module. The memory bank size M is a positive integer, preferably set to 40. The memory bank stores M low-frequency component-hint pairs in the form {(k1,v1),…,(k m ,v m ),…,(k M ,v M )},1≤m≤M,(k m ,v m ) represents the m-th low-frequency component - hint pair, where k m V represents the m-th low-frequency component. m This represents the m-th hint. k m ,vm ∈R (αH)×(αW)×C H, W, and C represent the height, width, and number of channels of the image, respectively. The scaling factor α controls the amplitude range of the cues, causing them to focus on the low-frequency part. α∈(0,1), preferably α=0.01;

[0027] 2.4 (k1, v1), ..., (k m ,v m ),…,(k M ,v M All are initialized to A tensor representing a set of all 1s, where the memory is initialized with M low-frequency components of all 1s – hint pairs;

[0028] 2.5 The Fourier Transform module receives the image x to be segmented from the user input. t , for x t Perform a Fast Fourier Transform on x t The low-frequency components are sent to the memory bank module, and x t The amplitude and phase are sent to the prompt update module and the inverse Fourier transform module, by means of:

[0029] 2.5.1 Regarding x t Perform a fast Fourier transform to obtain x t amplitude and phase As shown in formula (1);

[0030]

[0031] in Indicates Fast Fourier Transform;

[0032] 2.5.2 Take Chinese x t low-frequency components That is, the central region of the amplitude, with a size of αH×αW, will Send to the memory module;

[0033] 2.5.3 x t amplitude and phase Send to the prompt update module and the inverse Fourier transform module;

[0034] 2.6 The memory module receives x from the Fourier transform module. t low-frequency components Calculate the cosine similarity with the low-frequency components stored in the memory bank, and select the N values ​​with the highest cosine similarity. Use the hints in the memory bank corresponding to the indices of these N values ​​to initialize the hint, thus obtaining the result with x. tThe corresponding initialization prompt is sent to the prompt update module as follows:

[0035] 2.6.1 Let m = 1;

[0036] 2.6.2 Calculate the cosine similarity with the low-frequency component k stored in the memory bank, denoted as Cos(m), as shown in formula (2): m

[0037]

[0038] where "·" represents the dot product and ||·|| is the Euclidean norm;

[0039] 2.6.3 If m < M, let m = m + 1 and go to 2.6.2; if m = M, it means that all cosine similarities have been calculated, go to 2.6.4;

[0040] 2.6.4 Compare Cos(1), …, Cos(m), …, Cos(M), and take the largest N values, where N is a positive integer, preferably N = 16. Denote the indices (i.e., the values of m) corresponding to these N values as s1, …, s n , …, s N . Use the prompts corresponding to these indices to initialize the prompt and obtain the initialization prompt corresponding to x t as follows:

[0041] 2.6.4.1 Let n = 1;

[0042] 2.6.4.2 Calculate the weight w n of the prompt, and the calculation method is as shown in formula (3):

[0043]

[0044] 2.6.4.3 If n < N, let n = n + 1 and go to 2.6.4.2; if n = N, it means that all weights have been calculated, go to 2.6.4.4;

[0045] 2.6.4.4 Calculate the initialization prompt corresponding to x t as shown in formula (4):

[0046]

[0047] 2.6.4.5 Send the initialization prompt corresponding to x t to the prompt update module;

[0048] 2.7 The prompt update module receives from the memory bank module​​​​​ Amplitude received from the Fourier transform module and phase Calculate the loss based on the absolute difference of the BN layer statistics in the target domain. And it propagates the message back to the update module, thus obtaining...

[0049] to x t Corresponding update notification To achieve alignment of the low-level distribution, The method to send data to the memory module and the inverse Fourier transform module is as follows:

[0050] 2.7.1 Prompt to update module and Perform an inverse fast Fourier transform to obtain the initial image. The calculation method is as shown in formula (5):

[0051]

[0052] in This represents the inverse Fourier transform; ⊙ represents element-wise multiplication. Indicates to Perform full 1 padding (i.e., pad the tensor with 1s around its perimeter) until it matches the magnitude spatial resolution H×W.

[0053] 2.7.2 will The input suggestion update model in the suggestion update module is used to change the mean and variance of each batch normalization layer (BN layer) in the suggestion update model (i.e., the mean and variance of the target domain BN layer) from the mean and variance of the previous source domain. The suggestions are updated by minimizing the absolute difference in the mean and variance of the source and target domain BN layers, thus achieving alignment of the low-level distribution. The loss based on the absolute difference in the statistics of the target domain BN layer is calculated. As in formula (6):

[0054]

[0055] Where J represents the total number of BN layers in the model that prompts for updating. This represents the mean value of the j-th BN layer under the source domain input. This represents the variance of the j-th BN layer under the source domain input. This represents the mean value of the j-th BN layer under the target domain input. This represents the variance of the j-th BN layer under the target domain input;

[0056] 2.7.3 Usage Perform a backpropagation, update the parameters using the Adam optimizer, and update... To minimize When the parameters are updated, the learning rate is 0.05, while it is prompted to update that all parameters of the model remain frozen; Denote the updated prompt as

[0057] as

[0058] 2.7.4 Send the updated prompt to the memory bank module and the inverse Fourier transform module;

[0059] 2.8 The memory bank module receives from the prompt update module and combines it with the low-frequency components previously received from the Fourier transform module to form a new low-frequency component - prompt pair, and add it to the memory bank using the first-in, first-out (FIFO) strategy. The method is as follows:

[0060] 2.8.1 Let m = 1;

[0061] 2.8.2 Let (k m , v m ) in the memory bank be equal to (k m+1 , v m+1 );

[0062] 2.8.3 If m < M - 1, let m = m + 1 and go to 2.8.2; If m = M - 1, it means that the first to the second-to-last low-frequency component - prompt pairs in the memory bank have been assigned, then go to 2.8.4;

[0063] 2.8.4 Let The memory bank update is completed, then go to 2.9;

[0064] 2.9 The inverse Fourier transform module receives the updated prompt from the prompt update module receives the amplitude and phase from the Fourier transform module

[0065] performs an inverse fast Fourier transform on the updated prompt and the amplitude and phase to obtain the data-adapted image Send to the strong enhancement module, the weak enhancement module, and the image segmentation module. The method is as follows:

[0066] 2.9.1 Perform an inverse fast Fourier transform on and the amplitude phase to obtain the data-adapted image The calculation method is as shown in formula (7):

[0067]

[0068] denotes the pair of Perform full 1 padding until it matches the amplitude spatial resolution H×W;

[0069] 2.9.2 Image after data adaptation Send to the strong enhancement module, weak enhancement module, and image segmentation module;

[0070] 2.10 Both the strong enhancement module and the weak enhancement module simultaneously receive the adapted image from the inverse Fourier transform module. Strong enhancement module Perform strong enhancement (i.e., to) The image is then enhanced by adjusting brightness, contrast, applying Gamma transformation, Gaussian noise, and Gaussian blur. Will Send to the image segmentation module; simultaneously, the weak enhancement module... Perform five types of weak enhancement (i.e., to By performing horizontal flipping, vertical flipping, rotation by 90 degrees, rotation by 180 degrees, or rotation by 270 degrees, five weakly enhanced image sets R are obtained. set R set Includes five images with different weak enhancements. yes The image after horizontal flipping. yes The image after vertical flipping. yes The image after rotating 90 degrees. yes The image after rotating 180 degrees, yes After rotating the image by 270 degrees, R set Send to the image segmentation module;

[0071] 2.11 The image segmentation module receives the enhanced image from the enhancement module. The image segmentation model in the image segmentation module Segmentation is performed to obtain the probability map p of the strongly enhanced image. Q , will p Q Send to the consistency loss function calculation module;

[0072] 2.12 The image segmentation module receives the weakly enhanced image set R from the weak enhancement module. set The image segmentation model in the image segmentation module is for The images are segmented separately to obtain the probability map of the first weakly enhanced image. Probability map of the second weakly enhanced image Probability map of the third weakly enhanced image Probability map of the fourth weakly enhanced image Probability map of the fifth weakly enhanced image The average probability map p of the weakly enhanced image set is obtained by averaging these five probability maps. R , p R Send to the consistency loss function calculation module;

[0073] 2.13 The image segmentation module receives the adapted image from the inverse Fourier transform module. The image segmentation model in the image segmentation module Segmentation is performed to obtain the probability map p of the image. t , will p t Send to the entropy calculation module;

[0074] 2.14 The consistency loss function calculation module receives the probability map p of the strongly enhanced image from the image segmentation module. Q The average probability map p of the weakly enhanced image set R Calculate the consistency loss value The consistency loss value is then sent to the summation module, and the calculation method is as shown in formula (8):

[0075]

[0076] in p Q ,p R The cross entropy (CE) loss (see the reference: "Claude E. Shannon. A mathematical theory of communication [J]. The Bell System Technical Journal, 1948, 27(3): 379-423." Claude E. Shannon's paper: A mathematical theory of communication);

[0077] 2.15 The entropy calculation module receives the probability map p of the image from the image segmentation module. t Calculate the entropy loss value The entropy loss value is then sent to the summation module, and the calculation method is as shown in formula (9):

[0078]

[0079] 2.16 Summation module for consistency loss value and entropy loss value Weighted summation yields the final loss value. The calculation method is as shown in formula (10):

[0080]

[0081] Where ρ is a parameter that controls the trade-off between consistency loss and entropy minimization, ρ∈(0,1), and preferably ρ is 0.5;

[0082] 2.17 Backpropagation loss value of the summation module The Adam optimizer is used to update the parameters of the batch normalization (BN) layer of the image segmentation model in the image segmentation module (only updating the image segmentation model, not the prompting model). The learning rate is 1×10⁻⁶ during parameter updates. -5 The model adaptation is completed once (i.e. the image segmentation model completes model adaptation), aligning the low-level distribution with the high-level semantic features, and obtaining a test-time adaptive medical image segmentation system based on collaborative data model adaptation after model adaptation.

[0083] The third step involves testing the adapted medical image segmentation system based on collaborative data model adaptation against the adapted images. To perform segmentation and obtain the predicted image segmentation results, the method is as follows:

[0084] The image segmentation module, after model adaptation, receives data from the inverse Fourier transform module and adapts the resulting image. Segmentation is performed to obtain the user-input image x. t The image segmentation is complete once the segmentation results are obtained.

[0085] The following beneficial effects can be achieved by using the present invention:

[0086] This invention proposes a test-time adaptive medical image segmentation method based on collaborative data model adaptation. The invention uses a batch normalization (BN) layer to connect data adaptation and model adaptation, achieving synergy between the two, and aligning low-level distribution with high-level semantic features, thereby improving the performance of the image segmentation model and enabling adaptive segmentation of medical images. The following effects can be achieved using this invention:

[0087] 1. This invention constructs a test-time adaptive medical image segmentation system that integrates a Fourier transform module, a memory module, a hint update module, an inverse Fourier transform module, a strong enhancement module, a weak enhancement module, an image segmentation module, a consistency loss function calculation module, an entropy calculation module, and a summation module. In the second testing step, the system improves the test-time adaptive medical image segmentation performance by coordinating data and model adaptation through a batch normalization (BN) layer.

[0088] 2. This invention employs a collaborative data model adaptation. It introduces a Fourier transform module, a memory module, a hint update module, and an inverse Fourier transform module to adapt the input image data (see 2.5-2.9 for details), changing the appearance of the original image to adapt it as closely as possible to the source domain model at the distribution level. Furthermore, it proposes a strong enhancement module, a weak enhancement module, an image segmentation module, a consistency loss function calculation module, an entropy calculation module, and a summation module to adapt the model (see 2.10-2.17 for details), enabling the model to adapt to new data and achieving alignment between low-level distribution and high-level semantic features.

[0089] 3. The third step of this invention employs a test-time adaptive medical image segmentation system adapted to the model to segment the data-adapted images, achieving alignment of low-level data distribution with high-level semantic features of the model. Experiments were conducted using two-dimensional fundus image datasets from four different institutions to test this invention. The invention was compared with the TENT, CoTTA, and VPTTA methods described in the background, as well as several classic test-time adaptive methods such as DLTTA, DUA, SAR, and DomainAdaptor. The Dice coefficient of this invention is significantly improved compared to the above methods. Attached Figure Description

[0090] Figure 1 This is a logical structure diagram of the test-time adaptive medical image segmentation system constructed in the first step of this invention.

[0091] Figure 2 This is the overall flowchart of the present invention.

[0092] Figure 3 This image presents a qualitative comparison of the segmentation results of the present invention with those of the Visually Prompt Data Adaptive (VPTTA) test-time image segmentation method, based on visual cue data adaptation, and the ground truth labels. The images in each row represent fundus images from different medical institutions and their corresponding segmentation results, including the segmentation of the optic cup and optic disc. Since other methods in the background art are less effective than VPTTA, only the best-performing VPTTA method is selected for qualitative comparison. Detailed Implementation

[0093] Specific examples of the present invention will be described below with reference to the accompanying drawings. Figure 2 As shown, the present invention includes the following steps:

[0094] The first step is to construct a test-time adaptive medical image segmentation system based on a collaborative data model. For example... Figure 1 As shown, the test-time adaptive medical image segmentation system based on collaborative data model adaptation consists of a Fourier transform module, a memory module, a hint update module, an inverse Fourier transform module, a strong enhancement module, a weak enhancement module, an image segmentation module, a consistency loss function calculation module, an entropy calculation module, and a summation module.

[0095] The Fourier transform module is connected to the memory bank module, the prompt update module, and the inverse Fourier transform module. During testing, the Fourier transform module receives an image input at the current time, performs a Fast Fourier Transform on the image, and obtains the amplitude and phase. The low-frequency component of the amplitude is sent to the memory bank module, and the amplitude and phase are sent to the prompt update module and the inverse Fourier transform module.

[0096] The memory module is connected to the Fourier transform module and the prompt update module. During testing, the memory module receives low-frequency components from the Fourier transform module, calculates the similarity between the low-frequency components and the existing low-frequency components in the memory, selects the most similar low-frequency components, and obtains an initialized prompt by weighted averaging the prompts corresponding to the most similar low-frequency components. The initialized prompt is then sent to the prompt update module. The module receives the updated prompt from the prompt update module, adds the low-frequency components and the updated prompt as low-frequency component-prompt pairs to the memory, and pops the oldest low-frequency component-prompt pair from the memory, maintaining a fixed number of low-frequency component-prompt pairs in the memory.

[0097] The prompt update module is connected to the Fourier Transform module, the memory module, and the Inverse Fourier Transform module. The prompt update module contains a segmentation model (let's call it the prompt update model), whose parameters are taken from the source domain segmentation model. During testing, the prompt update module receives initialized prompts from the memory module and amplitude and phase from the Fourier Transform module. The prompt update module pads the initialized prompts with 1s around the edges until they match the amplitude, and then performs an Inverse Fast Fourier Transform on the initialized prompts along with the phase to obtain the initialized image. The initialized image is then input into the prompt update model within the prompt update module to obtain a loss function based on a batch normalization layer. This loss function is backpropagated through the prompt update module to update the prompts, resulting in the updated prompts. The updated prompts are then sent to the Inverse Fourier Transform module and the memory module. The prompt update model uses the same ResUNet architecture as the source domain segmentation model, representing a natural evolution of researchers combining ResNet and U-Net. ResNet networks are constructed by stacking a series of residual blocks, each consisting of convolutional layers and normalization layers. Depending on the network depth, variants of ResNet (such as ResNet34, ResNet50, ResNet101, etc.) are commonly used in specific applications. In ResNet, each convolutional layer, except for the output layer, is followed by a normalization layer. U-Net is a convolutional neural network specifically designed for biomedical image segmentation. Its core feature is its unique U-shaped symmetric structure, comprising an encoder (downsampling) and a decoder (upsampling). It fuses features from different levels through skip connections, extracting high-level semantic information while preserving spatial details. ResUNet is an improved segmentation network combining U-Net and ResNet. By introducing residual blocks into the traditional U-Net architecture, it effectively solves the gradient vanishing problem in deep networks, improving feature reuse and training stability. This network retains the encoder-decoder structure and skip connections of U-Net, while adding cross-layer short-circuit connections in each convolutional block. This allows the network to learn more complex feature representations, resulting in excellent performance in tasks requiring fine boundary prediction, such as medical image segmentation and remote sensing image analysis. It is particularly suitable for segmentation scenarios with small sample sizes and high accuracy. In ResUNet, in addition to the output layer and skip connections, each convolutional layer is followed by a normalization layer.

[0098] The Inverse Fourier Transform (IFT) module is connected to the Fourier Transform (FT) module, the cue update module, the strong enhancement module, the weak enhancement module, and the image segmentation module. During testing, the IFT module receives the updated cue from the cue update module and the amplitude and phase from the Fourier Transform (FT) module. It pads the edges of the updated cue with 1s until it matches the amplitude value. Then, it performs an IFT with the phase value after multiplying the updated cue by the amplitude, obtaining the data-adapted image. This data-adapted image is then sent to the strong enhancement module, the weak enhancement module, and the image segmentation module, respectively.

[0099] The enhanced image enhancement module is connected to the inverse Fourier transform module and the image segmentation module. During testing, the enhanced image enhancement module receives the adapted image from the inverse Fourier transform module, performs enhanced image enhancement on the adapted image to obtain a highly enhanced image, and then sends the highly enhanced image to the image segmentation module.

[0100] The weak enhancement module is connected to the inverse Fourier transform module and the image segmentation module. During testing, the weak enhancement module receives the adapted image from the inverse Fourier transform module and performs different weak enhancement processes on the adapted image, namely, horizontal flipping, vertical flipping, rotation by 90 degrees, rotation by 180 degrees, and rotation by 270 degrees, respectively, to obtain five weakly enhanced images. These five weakly enhanced images are designated as the weakly enhanced image set and sent to the image segmentation module.

[0101] The image segmentation module is connected to the inverse Fourier transform module, strong enhancement module, weak enhancement module, consistency loss function calculation module, entropy calculation module, and summation module. During testing, the image segmentation module receives the strongly enhanced image from the strong enhancement module, segments it to obtain a probability map of the strongly enhanced image, and sends this probability map to the consistency loss function calculation module. The image segmentation module receives the weakly enhanced image set from the weak enhancement module, segments each weakly enhanced image in the set, averages the segmentation results to obtain the average probability map of the weakly enhanced image set, and sends this average probability map to the consistency loss function calculation module. The image segmentation module receives the data-adapted image from the inverse Fourier transform module, segments it to obtain the image's probability map, and sends this probability map to the entropy calculation module. The image segmentation module consists of an image segmentation model, which, like the cue update model, uses the same ResUNet neural network architecture as the source domain model, and its initial parameters are also the same as those of the source domain segmentation model. After the convolutional layer of the image segmentation model, there is a batch normalization layer. The batch normalization layer needs to be adapted by the model to ensure accurate segmentation. After the image segmentation model is adapted, the image segmentation module segments the image after it is adapted by the data received from the inverse Fourier transform module and outputs the final segmentation result.

[0102] The consistency loss function calculation module is connected to the image segmentation module and the summation module. During testing, the consistency loss function calculation module receives the probability map of the strongly enhanced image and the average probability map of the weakly enhanced image set from the image segmentation module. It calculates the consistency loss value based on the probability map of the strongly enhanced image and the average probability map of the weakly enhanced image set, and then sends the consistency loss value to the summation module.

[0103] The entropy calculation module is connected to the image segmentation module and the summation module. During testing, the entropy calculation module receives the probability map of the image from the image segmentation module, calculates the entropy loss value based on the probability map, and sends the entropy loss value to the summation module.

[0104] The summation module is connected to the consistency loss function calculation module, the entropy calculation module, and the image segmentation module. During testing, the summation module receives consistency loss values ​​from the consistency loss function calculation module and entropy loss values ​​from the entropy calculation module. It calculates the weighted sum of the consistency loss value and the entropy loss value to obtain the final loss value, and then backpropagates the final loss value to the image segmentation model of the image segmentation module for model adaptation.

[0105] In the test-time adaptive medical image segmentation system based on collaborative data model adaptation, all modules operate during testing. For each image to be segmented, data adaptation is performed, and simultaneously, the model adapts to the image. The adapted system then segments the image to obtain the segmentation result.

[0106] The second step involves the test-time adaptive medical image segmentation system based on collaborative data model adaptation, constructed in the first step, receiving the user-input image to be segmented. This user-input image is then used to test the system. During testing, the image to be segmented undergoes data adaptation, and the adapted image is then used to adapt the image segmentation model within the system. The method is as follows:

[0107] 2.1 Initialize the model parameters in the test-time adaptive medical image segmentation system based on collaborative data model adaptation. The method is to load the model parameters of the user-provided source domain model (generally, the user will train the source domain model on labeled source domain data (fundus images of a certain institution) using deep learning methods) into the prompt update model in the prompt update module and the image segmentation model in the image segmentation module.

[0108] 2.2 The Adam optimizer was selected as the optimizer for both the cue update model and the image segmentation model. The learning rate for the cue update model was initialized to 0.05, and the learning rate for the image segmentation model was initialized to 1×10⁻⁶. -5 ;

[0109] 2.3 Construct the memory bank in the memory bank module. The memory bank size M is a positive integer, preferably set to 40. The memory bank contains M low-frequency component-hint pairs in the form {(k1,v1),…,(k m ,v m ),…,(k M ,v M )},1≤m≤M,(k m ,v m ) represents the m-th low-frequency component - hint pair, where k m V represents the m-th low-frequency component. m This represents the m-th hint. k m ,v m ∈R (αH)×(αW)×C H, W, and C represent the height, width, and number of channels of the image, respectively. The scaling factor α controls the amplitude region of the cues, prompting them to focus on the low-frequency part α∈(0,1), preferably α=0.01;

[0110] 2.4 (k1, v1), ..., (k m ,v m ),…,(k M ,v M All are initialized to This represents a tensor with all elements set to 1, i.e., the memory is initialized with M low-frequency components-hint pairs, all of which are all 1s.

[0111] 2.5 The Fourier Transform module receives the image x to be segmented from the user input. t , for x t Perform a Fast Fourier Transform on x t The low-frequency components are sent to the memory bank module, and x t The amplitude and phase are sent to the prompt update module and the inverse Fourier transform module, by means of:

[0112] 2.5.1 Regarding x t Perform a fast Fourier transform to obtain x t amplitude and phase As shown in formula (1);

[0113]

[0114] in Indicates Fast Fourier Transform;

[0115] 2.5.2 Take Chinese x t low-frequency components That is, the central region of the amplitude, with a size of αH×αW, will Send to the memory module;

[0116] 2.5.3 Send the amplitude t and phase of x to the hint update module and the inverse Fourier transform module;

[0117] 2.6 The memory bank module receives the low-frequency components t of x from the Fourier transform module, calculates the cosine similarity with the low-frequency components stored in the memory bank, selects the N values with the largest cosine similarity, and uses the hints in the memory bank corresponding to the indices of these N values to initialize the hint, obtaining the initialization hint corresponding to x t , and sends to the hint update module. The method is as follows:

[0118] 2.6.1 Let m = 1;

[0119] 2.6.2 Calculate the cosine similarity using and the low-frequency component k m stored in the memory bank, denoted as Cos(m), as shown in formula (2):

[0120]

[0121] where "·" represents the dot product and ||·|| is the Euclidean norm;

[0122] 2.6.3 If m < M, let m = m + 1 and go to 2.6.2; if m = M, it means that all cosine similarities have been calculated, go to 2.6.4;

[0123] 2.6.4 Compare Cos(1),..., Cos(m),..., Cos(M), select the largest N values, N is a positive integer, preferably N = 16, and denote the indices (i.e., the values of m) corresponding to these N values as s1,..., s n ,..., s N . Use the hints corresponding to these indices to initialize the hint, obtaining the initialization hint corresponding to x t . The method is as follows:

[0124] 2.6.4.1 Let n = 1;

[0125] 2.6.4.2 Calculate the weight w n of the hint, and the calculation method is as shown in formula (3):

[0126]

[0127] 2.6.4.3 If n < N, let n = n + 1 and go to 2.6.4.2; if n = N, it means that all weights have been calculated, go to 2.6.4.4;

[0128] 2.6.4.4 Calculation with x t Corresponding initialization prompts The calculation method is as shown in formula (4):

[0129]

[0130] 2.6.4.5 will be related to x t Corresponding initialization prompts Send to the update notification module;

[0131] 2.7 The update module receives information from the memory module. Amplitude received from the Fourier transform module and phase Calculate the loss based on the absolute difference of the BN layer statistics in the target domain. And it propagates the message back to the update module, thus obtaining...

[0132] to x t Corresponding update notification To achieve alignment of the low-level distribution, The method to send data to the memory module and the inverse Fourier transform module is as follows:

[0133] 2.7.1 Prompt to update module and Perform an inverse fast Fourier transform to obtain the initial image. The calculation method is as shown in formula (5):

[0134]

[0135] in This represents the inverse Fourier transform; ⊙ represents element-wise multiplication. Indicates to Perform full 1 padding (i.e., pad the tensor with 1s around its perimeter) until it matches the magnitude spatial resolution H×W.

[0136] 2.7.2 will The input suggestion update model in the suggestion update module is used to change the mean and variance of each BN layer in the suggestion update model (i.e., the mean and variance of the target domain BN layer) from the mean and variance of the previous source domain. The suggestions are updated by minimizing the absolute difference in the mean and variance of the source and target domain BN layers, thus achieving alignment of the low-level distribution. The loss based on the absolute difference in the statistics of the target domain BN layer is calculated. As in formula (6):

[0137]

[0138] Where J represents the total number of BN layers in the model that prompts for updating. represents the mean value of the j-th BN layer under the source domain input, represents the variance of the j-th BN layer under the source domain input, represents the mean value of the j-th BN layer under the target domain input, represents the variance of the j-th BN layer under the target domain input;

[0139] 2.7.3 Use to perform a single backpropagation. The parameter update uses the Adam optimizer to update to minimize The learning rate during parameter update is 0.05, while prompting to update all model parameters to remain frozen; Denote the updated prompt as

[0140] as

[0141] 2.7.4 Send the updated prompt to the memory bank module and the inverse Fourier transform module;

[0142] 2.8 The memory bank module receives from the prompt update module and combines it with the low-frequency component previously received from the Fourier transform module to form a new low-frequency component - prompt pair, which is added to the memory bank using the First-In, First-Out (FIFO) strategy. The method is as follows:

[0143] 2.8.1 Let m = 1;

[0144] 2.8.2 Let (k m , v m ) in the memory bank be equal to (k m+1 , v m+1 );

[0145] 2.8.3 If m < M - 1, let m = m + 1 and go to 2.8.2; If m = M - 1, it means that the first to the second-to-last low-frequency component - prompt pairs in the memory bank have been assigned, then go to 2.8.4;

[0146] 2.8.4 Let The memory bank update is complete, then go to 2.9;

[0147] 2.9 The inverse Fourier transform module receives the updated prompt from the prompt update module receives the amplitude and phase from the Fourier transform module

[0148] and performs an inverse fast Fourier transform on the updated prompt and the amplitude and phase to obtain the data-adapted image Send The data is sent to the strong enhancement module, the weak enhancement module, and the image segmentation module using the following method:

[0149] 2.9.1 and amplitude phase Perform an inverse fast Fourier transform to obtain the image after data adaptation. The calculation method is as shown in formula (7):

[0150]

[0151] Indicates to Perform full 1 padding until it matches the amplitude spatial resolution H×W;

[0152] 2.9.2 Image after data adaptation Send to the strong enhancement module, weak enhancement module, and image segmentation module;

[0153] 2.10 Both the strong enhancement module and the weak enhancement module simultaneously receive the adapted image from the inverse Fourier transform module. Strong enhancement module Perform strong enhancement (i.e., to) The image is then enhanced by adjusting brightness, contrast, applying Gamma transformation, Gaussian noise, and Gaussian blur. Will Send to the image segmentation module; simultaneously, the weak enhancement module... Perform five types of weak enhancement (i.e., to By performing horizontal flipping, vertical flipping, rotation by 90 degrees, rotation by 180 degrees, or rotation by 270 degrees, five weakly enhanced image sets R are obtained. set R set Includes five images with different weak enhancements. yes The image after horizontal flipping. yes The image after vertical flipping. yes The image after rotating 90 degrees. yes The image after rotating 180 degrees, yes After rotating the image by 270 degrees, R set Send to the image segmentation module;

[0154] 2.11 The image segmentation module receives the enhanced image from the enhancement module. The image segmentation model in the image segmentation module Segmentation is performed to obtain the probability map p of the strongly enhanced image. Q , will p Q Send to the consistency loss function calculation module;

[0155] 2.12 The image segmentation module receives the weakly enhanced image set R from the weak enhancement module. set The image segmentation model in the image segmentation module is for The images are segmented separately to obtain the probability map of the first weakly enhanced image. Probability map of the second weakly enhanced image Probability map of the third weakly enhanced image Probability map of the fourth weakly enhanced image Probability map of the fifth weakly enhanced image The average probability map p of the weakly enhanced image set is obtained by averaging these five probability maps. R , p R Send to the consistency loss function calculation module;

[0156] 2.13 The image segmentation module receives the adapted image from the inverse Fourier transform module. The image segmentation model in the image segmentation module Segmentation is performed to obtain the probability map p of the image. t , will p t Send to the entropy calculation module;

[0157] 2.14 The consistency loss function calculation module receives the probability map p of the strongly enhanced image from the image segmentation module. Q The average probability map p of the weakly enhanced image set R Calculate the consistency loss value The consistency loss value is then sent to the summation module, and the calculation method is as shown in formula (8):

[0158]

[0159] in p Q ,p R Cross-entropy CE loss;

[0160] 2.15 The entropy calculation module receives the probability map p of the image from the image segmentation module. t Calculate the entropy loss value The entropy loss value is then sent to the summation module, and the calculation method is as shown in formula (9):

[0161]

[0162] 2.16 Summation module for consistency loss value and entropy loss value Weighted summation yields the final loss value. The calculation method is as shown in formula (10):

[0163]

[0164] Where ρ is a parameter that controls the trade-off between consistency loss and entropy minimization, ρ∈(0,1), and preferably ρ is 0.5;

[0165] 2.17 Backpropagation loss value of the summation module Then, the Adam optimizer is used to update the Batch Normalization (BN) layer parameters of the image segmentation model in the image segmentation module (only the image segmentation model is updated, not the prompt to update the model). The learning rate is 1×10⁻⁶ during parameter updates. -5 The model adaptation of the image segmentation model is completed, and the alignment of low-level distribution with high-level semantic features is achieved, resulting in a test-time adaptive medical image segmentation system based on collaborative data model adaptation after model adaptation.

[0166] The third step involves testing the adapted medical image segmentation system based on collaborative data model adaptation against the adapted images. To perform segmentation and obtain the predicted image segmentation results, the method is as follows:

[0167] The image segmentation module, after model adaptation, receives data from the inverse Fourier transform module and adapts the resulting image. Segmentation is performed to obtain the user-input image x. t The image segmentation is complete once the segmentation results are obtained.

[0168] To verify the effectiveness of this invention, two-dimensional fundus image datasets from different institutions were selected: RIM-ONE (see the paper "Francisco Fumero, et al. RIM-ONE: An open retinal image database for opticnerve evaluation [C]. 2011 24th international symposium on computer-based medical systems (CBMS). IEEE, 2011: 1-6." Francisco Fumero et al.'s paper: RIM-ONE: An open retinal image database for opticnerve evaluation), REFUGE (see the paper "José Ignacio Orlando, et al. Refuge challenge: Aunified framework for evaluating automated methods for glaucoma assessment from fundus photographs [J]. Medical image analysis, 2020, 59: 101570." José Ignacio Orlando et al.'s paper: Refuge challenge: A unified framework for evaluating automated methods for glaucoma assessment from fundus photographs [J]. Medical image analysis, 2020, 59: 101570." José Ignacio Orlando et al.'s paper: Refuge challenge: A unified framework for evaluating automated methods for glaucoma assessment from fundus photographs), and ORIGA (see the paper "Zhuo Zhang, et al. Origa-light: Anonline retinal fundus image database"). For glaucoma analysis and research[C]. 2010 Annual international conference of the IEEE Engineering in Medicine and Biology. IEEE, 2010: 3065-3068. Zhuo Zhang et al.'s paper: Origa-light: An online retinal fundus image database for glaucoma analysis and research), Drishti-GS (see the literature "Jayanthi Sivaswamy, et al. Drishti-gs: Retinal image dataset for optic nerve head(onh) segmentation[C]. 2014 IEEE 11th international symposium on biomedical imaging (ISBI). IEEE, 2014: 53-56.The paper by Jayanthi Sivaswamy et al., "Drishti-gs: A Retinal Image Dataset for Optic Disc (ONH) Segmentation," uses user input images as the test set to perform qualitative and quantitative comparisons of the segmentation results of this invention. The quantitative comparison uses the Dice segmentation coefficient. These datasets include fundus images and their corresponding ground truth labels, with labels indicating the optic cup and optic disc regions, respectively. These four datasets contain 159, 400, 650, and 101 two-dimensional fundus images, respectively, all used as test set images, totaling 1310 images, with the image resolution uniformly adjusted to 512×512.

[0169] The experiment was conducted on a server using Ubuntu 20.04 (a version of Linux) and an NVIDIA Quadro RTX 6000 GPU with 24GB of RAM. All images from the RIM-ONE, REFUGE, ORIGA, and Drishti-GS medical fundus 2D image datasets from different institutions were streamed into the test-time adaptive image segmentation system of this invention. All data were segmented, and the Dice coefficients were calculated and compared with other methods to measure the effectiveness of the invention. During the experiment, the ResNet network used in this invention was ResNet34, with learning rates of 0.05 for the update model and 1×10⁻⁶ for the image segmentation model. -5 M is set to 40, α = 0.01, N = 16, and ρ is set to 0.5.

[0170] First, the segmentation results of this invention are qualitatively compared with the segmentation results of VPTTA, a test-time adaptive image segmentation method based on visual cue data adaptation in the background technology. Since other methods in the background technology are not as effective as VPTTA, only VPTTA is selected as a representative for comparison of visual results (see the paper "Ziyang Chen, et al. Each testimage deserves a specific prompt: Continual test-time adaptation for 2dmedical image segmentation[C]. Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2024:11184-11193. Ziyang Chen et al.'s paper: Each test image needs a specific prompt: Continual test-time adaptive method for 2D medical image segmentation"). Figure 3As shown, the first column contains user-input 2D fundus images, the second column contains segmentation prediction results from the VPTTA method, the third column contains segmentation prediction results from this invention, and the fourth column contains ground truth labels. The first row contains example images from the RIM-ONE dataset, the second row from the REFUGE dataset, the third row from the ORIGA dataset, and the fourth row from the Drishti-GS dataset. The yellow portion represents the segmented optic cup, outlined in red; the orange portion represents the segmented optic disc, outlined in green. From... Figure 3 It can be seen that the segmentation prediction results of the present invention are more complete and closer to the true labels than VPTTA, indicating that the present invention can effectively segment and predict the optic cup and optic disc in fundus images.

[0171] Then, a quantitative comparison is performed. First, a performance evaluation index for image segmentation algorithms is defined. This experiment uses a medical image segmentation performance evaluation method, specifically the index Dice. Dice represents the Dice coefficient, which is twice the number of correctly segmented regions divided by the union of the predicted result and the label. The larger the Dice value, the better the segmentation effect.

[0172] Based on the experimental results of this invention, the experimental results of the fundus two-dimensional image datasets RIM-ONE, REFUGE, ORIGA, and Drishti-GS are analyzed. The source domain image segmentation model was trained on the fundus image dataset REFUGE-Validation / Test, which comes from a different institution than RIM-ONE, REFUGE, ORIGA, and Drishti-GS, thus exhibiting domain bias. The average quantitative performance comparison of this invention on the four datasets RIM-ONE, REFUGE, ORIGA, and Drishti-GS is shown in Table 1. "No adaptation" represents directly using the source domain segmentation model to segment the test set data, where the segmentation performance is very poor, representing the lower bound of the segmentation performance and illustrating the necessity of domain adaptation; the seven methods in the middle (TENT-Continual, CoTTA, DLTTA, DUA, SAR, Domain Adaptor, VPTTA) are test-time adaptive methods, among which TENT-Continual is the TENT method used for continuous test-time adaptation. The results in the table are from the paper "Ziyang Chen, et al. Each test image deserves a specific prompt: Continual test-time adaptation for 2d medical image segmentation [C]. Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2024:11184-11193. Ziyang Chen et al.'s paper: Each test image deserves a specific prompt: Continual test-time adaptation for 2D medical image segmentation". This paper compares the performance of this invention with the current state-of-the-art (SOTA) test-time adaptive medical image segmentation method VPTTA, and previous test-time adaptive medical image segmentation methods TENT-Continual, CoTTA, DLTTA, DUA, SAR, and Domain Adaptor. The experimental results show that this invention can perform image segmentation quickly and accurately, achieving a consistent performance improvement. This invention outperforms all other TTA methods, achieving the highest average Dice coefficient of 71.78%.

[0173] Table 1

[0174]

[0175] In summary, on the RIM-ONE, REFUGE, ORIGA, and Drishti-GS datasets of 2D fundus images, this invention achieves new state-of-the-art (SOTA) performance in the field of time-of-test adaptive medical image segmentation. As shown in Table 1, in the comparison of the mean Dice, compared to 52.67% for "no adaptation," the Dice value of this invention is significantly improved, reaching 71.78%, demonstrating the necessity of domain adaptation. Compared to other time-of-test adaptive methods such as TENT-Continual, CoTTA, DLTTA, DUA, SAR, Domain Adaptor, and VPTTA, the Dice value of this invention is improved by 15.27% compared to the highest among them (56.51%), indicating a significant improvement and achieving better segmentation results. The Dice value of this invention achieves the best results compared to other time-of-test adaptive medical image segmentation methods, indicating that this invention achieves the best segmentation performance.

Claims

1. A test-time adaptive medical image segmentation method based on collaborative data model adaptation, characterized in that... Includes the following steps: The first step is to construct a test-time adaptive medical image segmentation system based on a collaborative data model. This system consists of a Fourier transform module, a memory module, a hint update module, an inverse Fourier transform module, a strong enhancement module, a weak enhancement module, an image segmentation module, a consistency loss function calculation module, an entropy calculation module, and a summation module. The Fourier transform module is connected to the memory bank module, the prompt update module, and the inverse Fourier transform module. During testing, the Fourier transform module receives an image input at the current time, performs a fast Fourier transform on the image, and obtains the amplitude and phase. The low-frequency component in the amplitude is sent to the memory bank module, and the amplitude and phase are sent to the prompt update module and the inverse Fourier transform module. The memory bank module is connected to the Fourier transform module and the prompt update module. During testing, the memory bank module receives low-frequency components from the Fourier transform module, calculates the similarity between the low-frequency components and the original low-frequency components in the memory bank, selects the most similar low-frequency components, and obtains the initial prompt by weighted averaging the prompts corresponding to the most similar low-frequency components. Send the initialization prompt to the prompt update module; Receive the updated prompt from the prompt update module, add the low-frequency component and the updated prompt as a low-frequency component-prompt pair to the memory bank, and pop the earliest low-frequency component-prompt pair from the memory bank; The prompt update module is connected to the Fourier transform module, the memory module, and the inverse Fourier transform module. The prompt update module contains a prompt update model. During testing, the prompt update module receives initial prompts from the memory module and amplitude and phase information from the Fourier transform module. The prompt update module processes the initial prompts to obtain an initial image. The initial image is input into the prompt update model to obtain a loss function based on a batch normalization (BN) layer. The loss obtained from the BN layer loss function is backpropagated to the prompt update module to update the prompts, resulting in the updated prompts. The updated prompts are then sent to the inverse Fourier transform module and the memory module. The prompt update model uses the same ResUNet architecture as the source domain segmentation model, with a normalization layer connected after each convolutional layer, except for the output layer and skip connections. The inverse Fourier transform module is connected to the Fourier transform module, the prompt update module, the strong enhancement module, the weak enhancement module, and the image segmentation module. During testing, the inverse Fourier transform module receives the updated prompt from the prompt update module and the amplitude and phase from the Fourier transform module. It performs an inverse fast Fourier transform on the updated prompt to obtain the data-adapted image. The data-adapted image is then sent to the strong enhancement module, the weak enhancement module, and the image segmentation module, respectively. The strong enhancement module is connected to the inverse Fourier transform module and the image segmentation module. During testing, the strong enhancement module receives the data-adapted image from the inverse Fourier transform module, performs strong enhancement processing on the data-adapted image to obtain a strongly enhanced image, and sends the strongly enhanced image to the image segmentation module. The weak enhancement module is connected to the inverse Fourier transform module and the image segmentation module. During testing, the weak enhancement module receives the data-adapted image from the inverse Fourier transform module, performs different weak enhancement processes on the data-adapted image, and obtains five weakly enhanced images. These five weakly enhanced images are called the weakly enhanced image set, and the weakly enhanced image set is sent to the image segmentation module. The image segmentation module is connected to the inverse Fourier transform module, the strong enhancement module, the weak enhancement module, the consistency loss function calculation module, the entropy calculation module, and the summation module. During testing, the image segmentation module receives the strongly enhanced image from the strong enhancement module, segments it to obtain the probability map of the strongly enhanced image, and sends the probability map of the strongly enhanced image to the consistency loss function calculation module. The image segmentation module receives the weakly enhanced image set from the weak enhancement module, segments each weakly enhanced image in the image set, averages the segmentation results to obtain the average probability map of the weakly enhanced image set, and sends the average probability map of the weakly enhanced image set to the consistency loss function calculation module. The image segmentation module receives the adapted image from the inverse Fourier transform module, segments it to obtain the probability map of the image, and sends the probability map of the image to the entropy calculation module. The image segmentation module consists of an image segmentation model, which adopts the same ResUNet architecture as the source domain model. After the image segmentation model is adapted, the image segmentation module segments the adapted image received from the inverse Fourier transform module and outputs the final segmentation result. The consistency loss function calculation module is connected to the image segmentation module and the summation module; during testing, the consistency loss function calculation module receives the probability map of the strongly enhanced image and the average probability map of the weakly enhanced image set from the image segmentation module. The consistency loss value is calculated based on the probability map of the strongly enhanced image and the average probability map of the weakly enhanced image set, and then sent to the summation module. The entropy calculation module is connected to the image segmentation module and the summation module. During testing, the entropy calculation module receives the probability map of the image from the image segmentation module, calculates the entropy loss value based on the probability map, and sends the entropy loss value to the summation module. The summation module is connected to the consistency loss function calculation module, the entropy calculation module, and the image segmentation module. During testing, the summation module receives the consistency loss value from the consistency loss function calculation module and the entropy loss value from the entropy calculation module. It calculates the weighted sum of the consistency loss value and the entropy loss value to obtain the final loss value, and then backpropagates the final loss value to the image segmentation model of the image segmentation module for model adaptation. The second step involves the test-time adaptive medical image segmentation system based on collaborative data model adaptation, constructed in the first step, receiving the user-input image to be segmented. This user-input image is then used to test the system. During testing, the image to be segmented undergoes data adaptation, and the adapted image is then used to adapt the image segmentation model within the system. The method is as follows: 2.1 Initialize the model parameters in the test-time adaptive medical image segmentation system based on the collaborative data model adaptation by loading the model parameters of the source domain model provided by the user into the prompt update model in the prompt update module and the image segmentation model in the image segmentation module; 2.2 Select the Adam optimizer as the optimizer for the prompt update model and the image segmentation model; 2.3 Construct the memory bank in the memory bank module. The size M of the memory bank is a positive integer. The memory bank stores M low-frequency component-hint pairs in the form {(k1,v1),…,(k m ,v m ),…,(k M ,v M )},1≤m≤M,(k m ,v m ) represents the m-th low-frequency component - hint pair, where k m V represents the m-th low-frequency component. m This indicates the m-th prompt; k represents the k-th prompt. m ,v m ∈R (αH)×(αW)×C H, W, and C represent the height, width, and number of channels of the image, respectively. The scaling factor α controls the amplitude range of the cue effect, causing it to focus on the low-frequency part. α∈(0,1); 2.4 (k1, v1), ..., (k m ,v m ),…,(k M ,v M All are initialized to A tensor representing a tensor whose elements are all 1s; 2.5 The Fourier Transform module receives the image x to be segmented from the user input. t , for x t Perform a Fast Fourier Transform on x t low-frequency components Send to the memory module, x t amplitude and phase Send to the prompt update module and the inverse Fourier transform module; 2.6 The memory module receives x from the Fourier transform module. t low-frequency components Calculate the cosine similarity with the low-frequency components stored in the memory bank, and select the N values ​​with the highest cosine similarity. Initialize the hint with the hints stored in the memory bank corresponding to the indices of these N values, thus obtaining the hint with x. t The corresponding initialization prompt will Send to the notification update module, where N is a positive integer; 2.7 The update module receives information from the memory module. Amplitude received from the Fourier transform module and phase Calculate the loss based on the absolute difference of the BN layer statistics in the target domain. And backpropagate the suggestion update module to obtain the result with x t Corresponding update notification To achieve alignment of the low-level distribution, Send to the memory module and the inverse Fourier transform module; 2.8 The memory module receives information from the prompt update module. Compared with the low-frequency components previously received from the Fourier transform module New low-frequency component-hint pairs are formed and added to the memory bank using a first-in-first-out (FIFO) strategy; 2.9 The inverse Fourier transform module receives information from the prompt update module. Amplitude received from the Fourier transform module and phase right and phase Perform an inverse fast Fourier transform to obtain the image after data adaptation. Will Send to the strong enhancement module, weak enhancement module, and image segmentation module; 2.10 Both the strong enhancement module and the weak enhancement module simultaneously receive images from the inverse Fourier transform module. Strong enhancement module Perform strong enhancement to obtain the strongly enhanced image. Will Send to the image segmentation module; simultaneously, the weak enhancement module... Five weak enhancements were performed to obtain the image set R after the five weak enhancements. set R set Includes five images with different weak enhancements. yes The image after horizontal flipping. yes The image after vertical flipping. yes The image after rotating 90 degrees. yes The image after rotating 180 degrees, yes After rotating the image by 270 degrees, R set Send to the image segmentation module; 2.11 The image segmentation module receives the enhanced image from the enhancement module. The image segmentation model in the image segmentation module Segmentation is performed to obtain the probability map p of the strongly enhanced image. Q , will p Q Send to the consistency loss function calculation module; 2.12 The image segmentation module receives the weakly enhanced image set R from the weak enhancement module. set The image segmentation model in the image segmentation module is for The images are segmented separately to obtain the probability map of the first weakly enhanced image. Probability map of the second weakly enhanced image Probability map of the third weakly enhanced image Probability map of the fourth weakly enhanced image Probability map of the fifth weakly enhanced image The average probability map p of the weakly enhanced image set is obtained by averaging these five probability maps. R , p R Send to the consistency loss function calculation module; 2.13 The image segmentation module receives data from the inverse Fourier transform module. The image segmentation model in the image segmentation module Segmentation is performed to obtain the probability map p of the image. t , will p t Send to the entropy calculation module; 2.14 The consistency loss function calculation module receives p from the image segmentation module. Q and p R Calculate the consistency loss value The consistency loss value is then sent to the summation module, and the calculation method is as shown in formula (8): in p Q ,p R Cross-entropy CE loss; 2.15 The entropy calculation module receives the probability map p of the image from the image segmentation module. t Calculate the entropy loss value The entropy loss value is then sent to the summation module, and the calculation method is as shown in formula (9): 2.16 Summation module for consistency loss value and entropy loss value Weighted summation yields the final loss value. The calculation method is as shown in formula (10): where ρ is a parameter that controls the trade-off between the consistency loss and entropy minimization, and ρ ∈ (0, 1); 2.17 Backpropagation loss value of the summation module The batch normalization layer parameters of the image segmentation model in the image segmentation module are updated using the Adam optimizer, with a learning rate of 1×10⁻⁶ during parameter updates. -5 A model adaptation is completed to align low-level distribution with high-level semantic features, resulting in a test-time adaptive medical image segmentation system based on collaborative data model adaptation after model adaptation. The third step involves testing the adapted medical image segmentation system based on collaborative data model adaptation against the adapted images. To perform segmentation and obtain the predicted image segmentation results, the method is as follows: The image segmentation module, after model adaptation, receives data from the inverse Fourier transform module and adapts the resulting image. Segmentation is performed to obtain the user-input image x. t The image segmentation is complete once the segmentation results are obtained.

2. The test-time adaptive medical image segmentation method based on collaborative data model adaptation as described in claim 1, characterized in that... The ResUNet architecture is a natural evolution of the combination of ResNet and U-Net; the ResNet network is constructed by stacking a series of residual modules, and the residual modules are composed of convolutional layers and normalization layers. The ResNet network uses ResNet34, ResNet50 or ResNet101 network; in ResNet, except for the output layer, a normalization layer is connected after each convolutional layer; U-Net is a convolutional neural network designed specifically for biomedical image segmentation, adopting a U-shaped symmetric structure, including an encoder and a decoder, and fusing features at different levels through skip connections.

3. The test-time adaptive medical image segmentation method based on collaborative data model adaptation as described in claim 1, characterized in that... When constructing the memory bank in the memory bank module in step 2.3, the memory bank size M is set to 40, and the scaling factor α = 0.01; ρ takes 0.5 in step 2.

16.

4. The test-time adaptive medical image segmentation method based on collaborative data model adaptation as described in claim 1, characterized in that... Step 2.5 The Fourier transform module receives the image x to be segmented from the user input. t , for x t Perform a Fast Fourier Transform on x t The low-frequency components are sent to the memory bank module, and x t The method for sending the amplitude and phase to the prompt update module and the inverse Fourier transform module is as follows: 2.5.1 Regarding x t Perform a fast Fourier transform to obtain x t amplitude and phase As shown in formula (1); in Indicates Fast Fourier Transform; 2.5.2 Take Chinese x t low-frequency components That is, the central region of the amplitude, with a size of αH×αW, will Send to the memory module; 2.5.3 x t amplitude and phase Send to the prompt update module and the inverse Fourier transform module.

5. The test-time adaptive medical image segmentation method based on collaborative data model adaptation as described in claim 1, characterized in that... In step 2.6, the memory module receives x from the Fourier transform module. t low-frequency components Calculate the cosine similarity with the low-frequency components stored in the memory bank, and select the N values ​​with the highest cosine similarity. Use the hints in the memory bank corresponding to the indices of these N values ​​to initialize the hint, thus obtaining the result with x. t The corresponding initialization prompt will The method to send the update notification to the module is: 2.6.1 Let m = 1; 2.6.2 Utilization With the low-frequency component k stored in the memory bank m The cosine similarity is calculated and denoted as Cos(m), as shown in formula (2): where "·" represents the dot product and ||·|| is the Euclidean norm; 2.6.3 If m < M, let m = m + 1 and go to 2.6.2; if m = M, it means that all cosine similarity calculations are completed, and go to 2.6.4; 2.6.4 Compare Cos(1), ..., Cos(m), ..., Cos(M), and take the N largest values, where N is a positive integer. Record the indices corresponding to these N values ​​as s1, ..., s n ,…,s N ; hints corresponding to these indexes To initialize the prompt, get the result with x t The corresponding initialization prompt is as follows: 2.6.4.1 Let n = 1; 2.6.4.2 Calculate the suggested weight w n The calculation method is as shown in formula (3): 2.6.4.3 If n < N, let n = n + 1 and go to 2.6.4.2; if n = N, it means that all weight calculations are completed, and go to 2.6.4.4; 2.6.4.4 Calculation with x t Corresponding initialization prompts The calculation method is as shown in formula (4): 2.6.4.5 will be related to x t Corresponding initialization prompts Send to the notification update module.

6. The test-time adaptive medical image segmentation method based on collaborative data model adaptation as described in claim 5, characterized in that... In step 2.6.4, N = 16.

7. The test-time adaptive medical image segmentation method based on collaborative data model adaptation as described in claim 1, characterized in that... Step 2.7 The prompt update module receives from the memory module. Amplitude received from the Fourier transform module and phase Calculate the loss based on the absolute difference of the BN layer statistics in the target domain. And backpropagate the suggestion to update the module, to obtain the result with x t Corresponding update notification To achieve alignment of the low-level distribution, The method for sending data to the memory module and the inverse Fourier transform module is as follows: 2.7.1 Prompt to update module and Perform an inverse fast Fourier transform to obtain the initial image. The calculation method is as shown in formula (5): in This represents the inverse Fourier transform; ⊙ represents element-wise multiplication. Indicates to Performing full 1 padding involves filling the tensor with 1s around its perimeter until it matches the magnitude space resolution H×W. 2.7.2 will The input suggestion update module is used to update the suggestion update model, so that the mean and variance of each batch normalization layer in the suggestion update model change with the mean and variance of the previous source domain. The suggestion is updated by minimizing the absolute difference between the mean and variance of the batch normalization layers of the source domain and the target domain, thereby achieving alignment of the low-level distribution. Calculate the loss based on the absolute difference of the BN layer statistics in the target domain. As in formula (6): Where J represents the total number of BN layers in the model that prompts for updating. This represents the mean value of the j-th BN layer under the source domain input. This represents the variance of the j-th BN layer under the source domain input. This represents the mean value of the j-th BN layer under the target domain input. This represents the variance of the j-th BN layer under the target domain input; 2.7.3 Usage Perform a backpropagation, update the parameters using the Adam optimizer, and update... To minimize The learning rate is 0.05 when updating parameters, while the suggestion is to keep all model parameters frozen; the updated suggestion is denoted as... 2.7.4 will update the prompts. Send it to the memory module and the inverse Fourier transform module.

8. The test-time adaptive medical image segmentation method based on collaborative data model adaptation as described in claim 1, characterized in that... Step 2.8 The memory module receives from the prompt update module. Compared with the low-frequency components previously received from the Fourier transform module The method for forming new low-frequency component-hint pairs and adding them to the memory bank using a first-in-first-out (FIFO) strategy is as follows: 2.8.1 Let m = 1; 2.8.2 Let (k) in the memory bank m ,v m )=(k m+1 ,v m+1 ); 2.8.3 If m < M - 1, let m = m + 1 and go to 2.8.2; if m = M - 1, it means that the first to the second-to-last low-frequency component - prompt pairs in the memory bank have been assigned, and go to 2.8.4; Order 2.8.4 The memory bank has been updated.

9. The test-time adaptive medical image segmentation method based on collaborative data model adaptation as described in claim 1, characterized in that... Step 2.9 The inverse Fourier transform module receives from the prompt update module Amplitude received from the Fourier transform module and phase right and phase Perform an inverse fast Fourier transform to obtain the image after data adaptation. Will The method for sending data to the strong enhancement module, weak enhancement module, and image segmentation module is as follows: 2.9.1 and amplitude phase Perform an inverse fast Fourier transform to obtain the image after data adaptation. The calculation method is as shown in formula (7): Indicates to Perform full 1 padding until it matches the amplitude spatial resolution H×W; 2.9.2 Image after data adaptation It is sent to the strong enhancement module, the weak enhancement module, and the image segmentation module.

10. The test-time adaptive medical image segmentation method based on collaborative data model adaptation as described in claim 1, characterized in that... The enhanced module described in step 2.10... Perform strong enhancement targeting The weak enhancement module performs brightness adjustment, contrast variation, Gamma transformation, Gaussian noise, and Gaussian blurring. Perform five types of weak enhancement Perform horizontal flip, vertical flip, rotation 90 degrees, rotation 180 degrees, or rotation 270 degrees.

Images