Cross-site incremental medical image segmentation system based on domain-specific regularization

Abstract

This paper discloses a cross-site medical image incremental segmentation system based on domain-specific regularization. This system analyzes the domain specificity of the network parameters of the medical image segmentation model by simulating changes in brightness, contrast, saturation, and image noise of medical images at different sites. By introducing a regularization term, it limits the update of network parameters with high domain specificity, preventing the segmentation model from overfitting to domain-specific features at new sites, thereby solving the catastrophic forgetting problem in the cross-site medical image incremental segmentation task. This invention improves the average performance of cross-site medical image segmentation and reduces the degree of knowledge forgetting during the incremental learning process.

Description

Technical Field

[0001] The present invention relates to the technical field of cross-site medical image segmentation, and in particular to a cross-site medical image incremental segmentation system based on domain-specific regularization. Background Art

[0002] Medical image segmentation is a technique that accurately separates structures or regions of interest from the background in medical images. It has a wide range of applications in the medical field. However, medical image segmentation tasks based on convolutional neural networks require a large number of annotated images. Training images need to be collected from multiple sites. However, privacy restrictions in the medical field prohibit the sharing of clinical data, making it difficult to obtain training data from multiple sites simultaneously.

[0003] Incremental learning is a machine learning method whose goal is to continue learning new data based on an already trained model without having to access previous training data. Incremental learning methods can effectively utilize the information of new samples and incorporate it into the existing model, allowing the model to adapt to new data without forgetting the knowledge from old data. Cross-site incremental medical image segmentation is a medical image segmentation task based on incremental learning. It can overcome the privacy limitations of medical image data. Its goal is to effectively use the data from the new site to update the model trained at the old site without sacrificing the segmentation accuracy of the previous site, so that the model has high segmentation accuracy at both the new and old sites.

[0004] The task of incremental segmentation of cross-site medical images is set as follows: medical image data from multiple sites will arrive sequentially and be used to train the segmentation model. In each training step, only the medical image and segmentation mask of the current site are available, while the medical image and segmentation mask of the previous site cannot be obtained. The segmentation model finally obtained by incremental training should have accurate segmentation performance for medical images of all sites. The difficulty lies in that, since only data from one site can be obtained in each training step, on the one hand, the segmentation model can easily overfit to the domain-specific features of the site, such as brightness, contrast, saturation, and image noise of the medical image; on the other hand, it is difficult for the segmentation model to learn the cross-domain invariant knowledge shared by multiple sites from the data of one site, which can easily reduce the generalization of the model and lead to catastrophic forgetting of the segmentation model at the previous site.

[0005] To mitigate catastrophic forgetting, previous incremental learning methods are mainly divided into three schools: 1) Data replay-based methods: data from old sites need to be replayed in the incremental learning of new sites, which violates the privacy restrictions of medical image data and is therefore not suitable for cross-site medical image incremental segmentation tasks; 2) Parameter isolation-based methods: This type of method explicitly assigns different model parameters to each site, but its effectiveness is limited by the model capacity, so it is not very practical; 3) Regularization-based methods can be further divided into two types: Data-based methods extract knowledge from the previous network model through distillation learning, and use distillation loss as a regularization term to guide network model learning to prevent the network model from forgetting the knowledge of old sites; Prior-based methods assign importance weights to the model parameters and introduce substitution loss as a regularization term in the process of network model update to penalize the offset of the corresponding parameters.

[0006] Although existing methods can alleviate forgetting to a certain extent, they ignore the inherent acquisition bias problem in the cross-site medical image incremental segmentation task, fail to decouple domain-specific knowledge and cross-domain invariant knowledge in the incremental learning process, and find it difficult to accurately extract cross-site invariant knowledge between different sites, thus resulting in poor segmentation performance in the task of cross-site medical image incremental segmentation process. Summary of the Invention

[0007] The purpose of the present invention is to overcome the shortcomings and deficiencies of the existing technology and propose a cross-site medical image incremental segmentation system based on domain-specific regularization. The system analyzes the domain specificity of medical image segmentation model parameters by simulating the changes in brightness, contrast, saturation and image noise of medical images from different sites. In the incremental training stage of the segmentation model, the regularization term is introduced to limit the update of network parameters with high domain specificity, prevent the segmentation model from overfitting to domain-specific features, and thus solve the catastrophic forgetting problem in the cross-site medical image incremental segmentation task.

[0008] To achieve the above objectives, the present invention provides a technical solution: a cross-site medical image incremental segmentation system based on domain-specific regularization, comprising:

[0009] The initial site data import module is used to obtain data from the first site, including medical images and segmentation masks, and preprocess the data to obtain images and segmentation masks of uniform size;

[0010] The segmentation model initialization training module is used to build a U-Net segmentation model and use the preprocessed medical image and segmentation mask of the first site to initialize the training segmentation model to obtain the initialized trained U-Net segmentation model;

[0011] The segmentation model network parameter analysis module is used to analyze the parameter domain specificity of the trained U-Net segmentation model. The module enhances the brightness, contrast, and saturation of the medical images used to train the U-Net segmentation model. The medical images before and after enhancement are fed into the U-Net segmentation model for prediction to obtain two outputs. The difference between the two outputs of the U-Net segmentation model is calculated and back-propagated to obtain the segmentation model gradient, which is the parameter domain specificity of the U-Net segmentation model.

[0012] The new site data import module is used to import the U-Net segmentation model trained on the previous site and the parameter domain specificity of each U-Net segmentation model, and import the medical images and segmentation masks from the new site. After preprocessing, the images and segmentation masks of uniform size are obtained, which are the preprocessed medical images and segmentation masks of the new site.

[0013] The segmentation model incremental training module uses the pre-processed medical images and segmentation masks of the new site and the U-Net segmentation model trained on the previous site as the pre-trained model for incremental training. During the incremental training process, a regularization term is defined based on the domain specificity of the segmentation model parameters. The domain-specific regularization term loss is used to constrain the update of model parameters with high domain specificity, resulting in an incrementally trained U-Net segmentation model.

[0014] The segmentation model prediction module uses the incrementally trained U-Net segmentation model for accurate prediction of medical image segmentation.

[0015] Furthermore, the initial site data import module specifically performs the following operations:

[0016] The medical image and the corresponding segmentation mask from the first site are resized, and the image and segmentation mask are proportionally resized as a whole so that the long side of the image and the segmentation mask is 384. If the length of the long side and the short side is the same, the size of all images and segmentation masks is 384×384. If the length of the short side is not 384, the short side of the image and the corresponding segmentation mask part are zero-filled so that the size of all input images and segmentation masks is a fixed size of 384×384. The preprocessed medical image x and segmentation mask y of the first site are obtained, where (x, y)∈D1, D1 represents the preprocessed data from the first site.

[0017] Furthermore, the segmentation model initialization training module includes a segmentation model construction module and a segmentation model training module, wherein:

[0018] The segmentation model building module builds a U-Net segmentation model for semantic segmentation. The model consists of a U-shaped encoder and decoder connected by a skip connection. The encoder is responsible for extracting image features and reducing the dimensionality, while the decoder resamples the low-dimensional features to the original size and generates the segmentation result.

[0019] The segmentation model training module uses the preprocessed medical image and segmentation mask of the first site to train the U-Net segmentation model, as the training loss function L of the segmentation loss seg The definition is as follows:

[0020]

[0021]

[0022] Where (x,y)∈D1 is the medical image and segmentation mask of the first site; F1(·) represents the U-Net segmentation model of the initial site, is the predicted output of the U-Net segmentation model, that is, the original segmentation model output, MSE(·,·) and DICE(·,·) represent the minimum square error loss and dice coefficient loss, respectively, which are specifically defined as follows:

[0023]

[0024]

[0025] Where N is and the number of elements in y, n is the element number, and y n The original segmentation model outputs are and the nth element of the segmentation mask y, yes The intersection of y and and |y| represent and the number of y elements;

[0026] After the above training loss function L seg After training, the initialized trained U-Net segmentation model F1(·) is obtained.

[0027] Furthermore, the segmentation model network parameter analysis module includes a domain difference simulation module and a domain specificity analysis module, wherein:

[0028] The domain difference simulation module is used to simulate the differences in medical images from different sites. These differences arise from the differences in the scanning instrument protocols used by different sites and are reflected in the brightness, contrast, saturation, and image noise of the medical images acquired by the scanning instruments. The domain difference simulation module implements the following data enhancement method on the input image:

[0029] a. Brightness data enhancement: adjust the brightness of the input image to [0.5, 1.5];

[0030] b. Contrast data enhancement: adjust the contrast of the input image to [0.5, 1.5];

[0031] c. Saturation data enhancement: adjust the saturation of the input image to [0.5, 1.5];

[0032] d. Gaussian noise data enhancement: add Gaussian noise to the input image with a variance of 0.1;

[0033] The domain-specific analysis module is used to introduce the data augmentation method of the domain difference simulation module, observe the difference in the U-Net segmentation model output before and after data augmentation, and then calculate the domain-specific characteristics of the network parameters of the U-Net segmentation model. The domain-specificity of the parameters of the U-Net segmentation model trained at each site needs to be calculated for use in incremental training of subsequent sites. The processing steps of the domain-specific analysis module are as follows:

[0034] 1) The preprocessed data (x,y)∈D from the tth site t As input, it is fed into the U-Net segmentation model F trained at the tth site t (·), get the original segmentation model output

[0035]

[0036] 2) For the preprocessed data (x,y)∈D from the tth site t , perform the following process: randomly select one of the four data augmentation modes of brightness data augmentation, contrast data augmentation, saturation data augmentation and Gaussian noise data augmentation from the domain difference simulation module, act on the medical image x from the t-th site, and then feed the data augmented image into the U-Net segmentation model F trained at the t-th site t (·); The above process is repeated H times to obtain H groups of domain deviation outputs:

[0037]

[0038] Where η h represents the mode of h-th data enhancement, represents the domain deviation output obtained after the h-th data augmentation, where h = {1, 2, 3, ..., H};

[0039] 3) Based on the original segmentation model output obtained in step 1) without domain difference simulation And the domain deviation output obtained in step 2) Use the least square error loss to measure the difference L in the segmentation model output dif , which is defined as follows:

[0040]

[0041] 4) The segmentation model output difference L obtained in step 3) dif As the loss function, the U-Net segmentation model F trained at the t-th site t (·) Perform a backpropagation to obtain the backpropagation gradient of each parameter as the domain specificity of the segmentation model parameters:

[0042]

[0043] Where i represents the U-Net segmentation model F trained at the tth site t The number of the parameters of (·), t represents the order of the sites, Ω i,t Represents the U-Net segmentation model F trained at the tth site t (·) network parameters θi i,t Domain specificity of the network parameters θ i,t The higher the domain specificity, the more sensitive the parameter is to the domain deviation of the input image, that is, it is easier to fit the domain specific features.

[0044] Furthermore, the new site data import module specifically performs the following operations:

[0045] 1) Import the U-Net segmentation model trained at the previous site, that is, the U-Net segmentation model F1(·), F2(·), ..., F trained at sites 1 to T-1 T-1 (·);

[0046] 2) Import the U-Net segmentation model F1(·), F2(·), ..., F of the previous site calculated by the segmentation model network parameter analysis module T-1 The parameter domain specificity of (·) {Ω i,1} i ,{Ω i,2} i ,…,{Ω i,t} i …,{Ω i,T-1} i , where i represents the serial number of the parameters of the U-Net segmentation model, {Ω i,t} i Represents the U-Net segmentation model F trained at the tth site t The domain-specific set of all parameters of (·);

[0047] 3) Resize the medical image and the corresponding segmentation mask from the new site T, and resize the image and segmentation mask proportionally as a whole so that the long side of the image and segmentation mask is 384. If the length of the long side and the short side are the same, the size of all images and segmentation masks is 384×384. If the length of the short side is not 384, the short side of the image and the corresponding segmentation mask are zero-filled so that the size of all input images and segmentation masks is a fixed size of 384×384. The preprocessed medical image x and segmentation mask y of the new site T are obtained, where (x, y)∈D T , D T represents the preprocessed data from the new site T.

[0048] Furthermore, the segmentation model incremental training module is based on the U-Net segmentation model F trained at the previous site T-1. T-1 (·) is used as a pre-trained model with the pre-processed medical image of the new site T and the segmentation mask (x,y)∈D T As training data, D T Represents preprocessed data from a new site T, incrementally training the U-Net segmentation model F T (·), including the following steps:

[0049] 1) Define the domain-specific regularization loss L reg , which is used to limit the update of network parameters with high domain specificity, preventing the U-Net segmentation model from overfitting to domain-specific features during incremental training, resulting in catastrophic forgetting of initial site knowledge:

[0050]

[0051] Where, Ω i.t Represents training of U-Net segmentation model F at site t t (·) network parameters θ i,t Domain specificity, θ i represents the network parameters of the U-Net segmentation model updated in the incremental training of site T;

[0052] 2) Define the incremental training loss function L total as follows:

[0053]

[0054]

[0055] L total =L seg +λL reg

[0056] Where, is the predicted output of the U-Net segmentation model for the new site, and λ represents the domain-specific regularization loss L reg Weights of incremental training loss function L total is the segmentation loss L seg and the weighted domain-specific regularization loss L reg the sum of;

[0057] 3) The U-Net segmentation model F trained at the previous site T-1 T-1 (·) As a pre-trained model, initialize the network parameter weights of the U-Net segmentation model to incrementally train the loss function L total As the loss function, the preprocessed medical image of the new site and the segmentation mask (x, y) ∈ D T As training data, perform incremental training;

[0058] 4) The incremental training is completed, and the U-Net segmentation model F that has completed the incremental training at the new site T is obtained T (·), this model will be used in the segmentation model prediction module to perform accurate prediction of medical image segmentation.

[0059] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0060] 1. This paper introduces a domain difference simulation module, which simulates possible domain deviations in the process of acquiring medical images through data augmentation methods to analyze the domain specificity of the trained medical image segmentation model parameters. In this way, the domain-specific knowledge and cross-domain invariant knowledge are decoupled during the incremental learning process, thereby solving the catastrophic forgetting problem in the cross-site incremental medical image segmentation task.

[0061] 2. The present invention defines a method for calculating the domain specificity of a network model. By defining and calculating the domain specificity for each parameter of a medical image segmentation model, and introducing a regularization term in the incremental training phase of the segmentation model to limit the update of network parameters with high domain specificity, the segmentation model is prevented from overfitting to domain-specific features, thereby improving the average performance of cross-site medical image segmentation and reducing the degree of knowledge forgetting during the incremental learning process.

[0062] 3. The present invention can overcome the privacy limitations of medical image data. In cross-site medical image segmentation tasks, there is no need to access data from previous sites, and the demand for computing resources such as CPU and GPU is low. The present invention is applicable to medical image segmentation models including U-Net networks, is plug-and-play, and has high availability. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 Flowchart of data processing of the system of the present invention.

[0064] Figure 2 This is the calculation flow chart of the segmentation model network parameter analysis module. DETAILED DESCRIPTION

[0065] The present invention will be described in further detail below with reference to the embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0066] This embodiment of the present invention utilizes a multi-site fundus image dataset, comprising medical image data from four sites: REFUGE-Zeiss-Visucam-500, REFUGE-Canon-CR-2, RIM-ONE-r3, and Drishti-GS. Due to the task of incremental cross-site medical image segmentation, image data from these four sites is acquired in the order described above. At each training step, only the image and segmentation mask for the current site are available; images and segmentation masks from previous sites are not.

[0067] like Figure 1 As shown, this embodiment discloses a cross-site medical image incremental segmentation system based on domain-specific regularization, including: an initial site data import module, a segmentation model initialization training module, a segmentation model network parameter analysis module, a new site data import module, a segmentation model incremental training module, and a segmentation model prediction module; the details are as follows:

[0068] 1) Initial site data import

[0069] The data of the first site (in this embodiment, the REFUGE-Zeiss-Visucam-500 site) is input into the initial site data import module of the present system. The initial site data import module is used to preprocess the data of the initial site, including the following steps: resizing the medical image and the corresponding segmentation mask from the first site, proportionally resizing the image and the segmentation mask as a whole so that the long side of the image and the segmentation mask is 384. If the long side and the short side are the same length, the size of all images and segmentation masks is 384×384. If the short side is not 384, the short side of the image and the corresponding segmentation mask are zero-padded so that the size of all input images and segmentation masks is a fixed size of 384×384, thereby obtaining the preprocessed medical image x and segmentation mask y of the first site, where (x, y)∈D1, D1 represents the preprocessed data from the first site.

[0070] 2) Segmentation model initialization training

[0071] The preprocessed medical image x and segmentation mask y of the first site D1 are fed into the segmentation model initialization training module of this system. The segmentation model construction module constructs a U-Net segmentation model for semantic segmentation. The model consists of a U-shaped encoder and decoder connected by a skip connection. The encoder is responsible for extracting image features and reducing the dimensionality, while the decoder resamples the low-dimensional features to the original size and generates the segmentation results. The segmentation model initialization training module includes a segmentation model construction module and a segmentation model training module. The specific steps are as follows:

[0072] 2.1) Segmentation model construction

[0073] The segmentation model building module constructs a U-Net segmentation model for semantic segmentation. This model consists of a U-shaped encoder and decoder connected by a skip connection. The encoder is responsible for extracting image features and reducing the dimensionality, while the decoder resamples the low-dimensional features to the original size and generates the segmentation result.

[0074] 2.2) Segmentation model training

[0075] The segmentation model training module uses the preprocessed medical image and segmentation mask of the first site to train the U-Net segmentation model, as the training loss function L of the segmentation loss seg The definition is as follows:

[0076]

[0077]

[0078] where (x,y)∈D1 is the preprocessed medical image and segmentation mask from the first site from the initial site data import module; F1(·) represents the U-Net segmentation model of the initial site, is the predicted output of the U-Net segmentation model, that is, the original segmentation model output, MSE(·,·) and DICE(·,·) represent the minimum square error loss and dice coefficient loss, respectively, which are specifically defined as follows:

[0079]

[0080]

[0081] Where N is and the number of elements in y, n is the element number, and y n The original segmentation model outputs are and the nth element of the segmentation mask y, yes The intersection of y and and |y| represent and the number of y elements.

[0082] After the above training loss function L seg After training, the initialized trained U-Net segmentation model F1(·) is obtained.

[0083] 3) Segmentation model network parameter analysis

[0084] The initialized trained U-Net segmentation model F1(·) and the preprocessed medical image x and segmentation mask y of the first site D1 are fed into the segmentation model network parameter analysis module of the present system. The segmentation model network parameter analysis module includes a domain difference simulation module and a domain specificity analysis module, wherein:

[0085] The domain difference simulation module simulates differences between medical images from different sites. These differences arise from differences in the scanning instrument protocols used by different sites, and are reflected in the brightness, contrast, saturation, and image noise of the medical images acquired by the scanning instruments. Furthermore, the domain difference simulation module applies the following data enhancement methods to the input images: brightness data enhancement: adjusting the brightness of the input image to a range of [0.5, 1.5]; contrast data enhancement: adjusting the contrast of the input image to a range of [0.5, 1.5]; saturation data enhancement: adjusting the saturation of the input image to a range of [0.5, 1.5]; and Gaussian noise data enhancement: adding Gaussian noise to the input image with a variance of 0.1.

[0086] like Figure 2 As shown, the domain-specific analysis module is used to introduce the data augmentation method of the domain difference simulation module mentioned above, observe the difference between the original model output and the domain deviation output before and after data augmentation, and then calculate the domain specificity of the network parameters of the U-Net segmentation model. The domain specificity of the parameters of the U-Net segmentation model trained at each site needs to be calculated for incremental training at subsequent sites. The processing steps of the domain-specific analysis module for the segmentation model trained at the first site are as follows:

[0087] 3.1) Original output calculation

[0088] The preprocessed data (x,y)∈D1 from the first site is fed into the U-Net segmentation model F1(·) trained at the first site to obtain the original segmentation model output

[0089]

[0090] 3.2) Domain Deviation Output Calculation

[0091] For the preprocessed data (x, y) ∈ D1 from the first site, the following process is performed: one of the four data augmentation modes (brightness data augmentation, contrast data augmentation, saturation data augmentation, and Gaussian noise data augmentation) from the domain difference simulation module is randomly selected and applied to the medical image x from the first site. The augmented image is then fed into the U-Net segmentation model F1(·) trained at the first site. The above process is repeated H times to obtain H sets of domain deviation outputs. In this example, H = 8:

[0092]

[0093] Among them, η h represents the mode of h-th data enhancement, represents the corresponding domain deviation output obtained after the h-th data augmentation input, where h = {1, 2, 3, …, H}.

[0094] 3.3) Segmentation model output difference calculation

[0095] Based on the original segmentation model output obtained in step 3.1) without domain difference simulation And the domain deviation output obtained in step 3.2) Use the least square error loss to measure the difference L in the segmentation model output dif The definition is as follows:

[0096]

[0097] 3.4) Domain-specific calculation of segmentation model parameters

[0098] The segmentation model output difference L obtained in step 3.3) dif As the loss function, the U-Net segmentation model F1(·) trained at the first site is back-propagated once to obtain the back-propagated gradient of each parameter as the domain specificity of the segmentation model parameters:

[0099]

[0100] Where i represents the parameter number of the U-Net segmentation model F1(·) trained at the first site, 1 represents the site order, and Ω i,1 represents the network parameters θ of the U-Net segmentation model F1(·) trained at the first site i,1 Domain specificity of the network parameters θ i,1 The higher the domain specificity, the more sensitive the parameter is to the domain deviation of the input image, that is, it is easier to fit domain-specific features such as brightness, contrast, saturation and image noise of medical images.

[0101] 4) Import new site data

[0102] The U-Net segmentation model trained on the previous site, the parameter domain specificity of each segmentation model, the medical image of the new site, and the segmentation mask are input into the new site data import module of this system. Since data of multiple new sites will be continuously acquired during the cross-site incremental learning process, this embodiment takes the new site T (1<T≤4) as an example. The processing steps of the new site data import module are as follows:

[0103] 4-1) Importing the previous site segmentation model

[0104] Import the U-Net segmentation model trained at the previous site, that is, the U-Net segmentation model F1(·), F2(·), ..., F trained at site 1 to site T-1 T-1 (·);

[0105] 4-2) Previous site parameter domain specific import

[0106] Import the segmentation model F1(·), F2(·),…, F of the previous site calculated by the segmentation model network parameter analysis module T-1 The parameter domain specificity of (·) {Ω i,1} i ,{Ω i,2} i ,…,{Ω i,t} i …,{Ω i,T-1} i , where i represents the serial number of the parameters of the U-Net segmentation model, {Ω i,t} i Represents the U-Net segmentation model F trained at the tth site t The domain-specific set of all parameters of (·).

[0107] 4-3) Importing new site data

[0108] The medical images and corresponding segmentation masks from the new site are resized. The image and segmentation mask are proportionally resized as a whole so that the long side of the image and segmentation mask is 384. If the long side and the short side are the same length, the size of all images and segmentation masks is 384×384. If the short side is not 384, the short side of the image and the corresponding segmentation mask are zero-filled so that the size of all input images and segmentation masks is a fixed size of 384×384. The preprocessed medical image x and segmentation mask y of the new site T are obtained, where (x, y)∈D T .

[0109] 5) Incremental training of segmentation model

[0110] The data imported in step 4) is input into the segmentation model incremental training module of this system. The segmentation model incremental training module is based on the U-Net segmentation model F trained at the previous site T-1. T-1 (·) is used as a pre-trained model with the pre-processed medical image of the new site T and the segmentation mask (x,y)∈D T As training data, incrementally train the U-Net segmentation model F T (·), including the following steps:

[0111] 5-1) Define domain-specific regularization loss

[0112] Domain-specific regularization loss L reg It is used to limit the update of network parameters with high domain specificity to prevent the U-Net segmentation model from overfitting to domain specific features during incremental training, resulting in catastrophic forgetting of previous site knowledge:

[0113]

[0114] Among them, Ω i.t Represents training of U-Net segmentation model F at site t t (·) network parameters θ i,t The domain specificity of θ i Represents the network parameters of the U-Net segmentation model updated in incremental training at site T.

[0115] 5-2) Define incremental training loss function

[0116] Incremental training loss function L total is defined as follows:

[0117]

[0118]

[0119] L total =L seg +λL reg

[0120] Where (x,y)∈D T Import the pre-processed medical images and segmentation masks from the new site into the module for data from the new site; F T (·) represents the U-Net segmentation model incrementally trained on the new site T, is the predicted output of the U-Net segmentation model for the new site. λ represents the domain-specific regularization loss L reg The weight of λ is 10 in this embodiment. The incremental training loss function L total is the segmentation loss L segand the weighted domain-specific regularization loss L reg The sum of .

[0121] 5-3) Incremental training model

[0122] The U-Net segmentation model F trained at the previous site T-1 T-1 (·) As a pre-trained model, initialize the network parameter weights of the U-Net segmentation model to incrementally train the loss function L total As the loss function, the preprocessed medical image of the new site and the segmentation mask (x, y) ∈ D T As training data, incremental training is performed. After the incremental training is completed, the U-Net segmentation model F that completes the incremental training at the new site T is obtained. T (·).

[0123] 6) Analysis of network parameters of the segmentation model for the new site

[0124] The U-Net segmentation model F that completes incremental training at site T T (·) and the pre-processed medical image x and segmentation mask y from site T are fed into the segmentation model network parameter analysis module of this system. The specific steps are as follows:

[0125] 6-1) Calculate the original output

[0126] The data (x,y)∈D from the Tth site T As input, it is fed into the U-Net segmentation model F trained at the Tth site T (·), get the original segmentation model output

[0127]

[0128] 6-2) Calculation domain deviation output

[0129] For the data (x,y)∈D from the Tth site T , perform the following process: randomly select one of the four data augmentation modes from the domain difference simulation module: brightness data augmentation, contrast data augmentation, saturation data augmentation, and Gaussian noise data augmentation, and apply it to the medical image x from the T-th site, and then feed the data augmented image into the U-Net segmentation model F trained at the T-th site T The above process is repeated H times to obtain H groups of domain deviation outputs. In this embodiment, H=8:

[0130]

[0131] Among them, η h represents the mode of h-th data enhancement, represents the corresponding domain deviation output obtained after the h-th data augmentation input, where h = {1, 2, 3, …, H}.

[0132] 6-3) Calculate the difference in segmentation model output

[0133] Based on the original segmentation model output obtained in step 6-1) without domain difference simulation And the domain deviation output obtained in step 6-2) Use the least square error loss to measure the difference L in the segmentation model output dif The definition is as follows:

[0134]

[0135] 6-4) Computing domain-specific segmentation model parameters

[0136] The U-Net segmentation model output difference L obtained in step 6-3) dif As the loss function, the segmentation model F trained at the Tth site T (·) Perform a backpropagation to obtain the backpropagation gradient of each parameter as the domain specificity of the segmentation model parameters:

[0137]

[0138] Where i represents the U-Net segmentation model F trained at the Tth site T The number of the parameters of (·), T represents the order of the sites, Ω i,T Represents the segmentation model F trained at the Tth site T (·) network parameters θ i,T Domain specificity of the network parameters θ i,T The higher the domain specificity, the more sensitive the parameter is to the domain deviation of the input image, that is, it is easier to fit domain-specific features such as brightness, contrast, saturation and image noise of medical images.

[0139] After the segmentation model network parameter analysis of the T-th site is completed, if there are still sites in the data set of the embodiment that have not completed training, that is, when T<4, repeat steps 4) to 6) and continue to import data from the T+1-th site to complete the segmentation model incremental training and segmentation model network parameter analysis of the T+1-th site.

[0140] If all sites of the data set of the embodiment have been trained, that is, when T=4, step 7) is executed.

[0141] 7) Segmentation model prediction

[0142] The U-Net segmentation model trained in step 6) is used in the segmentation model prediction module to perform accurate prediction of medical image segmentation.

[0143] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A cross-site medical image incremental segmentation system based on domain-specific regularization, characterized by: include: The initial site data import module is used to obtain data from the first site, including medical images and segmentation masks, and preprocess the data to obtain images and segmentation masks of uniform size; The segmentation model initialization training module is used to build a U-Net segmentation model and use the preprocessed medical image and segmentation mask of the first site to initialize the training segmentation model to obtain the initialized trained U-Net segmentation model; The segmentation model network parameter analysis module is used to analyze the parameter domain specificity of the trained U-Net segmentation model. The module enhances the brightness, contrast, and saturation of the medical images used to train the U-Net segmentation model. The medical images before and after enhancement are fed into the U-Net segmentation model for prediction to obtain two outputs. The difference between the two outputs of the U-Net segmentation model is calculated and back-propagated to obtain the segmentation model gradient, which is the parameter domain specificity of the U-Net segmentation model. The new site data import module is used to import the U-Net segmentation model trained on the previous site and the parameter domain specificity of each U-Net segmentation model, and import the medical images and segmentation masks from the new site. After preprocessing, the images and segmentation masks of uniform size are obtained, which are the preprocessed medical images and segmentation masks of the new site. The segmentation model incremental training module uses the pre-processed medical images and segmentation masks of the new site and the U-Net segmentation model trained on the previous site as the pre-trained model for incremental training. During the incremental training process, a regularization term is defined based on the domain specificity of the segmentation model parameters. The domain-specific regularization term loss is used to constrain the update of model parameters with high domain specificity, resulting in an incrementally trained U-Net segmentation model. The segmentation model prediction module uses the incrementally trained U-Net segmentation model for accurate prediction of medical image segmentation.

2. The cross-site medical image incremental segmentation system based on domain-specific regularization according to claim 1, characterized in that The initial site data import module specifically performs the following operations: The medical image and the corresponding segmentation mask from the first site are resized, and the image and segmentation mask are proportionally resized as a whole so that the long side of the image and the segmentation mask is 384. If the length of the long side and the short side is the same, the size of all images and segmentation masks is 384×384. If the length of the short side is not 384, the short side of the image and the corresponding segmentation mask part are zero-filled so that the size of all input images and segmentation masks is a fixed size of 384×384. The preprocessed medical image x and segmentation mask y of the first site are obtained, where (x, y)∈D1, D1 represents the preprocessed data from the first site.

3. The cross-site medical image incremental segmentation system based on domain-specific regularization according to claim 1, characterized in that The segmentation model initialization training module includes a segmentation model construction module and a segmentation model training module, wherein: The segmentation model building module builds a U-Net segmentation model for semantic segmentation. The model consists of a U-shaped encoder and decoder connected by a skip connection. The encoder is responsible for extracting image features and reducing the dimensionality, while the decoder resamples the low-dimensional features to the original size and generates the segmentation result. The segmentation model training module uses the preprocessed medical image and segmentation mask of the first site to train the U-Net segmentation model, as the training loss function L of the segmentation loss seg The definition is as follows: Where (x,y)∈D1 is the medical image and segmentation mask of the first site; F1(·) represents the U-Net segmentation model of the initial site, is the predicted output of the U-Net segmentation model, that is, the original segmentation model output, MSE(·,·) and DICE(·,·) represent the minimum square error loss and dice coefficient loss, respectively, which are specifically defined as follows: Where N is and the number of elements in y, n is the element number, and y n The original segmentation model outputs are and the nth element of the segmentation mask y, yes The intersection of y and and |y| represent and the number of y elements; After the above training loss function L seg After training, the initialized trained U-Net segmentation model F1(·) is obtained.

4. The cross-site medical image incremental segmentation system based on domain-specific regularization according to claim 1, characterized in that: The segmentation model network parameter analysis module includes a domain difference simulation module and a domain specificity analysis module, wherein: The domain difference simulation module is used to simulate the differences in medical images from different sites. These differences arise from the differences in the scanning instrument protocols used by different sites and are reflected in the brightness, contrast, saturation, and image noise of the medical images acquired by the scanning instruments. The domain difference simulation module implements the following data enhancement method on the input image: a. Brightness data enhancement: adjust the brightness of the input image to [0.5, 1.5]; b. Contrast data enhancement: adjust the contrast of the input image to [0.5, 1.5]; c. Saturation data enhancement: adjust the saturation of the input image to [0.5, 1.5]; d. Gaussian noise data enhancement: add Gaussian noise to the input image with a variance of 0.1; The domain-specific analysis module is used to introduce the data augmentation method of the domain difference simulation module, observe the difference in the U-Net segmentation model output before and after data augmentation, and then calculate the domain-specific characteristics of the network parameters of the U-Net segmentation model. The domain-specificity of the parameters of the U-Net segmentation model trained at each site needs to be calculated for use in incremental training of subsequent sites. The processing steps of the domain-specific analysis module are as follows: 1) The preprocessed data (x,y)∈D from the tth site t As input, it is fed into the U-Net segmentation model F trained at the tth site t (·), get the original segmentation model output 2) For the preprocessed data (x,y)∈D from the tth site t , perform the following process: randomly select one of the four data augmentation modes of brightness data augmentation, contrast data augmentation, saturation data augmentation and Gaussian noise data augmentation from the domain difference simulation module, act on the medical image x from the t-th site, and then feed the data augmented image into the U-Net segmentation model F trained at the t-th site t (·); The above process is repeated H times to obtain H groups of domain deviation outputs: Where η h represents the mode of h-th data enhancement, represents the domain deviation output obtained after the h-th data augmentation, where h = {1, 2, 3, ..., H}; 3) Based on the original segmentation model output obtained in step 1) without domain difference simulation And the domain deviation output obtained in step 2) Use the least square error loss to measure the difference L in the segmentation model output dif , which is defined as follows: 4) The segmentation model output difference L obtained in step 3) dif As the loss function, the U-Net segmentation model F trained at the t-th site t (·) Perform a backpropagation to obtain the backpropagation gradient of each parameter as the domain specificity of the segmentation model parameters: Where i represents the U-Net segmentation model F trained at the tth site t The number of the parameters of (·), t represents the order of the sites, Ω i,t Represents the U-Net segmentation model F trained at the tth site t (·) network parameters θ i,t Domain specificity of the network parameters θ i,t The higher the domain specificity, the more sensitive the parameter is to the domain deviation of the input image, that is, it is easier to fit the domain specific features.

5. The cross-site medical image incremental segmentation system based on domain-specific regularization according to claim 1, characterized in that: The new site data import module specifically performs the following operations: 1) Import the U-Net segmentation model trained at the previous site, that is, the U-Net segmentation model F1(·), F2(·), ..., F trained at sites 1 to T-1 T-1 (·); 2) Import the U-Net segmentation model F1(·), F2(·), ..., F of the previous site calculated by the segmentation model network parameter analysis module T-1 The parameter domain specificity of (·) {Ω i,1 } i ,{Ω i,2 } i ,…,{Ω i,t } i …,{Ω i,T-1 } i , where i represents the serial number of the parameters of the U-Net segmentation model, {Ω i,t } i Represents the U-Net segmentation model F trained at the tth site t The domain-specific set of all parameters of (·); 3) Resize the medical image and the corresponding segmentation mask from the new site T, and resize the image and segmentation mask proportionally as a whole so that the long side of the image and segmentation mask is 384. If the length of the long side and the short side are the same, the size of all images and segmentation masks is 384×384. If the length of the short side is not 384, the short side of the image and the corresponding segmentation mask are zero-filled so that the size of all input images and segmentation masks is a fixed size of 384×384. The preprocessed medical image x and segmentation mask y of the new site T are obtained, where (x, y)∈D T , D T represents the preprocessed data from the new site T.

6. The cross-site medical image incremental segmentation system based on domain-specific regularization according to claim 1, characterized in that: The segmentation model incremental training module is based on the U-Net segmentation model F trained at the previous site T-1. T-1 (·) is used as a pre-trained model with the pre-processed medical image of the new site T and the segmentation mask (x,y)∈D T As training data, D T Represents preprocessed data from a new site T, incrementally training the U-Net segmentation model F T (·), including the following steps: 1) Define the domain-specific regularization loss L reg , which is used to limit the update of network parameters with high domain specificity, preventing the U-Net segmentation model from overfitting to domain-specific features during incremental training, resulting in catastrophic forgetting of initial site knowledge: Where, Ω i.t Represents training of U-Net segmentation model F at site t t (·) network parameters θ i,t Domain specificity, θ i represents the network parameters of the U-Net segmentation model updated in the incremental training of site T; 2) Define the incremental training loss function L total as follows: THE total =L seg +λL reg Where, is the predicted output of the U-Net segmentation model for the new site, and λ represents the domain-specific regularization loss L reg Weights of incremental training loss function L total is the segmentation loss L seg and the weighted domain-specific regularization loss L reg the sum of; 3) The U-Net segmentation model F trained at the previous site T-1 T-1 (·) As a pre-trained model, initialize the network parameter weights of the U-Net segmentation model to incrementally train the loss function L total As the loss function, the preprocessed medical image of the new site and the segmentation mask (x, y) ∈ D T As training data, perform incremental training; 4) The incremental training is completed, and the U-Net segmentation model F that has completed the incremental training at the new site T is obtained T (·), this model will be used in the segmentation model prediction module to perform accurate prediction of medical image segmentation.

Images