Medical image adaptive segmentation method and device, medium and product
By introducing an adaptive learning rate adjustment method into the federated learning network, the problem of data distribution bias in different hospitals is solved, and rapid adaptive segmentation of medical images in external hospitals is realized, segmentation efficiency is improved and data privacy is protected.
Patent Information
- Application Number
- CN202510378994.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-27
- Publication Date
- 2025-07-11
AI Technical Summary
In medical image segmentation, due to the heterogeneity of data distribution in different hospitals, it is difficult to quickly deploy to external hospitals that are not involved in internal training, reducing the efficiency of medical image segmentation.
By introducing an adaptive learning rate adjustment method in the federated learning network, the multi-center unbiased prototype group, adaptive learning rate and model weight of the intraditional servers are used to build an extradomain model to realize the rapid adaptive segmentation of medical images and avoid data leakage.
It realizes rapid and efficient segmentation of medical images from external hospitals without relying on in-domain hospital data, improving segmentation efficiency and protecting data privacy.
Smart Images
Figure CN120298431A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of medical image segmentation, and particularly to a medical image adaptive segmentation method, device, medium and product. Background Art
[0002] A federated model refers to a machine learning model trained under the framework of federated learning. Federated learning is a distributed machine learning framework that allows multiple clients (such as mobile devices, hospitals, enterprises, etc.) to collaboratively train a global model without sharing local data. This framework is particularly suitable for scenarios of protecting data privacy and processing distributed data, and thus has potential application value in the field of medical image segmentation.
[0003] Medical image segmentation is a key area in the field of healthcare, which helps in the accurate diagnosis and treatment of various diseases. In medical image analysis, especially in the segmentation task, federated learning, as a key technology, can perform collaborative model training across multiple data sources without sharing the original data. This is crucial for processing sensitive medical data as it can improve the model's performance by leveraging data scattered across different institutions while protecting patient privacy.
[0004] Although federated learning has great potential in theory, it still faces some challenges in practice. Existing federated learning-based medical image segmentation methods usually assume that the data distributions of all participating parties are the same, which does not conform to the actual situation. Due to differences in imaging devices, patient distributions, and physicians' preferences among different hospital institutions, the medical images taken have strong distribution heterogeneity. This heterogeneity results in the difficulty of quickly deploying the federated model trained within the domain to external hospitals without dense labels for medical images that did not participate in the internal training, further reducing the segmentation efficiency of medical images. Summary of the Invention
[0005] The purpose of the present application is to provide a medical image adaptive segmentation method, device, medium and product, which can achieve the rapid adaptation of external hospitals that did not participate in the internal training and have no labels for medical images, as well as the efficient and rapid segmentation of medical images, and improve the segmentation efficiency of medical images by solving the problem of data distribution bias for medical image data in different hospitals.
[0006] To achieve the above purpose, the present application provides the following solutions:
[0007] In a first aspect, the present application provides a medical image adaptive segmentation method, and the medical image adaptive segmentation method is applied to a federated learning network; the federated learning network includes: a central server and multiple in-domain servers; the in-domain servers correspond to in-domain hospitals one by one; at least one out-of-domain server is connected to the central server;
[0008] The medical image adaptive segmentation method includes:
[0009] Call the model to be deployed and source domain knowledge, and configure the parameters of the model to be deployed on the local initial model of the off-domain server to obtain the off-domain initial model; the model to be deployed is obtained by the central server aggregating multiple in-domain models based on the model weights of each in-domain model; any in-domain model is obtained by the off-domain server where it is located based on the medical image set and label set of the corresponding in-domain hospital, and using the adaptive learning rate adjustment method to train the local initial model; the local initial model includes an encoder and a decoder; the output end of the encoder is connected to the input end of the decoder; the source domain knowledge includes: the multi-center unbiased prototype group, adaptive learning rate, and model weight of each in-domain model;
[0010] Obtain the medical image set corresponding to the off-domain server's corresponding hospital as the off-domain medical image set;
[0011] Use the off-domain medical image set to train the off-domain initial model to obtain the test adaptive loss;
[0012] Use the test adaptive loss and the adaptive learning rate to update the model weights of multiple in-domain models in the off-domain initial model to obtain the off-domain model;
[0013] Obtain the medical image corresponding to the off-domain server's corresponding hospital as the image to be segmented;
[0014] Input the image to be segmented into the off-domain model to obtain the segmentation result of the image to be segmented.
[0015] Optionally, before calling the model to be deployed and source domain knowledge, and configuring the parameters of the model to be deployed on the local initial model of the off-domain server to obtain the off-domain initial model, it further includes:
[0016] Determine any in-domain server as the current in-domain server;
[0017] Determine the local initial model of the current in-domain server as the current local initial model;
[0018] Obtain the medical image set corresponding to the current in-domain server's corresponding hospital as the in-domain medical image set;
[0019] Obtain the label set corresponding to the in-domain medical image set as the in-domain label set;
[0020] Use the in-domain medical image set to train the current local initial model to obtain the current in-domain model and the current in-domain model parameters; the current in-domain model parameters include: the multi-center unbiased prototype group and model weight of the current in-domain model;
[0021] Input the in-domain medical image set into the current in-domain model to obtain an in-domain pseudo-label set;
[0022] Use the in-domain pseudo-label set, the in-domain label set, and the model weights of the current in-domain model to determine the adaptive learning rate of the current in-domain model;
[0023] Update the current in-domain server, and return to the step of "determining the local initial model of the current in-domain server as the current local initial model" until all in-domain servers are traversed, obtaining the in-domain models of all in-domain hospitals, as well as the multi-center unbiased prototype groups, model weights, and adaptive learning rates of each in-domain model.
[0024] Optionally, use the in-domain medical image set to train the current local initial model to obtain the current in-domain model and the current in-domain model parameters, including:
[0025] Determine the current in-domain model at the 0th iteration as the current local initial model;
[0026] Let the iteration number t = 1;
[0027] Use the encoder in the current in-domain model at the (t - 1)th iteration to extract features from the in-domain medical image set to obtain the feature set at the tth iteration;
[0028] Randomly sample the feature set according to the in-domain label set to obtain the memory queue at the tth iteration;
[0029] Use the K-means algorithm to cluster the memory queue at the tth iteration to obtain the multi-center unbiased prototype group at the tth iteration;
[0030] Input the memory queue at the tth iteration into the decoder of the current in-domain model at the (t - 1)th iteration to obtain the in-domain pseudo-label set at the tth iteration;
[0031] Based on the feature set at the tth iteration, the memory queue at the tth iteration, the multi-center unbiased prototype group at the tth iteration, the in-domain label set, and the in-domain pseudo-label set at the tth iteration, determine the self-supervised loss at the tth iteration;
[0032] According to the self-supervised loss at the tth iteration, adjust the parameters of the current in-domain model at the (t - 1)th iteration to obtain the current in-domain model and the current in-domain model parameters at the tth iteration;
[0033] Increment the value of the iteration count \(t\) by 1, and return to the step of "using the encoder in the current domain model at the \((t - 1)\) - th iteration to extract features from the in - domain medical image set to obtain the feature set at the \(t\) - th iteration", until the iteration stop condition is reached. Determine the current domain model at the \(t\) - th iteration as the current domain model, and the parameters of the current domain model at the \(t\) - th iteration as the current domain model parameters; the iteration stop condition includes that the self - supervised loss is less than a preset loss threshold and the iteration count reaches a preset iteration count.
[0034] Optionally, the model weight of the current domain model at the \(t\) - th iteration is:
[0035]
[0036] where \(\theta\) t is the model weight of the current domain model at the \(t\) - th iteration; \(\theta\) t-1 is the model weight of the current domain model at the \((t - 1)\) - th iteration; \(\alpha_1\) is a learnable adaptive learning rate; \(L\) ssp is the self - supervised loss at the \(t\) - th iteration; is the partial derivative of the model weight \(\theta\).
[0037] Optionally, the adaptive learning rate of the current domain model is:
[0038]
[0039] where \(\alpha\) is the adaptive learning rate of the current domain model; \(\alpha_1\) is a learnable adaptive learning rate; \(\eta\) is the learning rate for adjusting the adaptive learning rate; \(y\) in is the in - domain label set; \(L\) Dice is the segmentation metric loss; \(f\) dec is the decoder; \(f\) enc is the encoder; \(x\) in is the in - domain hospital image; \(\theta\) t is the model weight of the current domain model at the \(t\) - th iteration; is the partial derivative of the learnable adaptive learning rate \(\alpha_1\).
[0040] Optionally, the test adaptive loss is:
[0041] \(L\) tta =L E +\(\gamma\) tta (L MCPCL +L CBCL +L cons );
[0042]
[0043] \(L\) cons =\(\vert\vert f\)enc (x w ) - f enc (x s ) || 2 ;
[0044]
[0045] Among them, L tta is the test adaptive loss; L E is the entropy regularization loss; γ tta is the trade-off coefficient; L MCPCL is the multi-prototype contrast learning loss; L CBCL is the intra-batch contrast learning loss; L cons is the consistency loss; E f,c is the pixel-level feature set of out-of-domain hospital images; P + is the unbiased prototype group with the same category as the pixel-level feature set E f,c of out-of-domain hospital images; P - is the unbiased prototype group with different categories from the pixel-level feature set E f,c of out-of-domain hospital images; E + is the feature set with the same category as the pixel-level feature set E f,c in the multi-prototype center of out-of-domain hospital images; E - is the feature set with different categories from the pixel-level feature set E f,c in the multi-prototype center of out-of-domain hospital images; γ is the temperature coefficient; B + is the batch-level feature with the same category as the pixel-level feature set E f,c of out-of-domain hospital images; B - is the batch-level feature with different categories from the pixel-level feature set E f,c of out-of-domain hospital images; E1 + is the pixel-level feature set of out-of-domain hospital images in the same category batch as the currently sampled pixel-level feature set E f,c of out-of-domain hospital images; E1 - is the pixel-level feature set of out-of-domain hospital images in a different category batch from the currently sampled pixel-level feature set E f,c of out-of-domain hospital images; is the pixel-level feature set of the current batch of out-of-domain hospital images; x w is the out-of-domain hospital medical image obtained through weak augmentation; x s is the out-of-domain hospital medical image obtained through strong augmentation; is the pseudo-label of the out-of-domain hospital image; cross_entropy(·) is the cross-entropy loss; f dec is the decoder; H is the height; W is the width; H×W is the spatial size; is the feature of the out-of-domain hospital image with a spatial index of 0; The features of the out-of-domain hospital images with a spatial index of 1; The features of the out-of-domain hospital images with a spatial index of H×W; The feature set of the out-of-domain hospital images.
[0046] Optionally, the model weights of multiple in-domain models in the updated out-of-domain initial model are:
[0047]
[0048] where θ is the model weight of multiple in-domain models in the updated out-of-domain initial model; θ0 is the model weight of the in-domain model; is the partial derivative of the model weight θ0 of the in-domain model; α is the adaptive learning rate; L tta is the test adaptive loss.
[0049] In a second aspect, the present application provides a computer device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement the medical image adaptive segmentation method described in any one of the above.
[0050] In a third aspect, the present application provides a computer-readable storage medium, on which a computer program is stored, wherein the computer program, when executed by a processor, implements the medical image adaptive segmentation method described in any one of the above.
[0051] In a fourth aspect, the present application provides a computer program product, including a computer program, wherein the computer program, when executed by a processor, implements the medical image adaptive segmentation method described in any one of the above.
[0052] According to the specific embodiments provided by the present application, the present application has the following technical effects:
[0053] The present application provides a method, device, medium and product for adaptive segmentation of medical images. By introducing an adaptive learning rate adjustment method in federated learning, the problem of distribution bias of medical image data in different hospitals is solved, the distribution difference of medical image data is reduced, and the rapid adaptation of external hospitals that do not participate in internal training and have no labels for medical images is achieved. The out-of-domain model constructed by the federated learning and the adaptive learning rate adjustment method can obtain the medical image segmentation results of the target out-of-domain hospital only through the medical images of the target out-of-domain hospital without relying on any medical images and labels of in-domain hospitals. This not only avoids the leakage of medical image data of in-domain hospitals and improves the confidentiality of medical image data, but also realizes the efficient and rapid segmentation of medical images of the target out-of-domain hospital, significantly improves the segmentation efficiency of medical images, and greatly promotes the application of the federated adaptive model in actual clinical auxiliary diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.
[0055] Figure 1 It is a schematic flowchart of a method for adaptive segmentation of medical images provided by an embodiment of the present application;
[0056] Figure 2 It is a schematic flowchart of constructing an in-domain model provided by an embodiment of the present application;
[0057] Figure 3 It is a schematic flowchart of refining the determination of the current in-domain model and the current in-domain model parameters provided by an embodiment of the present application;
[0058] Figure 4 It is a schematic diagram of constructing an out-of-domain model provided by an embodiment of the present application;
[0059] Figure 5 It is a schematic diagram of the structure of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS
[0060] The following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, rather than all embodiments. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.
[0061] To make the above objects, features, and advantages of the present application more obvious and understandable, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0062] In an exemplary embodiment of the present application, a method for adaptive segmentation of medical images is provided. This method is applied to a federated learning network. The federated learning network includes: a central server and multiple in-domain servers. Among them, the in-domain servers correspond one-to-one with in-domain hospitals, and at least one out-of-domain server is connected to the central server.
[0063] A method for adaptive segmentation of medical images, as Figure 1 shown, includes the following steps 101 to step 106.
[0064] Wherein:
[0065] Step 101: Invoke the model to be deployed and source domain knowledge, and configure the parameters of the model to be deployed in the local initial model of the out-of-domain server to obtain the out-of-domain initial model.
[0066] The model to be deployed is obtained by the central server aggregating multiple in-domain models based on the model weights of each in-domain model. Any in-domain model is obtained by the out-of-domain server where it is located training the local initial model based on the medical image set and label set of the corresponding in-domain hospital using an adaptive learning rate adjustment method. And the local initial model includes an encoder and a decoder, and the output end of the encoder is connected to the input end of the decoder. The source domain knowledge includes: the multi-center unbiased prototype group, adaptive learning rate, and model weight of each in-domain model.
[0067] Step 102: Obtain the medical image set of the hospital corresponding to the out-of-domain server as the out-of-domain medical image set.
[0068] Step 103: Use the out-of-domain medical image set to train the out-of-domain initial model to obtain the test adaptive loss.
[0069] Step 104: Use the test adaptive loss and the adaptive learning rate to update the model weights of multiple in-domain models in the out-of-domain initial model to obtain the out-of-domain model.
[0070] Step 105: Obtain the medical image of the hospital corresponding to the out-of-domain server as the image to be segmented.
[0071] Step 106: Input the image to be segmented into the out-of-domain model to obtain the segmentation result of the image to be segmented.
[0072] Implement the above-mentioned Step 101 to Step 106. Based on the federated learning network, first construct multiple in-domain models through the adaptive learning rate adjustment method and obtain the source domain knowledge. Then aggregate the multiple in-domain models into the model to be deployed, call the model to be deployed and the source domain knowledge, and configure the parameters of the model to be deployed on the out-of-domain server to obtain the initial out-of-domain local model. After training, obtain the test adaptive loss. Finally, according to the test adaptive loss and the adaptive learning rate in the source domain knowledge, update the model weights of the multiple in-domain models in the initial out-of-domain model to finally obtain the out-of-domain model.
[0073] Based on the above-mentioned Step 101 to Step 106, this application introduces the adaptive learning rate adjustment method in the federated learning network, which not only solves the problem of distribution bias of medical image data in different hospitals, reduces the distribution difference of medical image data, realizes the rapid adaptation of external hospitals that have not participated in internal training and have no labels for medical images, but also can obtain the medical image segmentation results of the target out-of-domain hospital without relying on any in-domain hospital medical images and labels, avoiding the leakage of in-domain hospital medical image data, improving the confidentiality of medical image data, and significantly improving the segmentation efficiency of medical images.
[0074] In another exemplary embodiment of this application, before calling the model to be deployed and the source domain knowledge, and configuring the parameters of the model to be deployed on the out-of-domain server local initial model to obtain the initial out-of-domain model, that is, before Step 101, as Figure 2 shown, it further includes Step 201 to Step 208. Among them:
[0075] Step 201: Determine any in-domain server as the current in-domain server.
[0076] Step 202: Determine the local initial model of the current in-domain server as the current local initial model.
[0077] Step 203: Obtain the medical image set corresponding to the current in-domain server as the in-domain medical image set.
[0078] Step 204: Obtain the label set corresponding to the in-domain medical image set as the in-domain label set.
[0079] Step 205: Use the in-domain medical image set to train the current local initial model to obtain the current in-domain model and the current in-domain model parameters. The current in-domain model parameters include: the multi-center unbiased prototype group and model weights of the current in-domain model.
[0080] Step 206: Input the in-domain medical image set into the current in-domain model to obtain the in-domain pseudo-label set.
[0081] Step 207: Determine the adaptive learning rate of the current in-domain model by using the in-domain pseudo-label set, the in-domain label set, and the model weights of the current in-domain model.
[0082] Step 208: Update the current in-domain server and return to Step 202 until all in-domain servers are traversed, obtaining the in-domain models of all in-domain hospitals, as well as the multi-center unbiased prototype groups, model weights, and adaptive learning rates of each in-domain model.
[0083] Based on the above Steps 201 to 208, not only the in-domain models of all in-domain hospitals are obtained, but also the multi-center unbiased prototype groups, model weights, and adaptive learning rates of each in-domain model are obtained.
[0084] In another exemplary embodiment of the present application, the current local initial model is trained by using the in-domain medical image set to obtain the current in-domain model and the current in-domain model parameters, that is, Step 205, as Figure 3 shown, which can be replaced by Steps 301 to 310:
[0085] Step 301: Determine that the current local initial model is the current in-domain model at the 0th iteration.
[0086] Step 302: Let the iteration number t = 1.
[0087] Step 303: Use the encoder in the current in-domain model at the (t - 1)th iteration to perform feature extraction on the in-domain medical image set to obtain the feature set at the tth iteration.
[0088] Step 304: Randomly sample the feature set according to the in-domain label set to obtain the memory queue at the tth iteration.
[0089] Step 305: Use the K-means algorithm to cluster the memory queue at the tth iteration to obtain the multi-center unbiased prototype group at the tth iteration.
[0090] Step 306: Input the memory queue at the tth iteration into the decoder of the current in-domain model at the (t - 1)th iteration to obtain the in-domain pseudo-label set at the tth iteration.
[0091] Step 307: Based on the feature set at the tth iteration, the memory queue at the tth iteration, the multi-center unbiased prototype group at the tth iteration, the in-domain label set, and the in-domain pseudo-label set at the tth iteration, determine the self-supervised loss at the tth iteration.
[0092] Step 308: According to the self-supervised loss at the tth iteration, adjust the parameters of the current in-domain model at the (t - 1)th iteration to obtain the current in-domain model and the current in-domain model parameters at the tth iteration.
[0093] Step 309: Increment the value of the iteration number t by 1, and return to Step 303 until the iteration stop condition is reached. Determine the current in-domain model at the t-th iteration as the current in-domain model, and the parameters of the current in-domain model at the t-th iteration as the current in-domain model parameters. The iteration stop condition includes that the self-supervised loss is less than a preset loss threshold and the iteration number reaches a preset iteration number.
[0094] Based on the above Steps 301 to 309, perform t iterations of training on the current local initial model. During the iteration, when the self-supervised loss is less than the preset loss threshold and the iteration number reaches the preset iteration number, it is possible to determine the current in-domain model at the t-th iteration as the current in-domain model, and determine the parameters of the current in-domain model at the t-th iteration as the current in-domain model parameters.
[0095] In another exemplary embodiment of the present application, the model weights of the current in-domain model at the t-th iteration are:
[0096]
[0097] where θ t is the model weight of the current in-domain model at the t-th iteration; θ t-1 is the model weight of the current in-domain model at the (t - 1)-th iteration; α1 is a learnable adaptive learning rate; L ssp is the self-supervised loss at the t-th iteration; is the partial derivative of the model weight θ.
[0098] In another exemplary embodiment of the present application, the adaptive learning rate of the current in-domain model is:
[0099]
[0100] where α is the adaptive learning rate of the current in-domain model; α1 is a learnable adaptive learning rate; η is the learning rate for adjusting the adaptive learning rate; y in is the in-domain label set; L Dice is the segmentation metric loss; f dec is the decoder; f enc is the encoder; x in is the in-domain hospital image; θ t is the model weight of the current in-domain model at the t-th iteration; is the partial derivative of the learnable adaptive learning rate α1.
[0101] In another exemplary embodiment of the present application, the test adaptive loss is:
[0102] L tta = L E + γ tta(L MCPCL + L CBCL + L cons )。
[0103]
[0104] L cons = ||f enc (x w ) - f enc (x s )|| 2 。
[0105]
[0106] Among them, L tta is the test adaptive loss; L E is the entropy regularization loss; γ tta is the trade-off coefficient; L MCPCL is the multi-prototype contrast learning loss; L CBCL is the intra-batch contrast learning loss; L cons is the consistency loss; E f,c is the pixel-level feature set of out-of-domain hospital images; P + is the unbiased prototype group with the same category as the pixel-level feature set E f,c of out-of-domain hospital images; P - is the unbiased prototype group with different categories from the pixel-level feature set E f,c of out-of-domain hospital images; E + is the feature set in the multi-prototype center with the same category as the pixel-level feature set E f,c of out-of-domain hospital images; E - is the feature set in the multi-prototype center with different categories from the pixel-level feature set E f,c of out-of-domain hospital images; γ is the temperature coefficient; B + is the batch-level feature with the same category as the pixel-level feature set E f,c of out-of-domain hospital images; B - is the batch-level feature with different categories from the pixel-level feature set E f,c of out-of-domain hospital images; E1 + is the pixel-level feature set of out-of-domain hospital images in the batch with the same category as the current sampled pixel-level feature set E f,c of out-of-domain hospital images; E1 - is the pixel-level feature set of out-of-domain hospital images in the batch with different categories from the current sampled pixel-level feature set E f,c of out-of-domain hospital images; is the pixel-level feature set of the current batch of out-of-domain hospital images; x w is the out-of-domain hospital medical image obtained by weak augmentation; x sis the medical image of an overseas hospital obtained through strong enhancement; is the pseudo-label of the overseas hospital image; cross_entropy(·) is the cross-entropy loss; f dec is the decoder; H is the height; W is the width; H×W is the spatial size; is the feature of the overseas hospital image with a spatial index of 0; is the feature of the overseas hospital image with a spatial index of 1; is the feature of the overseas hospital image with a spatial index of H×W; is the feature set of the overseas hospital image.
[0107] In another exemplary embodiment of the present application, the model weights of multiple in-domain models in the updated overseas initial model are:
[0108]
[0109] where θ is the model weight of multiple in-domain models in the updated overseas initial model; θ0 is the model weight of the in-domain model; is the partial derivative of the model weight θ0 of the in-domain model; α is the adaptive learning rate; L tta is the test adaptive loss.
[0110] In another exemplary embodiment of the present application, a determination process of an overseas model in a medical image adaptive segmentation method is provided. The publicly available fed-polyp dataset (a dataset dedicated to medical image analysis, mainly for colorectal polyp detection and segmentation tasks) and fed-prostate dataset (a dataset dedicated to medical image analysis, mainly for the segmentation task of prostate magnetic resonance images) are used as training data. The fed-polyp dataset is collected by four different hospital institutions, and the fed-prostate dataset is collected by six hospitals. In the specific implementation process, in order to meet the strategy of deploying models in in-domain and overseas hospitals, one hospital is selected as the overseas hospital, and the remaining all hospitals are used as in-domain hospitals for experiments. For example: Based on the fed-polyp dataset, there are a total of four hospitals participating in distributed training. Therefore, according to the division method of in-domain hospitals and overseas hospitals of 3∶1, three in-domain models can be obtained. The specific experimental process is as Figure 4 shown and includes:
[0111] S101: Input the medical image set x in and the label set y in in the fed-polyp dataset into the current local initial model in the in-domain server. The current local initial model is the current in-domain model at the 0th iteration.
[0112] S102: Let the iteration number t = 1.
[0113] S103: Use the encoder f in the current domain model during the (t - 1)-th iteration enc to extract features from the medical image set x in and obtain the feature set E at the t-th iteration.
[0114] S104: According to the label set y in , randomly sample the feature set to obtain the memory queue Q at the t-th iteration in . The memory queue Q at the t-th iteration in is:
[0115]
[0116] where Q is the size of the memory queue; is the internal feature numbered 0; is the internal feature numbered 1; is the internal feature numbered Q.
[0117] S105: Use the K-means algorithm to cluster the memory queue at the t-th iteration to obtain the multi-center unbiased prototype group at the t-th iteration. The multi-center unbiased prototype group P at the t-th iteration is:
[0118] P = cluster(Q in ).
[0119] where cluster(·) is the clustering operation using the K-means algorithm.
[0120] S106: Input the memory queue Q at the t-th iteration in into the decoder f of the current domain model during the (t - 1)-th iteration dec to obtain the in-domain pseudo-label set at the t-th iteration (f dec (f enc (x in )).
[0121] S107: Based on the feature set at the t-th iteration, the memory queue Q at the t-th iteration in , the multi-center unbiased prototype group P at the t-th iteration, the label set y in and the in-domain pseudo-label set at the t-th iteration, determine the self-supervised loss at the t-th iteration. The self-supervised loss L at the t-th iteration ssp is:
[0122] L ssp = L contrast (Q in ,P,E)+L E .
[0123] Among them, L contrast is for contrastive learning; L E is the cross-entropy loss.
[0124] The purpose is to enable the current domain model to avoid fitting noise without real label supervision.
[0125] S108: According to the self-supervised loss at the t-th iteration, adjust the parameters of the current domain model at the (t - 1)-th iteration to obtain the current domain model and the current domain model parameters at the t-th iteration. Further enhance the generalization ability of the current domain model at the (t - 1)-th iteration through the self-supervised fine-tuning method.
[0126] S109: Increase the value of the iteration number t by 1, and return to S3 until the iteration stop condition is reached. Determine the current domain model at the t-th iteration as the current domain model, and the parameters of the current domain model at the t-th iteration as the current domain model parameters. The iteration stop condition includes that the self-supervised loss is less than the preset loss threshold, and the iteration number reaches the preset iteration number.
[0127] S110: Input the medical image set x in into the current domain model to obtain the in-domain pseudo-label set.
[0128] S111: Adopt the gradient chain rule, and use the in-domain pseudo-label set, the in-domain label set y in and the model weights of the current domain model to determine the adaptive learning rate of the current domain model.
[0129] S112: Update the current domain server, and return to S1 until all in-domain servers are traversed, obtaining the in-domain models of three in-domain hospitals, as well as the multi-center unbiased prototype groups, model weights, and adaptive learning rates of the three in-domain models.
[0130] S201: Aggregate the in-domain models of the three in-domain hospitals into the model to be deployed, and collect the multi-center unbiased prototype groups, model weights, and adaptive learning rates of the three in-domain models.
[0131] S202: Call the model to be deployed and the multi-center unbiased prototype groups, model weights, and adaptive learning rates of the three in-domain models, and configure the parameters of the model to be deployed in the local initial model of the out-of-domain server to obtain the out-of-domain initial model.
[0132] S203: Input the out-of-domain medical image set, and use the encoder in the out-of-domain initial model to extract features from the medical image set to obtain the out-of-domain feature set Among them, is the out-of-domain feature with a spatial size of 0 in the out-of-domain medical image; is an out-of-domain feature with an out-of-domain medical image space size of 1; is the out-of-domain feature of the out-of-domain medical image space with a size of H×W.
[0133] S204: Randomly sample the out-of-domain feature set by category to obtain s pixel-level features Where c is the segmentation category (each image is divided into categories according to each pixel, such as lesion area and non-lesion area); is the s pixel-level features of the segmentation category 0; is the s pixel-level features of the segmentation category c.
[0134] S205: using the contrastive learning module of the multi-prototype center, adjusting the out-of-domain initial model, bringing the multi-prototype center of the source domain closer to the internal and external domains at the feature level, and obtaining the multi-prototype contrastive learning loss;
[0135] S206: By establishing intra-batch contrastive learning based on the pixel features in the current batch, the out-of-domain initial model is adjusted to obtain the intra-batch contrastive learning loss, thereby enhancing the intra-batch feature space.
[0136] S207: Use prediction consistency to improve the robustness of out-of-domain initial model fine-tuning and obtain consistency loss.
[0137] S208: The out-of-domain feature set is decoded using the decoder of the out-of-domain initial model to obtain an out-of-domain pseudo-label, and the out-of-domain pseudo-label is used to perform entropy regularization constraints on the out-of-domain initial model to obtain a cross entropy loss.
[0138] S209: According to the multi-prototype contrastive learning loss, the batch internal contrastive learning loss, the consistency loss and the cross entropy loss, the test adaptive loss is obtained.
[0139] S210: Using the test adaptive loss and the adaptive learning rates of the three in-domain models, the model weights of multiple in-domain models in the out-of-domain initial model are updated to obtain the out-of-domain model and output the segmentation result.
[0140] The goal of the above S101 to S210 is to quickly adapt the in-domain model pre-trained in the in-domain hospital to the out-domain hospital, and it does not need to use the labels of the out-domain hospital, but adopts the strategy of fine-tuning the model while testing the external hospital data for deployment. Therefore, in the in-domain model training stage, it is necessary to perform multi-prototype extraction on the privacy data of the in-domain hospital, extract multiple feature prototype centers of each in-domain hospital, and then construct a small number of feature groups representing the distribution of hospitals in the domain, and then add them to the adaptive learning rate adjustment module to learn the adaptive learning rate. Combined with the adaptive learning rate, the input medical images of the out-domain hospital are tested to achieve rapid deployment and adapt to the distribution of out-domain hospitals.
[0141] In an exemplary embodiment, a computer device is provided. The computer device can be a server or a terminal, and its internal structure diagram can be as shown in Figure 5 the following figure. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O), and a communication interface. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store medical image segmentation data. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it implements a medical image adaptive segmentation method.
[0142] Those skilled in the art can understand that Figure 5 the structure shown in the figure is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements. In an exemplary embodiment, a computer device is provided, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, the steps in the above method embodiments are implemented.
[0143] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by the processor, the steps in the above method embodiments are implemented.
[0144] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by the processor, the steps in the above method embodiments are implemented.
[0145] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with relevant regulations.
[0146] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memories. Non-volatile memory can include Read-Only Memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.
[0147] The databases involved in the embodiments provided in this application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in this application can be general-purpose processors, central processors, graphics processors, digital signal processors, programmable logics, data processing logics based on quantum computing, etc., without limitation.
[0148] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this specification.
[0149] Specific examples are used in this article to elaborate on the principles and implementation manners of this application. The descriptions of the above embodiments are only used to help understand the method and its core idea of this application; at the same time, for those of ordinary skill in the art, according to the idea of this application, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to this application.
Claims
1. An adaptive segmentation method for medical images, characterized in that, The medical image adaptive segmentation method is applied to a federated learning network; The federated learning network includes: a central server and multiple in-domain servers; the in-domain servers correspond to in-domain hospitals one by one; at least one out-of-domain server is connected to the central server; The medical image adaptive segmentation method includes: Invoking a model to be deployed and source domain knowledge, and configuring the parameters of the model to be deployed in the local initial model of the out-of-domain server to obtain an out-of-domain initial model; the model to be deployed is obtained by the central server aggregating multiple in-domain models based on the model weights of each in-domain model; any in-domain model is obtained by the out-of-domain server where it is located based on the medical image set and label set of the corresponding in-domain hospital, and training the local initial model using an adaptive learning rate adjustment method; the local initial model includes an encoder and a decoder; the output end of the encoder is connected to the input end of the decoder; the source domain knowledge includes: the multi-center unbiased prototype group, adaptive learning rate, and model weight of each in-domain model; Obtaining the medical image set of the hospital corresponding to the out-of-domain server as the out-of-domain medical image set; Using the out-of-domain medical image set to train the out-of-domain initial model to obtain a test adaptive loss; Using the test adaptive loss and the adaptive learning rate to update the model weights of multiple in-domain models in the out-of-domain initial model to obtain an out-of-domain model; Obtaining the medical image of the hospital corresponding to the out-of-domain server as the image to be segmented; Inputting the image to be segmented into the out-of-domain model to obtain the segmentation result of the image to be segmented.
2. The medical image adaptive segmentation method according to claim 1, wherein Before invoking the model to be deployed and source domain knowledge, and configuring the parameters of the model to be deployed in the local initial model of the out-of-domain server to obtain an out-of-domain initial model, it further includes: Determining any in-domain server as the current in-domain server; Determining the local initial model of the current in-domain server as the current local initial model; Obtaining the medical image set of the hospital corresponding to the current in-domain server as the in-domain medical image set; Obtaining the label set corresponding to the in-domain medical image set as the in-domain label set; Using the in-domain medical image set to train the current local initial model to obtain the current in-domain model and the current in-domain model parameters; the current in-domain model parameters include: the multi-center unbiased prototype group and model weight of the current in-domain model; Inputting the in-domain medical image set into the current in-domain model to obtain an in-domain pseudo-label set; Using the in-domain pseudo-label set, the in-domain label set, and the model weight of the current in-domain model to determine the adaptive learning rate of the current in-domain model; Updating the current in-domain server, and returning to the step of "determining the local initial model of the current in-domain server as the current local initial model" until all in-domain servers are traversed, obtaining the in-domain models of all in-domain hospitals, and the multi-center unbiased prototype group, model weight, and adaptive learning rate of each in-domain model.
3. The medical image adaptive segmentation method according to claim 2, wherein Using the in-domain medical image set to train the current local initial model to obtain the current in-domain model and the current in-domain model parameters, including: Determining the current local initial model as the current in-domain model at the 0th iteration; Letting the iteration number t = 1; Use the encoder in the current in-domain model at the (t - 1)-th iteration to extract features from the in-domain medical image set, obtaining the feature set at the t-th iteration; Randomly sample the feature set according to the in-domain label set, obtaining the memory queue at the t-th iteration; Use the K-means algorithm to cluster the memory queue at the t-th iteration, obtaining the multi-center unbiased prototype group at the t-th iteration; Input the memory queue at the t-th iteration into the decoder of the current in-domain model at the (t - 1)-th iteration, obtaining the in-domain pseudo-label set at the t-th iteration; Based on the feature set at the t-th iteration, the memory queue at the t-th iteration, the multi-center unbiased prototype group at the t-th iteration, the in-domain label set, and the in-domain pseudo-label set at the t-th iteration, determine the self-supervised loss at the t-th iteration; According to the self-supervised loss at the t-th iteration, adjust the parameters of the current in-domain model at the (t - 1)-th iteration, obtaining the current in-domain model and the current in-domain model parameters at the t-th iteration; Increment the value of the iteration number t by 1, and return to the step "Use the encoder in the current in-domain model at the (t - 1)-th iteration to extract features from the in-domain medical image set, obtaining the feature set at the t-th iteration", until the iteration stop condition is reached. Determine the current in-domain model at the t-th iteration as the current in-domain model, and the parameters of the current in-domain model at the t-th iteration as the current in-domain model parameters; The iteration stop condition includes that the self-supervised loss is less than a preset loss threshold, and the iteration number reaches a preset iteration number.
4. The medical image adaptive segmentation method according to claim 3, wherein The model weights of the current in-domain model at the t-th iteration are: Among them, θ t is the model weight of the current in-domain model at the t-th iteration; θ t-1 is the model weight of the current in-domain model at the (t - 1)-th iteration; α1 is a learnable adaptive learning rate; L ssp is the self-supervised loss at the t-th iteration; is the partial derivative with respect to the model weight θ.
5. The medical image adaptive segmentation method according to claim 1, wherein The adaptive learning rate of the current in-domain model is: Among them, α is the adaptive learning rate of the model in the current domain; α1 is the learnable adaptive learning rate; η is the learning rate for adjusting the adaptive learning rate; y in is the in-domain label set; L Dice is the segmentation metric loss; f dec is the decoder; f enc is the encoder; x in is the in-domain hospital image; θ t is the model weight of the current in-domain model at the t-th iteration; is the partial derivative of the learnable adaptive learning rate α1.
6. The medical image adaptive segmentation method according to claim 1, wherein The test adaptive loss is: L tta = L E + γ tta (L MCPCL + L CBCL + L cons ); L cons = ||f enc (x w ) - f enc (x s )|| 2 ; Among them, L tta is the test adaptive loss; L E is the entropy regularization loss; γ tta is the trade-off coefficient; L MCPCL is the multi-prototype contrast learning loss; L CBCL is the intra-batch contrast learning loss; L cons is the consistency loss; E f,c is the pixel-level feature set of out-of-domain hospital images; P + is the unbiased prototype group with the same category as the pixel-level feature set E f,c of out-of-domain hospital images; P - is the unbiased prototype group with different categories from the pixel-level feature set E f,c of out-of-domain hospital images; E + is the feature set with the same category as the pixel-level feature set E f,c in the multi-prototype center of out-of-domain hospital images; E - is the feature set with different categories from the pixel-level feature set E f,c in the multi-prototype center of out-of-domain hospital images; γ is the temperature coefficient; B + is the batch-level feature with the same category as the pixel-level feature set E f,c of out-of-domain hospital images; B - is the batch-level feature with different categories from the pixel-level feature set E f,c of out-of-domain hospital images; E1 + is the pixel-level feature set of out-of-domain hospital images with the same category f,c as the currently sampled pixel-level feature set E of out-of-domain hospital images; E1 - is the pixel-level feature set of out-of-domain hospital images with different categories f,c from the currently sampled pixel-level feature set E of out-of-domain hospital images; is the pixel-level feature set of the current batch of out-of-domain hospital images; x w is the out-of-domain hospital medical image obtained by weak augmentation; x s is the out-of-domain hospital medical image obtained by strong augmentation; is the pseudo-label of the out-of-domain hospital image; cross_entropy(·) is the cross-entropy loss; f dec is the decoder; H is the height; W is the width; H×W is the spatial size; is the feature of the out-of-domain hospital image with spatial index 0; is the feature of the out-of-domain hospital image with spatial index 1; is the feature of the out-of-domain hospital image with spatial index H×W; is the feature set of the out-of-domain hospital image.
7. The medical image adaptive segmentation method according to claim 1, wherein The model weights of multiple in-domain models in the updated out-of-domain initial model are: Among them, θ is the model weights of multiple in-domain models within the out-of-domain initial model after update; θ0 is the model weights of the in-domain model; is the partial derivative of the model weights θ0 of the in-domain model; α is the adaptive learning rate; L tta is the test adaptive loss.
8. A computer device, comprising: A memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that the processor executes the computer program to implement the medical image adaptive segmentation method according to any one of claims 1-7.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the medical image adaptive segmentation method according to any one of claims 1-7.
10. A computer program product comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the medical image adaptive segmentation method according to any one of claims 1-7.