Image segmentation method and device, electronic equipment and storage medium

Abstract

The application discloses an image segmentation method and device, electronic equipment and storage medium, applied to the field of image processing, comprising: acquiring real-time parameters of a second segmentation model corresponding to a plurality of second clients; obtaining a batch normalization layer according to a preset first segmentation model; dynamically sorting and cyclically knowledge distilling the first segmentation model and the second segmentation model according to the first segmentation model and the real-time parameters to obtain a general model; performing personalized training on the model of the first client according to the general model, the batch normalization layer and the real-time parameters to obtain an image segmentation model; acquiring an image to be segmented, and passing the image to be segmented through the image segmentation model to obtain a segmentation result. The application realizes image segmentation through the federated learning mode of cyclic knowledge distillation and personalized training, can effectively utilize effective information of other site models, obtain excellent image segmentation performance, improve the performance of the image segmentation models of a plurality of clients, and improve the image segmentation effect.

Description

Technical Field

[0001] This invention relates to the field of image processing, and more specifically to an image segmentation method, apparatus, electronic device, and storage medium. Background Technology

[0002] Image segmentation is a representative task supporting computer-aided image content analysis. For medical images, image segmentation can not only identify lesion categories but also locate specific regions, playing a crucial role in clinical diagnosis. Existing technologies typically employ federated learning for image segmentation. Federated learning can help users achieve collaborative modeling while ensuring data privacy and security, thereby enhancing model performance. However, due to the unavoidable distribution differences between data sites, federated learning-based image segmentation methods often degrade in performance when encountering different data distributions across different sites. These data differences cause site shifts, resulting in poor image segmentation results. Summary of the Invention

[0003] This application provides an image segmentation method, apparatus, electronic device, and storage medium, which can effectively improve the performance of image segmentation models at various sites.

[0004] In a first aspect, this application provides an image segmentation method applied to a first client, the first client being communicatively connected to multiple second clients, the image segmentation method comprising:

[0005] Obtain real-time parameters of the second segmentation model corresponding to multiple second clients;

[0006] Based on the preset first segmentation model, the batch standardization layer is obtained;

[0007] Based on the first segmentation model and the real-time parameters, the first segmentation model and the second segmentation model are dynamically sorted and cyclically distilled to obtain a general model;

[0008] The image segmentation model is obtained by performing personalized training on the model of the first client based on the general model, the batch normalization layer, and the real-time parameters.

[0009] The image to be segmented is obtained, and the image to be segmented is processed by the image segmentation model to obtain the segmentation result of the image to be segmented.

[0010] The image segmentation method provided in the first aspect of the embodiments of this application has at least the following beneficial effects: In the training process of the image segmentation model, a batch normalization layer is first obtained according to a preset first segmentation model to retain the individuality of the local data distribution. Then, the segmentation models of the first client and multiple second clients are dynamically sorted, and common knowledge is accumulated in the sorted models through cyclic knowledge distillation to obtain a general model with common knowledge of each client. The model of the first client is personalized trained according to the batch normalization layer, the general model and the real-time parameters of the second segmentation model to obtain the image segmentation model corresponding to the first client. This application realizes image segmentation through federated learning of cyclic knowledge distillation and personalized training. It can accumulate common knowledge and adaptive personalization of all clients without the need for a central site, i.e., a server, and can effectively utilize the effective information of other site models, thereby obtaining excellent image segmentation performance. At the same time, it improves the performance of image segmentation models of multiple clients and improves the image segmentation effect.

[0011] According to some embodiments of the first aspect of this application, the real-time parameters include performance parameters of the second segmentation model, and the step of dynamically sorting the first segmentation model and the second segmentation model based on the first segmentation model and the real-time parameters, and performing cyclic knowledge distillation to obtain a general model includes:

[0012] Acquire sample image data and input the sample image data into the first segmentation model to obtain the performance parameters and weakly supervised segmentation loss of the first segmentation model;

[0013] The first segmentation model and the second segmentation model are dynamically sorted according to their performance parameters.

[0014] Based on the results of dynamic sorting, the teacher model of the first segmentation model is determined;

[0015] Based on the first segmentation model and the teacher model, the knowledge distillation loss of the first segmentation model is obtained;

[0016] The first segmentation model is trained using the knowledge distillation loss and the weakly supervised segmentation loss until the performance parameters of the first segmentation model meet the first preset condition.

[0017] Based on the first segmentation model after training, a general model is obtained.

[0018] According to some embodiments of the first aspect of this application, the step of inputting the sample image data into the first segmentation model to obtain the performance parameters of the first segmentation model includes:

[0019] Obtain the true labels of the sample image data;

[0020] The sample image data is input into the first segmentation model to obtain the predicted image data and the normalized exponential probability vector.

[0021] Based on the predicted image data and the true labels, a set similarity measurement function is obtained;

[0022] Based on the normalized exponential probability vector, the prediction entropy of the first segmentation model is obtained, and the prediction entropy is used to characterize the uncertainty of the first segmentation model.

[0023] The performance parameters of the first segmentation model are obtained based on the set similarity metric function and the uncertainty.

[0024] According to some embodiments of the first aspect of this application, the step of inputting the sample image data into the first segmentation model to obtain the weakly supervised segmentation loss of the first segmentation model includes:

[0025] Based on the sample image data, the corresponding sparse labels are obtained;

[0026] The sample image data is input into the first segmentation model to obtain predicted image data, predicted probability, and image selection features. The image selection features are the image features of the sample image data passing through a preset layer of the first segmentation model.

[0027] Tree affinity is generated by performing tree affinity on the sample image data and the selected features of the image to obtain a low-order affinity matrix and a high-order affinity matrix;

[0028] The low-order affinity matrix and the high-order affinity matrix are passed through a cascaded filter to obtain the soft pseudo-labels corresponding to the sample image data;

[0029] The tree energy loss is obtained based on the predicted probability and the soft pseudo-label.

[0030] Based on the predicted image data and the sparse labels, a partial cross-entropy loss is obtained;

[0031] Based on the predicted probability and the preset source mask and preset target mask, the gated conditional random field loss is obtained;

[0032] The weakly supervised segmentation loss of the first segmentation model is obtained based on the tree energy loss, the partial cross-entropy loss, and the gated conditional random field loss.

[0033] According to some embodiments of the first aspect of this application, the step of performing personalized training on the model of the first client based on the general model, the batch normalization layer, and the real-time parameters to obtain an image segmentation model includes:

[0034] The model of the first client is initialized according to the general model and the batch normalization layer to obtain the third segmentation model;

[0035] Based on the third segmentation model and the real-time parameters, the similarity weight matrix is ​​obtained;

[0036] Based on the third segmentation model and the similarity weight matrix, the teacher model of the third segmentation model is obtained;

[0037] Based on the third segmentation model and the teacher model, the knowledge distillation loss and weakly supervised segmentation loss of the third segmentation model are obtained;

[0038] The third segmentation model is trained using the knowledge distillation loss and the weakly supervised segmentation loss until the performance parameters of the third segmentation model meet the second preset condition.

[0039] The image segmentation model is obtained based on the third segmentation model after training.

[0040] According to some embodiments of the first aspect of this application, the real-time parameters include statistics from the second segmentation model, and obtaining the similarity weight matrix based on the third segmentation model and the real-time parameters includes:

[0041] Based on the third segmentation model, the statistics of the third segmentation model are obtained;

[0042] Based on the statistics of the third segmentation model and the second segmentation model, the optimal transmission distance between the third segmentation model and multiple second segmentation models is obtained;

[0043] Based on multiple optimal transmission distances, the similarity between the third segmentation model and multiple second segmentation models is obtained;

[0044] The similarity scores are then normalized and updated using a moving average in sequence to obtain the similarity weights.

[0045] Based on the multiple similarity weights, a similarity weight matrix is ​​obtained.

[0046] According to some embodiments of the first aspect of this application, obtaining the teacher model of the third segmentation model based on the third segmentation model and the similarity weight matrix includes:

[0047] Based on the third segmentation model, the batch normalized layer parameters and other layer parameters of the third segmentation model are obtained;

[0048] Based on the batch normalization layer parameters of the third segmentation model, the approval layer parameters of the teacher model are obtained;

[0049] Based on the other layer parameters of the third segmentation model and the similarity weight matrix, the other layer parameters of the teacher model are obtained.

[0050] Based on the approval layer parameters and other layer parameters of the teacher model, the teacher model corresponding to the third segmentation model is obtained.

[0051] Secondly, this application provides an image segmentation apparatus applied to a first client, the first client being communicatively connected to a plurality of second clients, comprising:

[0052] The data communication module is used to obtain real-time parameters of the second segmentation model corresponding to multiple second clients;

[0053] The distributed processing module is used to obtain batch standardization layers based on a preset first segmentation model;

[0054] A general training module is used to dynamically sort the first segmentation model and the second segmentation model and perform cyclic knowledge distillation based on the first segmentation model and the real-time parameters to obtain a general model.

[0055] The personalized training module is used to perform personalized training on the model of the first client based on the general model, the batch normalization layer and the real-time parameters to obtain an image segmentation model;

[0056] The image segmentation module is used to acquire the image to be segmented and to obtain the segmentation result of the image to be segmented by passing the image segmentation model.

[0057] Since the image segmentation apparatus provided in the second aspect applies the image segmentation method of any one of the first aspects, it has all the beneficial effects of the first aspect of this application.

[0058] Thirdly, this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program as the image segmentation method of any of the first aspects.

[0059] Since the processor of the third aspect executes the computer program as the image segmentation method of any of the first aspects, it has all the beneficial effects of the first aspect of the embodiments of this application.

[0060] Fourthly, embodiments of this application provide a computer storage medium including computer-executable instructions stored thereon, the computer-executable instructions being used for the image segmentation method as described in any of the first aspects.

[0061] Since the computer storage medium of the fourth aspect can perform the image segmentation method as described in any of the first aspects, it has all the beneficial effects of the first aspect of the embodiments of this application.

[0062] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0064] Figure 1 This is a structural block diagram of the image segmentation apparatus provided in the embodiments of this application;

[0065] Figure 2 This is a flowchart of the main image segmentation method provided in the embodiments of this application;

[0066] Figure 3 This is a flowchart illustrating the general training process of the image segmentation method provided in the embodiments of this application;

[0067] Figure 4 This is a flowchart illustrating the performance analysis of the image segmentation method provided in the embodiments of this application.

[0068] Figure 5 This is a flowchart illustrating the loss calculation process of the image segmentation method provided in this application embodiment;

[0069] Figure 6 This is a flowchart of the personalized training of the image segmentation method provided in the embodiments of this application;

[0070] Figure 7 This is a flowchart illustrating the weight calculation process of the image segmentation method provided in this application embodiment;

[0071] Figure 8 This is a flowchart of the model calculation for the image segmentation method provided in the embodiments of this application;

[0072] Figure 9 This is a graph showing the prediction results and uncertainties of the image segmentation method provided in the embodiments of this application;

[0073] Figure 10 These are the images provided in the embodiments of this application and their corresponding weak and fully supervised labels;

[0074] Figure 11This is the weakly supervised training structure of the image segmentation model provided in the embodiments of this application;

[0075] Figure 12 This is the general framework for federated learning applied to the image segmentation method provided in the embodiments of this application;

[0076] Figure 13 These are the segmentation results corresponding to various image segmentation methods provided in the embodiments of this application;

[0077] Figure 14 This is an ablation experiment of the image segmentation method provided in the embodiments of this application;

[0078] Figure 15 This is a schematic diagram of a system architecture platform for image segmentation provided in an embodiment of this application. Detailed Implementation

[0079] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that the embodiments of this application can also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the embodiments of this application with unnecessary detail.

[0080] It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0081] It should also be understood that references to "one embodiment" or "some embodiments" in the specification of embodiments of this application mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0082] Image segmentation is a representative task supporting computer-aided image content analysis. For medical images, image segmentation can not only identify lesion categories but also locate specific regions, playing a crucial role in clinical diagnosis. Limited by the number of patients and annotation costs, individual medical institutions often face insufficient data. Therefore, collaborative training using data from multiple hospitals is essential for obtaining a powerful intelligent analysis model. Existing technologies typically employ federated learning for image segmentation. In common federated learning, each local client trains its model using local training data, and the parameters of each local model are aggregated at a central server and broadcast to all sites for coordination. Federated learning can help users achieve collaborative modeling while ensuring data privacy and security, thereby enhancing model performance. However, due to unavoidable differences in data distribution between sites—such as doctors' experience, imaging equipment models, imaging parameters, and patient group differences—the performance of federated learning-based image segmentation methods often deteriorates when encountering different data distributions across sites. These data differences cause site shifts, resulting in poor image segmentation results.

[0083] Based on this, this application provides an image segmentation method, apparatus, electronic device, and storage medium. The image segmentation method provided in this application achieves image segmentation through a federated learning approach of cyclic knowledge distillation and personalized training. It can accumulate common knowledge and adaptively personalize all clients without the need for a central site, i.e., a server, and can effectively utilize the effective information from other site models, thereby obtaining excellent image segmentation performance.

[0084] Reference Figure 1 , Figure 1 This is a structural block diagram of the image segmentation apparatus provided in the embodiments of this application. The image segmentation apparatus provided in the embodiments of this application includes a data communication module 100, a distributed processing module 200, a general training module 300, a personalized training module 400, and an image segmentation module 500.

[0085] The data communication module 100 is connected to the distributed processing module 200, the general training module 300, and the personalized training module 400. The data communication module 100 is used to obtain the real-time parameters of the second segmentation model corresponding to multiple second clients.

[0086] The distributed processing module 200 is connected to the data communication module 100 and the general training module 300 respectively. The distributed processing module 200 is used to obtain the batch standardization layer according to the preset first segmentation model.

[0087] The general training module 300 is connected to the data communication module 100 and the personalized training module 400 respectively. The general training module 300 is used to dynamically sort the first segmentation model and the second segmentation model and perform cyclic knowledge distillation based on the first segmentation model and real-time parameters to obtain the general model.

[0088] The personalized training module 400 is connected to the data communication module 100, the general training module 300, and the image segmentation module 500. The personalized training module 400 is used to perform personalized training on the model of the first client based on the general model, the batch normalization layer, and multiple real-time parameters to obtain the image segmentation model.

[0089] The image segmentation module 500 is communicatively connected to the personalized training module 400. The image segmentation module 500 is used to acquire the image to be segmented and to obtain the segmentation result of the image to be segmented by passing it through the image segmentation model.

[0090] It should be noted that the image segmentation apparatus provided in this application embodiment includes a data communication module 100, a distributed processing module 200, a general training module 300, a personalized training module 400, and an image segmentation module 500. The data communication module 100 acquires real-time parameters of the second segmentation models corresponding to multiple second clients and sends the real-time parameters of the multiple second segmentation models to the distributed processing module 200, the general training module 300, and the personalized training module 400, respectively. The distributed processing module 200 obtains a batch standardization layer based on a preset first segmentation model. The general training module 300 performs dynamic sorting and cyclic knowledge distillation on the first segmentation model and the multiple real-time parameters to obtain a general model. The personalized training module 400 performs personalized training based on the received general model, batch standardization layer, and real-time parameters of the multiple second segmentation models to obtain an image segmentation model. The image segmentation module 500 acquires the image to be segmented and passes it through the image segmentation model to obtain the segmentation result of the image to be segmented. This application achieves image segmentation through a federated learning approach of cyclic knowledge distillation and personalized training. It can accumulate common knowledge and adaptively personalize all clients without the need for a central site, i.e., a server, and can effectively utilize the effective information of other site models, thereby obtaining excellent image segmentation performance. At the same time, it improves the performance of image segmentation models on multiple clients and enhances the image segmentation effect.

[0091] The apparatus and application scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of system architecture and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0092] It will be understood by those skilled in the art that Figure 1 The device structure shown does not constitute a limitation on the embodiments of this application. It may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0093] exist Figure 1 In the device structure shown, each module can call its stored simulation analysis program to execute the image segmentation method.

[0094] Based on the above apparatus, various embodiments of the image segmentation method of this application are proposed.

[0095] Reference Figure 2 , Figure 2 This is a main flowchart of the image segmentation method provided in this application embodiment. The image segmentation method provided in this application embodiment is applied to a first client, which is connected to multiple second clients in a communication relationship. The image segmentation method includes, but is not limited to, the following steps:

[0096] Step S100: Obtain the real-time parameters of the second segmentation model corresponding to multiple second clients.

[0097] Step S200: Obtain the batch standardization layer according to the preset first segmentation model.

[0098] Step S300: Dynamically sort the first segmentation model and the second segmentation model and perform cyclic knowledge distillation based on the first segmentation model and real-time parameters to obtain a general model.

[0099] Step S400: Based on the general model, batch normalization layer and real-time parameters, perform personalized training on the model of the first client to obtain the image segmentation model.

[0100] Step S500: Obtain the image to be segmented, and use the image segmentation model to obtain the segmentation result of the image to be segmented.

[0101] It should be noted that during the training process of the image segmentation model, a batch normalization layer is first obtained based on a preset first segmentation model to preserve the individuality of the local data distribution. Then, the segmentation models of the first client and multiple second clients are dynamically sorted, and common knowledge is accumulated in the sorted models through cyclic knowledge distillation to obtain a general model with common knowledge of each client. Based on the real-time parameters of the batch normalization layer, the general model, and the second segmentation models, the model of the first client is trained in a personalized manner to obtain the image segmentation model corresponding to the first client. This application achieves image segmentation through a federated learning approach of cyclic knowledge distillation and personalized training. It can accumulate common knowledge and adaptively personalize all clients without the need for a central site, i.e., a server, and can effectively utilize the effective information of other site models, thereby obtaining excellent image segmentation performance. At the same time, it improves the performance of image segmentation models of multiple clients and improves the image segmentation effect.

[0102] It should be noted that in the image segmentation method of this application, it is assumed that there are N clients, that is, the total number of the first client and the second client is N, represented as {C1, C2, ..., C...}. N}, and use {D1,D2,…,D N} represents each client's data. Each client's data is divided into a training set and a test set. The validation set is then divided from the training set according to a certain proportion. Where (x) i,j ,y i,j ) represents a sample and its corresponding sparse label. For the training set, This is the test set, and the total sample size is... The number of training samples. The number of test samples is specified. Since there are certain distribution differences / domain biases among the data from different clients, this application will train a personalized model for each client, denoted as... The overall goal is to aggregate data knowledge from all clients without data leakage to obtain a personalized model φ that performs well on each client. i , represented as

[0103]

[0104] Where L is the loss function.

[0105] It should be noted that the image segmentation method in this application employs a personalized federated learning approach. While the first client performs general and personalized training, the second client simultaneously performs dynamic sorting, cyclic knowledge distillation, and personalized training. During federated learning, cyclic knowledge distillation is the model's accumulation of common knowledge; therefore, the general model obtained at the second client is the same as the general model obtained at the first client. The personalized training phase involves training based on the general model, the batch normalization layer, and the real-time parameters of the segmentation models from other clients. Therefore, each client obtains a different image segmentation model.

[0106] It should be noted that the architecture of the first segmentation model can adopt the architecture of an existing segmentation model. In this embodiment, the U-Net network is used as the segmentation model architecture.

[0107] It should be noted that, in the stage of public knowledge accumulation, in order to maintain the personalization of the client model for its data distribution, before the cyclic knowledge distillation, i.e. step S200, this application retains the batch normalization layer of the first segmentation model through FedBN.

[0108] Understandably, referring to Figure 3 , Figure 3 This is a flowchart of the general training process for the image segmentation method provided in this application embodiment. The real-time parameters obtained in step S300 include the performance parameters of the second segmentation model. Step S200 includes, but is not limited to, the following steps:

[0109] Step S310: Obtain sample image data and input the sample image data into the first segmentation model to obtain the performance of the first segmentation model and the weakly supervised segmentation loss.

[0110] Step S320: Dynamically sort the first segmentation model and the second segmentation model according to the performance parameters of the first segmentation model and the second segmentation model.

[0111] Step S330: Determine the teacher model of the first segmentation model based on the results of dynamic sorting.

[0112] Step S340: Based on the first segmentation model and the teacher model, obtain the knowledge distillation loss of the first segmentation model.

[0113] Step S350: Train the first segmentation model based on the knowledge distillation loss and the weakly supervised segmentation loss until the performance parameters of the first segmentation model meet the first preset condition.

[0114] Step S360: Based on the first segmentation model after training, obtain the general model.

[0115] It should be noted that the first client sorts the first segmentation model and multiple second segmentation models according to the performance parameters of the first segmentation model and multiple second segmentation models in the current training round. The performance parameters of the clients are different in different training rounds, and their dynamic sorting results are different. The teacher model corresponding to the first client is different. This application sorts according to the performance parameters of the current training round. The performance parameters can be used to describe the model performance and are a visualization of the model performance. The model with the best performance is used as the teacher model of the second best performance model, and so on. The model is dynamically adjusted in each round, which can achieve the goal of balanced improvement of the models of each client and effectively complete the distillation of common knowledge.

[0116] It should be noted that in each training round, after determining the cyclic distillation ranking, the KL divergence loss is used as the knowledge distillation loss to narrow the distance between the prediction levels of the first segmentation model and the teacher model, thereby achieving the transfer of common knowledge. The KL divergence loss is determined by the first segmentation model and the corresponding teacher model, and its expression is as follows:

[0117] L dist =KL(φ i (x),φ i-1 (x))

[0118] Among them, L dist For knowledge distillation loss, φ i (x) represents the first segmentation model, φ i-1 (x) represents the teacher model corresponding to the first segmentation model, and x represents the sample image data of the current client. The loss of the first segmentation model can be determined by the knowledge distillation loss and the weakly supervised segmentation loss. Therefore, the overall objective function for training the first segmentation model is:

[0119]

[0120] λ d To balance the weighting coefficients between the knowledge distillation loss and the supervision loss of the current sample image data, L seg Losses are segmented for weak supervision.

[0121] It should be noted that in the public knowledge accumulation stage, i.e., step S300, this application sets a threshold th1 to determine whether the first segmentation model has performance or has accumulated sufficient representational ability. When the relevant parameters of the first segmentation model are greater than the threshold th1, the application proceeds by... Otherwise, the first segmentation model is trained using local data. λ in d Set to 0.

[0122] It should be noted that the first preset condition is that the performance of the first segmentation model remains stable, that is, it retains the common knowledge of the first segmentation model and multiple second segmentation models. For ease of calculation, a sufficient number of training rounds can be set for general training as needed. In this embodiment, the number of training rounds is set to 50.

[0123] Understandably, referring to Figure 4 , Figure 4 This is a flowchart of the performance analysis of the image segmentation method provided in the embodiments of this application. Step S310 includes, but is not limited to, the following steps:

[0124] Step S3111: Obtain the real labels of the sample image data.

[0125] Step S3112: Input the sample image data into the first segmentation model to obtain the predicted image data and the normalized exponential probability vector.

[0126] Step S3113: Obtain the set similarity measurement function based on the predicted image data and the true labels.

[0127] Step S3114: Based on the normalized exponential probability vector, obtain the prediction entropy of the first segmentation model. The prediction entropy is used to characterize the uncertainty of the first segmentation model.

[0128] Step S3115: Obtain the performance parameters of the first segmentation model based on the set similarity measurement function and uncertainty.

[0129] It should be noted that the embodiments of this application use the Dice coefficient (i.e., the set similarity metric function) of the sample image data on the model, and uncertainty as a performance parameter to measure the performance of the segmentation model. The models are sorted according to the magnitude of the performance parameters, and the uncertainty of the first segmentation model is calculated. This uncertainty is then multiplied by a tradeoff parameter, and the result is added to the set similarity metric function to obtain the performance parameters of the segmentation model. Specifically, the performance parameters of the segmentation model can be expressed as follows:

[0130]

[0131] Among them, U i λ represents the uncertainty of the i-th segmentation model (either the first or second segmentation model) in the current round. u To balance the parameters, this application employs the Monte Carlo dropout method to estimate the uncertainty. Specifically, under different additive Gaussian noise and random dropout conditions, the sample image data is randomly forwarded T times through the model. Each pixel in the sample image data can obtain a set of softmax probability vectors, i.e., normalized exponential probability vectors. Therefore, predictive entropy is chosen as a measure to approximate uncertainty, which can be expressed as:

[0132]

[0133] in, This is the probability of predicting a class as c in the t-th prediction. Additionally, the uncertainty U for a single sample in the sample image data... I It can be represented as {u}∈R H×W The uncertainty of the segmentation model on the client side is measured by the average sample uncertainty of its corresponding sample image data. Therefore, the uncertainty of the segmentation model can be expressed as:

[0134]

[0135] Where, N val This represents the total number of samples in the sample image data. Additionally, the set similarity measure, the Dice coefficient, indicates the degree of overlap between the predicted image data and the true labels. The mathematical expression for the set similarity measure is:

[0136]

[0137] in, Represents predicted image data. This represents the true label. For the overall objective function of training the first segmentation model, when the Dice coefficient of the first segmentation model is greater than the threshold th1, then... Training, λ d If the default value is used, otherwise, the first segmentation model is trained using local data, i.e. λ in d Set to 0.

[0138] Reference Figure 9 , Figure 9 This is a graph showing the prediction results and uncertainties of the image segmentation method provided in the embodiments of this application. Figure 9 Examples of prediction results and uncertainty plots for the same sample using different segmentation models are shown, revealing a negative correlation between model performance and uncertainty. Therefore, the performance parameters representing the model are calculated using the reciprocal of the uncertainty.

[0139] Understandably, referring to Figure 5 , Figure 5 This is a flowchart of the loss calculation of the image segmentation method provided in the embodiments of this application. Step S310 also includes, but is not limited to, the following steps:

[0140] Step S3121: Obtain the corresponding sparse labels based on the sample image data.

[0141] Step S3122: Input the sample image data into the first segmentation model to obtain the predicted image data, the predicted probability, and the selected image features. The selected image features are the image features of the sample image data through the preset layer of the first segmentation model.

[0142] Step S3123: Perform tree affinity generation on the sample image data and selected image features to obtain a low-order affinity matrix and a high-order affinity matrix.

[0143] Step S3124: Pass the low-order affinity matrix and the high-order affinity matrix through a cascaded filter to obtain the soft pseudo-labels corresponding to the sample image data.

[0144] Step S3125: Obtain the tree energy loss based on the predicted probability and soft pseudo-labels.

[0145] Step S3126: Based on the predicted image data and sparse labels, obtain the partial cross-entropy loss.

[0146] Step S3127: Based on the predicted probability and the preset source mask and preset target mask, obtain the gated conditional random field loss.

[0147] Step S3128: Based on the tree energy loss, partial cross-entropy loss, and gated conditional random field loss, obtain the weakly supervised segmentation loss of the first segmentation model.

[0148] It should be noted that image segmentation is essentially a pixel-level classification task. The most classic weakly supervised segmentation method modifies the traditional cross-entropy function to a partial cross-entropy function, that is, it calculates the cross-entropy loss only on pixels in the labeled region. However, applying partial cross-entropy loss only to sparsely labeled regions often fails to provide sufficient supervision signals near semantic boundaries, thus leading to poor prediction results in unsupervised areas. The weak supervision provided in this application extends outward from the supervised region in a specific way, forming pseudo-labels for unlabeled regions, thereby providing more sufficient supervision signals for model training.

[0149] It should be noted that, referring to Figure 10 , Figure 10This application provides images and corresponding weak and fully supervised labels. Points, graffiti, and block annotations can all serve as pixel-level sparse category supervision signals. However, bounding boxes, as rectangular boxes describing the boundaries of targets, require certain preprocessing transformations before they can be used as category supervision signals. Based on prior knowledge such as the shape and size of the segmented target, it can be converted into points, graffiti, blocks, or a mixture of the three. For example, in the segmentation task of the optic cup and optic disc in the fundus, for the two elliptical structures of the optic cup and optic disc, the Hough circle fitting transform is first used to fit the inscribed ellipse of the bounding boxes of the two target regions. Then, by skeletonizing the inner ellipse and the annular region between the two ellipses, sparse skeleton annotations of the two target regions are obtained. At the same time, the same operation is performed on the background part outside the optic disc bounding box to obtain sparse background annotations, thus converting the bounding box annotations into graffiti-like annotations. For the segmentation task of the avascular region of the fovea of ​​the retina, due to its irregular shape, the four sides of the bounding box are expanded outward by a certain number of pixels. The outer pixels of the expanded rectangle are set as the background class. At the same time, the length and width of the bounding box are reduced by 80% respectively to obtain a reduced box. The pixels within the range of this box are set as the target class, thus converting the bounding box into block annotations. For other complex structures, similar preprocessing based on priors such as shape and size can be performed to convert the box annotations into sparse pixel annotations.

[0150] In some embodiments, based on experimental and verification needs, this application utilizes multiple publicly available medical image segmentation datasets and employs automated methods to generate their corresponding sparse annotations, i.e., sparse labels (using the same annotation format on the same client, including four forms: points, scribbles, bounding boxes, and blocks), constructing two multi-domain weakly supervised segmentation datasets for segmenting the optic cup and optic disc (ODOC) and the fovea vascular zone (FAZ) of the retina, respectively. For the ODOC dataset, the optic cup and optic disc region is cropped based on the mask annotations of the original fundus image and uniformly resized to 384×384; while for the FAZ, the original image range is maintained and uniformly resized to 256×256. Data preprocessing includes normalizing all image pixel values ​​to between 0 and 1, and data augmentation includes randomly flipping the image horizontally and vertically, and randomly rotating it by a certain angle (range -45° to 45°). Figure 10 As shown in the last column, block labels can be obtained by processing each type of region through morphological erosion transformation, graffiti labels can be obtained through skeletonization, graffiti labels of another style can be obtained by performing local deformation transformation on the obtained skeleton and randomly erasing it, bounding box labels can be obtained by calculating the bounding rectangle of the target region mask, target point labels can be obtained by calculating the midpoints of the four sides of the inscribed rectangle of the target region, and background point labels can be obtained by expanding the bounding box and taking the midpoints of the four sides. In this way, sparse labels corresponding to the sample image data of each client segmentation model can be obtained.

[0151] It should be noted that the sample image data is processed through the first segmentation model to obtain the predicted image data, the predicted probability, and the selected image features. The predicted image data is the result of image segmentation, the selected image features are the image features of the sample image data through the preset layer of the first segmentation model, and the predicted probability is the probability output by the softmax layer of the first segmentation model.

[0152] It should be noted that, referring to Figure 11 , Figure 11 This is the weakly supervised training structure of the image segmentation model provided in the embodiments of this application. Figure 11 This application primarily showcases the weakly supervised training paradigm employed in this study. During both general and personalized training phases, the internal processing of the segmentation model utilizes this paradigm. The weakly supervised training paradigm mainly comprises a segmentation branch that processes labeled pixels and an auxiliary branch that processes unlabeled pixels. The segmentation branch uses the sparse label Y as a supervision signal and employs partial cross-entropy loss for supervision. The auxiliary branch obtains a pair of affinity matrices A from the sample image data I and the selected image features F. low and A high A low A is a low-order affinity matrix. high The affinity matrix is ​​a high-order matrix, which can be used to refine the prediction network's predictions and generate soft pseudo-labels. These soft pseudo-labels are used to supervise unlabeled pixel regions. Furthermore, to further improve the model's prediction accuracy for target edges, this embodiment introduces a gated conditional random field loss. The gated conditional random field loss is used to suppress the excessive expansion or collapse of target region pseudo-labels that may occur when simply using tree energy loss. The overall objective function, i.e., the weakly supervised segmentation loss, can be expressed as:

[0153] L seg =L pCE +λ t L tree +λ g L gCRF

[0154] Among them, L pCE For partial cross-entropy loss, L tree For tree energy loss, L gCRF For the gated conditional random field loss, λ t λ is the weighting parameter for the energy loss of the balanced tree. g The weighting parameters are used to balance the loss of the gated conditional random field.

[0155] It should be noted that for tree energy loss L treeThe calculation first requires calculating tree affinity. An image can be represented by an undirected graph G = (V, E), where V represents the set of vertices formed by all pixels, and E represents the set of edges formed by the edges between adjacent vertices. This embodiment uses a four-connected graph construction, that is, only connecting the top, bottom, left, and right adjacent pixels of each vertex to form edges. For a given pair of adjacent pixels i and j, their low-order and high-order weights are defined as follows:

[0156]

[0157]

[0158] Among them, I(i)∈R 3×h×w Let F(i) ∈ R be the pixel values ​​of the sample image data. 256×h×w Feature map values ​​are selected for the image, where h and w represent the image's height and width. F(i) is obtained by changing the number of channels of the selected features through a 1x1 convolution. After obtaining the weights, the edges with the largest weights are sequentially removed from E while maintaining graph connectivity to construct a minimum spanning tree, generating both higher-order and lower-order minimum spanning trees. The distance between two vertices in the minimum spanning tree can be calculated by summing the weights of their connecting edges. The distance of the shortest path between vertices, denoted as the hyperedge E, constitutes the distance graph of the minimum spanning tree.

[0159]

[0160] Where i, j, k, and m are vertex indices, *∈{low,high}. To capture long-range relationships between vertices, the distance graph is projected onto the positive affinity matrix:

[0161]

[0162]

[0163] in, Here are the hyperparameters used to adjust the intensity information. Given an image, the low-order affinity matrix is ​​static while the high-order affinity matrix is ​​dynamic. These two matrices extract pairwise relationships at the feature level, allowing the network to be trained to acquire complementary knowledge. Since the low-order affinity matrix A... low And containing boundary information, while the higher-order affinity matrix A high To maintain semantic consistency, a cascaded filter module is used to refine the predicted image data and apply it as a soft pseudo-label.

[0164]

[0165] Where P is the predicted probability, i.e., the probability value output by the softmax layer of the first segmentation model, and F is the cascaded filter. It is calculated by multiplying with the low-order and high-order affinity matrices and passing through filter F, as follows:

[0166]

[0167] Where Ω represents the set of all pixels, and To create a normalized image. After obtaining the pseudo-labels, the tree energy loss can be expressed as:

[0168]

[0169] Where δ is the label assignment function, used to measure the prediction probability P and the pseudo-label. The distance between them is calculated using the L1 norm loss in this embodiment of the application. Therefore, the tree energy loss can be further expressed as:

[0170]

[0171] Among them, Ω U This represents the unlabeled pixel region of the image. Labeled regions are optimized using a partial cross-entropy loss through a supervised signal, while unlabeled regions are optimized using a tree energy loss, thus enabling each site model to achieve online self-training under weak supervision.

[0172] It should be noted that in federated learning, a more practical setting would be to allow different central sites to use heterogeneous sparse labels and train them in a unified and compatible manner, so that each site can benefit from the federation.

[0173] It should be noted that the selected image features are the image features of the sample image data through the preset layer of the segmentation model. In this embodiment, U-Net is used as the network architecture of the segmentation model, so the preset layer is the second layer of the U-Net decoder. The segmentation model can also use other network architectures, and the corresponding preset layer will also change with the network architecture.

[0174] Understandably, referring to Figure 6 , Figure 6 This is a flowchart of the personalized training of the image segmentation method provided in this application embodiment. Step S400 includes, but is not limited to, the following steps:

[0175] Step S410: Initialize the model of the first client according to the general model and the batch normalization layer to obtain the third segmentation model.

[0176] Step S420: Obtain the similarity weight matrix based on the third segmentation model and real-time parameters.

[0177] Step S430: Based on the third segmentation model and the similarity weight matrix, obtain the teacher model of the third segmentation model.

[0178] Step S440: Based on the third segmentation model and the teacher model, obtain the knowledge distillation loss and weakly supervised segmentation loss of the third segmentation model.

[0179] Step S450: Train the third segmentation model based on the knowledge distillation loss and the weakly supervised segmentation loss until the performance parameters of the third segmentation model meet the second preset condition.

[0180] Step S460: Obtain the image segmentation model based on the third segmentation model after training.

[0181] It should be noted that the batch normalization layer in step S410 corresponds to the preset first segmentation model. This batch normalization layer retains the individuality of the first client's data distribution. Therefore, the third segmentation model obtained by initialization based on the general model and the batch normalization layer retains the individuality of its own data and also includes the common knowledge of other second clients.

[0182] It should be noted that the personalized training phase still employs iterative training of the local model using cyclic distillation. To better achieve personalization and learn useful knowledge from similar second-level segmentation models, the teacher model in this phase is determined based on the weighted aggregation of similarity between the third-level segmentation model and multiple second-level segmentation models in each training round. Specifically, the first and second clients calculate and broadcast the statistics of the batch normalization layer of the segmentation model based on local data. This allows the first and second clients to calculate the similarity weight matrix, which guides the weighted aggregation of the teacher model in each training round. In each training round, each client broadcasts the parameters of the segmentation model and uses the similarity weight matrix to aggregate the knowledge distillation teacher model. After determining the teacher model corresponding to the third-level segmentation model, the knowledge distillation loss and weakly supervised segmentation loss of the third-level segmentation model are calculated. The objective training function of the segmentation model can be expressed as:

[0183]

[0184] Among them, L dist For knowledge distillation loss, φ i (x) represents the third segmentation model, φ i-1 (x) represents the teacher model corresponding to the third segmentation model, and x and y represent the sample image data of the current client. λ d To balance the weighting coefficients between the knowledge distillation loss and the supervision loss of the current sample image data, L seg The loss is for weakly supervised segmentation. The teacher model uses the Dice coefficient obtained on the validation data of the current center as a performance metric. If its value is lower than the performance metric of the first segmentation model, then λ...d Set to 0, otherwise its value is:

[0185]

[0186] Where λ0 is a hyperparameter. The Dice coefficients for the teacher model. is the Dice coefficient of the first segmentation model.

[0187] It should be noted that the second preset condition is that the first segmentation model converges. To facilitate calculation, a sufficient number of training rounds can be set for personalized training. In this embodiment, the number of training rounds is set to 1000.

[0188] Understandably, referring to Figure 7 , Figure 7 This is a flowchart illustrating the weight calculation process of the image segmentation method provided in this application embodiment. Real-time parameters include statistics from the second segmentation model. Step S420 includes, but is not limited to, the following steps:

[0189] Step S421: Obtain the statistics of the third segmentation model based on the third segmentation model.

[0190] Step S422: Based on the statistics of the third segmentation model and the second segmentation model, obtain the optimal transmission distance between the third segmentation model and multiple second segmentation models.

[0191] Step S423: Based on multiple optimal transmission distances, obtain the similarity between the third segmentation model and multiple second segmentation models.

[0192] Step S424: Normalize and update the multiple similarities sequentially using moving average to obtain the similarity weights.

[0193] Step S425: Obtain the similarity weight matrix based on multiple similarity weights.

[0194] It should be noted that personalized training also incorporates sample image data for training. For the similarity weights, it is necessary to first obtain the model's statistics. These statistics can be represented by the statistics of the batch normalization layer, using μ to represent the mean and σ to represent the variance. Therefore, the statistics of the i-th client model (either the third or second segmentation model) can be expressed as:

[0195] (μ i ,σ i )=[(μ i,1 ,σ i,1 ),(μ i,2 ,σ i,2 ),…,(μ i,L ,σ i,L )]

[0196] Where L represents the number of layers in the network. This application uses an approximation of the Wasserstein distance to measure the distance between two Gaussian distributions. The Wasserstein distance is a method used to measure the difference between two probability distributions, also known as the optimal transmission distance. The Wasserstein distance between two segmentation models can be expressed as:

[0197]

[0198] Where N represents the Gaussian distribution of the segmentation model, l represents the l-th network layer, and r represents the standard deviation. Assuming each channel is independent, then σ i,l It is a diagonal matrix, σ i,l =Diag(r i,l Therefore, the distance between two clients i and j can be expressed as:

[0199]

[0200] The greater the distance between two clients i and j, the greater the difference in the distributions of the two segmentation models, and the higher the similarity weight m. i,j The smaller it should be, the more we introduce... Used to describe the similarity weight m i,j ,Will Set to d i,j The reciprocal of, that is Will Normalization yields:

[0201]

[0202] The parameters of other layers in the teacher model are updated using a moving average method, therefore the similarity weight m i,j The moving average method is also used for updating. Additionally, m... i,i If the similarity weight is set to a constant α, then the similarity weight m i,j It can be represented as:

[0203]

[0204] Similarity weight m i,j These are the matrix elements of the similarity weight matrix, representing the similarity between two segmentation models. The larger the value, the higher the similarity. The similarity weights can be used to further confirm the similarity weight matrix of the first segmentation model.

[0205] Understandably, referring to Figure 8 , Figure 8 This is a flowchart of the model calculation for the image segmentation method provided in this application embodiment. Step S430 includes, but is not limited to, the following steps:

[0206] Step S431: Based on the third segmentation model, obtain the batch normalized layer parameters and other layer parameters of the third segmentation model.

[0207] Step S432: Obtain the approval layer parameters of the teacher model based on the batch normalization layer parameters of the third segmentation model.

[0208] Step S433: Based on the other layer parameters of the third segmentation model and the similarity weight matrix, obtain the other layer parameters of the teacher model.

[0209] Step S434: Based on the approval layer parameters and other layer parameters of the teacher model, obtain the teacher model corresponding to the third segmentation model.

[0210] It should be noted that the approval layer parameters of the teacher model are obtained based on the batch normalization layer parameters of the third segmentation model. The approval layer parameters of the teacher model use the approval layer parameters of the local model updated in the previous round, while the other layer parameters are obtained by multiplying the network layer parameters with the matrix parameters in the similarity weight matrix, which can be specifically expressed as:

[0211]

[0212] in, This represents the non-batch normalized layer parameters updated in the t-th training round.

[0213] It should be noted that, referring to Figure 12 , Figure 12 This is the general framework of federated learning provided in the embodiments of this application. The image segmentation method provided in the embodiments of this application is mainly divided into two stages. The first stage is an uncertainty-aware dynamic cyclic public knowledge accumulation stage, i.e., the general training stage. It dynamically ranks the performance of each client model in each round of training in an uncertainty-aware manner and accumulates public knowledge in the form of cyclic knowledge distillation. The second stage is a distribution similarity-aware personalized stage, i.e., the personalized training stage. It measures the similarity between each client through statistical information from the batch normalization layer, aggregates the teacher models of each client, and performs knowledge distillation. Each site internally... Figure 11The proposed application employs a weakly supervised learning paradigm for training. Image segmentation is achieved through a federated learning approach combining recurrent knowledge distillation and personalized training. This allows for the accumulation of common knowledge and adaptive personalization across all clients without the need for a central server. It also effectively utilizes information from other client models, resulting in superior image segmentation performance. Furthermore, it improves the performance of image segmentation models across multiple clients, enhancing the overall segmentation effect. In terms of weak supervision, the application utilizes a comprehensive optimization objective combining partial cross-entropy loss, gated conditional random field loss, and tree energy loss. This effectively unifies training with various sparse labeled data, enabling the online generation of more accurate pseudo-labels for edges, thus achieving superior segmentation performance without requiring additional supervised data, alternating optimization processes, or time-consuming post-processing.

[0214] It should be noted that this application uses a server equipped with eight NVIDIA GeForce RTX 3090 graphics cards, running CentOS 7, and with an Intel Xeon Gold 6242R CPU for experiments. The programming language is Python, and the deep learning framework is PyTorch. The architectures of the first and second segmentation models both adopt the U-Net network architecture, with the number of channels from top to bottom being 16, 32, 64, 128, and 256, respectively. The architecture of the third segmentation model, obtained through initialization using a general model and batch normalization layers, is also a U-Net network architecture. The preset first segmentation model is obtained through random initialization, and the optimizer uses the AdamW optimizer with default parameters and an initial learning rate of 10⁻², which is updated with each iteration step.

[0215]

[0216] Among them, l r Indicates the current learning rate, l r0 N represents the initial learning rate, e is the current iteration step, and N is the number of iterations. e The total number of iterations is 30,000. Regarding the selection of hyperparameters, this application, based on internal exploratory experimental experience, considers the aforementioned trade-off parameter λ. u , λ d λ0 and the constant α are all set to 0.5, and the tradeoff parameter λ is set to... t and λ gAll values ​​were set to 0.1. The designed federated method had 50 training epochs in the general training phase and 1000 epochs in the personalized training phase. All comparison methods were deployed in the same experimental environment and used the same learning rate, optimizer, optimization strategy, and model architecture to achieve the fairest possible comparison. Furthermore, this application's embodiments selected the Dice similarity coefficient, Hausdorff distance (HD), precision (Pre), and recall (Rec) as the main metrics, forming a set similarity measure function. The Dice coefficient represents the fullness of the predicted image data and the ground truth label. The mathematical expression of the set similarity measure function is:

[0217]

[0218] The mathematical expression for Hausdorff distance is:

[0219]

[0220]

[0221] in, Represents predicted image data. This indicates the actual label. To predict the pixels in image data relative to the ground truth labels, a larger Dice coefficient and a smaller HD represent better segmentation results. Precision and recall are respectively:

[0222]

[0223]

[0224] In this application, TP, FP, and FN represent true positives, false positives, and false negatives, respectively. The embodiments of this application compare the performance of the proposed image segmentation method with several representative federated learning frameworks, including traditional centralized federated methods (such as FedAvg and FedProx) and state-of-the-art personalized federated methods (such as FT, FedBN, FedAP, FedRep, etc.). In terms of implementation, since these methods were originally designed for image classification tasks, this application strives to maintain their design principles and adapt them to weakly supervised image segmentation tasks. Furthermore, this application also performs fully supervised local training, weakly supervised centralized training, and fully supervised centralized training. Centralized training refers to merging and directly training data from various sites on a central server without considering data privacy and security. Referring to Tables 1 and 2, Table 1 shows the Dice coefficients of different federated learning algorithms on the view cup / disc segmentation task, and Table 2 shows the Hausdorff distance of different federated learning algorithms on the view cup / disc segmentation task. Firstly, by focusing on the total metrics in the last column of the tables, it can be seen that, compared to local training, all federated learning methods can effectively improve the overall performance of the models at each site on their respective test data. Client D saw a significant performance improvement after joining federated learning, mainly because client D used point labeling, the weakest form of supervision signal; joining the federated learning allowed it to acquire effective knowledge from other sites. Client F also achieved a significant performance improvement after joining the federated learning, because the data distribution at this site varied greatly, making it difficult to train a powerful, generalized deep segmentation model using only data from a single site. From the perspective of method categories, personalized federated methods, because they can personalize the model to cope with the diverse data distributions at each site, generally outperform traditional centralized federated frameworks. It is worth noting that the personalized federated learning method proposed in this application achieves the best performance, with each client benefiting from it. Most clients achieve the best or second-best performance among all federated methods, and it is the only method to achieve a Dice coefficient higher than 90 in the overall performance of sight cup / disc segmentation, significantly outperforming the performance of centralized training with weak supervision, and achieving performance closest to local training with full supervision and centralized training with full supervision. This means that, while protecting privacy, each client can train a high-performance model through data collaboration among multiple clients, and each client can use different sparse annotation methods, greatly saving annotation costs.

[0225] Table 1

[0226]

[0227]

[0228] Table 2

[0229]

[0230] Referring to Table 3, which shows the performance of different federated learning algorithms on the foveal avascular region segmentation task, the results of each algorithm on the foveal avascular region segmentation task further verify the improvement brought by the proposed method. The foveal avascular region segmentation task is more challenging due to the problem of insufficient client data (such as client E). In addition, due to the different image quality issues and distribution differences among clients, there is only a small improvement or even a performance degradation in centralized training and some federated learning methods, and there is negative interference between clients. Compared with local training, not all federated methods can bring performance improvements to each client. For example, FedBN, FedAP and MetaFed show that simply personalizing some parameters does not bring significant positive effects to the models of each client. The shared part of the model is still susceptible to negative interference from other clients due to distribution differences, quality differences and other factors. At the same time, static circular knowledge distillation is also susceptible to the invalid knowledge brought by dissimilar client models acting as teacher models. The method proposed in this application, through a public knowledge accumulation stage and a similarity aggregation knowledge distillation stage, can condense and accumulate similar and effective knowledge from various client data. In the second stage, it assigns higher weights to similar clients, thereby distilling beneficial knowledge, and assigns lower weights to different clients to reduce interference. Similarly, the method proposed in this application can provide performance that rivals both locally trained fully supervised and centrally trained fully supervised methods.

[0231] Table 3

[0232]

[0233] Reference Figure 13 , Figure 13 These are the segmentation results corresponding to various image segmentation methods provided in the embodiments of this application. Figure 13 The image segmentation method of this application is visually compared with the segmentation results of other federated methods. Thanks to the proposed personalized method and superior weakly supervised loss, the embodiments of this application achieve better performance in terms of edge accuracy, topological structure, and outlier suppression in the target region.

[0234] It should be noted that, referring to Figure 14 , Figure 14This document presents an ablation experiment of the image segmentation method provided in this application. To demonstrate the effectiveness of the proposed weakly supervised loss and federated paradigm, Table 4 shows the results of the ablation experiments on key components. This application conducted ablation experiments on five key components on a dataset for segmenting avascular regions of the fovea, namely tree energy loss and gated conditional random field loss in the weakly supervised loss. Table 4 shows that the elimination of any loss term leads to a certain degree of performance degradation in segmentation, resulting in decreased edge accuracy or an increase in discrete prediction regions. Subsequently, ablation was performed on the two stages of the image segmentation method—general training and personalized training. It can be seen that with the removal of the general training stage, the overall performance of each client significantly decreases, indicating that dynamic cyclic distillation of knowledge from each client allows each client model to retain effective knowledge obtained from the central training data while discarding redundant knowledge, which is crucial for the final model performance. The elimination of the personalized training stage causes a smaller performance degradation, indicating that even using only the model from the public knowledge accumulation stage can achieve good segmentation results. Even so, the personalized stage still improves the overall performance of the models at each site by approximately 2.2 in Dice. Furthermore, by eliminating the similarity weights used in the teacher models during the personalized phase and replacing them with weights based on sample proportions in FedAvg, the models cannot efficiently learn effective knowledge from similar client segmentation models and are easily negatively affected by dissimilar segmentation models, resulting in a significant performance degradation. This application also analyzes the contribution of each client's data to the entire federation by sequentially ablating each client. Simultaneously, with the sequential ablation of each site, the similarity and mutual influence between data from different sites can be analyzed. Within the framework of the image segmentation model in this application, the comprehensive average Dice coefficients of sites without site A, B, C, D, and E, and those under the full-site setting, are 88.17, 88.63, 88.19, 87.98, 89.14, and 90.38, respectively, and the Hausdorff distances are 13.18, 12.59, 11.562, 12.42, 11.74, and 9.63, respectively. From... Figure 5The detailed results show that each client achieved optimal performance in the full client configuration, indicating that despite differences in data distribution across centers, all clients can learn better personalized models from as many federated clients as possible. For client A, the lowest performance was achieved when client D exited, indicating a high degree of data similarity between the two centers, which effectively promotes mutual performance improvement. Similarly, for client D, client A's exit caused the largest performance drop. Likewise, it can be seen that client E has a large data distribution gap with other clients. With client E's exit, most of the remaining clients achieved the best or second-best performance under the four-client federation condition. The practical implication is that client E may need to provide more samples or higher-quality annotations to increase its contribution to the federation. Conversely, client D's exit led to a significant drop in the overall performance of the remaining clients, possibly because client D contributed the largest sample size among all centers, and other centers should allow it to use a sparser (relaxed), efficient annotation method.

[0235] Table 4

[0236] ablation experiment Dice coefficient Hausdorf Distance Recall rate accuracy No tree energy loss 88.47 12.67 89.81 89.08 Ungated conditional random field loss 88.41 11.84 90.5 88.58 Stage without public knowledge accumulation 85.45 33.42 88.39 85.84 No similarity weight 87.67 15.49 89.77 87.75 Unpersonalized stage 88.19 11.49 89.73 89.26 This article's method 90.38 9.63 91.2 91.34

[0237] In summary, the image segmentation method proposed in this application has excellent performance. It can extract effective common information from each client and perform adaptive personalization. Under the setting that each client has different forms of weakly supervised sparse labels, it achieves performance close to that of local training or even centralized training using fully supervised labels.

[0238] Additionally, refer to Figure 15 This application provides a schematic diagram of a system architecture platform for image segmentation.

[0239] The system architecture platform of this application embodiment includes one or more processors and memory. Figure 15 The example consists of a processor and a memory.

[0240] The processor and memory can be connected via a bus or other means. Figure 15 Taking the example of a connection between China and Israel via a bus.

[0241] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the system architecture platform via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0242] Those skilled in the art will understand that this system architecture platform can be applied to existing communication network systems and subsequent evolved mobile communication network systems, and this embodiment does not specifically limit it.

[0243] Those skilled in the art will understand that Figure 15 The device structure shown does not constitute a limitation on the system architecture platform and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0244] The system architecture platform can be an independent system architecture platform, or it can be a cloud system architecture platform that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms.

[0245] In addition, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program as in the image segmentation method of steps S100 to S500.

[0246] The processor and memory can be connected via a bus or other means.

[0247] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0248] The non-transient software program and instructions required to implement the target tracking method of the above embodiments are stored in memory. When executed by a processor, the image segmentation method described above is executed, for example, the method described above. Figure 2 Method steps S100 to S500.

[0249] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0250] Furthermore, one embodiment of this application also provides a computer-readable storage medium storing computer-executable instructions. These instructions, executed by a processor or controller, cause the processor to perform the image segmentation method described above, for example, the method described above. Figure 2 Method steps S100 to S500 Figure 3 Method steps S310 to S360 in the text Figure 4 Method steps S3111 to S3115 in the text Figure 5 Method steps S3121 and S3128 in the text Figure 6 Method steps S410 to S460 in the text Figure 7 Method steps S421 to S425 in the text Figure 8 Method steps S431 to S434.

[0251] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0252] The above is a detailed description of the preferred embodiments of this application. However, the embodiments of this application are not limited to the above-described implementation methods. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the embodiments of this application. All such equivalent modifications or substitutions are included within the scope defined by the claims of the embodiments of this application.

Claims

1. An image segmentation method, characterized in that, The image segmentation method is applied to a first client, which communicates with multiple second clients, and includes: Obtain real-time parameters of the second segmentation model corresponding to multiple second clients; Based on the preset first segmentation model, the batch standardization layer is obtained; Based on the first segmentation model and the real-time parameters, the first segmentation model and the second segmentation model are dynamically sorted and cyclically distilled to obtain a general model; The image segmentation model is obtained by performing personalized training on the model of the first client based on the general model, the batch normalization layer, and the real-time parameters. The image to be segmented is obtained, and the image to be segmented is processed by the image segmentation model to obtain the segmentation result of the image to be segmented; The real-time parameters include the performance parameters of the second segmentation model. The step of dynamically ranking the first segmentation model and the second segmentation model based on the first segmentation model and the real-time parameters, and performing cyclic knowledge distillation to obtain a general model, includes: Acquire sample image data and input the sample image data into the first segmentation model to obtain the performance parameters and weakly supervised segmentation loss of the first segmentation model; The first segmentation model and the second segmentation model are dynamically sorted according to their performance parameters. Based on the results of dynamic sorting, the teacher model of the first segmentation model is determined; Based on the first segmentation model and the teacher model, the knowledge distillation loss of the first segmentation model is obtained; The first segmentation model is trained using the knowledge distillation loss and the weakly supervised segmentation loss until the performance parameters of the first segmentation model meet the first preset condition. Based on the first segmentation model after training, a general model is obtained.

2. The image segmentation method according to claim 1, characterized in that, The step of inputting the sample image data into the first segmentation model to obtain the performance parameters of the first segmentation model includes: Obtain the true labels of the sample image data; The sample image data is input into the first segmentation model to obtain the predicted image data and the normalized exponential probability vector. Based on the predicted image data and the true labels, a set similarity measurement function is obtained; Based on the normalized exponential probability vector, the prediction entropy of the first segmentation model is obtained, and the prediction entropy is used to characterize the uncertainty of the first segmentation model. The performance parameters of the first segmentation model are obtained based on the set similarity metric function and the uncertainty.

3. The image segmentation method according to claim 1, characterized in that, The step of inputting the sample image data into the first segmentation model to obtain the weakly supervised segmentation loss of the first segmentation model includes: Based on the sample image data, the corresponding sparse labels are obtained; The sample image data is input into the first segmentation model to obtain predicted image data, predicted probability, and image selection features. The image selection features are the image features of the sample image data passing through a preset layer of the first segmentation model. Tree affinity is generated by performing tree affinity on the sample image data and the selected features of the image to obtain a low-order affinity matrix and a high-order affinity matrix; The low-order affinity matrix and the high-order affinity matrix are passed through a cascaded filter to obtain the soft pseudo-labels corresponding to the sample image data; The tree energy loss is obtained based on the predicted probability and the soft pseudo-label. Based on the predicted image data and the sparse labels, a partial cross-entropy loss is obtained; Based on the predicted probability and the preset source mask and preset target mask, the gated conditional random field loss is obtained; The weakly supervised segmentation loss of the first segmentation model is obtained based on the tree energy loss, the partial cross-entropy loss, and the gated conditional random field loss.

4. The image segmentation method according to claim 1, characterized in that, The step of performing personalized training on the model of the first client based on the general model, the batch normalization layer, and the real-time parameters to obtain an image segmentation model includes: The model of the first client is initialized according to the general model and the batch normalization layer to obtain the third segmentation model; Based on the third segmentation model and the real-time parameters, the similarity weight matrix is ​​obtained; Based on the third segmentation model and the similarity weight matrix, the teacher model of the third segmentation model is obtained; Based on the third segmentation model and the teacher model, the knowledge distillation loss and weakly supervised segmentation loss of the third segmentation model are obtained; The third segmentation model is trained using the knowledge distillation loss and the weakly supervised segmentation loss until the performance parameters of the third segmentation model meet the second preset condition. The image segmentation model is obtained based on the third segmentation model after training.

5. The image segmentation method according to claim 4, characterized in that, The real-time parameters include statistics from the second segmentation model. The process of obtaining the similarity weight matrix based on the third segmentation model and the real-time parameters includes: Based on the third segmentation model, the statistics of the third segmentation model are obtained; Based on the statistics of the third segmentation model and the second segmentation model, the optimal transmission distance between the third segmentation model and multiple second segmentation models is obtained; Based on multiple optimal transmission distances, the similarity between the third segmentation model and multiple second segmentation models is obtained; The similarity scores are then normalized and updated using a moving average in sequence to obtain the similarity weights. Based on the multiple similarity weights, a similarity weight matrix is ​​obtained.

6. The image segmentation method according to claim 4, characterized in that, The step of obtaining the teacher model of the third segmentation model based on the third segmentation model and the similarity weight matrix includes: Based on the third segmentation model, the batch normalized layer parameters and other layer parameters of the third segmentation model are obtained; Based on the batch normalization layer parameters of the third segmentation model, the approval layer parameters of the teacher model are obtained; Based on the other layer parameters of the third segmentation model and the similarity weight matrix, the other layer parameters of the teacher model are obtained. Based on the approval layer parameters and other layer parameters of the teacher model, the teacher model corresponding to the third segmentation model is obtained.

7. An image segmentation apparatus, characterized in that, Applied to a first client, which communicates with multiple second clients, including: The data communication module is used to obtain real-time parameters of the second segmentation model corresponding to multiple second clients; The distributed processing module is used to obtain batch standardization layers based on a preset first segmentation model; A general training module is used to dynamically sort the first segmentation model and the second segmentation model and perform cyclic knowledge distillation based on the first segmentation model and the real-time parameters to obtain a general model. The personalized training module is used to perform personalized training on the model of the first client based on the general model, the batch normalization layer and the real-time parameters to obtain an image segmentation model; The image segmentation module is used to acquire the image to be segmented and to pass the image to be segmented through the image segmentation model to obtain the segmentation result of the image to be segmented; The real-time parameters include the performance parameters of the second segmentation model. The step of dynamically ranking the first segmentation model and the second segmentation model based on the first segmentation model and the real-time parameters, and performing cyclic knowledge distillation to obtain a general model, includes: Acquire sample image data and input the sample image data into the first segmentation model to obtain the performance parameters and weakly supervised segmentation loss of the first segmentation model; The first segmentation model and the second segmentation model are dynamically sorted according to their performance parameters. Based on the results of dynamic sorting, the teacher model of the first segmentation model is determined; Based on the first segmentation model and the teacher model, the knowledge distillation loss of the first segmentation model is obtained; The first segmentation model is trained using the knowledge distillation loss and the weakly supervised segmentation loss until the performance parameters of the first segmentation model meet the first preset condition. Based on the first segmentation model after training, a general model is obtained.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the image segmentation method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the image segmentation method according to any one of claims 1 to 6.

Images