Image segmentation method, image segmentation device, electronic device, and storage medium

Abstract

The application discloses an image segmentation method, an image segmentation device, an electronic device and a storage medium, and is applied to the field of image processing and comprises the following steps: acquiring local parameters and global parameters and personalized parameters from a server; initializing a global model according to the global parameters; judging whether to perform adaptive head aggregation, generating aggregation parameters, and initializing a personalized model through the aggregation parameters; performing local training on the global model and the personalized model to obtain an image segmentation model, and uploading the image segmentation model to the server; and performing iterative training on the image segmentation model according to the updated global parameters and personalized parameters until the image segmentation model obtained through the last round of iterative training reaches a convergence state. Through adaptive head aggregation, the personalized model is initialized, the common knowledge of multiple client models and the unique knowledge of a local client model can be effectively integrated, personalized modeling is promoted, and the performance of the image segmentation model is improved.

Description

Technical Field

[0001] This invention relates to the field of image processing, and more specifically to an image segmentation method, an image segmentation apparatus, an electronic device, and a storage medium. Background Technology

[0002] Image segmentation is a representative task supporting computer-aided image content analysis. For medical images, image segmentation can not only identify lesion categories but also locate specific regions, playing a crucial role in clinical diagnosis. Current technologies typically employ federated learning for image segmentation. Federated learning can help users collaboratively model while ensuring data privacy and security, thereby enhancing model performance. However, in federated learning, each client has its own learning task and dataset, meaning the data from different clients may not be independently and identically distributed, i.e., statistical heterogeneity. This impairs the generalization performance of the image segmentation model, resulting in significant inconsistencies in performance across client models and poor image segmentation results. Summary of the Invention

[0003] This application provides an image segmentation method, an image segmentation apparatus, an electronic device, and a storage medium, which can effectively promote personalized modeling and improve the image segmentation effect of various customer models.

[0004] In a first aspect, this application provides an image segmentation method applied to a first client, the first client being connected to a server, the image segmentation method comprising:

[0005] Obtain local parameters and global and personalized parameters from the server;

[0006] The global model is initialized based on the global parameters;

[0007] Based on the comparison results between the current training iteration and the preset iteration, determine whether to perform adaptive head aggregation on the local parameters and the personalized parameters to generate aggregated parameters, and initialize the personalized model using the aggregated parameters;

[0008] The initialized global model and the personalized model are trained locally to obtain an image segmentation model, and the image segmentation model is uploaded to the server so that the server can update the global parameters and the personalized parameters.

[0009] The image segmentation model is trained in a new round of iterations based on the updated global parameters and the personalized parameters until the image segmentation model obtained from the last round of iterations reaches a convergent state.

[0010] The image to be segmented is input into the image segmentation model that has reached a convergence state to obtain the segmentation result of the image to be segmented.

[0011] The image segmentation method provided by the first aspect of the embodiments of this application has at least the following beneficial effects: During the training process of the image segmentation model, it is determined whether to perform adaptive head aggregation based on the comparison result between the current iteration training round and the preset round. When the current iteration training round is less than the preset round, there is less common knowledge in the personalized parameters. In order to ensure the personalization of the image segmentation model, the local parameters are directly used as aggregation parameters. When the current iteration training round is equal to or greater than the preset round, in order to alleviate the over-personalization of the personalized model, adaptive head aggregation is performed, so that the obtained personalized model contains the common knowledge between multiple client models and the unique knowledge of the local client model. The personalized parameters come from the server and contain the common knowledge between multiple client models. The local parameters come from the first client and contain the unique knowledge of the local client model. Adaptive head aggregation can realize the aggregation of local parameters and personalized parameters, so that the obtained personalized model contains the common knowledge between multiple client models and the unique knowledge of the local client model, effectively promoting personalized modeling and alleviating over-personalization, thereby obtaining excellent image segmentation performance, while improving the performance of the image segmentation models of each client and improving the image segmentation effect.

[0012] According to some embodiments of the first aspect of this application, determining whether to perform adaptive head aggregation on the local parameters and the personalized parameters to generate aggregated parameters based on the comparison result of the current training iteration and the preset iteration includes:

[0013] Get the current training iteration round;

[0014] When the current training iteration is less than the preset iteration, the local parameters are used as aggregate parameters;

[0015] When the current training iteration round is equal to the preset round, obtain the weight matrix, and perform multiple adaptive head aggregations on the weight matrix, the personalized parameters, and the local parameters to obtain the aggregated parameters;

[0016] When the current training iteration round is greater than the preset round, the weight matrix is ​​obtained, and a single adaptive head aggregation is performed on the weight matrix, the personalized parameters, and the local parameters to obtain the aggregated parameters;

[0017] The adaptive head aggregation includes the following steps: After each adaptive head aggregation, the weight matrix is ​​updated.

[0018] Calculate the difference between the personalized parameter and the local parameter;

[0019] Calculate the Hadamard product of the difference between the personalized parameter and the local parameter and the weight matrix;

[0020] The Hadamard product is added to the local parameter to obtain the aggregate parameter.

[0021] According to some embodiments of the first aspect of this application, updating the weight matrix includes:

[0022] Use the weight matrix as the initial matrix;

[0023] Image training samples are obtained, and gradient descent is used to process the aggregation parameters, the initial matrix, and the image training samples to obtain an updated weight matrix.

[0024] When the current training iteration is equal to the preset iteration, the aggregate parameters are updated according to the updated weight matrix, the personalized parameters, and the local parameters, and the weight matrix is ​​updated according to the updated aggregate parameters, the initial matrix, and the image training samples, until the weight matrix converges.

[0025] According to some embodiments of the first aspect of this application, the server is further connected to a plurality of second clients, and the step of locally training the initialized global model and the personalized model to obtain an image segmentation model includes:

[0026] Obtain image training samples;

[0027] The image training samples are input into the global model to obtain the first site channel attention value and global image features;

[0028] Obtain multiple second site channel attention values ​​from the server, wherein the second site channel attention values ​​are generated by the second client;

[0029] Based on the first site channel attention value and multiple second site channel attention values, the site embedding difference loss is obtained;

[0030] The global image features are input into the personalized model to obtain multiple personalized image features;

[0031] Based on the image training samples and multiple personalized image features, a weakly supervised segmentation loss is obtained;

[0032] The global model and the personalized model are trained based on the site embedding difference loss and the weakly supervised segmentation loss until the global model and the personalized model satisfy the conditions of local gradient descent.

[0033] Based on the global model and the personalized model after training, an image segmentation model is obtained.

[0034] According to some embodiments of the first aspect of this application, the global model includes an encoder and a site contrast representation module, wherein inputting the image training samples into the global model to obtain first site channel attention values ​​and global image features includes:

[0035] The image training samples are input into the encoder to obtain the encoded image features;

[0036] Global average pooling is performed on the encoded image features to obtain global average features;

[0037] Obtain the site code, and pass the site code through two fully connected layers to obtain the site connection features;

[0038] The global average feature and the site connection feature are channel-connected, and the result after channel connection is activated to obtain the first site channel attention value;

[0039] The first site channel attention value is multiplied element-wise with the encoded image feature, and the result of the multiplication is added to the encoded image feature to obtain the global image feature.

[0040] According to some embodiments of the first aspect of this application, obtaining the site embedding difference loss based on the first site channel attention value and a plurality of second site channel attention values ​​includes:

[0041] Calculate the stopping gradient values ​​of multiple second-site channel attention values;

[0042] Calculate the difference between the first site channel attention value and the plurality of stopping gradient values;

[0043] Calculate the sum of the multiple said differences;

[0044] The site embedding difference loss for the first client is obtained based on the sum of multiple differences and the number of the second client.

[0045] According to some embodiments of the first aspect of this application, multiple personalized image features include a first personalized feature, a second personalized feature, and a predicted image feature. The step of obtaining a weakly supervised segmentation loss based on the image training samples and the multiple personalized image features includes:

[0046] Based on the image training samples and the predicted image features, determine the prediction probability of the image training samples;

[0047] The image training samples are subjected to sparse processing to obtain corresponding sparse labels;

[0048] Based on the sparse labels and the predicted probabilities, a partial cross-entropy loss is obtained;

[0049] Based on the predicted probability and the preset source mask and preset target mask, the gated conditional random field loss is obtained;

[0050] Tree affinity is generated from the image training samples to obtain a low-order affinity matrix;

[0051] Tree affinity is generated from the first personalized feature and the second personalized feature to obtain two high-order affinity matrices. The first personalized feature and the second personalized feature are the output results of different network layers of the personalization module.

[0052] Based on the predicted probability, the low-order affinity matrix and the two high-order affinity matrices are sequentially cascaded and filtered to obtain the pseudo-labels corresponding to the image training samples;

[0053] Based on the predicted probability and the pseudo-label, the multi-scale recursive tree energy loss is obtained;

[0054] The tree energy loss, the partial cross-entropy loss, and the gated conditional random field loss are added together according to preset trade-off parameters to obtain the weakly supervised segmentation loss.

[0055] Secondly, this application provides an image segmentation apparatus applied to a first client, the first client being connected to a server, the image segmentation apparatus comprising:

[0056] The data communication module is used to acquire local parameters and global and personalized parameters from the server.

[0057] A global initialization module is used to initialize the global model based on the global parameters;

[0058] The personalized initialization module is used to determine whether to perform adaptive head aggregation on the local parameters and the personalized parameters based on the comparison results between the current training iteration and the preset iteration, so as to generate aggregated parameters and initialize the personalized model through the aggregated parameters;

[0059] The local training module is used to train the initialized global model and the personalized model locally to obtain an image segmentation model, and upload the image segmentation model to the server so that the server can update the global parameters and the personalized parameters.

[0060] The iterative training module is used to perform a new round of iterative training on the image segmentation model based on the updated global parameters and the personalized parameters, until the image segmentation model obtained from the last round of iterative training reaches a convergent state.

[0061] The image segmentation module is used to obtain the segmentation result of the image to be segmented by passing it through the trained image segmentation model.

[0062] Since the image segmentation apparatus provided in the second aspect applies the image segmentation method of any one of the first aspects, it has all the beneficial effects of the first aspect of this application.

[0063] Thirdly, this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program as the image segmentation method of any of the first aspects.

[0064] Since the processor of the third aspect executes the computer program as the image segmentation method of any of the first aspects, it has all the beneficial effects of the first aspect of the embodiments of this application.

[0065] Fourthly, embodiments of this application provide a computer storage medium including computer-executable instructions stored thereon, the computer-executable instructions being used for the image segmentation method as described in any of the first aspects.

[0066] Since the computer storage medium of the fourth aspect can perform the image segmentation method as described in any of the first aspects, it has all the beneficial effects of the first aspect of the embodiments of this application.

[0067] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0068] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0069] Figure 1 This is a system architecture diagram of the image segmentation method provided in the embodiments of this application.

[0070] Figure 2 This is a flowchart of the main image segmentation method provided in the embodiments of this application;

[0071] Figure 3 This is a flowchart of the aggregation judgment of the image segmentation method provided in the embodiments of this application;

[0072] Figure 4This is a flowchart of the adaptive head aggregation of the image segmentation method provided in the embodiments of this application;

[0073] Figure 5 This is a flowchart of the weight matrix update process for the image segmentation method provided in this application embodiment;

[0074] Figure 6 A flowchart illustrating the steps of local training for the image segmentation method provided in this application embodiment;

[0075] Figure 7 A flowchart illustrating the steps of global model processing for the image segmentation method provided in this application embodiment;

[0076] Figure 8 A flowchart illustrating the steps of calculating the site embedding difference loss in the image segmentation method provided in this application embodiment;

[0077] Figure 9 A flowchart illustrating the steps of calculating the weakly supervised segmentation loss in the image segmentation method provided in this application embodiment;

[0078] Figure 10 This is a network architecture diagram of the image segmentation method provided in the embodiments of this application;

[0079] Figure 11 This is a network architecture diagram of the weakly supervised training paradigm provided in the embodiments of this application;

[0080] Figure 12 This is a schematic diagram of the training samples of each client provided in the embodiments of this application, as well as various sparse labels and fully supervised labels corresponding to the images.

[0081] Figure 13 This is a schematic diagram of sparse tag generation provided in an embodiment of this application;

[0082] Figure 14 This is a structural block diagram of the image segmentation apparatus provided in the embodiments of this application;

[0083] Figure 15 This is a schematic diagram of a system architecture platform for image segmentation provided in an embodiment of this application. Detailed Implementation

[0084] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that the embodiments of this application can also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the embodiments of this application with unnecessary detail.

[0085] It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0086] It should also be understood that references to "one embodiment" or "some embodiments" in the specification of embodiments of this application mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0087] Image segmentation is a representative task supporting computer-aided image content analysis. For medical images, image segmentation can not only identify lesion categories but also locate specific regions, playing a crucial role in clinical diagnosis. Current technologies typically employ federated learning for image segmentation. Federated learning can help users collaboratively model while ensuring data privacy and security, thereby enhancing model performance. However, in federated learning, each client has its own learning task and dataset, meaning the data from different clients may not be independently and identically distributed, i.e., statistical heterogeneity. This impairs the generalization performance of the image segmentation model, resulting in significant inconsistencies in performance across client models and poor image segmentation results.

[0088] Based on this, this application provides an image segmentation method, an image segmentation apparatus, an electronic device, and a storage medium. The image segmentation method provided in this application updates and initializes the personalized model at the element level through adaptive head aggregation, which can effectively integrate the common knowledge of the global model and the unique knowledge of the personalized model, effectively promoting personalized modeling while also alleviating over-personalization, thereby obtaining excellent image segmentation performance. At the same time, it improves the performance of the image segmentation model of each client and improves the image segmentation effect.

[0089] The present application will be further described below with reference to the accompanying drawings.

[0090] Reference Figure 1 , Figure 1This is a system architecture diagram of the image segmentation method provided in this application embodiment. The image segmentation method provided in this application embodiment is applied to a federated learning system, which includes a server and multiple clients. Each client has an image segmentation model, which is trained by receiving data from the server and local data. The trained image segmentation model has good image segmentation performance. The server can receive image segmentation models from multiple clients, integrate and process them, and distribute the processing results to the clients. This allows multiple clients to collaboratively train a powerful image segmentation model without affecting data privacy and security, and the image segmentation model can retain common knowledge among the client models.

[0091] A client is a terminal or program running on a terminal that can initiate requests to a server. Terminals include various forms such as desktop computers, laptops, PDAs (Personal Digital Assistants), mobile phones, in-vehicle clients, and dedicated clients. Furthermore, it can be a single device or a collection of multiple devices. For example, multiple desktop computers connected via a local area network, sharing a single monitor, can work collaboratively to form a single client. The client can communicate with the server and exchange data via wired or wireless means.

[0092] A server is a computer system that provides certain services to clients. Compared to ordinary clients, servers have higher requirements in terms of stability, security, and performance. A server can be a single high-performance computer in a network platform, a cluster of multiple high-performance computers, a portion of a single high-performance computer (such as a virtual machine), or a combination of portions of multiple high-performance computers (such as virtual machines).

[0093] This application provides an image segmentation method that uses adaptive head aggregation to aggregate local and personalized parameters. This results in a personalized model that incorporates common knowledge among multiple client models and unique knowledge of the local client model, effectively promoting personalized modeling and mitigating over-personalization. This leads to excellent image segmentation performance and improves the performance of image segmentation models on each client, thus enhancing the overall image segmentation effect.

[0094] The image segmentation method of this disclosure can be executed on a client, on a server, or partly on a client and partly on a server.

[0095] Reference Figure 2 , Figure 2This is a main flowchart of the image segmentation method provided in this application embodiment. The image segmentation method is applied to a first client, which is connected to a server. The image segmentation method includes, but is not limited to, the following steps:

[0096] Step S100: Obtain local parameters and global parameters and personalized parameters from the server.

[0097] Step S200: Initialize the global model based on the global parameters.

[0098] Step S300: Based on the comparison results between the current iteration training round and the preset round, determine whether to perform adaptive head aggregation on the local parameters and personalized parameters to generate aggregated parameters, and initialize the personalized model using the aggregated parameters.

[0099] Step S400: Train the initialized global model and personalized model locally to obtain an image segmentation model, and upload the image segmentation model to the server so that the server can update the global parameters and personalized parameters.

[0100] Step S500: Perform a new round of iterative training on the image segmentation model based on the updated global parameters and personalized parameters until the image segmentation model obtained from the last round of iterative training reaches a convergent state.

[0101] Step S600: Input the image to be segmented into the image segmentation model that has reached convergence to obtain the segmentation result of the image to be segmented.

[0102] It should be noted that during the training of the image segmentation model, the decision to perform adaptive head aggregation is based on a comparison between the current training iteration and a preset iteration. When the current training iteration is less than the preset iteration, there is less common knowledge in the personalized parameters. To ensure the personalization of the image segmentation model, local parameters are directly used as aggregation parameters. However, when the current training iteration is equal to or greater than the preset iteration, adaptive head aggregation is performed to mitigate over-personalization of the personalized model. This results in a personalized model that incorporates common knowledge among multiple client models and unique knowledge from the local client model. Personalized parameters come from the server and contain common knowledge among multiple client models, while local parameters come from the first client and contain unique knowledge from the local client model. Adaptive head aggregation effectively aggregates local and personalized parameters, resulting in a personalized model that incorporates both common and unique knowledge from multiple client models. This promotes personalized modeling and mitigates over-personalization, thereby achieving excellent image segmentation performance and improving the performance of image segmentation models on each client, ultimately enhancing the overall image segmentation effect.

[0103] It should be noted that steps S100 to S600 can be performed on any client in the federated learning system. For ease of description, one of the clients in the federated learning system is defined as the first client.

[0104] It should be noted that for step S100, the global and personalized parameters differ for different training iterations. In the initial training, the global and personalized parameters are generated by the server based on the initial model. In subsequent iterations, different clients have different weights in federated learning. The server performs a weighted average of the image segmentation models received from all clients based on their weights, and generates global and personalized parameters based on the weighted average model. The global and personalized parameters received by the first client contain model information from other clients. Furthermore, personalized model initialization requires the use of aggregated parameters, which are generated through adaptive header aggregation. These aggregated parameters retain the common knowledge of the global model and the unique knowledge of the personalized model, enabling element-level updates and initialization of the personalized model. The generation of the global and personalized models improves the performance of the client's image segmentation model and enhances the image segmentation effect.

[0105] It should be noted that local parameters are the parameters of the personalized model stored locally. Obtaining local parameters is beneficial for the aggregation of unique knowledge of the local client model, i.e., the model of the first client, during the initial process of personalized modeling.

[0106] Understandably, referring to Figure 3 , Figure 3 This is a flowchart of the aggregation judgment of the image segmentation method provided in the embodiments of this application. Step S300 includes, but is not limited to, the following steps:

[0107] Step S310: Obtain the current iteration training round.

[0108] Step S320: When the current iteration training round is less than the preset round, use the local parameters as the aggregation parameters.

[0109] Step S330: When the current iteration training round is equal to the preset round, obtain the weight matrix, and perform multiple adaptive head aggregations on the weight matrix, personalized parameters, and local parameters to obtain aggregated parameters.

[0110] Step S340: When the current iteration training round is greater than the preset round, obtain the weight matrix, and perform a single adaptive head aggregation on the weight matrix, personalized parameters and local parameters to obtain the aggregated parameters.

[0111] It should be noted that after each adaptive head aggregation, the weight matrix is ​​updated to facilitate the next round of adaptive head aggregation.

[0112] It should be noted that this application sets the preset number of rounds to two. In step S320, when the current iteration training round is less than the preset number of rounds, that is, when the current iteration training round is 1, there is less common knowledge in the personalized parameters. In order to ensure the personalization of the image segmentation model, the local parameters are directly used as aggregation parameters to initialize the personalized model through the aggregation parameters.

[0113] It should be noted that in step S330, when the current iteration training round is equal to the preset round, i.e., the current iteration training round is 2, in order to aggregate the common knowledge between multiple client models and the unique knowledge of the local client model without introducing multiple aggregate weight matrices, this application uses an adaptive head aggregation method to aggregate the common knowledge between multiple client models and the unique knowledge of the local client model to generate aggregation parameters. This method is similar to residual learning for updating the weight matrix. A single adaptive head aggregation is performed based on the weight matrix, personalized parameters, and local parameters to generate aggregation parameters. The aggregation parameters can update the weight matrix. Then, multiple adaptive head aggregations are performed to alternately update the weight matrix and aggregation parameters. The updated aggregation parameters can be used to initialize the personalized model.

[0114] It should be noted that the weight matrix obtained in step S330 is convergent. In subsequent training processes, in order to improve training speed and save training time, when the current training iteration is greater than the preset iteration, the weight matrix of the previous training iteration is obtained, and only the weight matrix, personalized parameters and local parameters are subjected to a single adaptive head aggregation to generate aggregated parameters and update the weight matrix.

[0115] It should be noted that the weight matrix contains multiple elements. Adaptive head aggregation through the weight matrix can achieve element-level updates of the personalized model, resulting in a more efficient personalized model.

[0116] Understandably, referring to Figure 4 , Figure 4 This is a flowchart of the adaptive head aggregation of the image segmentation method provided in this application embodiment. Adaptive head aggregation includes, but is not limited to, the following steps:

[0117] Step S351: Calculate the difference between the personalized parameters and the local parameters.

[0118] Step S352: Calculate the Hadamard product of the difference between the personalized parameters and the local parameters and the weight matrix.

[0119] Step S353: Add the Hadamard product to the local parameters to obtain the aggregate parameters.

[0120] It should be noted that the weight matrix is ​​represented as W. iThen, the aggregation parameters are calculated based on the personalized parameters, local parameters, and weight matrix. The aggregation parameters can be represented as:

[0121]

[0122] in, Here, k represents the first client, and t represents the current training iteration round. For local parameters, For personalized parameters, := is the assignment symbol, and ⊙ is the Hadamard product.

[0123] It should be noted that adaptive head aggregation can aggregate local parameters and personalized parameters, so that the resulting personalized model contains common knowledge among multiple client models and unique knowledge of the local client model. This effectively promotes personalized modeling and alleviates over-personalization, thereby achieving excellent image segmentation performance. At the same time, it improves the performance of image segmentation models of each client and improves the image segmentation effect.

[0124] It should be noted that, referring to Figure 5 , Figure 5 This is a flowchart of the weight matrix update process for the image segmentation method provided in this application embodiment. Step S320 includes, but is not limited to, the following steps:

[0125] Step S321: Use the weight matrix as the initial matrix;

[0126] Step S322: Obtain image training samples, and use gradient descent to process the aggregation parameters, initial matrix and image training samples to obtain the updated weight matrix.

[0127] Step S323: When the current iteration training round is equal to the preset round, update the aggregate parameters according to the updated weight matrix, personalized parameters and local parameters, and update the weight matrix according to the updated aggregate parameters, initial matrix and image training samples, until the weight matrix converges.

[0128] It should be noted that the weight matrix is ​​used as the initial matrix, and the initial matrix is ​​expressed as follows: The weight matrix is ​​represented as W i Where t is the current training iteration round, when the current iteration number equals the preset round number, each element in the weight matrix is ​​initialized to 1, and then the weight matrix is ​​iteratively updated using gradient descent, which can be expressed as:

[0129]

[0130] Where GRD represents update based on local gradients, D i Use images as training samples.

[0131] Afterwards, according to and The aggregation parameters and the initial matrix are updated alternately until the weight matrix converges.

[0132] It should be noted that after step S322, the weight matrix needs to be pruned to make the resulting weight matrix more conducive to the generation of aggregation parameters. In this embodiment, the aggregation matrix is ​​pruned using σ(w) = max(0, min(1, w)), where... That is, w represents each element of the aggregation matrix, and the value range of w is [0,1].

[0133] It should be noted that, for step S322, when the current iteration training round is equal to the preset round, the aggregation parameters are updated according to the updated weight matrix and the adaptive head aggregation performed in steps S351 to S353, and the weight matrix is ​​updated according to step S322 until the weight matrix converges.

[0134] It should be noted that in step S340, when the current iteration training round is greater than the preset round, steps S351 to S353 and steps S321 to S322 are executed only once, which can improve the training speed and save training time.

[0135] Understandably, referring to Figure 6 , Figure 6 This is a flowchart illustrating the steps of local training for the image segmentation method provided in this application embodiment. Step S400 includes, but is not limited to, the following steps:

[0136] Step S410: Obtain image training samples.

[0137] Step S420: Input the image training samples into the global model to obtain the first site channel attention value and global image features.

[0138] Step S430: Obtain multiple second site channel attention values ​​from the server. The second site channel attention values ​​are generated by the second client.

[0139] Step S440: Based on the first site channel attention value and multiple second site channel attention values, obtain the site embedding difference loss.

[0140] Step S450: Input the global image features into the personalized model to obtain multiple personalized image features.

[0141] Step S460: Based on the image training samples and multiple personalized image features, obtain the weakly supervised segmentation loss.

[0142] Step S470: Train the global model and the personalized model based on the site embedding difference loss and the weakly supervised segmentation loss until the global model and the personalized model satisfy the conditions for local gradient descent.

[0143] Step S480: Based on the global model and the personalized model after training, obtain the image segmentation model.

[0144] It should be noted that the server is also connected to multiple second clients. The first client, multiple second clients, and the server together constitute the federated learning system.

[0145] It should be noted that the attention value of the second site channel is generated by the global model at the second client. Based on the attention value of the first site channel and multiple attention values ​​of the second site channel, a site embedding difference loss is obtained, which can increase the embedding difference between multiple clients, thereby promoting personalized modeling. Furthermore, this application derives a weakly supervised segmentation loss based on image training samples and multiple personalized image features. Using this weakly supervised segmentation loss can further reduce the annotation cost for the client. The overall objective loss function of the image segmentation model, i.e., the global model and the personalized model, can be expressed as:

[0146] L i =L seg +λ1L con

[0147] Among them, L i L is the target loss of the image segmentation model. seg For weakly supervised segmentation loss, L con Let λ be the site embedding difference loss, and λ1 be the trade-off parameter corresponding to the site embedding difference loss.

[0148] It should be noted that the global model and the personalized model satisfy the condition of local gradient descent, that is, the local gradient descent is used to find the local minimum of the target loss of the image segmentation model. When the target loss is minimized, the global model and the personalized model reach their current optimality and their segmentation performance is the best.

[0149] Understandably, referring to Figure 7 , Figure 7 This is a flowchart illustrating the global model processing steps of the image segmentation method provided in this application embodiment. Step S420 includes, but is not limited to, the following steps:

[0150] Step S421: Input the image training samples into the encoder to obtain the encoded image features.

[0151] Step S422: Perform global average pooling on the encoded image features to obtain global average features.

[0152] Step S423: Obtain the site code and pass the site code through two fully connected layers to obtain the site connection feature.

[0153] Step S424: Connect the global average feature and the site connection feature through channels, and activate the result after channel connection to obtain the first site channel attention value.

[0154] Step S425: Multiply the first station channel attention value with the encoded image features one by one, and add the result of the multiplication to the encoded image features to obtain the global image features.

[0155] It should be noted that, referring to Figure 9 The global module includes an encoder and a site contrast representation module. Image training samples are input to the encoder F. e The encoded image features f are obtained. k Then, the encoded image features f k The input is processed by the site comparison and characterization module. Steps S422 to S425 are the processing steps of the site comparison and characterization module. First, the encoded image features f are processed. k Perform global average pooling to encode image features f k With dimensions of C×H×W, global average pooling encodes image features f. k The size is changed to C×1×1, meaning the size of the global average feature is C×1×1. Simultaneously with global average pooling, the site code c is obtained. k The site is encoded using two fully connected layers. k Extend the feature to obtain the site connection characteristics. The site connectivity feature has a size of C×1×1. Then, the global average feature and the site connectivity feature are calculated. Channel connections are performed, resulting in a size of 2C×1×1. The connection result is then activated by passing it through a fully connected layer with an activation function to obtain the first site channel attention value. First Station Passage Attention Values It can be used to calculate the site embedding difference loss, and then apply the residual to the attention value of the first site channel. With encoded image features f k Processing will be carried out, that is, the attention value of the first station channel will be adjusted. With encoded image features f k Perform successive multiplication and then combine the result with the encoded image features f. k By adding them together, we obtain the global image features f. k ', global image features f k It is then sent to the personalization module, namely encoder F. dThe final segmentation is then performed. The site comparison representation module adaptively aggregates the data distribution representations within the client and separates the distribution representations between multiple clients. The data distribution representations between different clients are closely related but distinct from each other, which effectively promotes model personalization.

[0156] It should be noted that the site encoding is one-hot encoding, meaning that the clients in the federated learning system are sorted, and different site codes are assigned to each client based on the sorting result. If the first client's sorting result is k (i.e., the kth client), then the total length of the site codes is the total number of clients in the federated learning system, where the kth position is 1 and the remaining positions are 0. Different clients are sequentially assigned different site codes, resulting in site channel attention values ​​with different styles and distributions. The site embedding difference loss can be obtained based on the first and second site channel attention values, and this loss serves as the basis for the image segmentation model. The site contrast representation module enhances the representation comparison between sites, i.e., between multiple clients, through channel attention.

[0157] Understandably, referring to Figure 8 , Figure 8 This is a diagram illustrating the steps of calculating the site embedding difference loss in the image segmentation method provided in this application embodiment. Step S440 includes, but is not limited to, the following steps:

[0158] Step S441: Calculate the stopping gradient value of the attention value of multiple second site channels.

[0159] Step S442: Calculate the difference between the first station channel attention value and multiple stopping gradient values.

[0160] Step S443: Calculate the sum of multiple differences.

[0161] Step S444: Based on the sum of multiple differences and the number of second clients, obtain the site embedding difference loss for the first client.

[0162] It should be noted that, for the first client, the stopping gradient values ​​of the second site channel attention values ​​corresponding to other clients in the federated learning system, i.e., the second client, are first calculated. Then, the differences between the first site channel attention value and the multiple stopping gradient values ​​are calculated, and these differences are summed. Based on the sum and the number of second clients, the site embedding difference loss of the first client is obtained. The site embedding difference loss can be expressed as:

[0163]

[0164] Among them, L con The site embedding difference loss is used, where K is the number of clients in the federated learning system, i.e., the sum of the number of the first client and the number of the second client. The first station channel attention value, The second site-level channel attention value is represented by `StopGradient()`, and the stopping gradient value is represented by `StopGradient()`. The calculated site embedding difference loss can be used to train the image segmentation model, which can increase the difference contrast between different clients, thereby promoting personalized modeling and improving the segmentation performance of the image segmentation model.

[0165] Understandably, referring to Figure 9 , Figure 9 This is a flowchart illustrating the steps of weakly supervised segmentation loss calculation in the image segmentation method provided in this application embodiment. Step S460 includes, but is not limited to, the following steps:

[0166] Step S462: Determine the prediction probability of the image training samples based on the image training samples and the predicted image features.

[0167] Step S462: Perform sparse processing on the image training samples to obtain the corresponding sparse labels.

[0168] Step S463: Based on the sparse labels and predicted probabilities, obtain the partial cross-entropy loss.

[0169] Step S464: Based on the predicted probability and the preset source mask and preset target mask, obtain the gated conditional random field loss.

[0170] Step S465: Generate tree affinity for the image training samples to obtain a low-order affinity matrix.

[0171] Step S466: Generate tree affinity for the first personalized feature and the second personalized feature to obtain two high-order affinity matrices. The first personalized feature and the second personalized feature are the output results of different network layers of the personalized module.

[0172] Step S467: Based on the predicted probability, perform cascade filtering on the low-order affinity matrix and the two high-order affinity matrices to obtain the pseudo-labels corresponding to the image training samples.

[0173] Step S468: Based on the predicted probability and pseudo-label, obtain the energy loss of the multi-scale recursive tree.

[0174] Step S469: Add the tree energy loss, partial cross-entropy loss and gated conditional random field loss according to the preset trade-off parameters to obtain the weakly supervised segmentation loss.

[0175] It should be noted that, referring to Figure 10 , Figure 10This application primarily showcases the weakly supervised training paradigm employed. This paradigm mainly comprises a segmentation branch for processing labeled pixels and an auxiliary branch for processing unlabeled pixels. The segmentation branch uses the sparse label Y as the supervision signal and employs a partial cross-entropy loss for supervision. For image training samples, this can be represented as the labeled set I. L and unlabeled set I U The partial cross-entropy loss, applied to labeled pixels, can be expressed as:

[0176]

[0177] Among them, Y i and P i This represents the sparse label and predicted probability of pixel i in the annotation set. The auxiliary branch obtains the triple affinity matrix A from the image training sample I and the first personalized feature F1 and the second personalized feature F2. low A high1 and A high2 A low Includes object boundary information, A high1 and A high2 To maintain semantic consistency, the low-order affinity matrix and the two high-order affinity matrices are then cascaded and filtered according to the predicted probabilities to obtain pseudo-labels corresponding to the image training samples.

[0178]

[0179] Where F() represents cascaded filtering, and then based on the predicted probability and pseudo-labels, the multi-scale recursive tree energy loss is obtained, which can be expressed as:

[0180]

[0181] in, and P i This represents the pseudo-label and predicted probability of pixel i in the unlabeled set. Furthermore, based on the predicted probability and preset source and target masks, a gated conditional random field loss is obtained. This loss is used to suppress the excessive expansion or collapse of pseudo-labels in the target region that might occur when using only tree energy loss, thereby further improving edge accuracy and reducing P and Outliers are identified. Then, based on preset tradeoff parameters, the tree energy loss, partial cross-entropy loss, and gated conditional random field loss are added together to obtain the weakly supervised segmentation loss, the formula of which is:

[0182] L seg =L pCE +λ2L MsTree +λ3L gCRF

[0183] Among them, L seg For weakly supervised segmentation loss, L pCE For partial cross-entropy loss, L MsTree For the energy loss of multi-scale recursive trees, L gCRF λ is the gated conditional random field loss, and λ² is the multi-scale recursion tree energy loss L. MsTree The corresponding trade-off parameter, λ3, is the gated conditional random field loss L. gCRF The corresponding trade-off parameters.

[0184] It should be noted that in federated learning, a more practical setting would be to allow different central sites to use heterogeneous sparse labels and train them in a unified and compatible manner, so that each site can benefit from the federation.

[0185] It should be noted that the selected image features are the image features of the sample image data after passing through the preset layers of the segmentation model. In this embodiment, U-Net is used as the network architecture for image segmentation. The first personalized feature is the output image feature of the second layer of the U-Net decoder, and the second personalized feature is the output image feature of the third layer of the U-Net decoder. By using the image features output by the multi-layer network of the decoder to generate pseudo-labels, the obtained pseudo-labels are more accurate and of higher quality, resulting in better training effects of the multi-scale recursive tree energy loss and weakly supervised segmentation loss on the model. In addition, the image segmentation model can also adopt other network architectures, and the corresponding preset layers will also change with the network architecture.

[0186] It should be noted that image segmentation is essentially a pixel-level classification task. The most classic weakly supervised segmentation method modifies the traditional cross-entropy function to a partial cross-entropy function, that is, it calculates the cross-entropy loss only on pixels in the labeled region. However, applying partial cross-entropy loss only to sparsely labeled regions often fails to provide sufficient supervision signals near semantic boundaries, thus leading to poor prediction results in unsupervised areas. The weak supervision provided in this application extends outward from the supervised region in a specific way, forming pseudo-labels for unlabeled regions, thereby providing more sufficient supervision signals for model training.

[0187] It should be noted that, referring to Figure 12 UA, OC, OD, FAZ, and BG represent the unlabeled region, optic cup, optic disc, avascular region of the fovea, and background, respectively. Point, graffiti, and block annotations can all serve as pixel-level sparse class supervision signals. However, bounding boxes, as rectangular boxes describing the boundaries of the target, require certain preprocessing transformations before they can be used as class supervision signals. Based on prior knowledge such as the shape and size of the segmented target, it can be converted into points, graffiti, blocks, or a mixture of the three.

[0188] It should be noted that, referring to Figure 13 In the segmentation task of the optic cup and optic disc in the fundus, for the two elliptical structures of the optic cup and optic disc, the Hough circle fitting transform is first used to fit the inscribed ellipse of the bounding box of the two target regions. Specifically, the largest ellipse in the rectangle is first obtained, and a small circular region in the ellipse is removed to obtain the inscribed ellipse. Then, the skeleton of the inner ellipse and the annular region between the two ellipses is skeletonized to obtain sparse labels for the skeleton of the two target regions. At the same time, the same operation is performed on the background part outside the optic disc bounding box to obtain sparse background labels, thus converting the bounding box annotation into a graffiti-like annotation. For the segmentation task of the avascular region of the fovea of ​​the retina, due to its irregular shape, the four sides of the bounding box are expanded outward by a certain number of pixels. The outer pixels of the expanded rectangle are set as the background class. At the same time, the length and width of the bounding box are reduced by 80% respectively to obtain a reduced box. The pixels within the range of this box are set as the target class, thus converting the bounding box into block annotation. For other complex structures, similar preprocessing based on priors such as shape and size can be performed to convert the box annotation into sparse pixel annotation.

[0189] It should be noted that, referring to Figure 10The network architecture of the image segmentation method provided in this application includes a server and multiple clients. For ease of description, the multiple clients are divided into a first client and multiple second clients. For the first client, it first obtains local parameters and global parameters and personalized parameters from the server. The local parameters are the parameters of the local personalized model. The global model is initialized based on the global parameters. When the current training iteration is 1, the local parameters are used as aggregation parameters. When the current training iteration is 2, the weight matrix is ​​obtained, and multiple adaptive head aggregations are performed on the weight matrix, personalized parameters, and local parameters to obtain aggregation parameters. When the current training iteration is greater than 2, the weight matrix is ​​obtained, and a single adaptive head aggregation is performed on the weight matrix, personalized parameters, and local parameters to generate aggregation parameters. Then, the personalized model is initialized using the aggregation parameters. The initialized global model and personalized model are trained locally to obtain an image segmentation model, and the image segmentation model is uploaded to the server so that the server can update the global parameters and personalized parameters. The global model comprises an encoder and a site contrast representation module. Image training samples are input into the global model to obtain first site channel attention values ​​and global image features. Based on the first site channel attention values ​​and multiple second site channel attention values, a site embedding difference loss is obtained. The second site channel attention values ​​are generated by a second client. The global image features are input into the personalized model to obtain multiple personalized image features. Based on the image training samples and multiple personalized image features, a weakly supervised segmentation loss is obtained. The global model and personalized models are trained using the site embedding difference loss and the weakly supervised segmentation loss until they satisfy the condition for local gradient descent. Based on the trained global and personalized models, an image segmentation model is obtained. Then, the image segmentation model is iteratively trained using the updated global and personalized parameters until it converges. Finally, the image to be segmented is acquired and processed by the trained image segmentation model to obtain the segmentation result.

[0190] It should be noted that during the training process of the image segmentation model, this embodiment adaptively aggregates the data distribution representation within the client and separates the distribution representation between multiple clients through the site comparison representation module. The data distribution representations between different clients are closely related and different from each other, which can effectively promote model personalization. In addition, by updating and initializing the personalized model at the element level through adaptive head aggregation, the common knowledge between multiple client models and the unique knowledge of the local client model can be effectively integrated, which can effectively promote personalized modeling and also alleviate over-personalization. Furthermore, this application uses site embedding difference loss and weakly supervised segmentation loss as the loss function of the image segmentation model, which can generate high-quality pseudo-labels for better training, thereby obtaining excellent image segmentation performance and improving the performance of the image segmentation models of each client, thus improving the image segmentation effect.

[0191] It should be noted that, referring to Figure 12 Domain bias exists between data samples from different clients, i.e., sites A to E, resulting in statistical heterogeneity. This is primarily caused by non-independent and identically distributed data, making it difficult for a single model to perform well across all clients. In medical image analysis scenarios, federated learning degradation can be even more severe due to the potential for more diverse image domain biases caused by differences in imaging equipment, protocols, patient populations, and physician expertise. To address this, this application experimentally verifies the beneficial effects of its image segmentation method using the following method. Extensive experiments were conducted to verify the effectiveness of the proposed image segmentation method on two medical image segmentation datasets: optic disc / cup (ODOC) segmentation from fundus images and foveal avascular zone (FAZ) segmentation from optical coherence tomography (OCTA) images. This application sets up five clients, with the five client datasets containing {101, 159, 400, 400, 200} samples respectively for fundus images. For optical coherence tomography (OCT) angiography images, the datasets for the five clients contain {304, 200, 300, 1012, and 39} samples, respectively. Fundus images are center-cropped and then resized to 384×384, while angiography images are directly resized to 256×256. This application utilizes automated algorithms to simulate points, graffiti, bounding boxes, and original segmentation masks. The image segmentation model in this application uses U-Net as the basic architecture of the image segmentation network, and adds a site contrast representation module between the encoder and decoder of U-Net. The number of channels in U-Net gradually increases from 16 to 256 from top to bottom, and an initial learning rate of 1×10⁻⁶ is used. -2The AdamW optimizer is used to optimize parameters, and a polynomial strategy with power = 0.9 is used to dynamically adjust the learning rate. The parameters λ1, λ2, λ3 and batch size are set to 1, 0.1, 0.1 and 12 respectively, and 500 rounds of federated training are used for all federated learning methods to ensure fair performance comparison. This application compares the performance of the proposed image segmentation method with several representative federated learning frameworks, including traditional centralized federated methods (such as FedAvg and FedProx) and state-of-the-art personalized federated methods (such as FT, FedBN, FedAP, FedRep, etc.). In terms of implementation, since these methods were originally designed for image classification tasks, this application tries to maintain their design principles and adapt them to weakly supervised image segmentation tasks. In addition, this application also performs fully supervised local training, weakly supervised centralized training, and fully supervised centralized training. Centralized training refers to merging and directly training data from various sites on a central server without considering data privacy and security. This application uses two metrics for evaluation: set similarity measure (Dice coefficient) and Hausdorff distance.

[0192] It should be noted that Table 1 shows the Dice coefficients of various methods at different sites for the optic disc / optic cup segmentation task. Compared with the local training method, all federated learning methods improved the overall segmentation performance of all clients. Clients C and D achieved significant performance improvements after joining federated learning because the local data distributions of clients C and D are diverse, making it difficult to train sufficiently powerful models using only local data. Due to the high heterogeneity of cross-site data, personalized federated learning methods benefit from training personalized models for different sites and generally achieve higher performance than centralized federated learning methods. FedAP utilizes batch normalization layers to measure the similarity of data distributions between sites, aggregating personalized models to extract useful information from other sites, thus achieving good performance. Notably, the image segmentation method in this embodiment achieves further performance improvements, with Dice coefficients approximately 2% higher than FedAP, approaching that of centralized training with fully supervised labels. Table 2 shows the Dice coefficients and Hausdorff distances of various methods at different sites for the segmentation task of the fovea avascular region from optical coherence tomography angiography images. Unlike image disc / cup segmentation tasks, most personalized federated learning methods suffer from confusion or over-personalization due to data consistency, performing even worse than traditional centralized federated methods. The image segmentation method provided in this application utilizes a site-specific contrastive representation module to enhance the contrast of data representations across clients, thereby mitigating these problems. Therefore, it is still able to train a high-performance segmentation model for each client and achieve optimal performance across all clients, significantly outperforming traditional centralized federated methods and approaching the performance of centralized training with weakly supervised labels.

[0193] Table 1

[0194]

[0195]

[0196] Table 2

[0197]

[0198] It should be noted that Table 3 shows the results of ablation experiments on the image segmentation method provided in this application embodiment. This application embodiment conducted ablation experiments on datasets for optic cup / optic disc segmentation and foveal avascular region segmentation, using the Dice coefficient as the evaluation criterion. As can be seen from Table 3, eliminating any one of the following—adaptive head aggregation, site comparison representation module, tree energy loss, and gated conditional random field loss—leads to a certain degree of performance degradation in segmentation.

[0199] Table 3

[0200] Adaptive Head Aggregation Site Comparison and Characterization Module Tree energy loss Gating loss Fundus cup fundus disc avascular area of ​​the fovea of ​​the retina - - - - 90.68 81.60 84.05 √ 92.98 84.67 88.96 √ 93.19 84.07 88.27 √ √ 93.65 85.10 89.20 √ √ √ 93.86 85.57 89.35 √ √ √ 94.63 85.82 92.47 √ √ √ √ 94.63 86.64 93.29

[0201] Additionally, refer to Figure 14 , Figure 14 The image segmentation device provided in this application embodiment is shown in the structural block diagram. The image segmentation device provided in this application embodiment includes a data communication module 100, a global initialization module 200, a personalized initialization module 300, a local training module 400, an iterative training module 500, and an image segmentation module 600.

[0202] The data communication module 100 is communicatively connected to the global initialization module 200. The data communication module 100 is used to obtain local parameters and global parameters and personalized parameters from the server.

[0203] The global initialization module 200 is connected to the data communication module 100 and the individual initialization module 300 respectively. The global initialization module 200 is used to initialize the global model according to the global parameters.

[0204] The personalized initialization module 300 is connected to the global initialization module 200 and the local training module 400. The personalized initialization module 300 is used to determine whether to perform adaptive head aggregation on the local parameters and the personalized parameters based on the comparison results between the current iteration training round and the preset round, so as to generate aggregated parameters and initialize the personalized model through the aggregated parameters.

[0205] The local training module 400 is connected to the personalized initialization module 300 and the iterative training module 500. The local training module 400 is used to train the initialized global model and the personalized model locally to obtain an image segmentation model, and upload the image segmentation model to the server so that the server can update the global parameters and the personalized parameters.

[0206] The iterative training module 500 is communicatively connected to the local training module 400 and the image segmentation module 600. The iterative training module 500 is used to perform a new round of iterative training on the image segmentation model based on the updated global parameters and personalized parameters, until the image segmentation model obtained from the last round of iterative training reaches the convergence state.

[0207] The image segmentation module 600 is communicatively connected to the iterative training module 500. The image segmentation module 600 is used to input the image to be segmented into the image segmentation model that has reached a convergence state, and obtain the segmentation result of the image to be segmented.

[0208] It should be noted that, during the training process of the image segmentation model, the personalized initialization module 300 of this application update and initializes the personalized model at the element level through adaptive head aggregation. This can effectively integrate the common knowledge between multiple client models and the unique knowledge of the local client model, effectively promoting personalized modeling while alleviating over-personalization, thereby obtaining excellent image segmentation performance and improving the performance of the image segmentation models of each client, thus improving the image segmentation effect.

[0209] Additionally, refer to Figure 15 This application provides a schematic diagram of a system architecture platform for image segmentation.

[0210] The system architecture platform of this application embodiment includes one or more processors and memory. Figure 15 The example consists of a processor and a memory.

[0211] The processor and memory can be connected via a bus or other means. Figure 15 Taking the example of a connection between China and Israel via a bus.

[0212] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the system architecture platform via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0213] Those skilled in the art will understand that this system architecture platform can be applied to existing communication network systems and subsequent evolved mobile communication network systems, and this embodiment does not specifically limit it.

[0214] Those skilled in the art will understand that Figure 15 The device structure shown does not constitute a limitation on the system architecture platform and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0215] The system architecture platform can be an independent system architecture platform, or it can be a cloud system architecture platform that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms.

[0216] In addition, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program as in the image segmentation method of steps S100 to S600.

[0217] The processor and memory can be connected via a bus or other means.

[0218] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0219] The non-transient software program and instructions required to implement the target tracking method of the above embodiments are stored in memory. When executed by a processor, the image segmentation method described above is executed, for example, the method described above. Figure 2 Method steps S100 to S600.

[0220] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0221] Furthermore, one embodiment of this application also provides a computer-readable storage medium storing computer-executable instructions. These instructions, executed by a processor or controller, cause the processor to perform the image segmentation method described above, for example, the method described above. Figure 2 Method steps S100 to S600 in the text Figure 3 Method steps S310 to S340 in the text Figure 4 Method steps S351 to S353 in the text Figure 5 Method steps S321 and S323 in the text Figure 6 Method steps S410 and S480, Figure 7 Method steps S421 to S425 in the text Figure 8 Method steps S441 to S444 in the text Figure 9 Method steps S461 to S469.

[0222] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0223] The above is a detailed description of the preferred embodiments of this application. However, the embodiments of this application are not limited to the above-described implementation methods. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the embodiments of this application. All such equivalent modifications or substitutions are included within the scope defined by the claims of the embodiments of this application.

Claims

1. An image segmentation method characterized by, The image segmentation method is applied to a first client connected with a server, and comprises the following steps: obtaining local parameters and global parameters and personalized parameters from the server; initializing a global model according to the global parameters; determining whether to perform adaptive head aggregation on the local parameters and the personalized parameters to generate aggregated parameters according to a comparison result of a current iteration training round and a preset round, and initializing a personalized model by using the aggregated parameters; performing local training on the initialized global model and the personalized model to obtain an image segmentation model, and uploading the image segmentation model to the server to enable the server to update the global parameters and the personalized parameters; performing a new round of iteration training on the image segmentation model according to the updated global parameters and the personalized parameters until the image segmentation model obtained in the last iteration training reaches a convergence state; inputting a to-be-segmented image into the image segmentation model reaching the convergence state to obtain a segmentation result of the to-be-segmented image; wherein the determining whether to perform adaptive head aggregation on the local parameters and the personalized parameters to generate aggregated parameters according to a comparison result of a current iteration training round and a preset round comprises the following steps: obtaining the current iteration training round; when the current iteration training round is less than the preset round, taking the local parameters as the aggregated parameters; when the current iteration training round is equal to the preset round, obtaining a weight matrix, performing multiple adaptive head aggregations on the weight matrix, the personalized parameters and the local parameters to obtain the aggregated parameters; when the current iteration training round is greater than the preset round, obtaining the weight matrix, performing single adaptive head aggregation on the weight matrix, the personalized parameters and the local parameters to obtain the aggregated parameters.

2. The image segmentation method of claim 1, wherein, After each adaptive head aggregation, the weight matrix is updated, and the adaptive head aggregation comprises the following steps: calculating a difference between the personalized parameters and the local parameters; calculating a Hadamard product of the difference between the personalized parameters and the local parameters and the weight matrix; adding the Hadamard product and the local parameters to obtain the aggregated parameters.

3. The image segmentation method of claim 2, wherein, The updating of the weight matrix comprises the following steps: taking the weight matrix as an initial matrix; obtaining image training samples and processing the aggregated parameters, the initial matrix and the image training samples in a gradient descent manner to obtain an updated weight matrix; when the current iteration training round is equal to the preset round, updating the aggregated parameters according to the updated weight matrix, the personalized parameters and the local parameters, and updating the weight matrix according to the updated aggregated parameters, the initial matrix and the image training samples until the weight matrix converges.

4. The image segmentation method of claim 1, wherein, The server is also connected with a plurality of second clients, and the local training on the initialized global model and the personalized model to obtain an image segmentation model comprises the following steps: obtaining image training samples; inputting the image training samples into the global model to obtain first site channel attention values and global image features; obtaining a plurality of second site channel attention values from the server, the second site channel attention values being generated by the second client; obtaining a site embedding difference loss according to the first site channel attention value and the plurality of second site channel attention values; inputting the global image feature into the personalized model to obtain a plurality of personalized image features; obtaining a weakly supervised segmentation loss according to the image training sample and the plurality of personalized image features; training the global model and the personalized model according to the site embedding difference loss and the weakly supervised segmentation loss until the global model and the personalized model satisfy a local gradient descent condition; obtaining an image segmentation model according to the global model and the personalized model after the training is completed.

5. The image segmentation method of claim 4, wherein, The global model comprises an encoder and a site contrast representation module, and the inputting of the image training sample into the global model to obtain the first site channel attention value and the global image feature comprises: inputting the image training sample into the encoder to obtain an encoded image feature; performing global average pooling on the encoded image feature to obtain a global average feature; obtaining a site code and obtaining a site connection feature through two fully connected layers; performing channel connection on the global average feature and the site connection feature, and performing activation on the result after the channel connection to obtain the first site channel attention value; performing element-by-element multiplication on the first site channel attention value and the encoded image feature, and adding the multiplication result to the encoded image feature to obtain the global image feature.

6. The image segmentation method of claim 4, wherein, The obtaining of the site embedding difference loss according to the first site channel attention value and the plurality of second site channel attention values comprises: calculating a stop gradient value of the plurality of second site channel attention values; calculating a difference value between the first site channel attention value and the plurality of stop gradient values; calculating a sum of the plurality of difference values; obtaining a site embedding difference loss of the first client according to the sum of the plurality of difference values and the number of the second clients.

7. The image segmentation method of claim 4, wherein, The plurality of personalized image features comprises a first personalized feature, a second personalized feature and a predicted image feature, and the obtaining of the weakly supervised segmentation loss according to the image training sample and the plurality of personalized image features comprises: determining a prediction probability of the image training sample according to the image training sample and the predicted image feature; performing sparse processing on the image training sample to obtain a corresponding sparse label; obtaining a partial cross-entropy loss according to the sparse label and the prediction probability; obtaining a gated conditional random field loss according to the prediction probability, a preset source mask and a preset target mask; generating a tree affinity for the image training sample to obtain a low-order affinity matrix; generating a tree affinity for the first personalized feature and the second personalized feature to obtain two high-order affinity matrices, the first personalized feature and the second personalized feature being output results of different network layers of the personalized model; Cascade filtering is sequentially performed on the low-order affinity matrix, the two high-order affinity matrices according to the predicted probability, to obtain pseudo-labels corresponding to the image training sample; According to the predicted probability and the pseudo-labels, a multi-scale recursive tree energy loss is obtained. According to a preset trade-off parameter, the tree energy loss, the partial cross-entropy loss and the GCRF loss are added to obtain a weakly supervised segmentation loss.

8. An image segmentation apparatus characterized by comprising: The image segmentation device is applied to a first client connected with a server, and comprises: A data communication module is configured to acquire local parameters and global parameters and personalized parameters from the server; A global initialization module is configured to initialize a global model according to the global parameters; A personalized initialization module is configured to compare a current iteration training round with a preset round, determine whether to perform adaptive head aggregation on the local parameters and the personalized parameters to generate aggregated parameters, and initialize a personalized model by using the aggregated parameters; A local training module is configured to locally train the initialized global model and the personalized model to obtain an image segmentation model, and upload the image segmentation model to the server to update the global parameters and the personalized parameters; An iteration training module is configured to perform a new round of iteration training on the image segmentation model according to the updated global parameters and personalized parameters until the image segmentation model obtained after the last iteration training reaches a convergence state; An image segmentation module is configured to input an image to be segmented into the image segmentation model that reaches the convergence state to obtain a segmentation result of the image to be segmented. The comparison of the current iteration training round with the preset round includes: acquiring the current iteration training round; when the current iteration training round is less than the preset round, taking the local parameters as the aggregated parameters; when the current iteration training round is equal to the preset round, acquiring a weight matrix, and performing multiple adaptive head aggregations on the weight matrix, the personalized parameters and the local parameters to obtain the aggregated parameters; when the current iteration training round is greater than the preset round, acquiring the weight matrix, and performing single adaptive head aggregation on the weight matrix, the personalized parameters and the local parameters to obtain the aggregated parameters.

9. An electronic device, comprising: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the image segmentation method of any one of claims 1 to 7 when executing the computer program.

10. A computer readable storage medium, the storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the image segmentation method of any one of claims 1 to 7.

Images