Medical image segmentation method and device, electronic equipment and storage medium

Abstract

The application provides a medical image segmentation method and device, electronic equipment and storage medium, and belongs to the technical field of medical image processing. The annotation information is obtained by fusing general knowledge prompts, data distribution prompts and annotation sparsity prompts. The aggregated global parameter distributed by the server is received. The aggregated global parameter is aggregated by the server based on the model parameters of the medical image segmentation model sent by multiple clients. The model parameters of the medical image segmentation model are initialized according to the aggregated global parameter. The sample medical image and the annotation information are input into the medical image segmentation model for image segmentation. The predicted segmentation label is obtained. The medical image segmentation model is adjusted according to the predicted segmentation label and the sample segmentation label. The target medical image segmentation model is obtained. The target medical image is obtained. The medical image segmentation of the target medical image is performed by the target medical image segmentation model. The image segmentation difficulty can be reduced while ensuring the accuracy of the medical image segmentation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image processing, and particularly relates to a medical image segmentation method and device, an electronic device and a storage medium. BACKGROUND

[0002] Medical image segmentation is to segment a region of interest, such as a physiological structure or a pathological region, from a medical image, so as to facilitate subsequent computer-aided diagnosis. In the related art, a deep learning technology is used for medical image segmentation. However, the deep learning method needs a large-scale fully annotated data set to train a medical image segmentation model, so as to realize accurate segmentation of a region of interest. The full annotation means that the annotation information of the image needs to be sufficient, for example, needs to include a mask of the image, a bounding box of the image, and a type of a pathological region of the image (such as a lung, a liver, etc.). On the one hand, the sharing of medical data is limited based on data security considerations. On the other hand, it is both expensive and time-consuming to fully annotate a large amount of medical data. The above factors increase the difficulty of obtaining a large-scale fully annotated data set, resulting in great difficulty in medical image segmentation. In addition, due to the difficulty in obtaining a large-scale fully annotated data set, the training effect of the medical image segmentation model is not good, which affects the accuracy of medical image segmentation. SUMMARY

[0003] The main purpose of the embodiments of the present application is to propose a medical image segmentation method, a medical image segmentation device, an electronic device and a storage medium, which aims to ensure the accuracy of medical image segmentation while reducing the difficulty of image segmentation.

[0004] To achieve the above-mentioned purpose, a first aspect of the embodiments of the present application proposes a medical image segmentation method applied to a client, comprising:

[0005] Obtaining a sample medical image set, the sample medical image set comprising a sample medical image, the sample medical image having a sample segmentation label, the sample segmentation label indicating the position of a region of interest of the sample medical image;

[0006] Based on the pixel value of each sample medical image in the sample medical image set, obtaining a data distribution prompt;

[0007] Obtaining the sample segmentation label and determining an annotation sparsity prompt; the annotation sparsity prompt is used to represent the annotation form of the sample segmentation label;

[0008] Obtaining a general knowledge prompt;

[0009] Fusing the general knowledge prompt, the data distribution prompt and the annotation sparsity prompt to obtain annotation information;

[0010] receive an aggregated global parameter assigned by a server end; the aggregated global parameter is aggregated by the server end based on model parameters of a medical image segmentation model sent by multiple clients;

[0011] initialize model parameters of the medical image segmentation model according to the aggregated global parameter;

[0012] input the sample medical image and the annotation information into the medical image segmentation model for image segmentation to obtain a predicted segmentation label;

[0013] adjust the medical image segmentation model according to the predicted segmentation label and the sample segmentation label to obtain a target medical image segmentation model;

[0014] obtain a target medical image, and perform medical image segmentation on the target medical image by using the target medical image segmentation model.

[0015] In some embodiments, the predicted segmentation label includes a first predicted segmentation sub-label and a second predicted segmentation sub-label, and the medical image segmentation model includes a global shared module, a local main decoder, and a local auxiliary decoder.

[0016] The inputting of the sample medical image and the annotation information into the medical image segmentation model for image segmentation to obtain a predicted segmentation label includes:

[0017] extracting context features of the sample medical image and the annotation information by using the global shared module to obtain context injection features;

[0018] performing image segmentation on the context injection features by using the local main decoder to obtain the first predicted segmentation sub-label;

[0019] performing image segmentation on the context injection features by using the local auxiliary decoder to obtain the second predicted segmentation sub-label;

[0020] The adjusting of the medical image segmentation model according to the predicted segmentation label and the sample segmentation label to obtain a target medical image segmentation model includes:

[0021] generating a pseudo predicted label according to the first predicted segmentation sub-label and the second predicted segmentation sub-label;

[0022] adjusting the medical image segmentation model according to the pseudo predicted label, the first predicted segmentation sub-label, the second predicted segmentation sub-label, and the sample segmentation label to obtain the target medical image segmentation model.

[0023] In some embodiments, the adjusting the medical image segmentation model according to the pseudo prediction label, the first prediction segmentation sub-label, the second prediction segmentation sub-label and the sample segmentation label to obtain the target medical image segmentation model comprises:

[0024] performing first loss calculation according to the first prediction segmentation sub-label, the second prediction segmentation sub-label and the sample segmentation label to obtain first loss data;

[0025] performing second loss calculation according to the pseudo prediction label, the first prediction segmentation sub-label and the second prediction segmentation sub-label to obtain second loss data;

[0026] adjusting the medical image segmentation model according to the first loss data and the second loss data to obtain the target medical image segmentation model.

[0027] In some embodiments, the performing second loss calculation according to the pseudo prediction label, the first prediction segmentation sub-label and the second prediction segmentation sub-label to obtain second loss data comprises:

[0028] performing first sub-loss calculation according to the pseudo prediction label and the first prediction segmentation sub-label to obtain first sub-loss data;

[0029] performing second sub-loss calculation according to the pseudo prediction label and the second prediction segmentation sub-label to obtain second sub-loss data;

[0030] performing loss fusion according to the first sub-loss data and the second sub-loss data to obtain the second loss data.

[0031] In some embodiments, the global shared module comprises an encoder and a dual attention fusion module, and the performing context feature extraction on the sample medical image and the annotation information by the global shared module to obtain context injection features comprises:

[0032] performing image encoding on the sample medical image by the encoder to obtain encoded features;

[0033] performing dual attention fusion feature extraction on the encoded features and the annotation information by the dual attention fusion module to obtain the context injection features.

[0034] To achieve the above object, a second aspect of the embodiments of the present application proposes a medical image segmentation method applied to a server side, the method comprising:

[0035] receive target model parameters of a medical image segmentation model sent by a plurality of clients; the medical image segmentation model comprises a global shared module, a local main decoder and a local auxiliary decoder, and the target model parameters comprise first global shared parameters of the global shared module and local main decoder parameters of the local main decoder;

[0036] aggregate the first global shared parameters of the plurality of clients to obtain second global shared parameters;

[0037] aggregate the local main decoder parameters of the plurality of clients to obtain global main decoder parameters;

[0038] determine global auxiliary decoder parameters based on a preset parameter selection strategy and the local main decoder parameters of the plurality of clients;

[0039] send the second global shared parameters, the global main decoder parameters and the global auxiliary decoder parameters as aggregated global parameters to the clients, so that the clients initialize model parameters of a medical image segmentation model according to the aggregated global parameters to perform medical image segmentation.

[0040] In some embodiments, the preset parameter selection strategy is a prompt similarity aggregation strategy, and the determination of the global auxiliary decoder parameters based on the preset parameter selection strategy and the local main decoder parameters of the plurality of clients comprises:

[0041] receive annotation information sent by a client; the annotation information comprises a general knowledge prompt and a data distribution prompt;

[0042] calculate similarity data between the clients according to the general knowledge prompt and the data distribution prompt;

[0043] aggregate the local main decoder parameters of the plurality of clients according to the similarity data to obtain the global auxiliary decoder parameters.

[0044] To achieve the above-mentioned purpose, a third aspect of the embodiments of the present application proposes a medical image segmentation device, applied to a client, the device comprising:

[0045] an image acquisition module, configured to acquire a sample medical image set, the sample medical image set comprising a sample medical image, the sample medical image having a sample segmentation label, the sample segmentation label indicating the position of a region of interest of the sample medical image;

[0046] a data distribution prompt acquisition module, configured to obtain a data distribution prompt based on the pixel value of each sample medical image in the sample medical image set;

[0047] The annotation sparse prompt acquisition module is configured to acquire the sample segmentation label and determine an annotation sparse prompt, wherein the annotation sparse prompt is used to represent a labeling form of the sample segmentation label.

[0048] The general knowledge prompt acquisition module is configured to acquire a general knowledge prompt.

[0049] The prompt fusion module is configured to fuse the general knowledge prompt, the data distribution prompt and the annotation sparse prompt to obtain annotation information.

[0050] The receiving module is configured to receive an aggregated global parameter distributed by a server end, wherein the aggregated global parameter is aggregated by the server end based on model parameters of a medical image segmentation model sent by multiple client ends.

[0051] The initialization module is configured to initialize model parameters of the medical image segmentation model according to the aggregated global parameter.

[0052] The input module is configured to input the sample medical image and the annotation information into the medical image segmentation model for image segmentation to obtain a predicted segmentation label.

[0053] The adjustment module is configured to adjust the medical image segmentation model according to the predicted segmentation label and the sample segmentation label to obtain a target medical image segmentation model.

[0054] The medical image segmentation module is configured to acquire a target medical image and perform medical image segmentation on the target medical image by using the target medical image segmentation model.

[0055] To achieve the above object, a fourth aspect of the embodiments of the present application provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the medical image segmentation method of the first aspect or the medical image segmentation method of the second aspect when executing the computer program.

[0056] To achieve the above object, a fifth aspect of the embodiments of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the medical image segmentation method of the first aspect or the medical image segmentation method of the second aspect.

[0057] The medical image segmentation method, the medical image segmentation device, the electronic equipment and the computer readable storage medium provided by the present application are characterized in that: a sample medical image set is obtained, the sample medical image set includes a sample medical image, the sample medical image has a sample segmentation label, the sample segmentation label indicates the position of the region of interest of the sample medical image, and the training of the medical image segmentation model is performed based on the sample medical image and the sample segmentation label. Based on the pixel value of each sample medical image in the sample medical image set, a data distribution prompt is obtained to determine the pixel distribution of each sample medical image in the sample medical image set. The sample segmentation label is obtained, and an annotation sparsity prompt is determined. The annotation sparsity prompt is used to represent the annotation form of the sample segmentation label, so that the medical image segmentation model can be trained using sample medical images with different annotation forms, and the dependence on completely annotated sample medical images in the model training process is reduced. A general knowledge prompt is obtained to guide the model to perform image segmentation. The general knowledge prompt, the data distribution prompt and the annotation sparsity prompt are fused to obtain annotation information. The training of the model is guided by introducing additional annotation information, so that the accuracy of image segmentation is ensured without the need for a large-scale completely annotated data set. If only local sample medical images and sample segmentation labels are used for model training, the generalization of the model will be reduced, and the accuracy of medical image segmentation will be reduced. In order to further improve the accuracy of medical image segmentation, the aggregated global parameters distributed by the server end are received, the aggregated global parameters are aggregated by the server end based on the model parameters of the medical image segmentation model sent by multiple clients, and the model parameters of the medical image segmentation model are initialized according to the aggregated global parameters, so that the local model can be trained without obtaining the data set of other clients, the safety of the data is ensured, the difficulty of obtaining a large-scale completely annotated data set is reduced, and the difficulty of medical image segmentation is reduced. The sample medical image and the annotation information are input into the medical image segmentation model for image segmentation to obtain a predicted segmentation label, the medical image segmentation model is adjusted according to the predicted segmentation label and the sample segmentation label, the optimal image segmentation model is determined, and a target medical image segmentation model is obtained. The target medical image is obtained, and the target medical image is segmented by the target medical image segmentation model, so that accurate segmentation of the target medical image is realized without the need for a large-scale completely annotated data set, and the difficulty of medical image segmentation is reduced. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 is a flowchart of the medical image segmentation method provided by the embodiment of the present application;

[0059] Figure 2a is an eye fundus image provided by the embodiment of the present application;

[0060] Figure 2b is an endoscope image provided by the embodiment of the present application;

[0061] Figure 3a This is a data distribution prompt provided in the embodiments of this application;

[0062] Figure 3b This is another data distribution prompt provided in the embodiments of this application;

[0063] Figure 3c This is another data distribution prompt provided in the embodiments of this application;

[0064] Figure 3d This is the correlation coefficient matrix provided in the embodiments of this application;

[0065] Figure 3e This is another correlation coefficient matrix provided in the embodiments of this application;

[0066] Figure 3f This is another correlation coefficient matrix provided in the embodiments of this application;

[0067] Figure 4 This is a schematic diagram of the sample annotation format provided in the embodiments of this application;

[0068] Figure 5 This is a schematic diagram of the network structure of a local model provided in an embodiment of this application;

[0069] Figure 6 yes Figure 1 The flowchart for step 108 in the document;

[0070] Figure 7 yes Figure 6 The flowchart for step 610 in the middle;

[0071] Figure 8 This is a schematic diagram of the three-prompt dual-attention fusion module provided in an embodiment of this application;

[0072] Figure 9 yes Figure 6 The flowchart for step 650 in the document;

[0073] Figure 10 yes Figure 9 The flowchart for step 920 in the middle;

[0074] Figure 11 This is another flowchart of the medical image segmentation method provided in the embodiments of this application;

[0075] Figure 12 yes Figure 11 The flowchart for step 1140 in the document;

[0076] Figure 13 This is a segmentation result diagram of the medical image segmentation method provided in the embodiments of this application;

[0077] Figure 14 is a structural schematic diagram of a medical image segmentation device provided by an embodiment of the present application;

[0078] Figure 15 is a hardware structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0079] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0080] It should be noted that although the functional modules are divided in the device schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a manner different from the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the description, claims and the above drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0081] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0082] Medical image segmentation is to segment the region of interest, such as physiological structure or pathological region, from the medical image, so as to facilitate subsequent computer-aided diagnosis. In the related art, deep learning technology is used for medical image segmentation. However, the deep learning method needs a large-scale fully annotated data set to train the medical image segmentation model, so as to realize accurate segmentation of the region of interest. Among them, full annotation means that the annotation information of the image should be sufficient, such as needing to include the mask of the image, the bounding box of the image, and the type of the pathological region of the image (such as lung, liver, etc.). On the one hand, based on the consideration of data security, the sharing of medical data is limited. On the other hand, fully annotating a large amount of medical data is not only expensive but also time-consuming. The above factors make it difficult to obtain a large-scale fully annotated data set, resulting in a greater difficulty in medical image segmentation. In addition, due to the difficulty in obtaining a large-scale fully annotated data set, the training effect of the medical image segmentation model is not good, which affects the accuracy of medical image segmentation.

[0083] Based on this, the embodiments of the present application provide a medical image segmentation method, a medical image segmentation device, an electronic device and a computer readable storage medium, which aims to ensure the accuracy of medical image segmentation while reducing the difficulty of image segmentation.

[0084] The medical image segmentation method, the medical image segmentation device, the electronic device, and the computer readable storage medium provided in the embodiments of the present application are described in detail through the following embodiments. First, the medical image segmentation method in the embodiments of the present application is described.

[0085] The medical image segmentation method provided in the embodiments of the present application relates to the technical field of medical image processing. The medical image segmentation method provided in the embodiments of the present application can be applied to a terminal, can be applied to a server end, and can also be software running in the terminal or the server end. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, or the like; the server end can be configured as a stand-alone physical server, can be configured as a server cluster or a distributed system formed by multiple physical servers, or can be configured as a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, and big data and artificial intelligence platforms; and the software can be an application that implements the medical image segmentation method, but is not limited to the above forms.

[0086] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0087] Figure 1 is an optional flowchart of the medical image segmentation method provided in the embodiments of the present application, applied to a client, Figure 1 The method in the above embodiment can include, but is not limited to, steps 101 to 110.

[0088] Step 101, obtaining a sample medical image set, the sample medical image set including a sample medical image, the sample medical image having a sample segmentation label, the sample segmentation label indicating a position of a region of interest of the sample medical image;

[0089] Step 102, obtaining a data distribution hint based on a pixel value of each sample medical image in the sample medical image set;

[0090] In step 103, a sample segmentation label is obtained, and an annotation sparse prompt is determined; the annotation sparse prompt is used to represent the annotation form of the sample segmentation label;

[0091] In step 104, a general knowledge prompt is obtained;

[0092] In step 105, the general knowledge prompt, the data distribution prompt, and the annotation sparse prompt are fused to obtain annotation information;

[0093] In step 106, an aggregated global parameter distributed by a server is received; the aggregated global parameter is aggregated by the server based on model parameters of the medical image segmentation model sent by multiple clients;

[0094] In step 107, the model parameters of the medical image segmentation model are initialized according to the aggregated global parameter;

[0095] In step 108, the sample medical image and the annotation information are input into the medical image segmentation model for image segmentation to obtain a predicted segmentation label;

[0096] In step 109, the medical image segmentation model is adjusted according to the predicted segmentation label and the sample segmentation label to obtain a target medical image segmentation model;

[0097] In step 110, a target medical image is obtained, and the target medical image is segmented by the target medical image segmentation model.

[0098] In step 101 of some embodiments, the client obtains a sample medical image set from a local database, the sample medical image set is used for training of a local medical image segmentation model of the client, and the sample medical image set includes multiple sample medical images. The sample medical images can be fundus images, endoscopic polyp images, etc. The fundus images of multiple clients are as shown in FIG. 1A, and the endoscopic polyp images of multiple clients are as shown in FIG. 1B. Figure 2a Figure 2b ​The sample medical images have sample segmentation labels indicating the locations and / or categories of the regions of interest of the sample medical images. The sample medical images are divided into regions of interest and background regions, and the regions of interest are regions that need to be segmented from the sample medical images and are helpful for computer-aided diagnosis, which are different from the background regions. The regions of interest segmented by different medical image segmentation tasks are different. For example, in an optical disc and cup segmentation (ODOC) task, the regions of interest are the optical disc and cup. In a foveal avascular zone (FAZ) segmentation task, the region of interest is an avascular zone located in the center of the macula. In a polyp segmentation task, the region of interest is a polyp region in a colonoscopy image. If the sample medical image is a fundus image, the sample segmentation label can be the category of a fundus-specific structure and the location of the fundus-specific structure in the fundus image, and the fundus-specific structure can be a cup, a macular region, and a retinal blood vessel.

[0099] In step 102 of some embodiments, Figure 2a and Figure 2b In some embodiments, the data samples from different clients show domain gaps in their distributions, and different clients have different domain biases due to differences in imaging equipment, patient populations, and physician expertise, etc., so that data heterogeneity is further intensified in medical image segmentation tasks. In this case, each client needs to adapt to the locally unique feature distribution or prediction pattern, and train a personalized local model based on personalized federated learning (pFL) technology. Existing personalized federated learning methods mainly include methods of local fine-tuning, segmenting the network and retaining the personalized layer parameters of the local model, similarity-based aggregation or knowledge distillation. However, these methods may suffer from catastrophic forgetting of public knowledge and overfitting to local distribution, fail to utilize valuable similarities from other clients, or result in inappropriate personalization due to coarse-grained aggregation. Embodiments of the present application introduce prompts to enable the model to better perceive contextual information such as different data distributions or downstream tasks through a set of manually set or learnable vectors, making it possible to train different data (different distributions, label forms, tasks, etc.) uniformly with almost no increase in computational complexity.

[0100] A Data Distribution Prompt (DDP) indicates the distribution of each medical image in a sample medical image set within the data space. The DDP allows the local model to be aware of the local data distribution and adaptively adjust its parameters. The sample medical image sets for N clients are {D1,...,D...}. N}, where D i For the sample medical image set of the i-th client, X i Y represents the sample medical image of the i-th client. i Let represent the sample segmentation label for the i-th client. Each client uses local data D. i Optimize the local target, and then update the local model. The data distribution for each client is different, i.e., P(D) i )≠P(D j ).

[0101] Data distribution cues can be determined based on different evaluation metrics, such as pixel mean, pixel variance, median, and similarity. In some embodiments, the sample medical images are converted to grayscale to obtain grayscale images. The pixel values ​​of all pixels in the grayscale image are summed, and the result is divided by the total number of pixels to obtain the pixel mean. The pixel mean of all sample medical images in the sample medical image set is used as the data distribution cues. In some embodiments, the squares of the differences between the pixel value of each pixel in the grayscale image and the pixel mean are summed, and the result is divided by the total number of pixels to obtain the pixel variance. The pixel variance of all sample medical images in the sample medical image set is used as the data distribution cues. In some embodiments, the pixel values ​​of all pixels in the grayscale image are arranged from smallest to largest to obtain the pixel median. The pixel median of all sample medical images in the sample medical image set is used as the data distribution cues. This application does not specifically limit the evaluation metrics; they can be selected according to actual conditions. The t-SNE visualization method is used to visualize the data distribution cues. Figure 3a , Figure 3b and Figure 3c This displays data distribution hints for different clients. Figure 3a Provide hints regarding the data distribution at Sites A, B, C, D, and E in the ODOC task. Figure 3b Provide hints regarding the data distribution at sites A, B, C, D, and E in the FAZ task. Figure 3cData distribution hints for sites A, B, C and D in the polyp segmentation task. The correlation coefficient matrix between different clients is calculated by methods such as Euclidean distance and cosine similarity, and the correlation coefficient matrix is used as the data distribution hint, wherein the correlation coefficient matrix is used to represent the similarity between any two clients. Figure 3d Correlation coefficient matrix between sites A, B, C, D and E in the ODOC task, Figure 3e Correlation coefficient matrix between sites A, B, C, D and E in the FAZ task, Figure 3f Correlation coefficient matrix between sites A, B, C and D in the polyp segmentation task. Each element in the correlation coefficient matrix is in the range of [0, 1], and the higher the element value, the greater the similarity between any two sites. If the element value is 0, it means that the data of the two sites is completely different, and if the element value is 1, it means that the data of the two sites is completely the same.

[0102] In step 103 of some embodiments, weakly supervised segmentation usually utilizes sparse granularity supervision through novel loss function design, consistency learning, adversarial learning or data synthesis. The embodiments of the application integrate weakly supervised segmentation into federated learning, allowing each client to have different forms of weak labels to further reduce the annotation cost of each client. Annotation sparsity prompts (ASP) are manually set to represent the annotation form of the sample segmentation label and reflect the annotation granularity of the sample segmentation label. As shown in Figure 4 , the annotation form of the sample segmentation label may be different, including point, scribble, bounding box, block and full mask. From point, scribble, bounding box, block to full mask, the annotation granularity gradually becomes dense. Among them, the point annotation method has the highest sparsity and the lowest density. The full mask annotation method has the lowest sparsity and the highest density. Obtain the sample segmentation label, determine the annotation granularity according to the sample segmentation label, and one hot encode the annotation granularity to obtain the annotation sparsity prompt. For example, the sample segmentation label is a bounding box, and the annotation granularity of the bounding box is set to 3, then the annotation sparsity prompt can be 011. Through sparse annotation, the difficulty of obtaining a large-scale fully annotated dataset is reduced.

[0103] In step 104 of some embodiments, in order to utilize the valuable similarities of other clients, a universal knowledge prompt (UKP) is obtained, the universal knowledge prompt being general knowledge and prompts for guiding medical image segmentation, the universal knowledge prompt can be a medical image segmentation algorithm, an evaluation index, an image feature that can obtain a better segmentation effect, etc. All clients jointly maintain two learnable parameters of data distribution prompts and universal knowledge prompts.

[0104] In step 105 of some embodiments, in order to obtain a better medical image segmentation effect, the universal knowledge prompt, the data distribution prompt and the annotation sparse prompt are spliced to integrate the triple prompt to obtain annotation information. Resource-intensive visual prompts or strict encoding or single-hot code output by a contrastive language-image pretraining (CLIP) usually aggravate calculation and privacy, and cannot understand feature information across clients. Through the triple latent prompt strategy, the learnable parameters are effectively combined with the pre-defined one-hot encoding, so that the model can perceive the common knowledge across multiple clients, the local data distribution and the supervision granularity (annotation granularity), realize the adaptive adjustment of the local model, and obtain more detailed and efficient responses. Each client can establish a personalized model, rather than building a single shared model like general federated learning, so that each client can independently perform different medical image segmentation tasks. At the same time, there is no need for data sharing between clients, ensuring the security of the data.

[0105] In step 106 of some embodiments, in order to establish a personalized model for each client, represented as the data island problem caused by the inability of the client to obtain the data of other clients is solved, and the support of a central server is needed, and the model update is coordinated and aggregated by the central server. Wherein, θ iThese are the model parameters for the client's local model. Each client has a different data distribution. If the model is trained using only local data, the trained model will perform well on the client's local device, but its generalization performance on unseen data will be poor. The involvement of the server can generate a global model update by aggregating the model parameters from local devices, thereby helping to improve the model's generalization performance. By aggregating model parameters from different devices and integrating different model information, the variance of model updates can be reduced, the differences between different devices can be narrowed, and the learning of the global model can be promoted. At the same time, the information exchanged between the client and the server is model parameters, not the client's local data, ensuring data security. Through client-server federated learning (FL), each client can train its own local model without the need for clients to share data, ensuring data security.

[0106] The server-side uses a certain aggregation method to aggregate the model parameters of medical image segmentation models sent from multiple clients to obtain aggregated global parameters, which are then sent to individual clients. The clients receive the aggregated global parameters assigned by the server. The client-side local model is considered a local model, and the server-aggregated model is considered a global model; both local and global models share the same network architecture. Due to data heterogeneity and label heterogeneity, a unified and easily deployable personalized federated learning framework is needed for weakly supervised segmentation. This application proposes a novel Personalized Federated Learning Framework with Learnable Prompt and Aggregation (FedLPPA) to uniformly utilize heterogeneous weak supervision for medical image segmentation. This framework maintains prompts representing general knowledge, data distribution, and annotation sparsity. The network architecture is as follows: Figure 5 As shown, N clients and servers exchange data, with each client having a local model. The local model is obtained by inserting an auxiliary decoder and a Tri-prompt Dual-attention Fusion (TDF) module into a U-shaped network. A Multilayer Perceptron (MLP) connects the TDF module to the segmentation head of the main decoder.

[0107] In step 107 of some embodiments, the medical image segmentation model is a local model that is local to the client. The global model and the local model are set in the same way, both setting the encoder and the TDF module as the global shared part, and setting the main decoder and the auxiliary decoder with MLP as the personalized component. The model parameters of the global shared part are the same for all clients, while the model parameters of the personalized component can be different for different clients. The model parameters of the global shared part are denoted as , the model parameters of the main decoder with MLP are denoted as and the model parameters of the auxiliary decoder are denoted as The training process of the medical image segmentation model involves multiple rounds of alternating updates between the clients and the server. In the t-th round, all clients receive the aggregated global parameters of the global model in the (t-1)-th round from the server, which include the model parameters of the global shared part of the global model , the model parameters of the main decoder of the global model and the model parameters of the auxiliary decoder of the global model where i represents the i-th client. The model parameters of the global shared part of the global model and the model parameters of the main decoder of the global model received by all clients are the same, and the model parameters of the auxiliary decoder of the global model received by each client are customized allocated by the server according to a preset parameter selection strategy. The global shared part of the local model is initialized using , the main decoder of the local model is initialized using and the auxiliary decoder of the local model is initialized using The model parameters of the main decoder of the local model of the i-th client in the t-th round are denoted as and the model parameters of the auxiliary decoder of the local model of the i-th client in the t-th round are denoted as These two parameters are derived based on and the calculation methods of these two parameters are shown in equations (1) and (2).

[0108]

[0109]

[0110] where is the model parameter of the main decoder of the local model of the i-th client in the (t-1)-th round; is the model parameter of the main decoder of the global model in the (t-1)-th round; W M，i is the learnable aggregation weight parameter of the main decoder of the local model of the i-th client; e is the Hadamard product; the model parameters of the local model-assisted decoder for the i-th client in the t-1th round; the model parameters of the global model-assisted decoder assigned to the i-th client in the t-1th round; A，i the learnable aggregation weight parameter of the local model-assisted decoder for the i-th client.

[0111] with σ(w) = max(0, min(1, w)), to ensure that the learnable aggregation weight parameter is in the range of [0, 1]. M，i , W A，i initialized as an identity matrix and iteratively updated during local data training.

[0112] Please refer to Figure 6 In some embodiments, the predicted segmentation label includes a first predicted segmentation sub-label and a second predicted segmentation sub-label, the medical image segmentation model includes a global shared module, a local main decoder and a local auxiliary decoder, and step 108 can include but is not limited to steps 610-630:

[0113] Step 610: performing context feature extraction on the sample medical image and the annotation information through the global shared module to obtain context injection features:

[0114] Step 620: performing image segmentation on the context injection features through the local main decoder to obtain the first predicted segmentation sub-label:

[0115] Step 630: performing image segmentation on the context injection features through the local auxiliary decoder to obtain the second predicted segmentation sub-label:

[0116] In step 610 of some embodiments, the global shared module is a global shared part of the local model. As shown in Figure 5 The global shared module includes a TDF module and an encoder, and the TDF module is used for unified and lightweight context learning, including a dual attention fusion module. The context injection features are context information in the sample medical image, and the context information is very important for accurately segmenting structures of interest such as lesions, tissues, etc. The context information can be features such as the position, shape, texture, pixel intensity, etc. of the structure of interest. Context learning aims to enable the network to adaptively optimize for various contexts by adopting conditions or prompts that encapsulate external factors such as scenarios, tasks or modalities, etc.

[0117] Weakly Supervised Segmentation (WSS) is to perform image segmentation by providing only weak labels or less supervision information. In WSS, an effective strategy is to generate reliable pseudo proposals in the unlabeled regions outside sparse labels. Since this strategy adopts a multi-level tree affinity method, it will cause a large local computation overhead. Another test is to train itself using the model's own predictions, but this is prone to update stagnation and noise accumulation. In order to alleviate these problems, the application adopts a dual-decoder strategy to personalize auxiliary decoders from sites with similar local distributions to reduce noise accumulation and generate excellent pseudo labels. The dual-decoder includes a local main decoder and a local auxiliary decoder, and the auxiliary decoder can obtain beneficial information from other clients to adjust the model parameters individually with the beneficial information of other clients, local data distribution and supervision granularity.

[0118] In step 620 of some embodiments, the local main decoder includes a main decoder, an MLP and a segmentation head of the local model, and the MLP has two linear layers. The context-injected features are input into the main decoder and the MLP respectively, the context-injected features are extracted by the main decoder, and the context-injected features are reduced in dimension by the MLP. The output of the main decoder and the output of the MLP are combined by Hadamard product, which can more directly transmit the context-injected features and enhance the flexibility of parameter optimization to enhance the ability of model context adjustment. The combined features are segmented by the segmentation head to obtain the first predicted segmentation sub-label. The first predicted segmentation sub-label is the segmentation result of the local main decoder output for the region of interest of the sample medical image.

[0119] In step 630 of some embodiments, the local auxiliary decoder and the local main decoder are two branches in parallel with each other, and the local auxiliary decoder includes an auxiliary decoder and a segmentation head. The context-injected features are extracted by the auxiliary decoder, and the extracted features are segmented by the segmentation head to obtain the second predicted segmentation sub-label. The second predicted segmentation sub-label is the segmentation result of the local auxiliary decoder output for the region of interest of the sample medical image.

[0120] Step 109 can include but is not limited to steps 640 to 650:

[0121] Step 640, generating a pseudo prediction label according to the first predicted segmentation sub-label and the second predicted segmentation sub-label;

[0122] Step 650, adjusting the medical image segmentation model according to the pseudo prediction label, the first predicted segmentation sub-label, the second predicted segmentation sub-label and the sample segmentation label to obtain a target medical image segmentation model.

[0123] In step 640 of some embodiments, in weakly supervised learning, the diversity of training samples can be increased by pseudo labels, and adding pseudo labels can provide additional supervision signals to make the model learn better feature representations, and self-training with pseudo labels can improve the generalization and robustness of the model. The pseudo prediction label is a pseudo suggestion output by the dual decoder, and is obtained by randomly mixing the outputs of the dual decoder. The first weight coefficient is multiplied by the first prediction segmentation sub-label, the second weight coefficient is multiplied by the second prediction segmentation sub-label, and the results of the two multiplications are added to obtain the pseudo prediction label. The first weight coefficient and the second weight coefficient add up to 1. The first weight coefficient and the second weight coefficient are both hyperparameters, used to randomly modulate the influence of the main decoder and the auxiliary decoder on the pseudo prediction label, and can give the main decoder a higher weight. The generation method of the pseudo prediction label is shown in formula (3).

[0124]

[0125] wherein, represents the pseudo prediction label; k is a sample identifier, representing the kth sample; arg max represents taking the maximum value; λ m is the first weight coefficient, λ m is a random number in the range of (0.7, 1); · represents multiplication; p M，k is the prediction probability of the kth sample output by the main decoder; (1-λ m ) is the second weight coefficient; p A，k represents the prediction probability of the kth sample output by the auxiliary decoder.

[0126] In step 650 of some embodiments, in order to stabilize the training of the dual decoder and promote more fine-grained personalized knowledge transfer, for each decoder of the client, a learnable aggregation method similar to residual learning is used to learn an aggregation weight matrix, which is used to personalize the decoder parameters on an element basis. The method of determining the decoder parameters according to the aggregation weight matrix can refer to the above formula (1) to formula (2), which will not be described here. The target loss is calculated according to the pseudo prediction label, the first prediction segmentation sub-label, the second prediction segmentation sub-label and the sample segmentation label, and the learnable aggregation weights of the local main decoder and the learnable aggregation weights of the local auxiliary decoder are adjusted to minimize the target loss, to obtain a target medical image segmentation model. The target medical image segmentation model is the final model obtained by training. Through the learnable aggregation, the model parameters of the dual decoder are further personalized, so that each client can train its own image segmentation model locally.

[0127] The steps 610 to 650 can obtain beneficial information from other clients by the auxiliary decoder and the main decoder, and learn the context by using the beneficial information of other clients, the local data distribution and the supervision granularity, so that each local model can adapt to the unique data distribution and supervision sparsity, and realize personalized learning. Meanwhile, the learnable aggregation mechanism can obtain fine personalized parameters, reduce local noise accumulation and generate excellent pseudo labels for weak supervision learning, so as to obtain an image segmentation model with high segmentation accuracy.

[0128] Referring to Figure 7 In some embodiments, the step 610 can include but is not limited to steps 710 to 720.

[0129] In step 710, the sample medical image is image encoded by an encoder to obtain encoded features.

[0130] In step 720, the encoded features and the annotation information are double-attention fused and feature extracted by a double-attention fusion module to obtain context injection features.

[0131] In step 710 of some embodiments, the sample medical image is input into the encoder for image encoding to obtain encoded features. The encoded features are image features of the sample medical image, and the encoded features can be texture, color, edge, corner, shape, structure and the like of the image. The encoder can be a transformer model, Resnet50 and the like, and the specific network structure of the encoder is not limited in the embodiments of the present application. The feature size of the encoded features is 256xHxW, 256 is the feature channel number, H is the height of the encoded features, and W is the width of the encoded features.

[0132] In step 720 of some embodiments, as Figure 8 shown, the annotation information is a local hint of the client, the local main decoder is a local main decoder, the local auxiliary decoder is a local auxiliary decoder, and the old aggregation weight is the learnable aggregation weight parameter before updating. The local model is trained by using the local data of the client to update the aggregation weight. For the i-th client, the local hint p i is obtained by connecting the shared general knowledge hint p U , the data distribution hint p D，i selected in the local channel dimension, and the annotation sparsity hint p S，i expanded in dimension. Wherein, p U ∈R 1×H×W , P D，i ∈R 1×H×W , P S，i ∈R 3×H×W , the local hint p i =cat(p U , pD，i , p S，i ). cat denotes concatenation operation, R denotes real number space, H and W denote the height and width of the encoder output encoding feature F respectively. These personalized cues are fused with data representation and perceived by the decoder through a dual attention (spatial and channel) fusion mechanism. The client also provides n one-hot annotation sparse cues, which are set to 3 in this application, to distinguish the point, scribble and block level sparsity, and the bounding box label is preprocessed into one of these categories.

[0133] It can be understood that the annotations do not provide direct pixel-level supervision, but act as indicators of location and extent. Therefore, they are converted into alternative types of sparse labels for paradigm alignment. Taking fundus images, OCT angiography (OCTA) images and endoscopic images as examples, the preprocessing process of the bounding box label is described. The fundus image and the OCTA image use traditional box annotation, and the endoscopic image uses rotated box annotation. The annotation sparse cues of the fundus image, the OCTA image and the endoscopic image are set to match the categories, and the moderate sparsity of the fundus image is converted into scribble and the dense sparsity of the other two images is converted into block. In the fundus image, the largest ellipse inside the annotation box rectangle is obtained, and a small circular region in the inner ring is removed, and the skeletonization is performed on the removed region to obtain the scribble label. In the OCTA image, the block label is the foveal avascular zone in the center, and the background is outside. In the endoscopic image, the block label is a polyp in the center, and the background is outside. Other methods, such as using SAM-Med2D segmentation (box as cue) and moderate erosion to reduce label noise, are also feasible.

[0134] The steps 710 to 720 described above learn the context by the encoding feature and the annotation information, determine the context injection feature, so that each local model can perceive the local data distribution, the supervised sparsity, the general knowledge and the features of the current sample medical image, and realize personalized learning.

[0135] In some embodiments, the step 720 can include but is not limited to steps 721 to 724:

[0136] Step 721, performing convolution processing on the encoding feature and the annotation information to obtain a convolution feature;

[0137] Step 722, performing spatial attention feature extraction on the convolution feature to obtain a spatial attention feature;

[0138] Step 723, performing channel attention feature extraction on the convolution feature to obtain a channel attention feature;

[0139] Step 724, performing feature fusion on the encoding feature, the convolution feature, the spatial attention feature and the channel attention feature to obtain the context injection feature.

[0140] In step 721 of some embodiments, the encoded features and annotation information are concatenated, and the concatenated features are convolved by two convolutional blocks to obtain convolutional features. These two convolutional blocks share the same network structure, both including convolutional layers (Conv), batch normalization (BN) layers, and ReLU activation layers. The feature size of the convolutional features is 64×H×W, where 64 is the number of feature channels, H is the height of the convolutional feature, and W is the width of the convolutional feature. The convolutional features are the initial fusion results. To further fuse and enhance functionality and improve the comprehensiveness of feature extraction, a dual attention mechanism is followed in both the spatial and channel dimensions.

[0141] In step 722 of some embodiments, the spatial attention value that measures the influence of the i-th location feature point on the j-th location feature point is represented as:

[0142]

[0143] Where Q represents the query weight matrix; K represents the key weight matrix; T represents the transpose of the matrix; and N is the number of elements in the query weight matrix. This represents matrix multiplication.

[0144] like Figure 8 As shown, in spatial attention, the query weight matrix Q, key weight matrix K, and value weight matrix V have the same matrix size, all being 64×H×W. The convolutional features... Multiplying the query weight matrix Q with the first query vector yields the convolutional feature vector. Multiplying the convolutional features by the key weight matrix K yields the first key vector, which is then used to generate the convolutional features. The first value vector is obtained by multiplying the first query vector by the value weight matrix V. The first query vector is transposed, and then multiplied by the first key vector to obtain the spatial attention. The spatial attention is then activated using the softmax function σ to obtain the activation feature map. This activation feature map is then multiplied by the first value vector to obtain the spatial attention feature. To ensure the feature size of the spatial attention feature meets the output size requirements, the spatial attention feature needs to be deformed. Through the spatial attention mechanism, the model's attention can be focused on specific regions of the sample medical image, making the model more attentive to important image areas, thereby improving the model's accuracy and robustness.

[0145] In step 723 of some embodiments, the channel attention value that measures the influence of the i-th location feature point on the j-th location feature point is represented as:

[0146]

[0147] Where Q′ represents the query weight matrix; K′ represents the key weight matrix; T represents the transpose of the matrix; and C is the number of elements in the key weight matrix. This represents matrix multiplication.

[0148] like Figure 8 As shown, in channel attention, the query weight matrix Q′, key weight matrix K′, and value weight matrix V′ have the same matrix size, all three being 64×H×W. The convolutional features... Multiplying the query weight matrix Q′ by the second query vector yields the convolutional feature vector. Multiplying the convolutional features by the key weight matrix K′ yields the second key vector, which in turn convolves the features. The second value vector is obtained by multiplying the value weight matrix V′. The second query vector is transposed, and then multiplied by the second key vector to obtain the channel attention. The channel attention is activated using the softmax function σ to obtain an activation feature map with a size of 64×64. This activation feature map is then multiplied by the second value vector to obtain the channel attention feature. Similarly, to ensure the feature size of the channel attention feature meets the output size requirements, it needs to be deformed. Through the channel attention mechanism, the feature map weights of each layer of the network can be dynamically adjusted according to the importance of each channel, allowing the model to more adaptively learn appropriate feature representations. The channel attention mechanism effectively reduces the influence of redundant features, improving the model's expressive power and generalization ability.

[0149] In step 724 of some embodiments, the convolutional features and spatial attention features are fused to obtain a first fused feature. The convolutional features and channel attention features are then fused to obtain a second fused feature. The first and second fused features are then fused to obtain a target fused feature. The encoded features and the target fused feature are then concatenated to obtain a context-injected feature.

[0150] Specifically, the convolutional features and the spatial attention features are weighted to enhance the spatial attention features and highlight important image regions, obtaining first fusion features. The convolutional features and the channel attention features are weighted to enhance the channel attention features and highlight important features, obtaining second fusion features. In order to comprehensively utilize important regions and important features, the first fusion features and the second fusion features are added to obtain target fusion features. In order to more comprehensively extract features and avoid the loss of small information caused by step-by-step feature extraction, the encoding features and the target fusion features are spliced through a skip connection to obtain context injection features. The calculation method of the target fusion features is shown in formula (6), and the calculation method of the context injection features is shown in formula (7).

[0151]

[0152]

[0153] wherein, is the target fusion feature; γ s and γ c are two learnable parameters for balancing the influence of the channel attention features and the spatial attention features; F s represents the spatial attention features; F c represents the channel attention features; represents the convolutional features: F * represents the context injection features; F represents the encoding features; and cat represents a splicing operation.

[0154] The steps 721 to 724 can make the model focus on important regions and important features of the image at the same time through the double attention mechanism, and perform feature fusion on the encoding features, the convolutional features, the spatial attention features and the channel attention features, so as to fully express the features of the sample medical image, ensure the comprehensiveness of feature extraction, and reduce the dependence on large-scale fully annotated data sets.

[0155] For the weakly supervised target loss of the client, it comes from the first loss data L sup supervised by the sparse label (sample segmentation label) and the second loss data L pse supervised by the pseudo prediction label.

[0156] Referring to Figure 9 , in some embodiments, the step 650 can include but is not limited to including steps 910 to 930:

[0157] The step 910 performs first loss calculation according to the first prediction segmentation sub-label, the second prediction segmentation sub-label and the sample segmentation label, obtaining first loss data;

[0158] Step 920, performing second loss calculation according to the pseudo prediction label, the first prediction segmentation sub-label and the second prediction segmentation sub-label, to obtain second loss data;

[0159] Step 930, adjusting the medical image segmentation model according to the first loss data and the second loss data, to obtain a target medical image segmentation model.

[0160] In step 910 of some embodiments, performing third sub-loss calculation according to the first prediction segmentation sub-label and the sample segmentation label to obtain third sub-loss data; performing fourth sub-loss calculation according to the second prediction segmentation sub-label and the sample segmentation label to obtain fourth sub-loss data; and performing weighted calculation on the third sub-loss data and the fourth sub-loss data to obtain the first loss data. The calculation method of the first loss data is shown in formula (8).

[0161] L sup = 0.5 x L pCE (p M，k , y k ) + 0.5 x L pCE (p A，k , y k ) Formula (8)

[0162] Wherein, L sup represents the first loss data; p M，k represents the prediction probability output by the main decoder for the kth sample medical image, i.e. the first prediction segmentation sub-label; y k represents the sample segmentation label; p A，k represents the prediction probability output by the auxiliary decoder for the kth sample medical image, i.e. the second prediction segmentation sub-label; L pCE is the local cross-entropy.

[0163] The third sub-loss data can determine the difference between the first prediction segmentation sub-label and the sample segmentation label, and the fourth sub-loss data can determine the difference between the second prediction segmentation sub-label and the sample segmentation label. The main decoder and the auxiliary decoder will both affect the image segmentation result, and the third sub-loss data and the fourth sub-loss data are weighted according to the size of the influence to determine the first loss data, so that the model parameters of the main decoder and the auxiliary decoder are adjusted according to the first loss data, so that the first prediction segmentation sub-label and the second prediction segmentation sub-label both tend to approach the sample segmentation label, and the accuracy of image segmentation by the main decoder and the auxiliary decoder is improved respectively.

[0164] In step 920 of some embodiments, the pseudo prediction label is generated by the first prediction segmentation sub-label and the second prediction segmentation sub-label. In order to be able to generate an excellent pseudo prediction label, it is necessary to perform a second loss calculation according to the pseudo prediction label, the first prediction segmentation sub-label and the second prediction segmentation sub-label, to obtain second loss data, so as to adjust the model parameters of the main decoder and the auxiliary decoder according to the second loss data.

[0165] In step 930 of some embodiments, the first loss data and the second loss data are summed to obtain a target loss, and the target loss is minimized to adjust the aggregable learning parameters of the medical image segmentation model to obtain a target medical image segmentation model. The calculation method of the target loss is shown in formula (9).

[0166] L wss = L sup + λL pse Formula (9)

[0167] Wherein, L wss is the target loss, i.e. the weakly supervised segmentation loss; L sup is the first loss data; L pse is the second loss data; λ is a weighting parameter for measuring the influence of the first loss data and the second loss data on the target loss.

[0168] The updating method of the aggregable learning parameters is shown in formula (10) and formula (11).

[0169]

[0170]

[0171] Wherein, η is the learning rate; W M，i is the learnable aggregation weight parameter of the local main decoder of the i-th client; W A，i is the learnable aggregation weight parameter of the local auxiliary decoder of the i-th client; and are the gradients of the learnable aggregation weight parameters. The formula (10) and formula (11) are executed alternately to update and (W M，i , W A，i ). After the medical image segmentation model converges, the initial parameters of the double decoder in each iteration are obtained and the local model is trained using formula (9). It should be noted that when t>2, only a few iterations are needed to obtain (W M，i , W A，i ).

[0172] The steps 910 to 930 can generate excellent first predicted segmentation sub-labels and second predicted segmentation sub-labels by jointly adjusting the model parameters based on the first loss data and the second loss data, thereby improving the accuracy of medical image segmentation. Meanwhile, the first predicted segmentation sub-labels and the second predicted segmentation sub-labels can be used to generate better pseudo predicted labels, so as to perform weakly supervised learning by using the pseudo predicted labels, thereby enhancing the generalization performance of the model.

[0173] Please refer to Figure 10 In some embodiments, the step 920 can include but is not limited to steps 1010 to 1030:

[0174] In step 1010, first sub-loss calculation is performed based on the pseudo predicted label and the first predicted segmentation sub-label, and first sub-loss data is obtained.

[0175] In step 1020, second sub-loss calculation is performed based on the pseudo predicted label and the second predicted segmentation sub-label, and second sub-loss data is obtained.

[0176] In step 1030, loss fusion is performed based on the first sub-loss data and the second sub-loss data, and the second loss data is obtained.

[0177] In step 1010 of some embodiments, the first sub-loss calculation is performed based on a Dice loss function and the pseudo predicted label and the first predicted segmentation sub-label p M,k , and the first sub-loss data is

[0178] In step 1020 of some embodiments, the second sub-loss calculation is performed based on a Dice loss function and the pseudo predicted label and the second predicted segmentation sub-label p A,k , and the second sub-loss data is

[0179] In step 1030 of some embodiments, the first sub-loss data and the second sub-loss data are loss weighted, and the second loss data is obtained. The calculation method of the second loss data is shown in formula (12).

[0180]

[0181] Wherein, L pse is the second loss data; L Dice is the Dice loss function; p M,k represents the prediction probability output by the main decoder for the kth sample medical image, i.e., the first predicted segmentation sub-label; and p A,k represents the prediction probability output by the auxiliary decoder for the kth sample medical image, i.e., the second predicted segmentation sub-label. The pseudo prediction label is obtained.

[0182] The second loss data is obtained through the pseudo prediction label, and the model parameters are collectively adjusted based on the first loss data and the second loss data, so as to refine the model parameters and improve the accuracy of medical image segmentation.

[0183] After obtaining the target medical image segmentation model, that is, after completing the local update, all clients need to send the model parameters (this round of parameters) to the server end for global model aggregation and personalized parameter allocation of the auxiliary decoder. As shown in formula (13).

[0184]

[0185] In step 110 of some embodiments, the target medical image is obtained through medical imaging technology such as nuclear magnetic resonance, optical coherence tomography, etc. The target medical image is a label-free image. The target medical image is segmented by the target medical image segmentation model to segment the region of interest from the target medical image to assist the computer in disease screening and diagnosis.

[0186] Figure 11 is an optional flowchart of the medical image segmentation method provided by the embodiments of the present application, which is applied to the server end, Figure 11 The method in the server end can include but is not limited to steps 1110-1150.

[0187] Step 1110, receiving target model parameters of a medical image segmentation model sent by a plurality of clients; the medical image segmentation model includes a global shared module, a local main decoder and a local auxiliary decoder, and the target model parameters include first global shared parameters of the global shared module and local main decoder parameters of the local main decoder;

[0188] Step 1120, aggregating the first global shared parameters of the plurality of clients to obtain second global shared parameters;

[0189] Step 1130, aggregating the local main decoder parameters of the plurality of clients to obtain global main decoder parameters;

[0190] Step 1140, determining global auxiliary decoder parameters based on a preset parameter selection strategy and the local main decoder parameters of the plurality of clients;

[0191] Step 1150, the second global shared parameter, the global main decoder parameter and the global auxiliary decoder parameter are aggregated as the aggregated global parameter, and the aggregated global parameter is sent to the client to initialize the model parameter of the medical image segmentation model for medical image segmentation according to the aggregated global parameter.

[0192] In step 1110 of some embodiments, the target model parameter is the model parameter of the client medical image segmentation model after convergence, the medical image segmentation model includes a global shared module, a local main decoder and a local auxiliary decoder, and the target model parameter includes a first global shared parameter of the global shared module and a local main decoder parameter of the local main decoder. The first global shared parameter is the model parameter of the global shared module The local main decoder parameter is the model parameter of the local main decoder

[0193] In step 1120 of some embodiments, the parameter acquisition of the double decoder involves two stages, which are the aggregation or selection of the server side and the local learnable aggregation of the client side respectively. On the server side, the first global shared parameters of multiple clients are aggregated to obtain the second global shared parameter. The second global shared parameter is the model parameter of the global shared module of the global model. The second global shared parameter is expressed as:

[0194]

[0195] Wherein, is the second global shared parameter; is the first global shared parameter of the i-th client; N is the number of clients; ω i is the aggregation weight, |D i |is the number of samples in the i-th client sample medical image set; is the total number of samples in all client sample medical image sets.

[0196] In step 1130 of some embodiments, the local main decoder parameters of multiple clients are aggregated to obtain the global main decoder parameter. The global main decoder parameter is the model parameter of the main decoder of the global model. The global main decoder parameter is expressed as:

[0197]

[0198] Wherein, is the global main decoder parameter; ω i is the aggregation weight of the i-th client; is the local main decoder parameter of the i-th client.

[0199] In step 1140 of some embodiments, the global auxiliary decoder parameters are model parameters of an auxiliary decoder of the global model. The preset parameter selection strategy is a parameter selection strategy of the server-side auxiliary decoder. The server-side provides four optional preset parameter selection strategies. Based on the preset parameter selection strategy, the local main decoder parameters of the plurality of clients are selected or calculated to obtain the global auxiliary decoder parameters.

[0200] In step 1150 of some embodiments, the second global shared parameters, the global main decoder parameters and the global auxiliary decoder parameters are aggregated as aggregated global parameters, and the aggregated global parameters are sent to the clients to initialize the model parameters of the medical image segmentation model for training of the local model to perform medical image segmentation.

[0201] The steps 1110 to 1150 described above can determine the model parameters of the global model by aggregating the plurality of model parameters uploaded by the clients on the server side, and initialize the model parameters of the local model using the model parameters of the global model, so that the local model can obtain beneficial information of other clients, and improve the image segmentation performance of the local model. At the same time, each client can train its own local model without obtaining a large-scale completely annotated dataset, which reduces the difficulty of obtaining the dataset, and further reduces the difficulty of medical image segmentation.

[0202] Please refer to Figure 12 In some embodiments, the preset parameter selection strategy is a prompt similarity aggregation strategy, and step 1140 can include but is not limited to steps 1210 to 1230:

[0203] Step 1210, receiving the annotation information sent by the client; the annotation information includes a general knowledge prompt and a data distribution prompt;

[0204] Step 1220, calculating the similarity data between the clients according to the general knowledge prompt and the data distribution prompt;

[0205] Step 1230, aggregating the local main decoder parameters of the plurality of clients according to the similarity data to obtain the global auxiliary decoder parameters.

[0206] In step 1210 of some embodiments, in the t-th round, the server side collects the local main decoder parameters and the local prompts from all clients

[0207] ​​​​In step 1220 of some embodiments, similarity data is used to measure the similarity between sample medical image sets between different clients, and the similarity data is calculated using cosine similarity. The calculation method for similarity data is shown in formulas (16) to (17).

[0208]

[0209] a ij =ReLu(sim(c i c j )) Formula (17)

[0210] Where, sim(c i c j ) indicates client c i With client c j Cosine similarity between them; a ij For client c i With client c j The similarity between them. ReLu(x) = max(0, x) is used to eliminate potential conflicts caused by different client models.

[0211] In step 1230 of some embodiments, as shown in formula (18), based on the similarity data a ij Local master decoder parameters for multiple clients Aggregation is performed to obtain global auxiliary decoder parameters. global auxiliary decoder parameters The auxiliary decoder assigned to the i-th client local model.

[0212]

[0213] Steps 1210 to 1230 above determine the global auxiliary decoder parameters based on a cue similarity aggregation strategy, and initialize the auxiliary decoder of the local model based on the global auxiliary decoder parameters to guide the training process of the local model, avoiding inaccurate model parameters caused by random initialization, thereby improving the accuracy of medical image segmentation. The introduction of a personalized dual-decoder mechanism based on cue similarity, coupled with learnable aggregation, efficiently and adaptively generates pseudo-labels for each site, thus promoting weakly supervised segmentation.

[0214] In some embodiments, if the preset parameter selection strategy is a random strategy, then the parameters are selected from the local master decoder parameters of multiple clients. Randomly select one parameter as a global auxiliary decoder parameter and assign it to a different client. That is, randomly assign the collected multiple local main decoder parameters to different clients. If the preset parameter selection strategy is a fixed sequence strategy, assign the local main decoder parameters of multiple clients according to a predefined cyclic sequence, that is, assign the local main decoder in the preset order as the global auxiliary decoder parameter. If the preset parameter selection strategy is a highest prompt similarity strategy, receive the prompt information sent by the client, the prompt information including general knowledge prompts and data distribution prompts, calculate the similarity data between the clients according to the general knowledge prompts and the data distribution prompts, and select the local main decoder parameter corresponding to the maximum similarity data as the global auxiliary decoder parameter. That is, when j≠i, select the maximum a ij Correspondingly As

[0215] The FedLPPA is benchmarked against several representative FL frameworks, including commonly used gFL methods such as FedAvg and FedProx, and state-of-the-art (SOTA) pFL methods including FT, FedBN, FedAP, FedRep, MetaFed, FedLC, FedALA, and FedlCRA. FedAvg computes a weighted average of each local model parameter according to the sample size to obtain a single global model. FedProx is an extension of FedAvg that incorporates proximal terms into the optimization to address system heterogeneity, thereby enhancing stability and performance in heterogeneous environments. Among the pFL methods, FT denotes FedAvg with site fine-tuning. FedBN and FedRep treat all batch normalization (BN) layers and task heads (e.g., final convolutional layers) as individualized components of the model, respectively. FedAP aggregates through BN layer statistics to assess similarity between sites while preserving individualized parameters for each site BN layer. MetaFed enables individualization at each site through static (fixed order) cyclic knowledge distillation. FedALA employs an adaptive local aggregation module to facilitate knowledge transfer between the global and local models, thereby enabling individualization. FedLC enables individualization through the use of local calibration that leverages inconsistencies in feature and prediction levels. FedICRA addresses the same setting as this paper by utilizing one-hot hints and designing a novel objective function for WSS. Given that some of the aforementioned methods were originally designed for classification tasks, efforts are made to preserve their original design principles while adapting them to the segmentation task. All comparative methods consistent with FedLPPA are trained in the WSS setting on the same dataset. Additionally, comparisons are made with some baselines and ideal settings, including local training (LT) with weak labels and full labels, and centralized training (CT). All methods are evaluated using two metrics: Dice Similarity Coefficient (DSC) and 95% Hausdorff Distance (HD95[px]).

[0216]

[0217] Table 1

[0218]

[0219]

[0220] Table 2

[0221] Tables 1 and 2 list the quantitative results of the ODOC segmentation task, with OD being the optic disc and OC being the optic cup. Overall, all FL methods enhance the overall segmentation performance at all sites compared to LT. Sites C and E achieve significant performance improvement after participating in FL, as their diverse local data distribution makes it challenging to train a robust model using only local data. Site D also experiences significant performance improvement after joining FL, as it uses point annotations where the supervision signal is insufficient to train a sufficient model. As shown in Figure 3a and Figure 3d , the pFL method that trains personalized models for different parts generally has higher performance than the gFL method. FedAP and FedICRA achieve significant performance, while the FedLPPA of the present application further enhances the performance. All clients benefit within the framework of the present application, showing clear advantages in overall performance over CL (weak) and approaching the performance of CL (full).

[0222]

[0223] Table 3

[0224]

[0225]

[0226] Table 4

[0227] Tables 3 and 4 list the quantitative results of the FAZ and polyp segmentation tasks, respectively. As shown in Figure 3b , 3eAs shown in Figs. 3c and 3f, unlike the fundus scene, the distribution of OCTA and endoscopic images of different clients varies relatively little. Some pFL methods (such as FedBN, FedAP, FedRep and MetaFed) even perform worse than typical gFL methods (such as FedAvg and FedProx) due to inappropriate or coarse-grained personalization. FedLPPA can make fine-grained personalized adjustments, thanks to the proposed TDF module (which can enable context learning to adapt to different scenarios) and the Prompt similarity Dual-decoder with Learnable Aggregation mechanism (PDLA). FedLPPA enables each client model to perceive different scenarios in a context learning manner and make more fine-grained personalized adjustments, thanks to the proposed TDF module and PDLA mechanism. Therefore, it achieves excellent performance in various data heterogeneous tasks. It is worth noting that FedLPPA significantly outperforms other federated learning methods in the FAZ segmentation task and is very close to the performance of the CT (complete) setting. It is also worth mentioning that in the polyp segmentation scenario, FedLPPA not only significantly outperforms other methods, but also exceeds the CT (complete) setting by 4.6% in DSC, fully demonstrating the potential of WSS combined with the FL paradigm. Figure 13 Visualization results of the three tasks are provided, and GT is the ground truth image.

[0228] Please refer to Figure 14 The embodiments of the present application also provide a medical image segmentation device, which is applied to a client and can implement the above medical image segmentation method. The medical image segmentation device comprises:

[0229] An image acquisition module 1401 is configured to acquire a sample medical image set, wherein the sample medical image set comprises a sample medical image, and the sample medical image has a sample segmentation label, and the sample segmentation label indicates the position of a region of interest of the sample medical image.

[0230] A data distribution prompt acquisition module 1402 is configured to obtain a data distribution prompt based on the pixel value of each sample medical image in the sample medical image set.

[0231] An annotation sparsity prompt acquisition module 1403 is configured to acquire the sample segmentation label and determine an annotation sparsity prompt. The annotation sparsity prompt is used to represent the annotation form of the sample segmentation label.

[0232] A general knowledge prompt acquisition module 1404 is configured to acquire a general knowledge prompt.

[0233] The prompt fusion module 1405 is configured to fuse the general knowledge prompt, the data distribution prompt, and the annotation sparsity prompt to obtain annotation information.

[0234] The receiving module 1406 is configured to receive an aggregated global parameter distributed by a server end; the aggregated global parameter is aggregated by the server end based on model parameters of a medical image segmentation model sent by multiple client ends.

[0235] The initialization module 1407 is configured to initialize model parameters of the medical image segmentation model according to the aggregated global parameter.

[0236] The input module 1408 is configured to input a sample medical image and the annotation information into the medical image segmentation model for image segmentation to obtain a predicted segmentation label.

[0237] The adjustment module 1409 is configured to adjust the medical image segmentation model according to the predicted segmentation label and a sample segmentation label to obtain a target medical image segmentation model.

[0238] The medical image segmentation module 1410 is configured to acquire a target medical image and perform medical image segmentation on the target medical image by using the target medical image segmentation model.

[0239] Embodiments of the present application further provide a medical image segmentation device applied to a server end and capable of implementing the medical image segmentation method described above. The medical image segmentation device comprises:

[0240] A receiving module is configured to receive target model parameters of a medical image segmentation model sent by multiple client ends; the medical image segmentation model comprises a global shared module, a local main decoder, and a local auxiliary decoder; the target model parameters comprise first global shared parameters of the global shared module and local main decoder parameters of the local main decoder.

[0241] A first aggregation module is configured to aggregate the first global shared parameters of the multiple client ends to obtain second global shared parameters.

[0242] A second aggregation module is configured to aggregate the local main decoder parameters of the multiple client ends to obtain global main decoder parameters.

[0243] A determination module is configured to determine global auxiliary decoder parameters based on a preset parameter selection strategy and the local main decoder parameters of the multiple client ends.

[0244] A sending module is configured to send the second global shared parameters, the global main decoder parameters, and the local main decoder parameters as an aggregated global parameter, and send the aggregated global parameter to the client ends, so that the client ends initialize model parameters of a medical image segmentation model according to the aggregated global parameter to perform medical image segmentation.

[0245] The specific implementation of the medical image segmentation apparatus is basically the same as the specific embodiment of the medical image segmentation method described above, and thus will not be described here again.

[0246] The embodiments of the present application further provide an electronic device including a memory and a processor. The memory stores a computer program. The processor implements the medical image segmentation method described above when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0247] Please refer to Figure 15 , Figure 15 The hardware structure of the electronic device of another embodiment is illustrated, which includes:

[0248] The processor 1510 can be implemented in a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, etc., and is used to execute related programs to implement the technical solutions provided by the embodiments of the present application.

[0249] The memory 1520 can be implemented in the form of a ROM (ReadOnly Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory), etc. The memory 1520 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 1520 and are called and executed by the processor 1510 to implement the medical image segmentation method of the embodiments of the present application.

[0250] The input / output interface 1530 is used to realize information input and output.

[0251] The communication interface 1540 is used to realize the communication interaction between the device and other devices. The communication can be realized by a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).

[0252] The bus 1550 is used to transmit information between various components (for example, the processor 1510, the memory 1520, the input / output interface 1530, and the communication interface 1540) of the device.

[0253] The processor 1510, the memory 1520, the input / output interface 1530, and the communication interface 1540 are connected to each other through the bus 1550 to realize the communication connection between them inside the device.

[0254] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the medical image segmentation method.

[0255] The memory, as a non-transitory computer readable storage medium, can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory disposed remotely relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0256] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0257] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than those shown in the figures, or combine certain steps, or different steps.

[0258] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, that is, can be located in one place, or can be distributed on multiple network units. According to actual needs, part or all of the modules can be selected to achieve the purpose of the embodiments of the present application.

[0259] Those skilled in the art can understand that all or some steps in the above disclosed method, the functions of the modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0260] The terms "first", "second", "third", "fourth", and the like in the description of this application and in the claims hereof, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so termed herein is solely for the convenience of the reader and does not limit the scope of the application. It is also to be understood that the description and examples in this application are intended to cover all possible combinations where any of the several elements can represent one or more elements.

[0261] It should be understood that, in the application, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" is used to describe the relationship between associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0262] In several embodiments provided in the application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the above-mentioned units is only a logical functional division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The coupling or direct coupling or communication connection between the displayed or discussed each other can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0263] The units described above as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on multiple network units. According to actual needs, part or all of the units can be selected to achieve the purpose of the embodiment of the application.

[0264] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0265] If the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in part, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions used to cause a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods in the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various other media that can store programs.

[0266] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not limited to the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A method of medical image segmentation, characterized by, The method applied to the client comprises: obtaining a sample medical image set comprising sample medical images having sample segmentation labels indicating the positions of regions of interest of the sample medical images; obtaining data distribution hints based on the pixel values of each of the sample medical images in the sample medical image set; obtaining the sample segmentation labels to determine annotation sparse hints; the annotation sparse hints are used to represent the annotation forms of the sample segmentation labels; obtaining general knowledge hints; fusing the general knowledge hints, the data distribution hints and the annotation sparse hints to obtain annotation information; receiving server-side assigned aggregated global parameters; the aggregated global parameters are aggregated by the server side based on the model parameters of medical image segmentation models sent by multiple clients; initializing the model parameters of the medical image segmentation model according to the aggregated global parameters; inputting the sample medical images and the annotation information into the medical image segmentation model for image segmentation to obtain predicted segmentation labels; adjusting the medical image segmentation model according to the predicted segmentation labels and the sample segmentation labels to obtain a target medical image segmentation model; obtaining a target medical image and performing medical image segmentation on the target medical image by the target medical image segmentation model; the predicted segmentation labels comprise first predicted segmentation sub-labels and second predicted segmentation sub-labels, and the medical image segmentation model comprises a global shared module, a local main decoder and a local auxiliary decoder; the inputting of the sample medical images, the annotation information into the medical image segmentation model for image segmentation to obtain predicted segmentation labels comprises: extracting context features of the sample medical images and the annotation information by the global shared module to obtain context injection features; performing image segmentation on the context injection features by the local main decoder to obtain the first predicted segmentation sub-labels; and performing image segmentation on the context injection features by the local auxiliary decoder to obtain the second predicted segmentation sub-labels; the adjusting of the medical image segmentation model according to the predicted segmentation labels and the sample segmentation labels to obtain a target medical image segmentation model comprises: generating pseudo predicted labels according to the first predicted segmentation sub-labels and the second predicted segmentation sub-labels; adjusting the medical image segmentation model according to the pseudo predicted labels, the first predicted segmentation sub-labels, the second predicted segmentation sub-labels and the sample segmentation labels to obtain the target medical image segmentation model.

2. The medical image segmentation method according to claim 1, characterized in that, the adjusting of the medical image segmentation model according to the pseudo predicted labels, the first predicted segmentation sub-labels, the second predicted segmentation sub-labels and the sample segmentation labels to obtain the target medical image segmentation model comprises: performing first loss calculation according to the first predicted segmentation sub-labels, the second predicted segmentation sub-labels and the sample segmentation labels to obtain first loss data; performing second loss calculation according to the pseudo prediction label, the first prediction segmentation sub-label and the second prediction segmentation sub-label to obtain second loss data; adjusting the medical image segmentation model according to the first loss data and the second loss data to obtain the target medical image segmentation model.

3. The medical image segmentation method according to claim 2, characterized in that, The second loss data is obtained by performing second loss calculation according to the pseudo prediction label, the first prediction segmentation sub-label and the second prediction segmentation sub-label, and the second loss data includes: performing first sub-loss calculation according to the pseudo prediction label and the first prediction segmentation sub-label to obtain first sub-loss data; performing second sub-loss calculation according to the pseudo prediction label and the second prediction segmentation sub-label to obtain second sub-loss data; performing loss fusion according to the first sub-loss data and the second sub-loss data to obtain the second loss data.

4. The medical image segmentation method according to claim 1, characterized in that, The global shared module includes an encoder and a double attention fusion module, and the context feature extraction of the sample medical image and the annotation information through the global shared module includes: performing image encoding on the sample medical image through the encoder to obtain an encoded feature; performing double attention fusion feature extraction on the encoded feature and the annotation information through the double attention fusion module to obtain the context injection feature.

5. The medical image segmentation apparatus characterized by comprising: The device is applied to a client and includes: an image acquisition module configured to acquire a sample medical image set, the sample medical image set including a sample medical image, the sample medical image having a sample segmentation label, the sample segmentation label indicating a position of a region of interest of the sample medical image; a data distribution hint acquisition module configured to obtain a data distribution hint based on a pixel value of each sample medical image in the sample medical image set; an annotation sparsity hint acquisition module configured to acquire the sample segmentation label and determine an annotation sparsity hint, the annotation sparsity hint being used to represent a labeling form of the sample segmentation label; a general knowledge hint acquisition module configured to acquire a general knowledge hint; a hint fusion module configured to fuse the general knowledge hint, the data distribution hint and the annotation sparsity hint to obtain annotation information; a receiving module configured to receive an aggregated global parameter distributed by a server end, the aggregated global parameter being aggregated by the server end based on model parameters of medical image segmentation models sent by multiple clients; an initialization module configured to initialize model parameters of the medical image segmentation model according to the aggregated global parameter; an input module configured to input the sample medical image and the annotation information into the medical image segmentation model to perform image segmentation and obtain a prediction segmentation label; an adjustment module configured to adjust the medical image segmentation model according to the prediction segmentation label and the sample segmentation label to obtain a target medical image segmentation model; a medical image segmentation module configured to acquire a target medical image and perform medical image segmentation on the target medical image through the target medical image segmentation model. The prediction segmentation label comprises a first prediction segmentation sub-label and a second prediction segmentation sub-label, and the medical image segmentation model comprises a global shared module, a local main decoder, and a local auxiliary decoder; The inputting of the sample medical image and the annotation information into the medical image segmentation model for image segmentation to obtain a prediction segmentation label comprises: Contextual feature extraction of the sample medical image and the annotation information by the global shared module to obtain context injection features; image segmentation of the context injection features by the local main decoder to obtain the first prediction segmentation sub-label; and image segmentation of the context injection features by the local auxiliary decoder to obtain the second prediction segmentation sub-label; The adjustment of the medical image segmentation model according to the prediction segmentation label and the sample segmentation label to obtain a target medical image segmentation model comprises: Generation of a pseudo prediction label according to the first prediction segmentation sub-label and the second prediction segmentation sub-label; Adjustment of the medical image segmentation model according to the pseudo prediction label, the first prediction segmentation sub-label, the second prediction segmentation sub-label, and the sample segmentation label to obtain the target medical image segmentation model.

6. An electronic device, characterized by The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the medical image segmentation method of any one of claims 1 to 4 when executing the computer program.

7. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 6. The computer program is executed by the processor to implement the medical image segmentation method of any one of claims 1 to 4.

Images