Model training method, image segmentation method, device and electronic equipment
By using the neural network model training method of gradient similarity determination parameter aggregation in multimodal image federated learning, the problem of poor convergence caused by large client training burden and data heterogeneity is solved, and efficient image segmentation model training is achieved to ensure that The personalization and versatility of the model.
Patent Information
- Application Number
- CN202411190668.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-28
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-08-28
AI Technical Summary
The existing multimodal image federated learning method has a high burden on the training of the client and relies more on the participation of the multimodal client to achieve performance improvement. This is not realistic in real scenarios and ignores the optimization and data of the global model. Poor convergence caused by heterogeneity.
A neural network model training method is proposed, including the first segmentation model and the second segmentation model. By obtaining the decoder parameters at the end of each iteration, determining the aggregation of sub-parameters based on gradient similarity, realizing personalized aggregation and updating of parameters , ensure common knowledge sharing of model parameters and alleviate poor convergence caused by data heterogeneity.
Without leaking data privacy, an image segmentation model that performs well on all-modal image data is obtained, and an optimal personalized image segmentation model is obtained when the client lacks a specific mode to complete the segmentation of image data. task, improves the efficiency and accuracy of federated learning.
Smart Images

Figure CN118709740B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology. Specifically, the present application relates to a model training method, an image segmentation method, a device, an electronic device, a computer-readable storage medium and a computer program product. Background Art
[0002] Multimodal imaging technology is an imaging technology that combines multiple different imaging modalities. It aims to more comprehensively understand the target object by integrating the complementary information provided by multiple imaging methods. It has applications in many fields.
[0003] Existing methods for analyzing multimodal images include Federated Learning, which aims to balance data privacy protection and data sharing computing. The core idea is to build a global model based on virtual fusion data by performing distributed model training between multiple data sources with local data without directly exchanging local data, and only by exchanging model parameters or intermediate results.
[0004] However, the federated learning method provided by related technologies places a heavy training burden on the client and relies more on the participation of multimodal clients to achieve performance improvements, which is not realistic in real scenarios. Summary of the invention
[0005] The embodiments of the present application provide a model training method, an image segmentation method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product, which can solve the above-mentioned problems of the prior art. The technical solution is as follows:
[0006] According to a first aspect of an embodiment of the present application, a training method for a neural network model is provided, wherein the neural network model includes a first segmentation model and at least one second segmentation model, wherein each first training sample of the first segmentation model includes full-modality image data of a target object, and each second training sample of the second segmentation model includes partial-modality image data of the target object;
[0007] The method includes:
[0008] Obtaining, at the end of the current round of iterative training, first decoder parameters of the first segmentation model and second decoder parameters of each of the second segmentation models;
[0009] For each second segmentation model, determining a first sub-parameter participating in aggregation in the first decoder parameter and a second sub-parameter participating in aggregation in the second decoder parameter according to the similarity between the first gradient and the second gradient at the end of each round of iteration in the most recent target round iteration; wherein the first gradient is an update gradient of a first decoder parameter of the first decoder, and the second gradient is an update gradient of a second decoder parameter of the second decoder;
[0010] For each second segmentation model, aggregating the first sub-parameter at the end of the current round of iterative training with the second sub-parameter of the second segmentation model to obtain second decoder parameters based on the next round of iterative training, and performing a round of training on the second segmentation model based on the obtained second decoder parameters and each second training sample of the second segmentation model to obtain second decoder parameters at the end of the next round of iterative training;
[0011] Aggregating the second sub-parameters of each second segmentation model at the end of the next round of iterative training and the first sub-parameters of the first segmentation model at the end of the current round of iterative training to obtain first decoder parameters based on the next round of iterative training, and performing a round of training on the first segmentation model based on the obtained first decoder parameters and each first training sample to obtain the first decoder parameters at the end of the next round of iterative training;
[0012] The first segmentation model at the end of the last round of iteration is used as the first image segmentation model, and the second segmentation model at the end of the last round of iteration is used as the second image segmentation model.
[0013] According to a second aspect of an embodiment of the present application, there is provided an image segmentation method, comprising:
[0014] Obtaining image data to be processed, wherein the image data to be processed is image data of a target modality of a target object, wherein the target modality is a full modality or a partial modality;
[0015] Inputting the image data to be processed into a pre-trained target image segmentation model to obtain the annotation information of the segmentation result of each pixel in the image data output by the image segmentation model;
[0016] Among them, the target image segmentation model is a first image segmentation model or a second image segmentation model trained according to the training method of the neural network model described in the first aspect, and includes an image segmentation model of an encoder corresponding to each modality in the target modality.
[0017] According to a third aspect of an embodiment of the present application, a training device for a neural network model is provided, wherein the neural network model includes a first segmentation model and at least one second segmentation model, wherein each first training sample of the first segmentation model includes full-modality image data of a target object, and each second training sample of the second segmentation model includes partial-modality image data of the target object;
[0018] The device comprises:
[0019] A parameter acquisition module, used to acquire first decoder parameters of the first segmentation model and second decoder parameters of each of the second segmentation models at the end of a current round of iterative training;
[0020] an aggregated sub-parameter determination module, configured to determine, for each second segmentation model, a first sub-parameter in the first decoder parameter that participates in aggregation and a second sub-parameter in the second decoder parameter that participates in aggregation according to the similarity between the first gradient and the second gradient at the end of each round of iteration in the most recent target round iteration; wherein the first gradient is an update gradient of a first decoder parameter of the first decoder, and the second gradient is an update gradient of a second decoder parameter of the second decoder;
[0021] A first parameter updating module is used for aggregating, for each second segmentation model, the first sub-parameter at the end of the current round of iterative training with the second sub-parameter of the second segmentation model to obtain second decoder parameters based on the next round of iterative training, and performing a round of training on the second segmentation model based on the obtained second decoder parameters and each second training sample of the second segmentation model to obtain the second decoder parameters at the end of the next round of iterative training;
[0022] A second parameter updating module is used to aggregate the second sub-parameters of each second segmentation model at the end of the next round of iterative training and the first sub-parameters of the first segmentation model at the end of the current round of iterative training to obtain the first decoder parameters based on the next round of iterative training, and to perform a round of training on the first segmentation model based on the obtained first decoder parameters and each first training sample to obtain the first decoder parameters at the end of the next round of iterative training;
[0023] The first segmentation model at the end of the last round of iteration is used as the first image segmentation model, and the second segmentation model at the end of the last round of iteration is used as the second image segmentation model.
[0024] According to a fourth aspect of the embodiments of the present application, there is provided an image segmentation device, including:
[0025] An image data acquisition module is used to obtain image data to be processed, wherein the image data to be processed is image data of a target modality of a target object, wherein the target modality is a full modality or a partial modality;
[0026] An image segmentation module, used for inputting the image data to be processed into a pre-trained target image segmentation model, and obtaining the annotation information of the segmentation result of each pixel in the image data output by the image segmentation model;
[0027] Among them, the target image segmentation model is a first image segmentation model or a second image segmentation model trained according to the training method of the neural network model provided in the first aspect, and includes an image segmentation model of an encoder corresponding to each modality in the target modality.
[0028] According to a fifth aspect of an embodiment of the present application, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory, and the processor executes the computer program to implement the steps of the method provided in the first aspect or the second aspect above.
[0029] According to a sixth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method provided in the first aspect or the second aspect are implemented.
[0030] According to a seventh aspect of an embodiment of the present application, a computer program product is provided, including a computer program, which implements the steps of the method provided in the first aspect or the second aspect when the computer program is executed by a processor.
[0031] The beneficial effects of the technical solution provided by the embodiment of the present application are:
[0032] The neural network model of the embodiment of the present application includes a first segmentation model and at least one second segmentation model, each training sample of the first segmentation model includes full-modality image data of the target object, and each second training sample of the second segmentation model includes image data of a specific modality of the missing target object, that is, image data of a partial modality of the target object. In each round of iterative training, the first decoder parameters of the first segmentation model and the second decoder parameters of each of the second segmentation models are obtained at the end of the current round of iterative training, and then for each second segmentation model, the first decoder parameters are determined according to the similarity between the first gradient and the second gradient at the end of each round of iteration in the most recent target round iteration. The first sub-parameter participating in the aggregation and the second sub-parameter participating in the aggregation in the second decoder parameter, the embodiment of the present application determines whether the update method of the second decoder parameter of the second segmentation model and the first decoder parameter of the first segmentation model are consistent based on the similarity of the gradient. If they are consistent, it means that the second segmentation model has a positive contribution to the update of the first segmentation model and should participate in the aggregation of the model parameters. If they are inconsistent, it means that the decoder parameter of the second segmentation model is highly sensitive to data differences, and the second segmentation model should be restricted from participating in the aggregation, thereby effectively realizing the personalized participation of the second decoder parameters of each second segmentation model in the aggregation, and determining the second sub-parameter participating in the aggregation for each second decoder parameter After the first sub-parameter participating in the aggregation in the first decoder parameter is obtained, for each second segmentation model, the first sub-parameter at the end of the current round of iterative training is aggregated with the second sub-parameter of the second segmentation model to obtain the second decoder parameter based on the next round of iterative training, and based on the obtained second decoder parameter and each second training sample of the second segmentation model, the second segmentation model is trained for one round to obtain the second decoder parameter at the end of the next round of iterative training. When updating the first decoder parameter of the first segmentation model, the embodiment of the present application uses the second sub-parameter of each second segmentation model at the end of the next round of iterative training. It can be found that the first segmentation model and The update of the second segmentation model is sequential. The first sub-parameters of each second segmentation model at the end of the current round of iterative training and the first sub-parameters of the first segmentation model are aggregated to obtain the first decoder parameters based on the next round of iterative training. This also reflects the purpose of personalized fusion of the second sub-parameters and the first sub-parameters of each second segmentation model to ensure common knowledge sharing between the first segmentation model and the second segmentation model. After that, a round of training is performed on the first segmentation model based on the obtained first decoder parameters and each first training sample to obtain the first decoder parameters at the end of the next round of iterative training. This can alleviate the problem of poor convergence of federated learning caused by data heterogeneity between clients.Without leaking data privacy, the embodiments of the present application can ultimately obtain not only a first image segmentation model that performs well on all-modal image data (that is, the first segmentation model at the end of the last round of iterations), but also an optimal personalized second image segmentation model that performs well when a specific modality is missing on the client (that is, the second segmentation model at the end of the last round of iterations), thereby completing the task of segmenting the image data of the target object. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in describing the embodiments of the present application.
[0034] Figure 1A A schematic diagram of the system architecture for implementing a neural network model training method provided in an embodiment of the present application;
[0035] Figure 1B A schematic diagram of a system architecture for implementing a neural network model training method provided in another embodiment of the present application;
[0036] Figure 2 A flowchart of a model training method provided in an embodiment of the present application;
[0037] Figure 3A The embodiments of this application are based on Figure 1A The system architecture shown is a schematic diagram of the process of training the neural network model in each round of iteration;
[0038] Figure 3B The embodiments of this application are based on Figure 1B The system architecture shown is a schematic diagram of the process of training the neural network model in each round of iteration;
[0039] Figure 4 A schematic diagram of a process for obtaining first decoder parameters based on which a first segmentation model is trained in a next round of iterations provided in an embodiment of the present application;
[0040] Figure 5 A schematic diagram of a process for updating a binary mask of a second segmentation model provided in an embodiment of the present application;
[0041] Figure 6 A schematic diagram of a processing flow of a second segmentation model provided in an embodiment of the present application;
[0042] Figure 7 A schematic diagram of a process for calibrating a first fusion feature provided in an embodiment of the present application;
[0043] Figure 8 A flowchart of an image segmentation method provided in an embodiment of the present application;
[0044] Fig. 9A A schematic diagram of an application scenario of an image segmentation system provided in an embodiment of the present application;
[0045] Fig. 9B A schematic diagram of another application scenario of an image segmentation system provided in an embodiment of the present application;
[0046] Fig. 9C A schematic diagram of an application scenario of another image segmentation system provided in an embodiment of the present application;
[0047] Fig.10 A schematic diagram of the structure of a training device for a neural network model provided in an embodiment of the present application;
[0048] Fig.11 A schematic diagram of the structure of an image segmentation device provided in an embodiment of the present application;
[0049] Fig.12 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0050] The embodiments of the present application are described below in conjunction with the drawings in the present application. It should be understood that the implementation methods described below in conjunction with the drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions of the embodiments of the present application.
[0051] It will be understood by those skilled in the art that, unless specifically stated, the singular forms "one", "an" and "the" used herein may also include plural forms. It should be further understood that the terms "including" and "comprising" used in the embodiments of the present application refer to that the corresponding features can be implemented as the presented features, information, data, steps, operations, elements and / or components, but do not exclude the implementation as other features, information, data, steps, operations, elements, components and / or combinations thereof supported by the technical field. It should be understood that when we call an element "connected" or "coupled" to another element, the one element can be directly connected or coupled to the other element, or it can refer to that the one element and the other element establish a connection relationship through an intermediate element. In addition, the "connection" or "coupling" used here may include wireless connection or wireless coupling. The term "and / or" used here indicates at least one of the items defined by the term, for example, "A and / or B" can be implemented as "A", or as "B", or as "A and B".
[0052] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.
[0053] When multimodal imaging technology is applied in the medical field, the federated learning methods for multimodal imaging mainly include the following solutions:
[0054] The modality-based normalization method (FedNorm) uses the mode normalization technology (Mode Normalization, MN) to privatize the normalization parameters and does not participate in the aggregation of modality parameters. This method is mainly proposed for CT and MRI modality images in federated multimodal liver segmentation. Its modality normalization technology is highly targeted and cannot be well generalized to other multimodal federated learning scenarios.
[0055] Based on the method of contrastive representation integration and aggregation, this method utilizes a public multimodal dataset, and the knowledge exchange between the client and the server is only completed on this public dataset. At the same time, a global-local cross-modal integration strategy is designed to aggregate the client's representation, and further intra-modal and inter-modal comparisons are proposed to standardize local training to make up for the missing modal information of single-modal clients and further improve client performance. This method only relies on a public dataset to complete the knowledge exchange between the client and the server, which will to some extent undermine the privacy protection restrictions under federated learning. At the same time, the client must not only complete training on its local private data, but also train on public data, which increases the training burden of the client.
[0056] Based on the separation network correlation adaptive method, this method first splits the client model into multiple smaller shareable blocks, and uses a dynamic multi-view graph structure to adaptively obtain the correlation between multimodal clients, thereby realizing local model training. Specifically, in each round, the server updates the current client relationship after receiving the client model parameters, and sends the aggregated relationship information to each client; at each client, the potential client relationship is considered and participates in the training update on the local data set in the form of a regularization term, thereby improving the model performance. However, in each iteration of this method, the server needs to calculate the relevant models of all blocks corresponding to each client, which requires a lot of computing resources. At the same time, the transmission of complete model parameters will also bring considerable communication overhead.
[0057] Based on the semi-supervised multimodal aggregation method (FedIoT), this method mainly considers the semi-supervised scenario. Under the premise that the local data of the client is not labeled, the client locally trains an autoencoder to upload it, and uses the server-side labeled auxiliary data set to train the model head (i.e. the last few layers of the model) for the final task. Finally, the client can use the model trained on the server to complete local tasks and obtain performance improvements. This method requires some clients to have aligned multimodal data, which is more stringent for the client.
[0058] The above four technologies only consider the client model performance, obtain a personalized model for each client, and ignore the optimization of the global model. At the same time, the correlation adaptation method based on the separation network and the multimodal aggregation method based on semi-supervision are more dependent on the participation of multimodal clients to obtain performance improvement, which is not realistic in real scenarios.
[0059] In addition, the related art also provides a method based on a federated modality-specific encoder and a multimodal anchor, which uses a federated encoder dedicated to each modality, followed by multimodal fusion decoding on the server side. At the same time, the client uses a personalized decoder to achieve personalization. In addition, the multi-anchor multimodal feature representation extracted from the server will be distributed to the client, and the modality-specific features will be locally adaptively calibrated through cross-attention. However, this method only considers the aggregation of modality-specific encoders, and the multimodal fusion decoder adopts a completely personalized strategy, which ignores the knowledge sharing of data distribution in the decoder between different clients, and only achieves suboptimal results.
[0060] The model training method, image segmentation method, device, electronic device, computer-readable storage medium and computer program product provided in this application are intended to solve the above technical problems in the prior art.
[0061] The following describes several exemplary embodiments to illustrate the technical solutions of the embodiments of the present application and the technical effects produced by the technical solutions of the present application. It should be noted that the following embodiments can refer to, draw on or combine with each other, and the same terms, similar features and similar implementation steps in different embodiments will not be described repeatedly.
[0062] Figure 1A A schematic diagram of the system architecture for implementing a neural network model training method provided in an embodiment of the present application is shown in the figure. The embodiment of the present application is aimed at a federated learning scenario in which the data modalities of multiple participants are incomplete and the data distribution is different. The system includes a server 1011 and multiple clients 1012.
[0063] The server side 1011 locally deploys a first segmentation model, and uses local training samples to train the first segmentation model. Each client 1012 locally deploys a second segmentation model, and uses local training samples to train the local second segmentation model. The server side and local training samples of each client are not interoperable.
[0064] The training samples used by the server 1011 to train the first segmentation model include full-modality image data of the target object, while the training samples used by each client to train the second segmentation model include partial-modality image data of the target object.
[0065] It should be understood that when the embodiments of the present application are applied to the medical field, the target object of the embodiments of the present application refers to a part located in the body of an organism, which may be a human body or other non-human animals, such as an orangutan, a fish, a parrot, etc. Taking the human body as an example, the part may be the brain, lungs, liver, etc. In order to facilitate the understanding of the technical solution provided by the embodiments of the present application, the following description will mainly take the implementation process when applied to the medical field as an example.
[0066] It should be understood that full-modality image data is relative to partial-modality image data. Full-modality image data refers to image data of all imaging modalities involved in all training samples, while partial-modality image data refers to image data of partial imaging modalities involved in all training samples. For example, if the imaging modalities involved in the training samples of all clients include four types: T1-weighted (T1), enhanced T1-weighted (T1e), T2-weighted (T2), and fluid attenuated inversion recovery (FLAIR), full-modality image data refers to images of four imaging modalities, namely, T1, T1e, T2, and FLAIR, while partial-modality image data refers to images of some imaging modalities among T1, T1e, T2, and FLAIR, such as images of two imaging modalities, namely, T1 and T1e. As a special case, some imaging modality image data also include full modality image data, that is, some training samples of the second segmentation model in the embodiment of the present application can also use full modality image data, but it should be understood that it is not allowed for all training samples of the second segmentation model to use full modality image data.
[0067] It should be noted that the training samples of each embodiment of the present application are heterogeneously distributed, that is, all training samples are not independent and identically distributed among various clients, which includes both specific acquisition settings (such as equipment and imaging protocols) of different image acquisition units (generally hospitals) and heterogeneous incomplete modalities (i.e., partial modalities).
[0068] The first segmentation model and the second segmentation model of the embodiment of the present application both include an encoder and a decoder. To distinguish them, the encoder in the first segmentation model is referred to as the first encoder, and the decoder is referred to as the first decoder; the encoder in the second segmentation model is referred to as the second encoder, and the decoder is referred to as the second decoder. Since the training samples of the first segmentation model use full-modal image data, the first segmentation model includes a first encoder corresponding to each modality in the full-modal image, such as Figure 1A As shown, the first encoder T1 represents the first encoder corresponding to the T1 modality, and the training samples of the second segmentation model adopt partial modality image data. Therefore, a second segmentation model only includes the second encoder corresponding to the modality involved in the partial modality image data.
[0069] For the convenience of description, Figure 1A There is only one second encoder in each second segmentation model, but it can be understood that if part of the modality image data of a second segmentation model involves more than one modality, then the number of second encoders in a second segmentation model is also consistent with the number of modalities involved in its training samples. It can be further known that for any second segmentation model, each training sample of the second segmentation model involves the same modality, and the modality involved in one part of the training samples cannot be different from the modality involved in another part of the training samples (for example, one part of the training samples involves T1 and T1e modalities, while another part of the training samples involves T2 modality).
[0070] exist Figure 1A In the system architecture shown, the server 1011 and each client 1012 are respectively the execution entities for training the local model, and each execution entity is independent of each other. When the server 1011 and each client 1012 are trained by federated learning, in each round of iteration, the server 1011 sends the model parameters of the first segmentation model at the end of the iterative training to each client 1012, so that the second segmentation model trained by the client 1012 can update the model parameters of the second segmentation model according to the model parameters of the first segmentation model, so as to perform the next round of iterative training according to the updated model parameters and the corresponding training samples, and each client 1012 also sends the model parameters of the second segmentation model trained by itself at the end of each round of iteration to the server, so that the server updates the model parameters of the first segmentation model according to the model parameters of each second segmentation model, so as to perform the next round of iterative training according to the updated model parameters and the corresponding training samples. The model parameters of the embodiment of the present application may include encoder parameters and decoder parameters in the corresponding model.
[0071] See also Figure 1B , which exemplarily shows a schematic diagram of the system architecture of another embodiment of the present application. As shown in the figure, the system includes a first server end 1021, a second server end 1022 and multiple clients 1023. The local training samples of the first server end 1021 and each second server end 1022 are not interoperable. The local training samples of the first server end all use full-modal images, and the local training samples of each client 1023 all use partial-modal images. The definitions of full-modal images and partial-modal images in the embodiments of the present application can refer to the above embodiments, and the embodiments of the present application will not be repeated. Accordingly, the first server end 1021 also locally includes a first segmentation model, and the structure of the first segmentation model can refer to the above Figure 1A In the embodiment shown in FIG. 1 , the client 1023 also includes a second segmentation model locally. The structure of the second segmentation model can be referred to in Figure 1AThe second segmentation model in the illustrated embodiment is not specifically limited in the embodiment of the present application.
[0072] and Figure 1A The difference from the embodiment shown is that in the embodiment of the present application, the model parameters of the first segmentation model of the first server end 1021 and the second segmentation model of each client end 1023 at the beginning of each round of iteration are provided by the second server 1022, and the model parameters of the first segmentation model of the first server end 1021 and the model parameters of each client end 1023 at the end of each round of iteration need to be sent to the second server 1022, and the second server end 1022 updates the model parameters of each model and returns the updated model parameters to the first server end 1021 and each client end 1023. The first server end 1021 and each client end 1023 perform the next round of iterative training based on the updated model parameters obtained from the second server end 1022.
[0073] Regarding the above Figure 1A and Figure 1B The provided system structure, the embodiment of the present application takes into account the heterogeneity problems between and within modalities. In order to alleviate the problem of poor convergence caused by data heterogeneity between each end, the embodiment of the present application proposes different parameter aggregation strategies for the encoder and decoder when updating the model parameters of each model: on the one hand, the encoders of different clients and server-side modality-specific encoders are aggregated in a manner of aggregating the encoders of the corresponding modality; on the other hand, a sparse aggregation strategy guided by gradient differences is adopted for the decoder. During the training process, it is adaptively determined which parameters of the client need to participate in the aggregation while maintaining the sharing of universal features between modalities.
[0074] The present application provides a model training method which can be applied to Figure 1A or Figure 1B In the system provided, Figure 2 As shown, the method includes:
[0075] S101. Obtain first decoder parameters of the first segmentation model and second decoder parameters of each of the second segmentation models at the end of a current round of iterative training.
[0076] At the end of each round of iterative training, the embodiment of the present application obtains the decoder parameters of the first decoder in the first segmentation model, that is, the first decoder parameters, and also obtains the decoder parameters of the second decoder of each second segmentation model, that is, the second decoder parameters.
[0077] It is understandable that the encoder parameters and decoder parameters of each segmentation model at the end of the first round of iterative training are obtained by training the initial decoder parameters of the model using the corresponding training samples. The embodiment of the present application sets the initial encoder parameters and decoder adoption numbers of each model in the initialization stage.
[0078] S102. For each second segmentation model, determine a first sub-parameter in the first decoder parameter that participates in aggregation and a second sub-parameter in the second decoder parameter that participates in aggregation according to the similarity between the first gradient and the second gradient at the end of each round of iteration in the most recent target round iteration.
[0079] The first gradient of the embodiment of the present application is the update gradient of the first decoder parameter of the first decoder, and the second gradient is the update gradient of the second decoder parameter of the second decoder. It should be understood that the update gradient of the decoder parameter at the end of the tth (t is a positive integer) round of iteration is used to describe the degree of change of the decoder parameter at the end of the tth round of iteration compared to the end of the t-1th round of iteration. The larger the update gradient, the greater the degree of change of the decoder parameter at the end of two adjacent iterations.
[0080] For each second segmentation model, at the end of each iterative training of the second segmentation model, the similarity between the first gradient and the second gradient of the second segmentation model at the end of each round of iteration in the most recent target round iteration is obtained. The embodiment of the present application does not limit the number of the most recent target round iteration, for example, it can be 5 rounds. It should be understood that when the number of iterations has not reached the number of target round iterations, the first sub-parameter participating in the aggregation in the first decoder parameter and the second sub-parameter participating in the aggregation in the second decoder adoption number both use the initial value.
[0081] Since different second segmentation models often have different second gradients at the end of each round of iteration, the similarity between the first gradient and the second gradient of each second segmentation model at the end of each round of iteration is also different, which in turn leads to differences in the first sub-parameters and the second sub-parameters involved in the aggregation for different second segmentation models.
[0082] The embodiment of the present application calculates the similarity between the second gradient of the second segmentation model and the first gradient. If the similarity between the two is high, it means that the update direction of the second decoder parameters of the second segmentation model is consistent with the first decoder parameters of the first segmentation model, and the second decoder parameters of the second segmentation model make a positive contribution to the update of the first decoder parameters of the first segmentation model. Therefore, the second decoder parameters should participate in the aggregation of decoder parameters. On the contrary, if the similarity between the second gradient and the first gradient is continuously low, it means that the current second decoder parameters are highly sensitive to the data difference between the second segmentation model and the first segmentation model, and the second decoder parameters should be restricted from participating in the aggregation of decoder parameters.
[0083] It should be understood that when a decoder performs a decoding operation, it is usually not completed by only one decoding unit, and usually multiple cascaded decoding units are required to work together to obtain the final output. The embodiment of the present application does not limit the type of decoding units contained in the decoder. For example, it may be a decoding layer. The decoder may be composed of multiple serially connected decoding layers. Each decoding layer inputs the output of the previous layer and obtains the output of the current decoding layer through the decoding operation. Therefore, the decoder parameters of a decoder also include the parameters of each decoding unit inside the decoder.
[0084] It should be noted that when using Figure 1A In the system architecture shown, step S102 can be executed by the server 1011, and the server 1011 sends the first information to each client 1012, wherein the first information includes the first sub-parameter participating in the aggregation in the first decoder parameter and the second sub-parameter participating in the aggregation in the second decoder parameter corresponding to the corresponding client 1012.
[0085] When using Figure 1B In the system architecture shown, step S102 may be executed by the second server 1022 .
[0086] S103. For each second segmentation model, aggregate the first sub-parameter at the end of the current round of iterative training with the second sub-parameter of the second segmentation model to obtain second decoder parameters based on the next round of iterative training, and perform a round of training on the second segmentation model based on the obtained second decoder parameters and each second training sample of the second segmentation model to obtain the second decoder parameters at the end of the next round of iterative training.
[0087] In the embodiment of the present application, the first decoder parameter and the second decoder parameter need to be aggregated. In order to ensure the consistency of feature scale, the model structure of the first decoder and each second decoder in the embodiment of the present application is the same. Therefore, when the first decoder parameter and a second decoder parameter are aggregated, the decoding unit corresponding to the first sub-parameter participating in the aggregation in the first decoder parameter is completely different from the decoding unit corresponding to the second sub-parameter participating in the aggregation in the second decoder parameter, and the set of the two is all the decoding units in the decoder. For example, if the decoder includes 5 decoding layers, namely decoding layers 1 to 5, each decoding layer corresponds to a sub-parameter, then if the decoding layers corresponding to the first sub-parameter participating in the aggregation in the first decoder parameter are decoding layers 1 and 2, then the decoding layers corresponding to the second sub-parameter participating in the aggregation in the second decoder parameter are decoding layers 3 to 5.
[0088] For the second segmentation model, the embodiment of the present application aggregates the first sub-parameters at the end of the current round of iterative training with the second sub-parameters of the second segmentation model to obtain the second decoder parameters based on the next round of iterative training, and the second decoder parameters based on the next round of iterative training include the first sub-parameters of the first decoder at the end of the current round of iterative training, and the second sub-parameters of the second decoder at the end of the current round of iterative training. Taking the decoder as an example including parameters of 5 decoding layers, for a second segmentation model, if it is determined that the first sub-parameters participating in the aggregation in the first decoder parameters are the parameters of decoding layers 1 and 2, and the second sub-parameters participating in the aggregation in the second decoder parameters are the parameters of decoding layers 3 to 5, then the second decoder parameters based on the next round of iterative training of the second segmentation model include the parameters of decoding layers 1 and 2 in the first decoder at the end of the current round of iterative training and the parameters of decoding layers 3 to 5 in the second decoder.
[0089] It should be noted that the embodiment of the present application uses a network structure including multiple decoding layers in the decoder to illustrate the relationship between the decoder parameters and the first sub-parameters and the second sub-parameters as an example. In actual applications, the decoder may also include other structures or finer-grained structures. When other structures or finer-grained structures are adopted, the first sub-parameter and the second sub-parameter correspond to the above-mentioned other structures or finer-grained structures.
[0090] It should be noted that when the embodiments of the present application are applied as follows Figure 1AIn the system architecture shown, the server 1011 can obtain the second decoder parameters given by the next round of iterative training of each second segmentation model, and send them to the corresponding client 1012. The corresponding client performs a round of training on the second segmentation model based on the obtained second decoder parameters and each second training sample of the second segmentation model, and obtains the second decoder parameters at the end of the next round of iterative training. In some embodiments, each client 1012 can also aggregate the first sub-parameter at the end of the current round of iterative training with the second sub-parameter of the second segmentation model to obtain the second decoder parameters based on the next round of iterative training, and then use the local training samples and the second decoder parameters to perform a round of training on the second segmentation model to obtain the second decoder parameters at the end of the next round of iterative training.
[0091] When the embodiments of the present application are applied to Figure 1B In the system structure shown, the second server end 1022 aggregates the first sub-parameter at the end of the current round of iterative training with the second sub-parameter of the second segmentation model for each second segmentation model, obtains the second decoder parameters based on the next round of iterative training, and sends them to the corresponding client 1023. The corresponding client 1023 performs a round of training on the second segmentation model based on the obtained second decoder parameters and each second training sample of the second segmentation model, and obtains the second decoder parameters at the end of the next round of iterative training.
[0092] S104. Aggregate the second sub-parameters of each second segmentation model at the end of the next round of iterative training and the first sub-parameters of the first segmentation model at the end of the current round of iterative training to obtain first decoder parameters on which the next round of iterative training is based, and perform a round of training on the first segmentation model based on the obtained first decoder parameters and each first training sample to obtain the first decoder parameters at the end of the next round of iterative training.
[0093] In each round of iterative training, the embodiment of the present application adopts a method of first training each second segmentation model. After the current round of iterative training of each second segmentation model is completed, the first decoder parameters at the end of the previous round of iteration are updated based on the second sub-parameters participating in aggregation of each second segmentation model at the end of the current round of iteration and the first sub-parameters participating in aggregation of the first segmentation model at the end of the previous round of iteration. The first segmentation model is trained based on the updated first decoder parameters and each first training sample to obtain the first decoder parameters when the current round of iterative training is completed.
[0094] Since the second sub-parameters participating in the aggregation of different second segmentation models are different, the first sub-parameters participating in the aggregation of the first segmentation model are also different for different second segmentation models. However, the number of first sub-parameters included in the first decoder parameters based on the next round of iterative training obtained after the final aggregation should be consistent with the number of first sub-parameters in the first decoder parameters at the end of this round of iterative training. Therefore, the embodiment of the present application can take the intersection of the first sub-parameters participating in the aggregation corresponding to each second segmentation model, determine the first sub-parameter participating in the aggregation for all second segmentation models as the first target sub-parameter, and then aggregate the second sub-parameters of each second segmentation model at the end of the next round of iterative training to obtain other sub-parameters in the first decoder parameters except the first target sub-parameter (also referred to as the second target sub-parameter), and summarize the first target sub-parameter and the second target sub-parameter to obtain the first decoder parameters based on the next round of iterative training.
[0095] For example, if the neural network model includes two second segmentation models, and each decoder includes 5 sub-parameters, if for the second segmentation model a, its second sub-parameters participating in the aggregation are sub-parameters 1 and 2, and the first sub-parameters participating in the aggregation of the first segmentation model are sub-parameters 3 to 5, and for the second segmentation model b, its second sub-parameters participating in the aggregation are sub-parameters 1 to 3, and the first sub-parameters participating in the aggregation of the first segmentation model are sub-parameters 4 and 5, then by taking the intersection of the first sub-parameters participating in the aggregation corresponding to the two second segmentation models, the first target sub-parameters are determined to be sub-parameters 4 and 5. Furthermore, since sub-parameter 1 only participates in the aggregation in the second segmentation model a, sub-parameter 3 only participates in the aggregation in the second segmentation model b, and sub-parameter 2 participates in the aggregation in both second segmentation models, the second target parameter includes sub-parameter 1 of the second segmentation model a at the end of the next round of iteration, sub-parameter 3 of the second segmentation model b at the end of the next round of iteration, and a new sub-parameter 2 after aggregating the sub-parameters 2 of the two second segmentation models at the end of the next round of iteration (for example, it can be obtained by taking the average). Therefore, the first decoder parameters based on the first segmentation model in the next round of iterative training include: sub-parameter 1 of the second segmentation model a at the end of the next round of iteration, new sub-parameter 2 after aggregating sub-parameter 2 of the two second segmentation models at the end of the next round of iteration, sub-parameter 3 of the second segmentation model b at the end of the next round of iteration, and sub-parameters 4 and 5 of the first segmentation model at the end of the current round of iterative training.
[0096] The training method of the neural network model provided in the embodiment of the present application, the neural network model includes a first segmentation model and at least one second segmentation model, each training sample of the first segmentation model includes full-modality image data of the target object, each second training sample of the second segmentation model includes image data of a specific modality of the missing target object, that is, image data of a partial modality of the target object, the first segmentation model includes a first encoder corresponding to each modality in the full modality, and a first decoder cascaded with each first encoder; the second segmentation model includes a second encoder corresponding to each modality in the partial modality, and a second decoder cascaded with each second encoder, in each round of iterative training, at the end of the current round of iterative training, The first decoder parameters of the first segmentation model and the second decoder parameters of each of the second segmentation models, and then for each second segmentation model, according to the similarity between the first gradient and the second gradient at the end of each round of iteration in the most recent target round iteration, determine the first sub-parameter participating in the aggregation in the first decoder parameter and the second sub-parameter participating in the aggregation in the second decoder parameter. The embodiment of the present application determines whether the update method of the second decoder parameter of the second segmentation model and the first decoder parameter of the first segmentation model is consistent based on the similarity of the gradient. If they are consistent, it means that the second segmentation model has a positive contribution to the update of the first segmentation model and should participate in the aggregation of the model parameters. If they are inconsistent, it means that the decoder parameters of the second segmentation model are for The data difference is highly sensitive, and the second segmentation model should be restricted from participating in the aggregation, so as to effectively realize the personalized participation of the second decoder parameters of each second segmentation model in the aggregation. After determining the second sub-parameters participating in the aggregation in each second decoder parameter and the first sub-parameters participating in the aggregation in the first decoder parameter, for each second segmentation model, the first sub-parameters at the end of the current round of iterative training are aggregated with the second sub-parameters of the second segmentation model to obtain the second decoder parameters based on the next round of iterative training. Based on the obtained second decoder parameters and each second training sample of the second segmentation model, the second segmentation model is trained for a round to obtain the second decoder parameters at the end of the next round of iterative training. In the embodiment of the present application, in the first segmentation When the first decoder parameters of the model are updated, the second sub-parameters of each second segmentation model at the end of the next round of iterative training are used. It can be found that the updates of the first segmentation model and the second segmentation model in each round of iteration of the present application are in sequence. The first sub-parameters of each second segmentation model at the end of the current round of iterative training are aggregated to obtain the first decoder parameters based on the next round of iterative training. This also reflects the purpose of personalized fusion of the second sub-parameters and the first sub-parameters of each second segmentation model, ensuring the common knowledge sharing between the first segmentation model and the second segmentation model, and then further based on the obtained first decoder parameters and each first training sample, a round of training is performed on the first segmentation model.Obtaining the first decoder parameters at the end of the next round of iterative training can alleviate the problem of poor convergence of federated learning caused by data heterogeneity between clients. Without leaking data privacy, the embodiment of the present application can not only obtain a first image segmentation model that performs well on all-modal image data (that is, the first segmentation model at the end of the last round of iteration), but also obtain an optimal personalized second image segmentation model that performs well when the client lacks a specific modality (that is, the second segmentation model at the end of the last round of iteration), completing the image data segmentation task of the target object. ,
[0097] The embodiment of the present application can be applied to a federated learning scenario, where a first segmentation model is deployed in a server, and a second segmentation model is deployed in a client, and the method is executed by the server, wherein obtaining a second decoder parameter of each second segmentation model at the end of a current round of iterative training includes:
[0098] Receiving model parameters of each second segmentation model sent by each of the clients, wherein the model parameters include second decoder parameters of the second segmentation model at the end of the current round of iterative training;
[0099] For each second segmentation model, after obtaining the second decoder parameters based on which the next round of iterative training is based, the method further includes:
[0100] The model parameters based on which the next round of iterative training of the second segmentation model is to be trained are sent to the corresponding client, so that the client performs a round of training on the second segmentation model based on the model parameters and the corresponding second training samples, and obtains the model parameters of the second segmentation model at the end of the next round of training.
[0101] See also Figure 3A , which exemplarily shows that the embodiment of the present application is based on Figure 1A The system architecture shown in the figure is a schematic diagram of the process of training the neural network model in each round of iteration, as shown in the figure, including:
[0102] S2101, the server receives first model parameters sent by each client, where the first model parameters include second decoder parameters of the second segmentation model deployed by the corresponding client at the end of the current round of iterative training;
[0103] S2102: The server determines, for each second segmentation model, a first sub-parameter involved in aggregation in the first decoder parameter and a second sub-parameter involved in aggregation in the second decoder parameter according to the similarity between the first gradient and the second gradient at the end of each round of iteration in the most recent target round iteration;
[0104] S2103. For each second segmentation model, the server aggregates the first sub-parameter at the end of the current round of iterative training with the second sub-parameter of the second segmentation model to obtain a second decoder parameter based on the next round of iterative training, and sends the second model parameters to each client, where the second model parameters include the second decoder parameter based on the next round of iterative training of the second segmentation model deployed by the corresponding client;
[0105] S2104, the client receives the second model parameters, performs a round of training on the second segmentation model based on the obtained second decoder parameters and each second training sample of the second segmentation model, obtains the second decoder parameters at the end of the next round of iterative training, and sends the third model parameters to the server, where the third model parameters include the second decoder parameters at the end of the next round of iterative training;
[0106] S2105. The server receives the third model parameters, aggregates the second sub-parameters of each second segmentation model at the end of the next round of iterative training and the first sub-parameters of the first segmentation model at the end of the current round of iterative training, and obtains the first decoder parameters based on the next round of iterative training;
[0107] S2106: The server performs a round of training on the first segmentation model based on the obtained first decoder parameters and each first training sample, and obtains the first decoder parameters at the end of the next round of iterative training.
[0108] See also Figure 3B , which exemplarily shows that the embodiment of the present application is based on Figure 1B The system architecture shown in the figure is a schematic diagram of the process of training the neural network model in each round of iteration, as shown in the figure, including:
[0109] S2201, the second server side obtains the first decoder parameters of the first segmentation model and the second decoder parameters of each of the second segmentation models at the end of the current round of iterative training;
[0110] S2202: The second server determines, for each second segmentation model, a first sub-parameter involved in aggregation in the first decoder parameter and a second sub-parameter involved in aggregation in the second decoder parameter according to the similarity between the first gradient and the second gradient at the end of each round of iteration in the most recent target round iteration;
[0111] S2203: For each second segmentation model, the second server aggregates the first sub-parameter at the end of the current round of iterative training with the second sub-parameter of the second segmentation model to obtain a second decoder parameter based on which a next round of iterative training is to be conducted, and sends a fourth model parameter to each client, where the fourth model parameter includes the second decoder parameter based on which a next round of iterative training of the second segmentation model deployed by the corresponding client is to be conducted;
[0112] S2204: The client receives the fourth model parameters, performs a round of training on the second segmentation model based on the obtained second decoder parameters and each second training sample of the second segmentation model, obtains the second decoder parameters at the end of the next round of iterative training, and sends the fifth model parameters to the second server, where the fifth model parameters include the second decoder parameters at the end of the next round of iterative training of the second segmentation model deployed by the corresponding client;
[0113] S2205. The second server receives the fifth model parameters, obtains the first decoder parameters based on the next round of iterative training by aggregating the second sub-parameters of each second segmentation model at the end of the next round of iterative training and the first sub-parameters of the first segmentation model at the end of the current round of iterative training, and sends the sixth model parameters to the first server, where the sixth model parameters include the first decoder parameters based on the next round of iterative training of the first segmentation model;
[0114] S2206. The first server receives the sixth model parameters, and performs a round of training on the first segmentation model based on the obtained first decoder parameters and each first training sample, to obtain the first decoder parameters at the end of the next round of iterative training.
[0115] Based on the above embodiments, as an optional embodiment, the network structure of the decoders of each segmentation model in the neural network model of the embodiment of the present application is the same, and the decoder of each segmentation model includes a first number of filters.
[0116] The embodiment of the present application does not specifically limit the connection relationship of each filter in the decoder. For example, all filters may be connected in series, or multiple filter clusters may be connected in series, and each filter in each filter cluster is connected in parallel. The embodiment of the present application does not specifically limit it. It is understandable that the decoder parameters of each decoder in the embodiment of the present application include the parameters of each filter in the corresponding decoder.
[0117] For each second segmentation model, determining a first sub-parameter participating in aggregation in the first decoder parameter and a second sub-parameter participating in aggregation in the second decoder parameter of the second segmentation model according to the similarity between the first gradient and the second gradient at the end of each round of iteration in the most recent target round iteration, including:
[0118] Determine, according to the similarity between the first gradient and the second gradient at the end of each round of iteration in the most recent target round iteration, a second filter participating in aggregation among the filters of the second decoder of the second segmentation model, and determine the parameters of each second filter as a second sub-parameter participating in aggregation among the parameters of the second decoder of the second segmentation model;
[0119] According to each of the second filters, the first filter participating in the aggregation among the filters of the first decoder is determined, and the parameters of each first filter are determined as the first sub-parameters participating in the aggregation among the parameters of the first decoder, wherein the first filter is a filter other than the filters corresponding to the second filters among the filters of the first decoder.
[0120] That is to say, the embodiment of the present application first determines the second filter participating in the aggregation among the filters of the second decoder of the second segmentation model according to the similarity between the first gradient and the second gradient at the end of each round of iteration in the most recent target round iteration, and then determines the parameters of each second filter as the second sub-parameter participating in the aggregation among the second decoder parameters of the second segmentation model. Since the model structures of each decoder are the same, the filter in the first decoder except the filter corresponding to the second filter is used as the first filter, and the parameters of each first filter are determined as the first sub-parameter participating in the aggregation among the first decoder parameters. For example, if the decoder includes 10 filters, if it is determined that the second filter participating in the aggregation in a second decoder is filter 1~3, then the first filter participating in the aggregation in the first decoder is filter 4~10.
[0121] The embodiments of the present application use the parameters of the filter as the most basic unit participating in aggregation, mainly for two reasons: first, the filters in the convolutional network usually learn to detect specific features in the input data. By personalizing at the filter level, the embodiments of the present application can maintain the integrity of these learned features, thereby achieving more consistent and reliable personalization; secondly, the embodiments of the present application need to transmit information indicating the first sub-parameter and the second sub-parameter participating in the aggregation between the server and the client. By using the filter as the unit, the data transmission load between the client and the server can be significantly reduced. In actual application, only one byte is needed to indicate whether the parameters of a filter participate in aggregation.
[0122] Based on the above embodiments, as an optional embodiment, the present application sets a corresponding binary mask for each second decoder. In some embodiments, the binary mask is used to indicate whether each filter in the corresponding second decoder participates in aggregation. In some embodiments, the binary mask is used for the parameters of the filters participating in aggregation in the corresponding second decoder (ie, the second sub-parameters).
[0123] The embodiment of the present application obtains a binary mask by numbering the filters in each decoder according to the same sorting logic, and at the same time, using a preset binary value, such as 0 or 1, to indicate whether each filter in the corresponding second decoder participates in aggregation. For example, the binary mask [0,1,1,1,0,0] indicates whether each filter in the second decoder with 6 filters participates in aggregation, where 0 represents not participating in aggregation and 1 represents participating in aggregation, so the binary mask indicates that the parameters of filters 2 to 4 participate in aggregation, while the parameters of filters 1, 5 and 6 do not participate in aggregation.
[0124] Since the parameters of all filters in each second decoder participate in the aggregation in the first round of iterations, each binary mask indicates that all filters participate in the aggregation.
[0125] It should be understood that, since the network structure of the first decoder and each second decoder is the same, the similarity between the first gradient and the second gradient at the end of each round of iteration in the embodiment of the present application includes the similarity between the first sub-gradient and the second sub-gradient of each corresponding filter in the first segmentation model and the second segmentation model. For example, if each decoder of the present application includes 3 filters, which are represented by the serial numbers as filters 1 to 3, then for a second decoder, the similarity between the second gradient of the second decoder and the first gradient of the first decoder in each round of iteration includes the following information:
[0126] The similarity between the first sub-gradient of filter 1 in the first decoder at the end of this iteration and the second sub-gradient of filter 1 in the second decoder at the end of this iteration;
[0127] Similarity between the first sub-gradient of filter 2 in the first decoder at the end of this iteration and the second sub-gradient of filter 2 in the second decoder at the end of this iteration;
[0128] The similarity between the first sub-gradient of filter 3 in the first decoder at the end of this iteration and the second sub-gradient of filter 3 in the second decoder at the end of this iteration;
[0129] It should be understood that the first sub-gradient is the update gradient of the filter in the first decoder, the first gradient is the set of first sub-gradients corresponding to each filter in the first decoder, and the second sub-gradient is the update gradient of the filter in the second decoder, and the second gradient is the set of second sub-gradients corresponding to each filter in the second decoder.
[0130] For each second segmentation model, the present application determines a first sub-parameter participating in aggregation in the first decoder parameter and a second sub-parameter participating in aggregation in the second decoder parameter according to the similarity between the first gradient and the second gradient at the end of each round of iteration in the most recent target round iteration, including:
[0131] If the similarity between the first gradient and the second gradient at the end of each round of iteration in the most recent target round iteration is less than a preset threshold, the binary mask of the second segmentation model at the end of this round of iteration is updated to obtain a binary mask based on which the next round of iterative training is performed; wherein the binary mask is used to indicate a second sub-parameter involved in aggregation in a second decoder parameter of the second segmentation model, and the binary mask based on which the first iterative training is performed is a preconfigured binary mask;
[0132] According to the binary mask on which the next round of iterative training is based, determine the first sub-parameter participating in aggregation in the first decoder parameters and the second sub-parameter participating in aggregation in the second decoder parameters of the second segmentation model, wherein the first sub-parameter is the other parameters in the first decoder parameters except the parameters corresponding to the second sub-parameter.
[0133] Specifically, for each filter in each second segmentation model, if the similarity of the filter at the end of each round of iteration in the most recent target round iteration is less than a preset threshold, the filter is used as the target filter;
[0134] In the binary mask at the end of this round of iteration of the second segmentation model, the parameter corresponding to the target filter is cancelled from participating in aggregation, so as to obtain the binary mask based on which the next round of iterative training is based.
[0135] When the binary mask of the second decoder is updated in the embodiment of the present application, the filter whose similarity at the end of each round of iteration in the most recent target round iteration is less than the preset threshold in the second segmentation model is used as the target filter, and then the value of the target filter in the binary mask of the second decoder is updated, and the updated value indicates that the parameter of the target filter does not participate in the aggregation, thereby obtaining the binary mask of the second decoder at the end of the next round of iteration. It can be understood that the second filter in the above embodiment is also the filter other than the target filter. After obtaining the binary mask of the second decoder at the end of the next round of iteration, the filters that indicate that they do not participate in the aggregation in the binary mask are called reference filters, and accordingly, the parameters of each reference filter in the first decoder parameters are used as the first sub-parameters, thereby reflecting that the filter corresponding to the second sub-parameter participating in the aggregation in the second decoder and the filter corresponding to the first sub-parameter participating in the aggregation in the first decoder constitute all the filters in the decoder.
[0136] Based on the above embodiments, as an optional embodiment, the second sub-parameters of each second segmentation model at the end of the next round of iterative training and the first sub-parameters of the first segmentation model at the end of the current round of iterative training are aggregated to obtain the first decoder parameters based on the next round of iterative training, including:
[0137] Obtaining a first weight corresponding to the first segmentation model and a second weight corresponding to each of the second segmentation models;
[0138] fusing the second sub-parameters of each of the second segmentation models at the end of the next round of iterative training to obtain a third sub-parameter;
[0139] Based on the first weight and the second weight, the first sub-parameter of the first segmentation model at the end of the current round of iterative training and the third sub-parameter are weightedly aggregated to obtain the first decoder parameter based on the next round of iterative training.
[0140] In the embodiment of the present application, a first weight may be set for the first segmentation model, and a second weight may be set for all second segmentation models. In some embodiments, the sum of the first weight and the second weight may be a preset value, such as 1.
[0141] In the embodiment of the present application, the third sub-parameter is obtained by fusing the second sub-parameters of each second segmentation model at the end of the next round of iterative training. It should be understood that, since the filters corresponding to the second sub-parameters of different second segmentation models in the present application are different, when the second sub-parameters of each second segmentation model at the end of the next round of iterative training are fused, if a second sub-parameter of different second segmentation models corresponds to the same filter, the second sub-parameters of the next round of iterative training corresponding to the same filter of each second segmentation model are fused (for example, the mean value can be adopted) to obtain the fusion result of the second sub-parameter corresponding to the filter. If the second sub-parameter of a second segmentation model does not correspond to the same filter as the second sub-parameters of other second segmentation models, the second sub-parameter of the second segmentation model at the end of the next round of iterative training is directly used as the fusion result of the second sub-parameter corresponding to the filter. The third sub-parameter can be obtained by counting the fusion results of the second sub-parameters corresponding to all filters.
[0142] For example, if there are two second segmentation models, respectively recorded as second segmentation model 1 and second segmentation model 2, the decoder parameters in each second segmentation model include sub-parameters 1 to 3. If the second sub-parameters participating in the aggregation in the second segmentation model 1 are sub-parameters 1 and 2, and the second sub-parameters participating in the aggregation in the second segmentation model 2 are sub-parameters 1 and 3, since sub-parameter 1 participates in the aggregation in both second segmentation models, and sub-parameters 2 and 3 participate in the aggregation only in one second segmentation model respectively, the third sub-parameter may include: the mean of sub-parameter 1 of the two second segmentation models at the end of the next round of iteration, sub-parameter 2 of the second segmentation model 1 at the end of the next round of iteration, and sub-parameter 3 of the second segmentation model 2 at the end of the next round of iteration.
[0143] Thus, the first sub-parameter is weighted by the first weight to obtain the weighted first sub-parameter, the third sub-parameter is weighted by the second weight to obtain the weighted third sub-parameter, and the weighted first sub-parameter and the weighted third sub-parameter are aggregated to obtain the first decoder parameter based on the next round of iterative training.
[0144] It should be noted that if the parameters of a certain filter in all second decoders do not participate in the aggregation of certain filters that are all zero, the embodiment of the present application specifically sets the second weight corresponding to the filter to 0, thereby ensuring that the parameters of the filter will not have a negative impact on the training of the neural network model.
[0145] Based on the above embodiments, as an optional embodiment, the second sub-parameters of each second segmentation model at the end of the next round of iterative training are fused to obtain the third sub-parameters, including:
[0146] For the target second sub-parameters of the filters with the same serial number in the second sub-parameters of the second decoders, the target second sub-parameters of the second segmentation models at the end of the next round of iterative training are fused to obtain the first fusion result corresponding to the filters with the same serial number in the next round;
[0147] The first fusion results of the filters of each sequence number are fused to obtain the third sub-parameter.
[0148] Although there are differences in the second sub-parameters participating in the aggregation of different second decoders, since the network structures of all the second decoders are the same, the embodiments of the present application can fuse the second sub-parameters of each second decoder based on the filter. Specifically, since each second sub-parameter uniquely corresponds to a filter, for each filter with a serial number, the second sub-parameter of each second decoder corresponding to the filter with the serial number is used as the target second sub-parameter, and the target second sub-parameters of each second decoder at the end of the next round of iterative training are fused to obtain the first fusion result corresponding to the filter in the next round.
[0149] The embodiment of the present application does not limit the method for fusing the target second sub-parameters of each second decoder at the end of the next round of iterative training. For example, it can be a method of taking the average of the target second sub-parameters of each second decoder at the end of the next round of iterative training. It should be understood that if there is only one target second sub-parameter of a filter, that is, only one parameter of the filter in the second decoder participates in the aggregation, and the parameters of the filter in other second decoders do not participate in the aggregation, then the target second sub-parameter can be used as the first fusion result corresponding to the filter in the next round of iteration.
[0150] After obtaining the first fusion results of each filter, the third sub-parameter can be obtained by summarizing the first fusion results of each filter. That is to say, the third sub-parameter of the embodiment of the present application is a collection of the first fusion results of filters of different serial numbers.
[0151] Based on the above embodiments, as an optional embodiment, the target second sub-parameters of each of the second segmentation models at the end of the next round of iterative training are fused to obtain the first fusion result corresponding to the filter with the same sequence number in the next round of iteration, including steps S301 to S303, specifically:
[0152] S301. Obtain update gradients of various second decoder parameters at the end of the current round of iterations.
[0153] In the embodiment of the present application, the update gradient of the second decoder parameter at the end of the current round of iteration is determined according to the first sub-parameter of the first decoder at the end of the previous round of iteration and the second sub-parameter of the second decoder parameter at the end of the current round of iteration, specifically:
[0154] First, the first sub-parameter of the first decoder parameter at the end of the previous round of iterations is aggregated with the second sub-parameter of the second decoder at the end of the current round of iterations to obtain a second aggregation result corresponding to the current round of iterations. Considering that for a second decoder, the decoding units corresponding to the first sub-parameter and the second sub-parameter do not overlap, therefore, in some embodiments, the first sub-parameter of the first decoder parameter at the end of the previous round of iterations and the second sub-parameter of the second decoder at the end of the current round of iterations are aggregated, and the first sub-parameter of the first decoder parameter at the end of the previous round of iterations and the second sub-parameter of the second decoder at the end of the current round of iterations can be simply summarized to obtain the second aggregation result corresponding to the current round of iterations.
[0155] In some other embodiments, the segmentation model also locally deploys an aggregation operator, and a second aggregation result output by the aggregation operator can be obtained by inputting the first sub-parameter of the first decoder parameter at the end of the previous round of iteration and the second sub-parameter of the second decoder at the end of the current round of iteration into the locally deployed aggregation operator.
[0156] According to the difference between the second decoder parameter at the end of the current round of iteration and the second aggregation result, the update gradient of the second decoder parameter at the end of the current round of iteration is obtained. Specifically, the embodiment of the present application uses the result of subtracting the second decoder parameter at the end of the current round of iteration from the second aggregation result as the update gradient of the second decoder parameter at the end of the current round of iteration.
[0157] In some embodiments, it is relatively simple to obtain the update gradient of the first decoder parameter at the end of the current round of iterations in the embodiment of the present application. It is only necessary to calculate the difference between the first decoder parameter at the end of the current round of iterations and the end of the previous round of iterations to obtain the update gradient of the first decoder parameter at the end of the current round of iterations. Specifically, in the embodiment of the present application, the first decoder parameter at the end of the previous round of iterations is subtracted from the first decoder parameter at the end of the current round of iterations, and the result of the subtraction is the update gradient of the first decoder parameter at the end of the current round of iterations.
[0158] S302. Obtain a third weight of each second decoder according to an update gradient of each second decoder parameter at the end of a current round of iterations, wherein the third weight is inversely correlated with a magnitude of a gradient of the corresponding second decoder parameter.
[0159] In the embodiment of the present application, after determining the update gradient of each second decoder parameter at the end of the current round of iteration, the third weight of each second decoder is set based on the principle that the larger the gradient of each second decoder parameter, the smaller the third weight of the second decoder. In some embodiments, the sum of the third weights of all second decoders is a preset value, and the embodiment of the present application does not limit the size of the preset value, for example, it can be 1.
[0160] S303. For the second sub-parameters of each second decoder, the target second sub-parameters corresponding to the filter with the same serial number are weighted summed with the third weight of the corresponding second decoder to obtain the first fusion result corresponding to the filter with the same serial number in the next round.
[0161] When fusing the target second sub-parameters of filters corresponding to the same serial number, the embodiment of the present application does not adopt a simple averaging method, but adopts a third weight set based on the gradient to perform weighted summation on each target second sub-parameter to obtain the first fusion result corresponding to the filter with the same serial number in the next round, thereby retaining more second decoder parameters with similar update gradient directions to the first decoder parameters, and providing more consistent and reliable parameter fusion.
[0162] See also Figure 4 , which exemplarily shows a flow chart of obtaining the first decoder parameters based on the first segmentation model in the next round of iterative training according to an embodiment of the present application, as shown in the figure, including:
[0163] S401, obtaining a first weight corresponding to the first segmentation model and a second weight corresponding to each of the second segmentation models;
[0164] S402, obtaining the update gradient of each second decoder parameter at the end of the current round of iteration;
[0165] S403, obtaining a third weight of each second decoder according to the update gradient of each second decoder parameter at the end of the current round of iteration, where the third weight is inversely correlated with the magnitude of the gradient of the corresponding second decoder parameter;
[0166] S404: For the second sub-parameters of the second decoders, the target second sub-parameters corresponding to the filters with the same serial number are weighted summed with the third weights of the corresponding second decoders to obtain the first fusion result corresponding to the filters with the same serial number in the next round;
[0167] S405, taking the mean of the first fusion results of the filters of each sequence number as the third sub-parameter;
[0168] S406: Based on the first weight and the second weight, weighted aggregation is performed on the first sub-parameter and the third sub-parameter of the first segmentation model at the end of the current round of iterative training to obtain a first decoder parameter based on which the next round of iterative training is based.
[0169] In some embodiments, the first decoder parameter based on the t+1th round of iterative training can be obtained by the following formula: :
[0170]
[0171] in, represents the first sub-parameter of the first segmentation model at the end of t rounds of iterative training, represents the first weight, represents the second weight, represents the binary mask of the second decoder i at the end of the tth iteration, represents the third weight of the second decoder i at the end of the tth iteration, represents the second decoder parameters of the second decoder i at the end of the t+1th iteration.
[0172] See also Figure 5 , which exemplarily shows a schematic diagram of a process of updating a binary mask of a second segmentation model provided in an embodiment of the present application, as shown in the figure:
[0173] First, according to the first decoder parameter of the first decoder at the end of the t-1th round of iteration , combined with the binary mask of the second decoder i at the end of the t-1th iteration and the second decoder parameters of the second decoder at the end of t-1 iterations Performing an update to obtain a second decoder parameter based on which the second decoder is based in t rounds of iterations;
[0174] Next, based on the obtained second decoder parameters and each second training sample of the second segmentation model, the second segmentation model is trained for one round to obtain the second decoder parameters at the end of the next round of iterative training. ;
[0175] Furthermore, the second decoder parameters of all second encoders at the end of the tth round of iterative training are used , combined with the binary mask set of all second encoders at the t-1th iteration , obtain the first decoder parameters based on which the first decoder is based in the tth round of iteration, perform the tth round of training on the first segmentation model based on the obtained first decoder parameters and each first training sample, and obtain the first decoder parameters at the end of the tth round of iterative training ;
[0176] Then, on the one hand, according to the first decoder parameter of the first decoder at the end of the tth round of iteration The first decoder parameters at the end of the t-1th iteration The difference between them is used to obtain the update gradient of the first decoder at the end of the tth round of iterations. ;
[0177] On the other hand, the second decoder parameters of the second decoder i at the end of the t-1th round of iteration are , the first decoder parameters of the first decoder at the end of the t-1th round of iteration and the binary mask of the second decoder i at the end of the t-1th iteration , input aggregation operator , obtain the second aggregation result, according to the decoder parameters of client i in the local training of this round The difference between the second aggregation result and the update gradient of the second decoder i at the end of the tth iteration is obtained ;
[0178] Finally, the update of the binary mask of the second decoder i is guided by utilizing the consistency of the update gradient of the second decoder i with the update gradient of the first decoder i. Specifically, the embodiment of the present application calculates the cosine similarity of each filter between the first gradient and the second gradient =cos( ), when the second decoder i l The cosine similarity of the kth filter of the layer continued P When the round is negative, the parameters of the filter are updated without referring to the aggregation. If the first lThe cosine similarity of the kth filter of the layer continued P If the value of the round is not negative, the parameters of the filter are kept and the aggregation is continued to obtain the binary mask of the second decoder i at the end of the tth round iteration. .
[0179] Based on the above embodiments, as an optional embodiment, each training sample also includes the annotation information of the segmentation result of each pixel in the corresponding image data. The embodiment of the present application does not specifically limit the type of segmentation result, for example, it can include tumor, edema, enhanced edema, flower color and non-enhanced tumor core, etc.
[0180] In this embodiment of the application, each second segmentation model further includes:
[0181] For the second encoder of the second segmentation model, the first encoder parameters of the first encoder of the same modality at the end of the previous round of iteration are used as the second encoder parameters of the second encoder after being updated in the current round of iteration;
[0182] The second segmentation model after updating the second encoder parameters and the second decoder parameters is used as the second segmentation model in this round of iteration, and the second segmentation model in this round of iteration is trained based on the training samples of the second segmentation model until it meets the second preset condition, so as to obtain the second segmentation model at the end of this round of iteration.
[0183] It can be clearly seen from the above embodiments that the present application adopts a method of first updating the model parameters of each second segmentation model and then updating the model parameters of the first segmentation model for one round of iteration. That is to say, after the tth round of iteration is completed, the first encoder parameters of the first segmentation model completed in the tth round of iteration are first passed to the second encoder corresponding to the same mode in each second segmentation model, so that the second encoder replaces the second encoder parameters of the second encoder when the tth round of iteration is completed with the first encoder parameters completed in the tth round of iteration, thereby obtaining the second encoder parameters of the second encoder after the update in the tth round of iteration.
[0184] Furthermore, since the second training sample of the second segmentation model carries the annotation information, through supervised learning, the second segmentation model after updating the second encoder parameters and the second decoder parameters can be trained, and when the preset conditions are met, it is considered that one round of iteration is completed. The loss function of the second segmentation model of the present application can be used to calculate the difference between the predicted segmentation result and the annotation information of each pixel in the image data output by the second decoder, and when the loss value of the loss function converges, it is considered that the preset conditions are met.
[0185] For each first split model, each iteration also includes:
[0186] For each first encoder, aggregating the second encoder parameters of all second encoders of the same mode at the end of this round of iteration to obtain updated first encoder parameters of the same mode of the first encoder;
[0187] The first segmentation model after updating the first encoder parameters and the first decoder parameters is used as the first segmentation model in this round of iteration. The first segmentation model in this round of iteration is trained according to the training samples of the first segmentation model until it meets the first preset conditions, so as to obtain the first segmentation model at the end of this round of iteration.
[0188] In the embodiment of the present application, the update of the first encoder parameter of the first encoder needs to be based on the second encoder parameters of all second encoders at the end of this round of iteration. The second encoder parameters of all second encoders at the end of this round of iteration are aggregated, which can be achieved by taking the average of all second encoder parameters.
[0189] In one embodiment, the embodiment of the present application can obtain the first encoder parameter after the first encoder is updated according to the following formula:
[0190]
[0191] in, represents the updated first encoder parameter of the first encoder corresponding to the m mode, N m represents the number of second encoders corresponding to m modes in all second segmentation models, represents the second encoder parameters of the second encoder i at the end of this round of iteration.
[0192] Furthermore, since the first training sample of the first segmentation model carries the annotation information, through supervised learning, it is possible to perform training based on the first segmentation model after updating the parameters of each first encoder and the parameters of the first decoder. When the preset conditions are met, it is considered that one round of iteration is completed. The loss function of the first segmentation model of the present application can be used to calculate the difference between the predicted segmentation result and the annotation information of each pixel in the image data output by the first decoder. When the loss value of the loss function converges, it is considered that the preset conditions are met.
[0193] It should be noted that the labeling information of the embodiment of the present application can represent different segmentation results through the color of pixel points. For example, green represents edema, blue represents enhanced tumors, red represents necrosis and non-enhanced tumor cores, and black represents normal target objects. Therefore, the output results of the first image segmentation model and the second image segmentation model obtained when the training is finally completed will be image data of pixel areas with at least one color.
[0194] Based on the above embodiments, as an optional embodiment, the decoder structure of the first decoder and each second decoder in the embodiment of the present application is the same, and both include at least one decoding layer. It can be understood that each decoding layer includes at least one filter. It should be noted that the number of filters included in different decoding layers in the embodiment of the present application may be the same or different, and the embodiment of the present application does not make specific limitations.
[0195] Each second encoder in the second segmentation model of the embodiment of the present application also includes an encoding layer that corresponds one-to-one to the decoding layer of the second decoder in the second segmentation model.
[0196] In the embodiment of the present application, the second segmentation model in the current iteration process is trained according to the training sample of the second segmentation model to meet the second preset condition, including:
[0197] For each decoding layer of the second segmentation model in the current round of iteration, according to each feature anchor point corresponding to the decoding layer at the end of the previous round of iteration, the attention mechanism is used to calibrate the first fused feature corresponding to the decoding layer, and the first fused features before and after the calibration of the decoding layer are fused to obtain the second fused feature corresponding to the decoding layer;
[0198] The second fused feature corresponding to the first decoding layer is used as the output result of the first decoding layer. For non-first decoding layers, feature decoding is performed according to the second fused feature corresponding to the non-first decoding layer and the output result of the previous decoding layer to obtain the output result of the non-first decoding layer;
[0199] According to the output results of the last decoding layer, the predicted segmentation results of each pixel in the corresponding image data of the training sample are obtained, and according to the difference between the predicted segmentation results of each pixel and the annotation information, the second encoder parameters and the second decoder parameters of the second segmentation model in the current round of iteration are optimized.
[0200] See also Figure 6, which exemplarily shows a processing flow diagram of the second segmentation model provided in an embodiment of the present application. As shown in the figure, the second segmentation model includes two second encoders, each of which has 4 encoding layers. Correspondingly, the second decoder also has 4 decoding layers. For the first encoding layer, the encoding features output by the two second encoders in the encoding layer are fused to obtain the first fused feature f1 corresponding to the first encoding layer. Since the encoding layer corresponds to the decoding layer one by one, the first fused feature f1 corresponding to the first decoding layer is obtained. Further, according to each feature anchor point A1 of the first decoding layer, the first fused feature f1 is calibrated using an attention mechanism, and the first fused features before and after calibration are further fused to obtain the second fused feature l1 corresponding to the first decoding layer. Based on the same logic, the second fused features l2~l4 corresponding to each decoding layer can be obtained.
[0201] The second fused features corresponding to the first decoding layer are taken as the output results of the first decoding layer. Starting from the second decoding layer, feature decoding is performed according to the second fused features corresponding to the decoding layer and the output results corresponding to the previous decoding layer, so as to obtain the output results of the decoding layer (d2~d4 in the figure). It should be noted that before feature decoding of the second fused features corresponding to the two vector decoding layers, considering that the features will be gradually downsampled during the encoding process, it is also necessary to upsample the output results of the previous decoding layer (represented as "×2" in the figure, that is, upsampling twice) so that the feature scales of the second fused features corresponding to the two decoding layers are the same.
[0202] The output of the last decoding layer of this application can be mapped to a larger vector space through a fully connected layer, which is also called a logits vector, and then the logits vector is sent to the Softmax layer. The function of the Softmax layer is to convert each element in the logits vector (that is, the classification result of each pixel) into a probability value, which represents the possibility of each pixel becoming the output of the current time step. The converted probability values add up to 1, and each probability value is between 0 and 1.
[0203] Although the segmentation result of each pixel is unique, the embodiment of the present application believes that even the pixels of the same segmentation result in each first training sample can still be further subdivided into multiple types in detail. For example, even if the segmentation results are all pixels of a tumor, there may be differences in the appearance of the tissues corresponding to the pixels. In addition, when the sampling machines are different, for the same segmentation result, the color information of the sampled pixels may also be different.
[0204] The overall processing logic of the second segmentation model is that in the encoder stage, the image data is compressed and reduced in dimension for more efficient representation and transmission. The decoder layer gradually increases the dimension of the data through operations such as upsampling and deconvolution to restore it to a size close to the original image, that is, each decoding layer in the decoder is used to reconstruct the image data in the training sample to varying degrees. Since the output of the decoder contains the reconstruction information of each pixel in the image data, combined with the difference in the feature information of each pixel in the same segmentation result mentioned above, the embodiment of the present application configures the same number of feature anchor points for each decoding layer, and each feature anchor point corresponding to each decoding layer is obtained by clustering the decoding features output by the corresponding decoding layer for the pixels of the same segmentation result in each first training sample. In the embodiment of the present application, each feature anchor point corresponding to the decoding layer can correspond to the cluster center of the clustering cluster of the decoding features of each pixel of the same segmentation result.
[0205] In the embodiment of the present application, for each decoding layer, based on the feature anchor points of the decoding layer at the end of the previous iteration, an attention mechanism is used to calibrate the first fused feature corresponding to the decoding layer, so that the calibrated first fused feature can be closer to similar feature anchor points, thereby improving the accuracy of segmentation.
[0206] The embodiment of the present application does not specifically limit the manner in which the first fused features before and after calibration of the current decoder are fused. For example, the first fused features before and after calibration may be added at the same feature scale.
[0207] Based on the above embodiments, as an optional embodiment, the embodiment of the present application can calibrate the first fusion feature corresponding to the decoding layer using an attention mechanism according to the feature anchor point at the end of the previous iteration based on a local adaptive calibration (Local Adaptive Calibration for Cross Attention, LACCA) module, including:
[0208] According to the feature anchor point at the end of the previous iteration, the first fusion feature corresponding to the decoding layer, and the number of filters in the decoding layer, an attention mechanism is used to perform normalization processing to obtain a fourth weight;
[0209] The first fusion feature corresponding to the decoding layer is weighted by the fourth weight to obtain a calibrated first fusion feature.
[0210] See also Figure 7 , which exemplarily shows a flow chart of calibrating the first fusion feature in an embodiment of the present application. As shown in the figure, l The first fusion feature corresponding to the layer decoding layer Fl ∈ , where R represents a real number set, for the first fusion feature F l Reshape the dimension as the query in the attention operation, and at the same time, the feature anchor point at the end of the previous iteration of the layer A l , as the key and value, the attention mechanism is used for planning processing to obtain the fourth weight, and then the fourth weight is used to weight the first fusion feature corresponding to the decoding layer to obtain the calibrated first fusion feature ,Will The dimension is reshaped to match the first fusion feature F l Consistent, the same characteristic scale and F l Add together to obtain the second fusion feature corresponding to the decoding layer.
[0211] The embodiment of the present application can transpose the feature anchor point at the end of the previous round of iteration, multiply the result of the transposition processing by the first fusion feature corresponding to the decoding layer, divide the multiplication result by the square root of the number of filters in the decoding layer, normalize the result through the softmax function to obtain a fourth weight, weight the first fusion feature corresponding to the decoding layer through the fourth weight, and obtain the calibrated first fusion feature.
[0212] In some embodiments, the first l The first fused feature after calibration corresponding to the layer decoding layer :
[0213]
[0214] Among them, Attn represents the attention mechanism, Indicates l Feature anchors of layer decoding layer A l The result of the transposition process is C l Indicates l The number of filters in the layer decoding layer, F l Indicates l The first fused feature corresponding to the layer decoding layer.
[0215] The embodiment of the present application calibrates the decoding features of the partial modality of the second segmentation model of the client to the feature anchor points of the full modality of the first segmentation model of the server, which also promotes the second segmentation model to adaptively maintain common knowledge and personalized parameters.
[0216] Based on the above embodiments, as an optional embodiment, the feature anchor point of each decoding layer is updated through the following steps S501 to S504, specifically:
[0217] S501: for each first training sample, determine, according to the annotation information of the first training sample, a segmentation result corresponding to the decoding feature of each pixel point output by the decoding layer at the end of the previous iteration;
[0218] S502: for each first training sample, the decoding features of each pixel point of the same segmentation result output by the decoding layer are fused to obtain the result features of the corresponding segmentation result of the decoding layer for the first training sample;
[0219] S503, for the same segmentation result, clustering the result features of the segmentation results of the decoding layer for each first training sample according to a preset number of clusters, and determining the features of the cluster center of each cluster cluster in the current round of iteration;
[0220] S504 , for each cluster, fuse the features of the cluster center of the current iteration and the previous iteration to obtain a feature anchor point corresponding to the cluster at the decoding layer at the end of the current iteration.
[0221] The present application embodiment defines l The dimension of the decoded features of the layer decoding layer is [ hl,wl,dl,c1 ],in, hl, wl,dl is the spatial dimension. The spatial dimensions of the decoded features of different decoding layers are different (they may be downsampled to the corresponding scale on the original size). c1 is the feature dimension, and the dimension of the annotation information is [ h,w,d,k ], h , w , d is the size of the image data, k is the number of types of segmentation results, that is, the annotation information contains the segmentation results of each pixel. By downsampling (scaling) the annotation information to the size of the decoding feature of the corresponding decoding layer, it becomes [ hl,wl,dl k ], then, decode each pixel in the feature (1* c1 The corresponding segmentation result can be obtained according to the annotation information.
[0222] In the embodiment of the present application, for each first training sample, the decoding features of each pixel point of the same segmentation result output by the decoding layer are averaged to achieve fusion, and the result features of the corresponding segmentation result of the decoding layer for the training sample are obtained. After obtaining the result features of each segmentation result in all the first training samples, for the same segmentation result, the embodiment of the present application clusters the result features of the segmentation result of the decoding layer for each first training sample according to a preset number of clusters (the empirical value is 3), and multiple clusters of each segmentation result can be obtained. In the embodiment of the present application, each cluster of each segmentation result corresponds to a feature anchor point. In the embodiment of the present application, the mean of each result feature in a cluster can be used as the feature of the cluster center.
[0223] In order to avoid the possible training crash caused by the frequent jumps of cluster centers in each round of re-clustering, the embodiment of the present application regards the feature anchor point as a memory bank and updates it by exponential sliding average. For each cluster cluster, the embodiment of the present application sets weights for the features of the cluster center of the current round iteration and the previous round iteration, respectively, and sums the weighted features of the cluster center of the current round iteration and the previous round iteration to obtain a feature anchor point corresponding to the cluster cluster at the end of the current round iteration of the decoding layer.
[0224] See also Figure 8 , which exemplarily shows a flow chart of an image segmentation method provided in an embodiment of the present application, as shown in the figure, including:
[0225] S601, obtaining image data to be processed, wherein the image data to be processed is image data of a target modality of a target object, and the target modality is a full modality or a partial modality;
[0226] S602: Input the image data to be processed into a pre-trained target image segmentation model to obtain the annotation information of the segmentation result of each pixel in the image data output by the image segmentation model.
[0227] After the neural network model is trained, the embodiment of the present application obtains a first image segmentation model capable of processing image data of the target object in all modalities and a second image segmentation model capable of processing image data of the target object in partial modality.
[0228] Since the encoder in each image segmentation model corresponds one-to-one to the modality of the image data that the image model can process, if the target modality is the full modality, the image data to be processed is input into the first image segmentation model, thereby obtaining the annotation information of the segmentation result of each pixel in the image data output by the first image segmentation model. If the target modality is a partial modality, the image data to be processed is input into the second image segmentation model including the encoder corresponding to each modality in the partial modality, thereby obtaining the annotation information of the segmentation result of each pixel in the image data output by the second image segmentation model.
[0229] Fig. 9A A schematic diagram of an application scenario of an image segmentation system provided for an embodiment of the present application. As shown in the figure, there is a tertiary hospital and four community hospitals in the region. The tertiary hospital has deployed a first nuclear magnetic resonance device that can acquire four-modality (T1, T1e, T2 and FLAIR) MRI brain tumor images, while each community hospital has deployed a second nuclear magnetic resonance device that can only acquire a single-modality MRI brain tumor image due to limited conditions. For the convenience of description, the corresponding modes of the four community hospitals are T1, T1e, T2 and FLAIR, respectively.
[0230] In the embodiment of the present application, a first segmentation model is deployed in a tertiary hospital, and the training samples of the first segmentation model are full-modality (quad-modality) brain tumor images collected in the tertiary hospital. A second segmentation model is deployed in each community hospital, and the first encoder in the second segmentation model corresponds to a single modality collected by the second nuclear magnetic resonance equipment of the community hospital, and the training samples are single-modality brain tumor images collected by the community hospital.
[0231] It should be noted that the relevant data collection and processing of each embodiment of this application should be strictly in accordance with the requirements of relevant national laws and regulations when applied in examples, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.
[0232] By using the training method of the neural network model provided in the above-mentioned embodiments of the present application, the first segmentation model and each second segmentation model are trained to obtain a first image segmentation model and each second image segmentation model. Not only does the segmentation performance of each second image segmentation model become stronger, but also the segmentation performance of the first image segmentation model is significantly improved through the training samples and model parameters of the second segmentation models of each heterogeneous modality.
[0233] When a patient visits a tertiary hospital for treatment, the first nuclear magnetic resonance equipment of the tertiary hospital collects a single-modality MRI image of the patient, and then the segmentation result image can be obtained through the first image segmentation model deployed by the community hospital.
[0234] When a patient visits a community hospital for treatment, after the second nuclear magnetic resonance equipment of the community hospital collects a single-modality MRI image of the patient, the segmentation result image can be obtained through the second image segmentation model deployed by the community hospital.
[0235] In some embodiments, when a patient visits a community hospital, after the community hospital's second nuclear magnetic resonance equipment acquires a single-modality MRI image of the patient, the MRI image can also be sent to the first image segmentation model deployed in the tertiary hospital. Since the first image segmentation model has a first encoder with the single modality, the segmentation result image can also be obtained through the first image segmentation model.
[0236] Fig. 9B A schematic diagram of an application scenario of another image segmentation system provided for an embodiment of the present application, as shown in the figure, this application scenario is a remote sensing image segmentation scenario, where multiple remote sensing satellites conduct remote sensing detection of a certain area and collect remote sensing images. Due to the functional differences of remote sensing satellites, the modalities of remote sensing images collected by different remote sensing satellites are different. The modalities involved in this application may include spectral modality, planning modality, spatial modality, etc. There is a national-level geographical institute (hereinafter referred to as the institute) and three provincial-level geographical research offices (hereinafter referred to as the offices) that analyze the remote sensing images of the area to provide public services such as urban planning, environmental monitoring, climate change, and disaster assessment. Among them, the institute has the authority to obtain remote sensing images of all modalities, while each office only has the authority to obtain remote sensing images of some modalities. For the convenience of description, the modalities corresponding to the three offices are spectral modality, planning modality, and spatial modality, respectively.
[0237] In the embodiment of the present application, a first segmentation model is deployed in the research institute, and the training samples of the first segmentation model are full-modal (tri-modal) remote sensing images. A second segmentation model is deployed in each office, and the first encoder in the second segmentation model corresponds to the single modality corresponding to the office, and the training samples are remote sensing images of the single modality.
[0238] Through the training method of the neural network model provided in the above-mentioned embodiments of the present application, the first segmentation model and each second segmentation model are trained to obtain the first image segmentation model and each second image segmentation model. Not only the segmentation performance of each second image segmentation model is stronger, but also the training samples and model parameters of the second segmentation model of each heterogeneous modality also significantly improve the segmentation performance of the first image segmentation model. It should be noted that the annotation information of the training samples in the embodiments of the present application is the segmentation result of the remote sensing image to determine the different information present in the image - water bodies, vegetation, soil, buildings, etc.
[0239] Researchers at the institute can use the first image segmentation model to obtain segmentation result images for full-modal remote sensing images collected by various remote sensing satellites.
[0240] Researchers in the office can obtain segmentation result images for remote sensing images of some modalities collected by corresponding remote sensing satellites through the second image segmentation model deployed by the office.
[0241] After obtaining the remote sensing images of some modalities collected by the corresponding remote sensing satellite, the researchers in the office send the remote sensing images to the first image segmentation model deployed by the institute. Since the first image segmentation model has a first encoder corresponding to the single modality, the segmentation result image can also be obtained through the first image segmentation model.
[0242] Multimodal video segmentation refers to the process of combining multiple modal information in the video (such as images, audio, text, etc.) and using their respective characteristics and mutual associations to segment objects or regions in the video. This segmentation not only considers the pixel or feature distribution in the video frame, but also incorporates additional information such as audio clues (such as sound positioning) and text descriptions (such as video titles and subtitles) to improve the accuracy and robustness of the segmentation.
[0243] Fig. 9C A schematic diagram of an application scenario of another image segmentation system provided in an embodiment of the present application is shown in the figure. The application scenario is a media editing scenario. By segmenting video frames, extracting key video frames from the video, and generating a video summary based on the key video frames, the efficiency of generating video summaries can be significantly improved.
[0244] The embodiment of the present application involves a film and television company headquarters (hereinafter referred to as the headquarters) and a number of multimedia processing organizations with cooperative relationships (hereinafter referred to as organizations). The embodiment of the present application deploys a first segmentation model at the headquarters. The training samples of the first segmentation model are short videos in all modalities (video, audio and text). That is, the first segmentation model includes three first encoders, and the three first encoders are used to extract visual features (color, texture, shape, motion, etc. in short video frames), audio features (sound direction, loudness, frequency and other characteristics in audio signals) and text features (using natural language processing technology to extract semantic information from short video titles, subtitles or descriptions to provide contextual support for segmentation), and each organization deploys a second segmentation model. For the convenience of description, each second segmentation model in this embodiment corresponds to a modality.
[0245] Workers at the headquarters can obtain segmentation result images for full-modal short videos through the first image segmentation model.
[0246] Workers in the institution can obtain segmentation result images for short videos of some modalities by deploying the second image segmentation model.
[0247] After obtaining short videos of some modalities, the workers of the institution send the short videos to the first image segmentation model deployed by the institute. Since the first image segmentation model has a first encoder for the single modality, the segmentation result image can also be obtained through the first image segmentation model.
[0248] After obtaining the segmentation result image, workers at the headquarters or institution can generate a video summary based on the segmentation result image, such as a cover of a short video.
[0249] In order to further verify the effectiveness of the model trained by the model training method of the embodiment of the present application, the embodiment of the present application uses PyTorch to implement the model framework of the embodiment of the present application, and trains on 5 GPUs of model RTX2080Ti, where one GPU is used on the server side and the clients are evenly distributed on the remaining GPUs. RFNet is used as the first segmentation model, and its modality-specific encoder and regularized decoder are used as the second encoder and second decoder of the client, respectively. The LACCA module uses 8 attention heads, the smoothing coefficient of the exponential moving average is set to 0.999, the input cropping size is 80×80×80 voxels, the batch size of the server and the client is set to 1, and other settings are the same for the server and the client.
[0250] The embodiment of the present application adopts the commonly used Dice loss plus the cross entropy loss for medical image segmentation. The learning rate and weight decay of the Adam optimizer are set to 0.0002 and 10e-5 respectively. The network is trained for 1000 rounds. In each round of federated learning, the server and the client train one epoch. The method of data preprocessing and enhancement is not specifically limited in this application, and reference may be made to the relevant technology.
[0251] The present application embodiment is experimented on the multimodal brain tumor segmentation BraTS 2018 dataset, which contains 285 multi-contrast MRI scans with four sequences: T1, T1c, T2, and FLAIR. The goal is to segment three sub-regions of the brain tumor: the entire tumor, the tumor core, and the enhancing tumor.
[0252] In the actual problem setting, the entire training set is initially divided into nine non-overlapping subsets. This is based on the official information of BraTS2018 to ensure that all cases from the same institution are grouped into the same subset.
[0253] Among these nine subsets, one is designated as the data set of the server side, and the remaining eight subsets are the data sets of the client side. Each subset is further divided into a training set, a validation set, and a test set in a ratio of 6:2:2. The embodiment of the present application allocates different modal combinations to each client, including at most full modality, which results in two clients for each single modality, dual modality, tri-modality, and full modality. It should be noted that each client can only access the specific modality assigned to its data, while the server's data contains full modality.
[0254] The mean Dice similarity coefficient (mDSC) of the three tumor subregions was used as the evaluation metric, and statistical significance was analyzed using the Wilcoxon signed-rank test.
[0255] First, we compared the traditional encoder-decoder architecture and the modality-specific encoder architecture on the Brats2018 validation set, as well as the impact of different feature anchor strategies on the final segmentation performance. The results are shown in Table 1:
[0256]
[0257] Table 1 Comparison of the impact of different training methods on the final segmentation performance
[0258] In Table 1, E&D represents the traditional encoder-decoder architecture. 4E&D represents the modality-specific encoder architecture. F, C, T1, and T2 represent that the corresponding client has the corresponding modality, S represents the server, and Avg represents the average segmentation performance of all clients. It can be seen that in the variant of FedAvg, the performance of the federated decoder and the personalized encoder (row (b)) is better than the opposite setting (row (a)). On this basis, the modality-specific encoder of the embodiment of the present application, that is, the second encoder (row (c)), achieves significant performance improvement compared with the FedAvg series. For example, compared with row (b), the average mDSC of the client is improved by about 4%, which shows that the FL architecture is effective in the problem setting, while the federated fusion decoder (row (d)) slightly reduces the average mDSC of the client. Row (e) adopts the first decoder and the second decoder proposed in the embodiment of the present application, and the average mDSC of the client and the DSC of the server are improved. Row (f) further combines the LACCA module, but uses a single anchor point representation obtained by averaging all server data to achieve further improvements on the client. This suggests that the multimodal representation is effective for adaptive calibration of local missing modal features using sparse shared parameters.
[0259] The last row is the complete model (also called the multi-anchor representation model) provided by the embodiment of the present application, which achieves the highest average missing modality performance on the client (72.84%) and full modality performance on the server (83.83%). These results show that multi-anchors are better able to represent 3D multimodal medical data than single anchors, as expected. Row (g) gives the results of the baseline method FedMEMA referenced by the embodiment of the present application on the validation set. It can be found that the embodiment of the present application achieves significant performance improvement on this basis (an average of 2.57% higher on the client and 0.6% higher on the server), which illustrates the effectiveness of the personalization strategy based on client-server gradient difference guidance proposed by the present method.
[0260] In order to verify the segmentation effect of the present application on the Brats2018 dataset, the present application embodiment compares the present application with various baselines and the latest federated learning (FL) algorithms. The present application embodiment designs two baselines. First, the server and client models are trained locally on the private data of each site as the result obtained as baseline 1 (local models in Table 2). Second, the present application embodiment trains it on the server data according to the original method of RFNet[6] and evaluates its performance on various test sets as baseline 2 (RFNet in Table 2). The present application embodiment reproduces the classic FedAvg and several related FL methods for multimodal data or personalized models in the application setting for comparison, including FedNorm, FedMSplit, CreamFL, FedIoT, perFL, IOP-FL, FedLC, and FedMEMA. The results are shown in Table 2, where F, C, T1, and T2 indicate that the corresponding client has the corresponding modality, S indicates the server, and Avg indicates the average segmentation performance of all clients.
[0261]
[0262] Table 2 Comparison of segmentation effects of various schemes on the Brats2018 dataset
[0263] Compared with the best results among various related methods, the method proposed in the embodiment of the present application is about 1% higher than FedLC in terms of average mDSC of the client and 0.5% higher in terms of mDSC of the server. The embodiment of the present application attributes the advantages to the sparse parameter sharing strategy and the adaptive calibration through cross-attention between the local missing modality and the global full modality representation. It is worth noting that in addition to improving the performance of the client through the locally personalized second segmentation model, the embodiment of the present application also significantly improves the performance of the server by effectively utilizing the client data of heterogeneous modalities.
[0264] The present application embodiment provides a training device for a neural network model, such as Fig.10 As shown, the neural network model includes a first segmentation model and at least one second segmentation model, each first training sample of the first segmentation model includes full-modality image data of the target object, and each second training sample of the second segmentation model includes partial-modality image data of the target object; the first segmentation model includes a first encoder corresponding to each modality in the full modality, and a first decoder cascaded with each first encoder; the second segmentation model includes a second encoder corresponding to each modality in the partial modality, and a second decoder cascaded with each second encoder;
[0265] The training device may include: a parameter acquisition module 1010, an aggregation sub-parameter determination module 1020, a first parameter updating module 1030 and a second parameter updating module 1040, wherein:
[0266] A parameter acquisition module 1010 is used to acquire the first decoder parameters of the first segmentation model and the second decoder parameters of each of the second segmentation models at the end of the current round of iterative training, wherein the decoder parameters of each segmentation model at the end of the first round of iterative training are obtained by performing one round of training on the initial decoder parameters of the model using the corresponding training samples;
[0267] The aggregated sub-parameter determination module 1020 is used to determine, for each second segmentation model, a first sub-parameter involved in aggregation in the first decoder parameter and a second sub-parameter involved in aggregation in the second decoder parameter according to the similarity between the first gradient and the second gradient at the end of each round of iteration in the most recent target round iteration; wherein the first gradient is an update gradient of a first decoder parameter of the first decoder, and the second gradient is an update gradient of a second decoder parameter of the second decoder;
[0268] A first parameter updating module 1030 is configured to aggregate, for each second segmentation model, the first sub-parameter at the end of the current round of iterative training with the second sub-parameter of the second segmentation model to obtain second decoder parameters based on the next round of iterative training, and to perform a round of training on the second segmentation model based on the obtained second decoder parameters and each second training sample of the second segmentation model to obtain the second decoder parameters at the end of the next round of iterative training;
[0269] The second parameter updating module 1040 is used to aggregate the second sub-parameters of each second segmentation model at the end of the next round of iterative training and the first sub-parameters of the first segmentation model at the end of the current round of iterative training to obtain the first decoder parameters based on the next round of iterative training, and to perform a round of training on the first segmentation model based on the obtained first decoder parameters and each first training sample to obtain the first decoder parameters at the end of the next round of iterative training.
[0270] The device of the embodiment of the present application can execute the training method of the neural network model provided in the embodiment of the present application, and the implementation principle is similar. The actions performed by each module in the device of each embodiment of the present application correspond to the steps in the method of each embodiment of the present application. For the detailed functional description of each module of the device, please refer to the description in the corresponding method shown in the previous text, which will not be repeated here.
[0271] The present application embodiment provides an image segmentation device, such as Fig.11 As shown, the device includes an image data acquisition module 1110 and an image segmentation module 1120, specifically:
[0272] The image data acquisition module 1110 is used to obtain the image data to be processed, wherein the image data to be processed is the image data of the target modality of the target object, and the target modality is the full modality or the partial modality;
[0273] The image segmentation module 1120 is used to input the image data to be processed into a pre-trained target image segmentation model to obtain the annotation information of the segmentation result of each pixel in the image data output by the image segmentation model;
[0274] Among them, the target image segmentation model is a first image segmentation model or a second image segmentation model trained according to the training method of the neural network model of the present application, and includes an image segmentation model of an encoder corresponding to each modality in the target modality.
[0275] The device of the embodiment of the present application can execute the image segmentation method provided by the embodiment of the present application, and the implementation principle is similar. The actions performed by each module in the device of each embodiment of the present application correspond to the steps in the method of each embodiment of the present application. For the detailed functional description of each module of the device, please refer to the description in the corresponding method shown in the previous text, which will not be repeated here.
[0276] In an embodiment of the present application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, the processor executes the above-mentioned computer program to implement the steps of at least one of the training method of a neural network model and the image segmentation method. Compared with the related art, it can be achieved that: the neural network model includes a first segmentation model and at least one second segmentation model, each training sample of the first segmentation model includes full-modal image data of the target object, each second training sample of the second segmentation model includes image data of a specific modality of the missing target object, that is, image data of a partial modality of the target object, the first segmentation model includes a first encoder corresponding to each modality in the full modality, and a first decoder cascaded with each first encoder The second segmentation model includes a second encoder corresponding to each mode in the partial mode, and a second decoder cascaded with each second encoder. In each round of iterative training, the first decoder parameters of the first segmentation model and the second decoder parameters of each second segmentation model are obtained at the end of the current round of iterative training. Then, for each second segmentation model, the first sub-parameter involved in aggregation in the first decoder parameter and the second sub-parameter involved in aggregation in the second decoder parameter are determined according to the similarity between the first gradient and the second gradient at the end of each round of iteration in the most recent target round iteration. The embodiment of the present application determines the update method of the second decoder parameters of the second segmentation model and the first decoder parameters of the first segmentation model based on the similarity of the gradients. Are they consistent? If they are consistent, it means that the second segmentation model has a positive contribution to the update of the first segmentation model and should participate in the aggregation of model parameters. If they are inconsistent, it means that the decoder parameters of the second segmentation model are highly sensitive to data differences, and the second segmentation model should be restricted from participating in the aggregation, so as to effectively realize the personalized participation of the second decoder parameters of each second segmentation model in the aggregation. After determining the second sub-parameter participating in the aggregation in each second decoder parameter and the first sub-parameter participating in the aggregation in the first decoder parameter, for each second segmentation model, the first sub-parameter at the end of the current round of iterative training is aggregated with the second sub-parameter of the second segmentation model to obtain the second decoder parameters based on which the next round of iterative training is based, and the second decoder parameters based on the second decoder parameter obtained are obtained. The first decoder parameters and the second training samples of the second segmentation model are used to perform a round of training on the second segmentation model to obtain the second decoder parameters at the end of the next round of iterative training. When updating the first decoder parameters of the first segmentation model, the second sub-parameters of the second segmentation models at the end of the next round of iterative training are used in the embodiment of the present application. It can be found that the updates of the first segmentation model and the second segmentation model in each round of iteration of the present application are sequential. The first sub-parameters of the second segmentation models at the end of the current round of iterative training and the first sub-parameters of the first segmentation model are aggregated to obtain the first decoder parameters based on the next round of iterative training. This also reflects the purpose of personalized fusion of the second sub-parameters of each second segmentation model with the first sub-parameters.Ensure common knowledge sharing between the first segmentation model and the second segmentation model, and then further perform a round of training on the first segmentation model based on the obtained first decoder parameters and each first training sample, and obtain the first decoder parameters at the end of the next round of iterative training, which can alleviate the problem of poor convergence of federated learning caused by data heterogeneity between clients. Without leaking data privacy, the embodiment of the present application can not only obtain a first image segmentation model that performs well on all-modal image data (that is, the first segmentation model at the end of the last round of iterations), but also obtain the optimal personalized second image segmentation model that performs well when the client lacks a specific modality (that is, the second segmentation model at the end of the last round of iterations), and complete the image data segmentation task of the target object. ,
[0277] In an alternative embodiment, an electronic device is provided, such as Fig.12 As shown, Fig.12 The electronic device 4000 shown includes: a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, such as through a bus 4002. Optionally, the electronic device 4000 may also include a transceiver 4004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application.
[0278] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.
[0279] The bus 4002 may include a path for transmitting information between the above components. The bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus 4002 is represented by only one thick line in the figure, but it does not mean that there is only one bus or one type of bus.
[0280] The memory 4003 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium that can be used to carry or store computer programs and can be read by a computer, without limitation herein.
[0281] The memory 4003 is used to store the computer program for executing the embodiment of the present application, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the computer program stored in the memory 4003 to implement the steps shown in the above method embodiment.
[0282] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps and corresponding contents of the aforementioned method embodiment can be implemented.
[0283] The embodiment of the present application also provides a computer program product, including a computer program, which can implement the steps and corresponding contents of the aforementioned method embodiment when executed by a processor.
[0284] The terms "first", "second", "third", "fourth", "1", "2", etc. (if any) in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than that shown or described in the drawings.
[0285] It should be understood that, although each operation step is indicated by arrows in the flowchart of the embodiment of the present application, the implementation order of these steps is not limited to the order indicated by the arrows. Unless clearly stated herein, in some implementation scenarios of the embodiment of the present application, the implementation steps in each flowchart can be performed in other orders according to demand. In addition, some or all of the steps in each flowchart may include multiple sub-steps or multiple stages based on actual implementation scenarios. Some or all of these sub-steps or stages may be executed at the same time, and each sub-step or stage in these sub-steps or stages may also be executed at different times respectively. In different scenarios of execution time, the execution order of these sub-steps or stages may be flexibly configured according to demand, and the embodiment of the present application does not limit this.
[0286] The above is only an optional implementation method for some implementation scenarios of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the technical concept of the solution of the present application, other similar implementation methods based on the technical ideas of the present application are also within the protection scope of the embodiments of the present application.
Claims
1. A training method for a neural network model, characterized in that: The neural network model includes a first segmentation model and at least one second segmentation model, each first training sample of the first segmentation model includes full-modality image data of the target object, and each second training sample of the second segmentation model includes partial-modality image data of the target object; The method comprises: Obtaining, at the end of the current round of iterative training, first decoder parameters of the first segmentation model and second decoder parameters of each of the second segmentation models; For each second segmentation model, determining a first sub-parameter participating in aggregation in the first decoder parameter and a second sub-parameter participating in aggregation in the second decoder parameter according to the similarity between the first gradient and the second gradient at the end of each round of iteration in the most recent target round iteration; wherein the first gradient is an update gradient of a first decoder parameter of the first decoder, and the second gradient is an update gradient of a second decoder parameter of the second decoder; For each second segmentation model, aggregating the first sub-parameter at the end of the current round of iterative training with the second sub-parameter of the second segmentation model to obtain second decoder parameters based on the next round of iterative training, and performing a round of training on the second segmentation model based on the obtained second decoder parameters and each second training sample of the second segmentation model to obtain second decoder parameters at the end of the next round of iterative training; Aggregating the second sub-parameters of each second segmentation model at the end of the next round of iterative training and the first sub-parameters of the first segmentation model at the end of the current round of iterative training to obtain first decoder parameters based on the next round of iterative training, and performing a round of training on the first segmentation model based on the obtained first decoder parameters and each first training sample to obtain the first decoder parameters at the end of the next round of iterative training; The first segmentation model at the end of the last round of iteration is used as the first image segmentation model, and the second segmentation model at the end of the last round of iteration is used as the second image segmentation model.
2. The method according to claim 1, characterized in that The network structures of the decoders of the segmentation models in the neural network model are the same, and the decoders of the segmentation models include a first number of filters; For each second segmentation model, determining, according to the similarity between the first gradient and the second gradient at the end of each round of iteration in the most recent target round iteration, a first sub-parameter participating in aggregation in the first decoder parameter and a second sub-parameter participating in aggregation in the second decoder parameter of the second segmentation model, comprises: Determine, according to the similarity between the first gradient and the second gradient at the end of each round of iteration in the most recent target round iteration, a second filter participating in aggregation among the filters of the second decoder of the second segmentation model, and determine the parameters of each second filter as a second sub-parameter participating in aggregation among the parameters of the second decoder of the second segmentation model; According to each of the second filters, the first filter participating in the aggregation among the filters of the first decoder is determined, and the parameters of each first filter are determined as the first sub-parameters participating in the aggregation among the parameters of the first decoder, wherein the first filter is a filter other than the filters corresponding to the second filters among the filters of the first decoder.
3. The method according to claim 1, characterized in that The step of aggregating the second sub-parameters of each second segmentation model at the end of the next round of iterative training and the first sub-parameters of the first segmentation model at the end of the current round of iterative training to obtain the first decoder parameters based on the next round of iterative training includes: Obtaining a first weight corresponding to the first segmentation model and a second weight corresponding to each of the second segmentation models; fusing the second sub-parameters of each of the second segmentation models at the end of the next round of iterative training to obtain a third sub-parameter; Based on the first weight and the second weight, the first sub-parameter of the first segmentation model at the end of the current round of iterative training and the third sub-parameter are weightedly aggregated to obtain the first decoder parameter based on the next round of iterative training.
4. The method according to claim 3, characterized in that The fusing the second sub-parameters of each of the second segmentation models at the end of the next round of iterative training to obtain the third sub-parameters includes: For the target second sub-parameters of the filters with the same serial number in the second sub-parameters of the second decoders, the target second sub-parameters of the second segmentation models at the end of the next round of iterative training are fused to obtain the first fusion result corresponding to the filters with the same serial number in the next round of iteration; The first fusion results of the filters of each sequence number are fused to obtain the third sub-parameter.
5. The method according to claim 4, characterized in that The step of fusing the target second sub-parameters of each of the second segmentation models at the end of the next round of iterative training to obtain the first fusion result corresponding to the filter with the same sequence number in the next round of iteration includes: Obtaining the updated gradients of the second decoder parameters at the end of the current round of iterations; Obtaining a third weight of each second decoder according to an update gradient of each second decoder parameter at the end of a current round of iterations, wherein the third weight is inversely correlated with a magnitude of a gradient of the corresponding second decoder parameter; For the second sub-parameters of the second decoders, the target second sub-parameters of the filters corresponding to the same serial number are weighted summed with the third weights of the corresponding second decoders to obtain the first fusion result corresponding to the filters with the same serial number in the next round; The step of fusing the first fusion results of the filters of the respective sequence numbers to obtain the third sub-parameter includes: The average value of the first fusion results of the filters of each sequence number is used as the third sub-parameter.
6. The method according to claim 1 or 2, characterized in that: For each second segmentation model, determining a first sub-parameter participating in aggregation in the first decoder parameter and a second sub-parameter participating in aggregation in the second decoder parameter according to the similarity between the first gradient and the second gradient at the end of each round of iteration in the most recent target round iteration, including: If the similarity between the first gradient and the second gradient at the end of each round of iteration in the most recent target round iteration is less than a preset threshold, the binary mask of the second segmentation model at the end of this round of iteration is updated to obtain a binary mask based on which the next round of iterative training is performed; wherein the binary mask is used to indicate a second sub-parameter involved in aggregation in a second decoder parameter of the second segmentation model, and the binary mask based on which the first iterative training is performed is a preconfigured binary mask; According to the binary mask on which the next round of iterative training is based, determine the first sub-parameter participating in aggregation in the first decoder parameters and the second sub-parameter participating in aggregation in the second decoder parameters of the second segmentation model, wherein the first sub-parameter is the other parameters in the first decoder parameters except the parameters corresponding to the second sub-parameter.
7. The method according to claim 6, characterized in that The similarity between the first gradient and the second gradient includes the similarity between the first sub-gradient and the second sub-gradient of each corresponding filter in the first segmentation model and the second segmentation model; If the similarity between the first gradient and the second gradient at the end of each round of iteration in the most recent target round iteration is less than a preset threshold, the binary mask of the second segmentation model at the end of this round of iteration is updated to obtain the binary mask based on the next round of iterative training, including: For each filter in each second segmentation model, if the similarity of the filter at the end of each round of iteration in the most recent target round iteration is less than the preset threshold, the filter is used as the target filter; In the binary mask at the end of this round of iteration of the second segmentation model, the parameter corresponding to the target filter is cancelled from participating in aggregation, so as to obtain the binary mask based on which the next round of iterative training is based.
8. The method according to claim 1, characterized in that The update gradient of the second decoder parameter at the end of the current round of iteration is obtained by: Aggregate the first sub-parameter of the first decoder parameter at the end of the previous iteration with the second sub-parameter of the second decoder parameter at the end of the current iteration to obtain a second aggregation result corresponding to the current iteration; Obtaining an update gradient of the second decoder parameter at the end of the current round of iteration according to a difference between the second decoder parameter at the end of the current round of iteration and the second aggregation result; The update gradient of the first decoder parameter at the end of the current round of iteration is obtained by: According to the difference between the time when the first decoder parameter ends at the end of the current round of iteration and the time when the previous round of iteration ends, the update gradient of the first decoder parameter at the end of the current round of iteration is obtained.
9. The method according to claim 1 or 2, characterized in that: Each training sample also includes the annotation information of the segmentation result of each pixel in the corresponding image data; For each second segmentation model, each iteration also includes: For the second encoder of the second segmentation model, the first encoder parameters of the first encoder of the same modality at the end of the previous round of iteration are used as the second encoder parameters of the second encoder after being updated in the current round of iteration; The second segmentation model after updating the second encoder parameters and the second decoder parameters is used as the second segmentation model in the current iteration process, so as to train the second segmentation model in the current iteration process until it meets the second preset condition based on the training samples according to the second segmentation model, and obtain the second segmentation model at the end of the current iteration process; For the first segmentation model, each round of iteration further includes: For each first encoder, aggregating the second encoder parameters of all second encoders of the same mode at the end of this round of iteration to obtain updated first encoder parameters of the same mode of the first encoder; The first segmentation model after updating the first encoder parameters and the first decoder parameters is used as the first segmentation model in this round of iteration. The first segmentation model in this round of iteration is trained according to the training samples of the first segmentation model until it meets the first preset conditions, so as to obtain the first segmentation model at the end of this round of iteration.
10. The method according to claim 9, characterized in that The first decoder and each second decoder have the same decoder structure, each including at least one decoding layer, and each second encoder of the second segmentation model includes an encoding layer corresponding one-to-one to the decoding layer of the second decoder in the second segmentation model; The training of the second segmentation model in the current iteration process based on the training samples of the second segmentation model until the second preset condition is met includes: For each decoding layer of the second segmentation model in the current round of iteration, the encoding features output by the corresponding encoding layers of each second encoder of the second segmentation model are fused to obtain a first fused feature corresponding to the decoding layer; For each decoding layer of the second segmentation model in the current round of iteration, according to each feature anchor point corresponding to the decoding layer at the end of the previous round of iteration, the attention mechanism is used to calibrate the first fused feature corresponding to the decoding layer, and the first fused features before and after the calibration of the decoding layer are fused to obtain the second fused feature corresponding to the decoding layer; The second fused feature corresponding to the first decoding layer is used as the output result of the first decoding layer. For non-first decoding layers, feature decoding is performed according to the second fused feature corresponding to the non-first decoding layer and the output result of the previous decoding layer to obtain the output result of the non-first decoding layer; According to the output result of the last decoding layer, the predicted segmentation result of each pixel in the corresponding image data of the training sample is obtained, and according to the difference between the predicted segmentation result of each pixel and the annotation information, each second encoder parameter and second decoder parameter of the second segmentation model in the current round of iteration is optimized; The feature anchor points corresponding to each decoding layer are obtained by clustering the decoding features output by the corresponding decoding layer for the pixel points of the same segmentation result in each first training sample.
11. The method according to claim 10, characterized in that According to the feature anchor point at the end of the previous iteration, the first fusion feature corresponding to the decoding layer is calibrated using the attention mechanism, including: According to the feature anchor point at the end of the previous iteration, the first fusion feature corresponding to the decoding layer, and the number of filters in the decoding layer, an attention mechanism is used to perform normalization processing to obtain a fourth weight; The first fusion feature corresponding to the decoding layer is weighted by the fourth weight to obtain a calibrated first fusion feature.
12. The method according to claim 9, characterized in that The feature anchors of each decoding layer are updated in the following way: For each first training sample, determining a segmentation result corresponding to a decoding feature of each pixel point output by the decoding layer at the end of the previous iteration according to the labeling information of the first training sample; For each first training sample, the decoding features of each pixel point of the same segmentation result output by the decoding layer are fused to obtain the result features of the corresponding segmentation result of the decoding layer for the first training sample; For the same segmentation result, clustering the result features of the segmentation results of the decoding layer for each first training sample according to a preset number of clusters, and determining the features of the cluster center of each cluster cluster in the current round of iteration; For each cluster, the features of the cluster center of the cluster in the current iteration and the previous iteration are fused to obtain a feature anchor point corresponding to the cluster at the decoding layer at the end of the current iteration.
13. The method according to claim 1, characterized in that The method is applied in a federated learning scenario, wherein the first segmentation model is deployed in a server, and one of the second segmentation models is deployed in a client, and the method is executed by the server, wherein obtaining the second decoder parameters of each of the second segmentation models at the end of the current round of iterative training includes: Receiving model parameters of each second segmentation model sent by each of the clients, wherein the model parameters include second decoder parameters of the second segmentation model at the end of the current round of iterative training; For each second segmentation model, after obtaining the second decoder parameters based on which the next round of iterative training is based, the method further includes: The model parameters based on which the next round of iterative training of the second segmentation model is to be trained are sent to the corresponding client, so that the client performs a round of training on the second segmentation model based on the model parameters and the corresponding second training samples, and obtains the model parameters of the second segmentation model at the end of the next round of training.
14. An image segmentation method, characterized in that: include: Obtaining image data to be processed, wherein the image data to be processed is image data of a target modality of a target object, wherein the target modality is a full modality or a partial modality; Inputting the image data to be processed into a pre-trained target image segmentation model to obtain the annotation information of the segmentation result of each pixel in the image data output by the image segmentation model; The target image segmentation model is a first image segmentation model or a second image segmentation model trained according to the training method of the neural network model according to any one of claims 1 to 13, and includes an image segmentation model of an encoder corresponding to each modality in the target modality.
15. A training device for a neural network model, characterized in that: The neural network model includes a first segmentation model and at least one second segmentation model, each first training sample of the first segmentation model includes full-modality image data of the target object, and each second training sample of the second segmentation model includes partial-modality image data of the target object; The device comprises: A parameter acquisition module, used to acquire first decoder parameters of the first segmentation model and second decoder parameters of each of the second segmentation models at the end of a current round of iterative training; an aggregated sub-parameter determination module, configured to determine, for each second segmentation model, a first sub-parameter in the first decoder parameter that participates in aggregation and a second sub-parameter in the second decoder parameter that participates in aggregation according to the similarity between the first gradient and the second gradient at the end of each round of iteration in the most recent target round iteration; wherein the first gradient is an update gradient of a first decoder parameter of the first decoder, and the second gradient is an update gradient of a second decoder parameter of the second decoder; A first parameter updating module is used for aggregating, for each second segmentation model, the first sub-parameter at the end of the current round of iterative training with the second sub-parameter of the second segmentation model to obtain second decoder parameters based on the next round of iterative training, and performing a round of training on the second segmentation model based on the obtained second decoder parameters and each second training sample of the second segmentation model to obtain the second decoder parameters at the end of the next round of iterative training; A second parameter updating module is used to aggregate the second sub-parameters of each second segmentation model at the end of the next round of iterative training and the first sub-parameters of the first segmentation model at the end of the current round of iterative training to obtain the first decoder parameters based on the next round of iterative training, and to perform a round of training on the first segmentation model based on the obtained first decoder parameters and each first training sample to obtain the first decoder parameters at the end of the next round of iterative training; The first segmentation model at the end of the last round of iteration is used as the first image segmentation model, and the second segmentation model at the end of the last round of iteration is used as the second image segmentation model.
16. An image segmentation device, characterized in that: include An image data acquisition module is used to obtain image data to be processed, wherein the image data to be processed is image data of a target modality of a target object, wherein the target modality is a full modality or a partial modality; An image segmentation module, used for inputting the image data to be processed into a pre-trained target image segmentation model, and obtaining the annotation information of the segmentation result of each pixel in the image data output by the image segmentation model; The target image segmentation model is a first image segmentation model or a second image segmentation model trained according to the training method of the neural network model according to any one of claims 1 to 13, and includes an image segmentation model of an encoder corresponding to each modality in the target modality.
17. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the method according to any one of claims 1 to 14.
18. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 14 is implemented.
19. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 14 is implemented.
