Image segmentation interpretation method based on multi-expert mixture, electronic device and storage medium

Abstract

The present invention discloses a multi-expert hybrid image segmentation interpretation method, electronic device, and storage medium. The method includes an independent multi-expert segmentation network and a hybrid-expert mechanism based on a gated network or a routing network. This model uses multiple homogeneous or heterogeneous image segmentation networks to form a multi-expert hybrid model, and segments labels by component, thereby providing targeted training and learning for the experts. This makes each expert network relatively independent. Compared to black-box interpretation and post-training model interpretation based on training results, this method can decouple different expert networks and achieve interpretable segmentation at the branch network level.

Description

Technical Field

[0001] The present invention belongs to the technical field of image segmentation and interpretability model, and in particular relates to an image segmentation interpretation method based on multi-expert mixture, an electronic device and a storage medium. Background Art

[0002] In recent years, with the development of deep learning, a growing number of researchers have devoted themselves to applying deep learning to the image field, including but not limited to image classification, object detection, image segmentation, and image generation. Many classic network models have been proposed to adapt to different image tasks, such as the commonly used classification network AlexNet, the classic object detection network Yolo, and the fully convolutional neural network (FCN). However, most of these networks are "black box models" whose learning and objectives are entirely adaptive, leaving technicians unable to understand the exact functioning of the network or its branches during training. Alternatively, researchers can simply perform reverse engineering based on the outputs or heatmaps of different layers of the trained network. However, neither approach achieves interpretability at the network training level.

[0003] The multi-expert mixture model was first proposed in 1991. In recent years, there has been some work based on this approach to enhance image processing tasks. However, most of this work has focused on increasing network width and improving performance by leveraging multiple experts, without analyzing and decoupling the work performed by each expert network. In fact, most of this work can be described as network aggregation, combining several "small black boxes" into a larger "black box" without leveraging the independence and specificity of the expert networks. Consequently, the network training process remains unexplained. Summary of the Invention

[0004] In order to overcome the deficiencies of the above-mentioned prior art, the purpose of the present invention is to provide an image segmentation interpretation method, electronic device and storage medium based on a mixture of multiple experts, so as to solve the problem that the existing image segmentation is not truly interpretable, and thereby further optimize the image segmentation effect.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] The image segmentation interpretation method, electronic device and storage medium based on multi-expert hybrid are realized by using a multi-expert hybrid network and additional loss in the learning process that allows experts to maintain independence, so as to achieve decoupling of the expert network and interpretability at the branch network level and the aggregation network level. Based on this method, the expert network can focus only on a small number of features it needs during feature extraction, and only focus on generating segmented images of the parts it is responsible for during decoding, giving full play to the "expertise" of each expert, so that the multi-segmentation problem can be differentiated into a single segmentation problem, which greatly improves the scalability of the multi-expert network in image segmentation tasks and the possibility of aggregating multiple heterogeneous networks.

[0007] The image segmentation interpretation method based on multi-expert mixture specifically includes the following steps:

[0008] Step 1: Build a multi-expert hybrid network;

[0009] Step 2: Segment and reconstruct the labels in the original image dataset into component label data, simplifying the multi-segmentation task into a single segmentation task.

[0010] In step 3, the encoders of each expert network, i.e., the feature extraction modules, are used to extract features from the original image data. During backpropagation, each expert optimizes parameters using the loss function calculation module, the component label data visible only to the expert, and the prediction graphs output by the expert. During the optimization process, each expert continuously extracts the features they need and discards useless features to minimize the losses of the multi-expert hybrid network and the network aggregation module.

[0011] In step 4, the decoders of each expert network, i.e., the image generation modules, are used to segment the component images based on the features extracted by each expert. Finally, the outputs of each expert network are aggregated to obtain the segmentation result of the multi-expert hybrid network, which is the final segmentation prediction map.

[0012] The multi-expert hybrid network includes multiple homogeneous or heterogeneous image segmentation networks, namely, an expert network, a network aggregation module composed of a gating network or a routing network, and a loss function calculation module modeling model loss;

[0013] The image segmentation network is a network model based on deep learning algorithms that divides an image into several specific regions with unique properties; for example, FCN, SegNet, and UNet;

[0014] The multi-expert mixture network is used to extract features of different components in the original image dataset and generate segmentation maps of different components in a targeted manner;

[0015] The original image dataset is composed of a data set of the required segmentation objects consisting of RGB or ARGB multi-channel color images and a data set of the corresponding labels of the required segmentation objects consisting of single-channel grayscale images.

[0016] The component label dataset is obtained by dividing the labels corresponding to the original image dataset into multiple different component labels according to certain rules.

[0017] The component is a part of the label corresponding to the segmented object in the original image data set, and all component labels together constitute an object label.

[0018] Furthermore, the network aggregation module is composed of a gating network or a routing network. The former is used to perform weighted aggregation on the prediction maps output by multiple experts in the multi-expert hybrid network according to the expert weights generated by taking the original image data or the feature maps output by each expert encoder as input to obtain the overall prediction map of the original image segmentation object; the latter is used to perform classified routing on the multiple block sub-maps obtained by dividing the original image into blocks according to a certain proportion and adopting the Overlap method, and give the routing score of the corresponding component of each block sub-map as the routing basis of the routing algorithm. Each expert in the multi-expert hybrid network obtains its own block sub-map as input according to the routing score corresponding to each component of the block sub-map and a certain routing algorithm, and outputs the prediction map of the corresponding block. The prediction maps output by all experts are linearly merged to obtain the overall prediction map of the original image segmentation object.

[0019] The block sub-image refers to a sub-image obtained by dividing the original image into blocks according to a certain ratio and the Overlap method. For example, if the original image is 224x224 in size, it is divided into blocks at a ratio of 1 / 4 to obtain 16 56x56 sub-blocks. The Overlap method refers to retaining the surrounding area of ​​the blocks at a specific size so that adjacent sub-blocks have overlapping information.

[0020] The loss function calculation module is used to calculate the segmentation loss of the multi-expert hybrid network for the input image data, as well as the segmentation loss of each expert network for each component, and feed the calculation results back to the multi-expert hybrid network and network aggregation module to optimize and update their parameters.

[0021] Furthermore, the loss function Loss of the multi-expert hybrid segmentation network model is as follows:

[0022]

[0023] Among them, Weight Moe Represents the loss weight of the final segmentation result of all components, Moe represents the final segmentation result of all components output, represents the total output of the multi-expert mixture network for the sample Loss, n represents the number of expert networks, E i represents the segmentation result output of the i-th expert network, Represents the loss of each expert for the sample.

[0024] Furthermore, the loss function The calculation of is as follows:

[0025]

[0026] in is the MSE (mean square error) function; h, w are the number of rows and columns of pixels in the output single-channel segmented image respectively; y_pred (x,y) Indicates the pixel value of the output segmented image at the horizontal coordinate x and the vertical coordinate y; y_true (x,y) The pixel value of the component label at the horizontal coordinate x and the vertical coordinate y;

[0027] Furthermore, the image segmentation process in step 3 is as follows:

[0028] Step 31: Divide the original image into n parts (if a routing network is used, the original image is divided into blocks and overlapped according to a certain ratio), and send them to the encoders of each expert network in the multi-expert hybrid network respectively. Each expert optimizes the parameters in the back propagation through the loss function calculation module and the component label data visible only to itself and the prediction map output by each expert. During the optimization process, each expert will continuously extract the feature encoder required by itself. i (x), discard useless features to minimize the loss of the multi-expert mixture network and network aggregation module.

[0029] Furthermore, the image segmentation process in step 4 is as follows:

[0030] Step 41, each expert network in the multi-expert hybrid network outputs its own encoder, i.e. Encoder i (x) Generate the segmented component graph, i.e. E i =Decoder i (Encoder i (x));

[0031] Step 42: The outputs of the encoders of each expert in the multi-expert hybrid network are fused in a certain way to form the input of the gating network, and the weight vectors of each expert are obtained through the global pooling layer, convolution layer and fully connected layer output of the gating network, i.e., W E =Gate(Concat(Encoder i (x)|i∈(0,n)));

[0032] Step 43: Finally, the outputs of each expert network in the multi-expert hybrid network are weighted and merged to obtain the final segmentation output result of the multi-expert hybrid network, that is,

[0033] Furthermore, the Gate is a simple gating network, which consists of GAP (global average pooling) module, Conv2D 1x1 (convolution module) and FC (feedforward neural network module).

[0034] An electronic device comprises a processor, a memory and a communication bus, wherein the processor and the memory communicate with each other via the communication bus;

[0035] Memory for storing computer programs;

[0036] The processor is configured to implement the above method steps when executing the program stored in the memory.

[0037] A computer-readable storage medium stores a computer program, which implements the above method steps when executed by a processor.

[0038] Beneficial effects of the present invention:

[0039] 1. The present invention utilizes a multi-expert hybrid network and corresponding additional losses to make the expert learning interpretable, controllable, and targeted;

[0040] 2. By decoupling the individual expert networks in the multi-expert hybrid network, this invention achieves the interpretability of each expert's segmentation of different components compared to the "black box" model of the multi-expert aggregate network, and balances and improves the overall network segmentation performance and accuracy;

[0041] 3. The present invention also provides a reference for the aggregation of multi-expert heterogeneous networks, making it possible for heterogeneous experts to be responsible for the components or tasks they are good at and aggregate them into a network as a whole. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is an interpretable image segmentation network graph based on a mixture of multiple experts (gated approach).

[0043] Figure 2 It is an interpretable image segmentation network graph based on a mixture of multiple experts (routing approach). DETAILED DESCRIPTION

[0044] The present invention will be described in further detail below with reference to the accompanying drawings.

[0045] As mentioned above, when existing deep learning is used for image processing, it lacks interpretability at the training level. The present invention proposes an interpretable image segmentation method based on a multi-expert mixture. In this method, the decoupling of the expert network and the interpretability at the branch network level and the aggregation network level are achieved through a multi-expert hybrid network and an additional loss in the learning process that allows experts to maintain independence. Based on this method, the expert network can only focus on a small number of features it needs during feature extraction, and only focus on generating segmented images of the components it is responsible for during decoding, giving full play to the "expertise" of each expert, allowing multi-dimensional segmentation problems to be differentiated into single segmentation problems, thereby greatly improving the scalability of multi-expert networks in image segmentation tasks and the possibility of aggregating multiple heterogeneous networks.

[0046] For ease of understanding, the concepts involved in the present invention are explained as follows:

[0047] Multiple experts:

[0048] Multiple experts also refer to multiple expert networks, which refer to multiple neural networks, i.e., multiple experts. Each expert is selected to be applied to a different part of the data set or to process different tasks or components, specifically different components in the present invention.

[0049] Interpretation method:

[0050] The interpretation method is an explanation method, and in the present invention, it specifically refers to an interpretable method applied to image segmentation.

[0051] like Figure 1 As shown in FIG, the image segmentation interpretation method based on multi-expert mixture includes the following steps:

[0052] Step S1, construct a multi-expert hybrid network, such as Figure 1 As shown, the multi-expert hybrid network includes multiple homogeneous or heterogeneous image segmentation networks, namely, expert networks;

[0053] The multi-expert mixture network is used to extract the features of different components in the original image data. i (x), and generate segmentation maps E of different components in a targeted manner i =Decoder i (Encoder i (x)); where Encoder i It is a feature extraction network composed of multiple convolution and maximum pooling feature extraction blocks. i It is a decoding module including but not limited to multiple upsampling and nonlinear convolution; E i Represents the output of the i-th expert network;

[0054] The network aggregation module is composed of a gated network or a routing network:

[0055] The former is used to generate expert weights based on the original image data or the feature maps output by each expert encoder as input, namely:

[0056] W E =Gate(Concat(Encoder i (x)|i∈(0,n)))

[0057] The prediction graphs output by multiple experts in the multi-expert mixture network are weightedly aggregated to obtain the overall prediction graph of the original image segmentation object; the generation process of the overall prediction graph of the original image segmentation object is as follows:

[0058] The original image input is x h*w*c , after Encoder i (x), and Concat processing to obtain the channel-wise spliced ​​gated network input [Encoder1, Encoder2, Encoder3, ..., Encoder n ], denoted as f n ; After the gated network, Gate(f n ), and then get the expert's weight vector W E =[W1,W2,W3,...,W n ]; Finally, the outputs of all expert network modules are weighted and merged, that is,

[0059] Among them, Gate(f n )=Softmax(FC n (FC 128 (Conv2D 1x1 (GAP(f n )))));

[0060] The latter is used to divide the original image into blocks according to a certain ratio and adopt the Overlap method to obtain multiple block sub-graphs, which are recorded as patches; the patches are classified and routed, and the routing score of the corresponding component of each block sub-graph is given, that is, As the routing basis of the routing algorithm. Where n is the number of sub-images of the original image, after classification, a probability subordination matrix with a dimension of NxC can be obtained:

[0061]

[0062] in Indicates that the subgraph with index i belongs to part j The probability of the classifier is sent to the router for routing and passed to the encoder of the expert network. Part of the dynamic routing algorithm can be written as:

[0063] Each expert in the multi-expert hybrid network obtains their own block subgraph as input based on the routing score and routing algorithm corresponding to each component of the block subgraph, and outputs the prediction graph of the corresponding block. The prediction graphs output by all experts are linearly merged to obtain the overall prediction graph of the original image segmentation object;

[0064] The routing algorithms include dynamic routing algorithms such as Top-K based on expert selection, Top-K based on sample selection, and static routing algorithms such as Hash-Router. Some dynamic routing algorithms can be written as:

[0065]

[0066] A sub-block image is created by dividing the original image into blocks using a specific ratio and the overlap method. For example, if the original image is 224x224, dividing it into blocks at a ratio of 1 / 4 will result in 16 56x56 sub-blocks. The overlap method preserves the surrounding area of ​​the blocks at a specific size, allowing adjacent sub-blocks to overlap.

[0067] The loss function calculation module is used to calculate the segmentation loss of the multi-expert mixture network for the segmented object in the input image, as well as the segmentation loss of each expert network for each component of the segmented object in the input image, and feed the calculation results back to the multi-expert mixture network and the network aggregation module to optimize and update their parameters;

[0068] Establish a loss function for image segmentation interpretation method based on multi-expert mixture;

[0069] The purpose of the image segmentation interpretation method based on multi-expert mixture is to complete the multi-component segmentation task under the premise of being able to interpret the work of experts, that is, to let each expert take full or main responsibility for the segmentation prediction of his or her own component. The loss function is to calculate the mean square error between the output of each expert network and the component label of the segmented object in the original image dataset. The total segmentation image obtained by merging the outputs of each expert through the network aggregation module is used as the evaluation of the overall segmentation effect. The loss function is to calculate the mean square error between the output and label of the multi-expert mixture. The determination process of each part of the loss function is as follows:

[0070] (1) In order to ensure that each expert has targeted learning, this example sets an additional loss for each expert, so that each expert only sees or focuses on the label data of a single component for the input data of all components, so that the expert's feature extraction and segmentation image generation are targeted and controllable. The additional loss function Expand to

[0071] (2) Only focusing on the component segmentation of each expert network does not necessarily guarantee the segmentation effect of the overall multi-expert hybrid network. Since the expert networks are independent of each other, the output of the multi-expert hybrid network may overlap or have gaps, which requires the feedback of the overall prediction output to optimize the parameters. Therefore, the aggregation loss Loss of the multi-expert hybrid network is set. Moe =MSE(label entire ,Moe), which is expanded to

[0072]

[0073] (3) By setting the weights of the total loss of the multi-expert hybrid network and the single component loss of each expert, the learning tendency of the network is adjusted, and the one that focuses more on the overall rather than the professionalism of the expert is given the loss. Moe A larger weight is given to each Assign smaller weights, and vice versa; the losses of different experts can also be different, reflecting that different experts have different requirements for the segmentation effect and independence of components; the overall loss function of the network is as follows:

[0074]

[0075] Step S2, multi-expert feature extraction;

[0076] Since the overall interpretability of the network requires the expert network to independently select the component features it needs when extracting features, this example sets up an independent encoder for each expert to flexibly extract the features required by each expert;

[0077] The shape of the input image is x = (B, H, W, C). The encoder is composed of a combination of feature extraction units including but not limited to multiple convolutional pooling blocks. The module can be formulated as:

[0078] Block=MaxPool2D(Conv2D(Conv2D(...(x))))

[0079] The encoder can be formulated as follows:

[0080] Encoder(x)=Block(Block(...(x)))

[0081] Step S3, generating gated network weights;

[0082] In order to optimize the aggregation effect of the component segmentation map output by the experts, a simple gating network is used in this example to sample and fuse the output of each expert's feature extraction module and output the weight vector of the expert aggregation after nonlinear transformation. The gating network structure includes but is not limited to a global average pooling module, a convolution module, a feedforward network module, and a probability normalization module. The output of each expert's independent feature extraction module is converted into an encoder. i (x) is spliced ​​into a feature map [Encoder1, Encoder2, Encoder3, ..., Encoder n ], denoted as f n , sent to the gate control network Gate, nonlinear transformation and normalization after the output weight vector W E =[W1,W2,W3,...,W n ];

[0083] Step S4, aggregation of multiple expert outputs;

[0084] The gating method performs weighted merging based on the component segmentation results output by the multi-expert hybrid network and the weight vector output by the gating network to obtain the overall segmentation result output Moe of the multi-expert hybrid network and the output E of each expert network in the multi-expert hybrid network. i , the formula is as follows:

[0085] E i =Decoder i (Encoder i (x))

[0086]

[0087] Most of the existing image segmentation work is carried out in a data-driven manner. Occasionally, there is interpretable work, which is also based on the results to reversely interpret the training process of the network model. However, there is a lack of direct supervision and restriction on the network model training process, which makes the network model always transparent to technical personnel. Although this may achieve good task results in some cases, it is not conducive to the interpretability of the network and the pruning, optimization or expansion of the network. The present invention, through multi-expert network aggregation and taking image segmentation tasks as examples, proposes a method based on multi-expert hybrid network decoupling in order to make the training of branch networks controllable and targeted during the training process of the network model. The present invention provides a basic model and concept of an interpretable multi-task network, which provides new ideas for the construction of multi-task or divisible single-task networks, and truly achieves complete interpretability of the network level corresponding to the training process and training results in terms of network interpretability.

[0088] The present invention constructs multiple independent and interpretable experts through the basic model of a multi-expert hybrid network. The experts have independent feature extraction modules and decoding modules, and the output of each expert is used as the basis for judging the loss of the entire network. The difference is that although each expert can see the entire data set (the routing expert can only see a subset of the data), the corresponding label is only a subset of the label set (the routing expert is also only responsible for the labels of the corresponding data subset). The loss of the entire system is obtained by a weighted linear combination of the overall network loss and the losses output by each expert, and is used for backpropagation and parameter optimization.

[0089] The present invention uses a simple gating network to integrate the component output results of experts, optimizes the cooperation and competition between experts, and makes the overall output effect of the system better. The problem of expert competition and cooperation is reflected in the overlapping of component segmentation images output by experts, and the collaborative processing of a single component by multiple experts, which results in each expert being unable to contribute their strength.

[0090] The present invention also includes an electronic device, including a memory and a processor, wherein the memory is used to store various computer program instructions, and the processor is used to execute the computer program instructions to complete all or part of the above steps; the electronic device can communicate with one or more external devices, and can also communicate with one or more devices that enable users to interact with the electronic device, and / or communicate with any device that enables the electronic device to communicate with one or more other computing devices. The electronic device can also communicate with one or more networks (such as local area networks, wide area networks and / or public networks) through a network adapter.

[0091] The present invention also includes a computer-readable storage medium storing a computer program that can be executed by a processor. The computer-readable storage medium may include but is not limited to magnetic storage devices, optical disks, digital versatile disks, smart cards and flash memory devices. In addition, the readable storage medium described in the present invention can represent one or more devices and / or other machine-readable media for storing information. The term "machine-readable medium" includes but is not limited to wireless channels and various other media (and / or storage media) that can store, contain and / or carry codes and / or instructions and / or data.

[0092] Each embodiment in this specification is described in a related manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiment is generally similar to the method embodiment, so the description is relatively simple. For related parts, refer to the description of the method embodiment.

[0093] The above description is only a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention are included in the scope of protection of the present invention.

Claims

1. Image segmentation interpretation method based on multi-expert mixture, characterized by: Specifically include the following steps: Step 1: Build a multi-expert hybrid network; Step 2: Segment and reconstruct the labels in the original image dataset into component label data, simplifying the multi-segmentation task into a single segmentation task. In step 3, the encoders of each expert network, i.e., the feature extraction modules, are used to extract features from the original image data. During backpropagation, each expert optimizes parameters using the loss function calculation module, the component label data visible only to the expert, and the prediction graphs output by the expert. During the optimization process, each expert continuously extracts the features they need and discards useless features to minimize the losses of the multi-expert hybrid network and the network aggregation module. In step 4, the decoders of each expert network, i.e., the image generation modules, are used to segment the features extracted by each expert to generate component images. Finally, the outputs of each expert network are aggregated to obtain the segmentation result of the multi-expert hybrid network, which is the final segmentation prediction map.

2. The image segmentation and interpretation method based on multi-expert mixture according to claim 1 is characterized in that: The multi-expert hybrid network includes multiple homogeneous or heterogeneous image segmentation networks, namely, a network aggregation module composed of an expert network, a gating network or a routing network, and a loss function calculation module; The image segmentation network is a network model based on a deep learning algorithm that divides an image into several specific regions with unique properties; The multi-expert mixture network is used to extract features of different components in the original image dataset and generate segmentation maps of different components in a targeted manner; The original image dataset is composed of a data set of the required segmentation objects consisting of RGB or ARGB multi-channel color images and a data set of the corresponding labels of the required segmentation objects consisting of single-channel grayscale images; The component label dataset is obtained by dividing the labels corresponding to the original image dataset into multiple different component labels according to certain rules; The component is a part of the label corresponding to the segmented object in the original image data set, and all component labels together constitute an object label.

3. The image segmentation and interpretation method based on multi-expert mixture according to claim 2, characterized in that: The network aggregation module is composed of a gating network or a routing network. The former is used to perform weighted aggregation on the prediction maps output by multiple experts in the multi-expert hybrid network according to the expert weights generated by taking the original image data or the feature maps output by each expert encoder as input to obtain the overall prediction map of the original image segmentation object; the latter is used to divide the original image into blocks according to a certain proportion and adopt the overlap method to obtain multiple block sub-maps for classification routing, and give the routing score of the corresponding component of each block sub-map as the routing basis of the routing algorithm. Each expert in the multi-expert hybrid network obtains its own block sub-map as input based on the routing score corresponding to each component of the block sub-map and a certain routing algorithm, and outputs the prediction map of the corresponding block. The prediction maps output by all experts are linearly merged to obtain the overall prediction map of the original image segmentation object. The block sub-image is a sub-image obtained by dividing the original image into blocks according to a certain ratio and the Overlap method. For example, if the original image is 224x224 in size, it is divided into 16 56x56 sub-blocks by a ratio of 1 / 4. The Overlap method means that the surrounding area of ​​the blocks is retained at a specific size so that adjacent sub-blocks have overlapping information. The loss function calculation module is used to calculate the segmentation loss of the multi-expert hybrid network for the input image data, as well as the segmentation loss of each expert network for each component, and feed the calculation results back to the multi-expert hybrid network and network aggregation module to optimize and update their parameters.

4. The image segmentation and interpretation method based on multi-expert mixture according to claim 1, characterized in that: The loss function Loss of the multi-expert mixture network is as follows: Among them, Weight Moe Represents the loss weight of the final segmentation result of all components, Moe represents the final segmentation result of all components output, represents the total output of the multi-expert mixture network for the sample Loss, n represents the number of expert networks, E i represents the segmentation result output of the i-th expert network, Represents the loss of each expert for the sample.

5. The image segmentation and interpretation method based on multi-expert mixture according to claim 4 is characterized in that: The loss function The calculation is as follows: in is the MSE (mean square error) function; h, w are the number of rows and columns of pixels in the output single-channel segmented image; y_pred (x,y) Indicates the pixel value of the output segmented image at the horizontal coordinate x and the vertical coordinate y; y_true (x,y) Indicates the pixel value of the component label at the horizontal coordinate x and the vertical coordinate y.

6. The image segmentation and interpretation method based on multi-expert mixture according to claim 1, characterized in that: The image segmentation process in step 3 is as follows: Step 31: Divide the original image into n parts (if a routing network is used, the original image is divided into blocks and overlapped according to a certain ratio), and send them to the encoders of each expert network in the multi-expert hybrid network respectively. Each expert optimizes the parameters in the back propagation through the loss function calculation module and the component label data visible only to itself and the prediction map output by each expert. During the optimization process, each expert will continuously extract the feature encoder required by itself. i (x), discard useless features to minimize the loss of the multi-expert mixture network and network aggregation module.

7. The image segmentation and interpretation method based on multi-expert mixture according to claim 1 is characterized in that: The image segmentation process in step 4 is as follows: Step 41, each expert network in the multi-expert hybrid network outputs its own encoder, i.e. Encoder i (x) Generate the segmented component graph, i.e. E i =Decoder i (Encoder i (x)); Step 42: The outputs of the encoders of each expert in the multi-expert hybrid network are fused in a certain way to form the input of the gating network, and the weight vectors of each expert are obtained through the global pooling layer, convolution layer and fully connected layer output of the gating network, i.e., W E =Gate(Concat(Encoder i (x)i∈(0,n))); Step 43: Finally, the outputs of each expert network in the multi-expert hybrid network are weighted and merged to obtain the final segmentation output result of the multi-expert hybrid network, that is, 8. The image segmentation and interpretation method based on multi-expert mixture according to claim 7 is characterized in that: The Gate is a simple gating network consisting of GAP module, Conv2D 1x1 and FC composition.

9. An electronic device, characterized in that: It includes a processor, a memory and a communication bus, wherein the processor and the memory communicate with each other via the communication bus; Memory for storing computer programs; A processor, configured to implement the method steps described in any one of claims 1 to 8 when executing a program stored in a memory.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps of any one of claims 1 to 8 are implemented.

Images