Universal medical segmentation method based on hybrid expert model and related equipment

Abstract

The invention discloses a general medical segmentation method based on a hybrid expert model and related equipment. The method comprises the following steps: acquiring an image to be segmented; inputting the to-be-segmented image into the trained image segmentation model, and outputting a segmentation result; the image segmentation model comprises an image encoder used for carrying out feature extraction on an input image to obtain image features; the prompt encoder is used for performing feature extraction on input user prompts to obtain prompt features; the image decoder is used for generating an image mask according to the image features and the prompt features; in a Transform block of the image encoder, an attention module is replaced by a MoPE module, and a proper expert is dynamically selected for processing according to characteristics of input data. According to the method, the parameter utilization efficiency of the model is optimized through the MoPE module, so that in different medical image segmentation tasks, proper experts can be dynamically selected for processing according to the characteristics of input data, the calculation cost is remarkably reduced, and the segmentation precision is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a universal medical segmentation method based on a hybrid expert model and related equipment. Background Art

[0002] Multimodal medical image segmentation is an important research area in computer vision and medical image processing. It is primarily used to extract structural information from different types of medical images, such as CT, MRI, and ultrasound. This technology is crucial for many medical applications, such as disease diagnosis, surgical planning, and postoperative evaluation.

[0003] Common medical image segmentation methods are primarily based on deep learning frameworks, such as U-Net and its variants. While these methods address the medical image segmentation challenge to some extent, due to their structural design and parameter limitations, they often fail to fully capture the complex features and rich contextual information in images, resulting in suboptimal performance when processing data from different modalities. In particular, these models' adaptability and generalization capabilities are particularly limited when applied to different patients or disease types. Large-scale pre-trained models based on Transformers, such as SAM and SAMMed2d, not only effectively model cross-modal feature interactions but also achieve zero-shot generalization through hinting mechanisms. However, directly increasing the model size leads to a surge in parameters, making full parameter fine-tuning computationally prohibitive. This problem is particularly prominent in computationally intensive tasks such as medical image segmentation, limiting the deployment of large-scale models in real-world applications. A mainstream view is that Mixture of Experts (MoE) models can achieve better performance and generalization without significantly increasing computational costs. VMoE was the first to bring MoE from the field of Natural Language Processing (NLP) to the field of Computer Vision (CV). VMoE introduces a MoE module in the Feed Forward Network (FFN) layer, increasing model size while keeping its training and inference costs roughly constant. ST-MoE introduces two new auxiliary loss functions to assist in routing training, addressing potential routing imbalance issues. However, extensive experiments have shown that the traditional FFN-MoE module performs poorly when applied to image segmentation tasks. For segmentation tasks, FFN-MoE requires the gating network to pre-classify different image regions, which poses a challenge for lightweight routing. Furthermore, FFN-MoE performs routing based on a single image patch (dimension d), while the attention layer needs to jointly process n patches. Directly transferring this mechanism to the attention layer would result in an expansion of the input dimension to n × d (the expert output remains E), leading to parameter explosion and dilution of routing signals. Summary of the Invention

[0004] In order to at least solve one of the technical problems existing in the prior art to a certain extent, the present invention aims to provide a general medical segmentation method and related equipment based on a hybrid expert model.

[0005] The first technical solution adopted by the present invention is:

[0006] A general medical segmentation method based on a mixture of experts model comprises the following steps:

[0007] Obtain the image to be segmented;

[0008] Input the image to be segmented into the trained image segmentation model and output the segmentation result;

[0009] The image segmentation model includes an image encoder, an image decoder and a prompt encoder. The image encoder is used to extract features from the input image to obtain image features; the prompt encoder is used to extract features from the input user prompts to obtain prompt features; the image decoder is used to generate an image mask based on the image features and prompt features; in the Transformer block of the image encoder, the MoPE module is used to replace the attention module, and a suitable expert is dynamically selected for processing based on the features of the input data.

[0010] Furthermore, obtaining the image to be segmented includes:

[0011] For 2D images, resize the image to the preset resolution; for 3D images, resize the image to the preset resolution after slicing.

[0012] Further, the image encoder includes an image embedding layer and a plurality of Transformer blocks;

[0013] The image embedding layer is used to convolve the image. The number of channels after convolution is 768, that is, the RGB image with a resolution of 256*256 is converted into an image token sequence of [768,16,16]. The token sequence will be processed by multiple Transformer blocks, but the structure will not change.

[0014] Finally, the output of the image encoder is merged with the user prompt output by the prompt encoder [768+x,16,16], where x depends on the number and type of prompts provided by the user. This sequence is then passed to the image decoder for decoding, outputting a [1,256,256] mask image.

[0015] Furthermore, the Transformer block includes a first layer normalization, a MoPE module, a second layer normalization and a feed-forward layer;

[0016] The MoPE module includes a prototype router and multiple expert groups. Each expert group includes three LoRA matrices, which are the projection matrices W corresponding to Q, V, and O in the attention. Q ,W V ,W O Adjustment is made; the prototype routing is responsible for distributing the input data to different experts.

[0017] Furthermore, the calculation formula of the MoPE module is as follows:

[0018]

[0019]

[0020] Where X is the input of the MoPE module; g e represents the routing score of the e-th expert; Concat represents the connection function, W O is the projection matrix of the Output module, is the correction matrix of the Output module of the e-th expert, E is the number of experts, here it is 8; Head i represents the i-th attention head, H is the number of attention heads; Q i is the Query of the i-th attention head, K i is the Key of the i-th attention head, V i is the Value of the i-th attention head, d k Represents K i The dimension of , T is the transpose; is the projection matrix of the Query module of the i-th attention head, is the correction matrix of the Query module of the e-th expert in the i-th attention head, is the projection matrix of the Value module of the i-th attention head, is the correction matrix of the Value module of the e-th expert in the i-th attention head, is the downsampling matrix of the Output module of the e-th expert, is the upsampling matrix of the Output module of the e-th expert, is the downsampling matrix of the Query module of the e-th expert in the i-th attention head, is the upsampling matrix of the Query module of the e-th expert in the i-th attention head, is the downsampling matrix of the Value module of the e-th expert in the i-th attention head, is the upsampling matrix of the Value module of the e-th expert in the i-th attention head.

[0021] Furthermore, the prototype router determines which group of experts to assign the input to by accepting the input and comparing it with the expert prototypes stored in itself by cosine similarity. The calculation formula is as follows:

[0022]

[0023] Score=[CosSim(p 1 ,X),...,CosSim(p 7 ,X)]

[0024] Where, CosSim(pe ,X) is used to calculate p e and X; TOP1 is an operation that sets all elements in a vector to zero except the element with the first maximum value; X is the input of the MoPE module; p e is the expert prototype of the e-th expert, Score e is the similarity score of the e-th expert.

[0025] Furthermore, during training, only the weights of the MoPE module in the image encoder are updated; for weights outside the MoPE module, they are initialized using the pre-trained weights of SAM ViT Base; the positional encoding is scaled in the same way as SAMMed2D; the prototypes in the MoPE module are randomly initialized; then, all layers, including the SAM layer, are frozen, and a round of prototype routing is updated using the prototype update strategy.

[0026] Furthermore, the loss function during the image segmentation model training process is:

[0027] L=L Dice +L ce

[0028] L Dice =1-Dice(A,B)

[0029]

[0030] Where A is the predicted segmentation area, B is the actual segmentation area; N is the total number of pixels in the image, C is the number of categories, and y i,c is the one-hot encoding of the true label, is the model’s predicted probability that pixel i belongs to category c.

[0031] The second technical solution adopted by the present invention is:

[0032] An electronic device comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, at least one program, the code set, or the instruction set is loaded and executed by the processor to implement a general medical segmentation method based on a hybrid expert model as described above.

[0033] The third technical solution adopted by the present invention is:

[0034] A computer-readable storage medium stores at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement a general medical segmentation method based on a hybrid expert model as described above.

[0035] The fourth technical solution adopted by the present invention is:

[0036] A computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned general medical segmentation method based on a mixture of experts model.

[0037] The beneficial effect of the present invention is that the present invention optimizes the parameter utilization efficiency of the model through the MoPE module, so that in different medical image segmentation tasks, it can dynamically select suitable experts for processing according to the characteristics of the input data, thereby significantly reducing the computational cost and improving the segmentation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following introduction is made to the drawings of the embodiments of the present invention or the related technical solutions in the prior art. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative work.

[0039] Figure 1 is a diagram showing the overall structure of an image segmentation model in an embodiment of the present invention;

[0040] Figure 2 is an architectural diagram of a Transformer block in an embodiment of the present invention;

[0041] Figure 3 2 is a structural diagram of the MoPE module in an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application and are not to be construed as limiting the present application. For the step numbers in the following embodiments, they are provided only for the convenience of explanation and are not intended to limit the order of the steps. The order of execution of the steps in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0043] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to limit the embodiments of the present application. The singular forms of "a", "said", and "the" used in the embodiments of the present application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. In addition, unless otherwise clearly defined, words such as setting, installing, and connecting should be understood in a broad sense, and those skilled in the art can reasonably determine the specific meanings of the above words in the present invention in combination with the specific content of the technical solution.

[0044] In the description of this application, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, they cannot be understood as limitations on this application.

[0045] In the description of this application, "several" means one or more, "many" means more than two, "greater than," "less than," and "exceed" are understood to exclude the number itself, while "above," "below," and "within" are understood to include the number itself. The terms "first" and "second" are used solely to distinguish technical features and are not to be construed as indicating or implying relative importance, or as implicitly specifying the number or order of the technical features indicated.

[0046] In the description of this application, "and / or" describes the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0047] Explanation of terms:

[0048] MoE: abbreviation of Mixture of Experts, mixed expert model.

[0049] MoPE: abbreviation of Mixture of Prototype Experts, mixed prototype expert model.

[0050] SAM-ViT-Base: The abbreviation of Segment Anything Model-Vision in Transformer-Base, which refers to the Segment Anything Model with the Vision in Transformer Base model as the backbone network.

[0051] PEFT: Parameter-Efficient Fine-Tuning, efficient parameter fine-tuning.

[0052] SAM: Segment Anything Model, an advanced image segmentation model.

[0053] In the field of medical image segmentation, especially the high training cost of large models, and the poor adaptability of existing models to different types of medical images (such as CT, MRI, fundus color photos, etc.), making it difficult to achieve effective cross-domain segmentation limitations and challenges, the present invention aims to establish an efficient multimodal medical image segmentation model. By introducing a mixture of experts (MoE) architecture, the parameter utilization efficiency of the model is optimized, so that it can dynamically select suitable experts for processing according to the characteristics of the input data in different medical image segmentation tasks, thereby significantly reducing the computational cost and improving the segmentation accuracy. Specifically, the present invention aims to solve the following technical problems:

[0054] 1) Improve model segmentation accuracy: Optimize the algorithm to achieve higher segmentation accuracy in multiple organs and complex backgrounds.

[0055] 2) Enhanced adaptability: Improve the algorithm's adaptability to medical images of various modalities to ensure good segmentation results in different imaging forms.

[0056] 3) Improve computational efficiency: By dynamically selecting experts and reducing unnecessary computations, the effect of large-scale medical image processing can be effectively improved by increasing only a very small amount of computing power.

[0057] The present invention aims to provide reliable technical support for disease monitoring and treatment by achieving high-precision multimodal medical image segmentation, promote the intelligence and automation of medical clinical practice, and thus improve the efficiency and accuracy of patient diagnosis and treatment.

[0058] Example 1

[0059] This embodiment provides a general medical segmentation method based on a hybrid expert model, comprising the following steps:

[0060] S1, obtain the image to be segmented;

[0061] S2. Input the image to be segmented into the trained image segmentation model and output the segmentation result;

[0062] The image segmentation model includes an image encoder, an image decoder and a prompt encoder. The image encoder is used to extract features from the input image to obtain image features; the prompt encoder is used to extract features from the input user prompts to obtain prompt features; the image decoder is used to generate an image mask based on the image features and prompt features; in the Transformer block of the image encoder, the MoPE module is used to replace the attention module, and a suitable expert is dynamically selected for processing based on the features of the input data.

[0063] The above method is explained in detail below with reference to the accompanying drawings and specific embodiments.

[0064] (1) Data preprocessing

[0065] This example uses seven datasets: ADAM, PALM, REFUGE, HMC-QU, ITIS2016, LITS2017, and ACDC as training data. Table 1 describes the corresponding data sets.

[0066] Table 1

[0067]

[0068]

[0069] For 2D images, resize the image to 256*256 resolution. For 3D images, resize the image to 256*256 resolution after slicing.

[0070] (2) Image segmentation model

[0071] See also Figure 1 The image segmentation model of this embodiment includes an image encoder, an image decoder and a prompt encoder. The image encoder is used to extract features of the input image to obtain image features; the prompt encoder is used to extract features of the input user prompt to obtain prompt features; and the image decoder is used to generate an image mask based on the image features and the prompt features.

[0072] Specifically, the image encoder includes an image embedding layer and multiple Transformer blocks. The image embedding layer performs a 2D convolution on the image with a kernel size of [16,16] and a stride of 16. The number of channels after the convolution is 768, that is, the RGB image with a resolution of 256*256 of [3,256,256] is converted into an image token sequence of [768,16,16]. This set of token sequences will be processed by multiple Transformer blocks, but the structure will not change. Ultimately, this structure will be merged with the encoded user prompt [768+x,16,16], where x depends on the number and type of prompts provided by the user. This set of sequences will be handed over to the decoder for decoding, and finally a mask image of [1,256,256] will be output.

[0073] In some embodiments, see Figure 2 The Transformer block in this embodiment includes a first-layer normalization, a MoPE module, a second-layer normalization, and a feedforward layer. Traditional Transformer blocks consist of three components: layer normalization, an attention module, and a feedforward layer. This embodiment replaces the attention module with the proposed MoPE module. The MoPE module also performs attention calculations.

[0074] Specifically, attention calculation depends on W Q ,W k ,W V ,W O The weights of the four modules. In traditional attention calculation, the weights of these four modules will not change due to changes in the image. Figure 3 ,After receiving the input of the module, the prototype routing ,will assign the image to different experts based on the matching degree between the ,image and the expert prototype, and the expert will modify ,W,. Q ,W V ,W O The weights of the three modules are modified and then the attention is calculated. k , because a large number of literatures show that W k The modification of W has no substantial improvement on the model performance. Q ,W V ,W O The modifications are different. In this way, the model achieves more specialized attention calculation for the image.

[0075] Among them, prototype routing is to compare the image sequence and the expert prototype through cosine similarity. First, each expert has an expert prototype and a set of W Q ,W V ,W OThe expert prototype and the image sequence have the same size, [768, 16, 16]. Therefore, cosine similarity can be calculated between the expert prototype and the image sequence. There are a total of 8 experts in the expert group. During the testing phase, the embodiment compares the image sequence with the eight expert prototypes, finds the expert group with the highest similarity, and modifies W according to the modification weight of this expert group. Q ,W V ,W O , followed by attention calculation.

[0076] (2.1) SAM model

[0077] SAM is an advanced image segmentation model. Designed to achieve efficient and accurate image segmentation using simple inputs (such as points, boxes, or masks), SAM is applicable to a wide range of computer vision tasks. Its high efficiency and high-quality segmentation capabilities have made it suitable for a wide range of computer vision tasks and have advanced the state of the art in image segmentation technology.

[0078] (2.2) Mixture of Experts (MoE)

[0079] MoE is a technique that can significantly scale up models while maintaining a consistent computational budget. It aims to enhance the overall model's expressive power by combining multiple expert models. Each expert can focus on a specific task or data distribution, thereby improving the model's performance on complex tasks.

[0080] (2.3) Prototype Router

[0081] Prototype routing is a routing component that implements expert specialization by leveraging learnable prototypes. Each expert maintains a prototype p e ∈R {n×d} , image tags are obtained by cosine similarity s e = cos(X,pe) matches the most relevant experts and only activates the experts with the highest scores to retain fine-grained features.

[0082] Furthermore, traditional MoE routing suffers from parameter coupling (joint backbone router updates) and reliance on auxiliary balancing losses, which can conflict with the primary mission objective. This example proposes a prototype-centric routing update strategy that decouples optimization through learnable cluster centers in the embedding space. Each expert dynamically updates its prototype using an exponential moving average (EMA, weight = 0.5) and its assigned data centroid, eliminating the auxiliary loss function while ensuring load balancing.

[0083] (3) Model training

[0084] This embodiment selects the SAM benchmark large model for subsequent fine-tuning. SAM has powerful image segmentation capabilities and is particularly suitable for processing multimodal images.

[0085] The pre-trained weights of SAM are trained on natural images. In order to allow the model to learn the relevant features of each medical modality image while avoiding excessive parameter size, this embodiment uses the LoRA matrix as the expert in the MoPE module and uses the MoPE module to replace the original attention module. The MoPE module is an attention module based on the hybrid expert model designed in this embodiment. It contains 1 prototype routing and 4 expert groups. Each expert group contains three LoRA matrices, which correspond to the projection matrix W corresponding to Q, V, and O in the attention. Q ,W V ,W O Make adjustments; routing is responsible for distributing the input data to different experts. Specifically, it can be calculated as follows:

[0086]

[0087] Where X is the input of the module, g e represents the routing score of the e-th expert, i represents the i-th attention head, and d k Represents K i dimension.

[0088] To determine which group of experts should be selected for attention calculation, this embodiment designs a new prototype routing. The prototype routing accepts input and compares the cosine similarity with the stored expert prototypes to determine which group of experts to assign the input to. Specifically, it can be calculated as follows:

[0089]

[0090] Score=[CosSim(p 1 ,X),...,CosSim(p 7 ,X)]

[0091] Among them, CosSim(p e ,X) is used to calculate p e and X, and TOP1 is an operation that sets all elements in a vector to zero except the element with the first maximum value.

[0092] To ensure smooth data distribution to the corresponding experts, the expert prototypes stored in the router need to be updated during the training phase. Each expert dynamically updates its prototype using an exponential moving average (EMA, weight = 0.5) and its assigned data centroid. The pseudo code is shown in Table 2 below:

[0093] Table 2

[0094]

[0095]

[0096] All parameters in the original image encoder were frozen, and the attention module of the Transformer block was replaced with the MoPE module proposed in this embodiment. During training, only the weights of the MoPE module in the image encoder were updated. The AdamW optimizer was used with an initial learning rate of 1e-4 for a total of 30 epochs, with the learning rate divided by 2 at the 5th and 10th epochs, and a batch size of 32. Weights outside the MoPE module were initialized using the pre-trained weights of SAM-ViT-Base. Positional encodings were scaled using the same method as for SAMMed2D. Prototypes in the MoPE module were randomly initialized. Then, all layers, including the SAM layer, were frozen, and the prototype routes were updated for one round using the prototype update strategy described above, with a batch size of 128, as a warm-up phase. After this, the formal fine-tuning process began. All layers in the SAM model were frozen, and only the MoPE module was updated. Furthermore, during the first 10 epochs, the general expert and routes were updated in odd-numbered epochs, and the special expert in even-numbered epochs. Over the next 20 epochs, the entire MoPE module was updated.

[0097] (3.1) Low-Rank Adaptation

[0098] LoRA is an efficient parameter fine-tuning technology that aims to achieve efficient adaptation and fine-tuning by performing low-rank decomposition on the weights of pre-trained models. LoRA can significantly reduce the number of parameters and computing resources required for fine-tuning while preserving the original performance of the model.

[0099] (4) Loss function

[0100] The following loss function is used to enable the model to achieve our training purpose.

[0101] (4.1)Dice Loss

[0102] Dice Loss is a loss function commonly used in image segmentation tasks, especially in fields such as medical image analysis. It is derived from the Dice coefficient, which is a metric used to measure the similarity between two samples and is particularly suitable for imbalanced class problems.

[0103] The calculation formula of Dice coefficient is:

[0104]

[0105] Among them, A is the predicted segmentation area and B is the actual segmentation area. The value of this coefficient is between 0 and 1, where 1 means complete overlap and 0 means no overlap.

[0106] Dice Loss is defined based on the Dice coefficient and is usually expressed as:

[0107] L Dice =1-Dice(A,B)

[0108] (4.2) Cross-Entropy Loss

[0109] The goal of semantic segmentation is to classify each pixel in an image into a specific category, so cross-entropy loss can be effectively used to evaluate the classification performance of the model at the pixel level. Cross-entropy loss is usually used to measure the difference between the probability distribution of each pixel predicted by the model and the true label.

[0110]

[0111] Where: N is the total number of pixels in the image; C is the number of categories; y i,c is the one-hot encoding of the true label. For pixel i and category c, if the pixel belongs to category c, then y i,c =1, otherwise y i,c =0. is the model’s predicted probability that pixel i belongs to category c.

[0112] (4.3) Total loss function

[0113] The final loss function is obtained by adding Dice Loss and Cross-Entropy Loss, that is:

[0114] L=L Dice +L ce

[0115] (5) Advantages and beneficial effects

[0116] The universal image segmentation method based on the hybrid expert model of the present invention solves the key challenge of multimodal image segmentation in the field of medical image segmentation. Compared with the existing technology, it has at least the following advantages and benefits:

[0117] 1) This paper proposes a universal segmentation model suitable for multimodal medical images, which not only improves the accuracy of diagnosis and treatment, but also provides important support for early detection of diseases, personalized treatment and prognosis assessment.

[0118] 2) This paper proposes a hybrid expert model, called MoPE, for the attention module. By assigning prototype routes, it effectively assigns data from different modalities to corresponding experts for processing. By activating the corresponding experts, the model achieves better performance while maintaining the same number of parameters during the inference phase.

[0119] 3) This invention enables the continued training of newly added modal data based on an already trained model without causing catastrophic forgetting. By adding experts to the MoPE training, the model can quickly adapt to new segmentation tasks, fully leveraging existing knowledge and significantly improving the model's segmentation performance, thus providing a reliable solution for clinical applications.

[0120] 4) This invention can adapt to the feature distribution of different modalities, capturing more comprehensive and detailed information about organs and lesion structures, thereby improving segmentation accuracy and robustness. It also enhances the ability to identify complex lesions and provides more accurate and reliable medical segmentation results for clinical use.

[0121] 5) Effectively improve cross-modal segmentation performance: Large models inherently possess strong generalization capabilities. Compared to traditional convolutional segmentation networks, large models can share and leverage features across different modalities. This feature sharing helps the model extract more representative features when processing cross-modal data, thereby improving segmentation performance. The Mixed Expert Model (MEM) uses routing to assign features from different modalities to corresponding experts for processing, reducing the required computation and model parameters.

[0122] 6) Efficient Prototype Routing: Learnable expert prototypes automatically encode the feature distributions of different imaging modalities (such as CT, MRI, and ultrasound). Prototype routing driven by cosine similarity achieves cross-modal feature decoupling. Compared to traditional FFN-MoE routing, this approach offers lower computational complexity, easier training, and greater interpretability.

[0123] 7) Scalability for Continuous Learning: MoPE's modular design and prototype routing mechanism give it significant scalability in transfer learning scenarios. When faced with a new task, simply add a new task-specific expert to the existing expert pool, freeze the original experts and backbone network, and train only the newly added experts and their associated prototype routing parameters. This maintains the performance stability of historical tasks while expanding performance for new tasks. This scalability not only accelerates the model training process but also enables the model to achieve reliable performance during iteration, providing a practical solution for clinical applications.

[0124] Example 2

[0125] An embodiment of the present invention also provides an electronic device, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement a general medical segmentation method based on a hybrid expert model as described in Example 1.

[0126] It is understood that the memory may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory may be used to store instructions, programs, codes, code sets, or instruction sets. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the various method embodiments described above, etc.; the data storage area may store data created based on the use of the server, etc.

[0127] The processor may include one or more processing cores. The processor utilizes various interfaces and circuits to connect various components within the server. It executes various server functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory, as well as accessing data stored in memory. Optionally, the processor may be implemented using at least one of the following hardware forms: digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor may integrate one or a combination of a central processing unit (CPU) and a modem. The CPU primarily processes the operating system and application programs, while the modem handles wireless communications. It is understood that the modem may not be integrated into the processor and may be implemented separately via a single chip.

[0128] Since the electronic device is an electronic device corresponding to a general medical segmentation method based on a hybrid expert model in an embodiment of the present invention, and the principle of solving the problem by the electronic device is similar to that of the method, the implementation of the electronic device can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0129] Example 3

[0130] An embodiment of the present invention also provides a computer-readable storage medium, which stores at least one instruction, at least one program, code set or instruction set. The at least one instruction, the at least one program, the code set or instruction set is loaded and executed by a processor to implement a general medical segmentation method based on a hybrid expert model as described in Example 1.

[0131] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program. The program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0132] Since the storage medium is a storage medium corresponding to a general medical segmentation method based on a hybrid expert model in an embodiment of the present invention, and the principle of solving the problem by the storage medium is similar to that of the method, the implementation of the storage medium can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0133] Example 4

[0134] In some possible implementations, various aspects of the methods of the embodiments of the present invention may also be implemented in the form of a program product, which includes program code. When the program product is executed on a computer device, the program code is used to cause the computer device to perform the steps of a general medical segmentation method based on a hybrid expert model according to various exemplary embodiments of the present application described above in this specification. The executable computer program code or "code" for performing each embodiment may be written in a high-level programming language such as C, C++, Python, Smalltalk, Java, JavaScript, Visual Basic, Structured Query Language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0135] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0136] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0137] The above embodiments are intended only to illustrate the technical concepts and features of the present invention. Their purpose is to enable those skilled in the art to understand the contents of the present invention and implement them accordingly. They are not intended to limit the scope of protection of the present invention. Any equivalent changes or modifications made based on the essence of the present invention are intended to be covered by the scope of protection of the present invention.

Claims

1. A general medical segmentation method based on a hybrid expert model, characterized in that: The following steps are involved: Obtain the image to be segmented; Input the image to be segmented into the trained image segmentation model and output the segmentation result; The image segmentation model includes an image encoder, an image decoder and a prompt encoder. The image encoder is used to extract features from the input image to obtain image features; the prompt encoder is used to extract features from the input user prompts to obtain prompt features; the image decoder is used to generate an image mask based on the image features and prompt features; in the Transformer block of the image encoder, the MoPE module is used to replace the attention module, and a suitable expert is dynamically selected for processing based on the features of the input data.

2. A general medical segmentation method based on a hybrid expert model according to claim 1, characterized in that: The step of obtaining the image to be segmented comprises: For 2D images, resize the image to the preset resolution; for 3D images, resize the image to the preset resolution after slicing.

3. A general medical segmentation method based on a hybrid expert model according to claim 1, characterized in that: The image encoder includes an image embedding layer and a plurality of Transformer blocks; The image embedding layer is used to convolve the image. The number of channels after convolution is 768, that is, the RGB image with a resolution of 256*256 is converted into an image token sequence of [768,16,16]. The token sequence will be processed by multiple Transformer blocks, but the structure will not change. Finally, the output of the image encoder is merged with the user prompt output by the prompt encoder [768+x,16,16], where x depends on the number and type of prompts provided by the user. This sequence is then passed to the image decoder for decoding, outputting a [1,256,256] mask image.

4. A general medical segmentation method based on a hybrid expert model according to claim 1, characterized in that: The Transformer block includes a first layer normalization, a MoPE module, a second layer normalization, and a feed-forward layer; The MoPE module includes a prototype router and multiple expert groups. Each expert group includes three LoRA matrices, which are the projection matrices W corresponding to Q, V, and O in the attention. Q ,W V ,W O Adjustment is made; the prototype routing is responsible for distributing the input data to different experts.

5. A general medical segmentation method based on a hybrid expert model according to claim 4, characterized in that: The calculation formula of the MoPE module is as follows: Where X is the input of the MoPE module; g e represents the routing score of the e-th expert; Concat represents the connection function, W O is the projection matrix of the Output module, is the correction matrix of the Output module of the e-th expert, E is the number of experts; Head i represents the i-th attention head, H is the number of attention heads; Q i is the Query of the i-th attention head, K i is the Key of the i-th attention head, V i is the Value of the i-th attention head, d k Represents K i The dimension of , T is the transpose; is the projection matrix of the Query module of the i-th attention head, is the correction matrix of the Query module of the e-th expert in the i-th attention head, is the projection matrix of the Value module of the i-th attention head, is the correction matrix of the Value module of the e-th expert in the i-th attention head, is the downsampling matrix of the Output module of the e-th expert, is the upsampling matrix of the Output module of the e-th expert, is the downsampling matrix of the Query module of the e-th expert in the i-th attention head, is the upsampling matrix of the Query module of the e-th expert in the i-th attention head, is the downsampling matrix of the Value module of the e-th expert in the i-th attention head, is the upsampling matrix of the Value module of the e-th expert in the i-th attention head.

6. A general medical segmentation method based on a hybrid expert model according to claim 4, characterized in that: The prototype router determines which group of experts to assign the input to by accepting the input and comparing it with the expert prototypes stored in the prototype router. The calculation formula is as follows: Score=[CosSim(p 1 ,X),...,CosSim(p 7 ,X)] Where, CosSim(p e ,X) is used to calculate p e and X; TOP1 is an operation that sets all elements in a vector to zero except the element with the first maximum value; X is the input of the MoPE module; p e is the expert prototype of the e-th expert, Score e is the similarity score of the e-th expert.

7. A general medical segmentation method based on a hybrid expert model according to claim 1, characterized in that: During training, only the weights of the MoPE module in the image encoder are updated; for weights outside the MoPE module, they are initialized using the pre-trained weights of SAM ViT Base; the positional encoding is scaled in the same way as SAMMed2D; the prototypes in the MoPE module are randomly initialized; then, all layers, including the SAM layer, are frozen, and a round of prototype routing is updated using the prototype update strategy.

8. The general medical segmentation method based on a hybrid expert model according to claim 1, characterized in that: The loss function during the image segmentation model training process is: L=L Dice +L ce L Dice =1-Dice(A,B) Where A is the predicted segmentation area, B is the actual segmentation area; N is the total number of pixels in the image, C is the number of categories, and y i,c is the one-hot encoding of the true label, is the model’s predicted probability that pixel i belongs to category c.

9. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that The storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method according to any one of claims 1 to 8.

Images