Multi-organ medical image segmentation method and device based on auxiliary network generation prompt

Abstract

The invention relates to the technical field of medical image processing, in particular to a multi-organ medical image segmentation method and device based on an auxiliary network generation prompt, and the method comprises the steps: obtaining a multi-organ medical image, inputting the multi-organ medical image into a pre-constructed auxiliary network, the auxiliary network outputs a first segmentation result of the multi-organ medical image; generating an input prompt box of the multi-organ medical image according to the first segmentation result; inputting the multi-organ medical image and the input prompt box into a pre-constructed visual basic model, wherein the visual basic model outputs a second segmentation result of the multi-organ medical image; and constructing a prediction fusion module based on pixel-by-pixel weighting, and performing pixel-by-pixel weighting fusion on the first segmentation result, the second segmentation result and the multi-organ medical image by using the prediction fusion module to obtain a final segmentation result. Therefore, the problem that the segmentation performance is severely reduced due to errors existing in automatic generation of input prompts in related technologies is solved.

Description

Technical Field

[0001] The present application relates to the field of medical image processing technology, and in particular to a multi-organ medical image segmentation method and device based on auxiliary network-generated prompts. Background Art

[0002] Medical image segmentation is a critical task in clinical diagnosis and treatment, aiming to accurately delineate the contours of organs or tumors. Manual segmentation performed by physicians has significant drawbacks, such as time-consuming processing and inconsistent and fluctuating segmentation standards. In contrast, automated segmentation algorithms demonstrate significant advantages in efficiency, accuracy, and consistency, and have been proven in clinical practice.

[0003] In recent years, deep learning-based segmentation algorithms have achieved significant advancements in medicine. However, they are typically designed for specific body parts or diseases, and lack adaptability to diverse tasks and data. SAM (Segment Anything Model), an emerging vision-based model, has demonstrated strong segmentation accuracy and generalization capabilities across various medical segmentation tasks after fine-tuning on medical data. As an interactive model, SAM requires only rough input prompts for segmentation. However, input prompts for medical segmentation tasks often require expert knowledge to ensure reliability and accuracy, limiting SAM's widespread application in the medical field. Summary of the Invention

[0004] The present application provides a multi-organ medical image segmentation method and device based on auxiliary network generation prompts to solve the problem that errors in the automatic generation of input prompts in related technologies can lead to serious degradation of segmentation performance.

[0005] The first aspect of the present application provides a multi-organ medical image segmentation method based on auxiliary network-generated prompts, comprising the following steps: acquiring a multi-organ medical image, inputting the multi-organ medical image into a pre-constructed auxiliary network, and the auxiliary network outputting a first segmentation result of the multi-organ medical image; generating an input prompt box for the multi-organ medical image based on the first segmentation result; inputting the multi-organ medical image and the input prompt box into a pre-constructed visual basic model, and the visual basic model outputting a second segmentation result of the multi-organ medical image; constructing a prediction fusion module based on pixel-by-pixel weighting, and using the prediction fusion module to perform pixel-by-pixel weighted fusion on the first segmentation result, the second segmentation result, and the multi-organ medical image to obtain a final segmentation result.

[0006] Optionally, the auxiliary network is formulated as:

[0007] y t =f(x t ,y t-1 )=Decoder(Encoder(x t))fort∈{1,2,…,T}

[0008] Among them, x t represents the input of stage t; y t is the output of the tth stage; T is the total number of segmentation stages, Encoder(·) is the encoder network of the auxiliary network; Decoder(·) is the decoder network of the auxiliary network.

[0009] Optionally, an input prompt box for a multi-organ medical image is generated based on the first segmentation result, including: traversing the non-zero foreground masks of all foreground categories of all images from the first segmentation result; extracting all existing foreground targets based on the non-zero foreground masks, and using a morphological erosion algorithm to filter the boundaries and details of areas other than the foreground targets in the image of the first segmentation result; using a morphological dilation algorithm to restore the original segmented area and details to obtain an optimized segmented area; extracting connected areas based on the optimized segmented area, generating an input prompt box based on the connected areas, and performing a non-maximum suppression operation on a subset of input prompt boxes of the same category in the same image.

[0010] Optionally, the formula for generating the input prompt box is:

[0011]

[0012] Among them, (x min ,y min ,x max ,y max ) is the coordinate of the input prompt box; R i ∈R (b,c) For each connected area; b is the index of the picture to which the box belongs; c is the index of the category to which the box belongs; I is the index record table of the picture to which the box belongs and the index record table of the category to which it belongs; Box is the list of box coordinate records.

[0013] Optionally, the fusion process of the pixel-by-pixel weighted prediction fusion module includes: splicing the first segmentation result, the second segmentation result and the multi-organ medical image in the channel dimension to obtain a fusion feature map, and generating a fusion weight map based on the fusion feature map; performing a softmax operation on the fusion weight map to normalize the weight of each pixel; determining the respective weights of the first segmentation result and the second segmentation result based on the normalized weight of each pixel, and performing weighted fusion based on the first segmentation result, the second segmentation result, and their respective weights to obtain the final segmentation result.

[0014] Optionally, before inputting the multi-organ medical images and the input prompt box into the pre-built visual base model, it also includes: obtaining a training data set; training the visual base model using the training data set, and during the training process, iteratively updating the network parameters of the visual base model using a mixed loss function.

[0015] Optionally, the hybrid loss function is:

[0016]

[0017] Among them, α and β are weights, y0 is the first segmentation result; y final is the second segmentation result; To calculate Dice loss; To calculate the cross entropy loss; is the coarse segmentation loss; is the fusion loss; is the final prediction mask.

[0018] The second aspect of the present application provides a multi-organ medical image segmentation device based on auxiliary network generation prompts, including: a first segmentation module, used to obtain a multi-organ medical image, input the multi-organ medical image into a pre-built auxiliary network, and the auxiliary network outputs a first segmentation result of the multi-organ medical image; a generation module, used to generate an input prompt box of the multi-organ medical image according to the first segmentation result; a second segmentation module, used to input the multi-organ medical image and the input prompt box into a pre-built visual basic model, and the visual basic model outputs a second segmentation result of the multi-organ medical image; a fusion module, used to construct a prediction fusion module based on pixel-by-pixel weighting, and use the prediction fusion module to perform pixel-by-pixel weighted fusion on the first segmentation result, the second segmentation result and the multi-organ medical image to obtain a final segmentation result.

[0019] The third aspect of the present application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein the processor executes the program to implement the multi-organ medical image segmentation method based on auxiliary network generation prompts as in the above embodiment.

[0020] The fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the multi-organ medical image segmentation method based on auxiliary network generation prompts as described in the above embodiment.

[0021] Therefore, this application has the following beneficial effects:

[0022] The embodiment of the present application obtains a first segmentation result by inputting a multi-organ medical image into a pre-built auxiliary network, generates an input prompt box for the multi-organ medical image based on the first segmentation result, then inputs the multi-organ medical image and the input prompt box into a pre-built visual base model, which outputs a second segmentation result for the multi-organ medical image. A prediction fusion module based on pixel-by-pixel weighting is constructed, and the prediction fusion module is used to perform pixel-by-pixel weighted fusion on the first segmentation result, the second segmentation result, and the multi-organ medical image to obtain a final segmentation result, providing high-quality input prompts and achieving ideal final segmentation performance. This solves the problem that errors in the automatic generation of input prompts in related technologies can lead to serious degradation of segmentation performance.

[0023] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0025] Figure 1 A flowchart of a multi-organ medical image segmentation method based on auxiliary network-generated prompts provided according to an embodiment of the present application;

[0026] Figure 2 A flowchart of a SAM multi-organ medical image segmentation method based on auxiliary network generation prompts provided according to one embodiment of the present application;

[0027] Figure 3 This is an architecture diagram of a prediction fusion module provided according to one embodiment of the present application;

[0028] Figure 4 This is a block diagram of an exemplary apparatus for multi-organ medical image segmentation based on auxiliary network-generated prompts according to an embodiment of the present application;

[0029] Figure 5 A schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] The following describes in detail embodiments of the present application. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0031] The following describes the multi-organ medical image segmentation method and device based on the auxiliary network generation prompt according to the embodiment of the present application with reference to the accompanying drawings. In response to the problem mentioned in the above background technology that the related art automatically generates input prompts with errors, which will lead to a serious decline in segmentation performance, the present application provides a multi-organ medical image segmentation method based on the auxiliary network generation prompt. In this method, a multi-organ medical image is input into a pre-constructed auxiliary network to obtain a first segmentation result, and an input prompt box of the multi-organ medical image is generated according to the first segmentation result. The multi-organ medical image and the input prompt box are then input into a pre-constructed visual basic model, and the visual basic model outputs a second segmentation result of the multi-organ medical image. A prediction fusion module based on pixel-by-pixel weighting is constructed, and the prediction fusion module is used to perform pixel-by-pixel weighted fusion on the first segmentation result, the second segmentation result and the multi-organ medical image to obtain a final segmentation result, providing high-quality input prompts and achieving ideal final segmentation performance. Thus, the problem that the related art automatically generates input prompts with errors, which will lead to a serious decline in segmentation performance, is solved.

[0032] Specifically, Figure 1 A flowchart of a multi-organ medical image segmentation method based on auxiliary network generation prompts provided in an embodiment of the present application.

[0033] like Figure 1 As shown, the multi-organ medical image segmentation method based on auxiliary network generation prompts includes the following steps:

[0034] In step S101, a multi-organ medical image is acquired, and the multi-organ medical image is input into a pre-built auxiliary network, which outputs a first segmentation result of the multi-organ medical image.

[0035] Among them, multi-organ medical images refer to medical imaging data containing multiple organs of the human body, which can be obtained through different imaging technologies, such as CT (Computed Tomography) and MRI (Magnetic Resonance Imaging); segmentation refers to dividing different tissues or organs in the image according to boundaries to facilitate subsequent analysis and diagnosis.

[0036] It is understandable that, in the embodiment of the present application, the multi-organ medical image to be processed is first input into a pre-built auxiliary network, which can perform a rough segmentation on the multi-organ medical image and output a first segmentation result of the multi-organ medical image. The auxiliary network is specifically as follows:

[0037] In the embodiment of the present application, the formula of the auxiliary network is:

[0038] y t =f(x t ,yt-1 )=Decoder(Encoder(x t ))fort∈{1,2,…,T}

[0039] Among them, x t represents the input of stage t; y t is the output of the tth stage; T is the total number of segmentation stages; Encoder(·) is the encoder network of the auxiliary network; Decoder(·) is the decoder network of the auxiliary network; the auxiliary network has a U-shaped structure with added skip connections. The U-shaped structure refers to a specific type of neural network architecture characterized by two main components: a contracting path (downsampling path) and an expanding path (upsampling path). The contracting path is used to capture contextual information in the image (i.e., feature extraction), while the expanding path is used for precise positioning (restoring the spatial size of the object). This structure can maintain high accuracy when performing pixel-level classification tasks. Skip connections are direct connections between different layers of the network, bypassing some intermediate layers, which can reduce information loss and enhance the ability to capture details.

[0040] In step S102, an input prompt box of the multi-organ medical image is generated according to the first segmentation result.

[0041] Among them, the input prompt box refers to the area identifier automatically generated based on the first segmentation result to guide subsequent processing or analysis. It is a rectangular or other shaped box on the graphical user interface, which is used to identify areas that require special attention in multi-organ medical images, such as the location and range of one or more organs.

[0042] It is understandable that the embodiment of the present application utilizes the first segmentation result to generate a stable input prompt box on a multi-organ medical image, which can highlight the position and boundary of each organ. The specific generation method is as follows:

[0043] In an embodiment of the present application, an input prompt box for a multi-organ medical image is generated based on a first segmentation result, including: traversing non-zero foreground masks of all foreground categories of all images from the first segmentation result; extracting all existing foreground targets based on the non-zero foreground mask, and using a morphological erosion algorithm to filter the boundaries and details of areas outside the foreground targets in the image of the first segmentation result; using a morphological dilation algorithm to restore the original segmented area and details to obtain an optimized segmented area; extracting connected areas based on the optimized segmented area, generating an input prompt box based on the connected areas, and performing a non-maximum suppression operation on a subset of input prompt boxes of the same category in the same image.

[0044] Among them, from the first segmentation result, the non-zero foreground masks of all foreground categories of all pictures are traversed, and for each batch of auxiliary network segmentation maps S∈R(B×C×H×W) , each foreground category in each picture extracts the possible foreground target, the formula is:

[0045] M (b,c) =S[b,c,:,:]forb∈1,2,…,B,c∈1,2,…,C,st|M (b,c) |0>0, where the batch size of the segmentation map is B, and the total number of categories including the background category (the background category is 0) is C; all existing foreground targets are extracted according to the non-zero foreground mask, and |M (b,c) |0>0 filters whether there is a foreground target related to category c in the input image; use the morphological erosion algorithm to filter the boundaries and details of the area other than the foreground target in the image of the first segmentation result, which is achieved by the following formula:

[0046] Where erode(·) is the morphological operation function erosion function; the morphological expansion algorithm is used to restore the original segmentation area and details, and the formula Where dilate(·) is the morphological operation function expansion function; based on the optimized segmented region, the connected region is extracted and the region attributes are calculated, which is achieved through the following two formulas:

[0047]

[0048] Among them L (b,c) is the extracted connected area; R (b,c) is the regional attribute; connected_components(·) is the connected region detection and labeling function, and region_properties(·) is the connected region attribute analysis function; the non-maximum suppression operation is a technique used to select the best bounding box to avoid repeated segmentation of the same target. The specific formula is Box,I=NMS(Box,I,τ IoU ), where Box is the box coordinate record table, I is the index record table of the image to which the box belongs and the index record table of the category to which it belongs, NMS(·) is the non-maximum suppression operation function, τ IoU The threshold for the input prompt box is pre-set according to actual needs and is not specifically limited here.

[0049] It can be understood that the embodiment of the present application can use the above formula to traverse all non-zero foreground masks belonging to the foreground category in all images, then use these non-zero foreground masks to extract all existing foreground targets, and use the morphological erosion algorithm to remove the boundary and detail noise in the external area of ​​the foreground target, and then apply the morphological expansion algorithm to restore the original segmented area and its details reduced by the erosion process to obtain the optimized segmented area. Based on this optimization result, the connected areas are further identified and extracted, and the input prompt boxes are generated according to these connected areas. Finally, the non-maximum suppression operation is performed on multiple prompt boxes of the same category in the same image to avoid repeated segmentation of the same target, ensuring that the final prompt box set is more accurate and concise.

[0050] In the embodiment of the present application, the formula for generating the input prompt box is:

[0051]

[0052] Among them, (x min ,y min ,x max ,y max ) is the coordinate of the input prompt box; R i ∈R (b,c) For each connected area; b is the index of the picture to which the box belongs; c is the index of the category to which the box belongs; I is the index record table of the picture to which the box belongs and the index record table of the category to which it belongs; Box is the list of box coordinate records.

[0053] It can be understood that the embodiment of the present application identifies and extracts connected areas through the above formula, generates input prompt boxes based on these connected areas, records the box coordinates in the list box, and obtains

[0054] In step S103 , the multi-organ medical image and the input prompt box are input into a pre-built visual basic model, and the visual basic model outputs a second segmentation result of the multi-organ medical image.

[0055] Among them, the pre-built visual basic model can be SAM, which has strong versatility and flexibility. It can not only perform segmentation tasks based on clear input prompt boxes, but also autonomously find and segment all objects in the image.

[0056] It can be understood that the embodiment of the present application inputs the multi-organ medical image and the input prompt box into a pre-built visual basic model such as the SAM model, and uses the SAM model to more accurately identify and segment the multi-organ medical image through the input prompt box, thereby obtaining a second segmentation result of the multi-organ medical image.

[0057] In an embodiment of the present application, before inputting the multi-organ medical image and the input prompt box into the pre-built visual basic model, it also includes: obtaining a training data set; using the training data set to train the visual basic model, and during the training process, using a mixed loss function to iteratively update the network parameters of the visual basic model.

[0058] Among them, the training dataset is the real segmentation mask, which is the key data used for supervised learning in image segmentation tasks. It provides a standard reference by marking the precise boundaries and areas of the target objects in each image. It usually requires a combination of manual annotation and automated tools to obtain.

[0059] It is understandable that, before inputting the multi-organ medical images and the input prompt box into the pre-built visual base model, the embodiment of the present application first needs to obtain a training data set consisting of real segmentation masks for training the visual base model. During the training process, a hybrid loss function is used to evaluate the difference between the model segmentation result and the real segmentation mask, and the network parameters of the model are iteratively updated based on this difference to optimize its performance. The hybrid loss function is specifically as follows:

[0060] In the embodiment of the present application, the hybrid loss function is:

[0061]

[0062] Among them, α and β are weights, y0 is the first segmentation result; y final is the second segmentation result; To calculate Dice loss; To calculate the cross entropy loss; is the coarse segmentation loss; is the fusion loss; is the final prediction mask.

[0063] It can be understood that the embodiment of the present application calculates the Dice loss and cross entropy loss of the first segmentation result and the final segmentation result through the above formula, combines the respective Dice loss and cross entropy loss according to the weights to obtain the coarse segmentation loss and the fusion loss, adds the two to obtain the final prediction mask, and uses the hybrid loss function to evaluate the difference between the final prediction mask and the true segmentation mask, and iteratively updates the network parameters of the model based on this difference, so that the visual base model can efficiently and accurately process multi-organ medical images.

[0064] In step S104, a prediction fusion module based on pixel-by-pixel weighting is constructed, and the prediction fusion module is used to perform pixel-by-pixel weighted fusion on the first segmentation result, the second segmentation result, and the multi-organ medical image to obtain a final segmentation result.

[0065] Among them, the pixel-by-pixel weighted prediction fusion module is a specially constructed module whose function is to assign different weights to each pixel according to its importance or reliability, and then fuse these segmentation results to generate a more accurate and reliable final segmentation result.

[0066] It can be understood that the embodiment of the present application constructs a pixel-by-pixel weighted prediction fusion module, which can analyze the first segmentation result, the second segmentation result, and the original multi-organ medical image itself of the multi-organ medical image, and calculate appropriate weights for each pixel, and use these weights to fuse the information from the three sources pixel by pixel, thereby obtaining a more accurate and detailed final segmentation result, making full use of the global information of the preliminary segmentation and the local refined information of the subsequent optimized segmentation, while taking into account the original image data to ensure the quality and reliability of the segmentation results.

[0067] In an embodiment of the present application, the fusion process of the pixel-by-pixel weighted prediction fusion module includes: splicing the first segmentation result, the second segmentation result and the multi-organ medical image in the channel dimension to obtain a fusion feature map, and generating a fusion weight map based on the fusion feature map; performing a softmax operation on the fusion weight map to normalize the weight of each pixel; determining the respective weights of the first segmentation result and the second segmentation result based on the normalized weight of each pixel, and performing weighted fusion based on the first segmentation result, the second segmentation result, and their respective weights to obtain the final segmentation result.

[0068] Among them, the first segmentation result, the second segmentation result and the multi-organ medical image are spliced ​​in the channel dimension to obtain the fusion feature map, and the formula X′=concatenate(X,y0,y sam ) is achieved, where X is a multi-organ medical image, y0 is the first segmentation result, y sam is the second segmentation result, concatenate(·) is the operation of superimposing multiple feature maps along the channel dimension, X ′ is the fusion feature map; Softmax operation is a commonly used mathematical function, which is used here to normalize the weight of each pixel into a probability distribution so that the weight values ​​of all pixels fall within the range of [0,1] and the sum is 1. The weights of the first segmentation result and the second segmentation result are determined according to the normalized weight of each pixel. In the formula Indicates that the fused feature map is input into the U-Net (a U-shaped convolutional neural network architecture specifically designed for biomedical image segmentation) structure network, softmax(·) is the Softmax operation, and dim=1 means that the Softmax function is applied to The second dimension of the result, i.e., the column, is weighted fusion based on the first segmentation result, the second segmentation result, and their respective weights to obtain the final segmentation result, which is obtained by formula y final =W0·y0+W1·y sam Implementation, where W0 and W1 are the weights of the first segmentation result and the second segmentation result respectively obtained by the above calculation.

[0069] It can be understood that the pixel-by-pixel weighted prediction fusion module of the embodiment of the present application first splices the first segmentation result and the second segmentation result of the multi-organ medical image in the channel dimension to form a fusion feature map. Based on this fusion feature map, a fusion weight map is generated that reflects the relative importance of each pixel point among different segmentation results. Then, a softmax operation is performed on this fusion weight map to normalize the weight value of each pixel. For a certain pixel point, two weight values ​​can be obtained after normalization, which correspond to the weights of the first segmentation result and the second segmentation result respectively. The weights of the first segmentation result and the second segmentation result are determined according to the normalized weights. Finally, the first segmentation result and the second segmentation result are weightedly fused according to these weights to obtain a more accurate and detailed final segmentation result, which fully utilizes the advantages of preliminary segmentation and optimized segmentation, and combines the original image information to improve the quality and accuracy of the overall segmentation.

[0070] According to the multi-organ medical image segmentation method based on auxiliary network prompt generation proposed in the embodiment of the present application, a first segmentation result is obtained by inputting the multi-organ medical image into a pre-constructed auxiliary network, an input prompt box of the multi-organ medical image is generated according to the first segmentation result, and then the multi-organ medical image and the input prompt box are input into a pre-constructed visual basic model, the visual basic model outputs a second segmentation result of the multi-organ medical image, and a prediction fusion module based on pixel-by-pixel weighting is constructed. The prediction fusion module is used to perform pixel-by-pixel weighted fusion on the first segmentation result, the second segmentation result and the multi-organ medical image to obtain a final segmentation result, providing high-quality input prompts and achieving ideal final segmentation performance.

[0071] The following is a further description of the multi-organ medical image segmentation method based on the auxiliary network generation prompt through a specific embodiment. Figure 2 、 Figure 3 The SAM multi-organ medical image segmentation method based on auxiliary network generation prompts proposed in this embodiment is further described in detail. Figure 2 As shown, the method specifically includes the following steps:

[0072] Step 1. Auxiliary network coarse segmentation: In order to drive the main network segmentation, an auxiliary network is constructed to perform segmentation so as to generate the input prompts required by the main network. The auxiliary network has a U-shaped structure and adds jump connections, as shown in formula (1):

[0073] y t =f(x t ,y t-1 )=Decoder(Encoder(x t ))fort∈{1,2,…,T}(1)

[0074] Among them, x t represents the input of stage t, y t is the output of the tth stage, T is the total number of segmentation stages, Encoder(·) is the encoder network of the auxiliary network; Decoder(·) is the decoder network of the auxiliary network.

[0075] Step 2, coarse segmentation post-processing and input prompt generation: The segmentation results generated by the auxiliary network need to be further post-processed to obtain high-quality, conservative, and stable input prompt boxes. The input prompt post-processing module involves morphological operation functions erode(·) and dilate(·), and a threshold τ of the input prompt box is pre-set. IoU , the subdivision steps are as follows:

[0076] Step 2.1, extract foreground targets: for each batch of auxiliary network segmentation map S∈R (B×C×H×W) , each foreground category in each picture extracts the possible foreground targets, as shown in formula (2):

[0077] M (b,c) =S[b,c,:,:]forb∈1,2,…,B,c∈1,2,…,C,st|M (b,c) |0>0(2)

[0078] The batch size of the segmentation map is B, the total number of categories including the background category (the background category is 0) is C, |M (b,c) |0>0 is used to filter whether there is a foreground target related to category c in the input image.

[0079] Step 2.2: Perform morphological erosion operation: eliminate discrete point areas and edge jaggedness in the original segmentation map, so that the boundary coordinates of the prompt box obtained based on the segmentation map are more stable and will not be affected by low-quality segmentation edges, as shown in formula (3):

[0080]

[0081] Step 2.3: Use the morphological dilation operation to restore the normal segmentation boundary and filter out the fragmented and outlier segmentation areas, as shown in formula (4):

[0082]

[0083] Step 2.4: Extract the connected areas based on the filtered segmentation map. The process of extracting the connected areas and calculating the regional attributes is shown in formulas (5) and (6):

[0084]

[0085] R (b,c) =region_properties(L (b,c) ) (6)

[0086] Step 2.5: Based on each connected area R i ∈R (b,c) , generate the corresponding minimum adjacent bounding box, record the box coordinates in the list Box, and record the index b of the picture to which the box belongs and the index c of the category to which it belongs in the list I, as shown in formulas (7) and (8):

[0087]

[0088] Step 2.6: Perform NMS operation on all boxes of the same category in the same image to avoid repeated segmentation of the same target by the SAM model, as shown in formula (9):

[0089] Box,I=NMS(Box,I,τ IoU ) (9)

[0090] Step 3, Medical SAM Segmentation: Input the input prompt box generated in Step 2 and the original input image into the frozen Medical SAM model to generate a segmentation mask.

[0091] Step 4: Prediction feature fusion: There are obvious differences in details between the rough segmentation map of the auxiliary network and the SAM segmentation map. It is possible to further integrate them to improve the segmentation performance. A prediction fusion module based on pixel-by-pixel weighting is constructed, such as Figure 3 The specific operations are as follows:

[0092] Step 4.1: Concatenate the original input image, the input hint segmentation map, and the SAM segmentation map along the channel dimension to form a fused feature map, as shown in formula (10):

[0093] X′=concatenate(X,y0,y sam ) (10)

[0094] Concatenate represents the operation of superimposing multiple feature maps along the channel dimension.

[0095] Step 4.2: Input the fused feature map into the U-Net network. Generate a pixel-level fusion weight map through feature extraction and structure recovery. Apply the softmax operation to the fusion weight map to normalize the weight of each pixel for subsequent weighted fusion, as shown in formula (11):

[0096]

[0097] Step 4.3: The two weight maps correspond to the input prompt segmentation map and the SAM segmentation map respectively. After weighted fusion of the two segmentation maps, the final segmentation result y is obtained. final , as shown in formula (12):

[0098] y final =W0·y0+W1·y sam (12)

[0099] Step 5: Design a hybrid loss function and iteratively update the network.

[0100] A hybrid supervised loss function is designed for medical segmentation scenarios. The true segmentation mask (GT masks) is used as supervision information to update the network parameters. The specific process is as follows: the loss of the initial segmentation map output by the auxiliary network and the true mask is calculated using a combination of Dice loss and cross entropy loss, with weights of 0.5 and 0.5 respectively, that is, α = β = 0.5. Supervising the segmentation results of the auxiliary network and the final fusion network at the same time can more directly and effectively improve the segmentation quality of the auxiliary network, thereby indirectly improving the segmentation quality of SAM, and at the same time fine-tuning the results of the segmentation map fusion. As shown in formulas (13), (14), and (15):

[0101]

[0102] Next, a multi-organ medical image segmentation device based on auxiliary network generation prompts proposed in an embodiment of the present application is described with reference to the accompanying drawings.

[0103] Figure 4 It is a block diagram of a multi-organ medical image segmentation device based on auxiliary network generation prompts according to an embodiment of the present application.

[0104] like Figure 4 As shown, the multi-organ medical image segmentation device 10 based on auxiliary network generation prompts includes: a first segmentation module 201, a generation module 202, a second segmentation module 203 and a fusion module 204.

[0105] Among them, the first segmentation module 201 is used to obtain multi-organ medical images, input the multi-organ medical images into a pre-built auxiliary network, and the auxiliary network outputs the first segmentation result of the multi-organ medical images; the generation module 202 is used to generate an input prompt box of the multi-organ medical image based on the first segmentation result; the second segmentation module 203 is used to input the multi-organ medical image and the input prompt box into a pre-built visual basic model, and the visual basic model outputs the second segmentation result of the multi-organ medical image; the fusion module 204 is used to construct a prediction fusion module based on pixel-by-pixel weighting, and use the prediction fusion module to perform pixel-by-pixel weighted fusion on the first segmentation result, the second segmentation result and the multi-organ medical image to obtain the final segmentation result.

[0106] In the embodiment of the present application, the formula of the auxiliary network is:

[0107] y t =f(x t ,y t-1 )=Decoder(Encoder(x t ))fort∈{1,2,…,T}

[0108] Among them, x t represents the input of stage t; y t is the output of the tth stage; T is the total number of segmentation stages; Encoder(·) is; Decoder(·) is.

[0109] In an embodiment of the present application, the generation module 202 is further used to: traverse the non-zero foreground masks of all foreground categories of all pictures from the first segmentation result; extract all existing foreground targets based on the non-zero foreground masks, and use a morphological erosion algorithm to filter the boundaries and details of the areas outside the foreground targets in the picture of the first segmentation result; use a morphological dilation algorithm to restore the original segmented areas and details to obtain optimized segmented areas; extract connected areas based on the optimized segmented areas, generate input prompt boxes based on the connected areas, and perform non-maximum suppression operations on a subset of input prompt boxes of the same category in the same picture.

[0110] In the embodiment of the present application, the formula for generating the input prompt box is:

[0111]

[0112] Among them, (x min ,y min ,x max ,y max ) is the coordinate of the input prompt box; R i ∈R (b,c)For each connected area; b is the index of the picture to which the box belongs; c is the index of the category to which the box belongs; I is the index record table of the picture to which the box belongs and the index record table of the category to which it belongs; Box is the list of box coordinate records.

[0113] In an embodiment of the present application, the fusion module 204 is further used to: splice the first segmentation result, the second segmentation result and the multi-organ medical image in the channel dimension to obtain a fusion feature map, and generate a fusion weight map based on the fusion feature map; perform a softmax operation on the fusion weight map to normalize the weight of each pixel; determine the respective weights of the first segmentation result and the second segmentation result based on the normalized weight of each pixel, and perform weighted fusion based on the first segmentation result, the second segmentation result, and their respective weights to obtain the final segmentation result.

[0114] In the embodiment of the present application, the device of the embodiment of the present application further includes: a training module.

[0115] Among them, the training module is further used to: obtain a training data set before inputting multi-organ medical images and input prompt boxes into a pre-built visual base model; use the training data set to train the visual base model, and during the training process, use a mixed loss function to iteratively update the network parameters of the visual base model.

[0116] In the embodiment of the present application, the hybrid loss function is:

[0117]

[0118] Among them, α and β are weights; y0 is the first segmentation result; y final is the final segmentation result; To calculate Dice loss; To calculate the cross entropy loss; is the coarse segmentation loss; is the fusion loss; is the final prediction mask.

[0119] It should be noted that the above explanation of the embodiment of the multi-organ medical image segmentation method based on auxiliary network generation prompts is also applicable to the multi-organ medical image segmentation device based on auxiliary network generation prompts in this embodiment, and will not be repeated here.

[0120] According to the multi-organ medical image segmentation device based on auxiliary network prompt generation proposed in the embodiment of the present application, a first segmentation result is obtained by inputting the multi-organ medical image into a pre-constructed auxiliary network, an input prompt box of the multi-organ medical image is generated according to the first segmentation result, and then the multi-organ medical image and the input prompt box are input into a pre-constructed visual basic model, the visual basic model outputs a second segmentation result of the multi-organ medical image, and a prediction fusion module based on pixel-by-pixel weighting is constructed. The prediction fusion module is used to perform pixel-by-pixel weighted fusion on the first segmentation result, the second segmentation result and the multi-organ medical image to obtain a final segmentation result, thereby providing high-quality input prompts and achieving ideal final segmentation performance.

[0121] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0122] Memory 301 , processor 302 , and computer programs stored in the memory 301 and executable on the processor 302 .

[0123] When the processor 302 executes the program, the multi-organ medical image segmentation method based on auxiliary network generation prompts provided in the above embodiment is implemented.

[0124] Furthermore, the electronic device further includes:

[0125] The communication interface 303 is used for communication between the memory 301 and the processor 302 .

[0126] The memory 301 is used to store computer programs that can be run on the processor 302 .

[0127] The memory 301 may include a high-speed RAM (Random Access Memory) memory, and may also include a non-volatile memory, such as at least one disk memory.

[0128] If the memory 301, processor 302, and communication interface 303 are implemented independently, the communication interface 303, memory 301, and processor 302 can be connected to each other via a bus and communicate with each other. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0129] Optionally, in a specific implementation, if the memory 301, the processor 302 and the communication interface 303 are integrated on a chip, the memory 301, the processor 302 and the communication interface 303 can communicate with each other through an internal interface.

[0130] The processor 302 may be a CPU (Central Processing Unit), or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application.

[0131] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned multi-organ medical image segmentation method based on auxiliary network-generated prompts.

[0132] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0133] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of this application, "N" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0134] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing a custom logical function or process step, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed in a different order than shown or discussed, including performing functions in a substantially simultaneous manner or in a reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application pertain.

[0135] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, the steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement the method: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array, a field programmable gate array, etc.

[0136] Those skilled in the art will understand that all or part of the steps carried out in the method for implementing the above-mentioned embodiment can be completed by instructing the relevant hardware through a program, and the above-mentioned program can be stored in a computer-readable storage medium. When the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0137] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A multi-organ medical image segmentation method based on auxiliary network generation prompts, characterized in that: The following steps are involved: Acquire a multi-organ medical image, input the multi-organ medical image into a pre-built auxiliary network, and the auxiliary network outputs a first segmentation result of the multi-organ medical image; generating an input prompt box for the multi-organ medical image according to the first segmentation result; Inputting the multi-organ medical image and the input prompt box into a pre-built visual basic model, wherein the visual basic model outputs a second segmentation result of the multi-organ medical image; A prediction fusion module based on pixel-by-pixel weighting is constructed, and the prediction fusion module is used to perform pixel-by-pixel weighted fusion on the first segmentation result, the second segmentation result, and the multi-organ medical image to obtain a final segmentation result.

2. The multi-organ medical image segmentation method based on auxiliary network generation prompts according to claim 1, characterized in that The formula of the auxiliary network is: y t =f(x t ,y t-1 )=Decoder(Encoder(x t ))fort∈{1,2,…,T} Among them, x t represents the input of stage t; y t is the output of the tth stage; T is the total number of segmentation stages; Encoder(·) is the encoder network of the auxiliary network; Decoder(·) is the decoder network of the auxiliary network.

3. The multi-organ medical image segmentation method based on auxiliary network generation prompts according to claim 1, characterized in that Generating an input prompt box for the multi-organ medical image according to the first segmentation result includes: Traversing the non-zero foreground masks of all foreground categories of all images from the first segmentation result; extracting all existing foreground objects according to the non-zero foreground mask, and filtering the boundaries and details of the area other than the foreground objects in the image of the first segmentation result using a morphological erosion algorithm; Use the morphological expansion algorithm to restore the original segmented area and details to obtain the optimized segmented area; Connected regions are extracted based on the optimized segmented regions, input prompt boxes are generated based on the connected regions, and non-maximum suppression is performed on a subset of input prompt boxes of the same category in the same image.

4. The multi-organ medical image segmentation method based on auxiliary network generation prompts according to claim 3 is characterized in that The generation formula of the input prompt box is: Among them, (x min ,y min ,x max ,y max ) is the coordinate of the input prompt box; R i ∈R (b,c) For each connected area; b is the index of the picture to which the box belongs; c is the index of the category to which the box belongs; I is the index record table of the picture to which the box belongs and the index record table of the category to which it belongs; Box is the list of box coordinate records.

5. The multi-organ medical image segmentation method based on auxiliary network generation prompts according to claim 1, characterized in that The fusion process of the pixel-by-pixel weighted prediction fusion module includes: splicing the first segmentation result, the second segmentation result, and the multi-organ medical image in a channel dimension to obtain a fusion feature map, and generating a fusion weight map according to the fusion feature map; Performing a softmax operation on the fusion weight map to normalize the weight of each pixel; The respective weights of the first segmentation result and the second segmentation result are determined according to the normalized weight of each pixel, and weighted fusion is performed according to the first segmentation result, the second segmentation result, and the respective weights to obtain a final segmentation result.

6. The multi-organ medical image segmentation method based on auxiliary network generation prompts according to claim 1, characterized in that: Before inputting the multi-organ medical image and the input prompt box into the pre-built visual basic model, the method further includes: Get the training dataset; The visual base model is trained using the training data set, and during the training process, the network parameters of the visual base model are iteratively updated using a hybrid loss function.

7. The multi-organ medical image segmentation method based on auxiliary network generation prompts according to claim 6, characterized in that: The hybrid loss function is: Among them, α and β are weights; y0 is the first segmentation result; y final is the final segmentation result; To calculate Dice loss; To calculate the cross entropy loss; is the coarse segmentation loss; is the fusion loss; is the final prediction mask.

8. A multi-organ medical image segmentation device based on auxiliary network generation prompts, characterized in that: include: a first segmentation module, configured to acquire a multi-organ medical image, input the multi-organ medical image into a pre-built auxiliary network, and output a first segmentation result of the multi-organ medical image; a generating module, configured to generate an input prompt box for the multi-organ medical image according to the first segmentation result; a second segmentation module, configured to input the multi-organ medical image and the input prompt box into a pre-built visual base model, wherein the visual base model outputs a second segmentation result of the multi-organ medical image; A fusion module is used to construct a prediction fusion module based on pixel-by-pixel weighting, and use the prediction fusion module to perform pixel-by-pixel weighted fusion on the first segmentation result, the second segmentation result and the multi-organ medical image to obtain a final segmentation result.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multi-organ medical image segmentation method based on auxiliary network-generated prompts as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed, the multi-organ medical image segmentation method based on auxiliary network generation prompts according to any one of claims 1 to 7 is implemented.

Images