Image segmentation method and device, electronic equipment and storage medium

Abstract

The invention discloses an image segmentation method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a to-be-processed original 3D image which comprises a plurality of 2D slice images; first prompt information is received, second prompt information is generated based on the first prompt information, and the first prompt information indicates a target object in the 2D slice image; and based on the second prompt information, performing segmentation processing on the original 3D image through the 3D medical basic large model to obtain a 3D segmentation result of the target object. According to the image segmentation method, a complete 3D segmentation result can be obtained only by giving a small amount of first prompt information on the 2D slice image, a user does not need to prompt each layer of image, the workload is reduced, a medical basic large model is adopted, the applicability is wider, and end-to-end training does not need to be carried out.

Description

Technical Field

[0001] This application relates to the technical fields of image processing, and in particular to an image segmentation method, apparatus, electronic device, and storage medium. Background Technology

[0002] In related technologies, there are two main directions for large-scale segmentation models based on medical images. One is to use the network structure and parameters of SAM (Segment Anything Model) as the base network, fine-tuned based on a large amount of medical data. However, the drawback of this method is that it can only process 2D images and cannot utilize the relationships between different layers of the medical image; the segmentation result is only for the current layer. The other approach is to redesign the network and train a 3D network using a large amount of medical data. The drawback of this method is that it requires consideration of computational load and resources, so the network input can only be small image patches. For larger target regions, it cannot cover the entire region of interest, resulting in incomplete segmentation. Summary of the Invention

[0003] Therefore, the purpose of this application is to propose an image segmentation method, apparatus, electronic device, storage medium, and computer program product that can obtain a complete 3D segmentation result by providing only a small amount of initial prompting information on the 2D slice image, without requiring the user to provide prompts for each layer of the image, thus reducing workload. Furthermore, it uses a large medical-based model, which has wider applicability and does not require end-to-end training.

[0004] This application provides an image segmentation method, the method comprising: acquiring an original 3D image to be processed, wherein the original 3D image includes a plurality of 2D slice images; receiving first prompt information and generating second prompt information based on the first prompt information, wherein the first prompt information indicates a target object in the 2D slice images; and performing segmentation processing on the original 3D image based on the second prompt information using a 3D medical basic model to obtain a 3D segmentation result of the target object.

[0005] For example, generating the second prompt information based on the first prompt information includes: segmenting the 2D slice image using a 2D medical basic large model based on the first prompt information to obtain a 2D segmentation result of the target object; and generating the second prompt information based on the 2D segmentation result.

[0006] For example, the 2D segmentation result includes a region of interest for the target object, and the step of generating a second prompt message based on the 2D segmentation result includes: dividing the region of interest for the target object to obtain an edge region and a center region; sampling the edge region and the center region respectively, and generating a second prompt message based on the samples of the edge region and the center region, wherein the number of samples in the edge region is greater than the number of samples in the center region.

[0007] For example, dividing the region of interest for the target object into an edge region and a center region includes: performing an erosion operation on the region of interest for the target object to obtain an eroded first region; and obtaining the edge region and the center region based on the region of interest and the eroded first region, wherein the edge region is the region in the region of interest that does not include the first region, and the center region is the region that includes the first region.

[0008] For example, the step of performing an erosion operation on the region of interest of the target object to obtain an eroded first region includes: performing an erosion operation on the region of interest of the target object to obtain a current first region; determining whether the area ratio of the current first region to the region of interest is greater than a preset ratio; if the area ratio of the current first region to the region of interest is greater than the preset ratio, performing an erosion operation on the current first region until the area ratio is less than or equal to the preset ratio, and determining the current first region as the eroded first region.

[0009] For example, the second prompt information includes multiple items. The step of segmenting the original 3D image based on the second prompt information using a 3D medical basic model to obtain a 3D segmentation result of the target object includes: when multiple second prompt information items are not all located within a preset area, dividing the original 3D image based on the size of the preset area to obtain multiple 3D image blocks; segmenting the multiple 3D image blocks based on the second prompt information corresponding to the 3D image blocks using multiple 3D medical basic models to obtain a segmentation result for the 3D image blocks, wherein the 3D medical basic model and the 3D image blocks correspond one-to-one; and fusing the segmentation results for the 3D image blocks to obtain a 3D segmentation result of the target object.

[0010] For example, the second prompt information includes sample coordinate information. Before segmenting the multiple 3D image blocks based on the second prompt information corresponding to the 3D image blocks using multiple 3D medical basic models, the method further includes: performing coordinate transformation on the second prompt information based on the 3D image blocks to obtain second prompt information corresponding one-to-one with the 3D image blocks; wherein, the coordinate transformation converts the sample coordinate information for the original 3D image into sample coordinate information for the 3D image blocks.

[0011] For example, the segmentation result for each 3D image block includes the coordinate result for each 3D image block, and the fusion of the segmentation results for each 3D image block to obtain the 3D segmentation result of the target object includes: performing coordinate transformation on the coordinate result for each 3D image block to obtain the coordinate result for the original 3D image; and fusing multiple 3D image blocks based on the coordinate result for the original 3D image to obtain the 3D segmentation result of the target object.

[0012] For example, the method further includes updating the 3D segmentation result based on the first prompt information.

[0013] For example, the first prompt information includes positive sample point information and / or negative sample point information, and updating the 3D segmentation result based on the first prompt information includes: updating the 3D segmentation result when the 3D segmentation result includes negative sample point information, and / or when the 3D segmentation result does not include positive sample point information.

[0014] Another embodiment of this application provides an image segmentation apparatus, the apparatus comprising: an acquisition module for acquiring an original 3D image to be processed, wherein the original 3D image includes a plurality of 2D slice images; a generation module for receiving first prompt information and generating second prompt information based on the first prompt information, wherein the first prompt information indicates a target object in the 2D slice images; and a segmentation module for segmenting the original 3D image based on the second prompt information using a 3D medical basic model to obtain a 3D segmentation result of the target object.

[0015] Another embodiment of the application provides a cloud computing device for performing the steps of the method of any of the above embodiments.

[0016] Another embodiment of this application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method of any of the above embodiments.

[0017] Another embodiment of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method of any of the above embodiments.

[0018] Another embodiment of this application provides a computer program product, which includes instructions that, when executed by a processor of a computer device, enable the computer device to perform the steps of the method described in any of the above embodiments.

[0019] In the above embodiments, the original 3D image to be processed includes multiple 2D slice images; a first prompt information is received, and a second prompt information is generated based on the first prompt information, wherein the first prompt information indicates the target object in the 2D slice image; based on the second prompt information, the original 3D image is segmented using a large-scale 3D medical model to obtain the 3D segmentation result of the target object. The image segmentation method of the present invention only requires a small amount of first prompt information on the 2D slice image to obtain a complete 3D segmentation result, eliminating the need for the user to provide prompts for each image layer, thus reducing workload. Furthermore, it uses a large-scale medical model, making it more widely applicable and eliminating the need for end-to-end training. Attached Figure Description

[0020] Figure 1 A flowchart illustrating the image segmentation method provided in this application embodiment;

[0021] Figure 2 A flowchart for generating second prompt information based on first prompt information is provided for an embodiment of this application;

[0022] Figure 3 A schematic diagram of a 2D segmentation model provided for an embodiment of this application;

[0023] Figure 4 A flowchart for generating the second prompt information provided in this application embodiment;

[0024] Figure 5 A schematic diagram illustrating random sampling and uniform sampling provided for embodiments of this application;

[0025] Figure 6 A flowchart illustrating the process of obtaining the edge region and the center region provided in this application embodiment;

[0026] Figure 7 A schematic diagram illustrating the division of the region of interest as provided in this application embodiment;

[0027] Figure 8 A flowchart illustrating the etched first region provided for an embodiment of this application;

[0028] Figure 9A flowchart for obtaining 3D segmentation results provided in this application embodiment;

[0029] Figure 10 A schematic diagram of the region division provided for an embodiment of this application;

[0030] Figure 11 A flowchart for fusing segmentation results for each 3D image block, provided for embodiments of this application;

[0031] Figure 12 A flowchart illustrating the overall implementation of the image segmentation method provided in this application.

[0032] Figure 13 A schematic diagram of the segmentation system provided in the embodiments of this application;

[0033] Figure 14 A schematic diagram of the 3D segmentation results provided for an embodiment of this application;

[0034] Figure 15 A schematic diagram of an image segmentation apparatus provided for an embodiment of this application;

[0035] Figure 16 A block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0036] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0037] In the field of image segmentation, especially in medical image segmentation, it is crucial to identify specific anatomical structures, including organs, lesions, and tissues, from medical images.

[0038] In some examples, the medical basic segmentation model includes both 2D and 3D models. However, the 2D basic segmentation model cannot produce 3D segmentation results; it can only provide segmentation results for the current slice image. Furthermore, using a 2D segmentation model to determine the segmentation result of a whole organ requires labeling the entire organ. For example, the user might label each slice image and then sequentially call the 2D segmentation model for prediction. This approach is inefficient because each slice image requires separate model prediction, resulting in long prediction times; it also requires numerous user operations and a significant workload; and the model cannot utilize the anatomical relationships between different layers during prediction, potentially leading to unreasonable results.

[0039] In some examples, 3D segmentation models can also be used for segmentation. However, for 3D segmentation models, it is impossible to input all the volume data of the image to be segmented into the network. It can only be input in blocks. When the target area is large, it will cause the target area to be incompletely segmented.

[0040] In some examples, the quality of 3D base model segmentation results is significantly influenced by the quality of user input prompts during interaction. For instance, the more evenly distributed and numerous the annotation points are within the target region, the better the segmentation performance. Conversely, if the user provides only a few points within a small local area when annotating, the segmentation performance will deteriorate considerably.

[0041] Based on this, the present invention proposes an application method for a large-scale medical image segmentation model, which can efficiently combine a 2D basic segmentation model and a 3D basic segmentation model. Based on the 2D basic segmentation model and a small amount of prompt information given by the user, more comprehensive prompt information is automatically generated as input to the 3D basic segmentation model. Then, the 3D image is segmented based on the 3D segmentation model to obtain a more complete 3D segmentation result, reducing the user's workload and achieving better segmentation results.

[0042] Furthermore, it adopts a large-scale medical segmentation model, which eliminates the need for end-to-end training. This means that there is no need to construct and label the dataset, nor to train and optimize the network model. Simply call the pre-trained large-scale segmentation model and parameters, and select the region of interest on the image to be tested for labeling to obtain the segmentation result of the region of interest. It has wider applicability and can be applied to any model.

[0043] Figure 1 This is a flowchart of an image segmentation method according to an embodiment of this application.

[0044] As an example, such as Figure 1 As shown, image segmentation methods include:

[0045] S101, Obtain the original 3D image to be processed, wherein the original 3D image includes multiple 2D slice images.

[0046] S102, receive the first prompt information and generate the second prompt information based on the first prompt information, wherein the first prompt information indicates the target object in the 2D slice image.

[0047] S103, based on the second prompt information, the original 3D image is segmented using a 3D medical basic model to obtain the 3D segmentation result of the target object.

[0048] For example, the original 3D image to be processed includes 3D images obtained by imaging methods such as computed tomography (CT), magnetic resonance imaging (MR), positron emission tomography (PET), and ultrasound examination (US). The original 3D image includes multiple 2D slice images; it can be understood that the original 3D image is composed of multiple layers of 2D images. A first prompt is received, indicating the target object in the 2D slice image. It can be understood that the user can annotate the first prompt on a 2D slice image of a certain layer in the original 3D image. The first prompt serves as input to the model, and a higher-quality second prompt is automatically generated based on the first prompt. The first prompt can be the coordinate information of a marker point. For example, when a user wants to segment the liver region in an image, the user can mark a point on the liver region as a positive sample point, and the first prompt includes the coordinate information of the positive sample point. Alternatively, a point can be marked on a non-liver region of the image as a negative sample point, and the first prompt includes the coordinate information of the negative sample point. The first prompt can also be in the form of a prompt box. For example, if the user marks a prompt box on the target object region, the first prompt can include the coordinates of the upper left and lower right corners of the prompt box.

[0049] For example, a second prompt is generated based on the first prompt. It can be understood that the generated second prompt is more comprehensive and of higher quality. The second prompt and the original 3D image serve as input to a large-scale 3D medical model. Segmentation of the original 3D image based on the second prompt results in a more complete 3D segmentation of the target object. The target object can be any organ, such as the liver.

[0050] The image segmentation method of this application only requires a small amount of initial cue information on a single 2D image layer to obtain a complete 3D segmentation result. It does not require the user to provide cue information for each image layer, greatly reducing the workload.

[0051] As an example, such as Figure 2 As shown, a second prompt message is generated based on the first prompt message, including:

[0052] S201, based on the first prompt information, the 2D slice image is segmented using a 2D medical basic large model to obtain the 2D segmentation result of the target object.

[0053] S202, Generate a second prompt message based on the 2D segmentation result.

[0054] For example, the 2D segmentation model can adopt the MedSAM model. SAM (Segment Anything model), as the first interactive image segmentation foundation model, has significant zero-shot generalization ability. The SAM network structure is based on a transformer architecture. Specifically, the SAM network structure can be divided into three parts: Image encoder, Prompt encoder, and Mask decoder. In the image encoder part, a Vision Transformer is used to extract features from the input image. In the prompt encoder part, the user-provided prompt information is encoded. Finally, a lightweight feature decoder is used to fuse and process the image features and prompt information features, outputting the final segmentation mask, i.e., the final 2D segmentation result. The network structure is as follows: Figure 3 As shown.

[0055] For example, the first prompt information and a 2D slice image from the original 3D image are used as inputs to a large 2D medical model. The image encoder extracts features from the 2D slice image to obtain 2D slice image features. The image feature encoder can adopt a network structure based on ViT (Vision Transformer) and use pre-trained parameters to better construct the feature representation of the input image. The prompt encoder extracts features from the input first prompt information to obtain first prompt information features. For the prompt encoder part, it can handle prompt information in various input formats, such as positive sample point labels, negative sample point labels, and prompt box information. SAM can use positional encoding and learnable embedded features to better represent the prompt information given by the user. Specifically, positive and negative sample points can use horizontal and vertical coordinates as input, and prompt boxes can use the horizontal and vertical coordinates of their upper left and lower right corners as input. Then, feature extraction is performed using learnable parameters. The mask prompt information is an image with the same resolution as the input image, and its features are extracted using convolution operations within this module. Then the model performs feature fusion and segmentation on the features of the first prompt information and the features of the 2D slice image. For example, it can generate candidate regions for the target object based on the feature map, perform category judgment and pixel segmentation on the candidate regions to obtain the segmentation mask. The mask decoder part can adopt a lightweight design, such as including two transformer layers to predict the segmentation mask and confidence respectively, and finally obtain the 2D segmentation result of the target object.

[0056] For example, a second prompt message is generated based on the 2D segmentation result of the target object. Since the first prompt message contains relatively little information and is insufficient to guide the 3D segmentation model, this application automatically generates a high-quality second prompt message based on the 2D segmentation result to guide the 3D segmentation model, resulting in better segmentation performance. The second prompt message can be in the form of multiple prompt points, prompt boxes, prompt surfaces, etc.

[0057] The 2D segmentation model of this application uses a large 2D medical image segmentation model to process images with better results and higher efficiency. It is applicable to any large 2D medical image segmentation model and has a wider range of applications.

[0058] As an example, such as Figure 4 As shown, the 2D segmentation result includes the region of interest for the target object. A second prompt is generated based on the 2D segmentation result, including:

[0059] S401, the region of interest for the target object is divided into edge regions and center regions.

[0060] S402, sample the edge region and the center region respectively, and generate a second prompt message based on the sampled edge region and the sampled center region, wherein the number of samples in the edge region is greater than the number of samples in the center region.

[0061] For example, this application comprehensively analyzes the principles of 3D segmentation models and, through numerous experiments, finds that the content of the prompt information input into the 3D segmentation model has a significant impact on the final segmentation result. If the quality of the prompt information is poor, the final segmentation effect will decrease significantly. Therefore, generating high-quality prompt information is crucial for the overall algorithm result.

[0062] For example, taking the second prompt information as multiple prompt points as an example, the methods for generating the second prompt information include random sampling and uniform sampling, etc. Figure 5 As shown, Figure 5 The left image is a schematic diagram of random sampling. It can be seen that the prompts generated by random sampling are unstable, sometimes showing dense sampling in small local areas. In comparison, Figure 5 The prompts generated by uniform sampling, as shown in the right figure, are quite effective. However, the problem with uniform sampling is that a large number of sampling points are needed for the final 3D segmentation model to produce the optimal segmentation result, leading to excessive computation and low efficiency.

[0063] Based on this, this application proposes a better generation strategy. First, to ensure coverage of the entire region of interest, uniform sampling is performed within the edge ring of the segmentation result generated by the 2D segmentation model to guarantee the integrity of the prompt information. Then, one or two prompt points are randomly sampled from the center region of the segmentation result generated by the 2D segmentation model, excluding the edge ring, to reduce the computational load of the model.

[0064] For example, the 2D segmentation result includes the region of interest (ROI) for the target object. It can be understood that the 2D segmentation model initially segments the ROI of the target object. To ensure the completeness of the sampled prompt information and improve computational efficiency, the ROI is divided into edge regions and a central region. Sampling is performed on both the edge and central regions separately. The edge regions can be sampled uniformly, while the central region only needs a preset number of samples, such as only one or two sample points. This ensures the completeness of the prompt information while reducing computation and improving efficiency. After sampling, a second prompt information is generated based on the samples from the edge and central regions. The samples can be sampling points; for example, the second prompt information can be generated jointly based on the sampling points from the edge and central regions. The number of samples in the edge regions is greater than the number of samples in the central regions. It can be understood that the amount of prompt information in the edge regions is greater than that in the central regions.

[0065] As an example, such as Figure 6 As shown, the region of interest for the target object is divided into edge regions and a central region, including:

[0066] S601, perform an erosion operation on the region of interest of the target object to obtain the first eroded region.

[0067] S602, based on the region of interest and the eroded first region, an edge region and a center region are obtained, wherein the edge region is the region of interest that does not include the first region, and the center region is the region that includes the first region.

[0068] For example, the erosion operation can be a morphological erosion operation. A morphological erosion operation is performed on the region of interest (ROI) of the target object to obtain the eroded first region. It can be understood that the edges of the ROI are eroded away, leaving the first region. Based on the ROI and the eroded first region, edge regions and center regions are obtained. For example, the edge region is obtained by performing a difference operation on the ROI and the eroded first region. The edge region is the region in the ROI that does not include the first region, and the center region is the region that includes the first region. Uniform sampling can be performed on the edge region. For the center region, considering the computational load of subsequent 3D segmentation model operations, the number of prompts can be appropriately reduced; uniform sampling of two points is sufficient.

[0069] Figure 7 This is a schematic diagram illustrating the division of the region of interest according to an embodiment of this application.

[0070] like Figure 7 As shown, for example, if the target object is the liver, and the region of interest (ROI) is the liver region, a 2D segmentation model is used to segment the target object to obtain a ROI mask. The effective value of the ROI mask can be set to 1, while the effective value of the non-ROI region can be set to 0. An erosion operation is performed on the ROI to obtain the eroded first region. The difference between the first region and the ROI is used to obtain the edge region, with the first region serving as the center region. Uniform sampling can be performed on the edge region, and 1-2 points can be sampled in the center region. The samples from the edge region and the center region are then merged to obtain the second prompt information.

[0071] As an example, such as Figure 8 As shown, an erosion operation is performed on the region of interest of the target object to obtain the first eroded region, which includes:

[0072] S801 performs an erosion operation on the region of interest of the target object to obtain the current first region.

[0073] S802, determine whether the area ratio of the current first region to the region of interest is greater than a preset ratio.

[0074] S803, if the area ratio of the current first region to the region of interest is greater than a preset ratio, perform an erosion operation on the current first region until the area ratio is less than or equal to the preset ratio, and determine the current first region as the eroded first region.

[0075] For example, an erosion operation is performed on the region of interest (ROI) of the target object to obtain the current first region. The current first region is the area remaining in the ROI after the erosion operation. If the ratio of the area of ​​the remaining current first region after erosion to the area of ​​the ROI in the original segmentation result is greater than a preset ratio, it indicates that the current first region is too large and another erosion operation is needed, along with uniform sampling of the edge regions. The preset ratio can be 60%. For example, if the ratio of the area of ​​the remaining current first region after erosion to the area of ​​the ROI in the original segmentation result is greater than 60%, another erosion operation is performed on the current first region until the area ratio is less than or equal to the preset ratio, at which point the current first region is determined to be the eroded first region. The current first region is updated with each erosion operation. It can be understood that the current first region is updated with each erosion operation.

[0076] The automatic prompting strategy proposed in this application has advantages in two aspects: First, its results can cover the entire region of interest of the target object, ensuring the integrity of the 3D segmentation model prediction results and avoiding the serious problem of incomplete segmentation of large organs. Second, the sampling rule proposed in this application minimizes the computational load while ensuring the subsequent segmentation effect. Since the more labeled points given for the prompt, the more points need to be extracted for the prompting information, which means more computation and resource consumption, the automatic prompting strategy proposed in this application minimizes the amount of prompting information while ensuring uniform coverage of all regions of interest.

[0077] After generating the second prompt information, the original 3D image is segmented based on the second prompt information using a 3D medical basic model.

[0078] As an example, such as Figure 9 As shown, the second prompt information includes multiple parts. Based on the second prompt information, the original 3D image is segmented using a large 3D medical model to obtain the 3D segmentation result of the target object, including:

[0079] S901, if multiple second prompt messages are not all located within a preset area, the original 3D image is divided based on the size of the preset area to obtain multiple 3D image blocks.

[0080] S902, based on the second prompt information corresponding to the 3D image block, multiple 3D medical basic large models are used to segment multiple 3D image blocks to obtain the segmentation result for the 3D image block, wherein the 3D medical basic large model and the 3D image block correspond one-to-one.

[0081] S903, fuse the segmentation results of the 3D image blocks to obtain the 3D segmentation result of the target object.

[0082] For example, after obtaining the second prompt information according to the above prompt information generation strategy, a 3D medical basic model is invoked to segment the original 3D image. Before invoking the 3D medical basic model to process the original 3D image, the number of 3D segmentation models to be invoked is determined. If all the second prompt information is within a preset region, which can be a 64*64 area centered on the midpoint of the second prompt information, then only one 3D segmentation model needs to be invoked. Otherwise, the original 3D image is divided into multiple 3D image blocks. It can be understood that the size of the preset region is determined by the input size of the 3D segmentation model. For example, in the pre-training stage, the input size of the 3D segmentation model is specified as 64*64*64, meaning the 3D segmentation model processes a 64*64*64 3D image.

[0083] For example, if multiple second prompts are not all located within a preset region, it indicates that the input channel of a single 3D segmentation model cannot fully accommodate the original 3D image, meaning that the volume data of the original 3D image to be processed cannot be completely input into a single 3D segmentation model. In this case, traditional 3D segmentation methods input the original 3D image to be processed in patches. However, this can lead to incomplete segmentation of the target region. For instance, if the region of interest is the kidney organ, but due to the limited size of a patch, after extracting an image patch centered on the user-provided interaction information and inputting it into the network, only a portion of the kidney region can be segmented. The image segmentation method of this application, when the region of interest is large, calls multiple 3D segmentation models for block segmentation, and each image block contains corresponding second prompts, ensuring both the image segmentation effect and the integrity of the image segmentation.

[0084] For example, the original 3D image is divided into multiple 3D image blocks based on the size of a preset region. For instance, if the input size of the 3D segmentation model is specified as 64*64*64, but the original 3D image is 128*128*128, the original 3D image can be divided into four image blocks. Figure 10 The diagram illustrates the region division. For each image patch, a 3D segmentation model is invoked to process that patch; that is, multiple 3D segmentation models are invoked in parallel to process multiple 3D image patches. Based on the second prompt information corresponding to each 3D image patch, multiple 3D medical basic models are used to segment the multiple 3D image patches, obtaining a segmentation result for each 3D image patch. The segmentation results of each 3D image patch are then fused to obtain the final 3D segmentation result.

[0085] It should be noted that although the generated second cue information is for a 2D image, when input into the 3D segmentation model, the second cue information is in 3D coordinates. The original 3D image and the second cue information serve as inputs to the 3D segmentation model, and the final output is the 3D segmentation result of the target object.

[0086] For example, the 3D segmentation model can be the SAM-Med3D model, which utilizes pre-trained parameters of the 3D segmentation model to predict the segmentation of the current image patch. Finally, the results of multiple 3D segmentation models running in parallel are merged to obtain a complete 3D segmentation result image of the region of interest.

[0087] The 3D segmentation model of this application uses a large 3D medical image basic segmentation model to process 3D images with better results and higher efficiency, and is applicable to any large 3D medical image basic segmentation model, thus having a wider range of applicability.

[0088] As an example, the second prompt information includes sample coordinate information. Before segmenting multiple 3D image blocks based on the second prompt information corresponding to the 3D image blocks using multiple 3D medical basic models, the process further includes: performing coordinate transformation on the second prompt information based on the 3D image blocks to obtain second prompt information that corresponds one-to-one with the 3D image blocks; wherein, the coordinate transformation converts the sample coordinate information for the original 3D image into sample coordinate information for the 3D image blocks.

[0089] For example, if the original 3D image is segmented for recognition, the second prompt information needs to be transformed in coordinates. This is because the original generated second prompt information uses coordinates for the entire original 3D image and needs to be converted to coordinates for each 3D image block, resulting in second prompt information that corresponds one-to-one with each 3D image block. The 3D medical basic model also corresponds one-to-one with each 3D image block. Based on the second prompt information corresponding to multiple 3D medical basic models and 3D image blocks, multiple 3D image blocks are segmented to obtain the segmentation result for each 3D image block. It can be understood that the input to each 3D medical basic model is a 3D image block and its corresponding second prompt information. The second prompt information indicates the target object in the image block, ensuring the segmentation effect of each image block. Then, the segmentation results of each 3D image block are fused to obtain the complete 3D segmentation result.

[0090] As an example, such as Figure 11 As shown, the segmentation result for each 3D image patch includes the coordinate result for each 3D image patch. The segmentation results for each 3D image patch are fused to obtain the 3D segmentation result of the target object, including:

[0091] S1101, perform coordinate transformation on the coordinate results for each 3D image block to obtain the coordinate results for the original 3D image.

[0092] S1102, based on the coordinate results of the original 3D image, multiple 3D image blocks are fused to obtain the 3D segmentation result of the target object.

[0093] For example, the output of each 3D segmentation model is the coordinate result for each 3D image patch. This coordinate result needs to be transformed to represent the coordinates of the entire image patch, so that the segmentation results of multiple 3D image patches can be fused to obtain a complete 3D segmentation result. This coordinate transformation can be understood as the inverse of the coordinate transformation described above.

[0094] The 3D segmentation method of this application divides the original 3D image into blocks for segmentation processing, and then fuses the segmentation results. When the region of interest of the target object is too large, it can ensure the segmentation effect of each small image block and the integrity of the segmentation result of the target object.

[0095] After obtaining the 3D segmentation results, it is necessary to optimize the segmentation results based on the initial prompt information.

[0096] As an example, image segmentation methods also include updating the 3D segmentation results based on the initial cue information.

[0097] For example, in the implementation of a large-scale segmentation model based on the SAM architecture, random Fourier features are used in the feature encoding part to better process the first prompt information given by the user. This method introduces random numbers, which will lead to a certain positional bias. This positional bias is not obvious in large-scale natural image model applications because the training set of natural images is large enough. However, due to the difficulty in acquiring and annotating medical images, the training set is much smaller than that of natural images, so this problem becomes apparent.

[0098] For example, the segmentation results obtained using the basic segmentation model may either lack positive sample information or contain negative sample information, which significantly weakens the model's reliability and interpretability. Therefore, this application checks and optimizes the 3D segmentation results after prediction based on the 3D model. If the 3D segmentation results are correct, the obtained 3D segmentation results are directly output. If there are errors in the 3D segmentation results, the 3D segmentation results are updated based on the first prompt information.

[0099] As an example, the first prompt information includes positive sample point information and / or negative sample point information. Updating the 3D segmentation result based on the first prompt information includes: updating the 3D segmentation result if the 3D segmentation result includes negative sample point information, and / or if the 3D segmentation result does not include positive sample point information.

[0100] For example, when annotating the first prompt information of a 2D slice image, the user can annotate positive sample point information and / or negative sample point information. It can be understood that positive sample points represent points annotated within the target object region, and negative sample points represent points annotated outside the target object region. If the 3D segmentation result includes negative sample point information, and / or does not include positive sample point information, it indicates that the 3D segmentation result is inaccurate and needs to be corrected and optimized.

[0101] For example, there are many methods that can be used to correct and optimize 3D segmentation results. For instance, region growing can be used as an example. Using the current segmentation result Mask as the seed point, a multi-neighbor traversal is performed on the image, such as a 9-neighborhood or a 26-neighborhood. A breadth-first search strategy (which covers a larger area) can be used to locate the boundary of the current region of interest. Because there must be differences in grayscale values ​​between different regions of interest, the above optimization of the 3D segmentation result is achieved by detecting the boundary of the region of interest. This operation is iterated until the 3D segmentation result includes positive sample point information and does not contain negative sample point information. For example, if the 3D segmentation result does not include positive sample points, it may be because the current segmentation result is too small; for example, it can be expanded outwards until positive sample points are included. Similarly, if the 3D segmentation result includes negative sample points, it may be because the current segmentation result is too large; it can be shrunk inwards until negative sample points are no longer included.

[0102] The image segmentation method of this application has the following advantages:

[0103] (1) By adopting the basic segmentation large model application method proposed in this application, only a small amount of prompting information is needed on one layer of 2D image to obtain the complete 3D segmentation result. The user does not need to provide prompts for each layer of image, which greatly reduces the workload.

[0104] (2) The 3D segmentation results obtained by using the basic segmentation large model application method proposed in this application can call multiple 3D segmentation models in parallel, resulting in better integrity and avoiding the problem of incomplete segmentation of large organs.

[0105] (3) The automatic generation strategy for prompt information proposed in this application can effectively reduce the number of low-quality and redundant prompt information, thereby reducing the computational load of the model and improving the running speed.

[0106] (4) The automatic generation strategy for prompt information proposed in this application can ensure the integrity of prompt coverage, thereby maximizing the capabilities of the basic segmentation model and obtaining the optimal segmentation result.

[0107] (5) The segmentation results obtained by using the basic segmentation big model application method proposed in this application can respond to the prompt information completely and accurately, and can guarantee that the segmentation results will definitely contain positive sample prompts and definitely not contain negative sample prompts. This avoids the influence of some random number operations introduced in the big model algorithm on the results, and greatly enhances the credibility and interpretability of the basic segmentation big model.

[0108] Figure 12 This is a flowchart illustrating the overall implementation of an image segmentation method according to an embodiment of this application.

[0109] like Figure 12As shown, the process first reads in the test image (the original 3D image) and the prompts annotated by the user on a certain layer of the image, and then uses this as the basis for predicting the 2D basic segmentation model. Based on the prediction result mask of the 2D basic segmentation model, the prompts (second prompts) required for the 3D basic segmentation model are automatically generated. There are many ways to generate these prompts, such as random sampling and uniform sampling. This application proposes a method for automatically generating effective second prompts, which can maximize the segmentation capability of the 3D basic segmentation model. Compared with random sampling, uniform sampling has more advantages in segmentation results. It should be noted that the application scenarios of the prompt generation strategy of this invention are not limited to this generation strategy, but are applicable to any method that can automatically generate the prompts required for subsequent basic segmentation model prediction. Then, it is determined whether an image patch (the input size of the 3D model) can completely contain all the prompts (second prompts). If so, the coordinate system of the prompts is directly transformed, converting the current coordinate system relative to the overall test image to the coordinate system relative to the current test image patch. If not, it indicates that the target region of interest is relatively large, requiring reasonable and efficient partitioning of image patches. Then, the basic 3D segmentation model is called in parallel to test each image patch, and finally, all results are merged. During the prompt coordinate system transformation, the model must be transformed to the corresponding image patch coordinate system. Based on the predicted 3D segmentation results, it is then checked whether the user-provided prompt information has been responded to completely accurately. If errors exist, they can be corrected to optimize the 3D segmentation results. For example, an error might mean that the 3D segmentation result does not contain the user-provided positive sample prompt information, or it might contain the user-provided negative sample prompt information. There are many methods for correction and optimization. Morphological methods, such as region growing, can be used, starting with the current 3D segmentation result as the seed point and growing until the positive sample prompt is fully contained, or the negative sample prompt is excluded. Alternatively, a lightweight deep learning segmentation model can be trained, taking an image patch and its corresponding coarse segmentation result as input, to predict fine segmentation.

[0110] It should be noted that the 2D and 3D basic segmentation models described in this invention do not refer to a specific algorithm or a few algorithms. The 2D basic segmentation model refers to all large-scale basic segmentation models that take 2D images as input, such as SAM and MedSAM. The 3D basic segmentation model refers to all large-scale basic segmentation models that take 3D image patches as input, such as SAM-Med3D. The application scenarios of this invention are not limited to combinations of two algorithms; it is applicable to any medical image basic segmentation model application method that uses a 2D basic model to generate a single-layer prediction mask, then automatically generates an efficient prompt based on this mask, and finally automatically calls the 3D basic segmentation model.

[0111] Figure 13 This is a schematic diagram of the structure of a segmentation system according to an embodiment of this application.

[0112] like Figure 13 As shown, the segmentation system includes a basic segmentation model (2D) module, an automatic prompt generation module, a prompt coordinate system transformation module, a basic segmentation model (3D) module, and a prompt response processing module. The basic segmentation model (2D) module is used to predict the segmentation of the current layer based on the prompt information provided by the user and the extracted 2D image of the current layer, utilizing the pre-trained parameters of the basic segmentation model (2D). The automatic prompt generation module automatically generates high-quality prompts based on the segmentation mask predicted by the basic segmentation model (2D). Morphological erosion is performed on the segmentation results, followed by a difference operation to obtain the edge ring region of the segmentation results. Uniform sampling is then performed within this region. The system then determines whether the above operation needs to be repeated based on the ratio of the area of ​​the remaining region after erosion to the area of ​​the original segmentation result. If this ratio is greater than 60%, it indicates that the region of interest is too large, and another uniform sampling operation within the edge ring is required. Finally, for internal regions, considering the computational load of subsequent basic segmentation model operations, the number of prompts can be appropriately reduced, with only two points being uniformly sampled.

[0113] The Prompt coordinate system transformation module analyzes the automatically generated Prompt, specifically including the division of the test image patch (original 3D image) and the transformation of the Prompt coordinate system. First, based on the coordinates of points in the Prompt, it determines whether an image patch can encompass the entire Prompt. If so, a single basic segmentation model (3D) is called; otherwise, the current Prompt needs to be appropriately divided so that multiple basic segmentation models (3D) can be called in parallel for more efficient operation. Finally, the reference coordinate system of the Prompt information is transformed to the current image patch to ensure the accuracy of the information provided by the Prompt.

[0114] The Basic Segmentation Model (3D) module: Within this module, segmentation prediction of the current image patch is achieved using pre-trained parameters of the basic segmentation model (3D) based on automatically generated and transformed Prompt information. Finally, the results of multiple parallel-running basic segmentation models (3D) are merged to obtain a complete 3D segmentation result map of the region of interest.

[0115] Prompt Response Processing Module: This module consists of two parts: a Prompt response checking module and a 3D segmentation result correction and optimization module. It ensures that the final 3D segmentation result accurately responds to all Prompt information. For example... Figure 14 As shown in the diagram, green dots represent positive sample points, and red dots represent negative sample points. For the following two error cases: either a positive sample prompt is not included, or a negative sample prompt is included, if neither appears in the current 3D segmentation result, the current segmentation result is directly output; otherwise, 3D segmentation result correction and optimization are required. There are many methods for correcting and optimizing 3D segmentation results; region growing is used as an example. Using the current segmentation result Mask as the seed point, a 26-neighborhood traversal is performed on the image, employing a breadth-first search approach to locate the boundary of the current region of interest. Because there must be differences in grayscale values ​​between different regions of interest, the above optimization of the 3D segmentation result is achieved by detecting the boundaries of the regions of interest. This operation is iterated until the 3D segmentation result contains all positive sample points and no negative sample points.

[0116] This application also proposes an image segmentation apparatus.

[0117] As an example, such as Figure 15As shown, the image segmentation device includes: an acquisition module 10, used to acquire the original 3D image to be processed, wherein the original 3D image includes multiple 2D slice images; a generation module 20, used to receive first prompt information and generate second prompt information based on the first prompt information, wherein the first prompt information indicates the target object in the 2D slice images; and a segmentation module 30, used to segment the original 3D image based on the second prompt information using a 3D medical basic large model to obtain the 3D segmentation result of the target object.

[0118] This application also proposes a computer-readable storage medium.

[0119] In this embodiment, a computer program is stored on a computer-readable storage medium, and when the computer program is executed by a processor, it implements the steps of the image segmentation method described above.

[0120] Figure 16 A block diagram of an electronic device provided in an embodiment of this application.

[0121] This application provides an electronic device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described image segmentation method.

[0122] like Figure 16 As shown, for ease of understanding, embodiments of this application illustrate a specific electronic device.

[0123] Electronic devices are intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0124] like Figure 16 As shown, the device includes a computing unit 1601, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 1602 or a computer program loaded into a random access memory (RAM) 1603 from a storage unit 1608. The RAM 1603 may also store various programs and data required for the operation of the electronic device. The computing unit 1601, ROM 1602, and RAM 1603 are interconnected via a bus 1604. An input / output (I / O) interface 1605 is also connected to the bus 1604.

[0125] Multiple components in the electronic device are connected to the I / O interface 1605. These components include: an input unit 1606, such as a keyboard or mouse; an output unit 1607, such as various types of displays or speakers; a storage unit 1608, such as a hard disk or optical disk; and a communication unit 1609, such as a network interface card (NIC), a modem, or a wireless transceiver. The communication unit 1609 allows the electronic device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0126] The computing unit 1601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1601 performs the various methods described above, such as image segmentation methods. For example, in some embodiments, the image segmentation method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1608. In some embodiments, part or all of the computer program may be loaded and / or installed on an electronic device via ROM 1602 and / or communication unit 1609. When the computer program is loaded into RAM 1603 and executed by the computing unit 1601, the image segmentation methods described above can be performed. Alternatively, in other embodiments, the computing unit 1601 may be configured to perform image segmentation methods by any other suitable means (e.g., by means of firmware).

[0127] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequential list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this application, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0128] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0129] In the description of this application, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this application, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0130] In the description of this application, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc., indicating the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application.

[0131] Furthermore, the terms "first," "second," etc., used in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying relative importance, or implicitly specifying the number of technical features indicated in this embodiment. Therefore, features defined with terms such as "first" and "second" in the embodiments of this application can explicitly or implicitly indicate that the embodiment includes at least one of those features. In the description of this application, the word "multiple" means at least two or more, such as two, three, four, etc., unless otherwise explicitly and specifically defined in the embodiments.

[0132] In this application, unless otherwise explicitly specified or limited in the embodiments, the terms "installation," "connection," "joining," and "fixing" appearing in the embodiments should be interpreted broadly. For example, a connection can be a fixed connection, a detachable connection, or an integral part; it can also be a mechanical connection, an electrical connection, etc. Of course, it can also be a direct connection, or an indirect connection through an intermediate medium, or it can be the internal communication between two components, or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific implementation.

[0133] In this application, unless otherwise expressly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "on top of," and "over" the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0134] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. An image segmentation method characterized by, The method comprises: acquiring an original 3D image to be processed, wherein the original 3D image comprises a plurality of 2D slice images; receiving first prompt information and generating second prompt information based on the first prompt information, wherein the first prompt information indicates a target object in the 2D slice image; performing segmentation processing on the original 3D image based on the second prompt information through a 3D medical base large model to obtain a 3D segmentation result of the target object.

2. The image segmentation method of claim 1, wherein, The method comprises: performing segmentation processing on the 2D slice image based on the first prompt information through a 2D medical base large model to obtain a 2D segmentation result of the target object; generating second prompt information based on the 2D segmentation result.

3. The image segmentation method of claim 2, wherein, The 2D segmentation result comprises a region of interest for the target object, and the method comprises: dividing the region of interest for the target object to obtain an edge region and a center region; sampling the edge region and the center region respectively, and generating second prompt information based on the samples of the edge region and the samples of the center region, wherein the number of samples of the edge region is greater than the number of samples of the center region.

4. The image segmentation method of claim 3, wherein, The method comprises: performing an erosion operation on the region of interest for the target object to obtain a first region after erosion; obtaining the edge region and the center region based on the region of interest and the first region after erosion, wherein the edge region is a region of the region of interest that does not include the first region, and the center region is a region that includes the first region.

5. The image segmentation method of claim 4, wherein, The method comprises: performing an erosion operation on the region of interest for the target object to obtain a current first region; determining whether the area ratio of the current first region to the region of interest is greater than a preset ratio; in the case where the area ratio of the current first region to the region of interest is greater than the preset ratio, performing an erosion operation on the current first region until the area ratio is less than or equal to the preset ratio, and determining the current first region as the first region after erosion.

6. The image segmentation method of claim 1, wherein, The second prompt information comprises a plurality of, and the method comprises: in the case where the plurality of second prompt information are not all located in a preset region, dividing the original 3D image based on the size of the preset region to obtain a plurality of 3D image blocks; performing segmentation processing on the plurality of 3D image blocks based on the second prompt information corresponding to the 3D image blocks through a plurality of 3D medical base large models to obtain segmentation results for the 3D image blocks, wherein the 3D medical base large models and the 3D image blocks correspond one by one; The segmentation results for the 3D image blocks are fused to obtain a 3D segmentation result of the target object.

7. The image segmentation method of claim 6, wherein, The second prompt information includes sample coordinate information, and before the segmentation processing of the plurality of 3D image blocks by the plurality of 3D medical base large models based on the second prompt information corresponding to the 3D image blocks, the method further includes: The second prompt information is coordinate-converted based on the 3D image blocks to obtain second prompt information corresponding to the 3D image blocks one by one. The coordinate conversion is to convert sample coordinate information for the original 3D image into sample coordinate information for the 3D image blocks.

8. The image segmentation method of claim 6, wherein, The segmentation result for each 3D image block includes a coordinate result for each 3D image block, and the fusion of the segmentation results for the 3D image blocks to obtain the 3D segmentation result of the target object includes: The coordinate result for each 3D image block is coordinate-converted to obtain a coordinate result for the original 3D image; The plurality of 3D image blocks are fused based on the coordinate result for the original 3D image to obtain the 3D segmentation result of the target object.

9. The image segmentation method of claim 1, wherein, The method further includes: The 3D segmentation result is updated based on the first prompt information.

10. The image segmentation method of claim 9, wherein, The first prompt information includes positive sample point information and / or negative sample point information, and the updating of the 3D segmentation result based on the first prompt information includes: In the case that the 3D segmentation result includes negative sample point information and / or the 3D segmentation result does not include positive sample point information, the 3D segmentation result is updated.

11. An image segmentation apparatus characterized by comprising: The apparatus includes: An acquisition module configured to acquire an original 3D image to be processed, wherein the original 3D image includes a plurality of 2D slice images; A generation module configured to receive first prompt information and generate second prompt information based on the first prompt information, wherein the first prompt information indicates a target object in the 2D slice images; A segmentation module configured to perform segmentation processing on the original 3D image based on the second prompt information by a 3D medical base large model to obtain a 3D segmentation result of the target object.

12. An electronic device, comprising: A memory and a processor, the memory stores a computer program, and the processor executes the computer program to implement the steps of the method in any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, A computer program is stored thereon, and the computer program is executed by a processor to implement the steps of the method in any one of claims 1-10.

Images