A general medical image interactive segmentation system based on a large model

Abstract

The application discloses a general medical image interactive segmentation method and system based on a large model, which adaptively selects a model according to an input image type, adjusts a prompt word for a poor network segmentation condition, continuously iterates a segmentation result, and improves a segmentation effect, and comprises the following steps: step S1, pre-processing data of different modalities; step S2, analyzing the size of an input image, selecting a suitable model according to a preset threshold; step S3, segmenting an image and using a prompt word to perform mask decoding to obtain mask output; and step S4, if the mask output is not satisfied, adjusting the prompt word and calling the segmentation model again to obtain more accurate results, and continuously iterating the process until a satisfactory result is obtained. The application trains a general medical image segmentation tool by using a large model and large data, can obtain a better segmentation result on any data, and more efficiently assists labeling and clinical analysis.

Description

Technical Field

[0001] This invention relates to the field of image recognition, and more particularly to a general interactive medical image segmentation system based on a large model. Background Technology

[0002] Image segmentation is a crucial branch of medical image analysis. Accurate segmentation can reveal precise anatomical structures, including human organs, tissues, and lesions, which is of great significance for clinical diagnosis. Manually annotated segmented images are time-consuming, labor-intensive, and inefficient. In recent years, with the development of artificial intelligence, numerous convolutional neural networks have been proposed and proven to demonstrate excellent segmentation results on many medical images. However, these networks often have limitations. On the one hand, they are designed for specific tasks, such as organ segmentation or lesion segmentation, and are trained and tested on a single type of data. Networks trained in this way often perform poorly on different modalities of data and cannot adapt to new data. On the other hand, the medical field lacks large-scale datasets that can serve as benchmarks, and these networks are relatively small and cannot learn from large datasets and their image features, thus limiting their capabilities and universality. While large-scale image-based models have emerged in recent years, they have not performed well in the medical image field. Differences in imaging methods mean that these large models cannot be directly used on medical data. Current medical image segmentation models are based on specific datasets and often perform poorly on other modalities. Furthermore, they suffer from poor segmentation results and require significant time and resources. In today's rapidly developing medical environment, with the ever-increasing amount of data, there is a need for a general-purpose model that can flexibly handle new needs and perform well in any new scenario. Summary of the Invention

[0003] The purpose of this invention is to provide a general interactive medical image segmentation method and system based on a large model to solve the problems mentioned in the background art.

[0004] To achieve the above-mentioned objectives, one aspect of the present invention provides a general interactive medical image segmentation method based on a large model. This method adaptively selects a model according to the input image type, and improves segmentation performance by adjusting prompts and iteratively refining the segmentation results when the network segmentation is inadequate. The method includes the following steps:

[0005] Step S1: Preprocess the data for different modalities;

[0006] Step S2: Analyze the size of the input image and select a suitable model based on a preset threshold;

[0007] Step S3: Segment the image and use prompt words to perform mask decoding to obtain the mask output;

[0008] Step S4: If the mask output is not satisfactory, adjust the prompt words and call the segmentation model again to obtain a more accurate result. Iterate this process until a satisfactory result is obtained.

[0009] Furthermore, for 2D images, step S3 involves selecting the region of interest on the interface, selecting the coordinates of the upper left and lower right corners of the prompt box as prompt words, inputting them into the prompt encoder for encoding, and using them together with the feature vector generated by the image encoder as input for mask decoding to obtain the mask output.

[0010] Furthermore, for 3D images, step S3 involves taking the layer with the largest area of ​​the 3D image as the starting layer for 2D segmentation to obtain a segmentation mask. Using the segmentation result of the starting layer as a prompt, the masks of adjacent layers are calculated in parallel from both the upward and downward directions, and so on until the image boundary.

[0011] Furthermore, the method is applied to an annotation platform to achieve semi-automatic segmentation of medical images.

[0012] Another aspect of the present invention provides a general interactive medical image segmentation system based on a large model. This system adaptively selects a model based on the input image type and improves segmentation performance by adjusting prompts and iterating through the segmentation results when the network segmentation is inadequate. The system includes a preprocessing module, a model selection module, a segmentation module, and an iteration module, wherein:

[0013] The preprocessing module is used to preprocess data for different modalities;

[0014] The model selection module is used to analyze the size of the input image and select a suitable model based on a preset threshold.

[0015] The segmentation module is used to segment the image and use prompt words to decode the mask, obtaining the mask output;

[0016] The iterative module adjusts the prompts and calls the segmentation model again if it is not satisfied with the mask output, in order to obtain a more accurate result. This process is iterated until a satisfactory result is obtained.

[0017] Furthermore, for 2D images, the segmentation module selects the region of interest on the interface, selects the coordinates of the upper left and lower right corners of the prompt box as prompt words and inputs them into the prompt encoder for encoding. Together with the feature vector generated by the image encoder, they are used as input for mask decoding to obtain the mask output.

[0018] Furthermore, for 3D images, the segmentation module takes the layer with the largest area of ​​the 3D image as the starting layer for 2D segmentation to obtain a segmentation mask. Using the segmentation result of the starting layer as a prompt, the mask of the adjacent layer is calculated in parallel from both the upward and downward directions, and so on until the image boundary.

[0019] Furthermore, the system is applied to a labeling platform to achieve semi-automatic segmentation of medical images.

[0020] Compared with existing technologies, this system and method have the following advantages:

[0021] 1. This invention uses a large model and big data to train a general medical image segmentation tool that can achieve good segmentation results on any data and more efficiently assist in annotation and clinical analysis.

[0022] 2. This invention makes segmentation more flexible through an interactive segmentation method. If the labeling is unsatisfactory, the prompt words can be easily adjusted to obtain satisfactory results.

[0023] 3. This invention is more efficient through its model selection function. Attached Figure Description

[0024] Figure 1 This is a flowchart of a general interactive medical image segmentation method based on a large model.

[0025] Figure 2 This is a schematic diagram of a 2D image segmentation process using a general interactive medical image segmentation method based on a large model.

[0026] Figure 3 This is a schematic diagram of the 3D image segmentation process using a general interactive medical image segmentation method based on a large model.

[0027] Figure 4 This is a schematic diagram of the interactive segmentation process using a general interactive medical image segmentation method based on a large model. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] like Figure 1The diagram shown is a flowchart of the method of this invention. This method adaptively selects a model based on the type of the input image. For cases where the network segmentation is poor, it adjusts the prompt words and iterates through the segmentation results to improve the segmentation effect. The method includes the following steps:

[0030] Step S1: Preprocess the data for different modalities;

[0031] Step S2: Analyze the size of the input image and select a suitable model based on a preset threshold;

[0032] Step S3: Segment the image and use prompt words to perform mask decoding to obtain the mask output;

[0033] Step S4: If the mask output is not satisfactory, adjust the prompt words and call the segmentation model again to obtain a more accurate result. Iterate this process until a satisfactory result is obtained.

[0034] like Figure 2 The diagram illustrates the segmentation process for a 2D image. For a 2D input image, different modalities undergo different preprocessing steps. The preprocessed image then passes through an image encoder, encoding it into a feature vector of size (256, 64, 64) for subsequent decoding. Simultaneously, the user selects a region of interest on the image. The system inputs the coordinates of the top-left and bottom-right corners of the selection prompt as a prompt word into the prompt encoder. These coordinates, along with the image feature vector, are used as input for mask decoding, ultimately yielding the mask output.

[0035] like Figure 3 The diagram illustrates the segmentation process for a 3D image. The input is a 3D image. This method performs different preprocessing steps for different modalities of data. The user selects the target region to be segmented on the flat scan image and adjusts the upper and lower boundaries on the images in two other directions to ensure integrity. The system analyzes the size of the input image and selects a model based on a preset threshold—either a large, full model or a smaller, lightweight model. The layer with the largest target area is used as the starting layer for segmentation. After 2D segmentation, a segmentation mask is obtained. Using the segmentation result of the starting layer as a prompt, the masks of adjacent layers are calculated in parallel in both the upward and downward directions, and so on until the boundary is reached.

[0036] like Figure 4 The diagram illustrates the interactive segmentation process. Users receive an intermediate result each time they use the system and can choose whether to adjust it. If dissatisfied with the result, users can refine it by adjusting the prompts, selecting more precise prompts, and then re-applying the segmentation model to obtain a more accurate result. This process iterates until a satisfactory result is achieved.

[0037] This invention applies big data and large models to a segmentation tool, making it applicable to various medical imaging platforms requiring image segmentation. It can be used on annotation platforms to achieve semi-automatic segmentation. Furthermore, it incorporates interactive methods into existing segmentation tools, significantly improving annotation efficiency and reducing manual adjustment time. The use of a lightweight model is determined based on the input image size, greatly reducing waiting time and resource consumption.

[0038] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A general interactive medical image segmentation method based on a large model, characterized in that, The model is adaptively selected based on the input image type. For cases where the network segmentation is poor, the prompt words are adjusted, and the segmentation results are iterated continuously to improve the segmentation effect, including the following steps: Step S1: Preprocess the data for different modalities; Step S2: Analyze the size of the input image and select a suitable model based on a preset threshold; Step S3: Segment the image and use prompt words to perform mask decoding to obtain the mask output; Step S4: If the mask output is not satisfactory, adjust the prompt words and call the segmentation model again to obtain a more accurate result. Iterate this process until a satisfactory result is obtained.

2. The general interactive medical image segmentation method based on a large model according to claim 1, characterized in that, For 2D images, step S3 involves selecting the region of interest on the interface, selecting the coordinates of the upper left and lower right corners of the prompt box as prompt words, inputting them into the prompt encoder for encoding, and using them together with the feature vector generated by the image encoder as input for mask decoding to obtain the mask output.

3. The general interactive medical image segmentation method based on a large model according to claim 1, characterized in that, For 3D images, step S3 involves taking the layer with the largest area of ​​the 3D image as the starting layer for 2D segmentation to obtain a segmentation mask. Using the segmentation result of the starting layer as a prompt, the masks of adjacent layers are calculated in parallel from both the upward and downward directions, and so on until the image boundary.

4. The general interactive medical image segmentation method based on a large model according to claim 1, characterized in that, The method is applied to a labeling platform to achieve semi-automatic segmentation of medical images.

5. A general interactive medical image segmentation system based on a large model, characterized in that, The model is adaptively selected based on the input image type. For cases where network segmentation is inadequate, the prompt words are adjusted, and the segmentation results are iterated continuously to improve the segmentation performance. This includes a preprocessing module, a model selection module, a segmentation module, and an iteration module, among which: The preprocessing module is used to preprocess data for different modalities; The model selection module is used to analyze the size of the input image and select a suitable model based on a preset threshold. The segmentation module is used to segment the image and use prompt words to decode the mask, obtaining the mask output; The iterative module adjusts the prompts and calls the segmentation model again if it is not satisfied with the mask output, in order to obtain a more accurate result. This process is iterated until a satisfactory result is obtained.

6. A general interactive medical image segmentation system based on a large model according to claim 5, characterized in that, For 2D images, the segmentation module selects the region of interest on the interface, selects the coordinates of the upper left and lower right corners of the prompt box as prompt words and inputs them into the prompt encoder for encoding. Together with the feature vector generated by the image encoder, they are used as input for mask decoding to obtain the mask output.

7. A general interactive medical image segmentation system based on a large model according to claim 5, characterized in that, For 3D images, the segmentation module takes the layer with the largest area of ​​the 3D image as the starting layer for 2D segmentation to obtain a segmentation mask. Using the segmentation result of the starting layer as a prompt, the mask of the adjacent layer is calculated in parallel from both the upward and downward directions, and so on until the image boundary.

8. A general interactive medical image segmentation system based on a large model according to claim 5, characterized in that, The system is used on a labeling platform to achieve semi-automatic segmentation of medical images.

Images