Image segmentation method and image processing system
By using the prompter model and the prompt enhancement program to generate the second prompt mark, combined with the segmentation basic model, the problem of high resource consumption in the image segmentation model training process in the prior art is solved, and efficient image segmentation is achieved while maintaining accuracy.
Patent Information
- Application Number
- CN202410975071.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-07-20
- Filing Date
- 2024-07-19
- Publication Date
- 2025-01-21
AI Technical Summary
During the training of existing image segmentation models, a large amount of label training data is required, a lot of time and computing resources are spent, especially when improvements or new functions are needed, repeating the training steps will consume a lot of resources.
By providing the input image to the prompter model, the first cue mark is generated according to the task type and converted into the second cue mark by the cue enhancement program. Finally, the input image and the second cue mark are provided to the segmentation base model to generate the output segmentation mask. .
Image segmentation is achieved without using a large amount of labeled data to train a large-scale model, simplifying the data annotation task, reducing training costs, and maintaining the accuracy of the output segmentation mask.
Smart Images
Figure CN119339068A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to an image segmentation method and an image processing system, and more particularly to an image segmentation method and an image processing system integrated with a segmentation basic model. Background Art
[0002] In recent years, artificial intelligence (AI) has developed rapidly, and various application fields such as industry, commerce, and medicine have begun to widely adopt AI technology to assist their operations. However, most companies lack the corresponding capabilities to build AI models that suit their needs. Therefore, most companies tend to use automated machine learning platforms or seek cooperation to build their AI models. Automated machine learning platforms enable enterprise users with basic knowledge of AI to build AI models dedicated to their companies through automated training methods. These platforms provide users with a variety of model options, allowing them to select models with corresponding capabilities according to their respective needs and train the selected models using their collected data sets.
[0003] Although the automated machine learning platform provides a variety of models for users to choose from, the key part required for training, that is, training data with accurate labels, still needs to be prepared by the user. How to establish a large amount of training data with accurate labels is a major burden for users, because it involves setting labeling rules, preparing unlabeled data sets, and requiring professionals (such as production line technicians, quality control personnel, or radiologists) to label according to established rules and generate labeled data. If the task involves image segmentation, it is even more time-consuming. Image segmentation tasks require labeling of each pixel, which takes a lot of time even for experienced labelers. Generally speaking, training machine learning models requires a large amount of labeled training data, a certain amount of time, and the computing equipment required for training.
[0004] As mentioned above, the existing image segmentation model training process includes the following steps: collecting data, annotating data, selecting a suitable model, loading a label data set, training the model, and obtaining a trained model. For example, the model can be trained based on image data with labels containing dogs, so that the position of the dog can be distinguished and located in the input image. If the trained model needs to be improved (for example, adding a segmentation function for cats), at least part of the training steps need to be repeated to retrain the model, and these retraining steps will consume a lot of time, manpower and computing resources. Summary of the invention
[0005] One aspect of the present disclosure document discloses an image segmentation method, comprising the following steps: providing an input image to a prompter model to generate a first prompt marker according to a task type of the prompter model; performing a prompt enhancement procedure on the first prompt marker with reference to the task type of the prompter model to generate a second prompt marker, wherein the prompt enhancement procedure includes converting a position, a size or a prompt type of the first prompt marker with reference to the task type to form the second prompt marker; and providing the input image and the second prompt marker to a segmentation base model, and generating an output segmentation mask on the input image according to the second prompt marker through the segmentation base model.
[0006] In some embodiments, the task type of the prompter model includes an object detection type, an image segmentation type, or a classification type.
[0007] In some embodiments, in response to the task type of the prompter model being an object detection type, the first prompt mark generated by the prompter model is a first bounding box, wherein the prompt enhancement procedure performed on the first prompt mark includes: moving the position of the first bounding box to form a second bounding box, enlarging a size of the first bounding box to the second bounding box, or reducing the size of the first bounding box to the second bounding box, wherein the second prompt mark includes the second bounding box.
[0008] In some embodiments, the task type in response to the prompter model is the object detection type, wherein the prompt enhancement procedure performed on the first prompt mark further includes: selecting a first node within the second bounding box, or selecting a second node outside the second bounding box, wherein the second prompt mark further includes a positive click at the first node or a negative click at the second node.
[0009] In some embodiments, in response to the task type of the prompter model being an image segmentation type, the first prompt mark generated by the prompter model is a first segmentation mask, wherein the prompt enhancement procedure performed on the first prompt mark includes: moving the position of the first segmentation mask to form a second segmentation mask, enlarging a size of the first segmentation mask to the second segmentation mask, reducing the size of the first segmentation mask to the second segmentation mask, or obtaining a bounding box based on multiple boundary nodes of the first segmentation mask, wherein the second prompt mark includes the second segmentation mask or the bounding box.
[0010] In some embodiments, the task type in response to the prompter model is the image segmentation type, wherein the prompt enhancement procedure performed on the first prompt mark further includes: selecting a first node within the second segmentation mask, or selecting a second node outside the second segmentation mask, wherein the second prompt mark further includes a positive click at the first node or a negative click at the second node.
[0011] In some embodiments, in response to the task type of the prompter model being a classification type, the first prompt mark generated by the prompter model is a heat map, wherein the prompt enhancement procedure performed on the first prompt mark includes: comparing the values of multiple feature points in the heat map with a threshold; locating multiple first feature points in the heat map that exceed the threshold, and locating multiple second feature points in the heat map that are below the threshold; and obtaining a bounding box based on multiple boundary nodes of the multiple first feature points, or obtaining a segmentation mask based on the multiple first feature points, wherein the second prompt mark includes the bounding box or the segmentation mask.
[0012] In some embodiments, the task type in response to the prompter model is the classification type, wherein the prompt enhancement procedure performed on the first prompt mark further includes: selecting a first node from the multiple first feature points, or selecting a second node from the multiple second feature points, wherein the second prompt mark further includes a positive click at the first node or a negative click at the second node.
[0013] In some embodiments, the image segmentation method further comprises: generating an output image according to outputting the segmentation mask and the input image.
[0014] In some embodiments, the prompter model is trained based on a training dataset to achieve the task type, wherein the training dataset includes a plurality of training images and label data corresponding to the plurality of training images.
[0015] In some embodiments, the prompter model is trained using a dataset of smaller size and specific type, whereas the segmentation base model is trained using a dataset of larger size and diverse type.
[0016] In some embodiments, the segmentation base model includes an image encoder, a hint encoder, and a mask decoder. The image encoder is used to receive the input image and convert the input image into an input image embedding vector. The hint encoder is used to receive the second hint tag and convert the second hint tag into a hint vector. The mask decoder is used to generate the output segmentation mask according to the input image embedding vector and the hint vector.
[0017] Another aspect of the disclosure document discloses an image processing system, comprising a storage unit and a processing unit. The storage unit is used to store a plurality of computer executable instructions. The processing unit is coupled to the storage unit, and the processing unit is used to execute the plurality of computer executable instructions, and the processing unit is used to: provide an input image to a prompter model to generate a first prompt mark according to a task type of the prompter model; perform a prompt enhancement procedure on the first prompt mark with reference to the task type of the prompter model to generate a second prompt mark, wherein the prompt enhancement procedure includes converting a position, a size or a prompt type of the first prompt mark with reference to the task type to form the second prompt mark; and provide the input image and the second prompt mark to a segmentation base model, and generate an output segmentation mask on the input image according to the second prompt mark through the segmentation base model.
[0018] In some embodiments, in response to the task type of the prompter model being an object detection type, the first prompt mark generated by the prompter model is a first bounding box, wherein the prompt enhancement procedure performed by the processing unit on the first prompt mark includes: moving the position of the first bounding box to form a second bounding box, enlarging a size of the first bounding box to the second bounding box, or reducing the size of the first bounding box to the second bounding box, wherein the second prompt mark includes the second bounding box.
[0019] In some embodiments, the task type in response to the prompter model is the object detection type, wherein the prompt enhancement procedure performed by the processing unit on the first prompt mark further includes: selecting a first node within the second bounding box, or selecting a second node outside the second bounding box, wherein the second prompt mark further includes a positive click at the first node or a negative click at the second node.
[0020] In some embodiments, in response to the task type of the prompter model being an image segmentation type, the first prompt mark generated by the prompter model is a first segmentation mask, wherein the prompt enhancement procedure performed by the processing unit on the first prompt mark includes: moving the position of the first segmentation mask to form a second segmentation mask, enlarging a size of the first segmentation mask to the second segmentation mask, reducing the size of the first segmentation mask to the second segmentation mask, or obtaining a bounding box based on multiple boundary nodes of the first segmentation mask, wherein the second prompt mark includes the second segmentation mask or the bounding box.
[0021] In some embodiments, the task type in response to the prompter model is the image segmentation type, wherein the prompt enhancement procedure performed by the processing unit on the first prompt mark further includes: selecting a first node within the second segmentation mask, or selecting a second node outside the second segmentation mask, wherein the second prompt mark further includes a positive click at the first node or a negative click at the second node.
[0022] In some embodiments, in response to the task type of the prompter model being a classification type, the first prompt mark generated by the prompter model is a heat map, wherein the prompt enhancement procedure performed by the processing unit on the first prompt mark includes: comparing the values of multiple feature points in the heat map with a threshold; locating multiple first feature points in the heat map that exceed the threshold, and locating multiple second feature points in the heat map that are below the threshold; and obtaining a bounding box based on multiple boundary nodes of the multiple first feature points, or obtaining a segmentation mask based on the multiple first feature points, wherein the second prompt mark includes the bounding box or the segmentation mask.
[0023] In some embodiments, the task type in response to the prompter model is the classification type, wherein the prompt enhancement procedure performed on the first prompt mark further includes: selecting a first node from the multiple first feature points, or selecting a second node from the multiple second feature points, wherein the second prompt mark further includes a positive click at the first node or a negative click at the second node.
[0024] In some embodiments, the processing unit is configured to generate an output image according to the output segmentation mask and the input image.
[0025] The image processing system of the present disclosure is capable of performing image segmentation without using a large amount of labeled data to train a large-scale model. The image processing system can train a relatively small-scale prompter model, and the image processing system uses the prompter model and a prompt enhancement procedure to generate a second prompt tag, which is sent to the segmentation base model to generate an output segmentation mask.
[0026] Under the framework of the present disclosure, a small model with low computational complexity can be selected as a prompter model, and the difficulty of the task of data annotation (used to train the prompter model) can be simplified. When a new task target needs to be added, the image processing system does not need to retrain a large-scale model, and the image processing system can use a limited amount of labeled data to train the prompter model, which helps to reduce the annotation cost (such as time cost and labor cost) of training the prompter model. At the same time, the segmentation result (such as output segmentation mask) can be output based on the appropriate prompt input (such as the second prompt mark) by the segmentation base model, thereby maintaining the accuracy of the output segmentation mask.
[0027] It should be noted that the above description and the following detailed description are illustrative of the present invention in the form of embodiments, and are used to assist in the explanation and understanding of the invention content claimed in the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] In order to make the above and other objects, features and embodiments of the present disclosure more clearly understood, the accompanying drawings are described as follows:
[0029] Figure 1 A schematic diagram showing an image processing system according to some embodiments of the present disclosure;
[0030] Figure 2 A flowchart showing an image segmentation method in some embodiments of the present disclosure is shown;
[0031] Figure 3 A schematic diagram showing a first prompt mark, a second prompt mark, an output segmentation mask and an output image generated based on an input image during execution of an image segmentation method in some embodiments;
[0032] Figure 4 A method flow chart showing a method for training a prompter model according to some embodiments of the present disclosure;
[0033] Figure 5 A schematic diagram showing a corresponding input image, a first prompt mark and a second prompt mark in an exemplary example when the task type of the prompter model is an object detection type;
[0034] Figure 6A schematic diagram showing a corresponding input image, a first prompt mark and a second prompt mark when the task type of the prompter model is an image segmentation type in another exemplary example;
[0035] Figure 7 A schematic diagram showing a corresponding input image, a first prompt mark, and a second prompt mark when the task type of the prompter model is a classification type in another exemplary example;
[0036] Fig. 8A as well as Figure 8B A schematic diagram showing different output segmentation masks generated by a segmentation base model according to different hint labels in some embodiments.
[0037] Explanation of symbols:
[0038] 100: Image processing system
[0039] 120: Storage unit
[0040] 140: Processing unit
[0041] 142: Prompt Model
[0042] 144: Prompt Enhancement Program
[0043] 146: Result Generator
[0044] 160: Interface
[0045] 180: Communication circuit
[0046] 190: Server
[0047] 192: Segmentation base model
[0048] 192a: Image Encoder
[0049] 192b: Prompt encoder
[0050] 192c: Mask decoder
[0051] 200: Image Segmentation Method
[0052] 400: Training Methods
[0053] IMGe: Input image embedding vector
[0054] IMGi: Input image
[0055] IMGo: Exporting Images
[0056] PM1: First prompt mark
[0057] PM1a: Bounding Box
[0058] PM1b: First segmentation mask
[0059] PM1c: Heatmap
[0060] PM2: Second prompt mark
[0061] PM2a: Forward Click
[0062] PM2b: Negative click
[0063] PM2c: Second Bounding Box
[0064] PM2d: Positive trajectory
[0065] PM2e: Negative trajectory
[0066] PM2f: Second Split Mask
[0067] PV: Prompt Vector
[0068] S210, S220, S230, S240: Steps
[0069] S410, S420, S430, S440, S450, S460: Steps
[0070] SMK,SMK1,SMK2: Output segmentation mask DETAILED DESCRIPTION
[0071] The following disclosure provides many different embodiments or examples for implementing different features of the present disclosure. The elements and configurations in the specific examples are used to simplify the present disclosure in the following discussion. Any examples discussed are used for illustrative purposes only and do not limit the scope and significance of the present disclosure or its examples in any way. Where appropriate, the same reference numerals are used between the drawings and in the corresponding text descriptions to represent the same or similar elements.
[0072] See also Figure 1 , which shows a schematic diagram of an image processing system 100 according to some embodiments of the present disclosure. In some embodiments, the image processing system 100 is capable of performing a specific image segmentation task on the input image IMGi. For example, the image processing system 100 can be used to mark potential tumor locations in magnetic resonance imaging (MRI) scan images, identify license plates in traffic monitoring images, mark cats in grass backgrounds, or separate foreground characters from background scenery. In some embodiments, the image processing system 100 is used to operate in conjunction with a segmentation foundation model (SFM) 192 to perform the above-mentioned image segmentation tasks.
[0073] The so-called segmentation base model 192 is a comprehensive deep learning framework for performing various image segmentation tasks. The segmentation base model 192 can be used as a general tool in computer vision to segment images into meaningful parts or localities. The segmentation base model 192 is usually based on advanced neural network architectures, such as transformers, convolutional neural networks (CNN) or fully convolutional networks (FCN). The segmentation base model 192 is trained based on a large-scale and diverse data set, so that it can be widely used in various types of images and various segmentation tasks. In some embodiments, the segmentation base model 192 can be run on an external server 190 outside the image processing system 100.
[0074] In some embodiments, the segmentation base model 192 can be implemented using a Segment Anything Model (SAM). The Segment Anything Model (SAM) can complete image segmentation tasks based on prompt word instructions. The Segment Anything Model supports multiple types of prompt tags (such as positive clicks, negative clicks, bounding boxes, or text descriptions). However, there are some limitations when the Segment Anything Model is applied to customized object segmentation tasks. First, the Segment Anything Model requires correct and appropriate prompt tags to clearly indicate the intent of the target segmentation task. Second, the Anything Segmentation Model requires additional specific training (related to the target segmentation task) to achieve the best segmentation effect.
[0075] The image processing system 100 in the present disclosure provides a solution for generating appropriate hint labels based on the input image IMGi and providing the appropriate hint labels to the segmentation base model 192. Figure 1As shown, in some embodiments, the image processing system 100 includes a storage unit 120, a processing unit 140, an interface 160, and a communication circuit 180. The processing unit 140 is coupled to the storage unit 120, the interface 160, and the communication circuit 180. In some embodiments, the storage unit 120 can be implemented by a memory, a cache, or a hard disk; the processing unit 140 can be implemented by a central processing unit (CPU), a graphic processing unit (GPU), a tensor processing unit (TPU), or an application-specific integrated circuit (ASIC); the interface 160 can include an input end (such as a connector, an image scanner, or a camera) and an output end (such as a display, a connector, or a projector); the communication circuit 180 can be implemented by a network adapter, a telecommunication communication circuit, or a wireless communication circuit.
[0076] The storage unit 120 is used to store computer executable instructions. These computer executable instructions can be executed by the processing unit 140 to perform the image segmentation method. Figure 2 and Figure 3 , Figure 2 A flowchart of an image segmentation method 200 in some embodiments of the present disclosure is shown. Figure 3 A schematic diagram showing a first prompt mark PM1, a second prompt mark PM2, an output segmentation mask SMK and an output image IMGo generated based on an input image IMGi during execution of the image segmentation method 200 in some embodiments.
[0077] Assume that the new task goal is to establish a new image segmentation model that can segment the cat from the background. In order to achieve this new task goal, the image processing system 100 uses the prompter model 142 and the prompt enhancement program 144 to generate the appropriate prompt marks (e.g., positive clicks for the cat, negative clicks for the background, a bounding box around the cat, or a rough segmentation mask for the cat) required for the segmentation base model 192 to run. In this example, the segmentation base model 192 can generate an accurate segmentation mask about the cat based on the appropriate prompt marks generated by the prompter model 142 and the prompt enhancement program 144. More details about the prompter model 142 and the prompt enhancement program 144 will be discussed in subsequent paragraphs.
[0078] like Figure 1 , Figure 2 as well as Figure 3As shown, the image processing system 100 receives the input image IMGi through the interface 160. In step S210, the processing unit 140 provides the input image IMGi to the prompter model 142 to generate a first prompt mark PM1 (e.g., a bounding box PM1a, a first segmentation mask PM1b, or a heat map PM1c) according to the task type of the prompter model 142. In step S220, the processing unit 140 executes the prompt enhancement program 144 on the first prompt mark PM1 according to the task type of the prompter model 142 to generate a second prompt mark PM2 (e.g., a positive click PM2a, a negative click PM2b, a second bounding box PM2c, a positive track PM2d, a negative track PM2e, or a second segmentation mask PM2f). In step S230 , the image processing system 100 provides the input image IMGi and the second hint mark PM2 to the segmentation basic model 192 through the communication circuit 180 , so that the segmentation basic model 192 can generate an output segmentation mask SMK on the input image IMGi according to the second hint mark PM2 .
[0079] By working in conjunction with the segmentation base model 192, the image processing system 100 can obtain an output segmentation mask SMK. Figure 1 , Figure 2 and Figure 3 As shown, in step S240, the result generator 146 executed by the processing unit 140 is capable of generating an output image IMGo according to the output segmentation mask SMK and the input image IMGi.
[0080] In some embodiments, the prompter model 142, the prompt enhancement program 144, and the result generator 146 can be implemented by software instructions executed by the processing unit 140. The task type of the prompter model 142 can be selected from a plurality of candidate task types, such as object detection type, image segmentation type, and classification type. The prompter model 142 can be trained according to a training data set to complete one of the object detection, image segmentation, or classification tasks. The training data set includes a plurality of training images and label data corresponding to the plurality of training images.
[0081] Please also read Figure 4 , Figure 4 1 is a flowchart showing a method 400 for training a prompter model 142 according to some embodiments of the present disclosure. Figure 4As shown, the training method 400 performs step S410 to collect a plurality of training images (e.g., images involving cats). Perform step S420 to annotate the training images to generate label data. Perform step S430 to select the task type of the prompter model 142. For example, the task type of the prompter model 142 can be selected as one of an object detection type (e.g., the prompter model 142 can use a YOLO or Faster R-CNN model), an image segmentation type (e.g., the prompter model 142 can use a Unet or Deeplab model), and a classification type (e.g., the prompter model 142 can use a VGGNet or ResNet model). Perform step S440 to load a data set (including a plurality of training images and corresponding label data) to prepare for training. Perform step S450 to train the prompter model 142. In some embodiments, step S450 includes: generating a prediction result of the training image by the prompter model 142, comparing the prediction result with the label data, and adjusting the parameters of the prompter model 142 according to the above comparison result. Execute step S460 to obtain the trained prompter model 142.
[0082] In some embodiments, the prompt model 142 is trained using a smaller scale and specific type of data set compared to the segmentation base model 192. This is because the prompt model 142 does not need to generate accurate image segmentation mask positions, and the output of the prompt model 142 is only used to generate prompt labels (as an indication / clue to the segmentation intention of the segmentation base model 192), while the relatively heavy task of generating accurate image segmentation mask positions is completed by the segmentation base model 192 based on the received prompt labels. In addition, the prompt model 142 can be specifically used to achieve a specific purpose (such as finding cats), so the prompt model 142 does not need to process a variety of different types of input images. Therefore, the prompt model 142 can be based on a relatively simple model and implemented with limited computing resources, and the prompt model 142 can complete training faster based on a smaller scale and specific type of data set.
[0083] like Figure 4 The prompter model 142 may be selected from various models corresponding to various task types, including object detection type, image segmentation type, or classification type. In some embodiments, since the prompter model 142 is selected from various task types, the first prompt mark PM1 generated by the prompter model 142 will have different formats with reference to the task type of the prompter model 142. Figure 3As shown, the first prompt mark PM1 generated by the prompter model 142 can be one of a bounding box PM1a, a first segmentation mask PM1b, or a heat map PM1c. Corresponding to the various types of the first prompt mark PM1 (i.e., bounding box PM1a, first segmentation mask PM1b, or heat map PM1c), the prompt enhancement program 144 used to process the first prompt mark PM1 will be different accordingly.
[0084] Please also read Figure 5 , which is a schematic diagram showing the corresponding input image IMGi, the first prompt mark PM1 and the second prompt mark PM2 when the task type of the prompter model 142 is the object detection type in an exemplary example.
[0085] In this example, the hint model 142 is an object detection model trained using a limited number of images involving cats, and the hint model 142 is used to mark the bounding box containing the cat in the input image IMGi. Figure 5 As shown, in some embodiments, when the task type of the prompt model 142 is the object detection type, the prompt model 142 generates a first prompt mark PM1 including a first bounding box PM1a, and the first bounding box PM1a corresponds to the approximate position of the object of interest (such as a cat).
[0086] like Figure 5 As shown, the hint enhancement program 144 is used to convert the first hint mark PM1 (ie, the first bounding box PM1a) to convert the first bounding box PM1a into the second bounding box PM2c.
[0087] In some embodiments, the aforementioned transformation may be (1) moving the position of the first bounding box PM1a to form the second bounding box PM2c, (2) enlarging the size of the first bounding box PM1a to the second bounding box PM2c, (3) reducing the size of the first bounding box PM1a to the second bounding box PM2c, or (4) copying the first bounding box PM1a to the second bounding box PM2c.
[0088] exist Figure 5 In the embodiment of the present invention, the size of the first bounding box PM1a is reduced to form the second bounding box PM2c of the second prompt mark PM2. However, the present disclosure is not limited thereto. The conversion method is determined according to the characteristics of the segmentation basic model 192.
[0089] like Figure 5As shown, the conversion of the first prompt mark PM1 by the prompt enhancement program 144 further includes selecting a node in the second bounding box PM2c as the position of the forward click PM2a. In some examples, the center point of the second bounding box PM2c can be selected as the position of the forward click PM2a, and the second prompt mark PM2 can also include the forward click PM2a.
[0090] like Figure 5 As shown, the conversion of the first prompt mark PM1 by the prompt enhancement program 144 further includes selecting one or more nodes outside the second bounding box PM2c as the position of the negative click PM2b. In some examples, a random node away from the second bounding box PM2c can be selected as the position of the negative click PM2b, and the second prompt mark PM2 can also include the negative click PM2b.
[0091] In some embodiments, the second prompt mark PM2 may include: Figure 5 The combination of the positive click PM2a, the negative click PM2b and the second bounding box PM2c shown. The positive click PM2a, the negative click PM2b and the second bounding box PM2c can be integrated together and sent to the segmentation base model 192 as a prompt input.
[0092] Please also read Figure 6 , which shows a schematic diagram of the corresponding input image IMGi, the first prompt mark PM1 and the second prompt mark PM2 when the task type of the prompter model 142 is the image segmentation type in another exemplary example.
[0093] In this example, the prompter model 142 is an image segmentation model trained using a limited number of images involving cats, and the prompter model 142 is used to mark a segmentation mask covering the area where the cat is located in the input image IMGi. Figure 6 As shown, in some embodiments, when the task type of the prompter model 142 is the image segmentation type, the first prompt mark PM1 generated by the prompter model 142 includes a first segmentation mask PM1b, and the first segmentation mask PM1b covers an approximate area of the object of interest (such as a cat).
[0094] like Figure 6 As shown, the hint enhancing program 144 is used to convert the first hint mark PM1 (ie, the first segmentation mask PM1b) to convert the first segmentation mask PM1b into the second segmentation mask PM2f.
[0095] In some embodiments, the aforementioned transformation may be (1) moving the position of the first segmentation mask PM1b to form the second segmentation mask PM2f, (2) enlarging the size of the first segmentation mask PM1b to the second segmentation mask PM2f, (3) reducing the size of the first segmentation mask PM1b to the second segmentation mask PM2f, or (4) copying the first segmentation mask PM1b to the second segmentation mask PM2f.
[0096] exist Figure 6 In the illustrated embodiment, the size of the first segmentation mask PM1b is reduced to form the second segmentation mask PM2f of the second prompt mark PM2. However, the present disclosure is not limited thereto.
[0097] like Figure 6 As shown, the conversion of the first prompt mark PM1 by the prompt enhancement program 144 further includes obtaining the second bounding box PM2c according to the position of the boundary node of the first segmentation mask PM1b. In some examples, the size and position of the second bounding box PM2c can be determined according to the top point, bottom point, rightmost point, and leftmost point of the first segmentation mask PM1b, and the second prompt mark PM2 can also include the second bounding box PM2c.
[0098] like Figure 6 As shown, the conversion of the first prompt mark PM1 by the prompt enhancement program 144 further includes selecting a node in the second segmentation mask PM2f as the position of the forward click PM2a. In some examples, the center point of the second segmentation mask PM2f can be selected as the position of the forward click PM2a, and the second prompt mark PM2 can also include the forward click PM2a.
[0099] like Figure 6 As shown, the conversion of the first prompt mark PM1 by the prompt enhancement program 144 further includes selecting one or more nodes outside the second segmentation mask PM2f as the location of the negative click PM2b. In some examples, a random node away from the second segmentation mask PM2f can be selected as the location of the negative click PM2b, and the second prompt mark PM2 can also include the negative click PM2b.
[0100] In some embodiments, the second prompt mark PM2 may include: Figure 6 The combination of the positive click PM2a, the negative click PM2b, the second bounding box PM2c and the second segmentation mask PM2f shown. The positive click PM2a, the negative click PM2b, the second bounding box PM2c and the second segmentation mask PM2f can be integrated together and sent to the segmentation base model 192 as a prompt input.
[0101] Please also read Figure 7, which shows a schematic diagram of the corresponding input image IMGi, the first prompt mark PM1 and the second prompt mark PM2 when the task type of the prompter model 142 is a classification type in another exemplary example.
[0102] In this example, the prompt model 142 is a classification model trained using a limited number of images involving cats, and the prompt model 142 is used to generate a heat map PM1c. The values of multiple feature points in the heat map PM1c will change depending on the probability value of whether they belong to the area where the object of interest (such as a cat) is located. For example, when a feature point has a higher probability of overlapping with a cat, it will be darker (such as a higher value) in the heat map, and when a feature point has a lower probability of overlapping with a cat, it will be lighter (such as a lower value) in the heat map, such as Figure 7 shown.
[0103] like Figure 7 As shown, the prompt enhancement procedure 144 for the first prompt mark PM1 (e.g., heat map PM1c) includes (1) comparing the values of multiple feature points in the heat map PM1c with a threshold; (2) locating multiple first feature points in the heat map PM1c that exceed the threshold, and locating multiple second feature points in the heat map PM1c that are below the threshold; (3) obtaining a second bounding box PM2c based on multiple boundary nodes of these first feature points, or obtaining a second segmentation mask PM2f based on these first feature points. In this example, the second bounding box PM2c and / or the second segmentation mask PM2f can be integrated and added to the combination of the second prompt mark PM2.
[0104] exist Figure 7 In the illustrated embodiment, the inner portion of the heat map PM1c corresponding to the higher probability value (ie, the first feature points exceeding the threshold) forms a second segmentation mask PM2f, and the boundary nodes of the second segmentation mask PM2f form a second bounding box PM2c.
[0105] like Figure 7 As shown, the conversion of the first prompt mark PM1 by the prompt enhancement program 144 further includes selecting the first node from the plurality of first feature points (corresponding to the area within the second segmentation mask PM2f) exceeding the threshold as the position of the positive click PM2a. In this example, the center point of the second segmentation mask PM2f can be selected as the position of the positive click PM2a. The second prompt mark PM2 can also include the positive click PM2a.
[0106] like Figure 7As shown, the conversion of the first prompt mark PM1 by the prompt enhancement program 144 further includes selecting one or more second nodes from the plurality of second feature points (corresponding to the area outside the second segmentation mask PM2f) below the threshold as the location of the negative click PM2b. In some examples, a random node away from the second segmentation mask PM2f can be selected as the location of the negative click PM2b, and the second prompt mark PM2 can also include the negative click PM2b.
[0107] In some embodiments, the second prompt mark PM2 may include: Figure 7 The combination of the positive click PM2a, the negative click PM2b, the second bounding box PM2c and the second segmentation mask PM2f shown. The positive click PM2a, the negative click PM2b, the second bounding box PM2c and the second segmentation mask PM2f can be integrated together and sent to the segmentation base model 192 as a prompt input.
[0108] like Figure 3 As shown, the segmentation basic model 192 includes an image encoder 192a, a hint encoder 192b, and a mask decoder 192c. The image encoder 192a is used to receive an input image IMGi and convert the input image IMGi into an input image embedding vector IMGe. The hint encoder 192b is used to receive a second hint marker PM2 and convert the second hint marker PM2 into a hint vector PV. The mask decoder 192c is used to generate an output segmentation mask SMK according to the input image embedding vector IMGe generated by the image encoder 192a and the hint vector PV generated by the hint encoder 192b.
[0109] In some embodiments, the segmentation base model 192 can be implemented by a segmentation any model (SAM). However, the present disclosure is not limited thereto. The segmentation base model 192 can be implemented by other similar base models based on a converter architecture, or other similar base models based on an encoder-decoder architecture.
[0110] Please also read Fig. 8A as well as Figure 8B , which is a schematic diagram showing different output segmentation masks generated by the segmentation base model 192 according to different hint labels in some embodiments.
[0111] like Fig. 8A 1 is an example of the segmentation base model 192 receiving the input image IMGi and the first hint mark PM1. In this example, the hint input to the segmentation base model 192 is the first hint mark PM1 that has not been processed by the hint enhancement program 144. Fig. 8AAs shown in FIG. 1 , the first hint PM1 only contains a bounding box that marks the approximate location of the liver. However, the bounding box also covers some areas around the liver. Fig. 8A The output segmentation mask SMK1 generated by the segmentation base model 192 may also cover other objects around the liver. In other words, the output segmentation mask SMK1 generated according to the first prompt marker PM1 is not accurate enough.
[0112] like Figure 8B Shown is another example where the segmentation base model 192 receives an input image IMGi and a second hint marker PM2. In this case, the hint input to the segmentation base model 192 is the second hint marker PM2 that has been processed by the hint enhancement program 144. As shown in FIG8B , the second hint marker PM2 includes a combination of a positive click, three negative clicks, and a bounding box. This bounding box marks the approximate location of the liver. The positive click marks the target to be segmented (i.e., the liver). These negative clicks mark the parts that should not be included in the segmentation target. Based on the hint provided by the second hint marker PM2, the output segmentation mask SMK2 generated by the segmentation base model 192 is shown in FIG8B . Figure 8B As shown, the target in this example (ie, the liver) can be segmented more accurately.
[0113] As discussed in the foregoing embodiments, the image processing system 100 is capable of performing image segmentation without using a large amount of labeled data to train a large-scale model. The image processing system 100 can train a relatively small-scale prompter model 142, and the image processing system 100 uses the prompter model 142 and the prompt enhancement program 144 to generate a second prompt mark PM2, and sends it to the segmentation base model 192 to generate an output segmentation mask SMK. The segmentation base model 192 can be a pre-trained model that runs externally. In general, the segmentation base model 192 has excellent versatility (i.e., the ability to process different images), and the image processing system 100 that integrates the segmentation base model 192 can also process different types of images (e.g., medical images, traffic images, face recognition images, fingerprint images).
[0114] Under the framework of the present disclosure, a small model with low computational complexity can be selected as the prompter model 142, and the difficulty of the task of data annotation (used to train the prompter model 142) can be simplified. In this example, the prompter model 142 can be trained with a limited number of labeled data and simplified label categories. When a new task target needs to be added (for example, image segmentation related to dogs), the image processing system 100 does not need to use training data related to dogs to retrain a large-scale model. The image processing system 100 can use a limited number of labeled data related to dogs to train the prompter model 142, which helps to reduce the annotation cost (such as time cost and labor cost) of training the prompter model 142. At the same time, the segmentation result (such as the output segmentation mask SMK) can be output based on the appropriate prompt input (such as the second prompt mark PM2) through the segmentation base model 192, thereby maintaining the accuracy of the output segmentation mask SMK.
[0115] Although specific embodiments of the present disclosure have been disclosed with respect to the above-mentioned embodiments, these embodiments are not intended to limit the present disclosure. Various substitutions and improvements can be performed in the present disclosure by a person of ordinary skill in the relevant art without departing from the principles and spirit of the present disclosure. Therefore, the protection scope of the present disclosure is determined by the appended claims.
Claims
1. An image segmentation method, characterized in that: Include: Providing an input image to a prompter model to generate a first prompt mark according to a task type of the prompter model; performing a prompt enhancement process on the first prompt mark with reference to the task type of the prompter model to generate a second prompt mark, wherein the prompt enhancement process includes converting a position, a size or a prompt type of the first prompt mark with reference to the task type to form the second prompt mark; as well as The input image and the second hint mark are provided to a segmentation base model, and an output segmentation mask is generated on the input image according to the second hint mark by the segmentation base model.
2. The image segmentation method of claim 1 , wherein the task type in response to the prompter model is an object detection type, the first prompt mark generated by the prompter model is a first bounding box, The prompt enhancement procedure performed on the first prompt mark includes: moving the position of the first boundary box to form a second boundary box, enlarging a size of the first boundary box to the second boundary box, or reducing the size of the first boundary box to the second boundary box. The second hint mark includes the second bounding box.
3. The image segmentation method of claim 2, wherein the task type in response to the prompter model is the object detection type, wherein the prompt enhancement procedure performed on the first prompt mark further comprises: selecting a first node within the second bounding box, or selecting a second node outside the second bounding box, The second prompt mark further includes a positive click located at the first node or a negative click located at the second node.
4. The image segmentation method of claim 1 , wherein the task type in response to the prompter model is an image segmentation type, the first prompt mark generated by the prompter model is a first segmentation mask, The prompt enhancement procedure performed on the first prompt mark includes: moving the position of the first segmentation mask to form a second segmentation mask, enlarging a size of the first segmentation mask to the second segmentation mask, reducing the size of the first segmentation mask to the second segmentation mask, or obtaining a boundary box according to a plurality of boundary nodes of the first segmentation mask. The second hint mark includes the second segmentation mask or the bounding box.
5. The image segmentation method of claim 4, wherein the task type in response to the prompter model is the image segmentation type, wherein the prompt enhancement procedure performed on the first prompt mark further comprises: selecting a first node within the second segmentation mask, or selecting a second node outside the second segmentation mask, The second prompt mark further includes a positive click located at the first node or a negative click located at the second node.
6. The image segmentation method of claim 1, wherein the task type in response to the prompter model is a classification type, and the first prompt mark generated by the prompter model is a heat map, The prompt enhancement procedure performed on the first prompt mark includes: Comparing the values of the plurality of feature points in the heat map with a threshold; Locating a plurality of first feature points in the heat map that exceed the threshold, and locating a plurality of second feature points in the heat map that are below the threshold; and A bounding box is obtained according to the plurality of boundary nodes of the plurality of first feature points, or a segmentation mask is obtained according to the plurality of first feature points, wherein the second hint mark includes the bounding box or the segmentation mask.
7. The image segmentation method of claim 6, wherein the task type in response to the prompter model is the classification type, wherein the prompt enhancement procedure performed on the first prompt mark further comprises: selecting a first node from the plurality of first feature points, or selecting a second node from the plurality of second feature points, The second prompt mark further includes a positive click located at the first node or a negative click located at the second node.
8. The image segmentation method according to claim 1, further comprising: An output image is generated according to the output segmentation mask and the input image.
9. The image segmentation method according to claim 1, wherein the segmentation basic model comprises: an image encoder, configured to receive the input image and convert the input image into an input image embedding vector; a hint encoder, configured to receive the second hint tag and convert the second hint tag into a hint vector; and A mask decoder is used to generate the output segmentation mask according to the input image embedding vector and the hint vector.
10. An image processing system, characterized in that: Include: a storage unit for storing a plurality of computer executable instructions; and a processing unit coupled to the storage unit, the processing unit being configured to execute the plurality of computer executable instructions, the processing unit being configured to: Providing an input image to a prompter model to generate a first prompt label according to a task type of the prompter model; performing a prompt enhancement process on the first prompt mark with reference to the task type of the prompter model to generate a second prompt mark, wherein the prompt enhancement process includes converting a position, a size or a prompt type of the first prompt mark with reference to the task type to form the second prompt mark; as well as The input image and the second hint mark are provided to a segmentation base model, and an output segmentation mask is generated on the input image according to the second hint mark by the segmentation base model.