Image division method and image processing system

Abstract

To provide an image segmentation method.SOLUTION: An image segmentation method is provided, comprising: providing an input image to a prompter model to generate a first prompt indicator according to a task type of the prompter model; performing a prompt enhancement procedure with reference to the task type of the prompter mode on the first prompt indicator to generate a second prompt indicator, the prompt enhancement procedure including converting the location, size or prompt type of the first prompt indicator into the second prompt indicator with reference to the task type; and providing the input image and the second prompt indicator to a segmentation foundation model to generate an output segmentation mask on the input image according to the second prompt indicator.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to an image segmentation method and an image processing system, and more particularly to an image segmentation method and an image processing system integrated with a segmentation base model. [Background technology]

[0002] In recent years, artificial intelligence has developed rapidly, and various application fields, such as industry, commerce, and medicine, have already begun to widely adopt artificial intelligence technology to support their operations. However, many companies lack the corresponding capabilities to build AI models that meet their needs. As a result, many companies tend to use automated machine learning platforms or seek assistance to build AI models. Automated machine learning platforms allow business users with basic knowledge of AI to build AI models tailored to their needs through automated training methods. These platforms provide users with a variety of model options, allowing them to select models with corresponding capabilities according to their needs and train the selected models using collected datasets. Summary of the Invention [Problem to be solved by the invention]

[0003] While automated machine learning platforms offer multiple models for users to choose from, users must provide a key component of training: accurately labeled training data. Establishing a large amount of accurately labeled training data places a significant burden on users, as it requires setting marking rules, preparing unmarked datasets, and having experts (e.g., production line engineers, quality control personnel, and radiologists) mark and generate labeled data based on predefined rules. Image segmentation tasks are even more time-consuming. Image segmentation tasks require pixel-by-pixel marking, which can be time-consuming even for experienced markers. Training a machine learning model typically requires a large amount of labeled training data, a certain amount of time, and the computing resources required for training.

[0004] From the above, the conventional image segmentation model training process includes the steps of collecting data, marking data, selecting an appropriate model, loading a label dataset, training the model, and obtaining the trained model. For example, a model can be trained based on image data whose labels include dogs, so that the model can identify and locate the position of the dog in the input image. If the trained model needs to be improved (e.g., adding a segmentation function for cats), at least some of the training steps need to be repeated to retrain the model, and these retraining steps will consume a lot of time, effort, and computing resources. [Means for solving the problem]

[0005] According to one aspect of the present disclosure, an image segmentation method is disclosed, comprising the steps of: providing an input image to an indicator model to generate a first indication mark based on a task type of the indicator model; executing an indication enhancement program to generate the second indication mark for the first indication mark by referring to the task type of the indicator model, and converting the position, size, or indication type of the first indication mark by referring to the task type to form a second indication mark; and providing the input image and the second indication mark to a segmentation basic model, and generating an output segmentation mask in the input image based on the second indication mark using the segmentation basic model.

[0006] According to another aspect of the present disclosure, an image processing system is disclosed, including: a memory unit for storing a plurality of computer-executable instructions; and a processing unit coupled to the memory unit and used to execute the plurality of computer-executable instructions, the processing unit being used to: provide an input image to an indicator model to generate a first indication mark based on a task type of the indicator model; generate the second indication mark by executing an indication enhancement program including: the indication enhancement program converting the position, size, or indication type of the first indication mark by referring to the task type of the indicator model to form a second indication mark; provide the input image and the second indication mark to a segmentation base model; and generate an output segmentation mask in the input image based on the second indication mark by the segmentation base model.

[0007] The above description and the following detailed description are used to exemplify the present application in the form of embodiments and to assist in the interpretation and understanding of the inventive content required by the present application. [Brief explanation of the drawings]

[0008] To make the above and other objects, features and embodiments of the present disclosure more clearly and comprehensibly, the accompanying drawings are described as follows: [Figure 1] FIG. 1 is a schematic diagram of an image processing system according to some embodiments of the present disclosure. [Figure 2] 1 is a flowchart of an image segmentation method according to some embodiments of the present disclosure. [Figure 3] 3A-3C are schematic diagrams of a first indication mark, a second indication mark, an output segmentation mask, and an output image generated based on an input image during an image segmentation method according to some embodiments. [Figure 4] 1 is a method flowchart of a method for training an indicator model according to some embodiments of the present disclosure. [Figure 5] 10 is a schematic diagram of an input image, a first indication mark, and a second indication mark corresponding to a case where a task type of an indicator model is an object detection type in one illustrative example. FIG. [Figure 6] FIG. 10 is a schematic diagram of an input image, a first indication mark, and a second indication mark corresponding to a case where the task type of the indicator model is an image division type in another illustrative example. [Figure 7] 10 is a schematic diagram of an input image, a first indication mark, and a second indication mark corresponding to a case where the task type of the indicator model is a classification type in another illustrative example. FIG. [Figure 8A] 10A-10C are schematic diagrams illustrating different output segmentation masks that a segmentation base model generates based on different indication marks in some embodiments. [Figure 8B] 10A-10C are schematic diagrams illustrating different output segmentation masks that a segmentation base model generates based on different indication marks in some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0009] The following disclosure provides many different embodiments or examples for implementing different features of the present disclosure. Elements and configurations in specific examples are used in the following discussion to simplify the disclosure. Any examples discussed are for illustrative purposes only and do not limit the scope and meaning of the disclosure or its examples in any way. Where appropriate, the same reference numerals are used between the drawings and in the corresponding text description to represent the same or similar elements.

[0010] Please refer to FIG. 1, which is a schematic diagram of an image processing system 100 according to some embodiments of the present disclosure. In some embodiments, the image processing system 100 can perform specific image segmentation tasks on input images IMGi. For example, the image processing system 100 can be used to mark potential tumor locations in magnetic resonance imaging (MRI) scan images, identify license plates in traffic monitoring images, mark cats in grassy backgrounds, or separate foreground people from background scenery. In some embodiments, the image processing system 100 is used in conjunction with a segmentation basis model (SFM) 192 to perform the above image segmentation tasks.

[0011] The so-called segmentation base model 192 is a comprehensive deep learning framework for performing various image segmentation tasks. The segmentation base model 192 serves as a general-purpose tool in computer vision and can be used to segment an image into meaningful parts or regions. The segmentation base model 192 is typically based on advanced neural network architectures, such as converters, convolutional neural networks (CNNs), or fully convolutional neural networks (FCNs). The segmentation base model 192 is trained on large and diverse datasets for broad applicability to a variety of different types of images and various segmentation tasks. In some embodiments, the segmentation base model 192 can run on an external server 190 outside of the image processing system 100.

[0012] In some embodiments, the segmentation base model 192 may be implemented by a segment anything model (SAM). The segment anything model can complete image segmentation tasks based on instruction word commands. The segment anything model supports multiple types of instruction marks (e.g., positive clicks, negative clicks, edge boxes, or text descriptions). However, the segment anything model has some limitations when used for customized object segmentation tasks. First, the segment anything model requires accurate and appropriate instruction marks to clearly indicate the intent of the target segmentation task. Second, the segment anything model requires additional specific training (related to the target segmentation task) to achieve optimal segmentation results.

[0013] The image processing system 100 of the present disclosure provides a solution capable of generating appropriate indication marks based on input images IMGi and providing the appropriate indication marks to a segmentation base model 192. As shown in FIG. 1 , in some embodiments, the image processing system 100 includes a storage unit 120, a processing unit 140, an interface 160, and a communication circuit 180. The processing unit 140 is coupled to the storage unit 120, the interface 160, and the communication circuit 180. In some embodiments, the storage unit 120 may be implemented by a memory, a fast-access memory, or a hard disk. The processing unit 140 may be implemented by a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), or an application-specific integrated circuit (ASIC). The interface 160 may include an input end (e.g., a connector, an image scanner, or a camera) and an output end (e.g., a display, a connector, or a projector). The communication circuit 180 may be implemented by a network adapter, a telecommunications circuit, or a wireless communication circuit.

[0014] The storage unit 120 is used to store computer-executable instructions. These computer-executable instructions may be executed by the processing unit 140 to perform an image segmentation method. Please refer to Figures 2 and 3 together. Figure 2 is a flowchart of an image segmentation method 200 according to some embodiments of the present disclosure, and Figure 3 is a schematic diagram of a first indication mark PM1, a second indication mark PM2, an output segmentation mask SMK, and an output image IMGo generated based on an input image IMGi during execution of the image segmentation method 200 according to some embodiments.

[0015] Assume that the new task goal is to establish a new image segmentation model that can segment the cat from the background. To achieve this new task goal, the image processing system 100 utilizes the indicator model 142 and the indicator reinforcement program 144 to generate appropriate indicators (e.g., a positive click for the cat, a negative click for the background, an edge box surrounding the cat, or a coarse segmentation mask for the cat) required for the operation of the segmentation base model 192. In this example, the segmentation base model 192 can generate an accurate segmentation mask for the cat based on the appropriate indicators generated by the indicator model 142 and the indicator reinforcement program 144. More details about the indicator model 142 and the indicator reinforcement program 144 are provided in subsequent paragraphs.

[0016] 1, 2, and 3, the image processing system 100 receives an input image IMGi via the interface 160. In step S210, the processing unit 140 provides the input image IMGi to the indicator model 142 based on the task type of the indicator model 142 to generate a first indication mark PM1 (e.g., an edge box PM1a, a first segmented mask PM1b, or a heat map PM1c). In step S220, the processing unit 140 executes the indication reinforcement program 144 on the first indication mark PM1 based on the task type of the indicator model 142 to generate a second indication mark PM2 (e.g., a positive click PM2a, a negative click PM2b, a second edge box PM2c, a positive trajectory PM2d, a negative trajectory PM2e, or a second segmented mask PM2f). In step S230, the image processing system 100 provides the input image IMGi and the second indication mark PM2 to the division basic model 192 via the communication circuit 180, so that the division basic model 192 can generate an output division mask SMK in the input image IMGi based on the second indication mark PM2.

[0017] By working with the segmentation base model 192, the image processing system 100 can obtain an output segmentation mask SMK. As shown in Figures 1, 2 and 3, in step S240, the result generation device 146 executed by the processing unit 140 can generate an output image IMGo based on the output segmentation mask SMK and the input image IMGi.

[0018] In some embodiments, the indicator model 142, the indicator reinforcement program 144, and the result generator 146 may be implemented by software instructions executed by the processing unit 140. The task type of the indicator model 142 may be selected from a plurality of candidate task types, such as an object detection type, an image segmentation type, and a classification type. The indicator model 142 is trained based on a training dataset and can further complete one of object detection, image segmentation, or classification tasks. The training dataset includes a plurality of training images and label data corresponding to the plurality of training images.

[0019] Please also refer to FIG. 4. FIG. 4 is a method flowchart of a training method 400 for the indicator model 142 according to some embodiments of the present disclosure. As shown in FIG. 4, the training method 400 performs step S410 to collect a plurality of training images (e.g., images of cats). Step S420 to mark the training images to generate label data. Step S430 to select a task type for the indicator model 142. For example, the task type for the indicator model 142 may be one of an object detection type (e.g., the indicator model 142 may use a YOLO or Faster R-CNN model), an image segmentation type (e.g., the indicator model 142 may use a Unet or DeepLab model), and a classification type (e.g., the indicator model 142 may use a VGGNet or ResNet model). Step S440 to load a dataset (including a plurality of training images and corresponding label data) and prepare it for training. Step S450 to train the indicator model 142. In some embodiments, step S450 includes the indicator model 142 generating a prediction result for the training image, comparing the prediction result with the label data, and adjusting parameters of the indicator model 142 based on the comparison result. Step S460 is performed to obtain the trained indicator model 142.

[0020] In some embodiments, the indicator model 142 is trained using a small, specific type of dataset compared to the segmentation base model 192. This is because the indicator model 142 does not need to generate accurate image segmentation mask positions; the output of the indicator model 142 is used only to generate indicator marks (as an indication / cue of the segmentation intention of the segmentation base model 192), and the heavy task of generating accurate image segmentation mask positions is performed by the segmentation base model 192 based on the received indicator marks. Note that the indicator model 142 can be specialized to achieve a specific purpose (e.g., cat hunting), and does not need to process various types of input images. Therefore, the indicator model 142 can be implemented based on a simple model with limited computational resources, and can complete training more quickly based on a small, specific type of dataset.

[0021] As shown in FIG. 4 , the indicator model 142 can be selected from a variety of models corresponding to a plurality of different task types, including an object detection type, an image segmentation type, or a classification type. In some embodiments, since the indicator model 142 is selected from a variety of different task types, the first indicator mark PM1 generated by the indicator model 142 has a different format based on the task type of the indicator model 142. As shown in FIG. 3 , the first indicator mark PM1 generated by the indicator model 142 may be one of an edge box PM1 a, a first segmentation mask PM1 b, or a heat map PM1 c. The instruction enhancement program 144 for processing the first indicator mark PM1 differs depending on the type of the first indicator mark PM1 (i.e., the edge box PM1 a, the first segmentation mask PM1 b, or the heat map PM1 c).

[0022] Please also refer to Figure 5, which is a schematic diagram of an input image IMGi, a first indication mark PM1, and a second indication mark PM2 corresponding to a case where the task type of the indicator model 142 is an object detection type in one illustrative example.

[0023] In this example, the indicator model 142 is an object detection model trained with a limited number of images of cats and is used to mark an edge box containing the cat in the input image IMGi. As shown in Figure 5, in some embodiments, when the task type of the indicator model 142 is an object detection type, the indicator model 142 generates a first indicator mark PM1 that includes a first edge box PM1a, which corresponds to the approximate location of the object of interest (e.g., a cat).

[0024] As shown in FIG. 5, the instruction reinforcement program 144 is used to convert the first instruction mark PM1 (ie, the first edge box PM1a) into a second edge box PM2c by converting the first edge box PM1a.

[0025] In some embodiments, the transformation may be (1) moving the position of the first edge box PM1a to form the second edge box PM2c, (2) increasing the size of the first edge box PM1a to the second edge box PM2c, (3) decreasing the size of the first edge box PM1a to the second edge box PM2c, or (4) copying the first edge box PM1a to the second edge box PM2c.

[0026] 5, the size of the first edge box PM1a is reduced to form the second edge box PM2c of the second indication mark PM2. However, the present disclosure is not limited to this. The conversion method is determined based on the characteristics of the division base model 192.

[0027] 5, the instruction reinforcement program 144 converts the first instruction mark PM1 further by selecting a node within the second edge box PM2c as the position of the forward click PM2a. In some examples, the center point of the second edge box PM2c may be selected as the position of the forward click PM2a, and the second instruction mark PM2 may also include the forward click PM2a.

[0028] 5, the instruction reinforcement program 144 converts the first instruction mark PM1 further includes selecting one or more nodes outside the second edge box PM2c as the location of the negative click PM2b. In some examples, a random node away from the second edge box PM2c may be selected as the location of the negative click PM2b, and the second instruction mark PM2 may also include the negative click PM2b.

[0029] In some embodiments, the second instruction mark PM2 may include a combination of the positive click PM2a, the negative click PM2b, and the second edge box PM2c shown in Figure 5. The positive click PM2a, the negative click PM2b, and the second edge box PM2c can be combined and sent to the divided basic model 192 as an instruction input.

[0030] Please also refer to Figure 6, which is a schematic diagram of an input image IMGi, a first indication mark PM1, and a second indication mark PM2 corresponding to a case where the task type of the indicator model 142 is an image segmentation type in another illustrative example.

[0031] In this example, the indicator model 142 is an image segmentation model trained using a limited number of images of cats, and is used to mark a segmentation mask covering the area where the cat is located in the input image IMGi. As shown in Figure 6, in some embodiments, when the task type of the indicator model 142 is an image segmentation type, the first indicator mark PM1 generated by the indicator model 142 includes a first segmentation mask PM1b, which covers the approximate area of ​​the object of interest (e.g., a cat).

[0032] As shown in FIG. 6, the instruction reinforcement program 144 is used to convert the first instruction mark PM1 (ie, the first divided mask PM1b) into the second divided mask PM2f.

[0033] In some embodiments, the transformation may be (1) moving the position of the first segmented mask PM1b to form the second segmented mask PM2f, (2) enlarging the size of the first segmented mask PM1b to the second segmented mask PM2f, (3) reducing the size of the first segmented mask PM1b to the second segmented mask PM2f, or (4) copying the first segmented mask PM1b to the second segmented mask PM2f.

[0034] 6, the size of the first segmented mask PM1b is reduced to form the second segmented mask PM2f of the second indication mark PM2, but the present disclosure is not limited to this.

[0035] 6, the instruction reinforcement program 144 converts the first instruction mark PM1 further includes obtaining a second edge box PM2c based on the positions of the edge nodes of the first segmented mask PM1b. In some examples, the size and position of the second edge box PM2c may be determined based on the apex, bottom, rightmost point, and leftmost point of the first segmented mask PM1b, and the second instruction mark PM2 may also include the second edge box PM2c.

[0036] 6, the instruction reinforcement program 144 converts the first instruction mark PM1 further by selecting a node in the second divided mask PM2f as the position of the forward click PM2a. In some examples, the center point of the second divided mask PM2f may be selected as the position of the forward click PM2a, and the second instruction mark PM2 may also include the forward click PM2a.

[0037] 6, the instruction reinforcement program 144 converts the first instruction mark PM1 further includes selecting one or more nodes outside the second divided mask PM2f as the location of the negative click PM2b. In some examples, a random node away from the second divided mask PM2f may be selected as the location of the negative click PM2b, and the second instruction mark PM2 may also include the negative click PM2b.

[0038] In some embodiments, the second instruction mark PM2 may include a combination of the positive click PM2a, the negative click PM2b, the second edge box PM2c, and the second segmentation mask PM2f shown in Figure 6. The four elements, the positive click PM2a, the negative click PM2b, the second edge box PM2c, and the second segmentation mask PM2f, can be combined and sent together to the segmentation base model 192 as an instruction input.

[0039] Please also refer to Figure 7, which is a schematic diagram of an input image IMGi, a first indication mark PM1, and a second indication mark PM2 corresponding to a case where the task type of the indicator model 142 is a classification type in another illustrative example.

[0040] In this example, the indicator model 142 is a classification model trained using a limited number of images of cats and is used to generate a heat map PM1c. The numerical values ​​of multiple feature points in the heat map PM1c vary depending on the probability value of whether or not the feature point belongs to a region where an object of interest (e.g., a cat) is located. For example, if a feature point has a high probability of overlapping with a cat, it will be displayed in a dark color (e.g., a high numerical value) in the heat map, while if a feature point has a low probability of overlapping with a cat, it will be displayed in a light color (e.g., a low numerical value) in the heat map, as shown in FIG. 7.

[0041] 7, the instruction reinforcement program 144 performed on the first instruction mark PM1 (e.g., heat map PM1c) includes: (1) comparing the values ​​of multiple feature points in the heat map PM1c with a threshold; (2) locating multiple first feature points in the heat map PM1c that exceed the threshold and locating multiple second feature points in the heat map PM1c that are lower than the threshold; and (3) obtaining a second edge box PM2c based on multiple edge nodes of the first feature points, or obtaining a second segmented mask PM2f based on the first feature points. In this example, the second edge box PM2c and / or the second segmented mask PM2f can be combined and added to the combination of the second instruction mark PM2.

[0042] In the example shown in Figure 7, the inner part corresponding to high probability values ​​in the heat map PM1c (i.e., those first feature points exceeding the threshold) forms a second segmentation mask PM2f, and the edge nodes of the second segmentation mask PM2f form a second edge box PM2c.

[0043] 7, the instruction reinforcement program 144 converts the first instruction mark PM1 further includes selecting a first node from these first feature points (corresponding to an area within the second divided mask PM2f) that exceed the threshold as the location of the forward click PM2a. In this example, the center point of the second divided mask PM2f may be selected as the location of the forward click PM2a. The second instruction mark PM2 may also include the forward click PM2a.

[0044] 7, the instruction reinforcement program 144 converts the first instruction mark PM1 further includes selecting one or more second nodes from these second feature points (corresponding to areas other than the second divided mask PM2f) that are lower than the threshold as positions of the negative click PM2b. In some examples, a random node away from the second divided mask PM2f may be selected as the position of the negative click PM2b, and the second instruction mark PM2 may also include the negative click PM2b.

[0045] In some embodiments, the second instruction mark PM2 may include a combination of the positive click PM2a, the negative click PM2b, the second edge box PM2c, and the second division mask PM2f shown in Figure 7. The four elements, the positive click PM2a, the negative click PM2b, the second edge box PM2c, and the second division mask PM2f, can be integrated and sent together to the division base model 192 as an instruction input.

[0046] 3, the segmentation base model 192 includes an image encoder 192a, a pointer encoder 192b, and a mask decoder 192c. The image encoder 192a is used to receive an input image IMGi and convert the input image IMGi into an input image embedding vector IMGe. The pointer encoder 192b is used to receive a second pointer mark PM2 and convert the second pointer mark PM2 into a pointer vector PV. The mask decoder 192c is used to further generate an output segmentation mask SMK based on the input image embedding vector IMGe generated by the image encoder 192a and the pointer vector PV generated by the pointer encoder 192b.

[0047] In some embodiments, the split base model 192 may be implemented by a segment-anything model (SAM). However, the present disclosure is not limited in this respect. The split base model 192 may also be implemented by other similar base models based on converter architectures or other similar base models based on encoder-decoder architectures.

[0048] 8A and 8B are schematic diagrams of different output segmentation masks that segmentation base model 192 generates based on different indicators in some embodiments.

[0049] FIG. 8A is an example in which the segmentation base model 192 receives an input image IMGi and a first instruction mark PM1. In this example, the instruction input to the segmentation base model 192 is the first instruction mark PM1, which has not been processed by the instruction reinforcement program 144. As shown in FIG. 8A, the first instruction mark PM1 includes only one edge box, which marks the approximate location of the liver. However, the edge box also covers some areas around the liver. Based on the instruction provided by the first instruction mark PM1 (i.e., the edge box), the output segmentation mask SMK1 generated by the segmentation base model 192 shown in FIG. 8A may also cover other objects around the liver. In other words, the output segmentation mask SMK1 generated based on the first instruction mark PM1 is not accurate.

[0050] FIG. 8B shows another example in which the segmentation base model 192 receives an input image IMGi and a second instruction mark PM2. In this case, the instruction input to the segmentation base model 192 is the second instruction mark PM2 processed by the instruction reinforcement program 144. As shown in FIG. 8B, the second instruction mark PM2 includes a combination of one positive click, three negative clicks, and one edge box. This edge box marks the approximate location of the liver. The positive clicks mark the target to be segmented (i.e., the liver). These negative clicks mark parts that should not be included in the segmentation target. Based on the instructions provided by the second instruction mark PM2, the output segmentation mask SMK2 generated by the segmentation base model 192 can more accurately segment the target (i.e., the liver) in this example, as shown in FIG. 8B.

[0051] As described in the above embodiments, the image processing system 100 can perform image segmentation without training a large-scale model using a large amount of label data. The image processing system 100 can train a small-scale indicator model 142, generate a second indicator mark PM2 using the indicator model 142 and the indicator reinforcement program 144, and send it to the segmentation base model 192 to generate an output segmentation mask SMK. The segmentation base model 192 can also be an externally operated pre-training model. Typically, the segmentation base model 192 has excellent versatility (i.e., the ability to process different images), and the image processing system 100 operating in conjunction with the segmentation base model 192 can also process different types of images (e.g., medical images, traffic images, face identification images, fingerprint images).

[0052] In the framework of the present disclosure, a small model with low computational complexity can be selected as the indicator model 142, simplifying the task difficulty of data marking (used to train the indicator model 142). In this example, the indicator model 142 can be trained with a limited number of label data and simplified label types. When a new task target (e.g., image segmentation related to dogs) needs to be added, the image processing system 100 can train the indicator model 142 using a limited number of label data related to dogs, without having to retrain a large-scale model using training data related to dogs, thereby contributing to reducing the marking costs (e.g., time costs and labor costs) of training the indicator model 142. In addition, the segmentation base model 192 can generate a segmentation result (e.g., an output segmentation mask SMK) based on an appropriate instruction input (e.g., the second instruction mark PM2), thereby maintaining the accuracy of the output segmentation mask SMK.

[0053] Although specific embodiments of the present disclosure have been disclosed in the above examples, these embodiments are not intended to limit the present disclosure. Various substitutions and improvements can be made in the present disclosure by those skilled in the art without departing from the principle and spirit of the present disclosure. Therefore, the scope of protection of the present disclosure is determined by the scope of the attached patent application. [Explanation of symbols]

[0054] 100: Image processing system 120: Memory unit 140: Processing unit 142: Indicator model 144: Instruction Reinforcement Program 146:Result generation device 160: Interface 180: Communication circuit 190:Server 192: Split Basic Model 192a: Image encoder 192b: instruction encoder 192c: Mask decoder 200: Image division method 400: Training Methods IMGe: Input image embedding vector IMGi: Input image IMGo: Output image PM1: First direction mark PM1a: Edge box PM1b: 1st division mask PM1c: Heatmap PM2: Second direction mark PM2a: Forward click PM2b: Negative click PM2c: Second edge box PM2d: Positive trajectory PM2e: Negative trajectory PM2f: Second division mask PV: Pointing Vector S210, S220, S230, S240: Step S410, S420, S430, S440, S450, S460: Step SMK, SMK1, SMK2: Output division mask

Claims

1. generating a first indicator mark based on a task type of the indicator model by providing an input image to the indicator model, the task type of the indicator model including an object detection type, an image segmentation type, or a classification type, and the first indicator mark including an edge box, a first segmentation mask, or a heat map; a step of generating the second indication mark by executing an indication reinforcement program, the step including converting a position, a size, or an indication type of the first indication mark by referring to the task type of the indicator model, to form the second indication mark, wherein the second indication mark includes a positive click, a negative click, a second edge box, a positive trajectory, a negative trajectory, or a second segmentation mask; providing the input image and the second indication marks to a segmentation base model, and generating an output segmentation mask in the input image based on the second indication marks by the segmentation base model.

2. the first indication mark generated by the indicator model in response to the task type of the indicator model being the object detection type is a first edge box; the instruction reinforcement program performed on the first instruction mark includes: moving a position of the first edge box to form the second edge box; enlarging a size of the first edge box to the second edge box; or reducing the size of the first edge box to the second edge box; The image segmentation method according to claim 1 , wherein the second indication mark includes the second edge box.

3. In response to the task type of the indicator model being the object detection type, the instruction reinforcement program for the first instruction mark further includes selecting a first node within the second edge box or selecting a second node outside the second edge box; The image division method according to claim 2 , wherein the second direction mark further includes the positive click located at the first node or the negative click located at the second node.

4. the first indication mark generated by the indicator model in response to the task type of the indicator model being the image segmentation type is the first segmentation mask; The instruction reinforcement program performed on the first instruction mark includes: moving a position of the first divided mask to form the second divided mask; enlarging a size of the first divided mask to the second divided mask; reducing the size of the first divided mask to the second divided mask; or obtaining the second edge box based on a plurality of edge nodes of the first divided mask; The image segmentation method according to claim 1 , wherein the second indication mark includes the second segmentation mask or the second edge box.

5. In response to the task type of the indicator model being the image segmentation type, the instruction reinforcement program for the first instruction mark further includes selecting a first node within the second segmentation mask or selecting a second node outside the second segmentation mask; The image dividing method according to claim 4 , wherein the second indication mark further includes the positive click located at the first node or the negative click located at the second node.

6. the first indicator mark generated by the indicator model in response to the task type of the indicator model being the classification type is the heat map; The instruction reinforcement program to be performed for the first instruction mark is comparing values ​​of a plurality of feature points in the heat map with a threshold; locating a plurality of first feature points in the heat map that are above the threshold and locating a plurality of second feature points in the heat map that are below the threshold; 2. The image segmentation method of claim 1, further comprising: obtaining an edge box based on a plurality of edge nodes of the plurality of first feature points; or obtaining the second segmentation mask based on the plurality of first feature points; and the second indication mark includes the second edge box or the second segmentation mask.

7. In response to the task type of the indicator model being the classification type, the instruction reinforcement program performed on the first instruction mark further includes selecting a first node from the plurality of first feature points or selecting a second node from the plurality of second feature points; The image dividing method according to claim 6 , wherein the second indication mark further includes the positive click located at the first node or the negative click located at the second node.

8. The image segmentation method of claim 1 , further comprising generating an output image based on the segmentation mask and the input image.

9. The divided basic model is an image encoder for receiving the input image and converting the input image into an input image embedding vector; a pointer encoder for receiving the second pointer marks and converting the second pointer marks into pointer vectors; a mask decoder for generating the output segmentation mask based on the input image embedding vector and the instruction vector.

10. a storage unit for storing a plurality of computer-executable instructions; a processing unit, coupled to the memory unit, for executing the plurality of computer-executable instructions; providing an input image to an indicator model to generate a first indicator mark based on a task type of the indicator model, wherein the task type of the indicator model includes an object detection type, an image segmentation type, or a classification type, and the first indicator mark includes an edge box, a first segmentation mask, or a heat map; generating the second indication mark by executing an indication reinforcement program including: converting a position, a size, or an indication type of the first indication mark by referring to the task type of the indicator model, and forming a second indication mark, wherein the second indication mark includes a positive click, a negative click, a second edge box, a positive trajectory, a negative trajectory, or a second division mask; a processing unit used for providing the input image and the second indication mark to a segmentation base model, and generating an output segmentation mask in the input image based on the second indication mark by the segmentation base model; an image processing system comprising:

Images