Image segmentation method and image processing system

Abstract

To provide an image segmentation method.SOLUTION: An image segmentation method is provided, comprising: providing an input image to a prompter model to generate a first prompt indicator according to a task type of the prompter model; performing a prompt enhancement procedure with reference to the task type of the prompter mode on the first prompt indicator to generate a second prompt indicator, the prompt enhancement procedure including converting the location, size or prompt type of the first prompt indicator into the second prompt indicator with reference to the task type; and providing the input image and the second prompt indicator to a segmentation foundation model to generate an output segmentation mask on the input image according to the second prompt indicator.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to an image segmentation method and an image processing system, and more particularly to an image segmentation method and an image processing system integrated with a segmentation basis model. [Background technology]

[0002] In recent years, artificial intelligence has developed rapidly, and various application fields such as industry, commerce, and medicine have already begun to widely adopt artificial intelligence technology to support their operations. However, many enterprises lack the corresponding capabilities to build artificial intelligence models that meet their needs. Therefore, many enterprises tend to use automated machine learning platforms or seek cooperation to build artificial intelligence models. Automated machine learning platforms enable enterprise users with basic knowledge of artificial intelligence to build enterprise-specific artificial intelligence models through automated training methods. These platforms provide users with a variety of model selections, so that they can select models with corresponding capabilities according to their respective needs and train the selected models with the collected datasets. Summary of the Invention [Problem to be solved by the invention]

[0003] Although automated machine learning platforms provide multiple models for users to choose from, the key part required for training, i.e., training data with accurate labels, must be provided by the user. How to establish a large amount of accurately labeled training data places a heavy burden on the user, as it requires setting marking rules, preparing unmarked datasets, and having experts (e.g., production line engineers, quality control personnel, radiologists) mark and generate label data based on predefined rules. If the task involves image segmentation, it takes even more time. Image segmentation tasks require marking pixel by pixel, which takes a large amount of time even for experienced markers. In general, training a machine learning model requires a large amount of labeled training data, a certain amount of time, and computing equipment required to perform the training.

[0004] From the above, the conventional image segmentation model training process includes the steps of collecting data, performing data marking, selecting an appropriate model, loading a label dataset, training the model, and obtaining the trained model. For example, the model can be trained based on image data whose label includes a dog, so that the dog's position can be identified and located in the input image. If the trained model needs to be improved (e.g., adding a segmentation function for cats), at least some training steps need to be repeated to retrain the model, and these retraining steps will consume a lot of time, effort, and computation resources. [Means for solving the problem]

[0005] According to one aspect of the present disclosure, an image segmentation method is disclosed that includes the steps of: providing an input image to an indicator model to generate a first indication mark based on a task type of the indicator model; executing an indication enhancement program to generate the second indication mark for the first indication mark by referring to the task type of the indicator model, and converting a position, size or indication type of the first indication mark by referring to the task type to form a second indication mark; and providing the input image and the second indication mark to a division base model, and generating an output division mask in the input image based on the second indication mark by the division base model.

[0006] According to another aspect of the present disclosure, an image processing system is disclosed, comprising: a memory unit for storing a plurality of computer-executable instructions; and a processing unit coupled to the storage unit and used to execute the plurality of computer-executable instructions, the processing unit being used to: provide an input image to an indicator model to generate a first indicator mark based on a task type of the indicator model; generate the second indicator mark by executing an indication enhancement program including the indication enhancement program converting a position, size or indication type of the first indicator mark by referring to the task type of the indicator model to form a second indicator mark; provide the input image and the second indication mark to a split base model; and generate an output split mask in the input image by the split base model based on the second indication mark.

[0007] It should be noted that the above description and the following detailed description are used to exemplarily explain the present application in the form of embodiments and to assist in the interpretation and understanding of the inventive contents required by the present application. [Brief description of the drawings]

[0008] To make the above and other objects, features and embodiments of the present disclosure more clearly and comprehensibly, the accompanying drawings are described as follows: [Figure 1] FIG. 1 is a schematic diagram of an image processing system according to some embodiments of the present disclosure. [Diagram 2] 1 is a flowchart of an image segmentation method in accordance with some embodiments of the present disclosure. [Diagram 3] 3A-3C are schematic diagrams of a first indication mark, a second indication mark, an output segmentation mask and an output image generated based on an input image during an image segmentation method in some embodiments. [Figure 4] 1 is a method flow diagram of a method for training an indicator model according to some embodiments of the present disclosure. [Diagram 5] 1 is a schematic diagram of an input image, a first indicator mark, and a second indicator mark corresponding to a case where a task type of an indicator model is an object detection type in one illustrative example; [Figure 6] 13 is a schematic diagram of an input image, a first indication mark, and a second indication mark corresponding to a case where the task type of the indicator model is an image division type in another illustrative example. FIG. [Figure 7] 11 is a schematic diagram of an input image, a first indicator mark, and a second indicator mark corresponding to a case where the task type of the indicator model is a classification type in another illustrative example. FIG. [Figure 8A] 11A-11C are schematic diagrams of different output segmentation masks that a segmentation base model generates based on different indication marks in some embodiments. [Figure 8B] 11A-11C are schematic diagrams of different output segmentation masks that a segmentation base model generates based on different indication marks in some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] The following disclosure provides many different embodiments or illustrations for implementing different features of the present disclosure. Elements and configurations in specific illustrations are used to simplify the present disclosure in the following discussion. Any illustrations discussed are used for illustrative purposes only and do not limit the scope and meaning of the present disclosure or its illustrations in any manner. Where appropriate, the same reference numerals are used between the drawings and in the corresponding text description to represent the same or similar elements.

[0010] Please refer to FIG. 1, which is a schematic diagram of an image processing system 100 according to some embodiments of the present disclosure. In some embodiments, the image processing system 100 can perform a specific image segmentation task on an input image IMGi. For example, the image processing system 100 can be used to mark potential tumor locations in a magnetic resonance imaging (MRI) scan image, identify license plates in a traffic monitoring image, mark a cat in a grass background, or separate foreground people from background scenery. In some embodiments, the image processing system 100 is used to work in conjunction with a segmentation basis model (SFM) 192 to perform the above image segmentation tasks.

[0011] The so-called segmentation base model 192 is a comprehensive deep learning framework for performing various image segmentation tasks. The segmentation base model 192 serves as a general-purpose tool in computer vision and can be used to segment an image into meaningful parts or localities. The segmentation base model 192 is generally based on advanced neural network architectures, such as converters, convolutional neural networks (CNNs) or fully convolutional networks (FCNs). The segmentation base model 192 is trained on large and diverse types of datasets so that it can be widely applied to various different types of images and various segmentation tasks. In some embodiments, the segmentation base model 192 can run on an external server 190 outside the image processing system 100.

[0012] In some embodiments, the segmentation basic model 192 may be realized by a segment anything model (SAM). The segment anything model can complete image segmentation tasks based on instruction word instructions. The segment anything model supports multiple types of instruction marks (e.g. positive click, negative click, edge box or text description). However, the segment anything model has some limitations when used for customized object segmentation tasks. First, the segment anything model needs accurate and appropriate instruction marks to clearly indicate the intention of the target segmentation task. Second, the segment anything model needs additional specific training (on the target segmentation task) to achieve optimal segmentation effect.

[0013] The image processing system 100 in the present disclosure provides a solution that can generate appropriate indication marks based on the input image IMGi and provide the appropriate indication marks to the segmentation base model 192. As shown in FIG. 1, in some embodiments, the image processing system 100 includes a storage unit 120, a processing unit 140, an interface 160, and a communication circuit 180. The processing unit 140 is coupled to the storage unit 120, the interface 160, and the communication circuit 180. In some embodiments, the storage unit 120 may be realized by a memory, a fast access memory, or a hard disk. The processing unit 140 may be realized by a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), or an application specific integrated circuit (ASIC), the interface 160 may include an input end (e.g., a connector, an image scanner, or a camera) and an output end (e.g., a display, a connector, or a projector), and the communication circuit 180 may be realized by a network adapter, a telecommunication circuit, or a wireless communication circuit.

[0014] The storage unit 120 is used to store computer-executable instructions. These computer-executable instructions may be executed by the processing unit 140 to perform an image segmentation method. Please refer to Figures 2 and 3 together. Figure 2 is a flowchart of an image segmentation method 200 in some embodiments of the present disclosure, and Figure 3 is a schematic diagram of a first indication mark PM1, a second indication mark PM2, an output segmentation mask SMK, and an output image IMGo generated based on an input image IMGi during execution of the image segmentation method 200 in some embodiments.

[0015] Assume that the new task goal is to establish a new image segmentation model that can segment the cat and the background. To achieve this new task goal, the image processing system 100 utilizes the indicator model 142 and the indicator reinforcement program 144 to generate appropriate indicators (e.g., positive clicks for the cat, negative clicks for the background, an edge box surrounding the cat, or a coarse segmentation mask for the cat) required for the operation of the segmentation base model 192. In this example, the segmentation base model 192 can generate an accurate segmentation mask for the cat based on the appropriate indicators generated by the indicator model 142 and the indicator reinforcement program 144. More details of the indicator model 142 and the indicator reinforcement program 144 are described in the following paragraphs.

[0016] As shown in FIG. 1, FIG. 2 and FIG. 3, the image processing system 100 receives an input image IMGi through the interface 160. In step S210, the processing unit 140 provides the input image IMGi to the indicator model 142 based on the task type of the indicator model 142 to generate a first indication mark PM1 (e.g., an edge box PM1a, a first segmented mask PM1b or a heat map PM1c). In step S220, the processing unit 140 executes the indication reinforcement program 144 on the first indication mark PM1 based on the task type of the indicator model 142 to generate a second indication mark PM2 (e.g., a positive click PM2a, a negative click PM2b, a second edge box PM2c, a positive trajectory PM2d, a negative trajectory PM2e or a second segmented mask PM2f). In step S230, the image processing system 100 provides the input image IMGi and the second indication mark PM2 to the divided basic model 192 via the communication circuit 180, so that the divided basic model 192 can generate an output divided mask SMK in the input image IMGi based on the second indication mark PM2.

[0017] In cooperation with the segmentation base model 192, the image processing system 100 can obtain an output segmentation mask SMK. As shown in Figures 1, 2 and 3, in step S240, the result generator 146 executed by the processing unit 140 can generate an output image IMGo based on the output segmentation mask SMK and the input image IMGi.

[0018] In some embodiments, the indicator model 142, the indicator reinforcement program 144, and the result generator 146 may be realized by software instructions executed by the processing unit 140. The task type of the indicator model 142 may be selected from a number of candidate task types, such as an object detection type, an image segmentation type, and a classification type. The indicator model 142 is trained based on a training dataset and can further complete one of object detection, image segmentation, or classification tasks. The training dataset includes a number of training images and label data corresponding to the number of training images.

[0019] Please also refer to FIG. 4. FIG. 4 is a method flow chart of a training method 400 of the indicator model 142 according to some embodiments of the present disclosure. As shown in FIG. 4, the training method 400 performs step S410 to collect a plurality of training images (e.g., images related to cats). Perform step S420 to mark the training images to generate label data. Perform step S430 to select a task type of the indicator model 142. For example, the task type of the indicator model 142 may select one of an object detection type (e.g., the indicator model 142 may use a YOLO or Faster R-CNN model), an image segmentation type (e.g., the indicator model 142 may use a Unet or Deeplab model), and a classification type (e.g., the indicator model 142 may use a VGGNet or ResNet model). Perform step S440 to load a dataset (including a plurality of training images and corresponding label data) and prepare it for training. Perform step S450 to train the indicator model 142. In some embodiments, step S450 includes the indicator model 142 generating a prediction result of the training image, comparing the prediction result with the label data, and adjusting parameters of the indicator model 142 based on the comparison result. Step S460 is performed to obtain the trained indicator model 142.

[0020] In some embodiments, compared with the segmentation base model 192, the indicator model 142 is trained using a small and specific type of dataset. This is because the indicator model 142 does not need to generate an accurate image segmentation mask position, and the output of the indicator model 142 is only used to generate an indication mark (as an indication / cue of the segmentation intention of the segmentation base model 192), and the heavy task of generating an accurate image segmentation mask position is performed by the segmentation base model 192 based on the received indication mark. Note that the indicator model 142 can be specialized to realize a specific purpose (e.g., cat search), so it does not need to process various types of input images. Therefore, the indicator model 142 can be realized with limited computational resources based on a simple model, and can complete training more quickly based on a small and specific type of dataset.

[0021] As illustrated in FIG. 4, the indicator model 142 can be selected from various models corresponding to a plurality of different task types, including an object detection type, an image segmentation type, or a classification type. In some embodiments, the indicator model 142 is selected from various different task types, so that the first indicator mark PM1 generated by the indicator model 142 has a further different format with reference to the task type of the indicator model 142. As illustrated in FIG. 3, the first indicator mark PM1 generated by the indicator model 142 may be one of an edge box PM1a, a first segmentation mask PM1b, or a heat map PM1c. The indication enhancement program 144 for processing the first indicator mark PM1 is different according to the various types of the first indicator mark PM1 (i.e., the edge box PM1a, the first segmentation mask PM1b, or the heat map PM1c).

[0022] Please also refer to Fig. 5, which is a schematic diagram of an input image IMGi, a first indication mark PM1, and a second indication mark PM2 corresponding to a case where the task type of the indicator model 142 is an object detection type in one illustrative example.

[0023] In this example, the indicator model 142 is an object detection model trained with a limited number of images of cats and is used to mark an edge box containing a cat in the input image IMGi. As shown in Fig. 5, in some embodiments, when the task type of the indicator model 142 is an object detection type, the indicator model 142 generates a first indicator mark PM1 including a first edge box PM1a, which corresponds to the approximate location of the object of interest (e.g., a cat).

[0024] As shown in FIG. 5, the instruction reinforcement program 144 is used to convert the first instruction mark PM1 (ie, the first edge box PM1a) into a second edge box PM2c by converting the first edge box PM1a.

[0025] In some embodiments, the transformation may be (1) moving the position of the first edge box PM1a to form a second edge box PM2c, (2) expanding the size of the first edge box PM1a to the second edge box PM2c, (3) reducing the size of the first edge box PM1a to the second edge box PM2c, or (4) copying the first edge box PM1a to the second edge box PM2c.

[0026] In the embodiment of Fig. 5, the size of the first edge box PM1a is reduced to form the second edge box PM2c of the second indication mark PM2. However, the present disclosure is not limited thereto. The conversion method is determined based on the characteristics of the division base model 192.

[0027] 5, the instruction reinforcement program 144 converts the first instruction mark PM1 further includes selecting one node in the second edge box PM2c as the position of the positive click PM2a. In some examples, the center point of the second edge box PM2c may be selected as the position of the positive click PM2a, and the second instruction mark PM2 may also include the positive click PM2a.

[0028] 5, the instruction reinforcement program 144 converts the first instruction mark PM1 further includes selecting one or more nodes outside the second edge box PM2c as the location of the negative click PM2b. In some examples, a random node away from the second edge box PM2c may be selected as the location of the negative click PM2b, and the second instruction mark PM2 may also include the negative click PM2b.

[0029] In some embodiments, the second instruction mark PM2 may include a combination of the positive click PM2a, the negative click PM2b, and the second edge box PM2c shown in Fig. 5. The positive click PM2a, the negative click PM2b, and the second edge box PM2c can be integrated and sent to the divided basic model 192 as an instruction input.

[0030] Please also refer to Fig. 6, which is a schematic diagram of an input image IMGi, a first instruction mark PM1, and a second instruction mark PM2 corresponding to a case where the task type of the indicator model 142 is an image division type in another illustrative example.

[0031] In this example, the indicator model 142 is an image segmentation model trained with a limited number of images of cats, and is used to mark a segmentation mask covering the area where the cat is located in the input image IMGi. As shown in Fig. 6, in some embodiments, when the task type of the indicator model 142 is an image segmentation type, the first indicator mark PM1 generated by the indicator model 142 includes a first segmentation mask PM1b, which covers the approximate area of ​​the object of interest (e.g., a cat).

[0032] As shown in FIG. 6, the indication reinforcement program 144 is used to convert the first indication mark PM1 (ie, the first divided mask PM1b) into the second divided mask PM2f.

[0033] In some embodiments, the transformation may be (1) moving the position of the first split mask PM1b to form the second split mask PM2f, (2) enlarging the size of the first split mask PM1b to the second split mask PM2f, (3) reducing the size of the first split mask PM1b to the second split mask PM2f, or (4) copying the first split mask PM1b to the second split mask PM2f.

[0034] 6, the size of the first divided mask PM1b is reduced to form the second divided mask PM2f of the second indication mark PM2. However, the present disclosure is not limited to this.

[0035] 6, the instruction reinforcement program 144 converts the first instruction mark PM1 further includes obtaining a second edge box PM2c based on the positions of the edge nodes of the first divided mask PM1b. In some examples, the size and position of the second edge box PM2c may be determined based on the apex, bottom point, rightmost point, and leftmost point of the first divided mask PM1b, and the second instruction mark PM2 may also include the second edge box PM2c.

[0036] 6, the instruction reinforcement program 144 converts the first instruction mark PM1 further includes selecting one node in the second divided mask PM2f as the position of the positive click PM2a. In some examples, the center point of the second divided mask PM2f may be selected as the position of the positive click PM2a, and the second instruction mark PM2 may also include the positive click PM2a.

[0037] 6, the instruction reinforcement program 144 converts the first instruction mark PM1 further includes selecting one or more nodes outside the second divided mask PM2f as the location of the negative click PM2b. In some examples, a random node away from the second divided mask PM2f may be selected as the location of the negative click PM2b, and the second instruction mark PM2 may also include the negative click PM2b.

[0038] In some embodiments, the second instruction mark PM2 may include a combination of the positive click PM2a, the negative click PM2b, the second edge box PM2c, and the second division mask PM2f shown in Fig. 6. The positive click PM2a, the negative click PM2b, the second edge box PM2c, and the second division mask PM2f can be integrated and sent together to the division base model 192 as an instruction input.

[0039] Please also refer to Fig. 7, which is a schematic diagram of an input image IMGi, a first instruction mark PM1, and a second instruction mark PM2 corresponding to a case where the task type of the indicator model 142 is a classification type in another illustrative example.

[0040] In this example, the indicator model 142 is a classification model trained with a limited number of images of cats and is used to generate a heat map PM1c. The values ​​of the feature points in the heat map PM1c vary according to the probability value of whether the feature point belongs to the region where the object of interest (e.g., cat) is located. For example, if the feature point overlaps with the cat with a high probability, it is a dark color (e.g., a high value) in the heat map, whereas if the feature point overlaps with the cat with a low probability, it is a light color (e.g., a low value) in the heat map, as shown in FIG. 7.

[0041] 7, the instruction reinforcement program 144 performed on the first instruction mark PM1 (e.g., heat map PM1c) includes (1) comparing the values ​​of a plurality of feature points in the heat map PM1c with a threshold value, (2) locating a plurality of first feature points in the heat map PM1c that are greater than the threshold value and locating a plurality of second feature points in the heat map PM1c that are less than the threshold value, and (3) obtaining a second edge box PM2c based on a plurality of edge nodes of the first feature points, or obtaining a second segmented mask PM2f based on the first feature points. In this example, the second edge box PM2c and / or the second segmented mask PM2f can be integrated and added to the combination of the second instruction mark PM2.

[0042] In the example shown in Fig. 7, the inner parts in the heat map PM1c that correspond to high probability values ​​(i.e., those first feature points that exceed the threshold) form a second segmentation mask PM2f. The edge nodes of the second segmentation mask PM2f form a second edge box PM2c.

[0043] 7, the instruction reinforcement program 144 converts the first instruction mark PM1 further includes selecting a first node from those first feature points (corresponding to an area within the second divided mask PM2f) that exceed a threshold as the location of the forward click PM2a. In this example, the center point of the second divided mask PM2f may be selected as the location of the forward click PM2a. The second instruction mark PM2 may also include the forward click PM2a.

[0044] 7, the instruction reinforcement program 144 converts the first instruction mark PM1 further includes selecting one or more second nodes from these second feature points (corresponding to areas other than the second divided mask PM2f) that are lower than a threshold as the location of the negative click PM2b. In some examples, a random node away from the second divided mask PM2f may be selected as the location of the negative click PM2b, and the second instruction mark PM2 may also include the negative click PM2b.

[0045] In some embodiments, the second instruction mark PM2 may include a combination of the positive click PM2a, the negative click PM2b, the second edge box PM2c, and the second division mask PM2f shown in Fig. 7. The four, the positive click PM2a, the negative click PM2b, the second edge box PM2c, and the second division mask PM2f, can be integrated and sent together to the division base model 192 as an instruction input.

[0046] As shown in Fig. 3, the split basic model 192 includes an image encoder 192a, a pointer encoder 192b, and a mask decoder 192c. The image encoder 192a is used to receive an input image IMGi and convert the input image IMGi into an input image embedding vector IMGe. The pointer encoder 192b is used to receive a second pointer mark PM2 and convert the second pointer mark PM2 into a pointer vector PV. The mask decoder 192c is further used to generate an output split mask SMK based on the input image embedding vector IMGe generated by the image encoder 192a and the pointer vector PV generated by the pointer encoder 192b.

[0047] In some embodiments, the split base model 192 may be implemented by a segment anything model (SAM). However, the present disclosure is not limited in this respect. The split base model 192 may be implemented by other similar base models based on converter architectures or other similar base models based on encoder-decoder architectures.

[0048] Please refer to Figures 8A and 8B together, which are schematic diagrams of different output segmentation masks that the segmentation base model 192 generates based on different indication marks in some embodiments.

[0049] FIG. 8A is an example in which the split base model 192 receives an input image IMGi and a first instruction mark PM1. In this example, the instruction input to the split base model 192 is the first instruction mark PM1 that has not been processed by the instruction reinforcement program 144. As shown in FIG. 8A, the first instruction mark PM1 includes only one edge box, which marks the approximate location of the liver. However, the edge box also covers some areas around the liver. Based on the instruction provided by the first instruction mark PM1 (i.e., the edge box), the output split mask SMK1 generated by the split base model 192 shown in FIG. 8A may also cover other objects around the liver. In other words, the output split mask SMK1 generated based on the first instruction mark PM1 is not accurate.

[0050] FIG. 8B is another example in which the segmentation base model 192 receives the input image IMGi and the second instruction mark PM2. In this case, the instruction input to the segmentation base model 192 is the second instruction mark PM2 processed by the instruction reinforcement program 144. As shown in FIG. 8B, the second instruction mark PM2 includes a combination of one positive click, three negative clicks and one edge box. This edge box marks the approximate location of the liver. The positive clicks mark the target to be segmented (i.e., the liver). These negative clicks mark parts that should not be included in the segmentation target. Based on the instruction provided by the second instruction mark PM2, the output segmentation mask SMK2 generated by the segmentation base model 192 can more accurately segment the target (i.e., the liver) in this example, as shown in FIG. 8B.

[0051] As described in the above embodiment, the image processing system 100 can perform image segmentation without training a large-scale model using a large amount of label data. The image processing system 100 can train a small-scale indicator model 142, and use the indicator model 142 and the indicator reinforcement program 144 to generate a second indicator mark PM2, and send it to the segmentation basic model 192 to generate an output segmentation mask SMK. The segmentation basic model 192 may be a pre-training model that operates externally. Typically, the segmentation basic model 192 has good versatility (i.e., the ability to process different images), and the image processing system 100 that operates in conjunction with the segmentation basic model 192 can also process different types of images (e.g., medical images, traffic images, face identification images, fingerprint images).

[0052] In the framework of the present disclosure, a small model with low computational complexity can be selected as the indicator model 142, and the task difficulty of data marking (used to train the indicator model 142) can be simplified. In this example, the indicator model 142 can be trained with a limited number of label data and a simplified label type. When a new task target (e.g., image segmentation related to dogs) needs to be added, the image processing system 100 can train the indicator model 142 with a limited number of label data related to dogs without the need to retrain a large-scale model with training data related to dogs, which contributes to reducing the marking cost (e.g., time cost and labor cost) of training the indicator model 142. In addition, the segmentation base model 192 can generate a segmentation result (e.g., output segmentation mask SMK) based on an appropriate instruction input (e.g., second instruction mark PM2), thereby maintaining the accuracy of the output segmentation mask SMK.

[0053] Although specific embodiments of the present disclosure are disclosed in the above embodiments, these embodiments are not intended to limit the present disclosure. Various substitutions and improvements can be made in the present disclosure by those skilled in the art without departing from the principle and spirit of the present disclosure. Therefore, the protection scope of the present disclosure is determined by the scope of the attached patent application. [Explanation of symbols]

[0054] 100: Image processing system 120: Storage unit 140: Processing unit 142: Indicator model 144: Instruction Reinforcement Program 146:Result generation device 160: Interface 180: Communication circuit 190:Server 192: Split Basic Model 192a: Image Encoder 192b: instruction encoder 192c: Mask Decoder 200: Image division method 400: Training Methods IMGe: Input image embedding vector IMGi: Input image IMGo: Output image PM1: First instruction mark PM1a: Edge box PM1b: 1st division mask PM1c: Heatmap PM2: Second Indicator Mark PM2a: Forward click PM2b: Negative click PM2c: Second edge box PM2d: Positive trajectory PM2e: Negative trajectory PM2f: 2nd division mask PV: Pointing Vector S210, S220, S230, S240: Step S410, S420, S430, S440, S450, S460: Step SMK, SMK1, SMK2: Output division mask

Claims

1. providing an input image to an indicator model to generate a first indicator mark based on a task type of the indicator model; generating the second indication mark by executing an indication reinforcement program including converting a position, a size, or an indication type of the first indication mark by referring to the task type of the indicator model, and forming the second indication mark; providing the input image and the second indication mark to a segmentation base model, and generating an output segmentation mask in the input image based on the second indication mark by the segmentation base model.

2. the first indicator mark generated by the indicator model in response to the task type of the indicator model being an object detection type is a first edge box; The instruction reinforcement program performed on the first instruction mark includes moving a position of the first edge box to form a second edge box, enlarging a size of the first edge box to the second edge box, or reducing the size of the first edge box to the second edge box, The image segmentation method according to claim 1 , wherein the second indication mark includes the second edge box.

3. In response to the task type of the indicator model being the object detection type, the indication reinforcement program for the first indication mark further includes selecting a first node within the second edge box or selecting a second node outside the second edge box; The image division method according to claim 2 , wherein the second indication mark further includes a positive click located at the first node or a negative click located at the second node.

4. the first indicator mark generated by the indicator model in response to the task type of the indicator model being an image segmentation type is a first segmentation mask; The instruction reinforcement program performed on the first instruction mark includes: moving a position of the first divided mask to form a second divided mask; enlarging a size of the first divided mask to the second divided mask; reducing the size of the first divided mask to the second divided mask; or obtaining an edge box based on a plurality of edge nodes of the first divided mask; The image segmentation method according to claim 1 , wherein the second indication mark includes the second segmentation mask or the edge box.

5. In response to the task type of the indicator model being the image segmentation type, the instruction reinforcement program for the first indication mark further includes selecting a first node within the second segmentation mask or selecting a second node outside the second segmentation mask; The image division method according to claim 4 , wherein the second indication mark further includes a positive click located at the first node or a negative click located at the second node.

6. the first indicator mark generated by the indicator model in response to the task type of the indicator model being a classification type is a heat map; The instruction reinforcement program to be performed for the first instruction mark is comparing values ​​of a plurality of feature points in the heat map with a threshold; locating a plurality of first feature points in the heat map that are above the threshold and locating a plurality of second feature points in the heat map that are below the threshold; 2. The image segmentation method of claim 1, further comprising: obtaining an edge box based on a plurality of edge nodes of the plurality of first feature points; or obtaining a segmentation mask based on the plurality of first feature points; and the second indication mark includes the edge box or the segmentation mask.

7. In response to the task type of the indicator model being the classification type, the instruction reinforcement program for the first instruction mark further includes selecting a first node from the plurality of first feature points or selecting a second node from the plurality of second feature points; The image division method according to claim 6 , wherein the second indication mark further includes a positive click located at the first node or a negative click located at the second node.

8. The image segmentation method of claim 1 , further comprising: generating an output image based on the output segmentation mask and the input image.

9. The division basic model is an image encoder for receiving the input image and converting the input image into an input image embedding vector; a direction encoder for receiving the second direction marks and converting the second direction marks into direction vectors; and a mask decoder for generating the output segmentation mask based on the input image embedding vector and the instruction vector.

10. a storage unit for storing a plurality of computer executable instructions; a processing unit, coupled to the memory unit, configured to execute the plurality of computer-executable instructions; providing an input image to an indicator model to generate a first indicator mark based on a task type of the indicator model; generating the second indication mark by executing an indication reinforcement program including: converting a position, a size, or an indication type of the first indication mark by referring to the task type of the indicator model, and forming the second indication mark; a processing unit adapted to provide the input image and the second indication mark to a segmentation base model and generate an output segmentation mask in the input image based on the second indication mark by the segmentation base model; 2. An image processing system comprising:

Images