Image prompt propagation method

By employing an image-based cue propagation method, utilizing the SAM model and augmented point correspondence algorithm to select appropriate cue points, the problem of low efficiency in generating high-quality results in unknown scenes using basic computer vision models is solved, enabling fast and automated image understanding and multi-task extension.

CN119048804AActive Publication Date: 2024-11-29NANJING AGRICULTURAL UNIVERSITY
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411046062.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-01
Publication Date
2024-11-29
Estimated Expiration
2044-08-01

AI Technical Summary

Technical Problem

Existing computer vision models require extensive manual annotation to generate high-quality results, resulting in low efficiency and limited performance improvement in unknown scenarios.

Method used

By employing an image-based cue propagation method, an anchor point set is initialized using the SAM model. An enhanced point-to-image mapping method is used to transfer anchor points to images in the sequence. A cue selector is designed to select appropriate cue points from the anchor point set. The alpha-shape algorithm is then used to evaluate the cue results to ensure their accuracy.

Benefits of technology

It automates the process of quickly understanding images in similar scenarios, reduces the cost of manual annotation, improves performance to adapt to different scenarios, and extends to multiple computer vision tasks such as image segmentation and video tracking.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119048804A_ABST
    Figure CN119048804A_ABST
Patent Text Reader

Abstract

The invention discloses an image prompt propagation method, which comprises the following steps of: 1, initializing an anchor point set according to a prompt provided by a user and a template picture; 2, transmitting the anchor points from the template to pictures except the template in the sequence by using an enhanced point correspondence method; 3, for each picture, selecting a prompt point from the anchor points corresponding to the picture, and inputting the prompt point and the picture into the SAM for prompt selection analysis; and 4, evaluating a prompt selection result, directly outputting the result after the result reaches the standard, otherwise, re-selecting a more appropriate prompt. According to the method, training for different scenes is not needed, prompts in a large number of pictures can be directly obtained for understanding the pictures, meanwhile, the method has high adaptability and can adapt to any new technology generated along with continuous development of a basic AI model, and therefore continuous performance improvement is achieved, and the method can be expanded to other visual basic models needing prompts.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to an image prompt propagation method and belongs to the field of computer vision. Background Art

[0002] Computer Vision Basic Models: Visual basic models are basic models for performing image processing tasks. These models are trained on huge data sets, enabling them to understand complex features, patterns, and representations in vision, and can output the desired results based on the prompts given by the user.

[0003] Prompt Engineering: Guiding the base AI model system towards the desired output through the design and optimization of prompts and underlying data structures. Prompts serve as contextual clues that guide the model’s focus, significantly improving the effectiveness of the base AI model.

[0004] The basic models in the field of computer vision have advantages such as strong image understanding ability and strong adaptability. Through efficient feature learning modules and large amounts of data for training, the basic models have good performance in multiple tasks. However, since basic models usually require prompts to generate high-quality results, their efficiency is often not high. Existing solutions such as fine-tuning models can partially solve this problem, but fine-tuning also requires specialized training for different scenarios, and the performance improvement for unknown scenarios is still limited.

[0005] In summary, the basic model of computer vision has the characteristics of strong versatility and ease of use, and can achieve good performance without special training for specific scenes. However, it requires user prompts to improve accuracy, and the generation of prompts often requires a large amount of manual annotation, which poses a huge challenge to the rapid batch understanding of images in similar scenes. In order to quickly and accurately understand each picture in a group of similar scenes, the present invention develops a prompt propagation method for the basic model of computer vision. Summary of the invention

[0006] The purpose of the present invention is to provide an image cue propagation method that can utilize the excellent performance of existing basic models and expand to a large number of downstream tasks such as image understanding, image segmentation, video tracking, image annotation and the like.

[0007] To achieve the above object, the technical solution adopted by the present invention is: an image prompt propagation method, which comprises the following steps:

[0008] Step 1: Initialize the anchor point set based on the prompts and template images provided by the user;

[0009] Step 2: Use the enhanced point correspondence method to transfer the anchor point from the template to the pictures in the sequence other than the template;

[0010] Step 3: For each image, select the cue point from the anchor point corresponding to the image and input it into SAM together with the image for cue selection analysis;

[0011] Step 4: Evaluate the prompt selection results and directly output them if they meet the requirements; otherwise, reselect more appropriate prompts.

[0012] Furthermore, in step 1, based on the image and prompt input by the user, the SAM model is used to process the mask of the template image, and then a feature point extraction algorithm is used to obtain all feature points inside the mask, together with the prompt input by the user itself, as the initial anchor point set.

[0013] Furthermore, in step 2, a set of nodes V is first established in the image using a traditional point correspondence algorithm:

[0014]

[0015] Among them l i represents the i-th image in the sequence, j represents the j-th point in the query image, (x j ,y j ) are the coordinates of the points inside the image;

[0016] If the xth point in the template can be connected to the image l i A set of nodes in , then create an inter-layer edge, the edge set is expressed as:

[0017]

[0018] in Represents the query point in the template, Represents all points in each image that are associated with the template query point; connects the points in the template with the points in other images by adding inter-layer edges. For multi-classification tasks, each point has its corresponding label. Traverse each point in the template that is connected to the point and assign the label of the corresponding point in the template to the point:

[0019]

[0020] in Represents any point connected to the template, and L represents the label of the point; in this way, a correspondence is established between the points or marked areas in the template and the points in other images, thereby generating a set of points with the same label and serving as prompt candidates for subsequent selection.

[0021] Furthermore, in step 3, a cue selector is designed to select points suitable as cues from the anchor point set.

[0022] Furthermore, the hint selector first classifies the anchor point set obtained in the previous step according to the label and obtains a first-class anchor point set and a second-class anchor point set:

[0023]

[0024] In the formula For the first i A set of anchor points for each image, For the first i The second anchor point set of the image, L interest Indicates the tags of interest for the current query;

[0025] For a class of anchor point sets, the prompt selector uses a divide-and-conquer algorithm to build a convex hull on all points and then removes points that lie on the convex hull:

[0026]

[0027] ConvexHull is a convex hull constructed using a divide-and-conquer algorithm. After this part is processed, the prompts will be screened in two situations: the first time entering the prompt selector and re-screening if the scorer evaluates it as unqualified.

[0028] Furthermore, when entering the hint selector for the first time, the point closest to the geometric center is directly selected from a set of anchor points as the initial hint:

[0029]

[0030] where p center represents the geometric center of a set of anchor points, Represents the initial type of prompt selected this time; the initial prompt will be input into the SAM model together with the image itself to generate a mask and analyze whether the prompt can correctly reflect the image characteristics.

[0031] Furthermore, for the case where the scorer evaluates unqualified re-screening, the input of the hint selector includes the mask generated by SAM and the currently used hint. If the mask contains the second-class anchor points, the hint selector first identifies these second-class anchor points contained in the mask:

[0032]

[0033] in For the first i The mask area generated by SAM segmentation of the image, Indicates that the mask contains two types of anchor points;

[0034] If there is a class of anchors that are not covered by the mask, the selector is prompted to calculate a class of anchors that are not contained in the masked area:

[0035]

[0036] in Indicates a type of anchor point not included in the mask;

[0037] Finally, the prompt selector is opened by selecting The point closest to the center and the point from Update the hint set to the point farthest from the center:

[0038]

[0039] in Indicates the second type of prompt used in the last segmentation. Indicates the type of prompt used in the last split. Indicates the updated second category prompt. Represents an updated type of prompt; the updated prompt set will be input into the SAM model together with the image to generate a mask and analyze whether the updated prompt set meets the standard.

[0040] Furthermore, in step 4, the prompt selection result is evaluated:

[0041] Firstly, the projection boundary of a class of anchor points is extracted using the alpha-shape algorithm to obtain the approximate outline of the region of interest;

[0042] Then, the intersection-over-union ratio between the approximate contour and the mask result obtained by SAM is calculated. If the intersection-over-union ratio exceeds the preset threshold, the selected prompt set meets the standard and is directly output. If the result fails to meet the standard, the mask coverage area and the anchor point set will be analyzed. If the mask area contains two types of anchor points or does not contain one type of anchor points, the prompt selector will reselect a more reasonable prompt.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] The present invention draws on the excellent achievements of existing basic AI models. It does not require training for different scenarios and can directly obtain prompts from a large number of pictures for understanding the pictures.

[0045] The present invention exhibits strong adaptability and can adapt to any new technologies that arise as the basic AI model continues to evolve, thereby achieving continuous performance improvements.

[0046] The present invention can be extended to other computer vision applications. By propagating cues across a group of similar scenes, the image annotation process can be simplified and the cost of manual labeling can be reduced.

[0047] The present invention realizes object tracking using SAM by consistently utilizing cues to perform target object segmentation over multiple frames of a video, providing an alternative method for the VOS task.

[0048] The present invention utilizes a mature solution of a two-dimensional image base model and can be extended to all existing hint-based base models.

[0049] The invention provides a simple and intuitive interactive interface. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a schematic diagram of the specific implementation of the prompt propagation of the present invention; for a given template and its prompts, the prompt propagation first uses SAM to obtain the mask of the template and extracts important feature points inside the target to initialize the anchor point set, and then the prompt propagation uses the enhanced point matching method to match the anchor point set and obtain the point correspondence. The prompt selector selects a set of valid prompts and inputs them into SAM for analyzing the relationship between the selected prompts and the image. The evaluator will evaluate the selected prompt set and decide whether to accept the prompt selection or update it.

[0051] Figure 2 This is the visual effect of the instance-level video segmentation task based on the hint propagation method on the DAVIS-17 dataset.

[0052] Figure 3 It is the visual effect of the target 3D reconstruction task based on the hint propagation method on the Tanks-and-Temples dataset. DETAILED DESCRIPTION

[0053] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0054] like Figure 1 , an image prompt propagation method, comprising the following steps:

[0055] Step 1: Initialize the anchor point set based on the prompts and template images provided by the user.

[0056] In this step, the SAM model is used to process the mask of the template image, and then the feature point extraction algorithm is used to obtain all the feature points inside the mask, together with the prompt input by the user, as the initial anchor point set.

[0057] Step 2: Use the enhanced point correspondence method to transfer the anchor points from the template to the images in the sequence other than the template.

[0058] In this step, the existing point association method is enhanced to connect the anchor points between different images. First, a set of nodes V is established in the image using the traditional point correspondence algorithm:

[0059]

[0060] Among them l irepresents the i-th image in the sequence, j represents the j-th point in the query image, (x j ,y j ) is the coordinate of the point in the image. If the xth point in the template can be connected to the image l i If we create a set of nodes in , we create an inter-layer edge set represented by:

[0061]

[0062] in Represents the query point in the template, Represents all points in each image that are associated with the template query point. By adding inter-layer edges, we connect the points in the template with the points in other images. For multi-classification tasks, each point has its corresponding label. We traverse each point in the template that is connected to the point and assign the label of the corresponding point in the template to the point:

[0063]

[0064] in Represents any point connected to the template, and L represents the label of the point. In this way, we establish the correspondence between the points or marked areas in the template and the points in other images, thereby generating a set of points with the same label and serving as prompt candidates for subsequent selection.

[0065] Step 3: For each image, select cue points from the anchor points corresponding to the image and input them into SAM together with the image for cue selection analysis.

[0066] In this step, a hint selector is designed to select points suitable as hints from the anchor point set. The hint selector first classifies the anchor point set obtained in the previous step according to the label and obtains the first-class anchor point set and the second-class anchor point set:

[0067]

[0068] In the formula For the first i A set of anchor points for each image, For the first i The second anchor point set of the image, L interest Represents the interesting label of the current query. For a set of anchor points, the prompt selector uses a divide-and-conquer algorithm to build a convex hull on all points, and then deletes the points on the convex hull:

[0069]

[0070] ConvexHull is a convex hull construction method that uses a divide-and-conquer algorithm. After this part of the processing is completed, the prompts will be screened in two situations: the first time entering the prompt selector and re-screening after being evaluated as unqualified by the scorer.

[0071] When entering the hint selector for the first time, the point closest to the geometric center is directly selected from a set of anchor points as the initial hint:

[0072]

[0073] where p center represents the geometric center of a set of anchor points, Indicates the initial type of prompt selected this time. The initial prompt will be input into the SAM model together with the image itself to generate a mask and analyze whether the prompt can correctly reflect the image features.

[0074] For cases where the scorer indicates substandard performance, the input to the hint selector includes the mask generated by SAM and the currently used hint. If the mask contains class II anchors, the hint selector first identifies these class II anchors contained in the mask:

[0075]

[0076] in For the first i The mask area generated by SAM segmentation of the image, Indicates that the mask contains a set of two types of anchor points. If there is a type of anchor point that is not covered by the mask, the selector is prompted to calculate the type of anchor point that is not included in the mask area:

[0077]

[0078] in represents a class of anchor points not included in the mask. Finally, the prompt selector selects The point closest to the center and the point from Update the hint set to the point farthest from the center:

[0079]

[0080] in Indicates the second type of prompt used in the last segmentation. Indicates the type of prompt used in the last split. Indicates the updated second category prompt. Represents an updated type of prompt. The updated prompt set will be input into the SAM model together with the image to generate a mask and analyze whether the updated prompt set meets the standard.

[0081] Step 4: Evaluate the prompt selection results and directly output them if they meet the requirements; otherwise, reselect more appropriate prompts.

[0082] In this step, the alpha-shape algorithm is first used to extract the projection boundary of the first type of anchor point to obtain the approximate contour of the region of interest. Then the intersection over union (IoU) between the approximate contour and the mask result obtained by SAM is calculated. If the IoU exceeds the preset threshold (usually set to 0.8), the selected prompt set meets the standard and is directly output. If the result fails to meet the standard, the mask coverage area and the anchor point set will be analyzed. If the mask area contains the second type of anchor point or does not contain the first type of anchor point, the prompt selector will reselect a more reasonable prompt.

[0083] In summary, the implementation process of the present invention is as follows: load an image sequence or video file, start the interactive selection module, and the user selects the target area of ​​interest by clicking. Use the SAM model to obtain the mask of the area of ​​interest, and extract the feature points inside the mask to initialize the anchor point set of the template image.

[0084] The traditional point matching algorithm is used to obtain the points that are related to each other between all images, and the enhancer is used to enhance the related results to obtain richer point matching relationships.

[0085] The cue selector selects the most suitable cue set based on the point matching relationship and inputs it into SAM to obtain a mask for subsequent analysis of the selected cue.

[0086] Evaluate the selected cue, and output it directly if the result meets the criteria, otherwise return to the selector to reselect a more appropriate set of cue points.

[0087] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the protection scope of the present invention in any form, and all technical solutions obtained by equivalent replacement and the like fall within the protection scope of the present invention.

[0088] The parts not involved in the present invention are the same as the prior art or can be implemented by using the prior art.

Claims

1. An image prompt propagation method, characterized in that: The steps include: Step 1: Initialize the anchor point set based on the prompts and template images provided by the user; Step 2: Use the enhanced point correspondence method to transfer the anchor point from the template to the pictures in the sequence other than the template; Step 3: For each image, select the cue point from the anchor point corresponding to the image and input it into SAM together with the image for cue selection analysis; Step 4: Evaluate the prompt selection results and directly output them if they meet the requirements; otherwise, reselect more appropriate prompts.

2. The image prompting dissemination method according to claim 1, characterized in that: In step 1, based on the picture and prompt input by the user, the SAM model is used to process the mask of the template picture, and then a feature point extraction algorithm is used to obtain all feature points inside the mask, together with the prompt input by the user itself as the initial anchor point set.

3. The image prompting propagation method according to claim 1, characterized in that: In step 2, a set of nodes V is first established in the image using a traditional point correspondence algorithm: Among them l i represents the i-th image in the sequence, j represents the j-th point in the query image, (x j ,y j ) are the coordinates of the points inside the image; If the xth point in the template can be connected to the image l i A set of nodes in , then create an inter-layer edge, the edge set is expressed as: in Represents the query point in the template, Represents all points in each image that are associated with the template query point; connects the points in the template with the points in other images by adding inter-layer edges. For multi-classification tasks, each point has its corresponding label. Traverse each point in the template that is connected to the point and assign the label of the corresponding point in the template to the point: in Represents any point connected to the template, and L represents the label of the point; in this way, a correspondence is established between the points or marked areas in the template and the points in other images, thereby generating a set of points with the same label and serving as prompt candidates for subsequent selection.

4. The image prompting dissemination method according to claim 1, characterized in that: In step 3, a cue selector is designed to select points suitable as cues from the anchor point set.

5. The image prompting transmission method according to claim 4, characterized in that: The hint selector first classifies the anchor point set obtained in the previous step according to the label and obtains a first-class anchor point set and a second-class anchor point set: In the formula For the first i A set of anchor points for each image, For the first i The second anchor point set of the image, L interest Indicates the tags of interest for the current query; For a class of anchor point sets, the prompt selector uses a divide-and-conquer algorithm to build a convex hull on all points and then removes points that lie on the convex hull: ConvexHull is a convex hull constructed using a divide-and-conquer algorithm. After this part is processed, the prompts will be screened in two situations: the first time entering the prompt selector and re-screening if the scorer evaluates it as unqualified.

6. The image prompting transmission method according to claim 5, characterized in that: When entering the hint selector for the first time, the point closest to the geometric center is directly selected from a set of anchor points as the initial hint: where p center represents the geometric center of a set of anchor points, Represents the initial type of prompt selected this time; the initial prompt will be input into the SAM model together with the image itself to generate a mask and analyze whether the prompt can correctly reflect the image characteristics.

7. The image prompting transmission method according to claim 5, characterized in that: For the case where the scorer evaluates unqualified re-screening, the input of the hint selector includes the mask generated by SAM and the currently used hint. If the mask contains the second-class anchor points, the hint selector first identifies these second-class anchor points contained in the mask: in For the first i The mask area generated by SAM segmentation of the image, Indicates that the mask contains two types of anchor points; If there is a class of anchors that are not covered by the mask, the selector is prompted to calculate a class of anchors that are not contained in the masked area: in Indicates a type of anchor point not included in the mask; Finally, the prompt selector is opened by selecting The point closest to the center and the point from Update the hint set to the point farthest from the center: in Indicates the second type of prompt used in the last segmentation. Indicates the type of prompt used in the last split. Indicates the updated second category prompt. Indicates a type of prompt after update; The updated prompt set will be input into the SAM model together with the image to generate a mask and analyze whether the updated prompt set meets the requirements.

8. The image prompting dissemination method according to claim 1, characterized in that: In step 4, the prompt selection result is evaluated: Firstly, the projection boundary of a class of anchor points is extracted using the alpha-shape algorithm to obtain the approximate outline of the region of interest; Then, the intersection-over-union ratio between the approximate contour and the mask result obtained by SAM is calculated. If the intersection-over-union ratio exceeds the preset threshold, the selected prompt set meets the standard and is directly output. If the result fails to meet the standard, the mask coverage area and the anchor point set will be analyzed. If the mask area contains two types of anchor points or does not contain one type of anchor points, the prompt selector will reselect a more reasonable prompt.

Citation Information

Patent Citations

  • Multi-mode output intelligent vibration detection method and device

    CN110956645A

  • Interactive automatic labeling system and method based on graffiti prompt coding large model

    CN117934512A

  • Computer-aided detection with enhanced workflow

    US20100135562A1

  • System and Method for Extremely Efficient Image and Pattern Recognition and Artificial Intelligence Platform

    US20180204111A1