Image prompting propagation method

Abstract

The application discloses an image prompt propagation method, which comprises the following steps: step 1, initializing an anchor point set according to a user-provided prompt and a template picture; step 2, using an enhanced point correspondence method to transfer the anchor point from the template to pictures in a sequence except the template; step 3, for each picture, selecting a prompt point from the anchor point corresponding to the picture and inputting the picture and the prompt point into a SAM for prompt selection analysis; and step 4, evaluating the prompt selection result, directly outputting if the result meets the requirement, or reselecting a more suitable prompt. The application does not need to be trained for different scenes, can directly obtain prompts in a large number of pictures for understanding the pictures, has strong adaptability, can adapt to any new technology generated along with the continuous development of a basic AI model, thereby realizing continuous performance improvement, and can be extended to other visual basic models needing prompts.

Description

Technical Field

[0001] This invention relates to an image cues propagation method, belonging to the field of computer vision. Background Technology

[0002] Computer vision foundational models: These are the basic models for performing image processing tasks. Trained on massive datasets, these models are able to understand complex features, patterns, and representations in vision and output the desired results based on user-provided prompts.

[0003] Hint engineering: Through the design and optimization of hints and underlying data structures, basic artificial intelligence model systems are guided towards the desired output direction. Hints, as contextual cues guiding the model's focus, significantly improve the effectiveness of basic artificial intelligence models.

[0004] Basic models in computer vision possess advantages such as strong image understanding capabilities and adaptability. Trained with efficient feature learning modules and large amounts of data, these basic models demonstrate good performance across multiple tasks. However, because they typically require prompts to generate high-quality results, their efficiency is often low. Existing solutions, such as fine-tuning models, can partially address this issue, but fine-tuning also requires specialized training for different scenarios, and performance improvements remain limited for unknown scenarios.

[0005] In summary, computer vision foundational models are characterized by their versatility and ease of use, achieving good performance without requiring specialized training for specific scenarios. However, they rely on user prompts to improve accuracy, and the generation of prompts often necessitates extensive manual annotation, posing a significant challenge to the rapid batch understanding of images in similar scenes. To quickly and accurately understand each image in a set of similar scenes, this invention develops a prompt propagation method for computer vision foundational models. Summary of the Invention

[0006] The purpose of this invention is to provide an image cue propagation method that can leverage the superior performance of existing basic models and be extended to a large number of downstream tasks such as image understanding, image segmentation, video tracking, and image annotation.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is: an image prompt propagation method, which includes the following steps:

[0008] Step 1: Initialize the anchor point set based on the user-provided prompts and template images;

[0009] Step 2: Use the enhanced point-to-point method to pass the anchor points from the template to the images in the sequence other than the template;

[0010] Step 3: For each image, select a cue point from the corresponding anchor point and input it along with the image into SAM for cue selection analysis;

[0011] Step 4: Evaluate the selected results. If the results meet the criteria, output them directly; otherwise, select a more suitable result.

[0012] Furthermore, in step 1, based on the image and prompts input by the user, a mask for the template image is obtained using the SAM model. Then, a feature point extraction algorithm is used to obtain all feature points inside the mask, which, together with the prompts input by the user, are used as the initial anchor point set.

[0013] Furthermore, in step 2, a set of nodes V is first established in the image using a traditional point correspondence algorithm:

[0014]

[0015] Among them l i Let represent the i-th image in the sequence, and j represent the j-th point in the query image. j ,y j () represents the coordinates of a point within the image;

[0016] If the x-th point in the template can be connected to the image l i For a set of nodes in the set, an inter-layer edge is created, which is represented as:

[0017]

[0018] in This represents the query point in the template. This represents all points in each image associated with the query point in the template. By adding inter-layer edges, the points in the template are connected to points in other images. For multi-class classification tasks, each point has its corresponding label. The process iterates through each point in the template that is connected to that point and assigns the corresponding label from the template to that point.

[0019]

[0020] in L represents any point connected to the template; thus, a correspondence is established between points or marked regions in the template and points in other images, thereby generating a set of points with the same label, which serve as prompts for subsequent selection.

[0021] Furthermore, in step 3, a prompt selector is designed to select suitable points from the set of anchor points as prompts.

[0022] Furthermore, the prompt selector first classifies the anchor set obtained in the previous step according to the label, resulting in a first-class anchor set and a second-class anchor set:

[0023]

[0024] In the formula Indicates that for the lth i A set of anchor points for an image. Indicates that for the lth i The second set of anchor points for the image, L interest Indicates the tags of interest in the current query;

[0025] For a set of anchor points, the hint selector uses a divide-and-conquer algorithm to construct a convex hull over all points, and then deletes points located on the convex hull:

[0026]

[0027] ConvexHull is constructed using a divide-and-conquer algorithm for convex hull construction. After this part is processed, there are two cases for filtering prompts: entering the prompt selector for the first time and being re-filtered if the prompt is deemed unqualified by the scorer.

[0028] Furthermore, for the first entry into the cue selector, the point closest to the geometric center will be directly selected from a set of anchor points as the initial cue:

[0029]

[0030] Where p center Describes the geometric center of a set of anchor points. This indicates the initial type of cue selected for this operation. The initial cue, along with the image itself, will be input into the SAM model to generate a mask and analyze whether the cue can accurately reflect the image features.

[0031] Furthermore, for cases where the scorer's evaluation fails and re-screening is required, the input to the selector includes the mask generated by SAM and the currently used hint. If the mask contains type II anchors, the selector first identifies these type II anchors contained in the mask.

[0032]

[0033] in Indicates that for the lth i The mask region generated by SAM segmentation of the image. This indicates that the mask contains a set of two types of anchor points;

[0034] If there is a class of anchor points not covered by the mask, the selector is prompted to calculate the class of anchor points not included in the mask area:

[0035]

[0036] in This indicates a type of anchor point not included in the mask;

[0037] Finally, the prompt selector allows you to select from... To the nearest point to the center and from Update the hint set to the point furthest from the center:

[0038]

[0039] in This indicates the type II cue used in the previous segmentation. This indicates a type of cue used in the previous segmentation. This indicates the updated type II prompt. This represents an updated set of prompts; the updated prompt set will be input into the SAM model along with the image to generate a mask and analyze whether the updated prompt set meets the criteria.

[0040] Furthermore, in step 4, the selection result is evaluated:

[0041] First, the alpha-shape algorithm is used to extract the projection boundary of a class of anchor points to obtain the outline of the approximate region of interest.

[0042] Then, the intersection-union ratio (CUI) between the approximate contour and the mask result obtained by SAM is calculated. If the CUI exceeds the preset threshold, the selected cue set meets the standard and is directly output. If the result fails to meet the standard, the mask coverage area and anchor point set will be analyzed. If the mask area contains two types of anchor points or does not contain one type of anchor point, then the cue selector will reselect a more reasonable cue.

[0043] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0044] This invention draws on the excellent achievements of existing basic AI models, eliminating the need for training for different scenarios and allowing direct acquisition of cues from a large number of images for image understanding.

[0045] This invention demonstrates strong adaptability and can adapt to any new technologies that emerge as the underlying AI model continues to evolve, thereby achieving continuous performance improvements.

[0046] This invention can be extended to other computer vision applications. By propagating cues in a set of similar scenes, it can simplify the image annotation process and reduce the cost of manual annotation.

[0047] This invention provides an alternative method for VOS tasks by consistently using cues to perform target object segmentation across multiple frames of a video and enabling object tracking using SAM.

[0048] This invention utilizes a mature solution for a two-dimensional image-based basic model and can be extended to all existing prompt-based basic models.

[0049] This invention provides a simple and intuitive interactive interface. Attached Figure Description

[0050] Figure 1 This is a schematic diagram illustrating the specific implementation of the cue propagation of the present invention. For a given template and its cue, cue propagation first uses SAM to obtain the mask of the template and extracts important feature points inside the target to initialize the anchor point set. Then, cue propagation uses an enhanced point matching method to match the anchor point set and obtain the point correspondence. The cue selector selects a set of valid cue inputs to SAM to analyze the relationship between the selected cue and the image. The evaluator evaluates the selected cue set and decides whether to accept the cue selection or update it.

[0051] Figure 2 This is the visual effect of an instance-level video segmentation task based on the cue propagation method on the DAVIS-17 dataset.

[0052] Figure 3 This is the visual effect of a target 3D reconstruction task based on the cue propagation method on the Tanks-and-Temples dataset. Detailed Implementation

[0053] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0054] like Figure 1 A method for propagating image cues includes the following steps:

[0055] Step 1: Initialize the anchor point set based on the user-provided prompts and template images.

[0056] In this step, the SAM model is used to process and obtain the mask of the template image. Then, the feature point extraction algorithm is used to obtain all the feature points inside the mask, which, together with the prompts input by the user, are used as the initial anchor point set.

[0057] Step 2: Use the enhanced point mapping method to pass the anchor points from the template to the images in the sequence other than the template.

[0058] In this step, existing point association methods are enhanced to connect anchor points between different images. First, a set of nodes V is established in the image using a traditional point correspondence algorithm:

[0059]

[0060] Among them l iLet represent the i-th image in the sequence, and j represent the j-th point in the query image. j y j Let x be the coordinates of a point within the image. If the x-th point in the template can be connected to the image l... i If a set of nodes is defined in the given set, then we create an inter-layer edge, which is represented as:

[0061]

[0062] in This represents the query point in the template. This represents all points in each image associated with the query point in the template. By adding inter-layer edges, we connect the points in the template to points in other images. For multi-class classification tasks, each point has a corresponding label. We iterate through each point in the template connected to that point and assign the corresponding label from the template to that point:

[0063]

[0064] in Let L represent any point connected to the template, and L represent the label of that point. In this way, we establish a correspondence between points or labeled regions in the template and points in other images, thereby generating a set of points with the same label, which serve as candidate suggestions for subsequent selection.

[0065] Step 3: For each image, select a cue point from the corresponding anchor point and input it along with the image into SAM for cue selection analysis.

[0066] In this step, a cue selector is designed to select suitable points as cue points from the anchor point set. The cue selector first classifies the anchor point set obtained in the previous step according to the label, resulting in a first-class anchor point set and a second-class anchor point set:

[0067]

[0068] In the formula Indicates that for the lth i A set of anchor points for an image. Indicates that for the lth i The second set of anchor points for the image, L interest This indicates the tags of interest for the current query. For a set of anchor points, the hint selector uses a divide-and-conquer algorithm to construct a convex hull over all points, and then deletes points located on the convex hull.

[0069]

[0070] ConvexHull employs a divide-and-conquer algorithm for convex hull construction. After this processing, the suggestions will be filtered in two ways: first entry into the suggestion selector and re-filtering if deemed unsuitable by the scorer.

[0071] For the first entry into the cue selector, the point closest to the geometric center will be directly selected from a set of anchor points as the initial cue:

[0072]

[0073] Where p center Describes the geometric center of a set of anchor points. This indicates the initial cue selected for this application. The initial cue, along with the image itself, will be input into the SAM model to generate a mask and analyze whether the cue accurately reflects the image features.

[0074] For cases where the scorer indicates substandard performance, the cue selector's input includes the mask generated by SAM and the currently used cue. If the mask contains type II anchors, the cue selector first identifies these type II anchors contained in the mask:

[0075]

[0076] in Indicates that for the lth i The mask region generated by SAM segmentation of the image. This indicates that the mask contains two sets of anchor points. If there are anchor points of the first type not covered by the mask, the selector is prompted to calculate the anchor points of the second type not included in the mask region.

[0077]

[0078] in This indicates a class of anchor points not included in the mask. Finally, the prompt selector selects from... To the nearest point to the center and from Update the hint set to the point furthest from the center:

[0079]

[0080] in This indicates the type II cue used in the previous segmentation. This indicates a type of cue used in the previous segmentation. This indicates the updated type II prompt. This represents an updated set of cues. The updated cue set, along with the image, will be input into the SAM model to generate a mask and analyze whether the updated cue set meets the criteria.

[0081] Step 4: Evaluate the selected results. If the results meet the criteria, output them directly; otherwise, select a more suitable result.

[0082] In this step, the alpha-shape algorithm is first used to extract the projected boundaries of one type of anchor points, obtaining an approximate contour of the region of interest. Then, the intersection-over-union (IoU) ratio between the approximate contour and the mask result obtained from SAM is calculated. If the IoU exceeds a preset threshold (usually set to 0.8), the selected cue set is considered acceptable and directly output. If the result fails to meet the threshold, the mask-covered area and the anchor point set are analyzed. If the mask area contains two types of anchor points or does not contain one type of anchor point, the cue selector will reselect a more suitable cue.

[0083] In summary, the implementation process of this invention is as follows: Load an image sequence or video file, activate the interactive selection module, and allow the user to select the target region of interest by clicking. Use the SAM model to obtain a mask for the region of interest, and extract feature points within the mask to initialize the anchor point set of the template image.

[0084] The traditional point matching algorithm is used to obtain the points that are related to each other in all images, and the association results are enhanced by an enhancer to obtain richer point matching relationships.

[0085] The cue selector filters the most suitable cue set based on point matching relationships and inputs it into SAM to obtain a mask for subsequent analysis of the selected cue.

[0086] The selected prompts are evaluated. If the result meets the criteria, the prompts are output directly; otherwise, the selection process returns to the selector to choose a more suitable set of prompts.

[0087] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the scope of protection of the present invention in any way, and all technical solutions obtained by equivalent substitution or other means fall within the scope of protection of the present invention.

[0088] All parts not covered in this invention are the same as or can be implemented using existing technologies.

Claims

1. A method for propagating image cues, characterized in that, Includes the following steps: Step 1: Initialize the anchor point set based on the user-provided prompts and template images; Step 2: Use an enhanced point-to-point mapping method to transfer anchor points from the template to the images in the sequence other than the template; first, use the traditional point-to-point mapping algorithm to establish a set of nodes V in the image: Among them l i Let i represent the i-th image in the sequence, and j represent the j-th point in the query image. The coordinates of a point within the image; If the x-th point in the template can be connected to the image l i For a set of nodes in the set, an inter-layer edge is created, which is represented as: in This represents the query point in the template. This represents all points in each image associated with the query point in the template. By adding inter-layer edges, the points in the template are connected to points in other images. For multi-class classification tasks, each point has its corresponding label. The process iterates through each point in the template that is connected to that point and assigns the corresponding label from the template to that point. in This represents any point connected to the template, where L represents the label of that point; In this way, a correspondence is established between points or marked regions in the template and points in other images, thereby generating a set of points with the same labels, which serve as prompts for subsequent selections; Step 3: For each image, select a cue point from the corresponding anchor point and input it along with the image into SAM for cue selection analysis; Step 4: Evaluate the selected results. If the results meet the criteria, output them directly; otherwise, select a more suitable result.

2. The image prompt propagation method according to claim 1, characterized in that: In step 1, based on the image and prompts input by the user, a mask for the template image is obtained using the SAM model. Then, a feature point extraction algorithm is used to obtain all feature points inside the mask, which, together with the prompts input by the user, are used as the initial anchor point set.

3. The image prompt propagation method according to claim 1, characterized in that: In step 3, a prompt selector is designed to select a suitable point from the set of anchor points as a prompt.

4. The image prompt propagation method according to claim 3, characterized in that: The prompt selector first classifies the anchor set obtained in the previous step according to the label, resulting in a first-class anchor set and a second-class anchor set: In the formula Indicates for the first A set of anchor points for an image. Indicates for the first The second set of anchor points for the image. Indicates the tags of interest in the current query; For a set of anchor points, the hint selector uses a divide-and-conquer algorithm to construct a convex hull over all points, and then deletes points located on the convex hull: in It uses a divide-and-conquer algorithm to construct the convex hull. After this part is processed, there are two cases for filtering prompts: entering the prompt selector for the first time and being re-filtered if the scorer evaluates it as unqualified.

5. The image prompt propagation method according to claim 4, characterized in that: For the first entry into the cue selector, the point closest to the geometric center will be directly selected from a set of anchor points as the initial cue: in Describes the geometric center of a set of anchor points. This indicates the initial type of cue selected for this operation. The initial cue, along with the image itself, will be input into the SAM model to generate a mask and analyze whether the cue can accurately reflect the image features.

6. The image prompt propagation method according to claim 4, characterized in that: For cases where the scorer's evaluation fails and re-screening is required, the input to the selector includes the mask generated by SAM and the currently used hint. If the mask contains type II anchors, the selector first identifies these type II anchors contained in the mask: in Indicates for the first The mask region generated by SAM segmentation of the image. This indicates that the mask contains a set of two types of anchor points; If there is a class of anchor points not covered by the mask, the selector is prompted to calculate the class of anchor points not included in the mask area: in This indicates a type of anchor point not included in the mask; Finally, the prompt selector allows you to select from... The point closest to the center and Update the hint set to the point furthest from the center: in This indicates the type II cue used in the previous segmentation. This indicates a type of cue used in the previous segmentation. This indicates the updated type II prompt. This indicates a type of updated notification; The updated cue set will be fed into the SAM model along with the images to generate masks and analyze whether the updated cue set meets the requirements.

7. The image prompt propagation method according to claim 1, characterized in that: In step 4, the selection result is evaluated: First, the alpha-shape algorithm is used to extract the projection boundary of a class of anchor points to obtain the outline of the approximate region of interest. Then, the intersection-union ratio (CUI) between the approximate contour and the mask result obtained by SAM is calculated. If the CUI exceeds the preset threshold, the selected cue set meets the standard and is directly output. If the result fails to meet the standard, the mask coverage area and anchor point set will be analyzed. If the mask area contains two types of anchor points or does not contain one type of anchor point, then the cue selector will reselect a more reasonable cue.

Images