Weak supervision visual space positioning method and system

Abstract

The invention relates to a weak supervision visual space positioning method and system. The method comprises the following steps of: generating various and real pseudo queries for each candidate box by using three complementary methods, and forming pseudo query region pairs; by calculating the single-mode similarity of the pseudo query and the real query, taking the candidate box corresponding to the pseudo query with the highest similarity as a pseudo label of the real query; secondly, estimating a confidence coefficient score of a query region pair by utilizing an image-text matching pre-training model; and finally, a visual space positioning model based on Transform is adopted to cooperate with the designed selective positioning loss based on confidence score, and supervised training is carried out on the region query pair. According to the method, a matching method based on single-mode similarity is used, a model can be helped to establish more reliable region query pairs, distribution drift between real query and pseudo query is avoided, and meanwhile, the designed selective positioning loss based on confidence can help effectively avoid overfitting of query region pairs which are wrongly associated.

Description

Technical Field

[0001] The present invention relates to the field of visual space positioning, and in particular to a weakly supervised visual space positioning method and system. Background Art

[0002] Visual spatial localization is an important task with a large number of potential applications. It has great application value in visual question answering, robot operation, visual language navigation and other fields. Visual spatial localization technology is a technology that analyzes and understands the content of an image and locates the object in the image that is most relevant to the query statement based on a given natural language query. This technology is mainly divided into three categories: fully supervised, weakly supervised and unsupervised. Under the fully supervised task, the training set gives the natural language query and the corresponding image and the corresponding constraint box (given in the form of upper left corner, length and width); under the weakly supervised condition, the training set gives the corresponding image and natural language query without the corresponding constraint box; under the unsupervised condition, the training set only has a bunch of images and queries and the images do not correspond to the queries.

[0003] Fully supervised methods are generally divided into two categories: one-stage and two-stage methods. Representative two-stage methods include: Mao et al. proposed an algorithm that generates explicit descriptions for specific objects or regions in an image, and can also preprocess or interpret such expressions to infer the object being described; Yu et al. proposed an algorithm that models the subject and object in natural language descriptions and the corresponding relationship; Yang et al. proposed an algorithm that uses graph algorithms to better model object relationships and implements association learning. Representative one-stage methods include: Yang et al. proposed an algorithm that encodes language features and fuses them into the YOLOv3 detector and then locates the object being referred to; Liao et al. proposed an algorithm that regards the visual space positioning task as a correlation filtering process and selects the peak of the relevant hotspot map as the target object; Deng et al. proposed an algorithm that inputs visual and text tokens into a series of stacked Transformers for cross-modal fusion and directly outputs the target box. However, fully supervised methods require a lot of time and effort to annotate images, which seriously limits the practicality and scalability of these methods.

[0004] In the weakly supervised setting of visual spatial localization, there are algorithms based on contrastive learning mechanisms and algorithms based on reconstruction mechanisms. The former is represented by: Gupta et al. proposed a contrastive learning framework based on mutual information and infoNCE (info noise contrastive estimation), which regards the visual spatial localization task as estimating the mutual information between the image region and the natural language description. Wang et al. proposed to use the knowledge distillation method with the contrastive learning loss function to model a scoring function that can calculate the similarity between the image region features and the text features. The latter is represented by: Liu et al. proposed a reconstruction architecture based on knowledge distillation, which aims to model the mapping of the input query to the candidate box. Sun et al. proposed a strategy of distinguishing triples and a scalable query parsing to better identify the subject and object. However, both of the above methods ignore the heterogeneous gap between regional features and text features, which makes the method of using cross-modal matching to select candidate boxes to reconstruct the original query or align regional query features unreliable; secondly, both methods ignore the situation of wrong association in the reconstruction process or contrastive learning process, which makes the model fall into the situation of error propagation and error accumulation, thus seriously hindering the performance of the model.

[0005] Representative unsupervised methods include: Jiang et al. proposed an algorithm for supervised training based on a set of manual rules to generate pseudo queries for candidate boxes. However, the pseudo queries generated by this method have a single style and ignore the distribution drift between pseudo queries and real queries. Summary of the invention

[0006] In view of the deficiencies in the prior art, the present invention proposes a confidence-aware pseudo-label weakly supervised visual space positioning method and system.

[0007] In order to achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0008] A weakly supervised visual space localization method for localizing objects most relevant to a natural language query comprises the following steps:

[0009] (1) Pseudo-query generation: Generate diverse and realistic pseudo-queries for each candidate box, forming a <pseudo-query, region> pair;

[0010] (2) Real query assignment: Using the unimodal similarity between the real query and the pseudo query, the candidate box corresponding to the pseudo query with the highest similarity is selected as the pseudo label of the real query, and it forms a <real query, region> pair with the real query;

[0011] (3) Confidence-aware association verification: Use the pre-trained model of image-text matching to estimate the confidence score for all <query, region> pairs, indicating the degree of association between the <query, region> pairs.

[0012] (4) Visual spatial localization model: A fully supervised visual spatial localization model based on Transformer is used with a selective localization loss based on confidence scores to perform supervised training on the <query, region> pairs. The trained fully supervised visual spatial localization model is then used to perform visual spatial localization.

[0013] Furthermore, step (1) generates diverse and realistic pseudo queries for each candidate box using three complementary methods, wherein the three complementary methods include: a heuristic method, an object feature method, and a relationship description method.

[0014] Furthermore, step (1) uses a pre-trained unimodal model (detector, classifier, attribute extractor, etc.) to extract multiple candidate boxes, categories, and attributes, and filters out candidate boxes and attributes that deviate from the true query distribution. Then, a heuristic rule-based terminal and a pre-trained description model are used to generate diverse and realistic pseudo queries for each candidate box and form pseudo query region pairs.

[0015] Furthermore, in order to make the pre-trained description model in step (1) better meet the description requirements, the present invention proposes a conditional prompting project. Two methods are used to generate pseudo queries for the description model: a. Input the candidate box and conditional prompts into the model, and let the description model complete the rest. b. Input the entire image and conditional prompts into the model, and let the description model complete the rest.

[0016] Furthermore, the heuristic method generates a pseudo query for each candidate box based on a fixed sentence structure of <category><attribute><relationship>; for categories, by counting the nouns that have appeared in the real query, some redundant object categories detected by the detector are filtered out based on these nouns; for attributes, in addition to adjectives, considering that objects such as clothes and hair can also be used as attributes, the contained objects can be used as attributes of another object by comparing the degree of inclusion of the box; for relationships, by comparing the size order of coordinates of candidate boxes of similar objects, single or multiple directional words such as front and back, left and right, up and down, and ordinal words such as left one and right two are added to each box.

[0017] Furthermore, the object feature method sends the cropped candidate boxes to the description model to generate various pseudo queries. In order to enable the model to better describe the features of the object, the conditional prompt project (the category name of the candidate box is used as the beginning of the sentence) is input into the description model to complete the entire sentence.

[0018] Furthermore, the relationship description method inputs the entire image into the description model to generate the relationship between the object in the candidate box and other objects in the image. In order to reduce the ambiguity of the model description, the location information obtained by the heuristic method is combined with the conditional prompt project (the category name of the candidate box plus the location information as the beginning of the sentence) and input into the description model to complete the entire sentence.

[0019] Furthermore, step (2) uses all pseudo queries obtained in step (1) and real queries to calculate the similarity between the texts, and assigns the candidate box corresponding to the pseudo query with the highest similarity to the real query to form a real query region pair.

[0020] Furthermore, step (3) uses a pre-trained image-text matching model to estimate a confidence score for the pseudo query region pair obtained in step (1) and the real query region pair obtained in step (2). The lower the score, the lower the degree of association.

[0021] Furthermore, step (4) utilizes a Transformer-based model to perform fully supervised training on all query region pairs.

[0022] Furthermore, step (4) adopts a norm regularization loss and giou (generalized intersection-over-union) loss, and, in order to suppress the damage of wrong association to the model, a selective localization loss is proposed based on the confidence score, which adjusts the loss weight of each sample in a batch based on the confidence score.

[0023] A weakly supervised visual space positioning system using the above method comprises:

[0024] The pseudo-query generation module is used to generate diverse and realistic pseudo-queries for each candidate box, forming a <pseudo-query, region> pair;

[0025] The real query assignment module is used to calculate the text feature similarity between the real query and the pseudo query, and assign the candidate box corresponding to the pseudo query with the highest similarity to the real query to form a <real query, region> pair;

[0026] The confidence-aware association verification module is used to estimate the association degree of all <query, region> pairs and obtain a confidence score for the subsequent fully supervised training of the model;

[0027] The visual-spatial localization module uses a Transformer-based visual-spatial localization model to perform supervised training on all <query, region> pairs, and cooperates with a selective localization loss based on confidence scores to avoid false associations that hinder the performance of the model.

[0028] The beneficial effects of the present invention are:

[0029] On the one hand, the present invention can make full use of the advantages of the pre-trained model, combining three complementary methods to generate diverse and realistic pseudo-queries for each candidate box, while introducing unimodal matching between texts to replace the cross-modal matching between text and image areas, and assigning a pseudo-labeled constraint box to each real query to form a real query area pair, so that the subsequent model can be supervised for training; on the other hand, a confidence-aware association verification module is introduced, which estimates a confidence score for each query area pair and cooperates with the designed selective positioning loss function to suppress the influence of some erroneous associations on model training, thereby preventing the model from falling into error propagation and error accumulation.

[0030] The reason why the present invention has the above-mentioned inventive effects is that: first, by improving the pseudo-query terminal generated based on rules and introducing a pre-trained description model and prompt engineering to generate diverse, real and reliable pseudo-queries for each candidate box, the distribution drift between pseudo-queries and real queries is reduced; secondly, the cross-modal similarity matching method of text and image areas is replaced by a unimodal similarity matching method between texts, making the matching process more reliable; finally, the confidence perception method proposed in the present invention can effectively estimate the association between query area pairs, and combined with the selective positioning loss function, the reliability of the samples learned by the model is guaranteed, avoiding error propagation and error accumulation. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a flow chart of the weakly supervised visual space positioning method and system of the present invention.

[0032] Figure 2 It is a schematic diagram of the object feature and relationship description method in the pseudo query generation unit of the present invention. DETAILED DESCRIPTION

[0033] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0034] The present invention provides a weakly supervised visual space positioning method and system, the process of which is as follows: Figure 1 As shown, the specific steps include:

[0035] (1) Pseudo query generation unit

[0036] The pseudo-query generation unit generates diverse and realistic candidate boxes for a series of candidate boxes extracted by the detector. The unit includes three complementary generation methods: heuristic, object features, and relationship description.

[0037] Heuristic: This method first extracts the categories, attributes and relationships of candidate boxes through some detectors. Then, for categories, objects that are far from the true query category are filtered out; for attributes, except for color, height, etc., when the overlapping area of ​​a box with other boxes accounts for more than 80% of its own area, it is considered that the box can be used as an attribute of other boxes (such as Figure 2 (b) shows that black hair can be used as an attribute of the man on the left); for relationships, by comparing the coordinates and area sizes of similar objects, position relationships are added to each frame. This method can generate some pseudo queries with relative and absolute position relationships (left and right, up and down, front and back, etc.). The detector includes an object detector and an attribute detector, which uses the detector to extract object candidate frames and object categories and attributes in the candidate frames for each image.

[0038] Object features: This method uses conditional cue engineering and pre-trained description models, such as Figure 2 As shown in (a), the candidate region and the conditional hint "man {}" can be input into the pre-trained description model BLIP to complete the text in the brackets. This method can generate a more specific description for the candidate box, making up for the shortcomings of the heuristic method.

[0039] Relational description: This method combines heuristic location information and pre-trained description models. Specifically, this method attempts to feed the entire image into the description model to generate a pseudo query. In order to improve the accuracy of the description, this method combines the location information of the object and conditional hint engineering. Figure 2 As shown in (b), this method inputs the conditional prompt "the man on the left {}" and the entire image into the model to complete the text in the brackets. This method can generate relationships with other objects for the candidate box, for example: the man on the left is listening to another man, etc.

[0040] (2) Real query allocation

[0041] Different from the previous method of using cross-modal similarity matching, the present invention designs a simpler and more reliable single-modal similarity matching method. By using word2vec to encode the real query and pseudo query and calculate the cosine similarity, the candidate box corresponding to the pseudo query with the highest similarity is assigned to the real query to form a real query region pair. The single-modal similarity matching method is as follows:

[0042]

[0043] Among them, r i is the i-th candidate box of the image, t is the real query, and p ij is the jth pseudo query generated by the i-th candidate box, Sim(·) is the cosine similarity function, which measures the similarity of two vectors:

[0044]

[0045] Here, φ(·) represents the function that converts the query into a vector.

[0046] (3) Confidence-aware association verification module

[0047] Since the query region pairs generated in step (1) contain some erroneous and unreal queries, and the real query and region matched in step (2) have mismatching problems, the present invention uses a pre-trained image-text matching model to estimate the degree of association between the pseudo query region pairs (generated in step (1)) and the real query region pairs (generated in step (2)), and estimates a confidence score c for each query region pair. i The higher the score, the higher the correlation degree of the query region pair. The pre-trained text matching model takes the image and text as input and outputs a two-dimensional vector to represent the matching and non-matching scores of the image and text. Here, the present invention uses the matching score of the image and text, that is, the value of the first dimension of the output two-dimensional vector, as the confidence score of the query region pair.

[0048] (4) Visual spatial positioning model

[0049] In order to train the generated query region pairs well, the present invention uses a simple Transformer structure to implement a fully supervised visual space localization model. The visual space localization model directly outputs the coordinates of the target box under given query conditions and images. The objective function of the training is:

[0050] The fully supervised visual spatial localization model is composed of a visual encoder, a language encoder, a visual language fusion Transformer and a coordinate regression network. It uses a bounding box loss function to perform supervised training on all <query, region> pairs. The bounding box loss function of the model is obtained by the following formula:

[0051]

[0052] Among them, b i =(x i ,y i ,w i ,h i ) is the normalized true label, is the coordinate of the prediction box, and Represents the smooth L1 norm loss function and giou loss function;

[0053] At the same time, in order to suppress error propagation and error accumulation, this method proposes a selective localization loss function based on confidence score, which is used in the model training process to make the model pay more attention to the relatively high correlation <query, region> pairs according to the confidence score of the <query, region> pair, while ignoring the <query, region> pairs with extremely low correlation; the entire selective localization loss based on confidence score is obtained by the following formula:

[0054]

[0055] in,

[0056] Among them, D is a sample in a batch, α i is the loss participation weight of the ith <query, region> pair, τ is a confidence score threshold, by which some region query pairs that are considered to be incorrectly associated can be ignored to avoid the model from overfitting these incorrect <query, region> pairs, and c i The confidence score of the <query, region> pair is output by the pre-trained graph-text matching model.

[0057] The following experimental results show that compared with existing methods, the weakly supervised visual space localization method and system of the present invention can achieve higher localization accuracy.

[0058] This embodiment uses the RefCOCO dataset. This data is collected from the MSCOCO image dataset. It contains 16,994 images, 50,000 query objects, each of which has one or more natural language queries, and a total of 142,210 natural language queries. The descriptions and images of this dataset contain multiple reference objects and diverse language query descriptions, so it has a certain degree of wide application and challenge, which helps to verify the effectiveness and superiority of this method.

[0059] Existing method 1: The method proposed by Jiang et al. in the paper “Pseudo-Q: Generating Pseudo Language Queries for Visual Grounding”.

[0060] Existing method 2: The method proposed by Sun et al. in the paper “Discriminative triad matching and reconstruction for weakly referring expression grounding”.

[0061] Existing method three: the method proposed by Subramanian et al. in the paper “ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehension”.

[0062] The present invention: the method of this embodiment.

[0063] Usually, in visual space positioning tasks, the top-1 target box is selected as the final prediction box and the accuracy is calculated. The experiment uses the Intersection over Union (IoU) metric to evaluate the accuracy of the target box. This metric measures the percentage of IoU between the predicted constraint box and the true label that is greater than or equal to the threshold of 0.5, that is, when the overlap area between the predicted box and the true label exceeds 0.5, the predicted box is considered correct.

[0064] As can be seen from Table 1, the present invention achieves better visual space positioning effect. The existing methods either ignore the distribution drift between real queries and pseudo queries, or use unreliable matching methods based on cross-modal similarity to reconstruct or compare and learn. At the same time, the existing methods also ignore the error propagation and error accumulation caused by the incorrect association of query area pairs, which hinders the performance of the model. The present invention first generates a variety of real pseudo queries for each candidate box, and assigns a candidate box to each real query through a unimodal similarity matching algorithm between real queries and pseudo queries, thereby improving the reliability of the constructed query area pairs; then, by designing a confidence-aware association verification module, the confidence score of each query area pair is estimated and combined with a selective positioning loss function, thereby effectively suppressing the error propagation effect caused by some incorrectly associated area query pairs; and improving the accuracy of visual space positioning.

[0065] Table 1. Comparative experimental results with existing methods (unit: %)

[0066] method Val TestA TestB Existing method 1 56.02 58.25 54.13 Existing method 2 39.21 41.14 37.72 Existing method three 45.78 46.10 47.07 The present invention 70.67 74.58 67.19

[0067] Based on the same inventive concept, another embodiment of the present invention provides a weakly supervised visual space positioning system using the above method, which includes:

[0068] The pseudo-query generation module is used to generate diverse and realistic pseudo-queries for each candidate box, forming a <pseudo-query, region> pair;

[0069] The real query assignment module is used to calculate the text feature similarity between the real query and the pseudo query, and assign the candidate box corresponding to the pseudo query with the highest similarity to the real query to form a <real query, region> pair;

[0070] The confidence-aware association verification module is used to estimate the association degree of all <query, region> pairs and obtain a confidence score for the subsequent fully supervised training of the model;

[0071] The visual-spatial localization module uses a Transformer-based visual-spatial localization model for supervised training on all <query, region> pairs, and cooperates with a selective localization loss based on confidence scores to avoid false associations that hinder the performance of the model.

[0072] Based on the same inventive concept, another embodiment of the present invention provides an electronic device (computer, server, smart phone, etc.), which includes a memory and a processor, the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program includes instructions for executing each step in the method of the present invention.

[0073] Based on the same inventive concept, another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, CD), which stores a computer program. When the computer program is executed by a computer, it implements the various steps of the method of the present invention.

[0074] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A weakly supervised visual space positioning method and system, characterized in that: The following steps are involved: Generate diverse and realistic pseudo-queries for candidate boxes, forming <pseudo-query, region> pairs; Using the unimodal similarity between the real query and the pseudo query, the candidate box corresponding to the pseudo query with the highest similarity is selected as the pseudo label of the real query, and it forms a <real query, region> pair with the real query; Use the pre-trained model of image-text matching to estimate the confidence score for all <query, region> pairs, indicating the degree of association between <query, region> pairs; Using the Transformer-based fully supervised visual spatial localization model and the selective localization loss based on confidence score, supervised training is performed on the composed <query, region> pairs, and visual spatial localization is performed using the trained fully supervised visual spatial localization model.

2. The method according to claim 1, characterized in that Three complementary methods are used to generate diverse and realistic pseudo queries for each candidate box, and the three complementary methods include: a heuristic method, an object feature method, and a relationship description method.

3. The method according to claim 2, characterized in that The heuristic method generates a pseudo query for each candidate box based on a fixed sentence structure of <category><attribute><relationship>; for the category, the nouns that appear in the real query are counted, and the redundant object categories detected by the detector are filtered out based on these nouns; For attributes, by comparing the degree of inclusion of the boxes, the contained object is used as the attribute of another object; for relations, by comparing the size order of the coordinates of the candidate boxes of the same object, single or multiple directional words and ordinal words are added to each box.

4. The method according to claim 2, characterized in that The object feature method sends the cropped candidate boxes to the description model to generate various pseudo queries. In order to enable the model to better describe the features of the object, the conditional prompt project (the category name of the candidate box is used as the beginning of the sentence) is input into the description model to complete the entire sentence.

5. The method according to claim 2, characterized in that The relationship description method inputs the entire image into the description model to generate the relationship between the object in the candidate box and other objects in the image. In order to reduce the ambiguity of the model description, the location information obtained by the heuristic method is combined with the conditional prompt project (the category name of the candidate box plus the location information as the beginning of the sentence) and input into the description model to complete the entire sentence.

6. The method according to claim 1, characterized in that The method utilizes the unimodal similarity between the real query and the pseudo query to select the candidate box corresponding to the pseudo query with the highest similarity as the pseudo label of the real query, and forms a <real query, region> pair with the real query, including: The word2vec model is used to extract text features from real queries and pseudo queries, and the cosine similarity is calculated to represent the similarity between the real query and the pseudo query. The candidate box corresponding to the pseudo query with the highest similarity is used as the pseudo label of the real query. The whole process is obtained by the following formula: Among them, r i is the i-th candidate box of the image, t is the real query, and p ij is the jth pseudo query generated by the i-th candidate box, Sim(·) is the cosine similarity function, which measures the similarity of two vectors: Here, φ(·) represents the function that converts the query into a vector.

7. The method according to claim 1, characterized in that The method of estimating the confidence scores of all <query, region> pairs using the pre-trained graph-text matching model is to send all the <query, region> pairs to a pre-trained graph-text matching model, estimate the correlation between the query and the candidate region, and output a confidence score c i .

8. The method according to claim 1, characterized in that The fully supervised visual spatial localization model is composed of a visual encoder, a language encoder, a visual language fusion Transformer and a coordinate regression network. It uses a bounding box loss function to perform supervised training on all <query, region> pairs. The bounding box loss function of the model is obtained by the following formula: Among them, b i =(x i ,y i ,w i ,h i ) is the normalized true label, is the coordinate of the prediction box, and Represents the smooth L1 norm loss function and giou loss function; The confidence score-based selective localization loss is used to allow the model to pay more attention to the <query, region> pairs with relatively high correlation, and ignore the <query, region> pairs with extremely low correlation, according to the confidence scores of the <query, region> pairs during the model training process. The entire confidence score-based selective localization loss is obtained by the following formula: in, Among them, D is a sample in a batch, α i is the loss participation weight of the ith <query, region> pair, τ is a confidence score threshold, by which some region query pairs that are considered to be incorrectly associated can be ignored to avoid the model from overfitting these incorrect <query, region> pairs, and c i The confidence score of the <query, region> pair is output by the pre-trained graph-text matching model.

9. A weakly supervised visual space positioning system using the method of any one of claims 1 to 8, characterized in that: include: The pseudo-query generation module is used to generate diverse and realistic pseudo-queries for each candidate box, forming a <pseudo-query, region> pair; The real query assignment module is used to calculate the text feature similarity between the real query and the pseudo query, and assign the candidate box corresponding to the pseudo query with the highest similarity to the real query to form a <real query, region> pair; The confidence-aware association verification module is used to estimate the association degree of all <query, region> pairs and obtain a confidence score for the subsequent fully supervised training of the model; The visual-spatial localization module uses a Transformer-based visual-spatial localization model to perform supervised training on all <query, region> pairs, and cooperates with a selective localization loss based on confidence scores to avoid false associations that hinder the performance of the model.

10. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the method according to any one of claims 1 to 8.

Images