Homogeneous object recognition method based on visual question answering module and color mask

Abstract

The application discloses a kind of same kind object identification method based on visual question answering module and color mask, comprising: user input image, all objects in image are labeled, the name of all objects is obtained;The subject of visual question is specified;Label placeholder in visual question;Substitute placeholder to obtain new visual question;Visual question and image are input to visual question answering module to obtain candidate answer;Different colors are used to color candidate answer;With the color of object name, the placeholder of visual question is replaced again, and new visual question is obtained again;The latest visual question and the colored image are input into visual question answering module, and the final answer of visual question is obtained.The application can effectively distinguish and identify the objects belonging to the same category in scene.

Description

Technical Field

[0001] The present invention relates to the technical field of object recognition, and in particular to a method for recognizing similar objects based on a visual question-answering module and a color mask. Background Art

[0002] In visual question answering (VQA), identifying and localizing entities presents significant challenges. Pre-trained visual language models address this problem by providing textual descriptions as answers. However, in visual scenes with multiple entities of the same category, textual descriptions often struggle to effectively distinguish objects belonging to the same category. Consequently, VQA datasets are limited by the limitations of textual descriptions and fail to adequately cover scenes involving multiple objects of the same category. Summary of the Invention

[0003] The purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art and provide a method for identifying similar objects based on a visual question-answering module and color masking, which can effectively distinguish objects belonging to the same category.

[0004] To achieve the above objectives, the present invention provides a technical solution: a method for identifying similar objects based on a visual question-answering module and a color mask, comprising the following steps:

[0005] S1: Label all objects in the image and obtain the names of all objects;

[0006] S2: Among all the object names obtained, specify the subject of the visual problem, which is called the problem subject;

[0007] S3: Modify the visual question to mark the placeholder as the object name;

[0008] S4: Fill all object names into the placeholders of the modified visual question in sequence to obtain a visual question containing different object names;

[0009] S5: Input visual questions and images containing different object names into the visual question answering module, obtain the confidence score between each object and the question subject, and select the object with the highest confidence score as the candidate answer;

[0010] S6: Use different colors to color each candidate answer in the image to obtain a candidate answer with color modification;

[0011] S7: Fill the color-modified candidate answers into the placeholders of the modified visual question in sequence, to obtain multiple visual questions containing the same candidate answers but different colors;

[0012] S8: Input the visual question and the colored image containing the same candidate answers in different colors into the visual question answering module, obtain the confidence scores between the candidate answers in different colors and the question body, and select the candidate answer with the highest confidence score as the final answer to the visual question.

[0013] Furthermore, in step S1, the image is automatically annotated using Labelmg, which is represented by M tag , define the input image as i, and get the set of all object names A={a1,a2,...,a n}, where a1, a2, ..., a n is the name of each object in the image, and the formula is as follows:

[0014] A=M tag (i).

[0015] Furthermore, in step S2, the problem subject is specified and the subject bounding box b is overlaid on the problem subject. The overlay process is defined as Get the image i with the subject frame b , the formula is as follows:

[0016]

[0017] Furthermore, in step S3, the visual question Q is modified. According to the subject-verb-object relationship, the subject or object corresponding to the question body of the visual question is marked as a placeholder, and the visual question Q1 is obtained after the modification.

[0018] Further, in step S4, the object name set A={a1, a2, ..., a n} replace the placeholder in Q1 with the object name in the Q1 to get a new visual problem

[0019] Furthermore, in step S5, the CLIP model is used for zero-shot visual question answering, and the visual question answering process is represented as M vqa , and the questions and image i b Input into the visual question answering module to obtain the confidence score between the object and the question subject Select the highest confidence score Then the following process is performed: From the highest confidence score Work backwards to the corresponding question, and then use the name of the object in the question as the candidate answer A * , define this process as M getN , the formula is as follows:

[0020]

[0021]

[0022]

[0023]

[0024]

[0025] Further, in step S6, the candidate answer A obtained in step S5 is * , there are many candidate answers in the image, let these candidate answers be In order to distinguish these candidate answers, b The mask areas corresponding to these candidate answers are covered with different colors to obtain image i c , these colors are represented by C1, C2, ..., C m Meaning, the process of overwriting the color mask is as follows:

[0026]

[0027] Further, in step S7, the candidate answers are modified with colors. Replace the placeholders in Q1 to get a series of Visual problems

[0028] Furthermore, in step S8, the CLIP model is used again for zero-shot visual question answering, and the visual question answering process is represented as M vqa , and visual problems and image i c Input into the visual question answering module to obtain the confidence score between the candidate answer and the question subject Select the highest confidence score S * , then perform the following process: from the highest confidence score S * Work backwards to the corresponding question, and then use the color and candidate answer in the question as the final answer CA last , define this process as M getCN , the formula is as follows:

[0029]

[0030]

[0031]

[0032]

[0033] CA last=M getCN (S * ).

[0034] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0035] 1. The present invention adds color masks of different colors during object recognition, which can effectively distinguish similar objects in the image.

[0036] 2. This invention can perform zero-sample visual question answering on the pre-trained CLIP model and can be applied to a variety of scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 4 is a framework diagram of the similar object recognition method of this embodiment.

[0038] Figure 2 Graph showing the experimental results of this embodiment. DETAILED DESCRIPTION

[0039] The present invention will be described in further detail below with reference to the embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0040] This embodiment discloses a method for identifying similar objects based on a visual question answering module and a color mask, using the CLIP model as the base model of the visual question answering module. Figure 1 The specific situation is as follows:

[0041] 1) The user inputs an image i and a visual question Q: “What is this person looking at?”

[0042] 2) Label all objects in the image and obtain the names of all objects:

[0043] The image is automatically annotated using Labelmg, denoted as M tag , the obtained set of all object names A = {banana, stem, person}, the formula is as follows:

[0044] A=M tag (i)

[0045] 3) Among all the object names obtained, specify the subject of the visual problem, called the problem subject:

[0046] The user specifies the subject of the problem. In this example, the subject of the problem is the person in image i. The subject bounding box b is overlaid on this person. The overlay process is defined as Get the image i with the subject frame b , the formula is as follows:

[0047]

[0048] 4) Modify the visual problem and mark the placeholder as the object name:

[0049] The visual question Q is modified to obtain Q1: "What is this person looking at?" which has been marked as a placeholder for the name of the object.

[0050] 5) Replace the placeholder:

[0051] Replace the placeholders in the visual question Q1 with the object names in the object name set A = {banana, stem, person} to get the visual question "Is this person looking at a banana?", a visual problem "Is this person looking at the stalk?", a visual problem "Is this person looking at someone?"

[0052] 6) Use the question-answering module to get confidence scores:

[0053] Use the CLIP model for zero-shot visual question answering and represent the visual question answering process as M vqa , and visual problems and image i b Input into the visual question answering module to get the confidence score between the object and the person Select the highest confidence score Then do the following: From the highest confidence score Reverse the question to: "Is this person looking at a banana?", and then take the name of the object in the question as the candidate answer A. * , define this process as M getN , the formula is as follows

[0054]

[0055]

[0056]

[0057]

[0058]

[0059] 7) Color the object according to the candidate answer and obtain the colored object:

[0060] There are many candidate answers in image i: banana, which is divided into banana 1, banana 2, and banana 3 from left to right. In order to distinguish these bananas, in image i b Cover the mask areas corresponding to these bananas with different colors to obtain image ic , use red C from left to right r Green C g and blue C b The process of coloring and overwriting the color mask is as follows:

[0061]

[0062] Get bananas with color modifications: red banana; green banana; blue banana.

[0063] 8) Replace the placeholder again:

[0064] Replace the placeholders in Q1 with the names of the objects with color modifications to get question Q 2,rb : "Is this person looking at a red banana?", Question Q 2,gb : "Is this person looking at a green banana?", Question Q 2,bb : "Is this person looking at a blue banana?".

[0065] 9) Use the question-answering module to get confidence scores:

[0066] The CLIP model is used again for zero-shot visual question answering, and the visual question answering process is represented as M vqa , and then ask the question Q 2,rb ,Q 2,gb ,Q 2,bb and image i c Input into the visual question answering module to obtain the confidence score S between bananas of different colors and the question subject rb ,S gb ,S bb , select the highest confidence score S * , then perform the following process: from the highest confidence score S * Reverse the question to: "Is this person looking at a red banana?", and then take the red banana in the question as the final answer CA last , define this process as M getCN , the formula is as follows:

[0067] S rb =M vaq (Q 2,rb ,i C )

[0068] S gb =M vaq (Q 2,gb ,i c )

[0069] S bb =M vaq (Q2,bb ,i C )

[0070] S * =max{S rb ,S gb ,S bb}

[0071] CA last =M getCN (S * )

[0072] To make the final answer more accurate, we repeat this visual question answering multiple times and select the answer with the highest frequency as the final candidate answer. In this example, the final output answer is that the person looks at the banana on the far left again.

[0073] In order to verify the effectiveness of the method of the present invention, experimental tests were conducted on multiple scenarios, and the experimental results are as follows: Figure 2 As shown in the figure, the experimental results fully demonstrate the effectiveness of the proposed method. By using color masks in the visual question answering process and implementing a zero-shot object recognition method on the CLIP model, it can be used in different scenarios.

[0074] Experimental conclusion: Due to the limitations of text descriptions, visual question answering datasets cannot fully cover scenes involving multiple objects of the same category. This paper proposes a similar object recognition method based on a visual question answering module and color mask. Experimental evaluations on three public datasets, MS COCO, Places, and PASCAL, show that the method of the present invention can achieve repeated recognition of scenes involving multiple objects of the same category, which is superior to existing methods, has good application prospects, and is worthy of promotion.

[0075] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A similar object recognition method based on a visual question answering module and color mask, characterized in that: The following steps are involved: S1: Label all objects in the image and obtain the names of all objects; S2: Among all the object names obtained, specify the subject of the visual problem, which is called the problem subject; Specify the problem subject and cover the subject bounding box b on the problem subject. The covering process is defined as Get the image i with the subject frame b , the formula is as follows: Where, represents the input image; S3: Modify the visual question to mark the placeholder as the object name; Modify the visual question Q. According to the subject-verb-object relationship, mark the subject or object corresponding to the question body of the visual question as a placeholder. After modification, the visual question Q1 is obtained. S4: Fill all object names into the placeholders of the modified visual question in sequence to obtain a visual question containing different object names; Use the object name set A={a1,a2,...,a n } replace the placeholder in Q1 with the object name in the Q1 to get a new visual problem S5: Input visual questions and images containing different object names into the visual question answering module, obtain the confidence score between each object and the question subject, and select the object with the highest confidence score as the candidate answer; Use the CLIP model for zero-shot visual question answering and represent the visual question answering process as M vqa , and the questions and image i b Input into the visual question answering module to obtain the confidence score between the object and the question subject Select the highest confidence score Then the following process is performed: From the highest confidence score Work backwards to the corresponding question, and then use the name of the object in the question as the candidate answer A * , define this process as M getN , the formula is as follows: S6: Use different colors to color each candidate answer in the image to obtain a candidate answer with color modification; S7: Fill the color-modified candidate answers into the placeholders of the modified visual question in sequence, to obtain multiple visual questions containing the same candidate answers but different colors; S8: Input the visual question and the colored image containing the same candidate answers in different colors into the visual question answering module, obtain the confidence scores between the candidate answers in different colors and the question body, and select the candidate answer with the highest confidence score as the final answer to the visual question.

2. The method for identifying similar objects based on a visual question answering module and a color mask according to claim 1, characterized in that: In step S1, the image is automatically annotated using Labelmg, denoted as M tag , define the input image as i, and get the set of all object names A={a1,a2,...,a n }, where a1, a2, ..., a n is the name of each object in the image, and the formula is as follows: A=M tag (i)。 3. The method for identifying similar objects based on a visual question answering module and a color mask according to claim 2, wherein: In step S6, the candidate answer A obtained in step S5 is * , there are many candidate answers in the image, let these candidate answers be A1 * ,A2 * ,...,A m * , in order to distinguish these candidate answers, in image i b The mask areas corresponding to these candidate answers are covered with different colors to obtain image i c , these colors are represented by C1, C2, ..., C m Meaning, the process of overwriting the color mask is as follows:

4. The method for identifying similar objects based on a visual question answering module and a color mask according to claim 3, wherein: In step S7, the candidate answers are modified with colors. Replace the placeholders in Q1 to get a series of Visual problems 5. The method for identifying similar objects based on a visual question answering module and a color mask according to claim 4, characterized in that: In step S8, the CLIP model is used again for zero-shot visual question answering, and the process of visual question answering is represented as M vqa , and visual problems and image i c Input into the visual question answering module to obtain the confidence score between the candidate answer and the question subject Select the highest confidence score S * , then perform the following process: from the highest confidence score S * Work backwards to the corresponding question, and then use the color and candidate answer in the question as the final answer CA last , define this process as M getCN , the formula is as follows: THAT last =M getCN (S * )。

Images