Performance Evaluation Method and Device for Visual Language Models in Localization Tasks

Abstract

This invention provides a method and apparatus for performance evaluation of a visual language model in a localization task. The localization task includes attribute-based localization, relation-based localization, and primary / secondary localization. The method includes: constructing an evaluation dataset for performance evaluation of the visual language model; performing a localization task on the evaluation samples in the evaluation dataset using the visual language model to obtain a localization heatmap of the evaluation samples; determining the performance evaluation score of the evaluation samples based on the localization heatmap; and determining the average performance evaluation score as the average accuracy of the evaluation dataset as the performance evaluation result of the visual language model. This application verifies the combinatorial reasoning ability of the visual language model in distinguishing attributes, relationships, and primary / secondary dimensions, thus addressing the deficiency of poor performance in combinatorial reasoning ability of pre-trained visual language models in the prior art.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method and apparatus for evaluating the performance of a visual language model in a localization task. Background Technology

[0002] Visual language combination evaluation aims to gain a deeper understanding of the capabilities of pre-trained visual language models in processing image and text information combinations. However, in localization tasks, pre-trained visual language models exhibit the following problems when processing images and descriptive text for some complex visual language combinations: First, attribute discrimination is inaccurate, failing to accurately distinguish target objects in images, particularly in densely packed objects or complex image scenes, making it difficult to accurately count and identify attributes such as color and shape of target objects. Second, relationship discrimination is inadequate, failing to understand and process relationships between target objects in images, such as the relative position, orientation, and combination of target objects with other objects in the image. Third, hierarchy discrimination is problematic, often neglecting the importance of word order when processing descriptive text, failing to distinguish the primary and secondary order of target objects in the image, leading to misunderstandings of the image content.

[0003] Therefore, based on the above performance in distinguishing attributes, relationships, and primary and secondary aspects, it can be seen that the pre-trained visual language model performs poorly in terms of combinatorial reasoning ability in localization tasks. Summary of the Invention

[0004] This invention provides a method and apparatus for evaluating the performance of a visual language model in a localization task. It can verify the combined reasoning ability of a visual language model in distinguishing attributes, distinguishing relationships, and distinguishing primary and secondary aspects, thereby addressing the shortcomings of existing pre-trained visual language models in terms of combined reasoning ability.

[0005] This invention provides a method for evaluating the performance of a visual language model in a localization task. The localization task includes attribute-based localization, relation-based localization, and primary / secondary localization. The method comprises the following steps: An evaluation dataset is constructed for performance evaluation of the visual language model. The evaluation dataset includes multiple evaluation samples. Each evaluation sample includes a target image and two descriptive texts. The target image is marked with two target objects that correspond one-to-one with the descriptive texts. For each evaluation sample in the evaluation dataset, the localization task is performed on the evaluation sample using the visual language model to obtain a localization heatmap of the target image in the evaluation sample. The localization task involves determining the corresponding target object from the target image based on the descriptive text, under the conditions of distinguishing attributes, distinguishing relationships, and distinguishing primary and secondary attributes. The performance evaluation score of the evaluation sample is then determined based on the localization heatmap. The average of the performance evaluation scores is determined as the average accuracy of the evaluation dataset, and the average accuracy is used as the performance evaluation result of the visual language model.

[0006] In some embodiments, the evaluation dataset includes an attribute evaluation sample set, a relationship evaluation sample set, and a primary and secondary evaluation sample set; the performance evaluation results of the visual language model include: attribute performance evaluation results, relationship evaluation results, and primary and secondary performance evaluation results; The attribute performance evaluation result is obtained by the visual language model after performing the attribute-based localization task using the attribute evaluation sample set. The relation performance evaluation result is obtained by the visual language model after performing the localization task based on genus relation discrimination using the relation evaluation sample set; The primary and secondary performance evaluation results are obtained by the visual language model after performing the primary and secondary differentiation-based localization task using the primary and secondary evaluation sample set.

[0007] In some embodiments, the two target objects in the evaluation samples of the attribute evaluation sample set are two objects with different attributes in the target image; the two target objects in the evaluation samples of the relationship evaluation sample set are two objects with different relationships in the target image, wherein the relationship is the association relationship between the target object and other objects, and the other objects are objects in the target image other than the two target objects; the two target objects in the evaluation samples of the primary and secondary evaluation sample set are two objects in the target image with a primary and secondary order.

[0008] In some embodiments, performing the localization task on the evaluation sample using the visual language model to obtain a localization heatmap of the target image in the evaluation sample includes: Obtain a target attention map, wherein the target attention map is any intermediate attention map obtained in the visual language model during the execution of the localization task; Determine the image-text matching score between the target image and the descriptive text, and for each image-text matching score, determine the gradient of the target attention map relative to the image-text matching score; For each gradient, a localization heatmap of the target image in the evaluation sample is determined by the target attention map and the gradient.

[0009] In some embodiments, determining the performance evaluation score of the evaluation sample based on the location heatmap includes: determining a mask map of the target image for the target object; Based on the location heatmap and the mask map, the average activation value of the evaluation sample is determined; The performance evaluation score of the evaluation sample is determined based on the preset activation value conditions satisfied by the average activation value.

[0010] In some embodiments, the visual language model is obtained by training it through performing an agent task, and the training process of the visual language model includes: Obtain training image samples and text pairs corresponding to the training image samples; For each training image sample, the visual language model is invoked to perform the proxy task to obtain a training heatmap of the training image sample for the text pair. The proxy task is to determine the training target object in the training image sample based on the text pair and to construct a training heatmap based on the training target object. The loss function of the visual language model is constructed based on the training heatmap, and backpropagation is performed on the visual language model through the loss function to update the parameters of the visual language model.

[0011] In some embodiments, obtaining the text pairs corresponding to the training image samples includes: Multiple candidate descriptive texts are determined based on the training image samples; Dependency structure parsing is performed on the candidate description text to obtain a tree structure of the candidate description text, wherein the tree structure includes nodes and directed arcs connecting the nodes, the nodes represent each word in the candidate description text, and the directed arcs represent the connection relationship between each word; The training target object in the training image sample is determined based on the root node in the tree structure, and a text pair corresponding to the training image sample is constructed based on the root node corresponding to the training target object.

[0012] In some embodiments, the loss function for constructing the visual language model based on the training heatmap includes: determining the element-wise multiplication result of the training heatmap and determining the area of ​​the training heatmap; Determine the ratio of the element-wise multiplication result to the area, and construct the loss function of the visual language model based on the expected value of the ratio.

[0013] The present invention also provides a performance evaluation device for a visual language model in a localization task, comprising the following modules: a construction module, used to construct an evaluation dataset for performance evaluation of the visual language model, the evaluation dataset comprising multiple evaluation samples, the evaluation samples comprising a target image and two descriptive texts, the target image being marked with two target objects corresponding one-to-one with the descriptive texts; An execution module is used to perform the localization task on each evaluation sample in the evaluation dataset using the visual language model to obtain a localization heatmap of the target image in the evaluation sample. The localization task is to determine the corresponding target object from the target image based on the descriptive text under the conditions of distinguishing attributes, distinguishing relationships, and distinguishing primary and secondary attributes. The determination module is used to determine the performance evaluation score of the evaluation sample based on the location heatmap; The determining module is used to determine that the average value of the performance evaluation score is the average accuracy of the evaluation dataset, and to use the average accuracy as the performance evaluation result of the visual language model.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the performance evaluation method of the visual language model in the localization task as described above.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the performance evaluation method for the visual language model in the localization task as described above.

[0016] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the performance evaluation method of the visual language model in the localization task as described above.

[0017] The present invention provides a performance evaluation method for visual language models in localization tasks. This method involves setting localization tasks based on attribute discrimination, relation discrimination, and primary / secondary discrimination for the visual language model. Under a constructed evaluation dataset, the visual language model is invoked to perform the corresponding localization tasks. The performance evaluation score of the visual language model is then determined based on the localization heatmap obtained after the tasks are completed. This verifies the visual language model's combinatorial reasoning ability in distinguishing attributes, relations, and primary / secondary relationships, thus addressing the deficiency of poor combinatorial reasoning ability in pre-trained visual language models in existing technologies. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the performance evaluation method of the visual language model in the localization task provided by the present invention. Figure 2 This is a sample diagram of the evaluation data provided by the present invention; Figure 3 This is a schematic diagram of the positioning heatmap of the target object provided by the present invention; Figure 4 This is a schematic diagram of the principle of text dependency parsing provided by the present invention; Figure 5 This is a schematic diagram of the performance evaluation device for the visual language model in the localization task provided by the present invention. Figure 6 This is a schematic diagram of the physical structure of an electronic device provided by the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0021] Figure 1 This is a flowchart illustrating the performance evaluation method of the visual language model in localization tasks provided by this invention, as shown below. Figure 1 As shown, the method includes the following steps: Step 101: Construct an evaluation dataset for performance evaluation of the visual language model.

[0022] In this embodiment of the invention, to further explore the capabilities of the visual language pre-trained model in combinatorial reasoning, a localization task for the visual language model was constructed as an evaluation task, and a corresponding evaluation dataset was built, denoted as the ARPGrounding dataset. The capabilities in combinatorial reasoning are mainly reflected in three dimensions: the attributes, relationships, and primary / secondary relationships of the target object. Accordingly, the localization task is also divided into attribute-based localization tasks, relationship-based localization tasks, and primary / secondary localization tasks, used to evaluate the visual language model's ability to distinguish object attributes, object relationships, and the primary / secondary relationships of objects, respectively.

[0023] The evaluation dataset includes multiple evaluation samples. Each sample consists of a target image and two descriptive texts. The target image is labeled with two target objects that correspond one-to-one with the descriptive texts, using bounding boxes. In other words, the target objects are selected within the target image using bounding boxes.

[0024] In this embodiment of the invention, the evaluation dataset is divided into three categories: attribute evaluation sample set, relationship evaluation sample set, and primary and secondary evaluation sample set. These sample sets will be introduced one by one below.

[0025] In the attribute evaluation sample set, the two target objects in the evaluation sample are objects with two different attributes in the target image, and the corresponding localization task is a localization task based on attribute discrimination. In the image of the evaluation sample, object attributes can be attributes such as color, state, and size. The two target objects belong to the same category of objects. See details... Figure 2 , Figure 2 This is a sample diagram of the evaluation data provided by the present invention, such as... Figure 2 As shown in (a) and (b), the two descriptive texts are "brown dog" and "white dog". The target objects in the images both belong to the dog category, but their color attributes are different. The localization task of the visual language model is to distinguish between white and brown, and to identify and select the dog with the correct color in the image.

[0026] In terms of attributes, this embodiment of the invention selects two objects of suitable size, non-overlapping nature, and belonging to the same category by filtering the Visual Genome (VG) dataset. To ensure that these attributes are distinguishable, this embodiment of the invention also utilizes WordNet to confirm the differences in attributes between objects and thereby generate descriptive text with discriminative capabilities.

[0027] In the relation evaluation sample set, the two target objects in the evaluation sample are two objects in the target image with different relationships but the same attributes. Here, the relationship is the relationship between the target object and other objects, which are objects in the target image other than the two target objects. The corresponding localization task is a localization task based on relation discrimination. In the image of the evaluation sample, the relationship is the association between the target object and other objects in the image. The association can be the positional relationship, directional relationship, connection relationship, or combination relationship between the target object and other objects. Here, the two target objects in the image may have the same attributes and the same category. For example... Figure 2 As shown in (c) and (d), the two descriptive texts are "bear under car" and "bear in car". The two target objects in the image are both yellow bears with the same color attribute and the same bear category, but their relationships are different. That is, the bears have different relationships with other objects (cars) in the image (different positional relationships, inside the car and under the car). The localization task of the visual language model is to distinguish the positional relationship between the two bears and the car (inside the car or under the car) and to identify and select the bear in the correct position in the image.

[0028] In terms of relational dimension, in order to ensure that the visual language model can distinguish target objects based solely on the relationship between them according to the descriptive text, this embodiment of the invention filters out triplet relation pairs that meet the conditions (e.g., bear in the car, bear under the car, car) in the VG dataset, and ensures that these relations are semantically distinguishable, excluding attribute information that may cause confusion.

[0029] In the primary and secondary evaluation sample set, the two target objects in the evaluation sample are two objects in the target image with a specific primary and secondary order. The corresponding localization task is a localization task based on the distinction between primary and secondary. In the image of the evaluation sample, primary and secondary refer to main and secondary importance. The target objects can be two objects with different attributes and relationships, and their emphasis differs in the image and descriptive text. For example... Figure 2 As shown in (e) and (f), the two descriptive texts are "bathroom sink under bathroom mirror" and "bathroom mirror above bathroom sink." The target objects in the image are a bathroom mirror and a bathroom sink. However, the first descriptive text emphasizes the bathroom sink, making it the primary focus, while the second descriptive text emphasizes the bathroom mirror, making it the primary focus, and the bathroom sink the secondary focus. That is, the two descriptive texts describe target objects with different levels of importance. The localization task of the visual language model aims to distinguish the primary and secondary order of the bathroom mirror and bathroom sink, identifying and selecting the primary bathroom mirror or bathroom sink in the image based on the corresponding descriptive text.

[0030] In terms of primary and secondary dimensions, this embodiment of the invention constructs a test set to test the model's understanding of the order of importance by reversing the relationship between the target object and the descriptive text in the image. For example, in a descriptive text that emphasizes a bathroom mirror (i.e., the bathroom mirror is primary and the bathroom sink is secondary), the bathroom sink is intentionally marked with an inverted bounding box in the target image to allow the visual language model to recognize it, thereby evaluating the model's ability to distinguish the order of importance.

[0031] Therefore, through step 101 above, this embodiment of the invention constructs an attribute sample set containing 6632 evaluation samples, a relationship sample set containing 370 evaluation samples, and a primary / secondary sample set containing 4423 evaluation samples. Each evaluation sample involves different target object examples in the image, and some target objects have ambiguous relationships. Compared with the other two dimensions, the relationship sample set has fewer evaluation samples because when two ambiguous target objects are involved in the image, they are more likely to have distinguishable attributes but are less likely to show distinguishable relationships with other target objects.

[0032] For example, given the descriptive texts "black car" and "white car," a model can typically select the correct target object based on one descriptive text, but not the other. This is because visual language models are influenced by the more salient target object in the target image ("black car" is more salient in the target image) during reasoning, and may also be affected by data bias during training ("black car" is more common in the training dataset). Therefore, this embodiment of the invention comprehensively evaluates the combinatorial reasoning performance of the visual language model by using the mutual interference of two combinatorially ambiguous target objects.

[0033] Step 102: For each evaluation sample in the evaluation dataset, perform a localization task on the evaluation sample using a visual language model to obtain a localization heatmap of the target image in the evaluation sample.

[0034] After constructing the ARPGrounding dataset, the dataset is then input into a visual language model for evaluation. The visual language model performs corresponding localization tasks (i.e., attribute-based, relation-based, and primary / secondary localization tasks), identifying and determining the corresponding target objects from the target images based on the descriptive text, under the conditions of distinguishing attributes, relations, and primary / secondary elements, respectively. This evaluates the visual language model's ability to distinguish attributes, relations, and primary / secondary elements. The visual language model is pre-trained and can be directly used to perform localization tasks on the dataset.

[0035] In this embodiment of the invention, the recognition result of the visual language model is represented by a location heatmap. Therefore, the visual language model performs a location task for the evaluation sample. That is, after determining the corresponding target object from the target image based on the description text, the location heatmap of the target image in the evaluation sample can be obtained.

[0036] The following describes the process of using a visual language model to perform a localization task to identify target objects and obtain a localization heatmap.

[0037] In this embodiment of the invention, the Grad-GAM algorithm is used to calculate the localization heatmap, and the visual language model can specifically be an image recognition network (e.g., Transformer). The target image and descriptive text in the evaluation samples are input into the visual language model. The Grad-GAM algorithm guides the visual language model to locate the target object described in the descriptive text within the target image by utilizing the gradient flow from the descriptive text to the intermediate layer of the Transformer in the visual language model.

[0038] First, the target attention map is obtained. This target attention map is any intermediate attention map obtained during the localization task in the visual language model. The visual language model contains multiple Transformer intermediate layers. Each intermediate layer processes the target image and obtains a corresponding feature map (attention map). This attention map can be randomly selected from any Transformer intermediate layer (denoted as...). f Extract an intermediate attention map A from ) Z As the target attention map, if the processing function of the intermediate layer f of the Transformer is denoted as f... Z So, attention graph A Z This can be expressed as the following formula: A Z =f Z (φ f (v,t))

[0039] In the above formula, v represents the target image in the input evaluation sample, and t represents the descriptive text.

[0040] Next, we determine the image-text matching scores between the target image and the descriptive text, and for each image-text matching score, we determine the gradient of the target attention map relative to the image-text matching score. Here, because the visual language model needs to identify the target object in the image based on the descriptive text, image-text matching is required during the recognition process. This is done by extracting the semantic features of the descriptive text and the image features of the target image, and then calculating the similarity between the semantic features and the image features to obtain the image-text matching score. This matching score can be used to represent the image-text similarity or image-text matching score when the visual language model is trained using image-text contrast loss, denoted as y.

[0041] Next, we calculate the gradient of the text-image matching score corresponding to the target attention map. The formula for calculating the gradient is as follows:

[0042] Finally, for each gradient, the localization heatmap of the target image in the evaluation sample is determined by the target attention map and the gradient. Since there are two descriptive texts, the localization and recognition result of the visual language model (i.e., the target object with the larger image-text matching score) can be obtained through the image-text matching score. This yields two gradients, which are then used to calculate the localization heatmap of the target image in the evaluation sample, denoted as H. The calculation formula is as follows: H = ReLU(A) z ☉G z )

[0043] In the above formula, ⊙ represents element-wise multiplication, and A Z It is a target attention map, G ZIt represents the gradient. ReLU is a commonly used ReLU activation function in fully connected layers of neural networks.

[0044] For example, see Figure 3 As shown, Figure 3 This is a schematic diagram of the location heatmap of the target object provided by the present invention, such as... Figure 3 As shown in (b), a black-and-white dog is marked with a yellow bounding box in the target image. Based on the descriptive text "black-and-white dog," the visual language model identifies the black-and-white dog in the target image and generates a [image / description] at the dog's location. Figure 3 The location heatmap shown in (a) is shown in the image. According to... Figure 3 As shown in (d), the dog behind the flowerpot in the target image is marked with a yellow bounding box. Based on the descriptive text "pot behind dog," the visual language model identifies the dog behind the flowerpot in the target image and generates an image at the dog's location. Figure 3 The location heatmap shown in (c) is shown in the middle.

[0045] Step 103: Determine the performance evaluation score of the evaluation sample based on the location heatmap.

[0046] After calculating the location heatmaps (denoted as H0 and H1) of the target objects corresponding to the two descriptive texts in step 102, the next step is to determine the performance evaluation score of the evaluation sample based on the location heatmaps. The specific process for determining the performance evaluation score is described below.

[0047] First, a mask image for the target object is determined. This is achieved by masking the pixel values ​​of the target object using a binary mask. Pixel values ​​within the bounding box of the target object are set to a mask value of 1, while those outside the bounding box are set to a mask value of 0. By masking all pixels in the target image, the corresponding mask image is obtained. Since there are two target objects in the target image, two separate mask images are obtained, denoted as M0 and M1. In the location heatmap, each pixel value is any value within the range of 0 to 1.

[0048] Next, based on the location heatmap and mask image, the average activation value of the evaluation sample is determined. Specifically, the average activation value of the current evaluation sample is calculated based on the location heatmaps H0 and H1 and the mask images M0 and M1, denoted as act(M). i H j The calculation formula is as follows:

[0049] In the above formula, ⊙ represents element-wise multiplication. When i equals j, it means the mask bounding box matches the description text, and in this case, the average activation value within the bounding box should be high. When i and j are not equal, it means the mask bounding box does not match the description text, and in this case, the average activation value within the bounding box should be low.

[0050] Since the localization heatmap has two values, H0 and H1, and the mask image also has two values, M0 and M1, four average activation values ​​can be obtained through free composition: act(M0, H1), act(M1, H0), act(M0, H1), and act(M1, H1). Based on the preset activation value conditions satisfied by the average activation values, the performance evaluation score of the evaluation sample is determined. The activation value conditions are act(M0, H1) > act(M1, H0) and act(M0, H1) < act(M1, H1). Therefore, the performance evaluation score is denoted as f(H0, H1, M0, M1), and the calculation formula is as follows:

[0051] As can be seen from the above formula, the performance evaluation score is 0 or 1. The score of this evaluation method determines whether the target object in the descriptive text has a higher average activation value than another target object in the image, and whether the same applies to another target object and the descriptive text in the evaluation sample. In this way, it can be determined whether the visual language model has made different activation responses based on different descriptive texts, that is, whether it has understood the difference between different combinations of target objects.

[0052] Step 104: Determine the average performance evaluation score as the average accuracy of the evaluation dataset, and use the average accuracy as the performance evaluation result of the visual language model.

[0053] In step 103, a performance evaluation score can be calculated for each evaluation sample in the ARPGrounding dataset. Finally, the average performance evaluation score is determined as the average accuracy of the evaluation dataset, which can be denoted as acc. The average accuracy acc is used as the performance evaluation result of the visual language model.

[0054] The average performance evaluation score (acc) for the entire ARPGrounding dataset is calculated as follows:

[0055] In the above formula, N is the total number of evaluation samples in the ARPGrounding dataset.

[0056] In some embodiments, the performance evaluation results (average accuracy) of the visual language model are also obtained based on the corresponding three-dimensional sample sets, resulting in performance evaluation results for each of the three dimensions. Therefore, in this embodiment of the invention, the performance evaluation results of the visual language model include: attribute performance evaluation results, relation evaluation results, and primary / secondary performance evaluation results; wherein, the attribute performance evaluation results are obtained by the visual language model performing a localization task based on attribute differentiation using the attribute evaluation sample set; the relation performance evaluation results are obtained by the visual language model performing a localization task based on kinship differentiation using the relation evaluation sample set; and the primary / secondary performance evaluation results are obtained by the visual language model performing a localization task based on primary / secondary differentiation using the primary / secondary evaluation sample set. In subsequent processes, the corresponding performance evaluation results can be selected for experimental verification or comparison with other models based on the focus of the visual language model's capabilities.

[0057] To verify the technical effectiveness of the evaluation method, a verification experiment was also set up in this embodiment of the invention. The experiment evaluated the performance (performance evaluation results, i.e., average accuracy) of four state-of-the-art pre-trained visual language models on a standard visual localization dataset and the ARPGrounding dataset provided by this invention, using the stochastic performance of the SOTA algorithm on the corresponding dataset as a reference. The experimental results (i.e., attribute performance evaluation results, relation evaluation results, primary / secondary performance evaluation results, and the average of the three) show that the pre-trained visual language models perform excellently on the standard visual localization task, but the ARPGrounding dataset reveals their limitations. The performance of the pre-trained visual language models on the VG dataset, Flickr dataset, and ReferIt dataset can reach or surpass the stochastic performance of the SOTA algorithm, but their performance on the ARPGrounding dataset is only around the stochastic performance level. This demonstrates that the ARPGrounding dataset can effectively verify the combinatorial reasoning ability of visual language models, providing guidance for training the combinatorial reasoning ability of visual language models, and addressing the deficiency of poor performance of pre-trained visual language models in combinatorial reasoning ability in existing technologies.

[0058] In order to conform to the present invention Figure 1 The performance evaluation method proposed in this paper also includes fine-tuning of the pre-trained visual language model. The fine-tuning process first generates a text pair for each target image, where each descriptive text in the text pair describes a different target object. Then, a proxy task is introduced to increase the diversity of the localization heatmaps of these two descriptive texts. This fine-tuning process plays a crucial role in improving the visual language model's understanding of combinatorial semantics. The text pair generation method will be described in detail below, along with the proxy task used during the training of the visual language model.

[0059] First, training image samples and corresponding text pairs are obtained. In this embodiment of the invention, multiple candidate descriptive texts are generated based on the training image samples. The generation method can employ an image-text generation model. Then, the candidate descriptive texts are sampled. The process involves performing dependency structure parsing on the candidate descriptive texts to obtain a tree structure (dependency tree) of the candidate descriptive texts. The tree structure includes nodes and directed arcs connecting the nodes. Nodes represent each word in the candidate descriptive text, and directed arcs represent the connection relationships between each word.

[0060] For example, see Figure 4 , Figure 4 This is a schematic diagram of the text dependency parsing principle provided by the present invention. For example... Figure 4 As shown, dependency parsing is introduced during the sampling phase. The parsing method can use the spacy library to analyze the grammatical structure of the candidate description text to identify related words and determine the connections between words. For training image samples, all candidate description texts are parsed into a tree structure, where directed arcs represent the connections between each word in the candidate description text.

[0061] Next, the training target objects in the training image samples are determined based on the root node in the tree structure, and text pairs corresponding to the training image samples are constructed based on the root node corresponding to the training target objects. Here, the root node of the dependency tree is used to construct text pairs of different target objects, so that these texts contain different target objects with various attributes, relationships, and priorities, providing rich combinatorial semantics for subsequent training stages.

[0062] For example, such as Figure 4 As shown, for Figure 4 In (a), the corresponding candidate description text is "man wearing a red shirt". Figure 4 In (b), the corresponding candidate description text is "tall green clock". The candidate description texts "man wearing a red shirt" and "tall green clock", after dependency tree parsing, are as follows: Figure 4 As shown in (c), dependency tree parsing can distinguish nouns, verbs, adjectives, etc., in the descriptive text. Names generally represent the target object, adjectives generally represent the attributes of the target object, and there may be relationships or hierarchical order among multiple nouns. Here, the root node of the dependency tree generally represents the nouns in the descriptive text, that is, the target object. Based on the root node of the dependency tree (such as shirt, man, clock), the target object in the image can be determined, and thus the text pair corresponding to the image can be constructed.

[0063] After constructing training image samples and text pairs, for each training image sample, a visual language model is invoked to perform a proxy task, resulting in a training heatmap of the training image sample for the text pair. The proxy task involves identifying the training target object in the training image sample based on the text pair and constructing a training heatmap based on the training target object. The proxy task aims to introduce diversity into the heatmap generation process. This diversity is represented by simultaneously training the ability to distinguish attributes, relationships, and primary / secondary elements. The training heatmap is generated by simultaneously training these three dimensions.

[0064] The training heatmap represents the training results of the visual language model. Next, the loss function of the visual language model is constructed based on the training heatmap. Since it involves text pairs, two training heatmaps are generated, denoted as H0 and H1. The element-wise multiplication result H0⊙H1 of the training heatmap is calculated, and the area of ​​the training heatmap is calculated. The area is determined by the width w and height h of the training heatmap, i.e., w×h. Then, the ratio of the element-wise multiplication result to the area, H0⊙H1 / (w×h), is determined. Finally, the loss function L of the visual language model is constructed based on the expected value E(H0⊙H1 / (w×h)). The specific calculation formula is as follows:

[0065] In the above formula, B represents the training sample batch, B = {v} i , t i 0, t i 1} n i＝1 , where v i t represents the training image samples. i 0, t i 1 represents a text pair, where H0 and H1 are the pairs corresponding to t respectively. i 0, t i The training heatmap is 1, where w and h represent the width and height of the training heatmap, respectively.

[0066] During training, in each training sample batch B, backpropagation is performed in the visual language model using the loss function L, and the gradient optimization algorithm is used to optimize the gradient to update the weight parameters of the visual language model. After continuous iteration of each training sample batch, training of the visual language model stops when the loss function converges or when the specified number of iterations is reached.

[0067] The loss function L provided in this embodiment of the invention uses coarse-grained text descriptions rather than dense bounding box annotations to provide weakly supervised guidance for heatmaps. This allows the loss function L to guide the visual language model to generate different training heatmaps for different descriptive text inputs in text pairs. Furthermore, the dependency tree of the descriptive text covers various combinations of target objects, attributes, relationships, and priorities. Therefore, it encourages the output of the visual language model to be more discriminative, while mitigating noise caused by combinatorial ambiguity during the generation of training heatmaps.

[0068] To verify the effectiveness of the fine-tuning, this invention also applied the fine-tuning process provided by this invention to two visual language models with different architectures. Experiments verified the performance evaluation results among the CLIP model, the CLIP fine-tuning process, the ALBEF model, and the ALBEF fine-tuning process. The performance evaluation results (i.e., attribute performance evaluation results, relation evaluation results, primary / secondary performance evaluation results, and the average of the three) show that for two pre-trained visual language models with different model structures, the fine-tuning process of this invention can significantly improve the combinatorial reasoning ability of the pre-trained visual language models. Furthermore, in the visual localization dataset, the fine-tuning process significantly enhanced the performance of both visual language models.

[0069] In summary, this invention focuses on pre-trained visual language models, revealing their combinatorial reasoning problems. It proposes an attribute, relation, and priority grounding benchmark scheme to evaluate the combinatorial reasoning ability of pre-trained visual language models in visual localization tasks. Specifically, it constructs an ARPGrounding dataset containing 11,425 samples and evaluates the combinatorial understanding ability of pre-trained visual language models from three dimensions. Several mainstream pre-trained visual language models are evaluated using the ARPGrounding dataset. Experimental results show that these visual language models exhibit significant deficiencies in combinatorial reasoning. Furthermore, this invention proposes a fine-tuning process for combinatorial awareness, demonstrating how low-cost image-text annotation can improve the combinatorial understanding ability of pre-trained visual language models in localization tasks.

[0070] The performance evaluation device for the visual language model in the localization task provided by the present invention is described below. The performance evaluation device for the visual language model in the localization task described below can be referred to in correspondence with the performance evaluation method for the visual language model in the localization task described above.

[0071] Figure 5 This is a schematic diagram of the performance evaluation device for the visual language model in the localization task provided by the present invention, as shown below. Figure 5As shown, this invention provides a performance evaluation device for a visual language model in a localization task. The localization task includes a localization task based on attribute differentiation, a localization task based on relation differentiation, and a localization task based on primary / secondary differentiation. The device includes: a construction module 501, an execution module 502, and a determination module 503. The construction module 501 is used to construct an evaluation dataset for performance evaluation of the visual language model. The evaluation dataset includes multiple evaluation samples, each including a target image and two descriptive texts. The target image is marked with two target objects corresponding one-to-one with the descriptive texts. The execution module 502 is used to perform a localization task on each evaluation sample in the evaluation dataset using the visual language model, obtaining a localization heatmap of the target image in the evaluation sample. The localization task involves determining the corresponding target object from the target image based on the descriptive text, under the conditions of differentiating attributes, differentiating relations, and differentiating primary / secondary differentiation, respectively. The determination module 503 is used to determine the performance evaluation score of the evaluation sample based on the localization heatmap. The determination module 503 is also used to determine that the average performance evaluation score is the average accuracy of the evaluation dataset, and uses the average accuracy as the performance evaluation result of the visual language model.

[0072] Specifically, the performance evaluation device for the visual language model in the localization task provided by the present invention can implement all the method steps implemented in the above-mentioned performance evaluation method embodiment for the visual language model in the localization task, and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0073] Figure 6 This is a schematic diagram of the physical structure of an electronic device provided by the present invention, such as... Figure 6As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a performance evaluation method for a visual language model in a localization task. This method includes: constructing an evaluation dataset for performance evaluation of the visual language model, the evaluation dataset including multiple evaluation samples, each evaluation sample including a target image and two descriptive texts, the target image being marked with two target objects corresponding one-to-one with the descriptive texts; for each evaluation sample in the evaluation dataset, performing the localization task on the evaluation sample using the visual language model to obtain a localization heatmap of the target image in the evaluation sample, wherein the localization task involves determining the corresponding target object from the target image based on the descriptive text, under the conditions of distinguishing attributes, distinguishing relationships, and distinguishing primary and secondary attributes; determining the performance evaluation score of the evaluation sample based on the localization heatmap; determining the average of the performance evaluation scores as the average accuracy of the evaluation dataset, and using the average accuracy as the performance evaluation result of the visual language model.

[0074] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0075] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the performance evaluation method for the visual language model in the localization task provided by the above methods. The method includes: constructing an evaluation dataset for performance evaluation of the visual language model, the evaluation dataset including multiple evaluation samples, each evaluation sample including a target image and two descriptive texts, the target image being marked with two target objects corresponding one-to-one with the descriptive texts; for each evaluation sample in the evaluation dataset, performing the localization task on the evaluation sample through the visual language model to obtain a localization heatmap of the target image in the evaluation sample, wherein the localization task is to determine the corresponding target object from the target image based on the descriptive text under the conditions of distinguishing attributes, distinguishing relationships, and distinguishing primary and secondary attributes; determining the performance evaluation score of the evaluation sample based on the localization heatmap; determining the average of the performance evaluation scores as the average accuracy of the evaluation dataset, and using the average accuracy as the performance evaluation result of the visual language model.

[0076] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a method for evaluating the performance of a visual language model in a localization task provided by the methods described above. This method includes: constructing an evaluation dataset for performance evaluation of the visual language model, the evaluation dataset including multiple evaluation samples, each evaluation sample including a target image and two descriptive texts, the target image being labeled with two target objects corresponding one-to-one with the descriptive texts; for each evaluation sample in the evaluation dataset, performing the localization task on the evaluation sample using the visual language model to obtain a localization heatmap of the target image in the evaluation sample, wherein the localization task involves determining the corresponding target object from the target image based on the descriptive text, under the conditions of distinguishing attributes, distinguishing relationships, and distinguishing primary and secondary attributes; determining a performance evaluation score for the evaluation sample based on the localization heatmap; determining the average of the performance evaluation scores as the average accuracy of the evaluation dataset, and using the average accuracy as the performance evaluation result of the visual language model.

[0077] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0078] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0079] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for evaluating the performance of a visual language model in a localization task, characterized in that, The localization task includes attribute-based localization tasks, relationship-based localization tasks, and primary / secondary localization tasks, and the method includes: An evaluation dataset is constructed for performance evaluation of the visual language model. The evaluation dataset includes multiple evaluation samples. Each evaluation sample includes a target image and two descriptive texts. The target image is marked with two target objects that correspond one-to-one with the descriptive texts. For each evaluation sample in the evaluation dataset, the localization task is performed on the evaluation sample using the visual language model to obtain a localization heatmap of the target image in the evaluation sample. The localization task involves determining the corresponding target object from the target image based on the descriptive text, under the conditions of distinguishing attributes, distinguishing relationships, and distinguishing primary and secondary attributes. The step of performing the localization task on the evaluation sample using the visual language model to obtain a localization heatmap of the target image in the evaluation sample includes: Obtain a target attention map, wherein the target attention map is any intermediate attention map obtained in the visual language model during the execution of the localization task; Determine the image-text matching score between the target image and the descriptive text, and for each image-text matching score, determine the gradient of the target attention map relative to the image-text matching score; For each gradient, a localization heatmap of the target image in the evaluation sample is determined by the target attention map and the gradient; The performance evaluation score of the evaluation sample is determined based on the location heatmap, including: Determine the mask image of the target image for the target object; Based on the location heatmap and the mask map, the average activation value of the evaluation sample is determined; The performance evaluation score of the evaluation sample is determined based on the preset activation value conditions satisfied by the average activation value. The average of the performance evaluation scores is determined as the average accuracy of the evaluation dataset, and the average accuracy is used as the performance evaluation result of the visual language model.

2. The performance evaluation method for visual language models in localization tasks according to claim 1, characterized in that, The evaluation dataset includes an attribute evaluation sample set, a relationship evaluation sample set, and a primary and secondary evaluation sample set; The performance evaluation results of the visual language model include: attribute performance evaluation results, relation performance evaluation results, and primary / secondary performance evaluation results; The attribute performance evaluation result is obtained by the visual language model after performing the attribute-based localization task using the attribute evaluation sample set. The relation performance evaluation result is obtained by the visual language model after performing the localization task based on genus relation discrimination using the relation evaluation sample set; The primary and secondary performance evaluation results are obtained by the visual language model after performing the primary and secondary differentiation-based localization task using the primary and secondary evaluation sample set.

3. The performance evaluation method for visual language models in localization tasks according to claim 2, characterized in that, The two target objects in the evaluation samples of the attribute evaluation sample set are two objects with different attributes in the target image; the two target objects in the evaluation samples of the relationship evaluation sample set are two objects with different relationships in the target image, where the relationship is the association between the target object and other objects, and the other objects are objects in the target image other than the two target objects; the two target objects in the evaluation samples of the primary and secondary evaluation sample set are two objects in the target image with a primary and secondary order.

4. The performance evaluation method for the visual language model in the localization task according to claim 1, characterized in that, The visual language model is obtained by training it through performing a proxy task. The training process of the visual language model includes: Obtain training image samples and text pairs corresponding to the training image samples; For each training image sample, the visual language model is invoked to perform the proxy task to obtain a training heatmap of the training image sample for the text pair. The proxy task is to determine the training target object in the training image sample based on the text pair and to construct a training heatmap based on the training target object. The loss function of the visual language model is constructed based on the training heatmap, and backpropagation is performed on the visual language model through the loss function to update the parameters of the visual language model.

5. The performance evaluation method for the visual language model in the localization task according to claim 4, characterized in that, The step of obtaining the text pairs corresponding to the training image samples includes: Multiple candidate descriptive texts are determined based on the training image samples; Dependency structure parsing is performed on the candidate description text to obtain a tree structure of the candidate description text, wherein the tree structure includes nodes and directed arcs connecting the nodes, the nodes represent each word in the candidate description text, and the directed arcs represent the connection relationship between each word; The training target object in the training image sample is determined based on the root node in the tree structure, and a text pair corresponding to the training image sample is constructed based on the root node corresponding to the training target object.

6. The performance evaluation method for the visual language model in the localization task according to claim 4, characterized in that, The loss function for constructing the visual language model based on the training heatmap includes: Determine the element-wise multiplication result of the training heatmap and determine the area of ​​the training heatmap; Determine the ratio of the element-wise multiplication result to the area, and construct the loss function of the visual language model based on the expected value of the ratio.

7. A performance evaluation device for a visual language model in a localization task, characterized in that, The localization task includes location tasks based on attribute differentiation, location tasks based on relationship differentiation, and location tasks based on primary and secondary differentiation. The device includes: The construction module is used to construct an evaluation dataset for performance evaluation of the visual language model. The evaluation dataset includes multiple evaluation samples, each of which includes a target image and two descriptive texts. The target image is marked with two target objects that correspond one-to-one with the descriptive texts. An execution module is configured to perform the localization task on each evaluation sample in the evaluation dataset using the visual language model to obtain a localization heatmap of the target image in the evaluation sample. The localization task involves determining the corresponding target object from the target image based on the descriptive text, under the conditions of distinguishing attributes, distinguishing relationships, and distinguishing primary and secondary elements. The step of performing the localization task on the evaluation sample using the visual language model to obtain the localization heatmap of the target image in the evaluation sample includes: obtaining a target attention map, where the target attention map is any intermediate attention map obtained in the visual language model during the execution of the localization task; determining the image-text matching score between the target image and the descriptive text, and for each image-text matching score, determining the gradient of the target attention map relative to the image-text matching score; and for each gradient, determining the localization heatmap of the target image in the evaluation sample using the target attention map and the gradient. The determination module is used to determine the performance evaluation score of the evaluation sample based on the location heatmap, including: determining the mask map of the target image for the target object; determining the average activation value of the evaluation sample based on the location heatmap and the mask map; and determining the performance evaluation score of the evaluation sample according to the preset activation value conditions satisfied by the average activation value. The determining module is used to determine that the average value of the performance evaluation score is the average accuracy of the evaluation dataset, and to use the average accuracy as the performance evaluation result of the visual language model.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the performance evaluation method for the visual language model in the localization task as described in any one of claims 1 to 6.

Images