A multimodal visual language understanding and positioning method, device, terminal and medium

Abstract

The present invention discloses a multimodal visual language understanding and positioning method, device, terminal and medium, comprising: constructing a visual language understanding and positioning model; processing unlabeled image data to generate multiple pseudo-label sources; learning a visual language understanding and positioning model for each source respectively; selecting all pseudo data sources step by step from easy to difficult according to the average number of entities in the expression text of each source based on a multi-source synchronous adaptive algorithm; performing sample instance-level reliability calculation on each paired data of the image and the pseudo-label based on the multi-source synchronous adaptive algorithm, obtaining an optimal sample subset through reliability evaluator selection, reliability threshold selection and a greedy algorithm, and learning an optimal visual language understanding and positioning model based on the optimal sample subset; the present invention proposes an unsupervised visual language understanding and positioning method, which can adapt a multimodal basic large model CLIP based on multi-source pseudo-labels to solve the problem of unsupervised visual positioning.

Description

Technical Field

[0001] The present invention relates to the field of unsupervised visual positioning technology, and in particular to a multimodal visual language understanding and positioning method, device, terminal and medium. Background Art

[0002] Visual grounding (VG), also known as reference expression comprehension (REC) or phrase grounding (PG), involves locating the bounding box region described by a textual expression within a specific image. This technology has become a key technology for AI-powered human-computer interaction. Due to the cross-modal nature of VG models, they require simultaneous understanding of both the language and the semantics of the image, making this a complex and challenging task.

[0003] Considering the complexity of the visual localization task, most existing methods focus on the fully supervised setting (i.e., using manually annotated triples of image-text-bounding box data as supervisory signals). However, in supervised visual localization, there are strict requirements on the quality of annotations. Specifically, the textual expression needs to meet the following requirements: first, it needs to be paired with the bounding box; second, it needs to refer to a unique object; and third, the semantics of the expression needs to be rich. In order to reduce the dependence on annotated data, weakly supervised settings (i.e., only given an image and text query pair, without a bounding box) and unsupervised visual localization (i.e., not using any task-related annotations to achieve the localization of the target area in the image) have recently attracted increasing attention.

[0004] Existing unsupervised visual localization methods mainly use pre-trained detectors and specific corpora to localize the referential text of unpaired data. The most advanced methods use hand-designed templates to match the results detected by off-the-shelf object and attribute detectors (e.g., "attribute-relation-noun", "Attr-Rela-Noun", "standingrightman"). In this way, pseudo-paired data of expression text and bounding boxes are generated and used as pseudo-annotations to learn visual language understanding and localization models in a supervised manner. However, the effectiveness of these pseudo-annotations in existing methods depends heavily on object or attribute detectors, which are always pre-trained on specific datasets, such as the MSCOCO dataset, which may limit the diversity of matching patterns and the richness of contextual semantics.

[0005] In the past few years, vision and language base models (e.g., CLIP model, i.e. Contrastive Language-Image Pre-Training) have achieved excellent results on many downstream tasks by adapting or prompting with a small amount of task-related data. The main advantage of these base models is that they can learn generalized knowledge from readily available network data with self-supervisory constraints. This can be considered to solve the problem of unsupervised visual localization using pre-trained models. However, due to the lack of task-related labeled data, this is a challenging task. A straightforward solution is to fine-tune the pre-trained model using pseudo labels generated in previous unsupervised visual localization methods. However, due to the difference between the pseudo labels and the corresponding task-specific ground truth labels, this will inevitably affect the generalization ability of the pre-trained model.

[0006] In fact, in addition to object or attribute detection, many other tasks, such as Scene Graph Generation and Image Caption, are closely related to visual language understanding and localization. Although these tasks have different annotations from visual localization, the prediction results of these models can be easily extended to pseudo-paired data expressing text and bounding boxes, which can help CLIP achieve adaptation in unsupervised visual localization problems. The off-the-shelf models learned in these tasks can easily expand the pseudo-label source for unsupervised visual localization, thereby providing zero-cost annotation information. Although these labels are zero-cost, they may also introduce noise that is harmful to VG.

[0007] Therefore, it is necessary to exploit the diversity brought by multi-source pseudo-labels on the one hand, and to mitigate the impact of noise on the other. Summary of the Invention

[0008] The technical problem to be solved by the present invention is that, in response to the defects of the existing technology, the present invention provides a multimodal visual language understanding and positioning method, device, terminal and medium to solve the technical problems of poor pseudo-label generalization ability and high noise in the existing unsupervised visual positioning method.

[0009] The technical solutions adopted by the present invention to solve the technical problems are as follows:

[0010] In a first aspect, the present invention provides a multimodal visual language understanding and positioning method, comprising:

[0011] Build a visual language understanding and positioning model;

[0012] Process unlabeled image data to generate multiple pseudo-label sources;

[0013] Learning a visual language understanding and localization model for each source separately, and using the learned model as a reliability evaluator; wherein the reliability evaluator is used to evaluate and obtain the specific source reliability and cross-source reliability of the sample;

[0014] According to the multi-source self-adaptive algorithm, the plurality of pseudo-label sources are sorted according to the average number of entities in the expression text of each source, and all pseudo-data sources are gradually selected from easy to difficult;

[0015] According to the multi-source adaptive algorithm, in each currently selected pseudo data source, sample instance-level reliability calculation is performed on each paired data of the image and pseudo label from the perspective of specific source reliability and cross-source reliability, and the optimal reliability evaluator and the optimal reliability threshold are selected in sequence by fixing the reliability threshold and fixing the reliability evaluator;

[0016] The current source pseudo sample data is filtered out in a greedy manner and added to the total sample subset, gradually expanding and obtaining the optimal sample subset, and learning the optimal visual language understanding and positioning model based on the optimal sample subset.

[0017] In one implementation, the processing of the unlabeled image data includes:

[0018] Define an unlabeled image dataset I and n pseudo triplet data from different sources

[0019] Among them, S i =(I,E i ,B i ), E i represents the set of pseudo expressions of the i-th source, B i Represents a collection of bounding boxes;

[0020] Define the test dataset D t =(I t ,E t ,B t ) and learning goal models;

[0021] Wherein, the learning target model is:

[0022]

[0023] F θ :(I,E)→B is based on D s Model;

[0024] l represents the loss function.

[0025] In one implementation, processing the unlabeled image data to generate multiple pseudo-label sources includes:

[0026] Processing the unlabeled image data through an object detector to generate a template expression pseudo label source;

[0027] Processing the unlabeled image data through a scene graph generator to generate a relational expression pseudo label source;

[0028] The unlabeled image data is processed by an image title generator to generate a title expression pseudo label source.

[0029] In one implementation, processing the unlabeled image data by the object detector to generate a template expression pseudo label source includes:

[0030] Generate several templates through the target detector and the attribute detector;

[0031] A vocabulary expression of the template expression pseudo-label source is generated according to several templates.

[0032] In one implementation, processing the unlabeled image data by the scene graph generator to generate a relational expression pseudo-label source includes:

[0033] The scene graph generation model uses the scene graph relationship as the expression text and the bounding box of the subject in the scene graph as the bounding box of the expression text to generate the relationship expression pseudo label source.

[0034] In one implementation, the processing of the unlabeled image data by the image title generator to generate a title expression pseudo-label source includes:

[0035] Parsing the generated pseudo-sentence subject through a language parser, and pairing the subject with the detection result of the image;

[0036] According to the pairing result, the bounding box detected by the detector is paired with the corresponding pseudo sentence to obtain the title expression pseudo label source.

[0037] In one implementation, the visual language understanding and positioning model includes: an image encoder, a text encoder, and a visual language cross-modal fusion positioning module.

[0038] In one implementation, learning the visual language understanding and localization model for each source separately includes:

[0039] Extracting and concatenating the layer features of the image encoder, and fusing the layer features into the feature size of the original single layer according to linear projection;

[0040] Input the language tag, visual tag, category tag and region tag into the visual language cross-modal fusion positioning module:

[0041]

[0042] in, is the language tag, is the visual label, [cls] is the category label generated by the image encoder, and p r Mark the area.

[0043] In one implementation, according to the multi-source adaptive algorithm, in each currently selected pseudo data source, sample instance-level reliability calculation is performed on each paired data of the image and the pseudo label based on the specific source reliability and the cross-source reliability, and the optimal reliability evaluator and the optimal reliability threshold are sequentially selected by fixing the reliability threshold and fixing the reliability evaluator, including:

[0044] According to the defined reliability threshold h0, a subset of pseudo samples is selected from the current data source;

[0045] Add the selected subset to D χ Perform model training and select the reliability evaluator M with the best performance on the validation set i* :

[0046]

[0047] Among them, D χ is the entire set of currently selected pseudo samples;

[0048] According to the reliability histogram H corresponding to the optimal reliability evaluator i*j* , from the current data source S j* Find the reliability threshold h with the best performance on the validation set * :

[0049]

[0050] Select the reliability value belonging to the reliability histogram H i*j* The interval [0,h * ] and add the selected pseudo samples to the entire sample subset D χ middle:

[0051] D χ =D χ ∪percent(H i*j* ,h * )

[0052] In one implementation, the method greedily filters out the current source pseudo sample data and adds it to the total sample subset, gradually expands and obtains the optimal sample subset, and learns the optimal visual language understanding and localization model based on the optimal sample subset, including:

[0053] A greedy sample selection algorithm based on binary search is used to solve the problem and obtain a set of final pseudo samples.

[0054] Based on pseudo samples Learn the final visual localization model F θ * :

[0055]

[0056] in, are pseudo samples selected by the greedy algorithm.

[0057] In a second aspect, the present invention provides a multimodal visual language understanding and positioning device, comprising:

[0058] A pseudo-label source generation module is used to process unlabeled image data and generate multiple pseudo-label sources;

[0059] a visual language understanding and positioning module, configured to select several sources from the plurality of pseudo-label sources as specific sources, and respectively learn visual language understanding and positioning models for the several specific sources;

[0060] A multi-source step-by-step adaptive algorithm module is used to gradually select pseudo data sources using the average number of entities in each expression text according to a multi-source step-by-step adaptive algorithm;

[0061] The calculation and learning module is used to calculate the sample instance-level reliability of each paired data of image and pseudo-label from the specific source reliability and cross-source reliability, and learn the visual language understanding and positioning model based on the calculation results.

[0062] In a third aspect, the present invention provides a terminal comprising: a processor and a memory, wherein the memory stores a multimodal visual language understanding and positioning program, and when the multimodal visual language understanding and positioning program is executed by the processor, it is used to implement the operation of the multimodal visual language understanding and positioning method as described in the first aspect.

[0063] In a fourth aspect, the present invention also provides a medium, which is a computer-readable storage medium, and the medium stores a multimodal visual language understanding and positioning program. When the multimodal visual language understanding and positioning program is executed by a processor, it is used to implement the operation of the multimodal visual language understanding and positioning method described in the first aspect.

[0064] The present invention adopts the above technical solution to achieve the following effects:

[0065] The present invention applies CLIP to an unsupervised visual language understanding and localization system, enabling the entire system to adapt to diverse pseudo data while retaining the generalization capability of the multimodal base large model CLIP. Furthermore, through specific source reliability and cross-source reliability, the deep distribution feature information of visual localization data can be mined, thereby enabling the model to achieve stronger generalization capabilities. Furthermore, through a greedy multi-source self-adaptive sample selection algorithm, paired data of images and pseudo labels can be screened to retrain the visual localization model. In the sample selection process at each step of the self-adaptive learning, an optimal balance can be found between reliability and diversity. The present invention can adapt the multimodal base large model CLIP based on multi-source pseudo labels to solve the problem of unsupervised visual localization. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0067] Figure 1 It is a flow chart of a multimodal visual language understanding and positioning method in one implementation of the present invention.

[0068] Figure 2 It is an adaptive schematic diagram of CLIP-based unsupervised visual language understanding and positioning tasks in one implementation of the present invention.

[0069] Figure 3 It is a schematic diagram of a multimodal visual language understanding and positioning model framework in one implementation of the present invention.

[0070] Figure 4 It is a schematic diagram of a visual language localization model in one implementation of the present invention.

[0071] Figure 5 It is a schematic diagram of a multi-source adaptive algorithm and its pseudo-label source, reliability evaluator and sample selection scheme in one implementation of the present invention.

[0072] Figure 6 It is a flow chart of a multi-source synchronous adaptive algorithm in one implementation of the present invention.

[0073] Figure 7 It is a flow chart of a single-source self-adaptive algorithm in one implementation of the present invention.

[0074] Figure 8 It is a functional principle diagram of a terminal in one implementation of the present invention.

[0075] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0076] In order to make the purpose, technical solutions and advantages of the present invention more clear and distinct, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0077] Exemplary Methods

[0078] Unsupervised visual grounding can learn and locate the region of the corresponding description referred to by the input text expression without using any task-related manual annotation information, aiming to alleviate the heavy reliance on manually annotated data. However, the pseudo-annotations generated by existing methods mainly rely on object or attribute detectors, which may limit the diversity of matching patterns and the richness of contextual semantics.

[0079] In order to utilize the pre-trained basic large model and rationally utilize the multi-source pseudo-labels generated by the off-the-shelf models learned in different tasks, this embodiment proposes a novel unsupervised multimodal visual language understanding and localization system and method based on self-paced curriculum learning. This system and method can adapt the multimodal basic large model CLIP with multi-source pseudo-labels, thereby solving the problem of unsupervised visual language understanding and localization. For unlabeled image data, this embodiment first generates multiple pseudo-label sources through different off-the-shelf models, and then proposes a multi-source adaptive unsupervised visual language understanding and localization method. In a self-paced manner, samples with more reliable pseudo-labels are gradually found, and finally a balance between the reliability and diversity of the selected pseudo-samples is achieved. The method in this embodiment significantly surpasses the existing state-of-the-art unsupervised visual language understanding and localization methods, and even outperforms existing weakly supervised methods.

[0080] like Figure 1 As shown, an embodiment of the present invention provides a multimodal visual language understanding and positioning method, comprising the following steps:

[0081] Step S100: constructing a visual language understanding and positioning model.

[0082] In this embodiment, the multimodal visual language understanding and positioning method is applied to a terminal, which includes but is not limited to: a computer and other devices.

[0083] like Figure 2As shown, in this embodiment, a new unsupervised visual language understanding and localization method is proposed. The method is implemented through the CLIP-VG framework, which uses multi-source pseudo labels in a self-paced manner to achieve CLIP's adaptation in the unsupervised visual language understanding and localization tasks.

[0084] In this embodiment, the CLIP-VG framework diagram is as follows: Figure 3 The method proposed in this embodiment includes a Visual Language Grounding Model (VLGM), a Multi-source Pseudo Labels Generation Module (MPGM), and a Multi-source Self-paced Adapting (MSA) module, which includes three important selection modules.

[0085] The method in this embodiment can adapt CLIP based on multi-source pseudo labels to solve the problem of unsupervised visual positioning. First, a visual language understanding and positioning model VLGM is constructed. In view of the diversity of different source information of pseudo labels, a CLIP-VG model is proposed. The model uses CLIP's image and text encoders as the visual and language backbone of the framework, thereby improving the generalization ability of various label sources. CLIP-VG consists of two encoders of CLIP and a visual language cross-modal fusion positioning module Transformer, that is, the visual language understanding and positioning model includes: image encoder, text encoder and visual language cross-modal fusion positioning module. The visual language understanding and positioning model can realize the understanding and positioning of the expression text during training and inference. In order to better mine visual scale information, this embodiment proposes to extract the intermediate layer features of the CLIP encoder, thereby realizing multi-level perception of visual representation.

[0086] After constructing the visual language understanding and localization model (VLGM), this embodiment uses different types of off-the-shelf models, namely, an object detector, a scene graph generator, and an image caption generator, to generate multiple pseudo-label sources for unlabeled image data. Then, this embodiment first independently learns multiple source-specific visual language understanding and localization models, with CLIP serving as the backbone of each pseudo-data source model. Next, this embodiment uses multi-source self-paced learning (SPL) to gradually find samples with more reliable pseudo-labels to improve the performance of the visual localization model. Specifically, in the step of executing the SPL algorithm, this embodiment gradually selects pseudo data sources from simple to complex based on the average number of entities per expression text. At the same time, within the data sources selected in each step of the SPL algorithm, this embodiment proposes to calculate the sample instance-level reliability of each paired image and pseudo-label data from two aspects: source-specific reliability and cross-source reliability.

[0087] The reliability of a specific source is calculated by the probability that the visual localization model learned from the current label source correctly predicts the label of the current source. In contrast, cross-source reliability is calculated by the probability that the visual localization model learned from other label sources correctly predicts the label of the current source. In order to effectively select a subset of pseudo-paired data to retrain the visual localization model, this embodiment designs a greedy sample selection algorithm based on binary search to achieve the best balance between reliability and diversity. The entire method can gradually learn visual language understanding and localization models using pseudo-paired data from different sources under the paradigm of curriculum learning from easy to difficult.

[0088] Visual grounding (VG) refers to locating the bounding box region described by a textual expression within a specific image. Both fully supervised and weakly supervised VG models rely heavily on costly manual annotation. The method proposed in this example is unsupervised, meaning no task-specific manual annotations are used during training. Therefore, this example requires defining the task.

[0089] like Figure 1 As shown, an embodiment of the present invention provides a multimodal visual language understanding and positioning method, comprising the following steps:

[0090] Step S200: Process the unlabeled image data to generate multiple pseudo label sources.

[0091] In one implementation of this embodiment, the following steps are included before step S200:

[0092] Step S001: define an unlabeled image dataset I and n pseudo triplet data from different sources

[0093] Step S002: define the test data set D t =(I t ,E t ,B t ) and learning goal models.

[0094] In this embodiment, we first define I as an unlabeled image dataset. Assume that there are n pseudo triplet data from different sources, denoted as Among them, S i =(I,E i ,B i ), E i represents the set of pseudo expressions from the i-th source, B i Denotes the set of bounding boxes. Note that pseudo labels from different sources have different distribution characteristics. The test dataset is defined as D t =(I t ,E t ,B t ). The goal of this embodiment is to learn a D-based s Model F θ :(I,E)→B, so that it can generalize well to the test data D t :

[0095]

[0096] Among them, l represents the loss function, which uses SmoothL1 loss and Giou loss with λ coefficient to measure the distance between the predicted box and the pseudo bounding box:

[0097] l=L smooth-l1 (F θ (I,E i ),B i )+λ·L giou (F θ (I,E i ),B i ). (2)

[0098] In this embodiment, during the training phase, for unlabeled images, the MPGM module uses an existing model to generate different sets of pseudo-labels to construct pseudo-triplets, which serve as pseudo-supervisory signals for training. The CLIP model then undergoes self-paced adaptation, progressively selecting reliable pseudo-paired data from different label sources based on their reliability, dynamically expanding the training data set. Through this self-paced curriculum learning paradigm, an optimal visual-language localization model (VLGM) is ultimately learned.

[0099] Specifically, in one implementation of this embodiment, step S100 includes the following steps:

[0100] Step S201 : Processing the unlabeled image data by the object detector to generate a template expression pseudo label source.

[0101] In one implementation of this embodiment, step S201 includes the following steps:

[0102] Step S201a, generating several templates through the object detector and attribute detector;

[0103] Step S201b: generating a vocabulary expression of the template expression pseudo-tag source according to several templates.

[0104] In this example, the MPGM module uses three off-the-shelf models: a detector, a scene graph generator, and an image descriptor. These models provide three pseudo-label sources: template representation, relation representation, and title representation, thereby constructing pseudo-triplets for visual localization. The template representation is based on the Pseudo-Q model, where representations are generated from 11 templates whose vocabulary and bounding boxes are derived from object detectors and attribute detectors pre-trained on the Visual Genome dataset.

[0105] Step S202, processing the unlabeled image data by the scene graph generator to generate a relational expression pseudo label source;

[0106] In one implementation of this embodiment, step S202 includes the following steps:

[0107] Step S202a: Using the scene graph generation model, the scene graph relationship is used as the expression text, and the bounding box of the subject in the scene graph is used as the bounding box of the expression text to generate the relationship expression pseudo label source.

[0108] In this example, relational expressions are phrases with a (for example, subject-verb-object) structure generated by RelTR, a scene graph generation model pre-trained on the Visual Genome dataset. In this example, scene graph relations are used as the expression text, and the bounding box of the subject in the scene graph is used as the bounding box of the expression text to construct pseudo-label pairs. In this example, the top 10 scene relations from RelTR are selected for each image.

[0109] Step S203: Process the unlabeled image data through the image title generator to generate a title expression pseudo label source.

[0110] In one implementation of this embodiment, step S203 includes the following steps:

[0111] Step S203a, parsing the generated pseudo-sentence subject through a language parser, and pairing the subject with the image detection result;

[0112] Step S203b: Pair the bounding box detected by the detector with the corresponding pseudo sentence according to the pairing result to obtain the title expression pseudo label source.

[0113] In this embodiment, the caption expression is a pseudo-sentence generated by the Image Caption model M2 or ClipCap (M2 is used for the RefCOCO / + / g dataset, and ClipCap is used for the ReferIt Game and Flickr datasets). In this embodiment, a language parser (e.g., spaCy) is used to parse the subject of the generated pseudo-sentence and pair it with the image detection results. If the subject of the pseudo-sentence matches the detected label, the bounding box detected by the detector can be paired with the corresponding pseudo-sentence to obtain a pseudo-label pair.

[0114] like Figure 1 As shown, in one implementation of the embodiment of the present invention, the multimodal visual language understanding and positioning method further includes the following steps:

[0115] Step S300 , learning the visual language understanding and positioning model of each source respectively, and using the learned model as a reliability evaluator.

[0116] In one implementation of this embodiment, step S300 includes the following steps:

[0117] Step S301, extracting and concatenating the layer features of the image encoder, and fusing the layer features into the feature size of the original single layer according to linear projection;

[0118] Step S302: inputting language tags, visual tags, category tags, and region tags into the visual language cross-modal fusion positioning module.

[0119] In this embodiment, the network structure of the proposed visual language localization system CLIP-VG is shown in Table 1 below:

[0120]

[0121] The schematic diagram of CLIP-VG is as follows Figure 4As shown. In this embodiment, the image and text encoders of CLIP are used as the backbone of vision and language, and the generalization ability of the model is improved by adapting to multi-source pseudo labels. In order to better explore scale information, this embodiment proposes to extract the intermediate layer features of the CLIP encoder to achieve multi-level perception of visual representation. Specifically, the [1, 4, 8, 12] layer features of the image encoder are extracted and spliced, and then fused into the feature size of the original single layer using linear projection. The detailed network structure of CLIP-VG is shown in Table 1. In order to prevent catastrophic forgetting, the parameters of the CLIP encoder are frozen during training in this embodiment, so that only a small number of parameters need to be adapted. The order of tokens input to the cross-modal Transformer is:

[0122]

[0123] in, CLIP language token (i.e. language tag), is the CLIP visual token (i.e., visual tag), [cls] represents the category token (i.e., category tag) generated by the CLIP image encoder, and p r The Region token (i.e., region label) of the final output is used to output the regression bounding box. The [Reg] token is randomly initialized at the beginning of training and optimized with the entire model.

[0124] In this embodiment, reliability and diversity are used to describe the characteristics of samples required by the model. Reliability represents the reliability of the sample, and diversity represents the richness of the semantic information contained in the sample subset.

[0125] Pseudo-labels from different sources have different distribution characteristics and may also be noisy. First, at the source level, different label sources have different distributions due to the specific matching pattern or quality when generating pseudo-labels. Second, different pseudo-labels from the same source also have differences in the reliability of visual localization model learning. In order to make full use of pseudo-labels, it is necessary to find more reliable data, while also avoiding losing too much data diversity, so as to learn a visual localization model with good generalization ability. To address these issues, this embodiment first introduces three definitions and calculation methods of reliability from the source level and instance level.

[0126] A difficulty measurer is one of the two core components of traditional curriculum learning, used to assess the relative ease or difficulty of each example. In unimodal tasks, difficulty is typically measured using pre-defined, quantifiable criteria, such as sentence length and POS entropy in natural language processing (NLP) or the number of objects in computer vision (CV). However, due to the correlation between cross-modal data, the difficulty of visual localization examples cannot be directly assessed.

[0127] In order to obtain the specific reliability of multimodal data paired with images and pseudo labels, this embodiment defines a set of initial visual localization models as reliability measures. Each model is learned from a specific pseudo data source:

[0128]

[0129] Then, the reliability of a single sample can be obtained, which is defined as follows:

[0130] r=1-IOU(M i (i,e),b) (5)

[0131] Among them, i, e, b represents a triplet of data image, expression text, and target candidate box.

[0132] Thus, the reliability of all samples in a specific pseudo data source can be calculated as follows:

[0133] R ij =1-IOU(M i (I,E j ),B j ),i∈[1,n],j∈[1,n], (6)

[0134] Among them, R ij is the i-th reliability evaluator M i The set of reliability values obtained from all samples of the jth data source. When i=j, R ij is the source-specific reliability. When i≠j, R ij IOU is a metric function that calculates the Jaccard overlap between the predicted bounding box and the pseudo bounding box of each sample.

[0135] It is worth noting that in this embodiment, reliability is defined as R ij =1-IOU(Mi (I,E j ),B j ) instead of using IOU directly. The main consideration is to facilitate the selection of pseudo data. With this definition, pseudo data can be selected by incrementally increasing the reliability value from 0.0 to 1.0 based on the reliability histogram.

[0136] Due to the quality and semantic characteristics of pseudo-labels, different sources have different distribution characteristics, and the same reliability evaluator has different recognition capabilities for different sources. On the one hand, the model learned on the current source can capture the main characteristics of the pseudo-paired data in that labeled source, allowing source-specific reliability to be used to select more reliable data. On the other hand, due to the difference between pseudo-labels and ground-truth labels, the model obtained on the current source can easily deviate from the ideal model, which may also affect the effectiveness of data selection. By considering cross-source reliability, models learned on other pseudo-label sources can be used to guide the selection of samples in the current labeled source, thereby selecting more general pseudo-data. Therefore, it is necessary to simultaneously utilize source-specific reliability and cross-source reliability during self-paced curriculum learning to improve the model's generalization ability.

[0137] In order to facilitate the data selection in the self-paced learning process, this embodiment is based on R ij A reliability histogram H is defined for each pseudo data source ij The reliability histogram has m bins, covering the entire reliability range, i.e. [0, 1.0]. Each bin represents the number of samples within the corresponding reliability value interval.

[0138] like Figure 1 As shown, in one implementation of the embodiment of the present invention, the multimodal visual language understanding and positioning method further includes the following steps:

[0139] Step S400 , using a multi-source adaptive algorithm, sorting the plurality of pseudo label sources according to the average number of entities in the expression text of each source, and gradually selecting all pseudo data sources from easy to difficult.

[0140] In this embodiment, a Multi-source Self-paced Adapting (MSA) algorithm is proposed to collect reliable pseudo samples and carefully select samples in a simple to complex curriculum learning paradigm. The process of the MSA algorithm is as follows: Figure 5 As shown, the formula of MSA algorithm is as follows Figure 6 shown.

[0141] Specifically, in one implementation of this embodiment, the process of selecting all pseudo data sources includes:

[0142] First, the average number of entities per expression in each label source is calculated, and the calculated average number of entities is used as the evaluation criterion of the difficulty of the source level;

[0143] Secondly, the label sources are sorted from simple to complex according to the evaluation criteria, and the pseudo data source is obtained by gradually adding them according to the sorting.

[0144] In this example, for the single-source fully supervised visual-language localization task, it is straightforward to find the optimal model that best generalizes to the target data. However, for multi-source pseudo-labeling, this problem is not easily solved due to the differences between different label sources. To ensure that the model can achieve robust visual localization capabilities under multi-source noisy data, this example uses a self-paced curriculum learning (SPL) approach to absorb knowledge from different sources.

[0145] In each step of SPL, it is necessary to decide which label source to use to retrain the VG model. Therefore, this embodiment proposes to calculate the average number of entities per expression in each label source as the evaluation criterion for the difficulty of the source level, so that the label sources can be sorted from simple to complex. Assume that the data source selected in the current SPL step is S j* Then, in each step of SPL, a new label source can be gradually added from simple to complex to learn the VG model, and the selection of pseudo label sources can be completed in this way.

[0146] like Figure 1 As shown, in one implementation of the embodiment of the present invention, the multimodal visual language understanding and positioning method further includes the following steps:

[0147] In step S500, according to the multi-source adaptive algorithm, in each currently selected pseudo data source, sample instance-level reliability calculation is performed on each paired data of the image and the pseudo label from the perspective of specific source reliability and cross-source reliability, and the optimal reliability evaluator and the optimal reliability threshold are selected in sequence by fixing the reliability threshold and fixing the reliability evaluator.

[0148] In this embodiment, due to the different quality and semantic features of the labels, different pseudo-label sources contain different feature distributions. Therefore, different reliability evaluators learned from different pseudo-label sources also have different discrimination capabilities for specific different sources. When selecting a reliability evaluator, different reliability evaluators can be used as the data source S. j* Obtain multiple different reliabilities (i.e., ). Therefore, it is necessary to select an optimal reliability evaluator for the data source learned in the current SPL step to select pseudo samples.

[0149] Specifically, in one implementation of this embodiment, step S500 includes the following steps:

[0150] Step S501, selecting a subset of pseudo samples from the current data source according to the defined reliability threshold h0;

[0151] Step S502: Add the selected subset to D χ Perform model training and select the reliability evaluator M with the best performance on the validation set i* .

[0152] In this embodiment, we first define a reliability threshold h0 and use it to select a subset of pseudo samples from the current data source. Specifically, we define percent(H ij* ,h0) as the basis of H ij* From the j * A subset extracted from a data source, where each extracted sample belongs to the reliability histogram H ij* In the reliability value interval [0,h0]. The number of samples in this subset can be described by the mathematical formula as follows:

[0153]

[0154] In particular, when h0=0 / 1, all data are not selected / selected.

[0155] Then, by adding the selected subset to D χ Perform model training and select the best performing reliability evaluator M on the validation set i* The details are as follows:

[0156]

[0157] Among them, D χ is the entire set of pseudo samples selected before the current SPL step, which is an empty set at initialization.

[0158] In determining the optimal reliability evaluator M i* Afterwards, according to the corresponding reliability histogram H i*j* , it is necessary to further extract the current data source S j* Select pseudo samples (i.e. greedy samples) in .

[0159] Specifically, in one implementation of this embodiment, step S500 further includes the following steps:

[0160] Step S503: According to the reliability histogram H corresponding to the optimal reliability evaluator i*j* , from the current data source S j* Find the reliability threshold h with the best performance on the validation set * ;

[0161] Step S504: Select the reliability value that belongs to the reliability histogram H i*j* The interval [0,h * ] and add the selected pseudo samples to the entire sample subset D χ middle.

[0162] In this embodiment, we need to find the reliability threshold h with the best performance on the validation set. * :

[0163]

[0164] Then, the histogram H corresponding to the optimal reliability evaluator is selected i*j* The interval [0,h * ] on the pseudo sample. Finally, the selected pseudo sample is added to the entire sample subset D χ , as shown below:

[0165] D χ =D χ ∪percent(H i*j* ,h * ) (10)

[0166] It is worth noting that if we solve formula (8) directly, the cost is unbearable. Therefore, a greedy sample selection algorithm based on binary search is proposed.

[0167] like Figure 1 As shown, in one implementation of the embodiment of the present invention, the multimodal visual language understanding and positioning method further includes the following steps:

[0168] In step S600, the current source pseudo sample data is filtered out in a greedy manner and added to the total sample subset, and the optimal sample subset is gradually expanded to obtain the optimal sample subset, and the optimal visual language understanding and positioning model is learned based on the optimal sample subset.

[0169] In one implementation of this embodiment, step S600 includes the following steps:

[0170] Step S601: A greedy sample selection algorithm based on binary search is used to solve the problem and obtain a set of final pseudo samples.

[0171] Step S602: Based on the pseudo sample Learn the final visual localization model Fθ * .

[0172] This embodiment proposes a greedy sample selection algorithm based on binary search. Specifically, h r 、h m and h l Defined as three temporary reliability thresholds. At the same time, h m Initialized to the initial value h m =h0, and fix h r =h m +Δ, and h l =h m -Δ.

[0173] Then, by trying different h m The value is greedily solved in formula (8). By continuously updating h m =h r or h l , until h m Reach h r and h l Better performance. Based on the idea of binary search, we can quickly find the appropriate reliability threshold for suboptimal performance, thereby reducing the cost of model training and ensuring that the selected pseudo samples achieve a balance between reliability and diversity. At the end of the self-paced learning, a final set of pseudo samples will be obtained. It can be used to learn the final visual positioning model F θ * :

[0174]

[0175] The method in this embodiment is also applicable to the scenario of single-source full supervision. In this case, MSA degenerates into single-source synchronous adaptation (SSA), as shown in Figure 7 As shown in the figure, compared to MSA, SSA lacks cross-source reliability. It only requires a single-source model to perform a self-paced reliability assessment on the training data and then find an appropriate reliability threshold. Note that due to the higher reliability of manual annotations, h0 is set to 0.8 in the single-source fully supervised scenario. Future research on fully supervised visual localization can draw on this plug-in algorithm to further improve the performance of the original model.

[0176] This embodiment achieves the following technical effects through the above technical solution:

[0177] This embodiment applies CLIP to an unsupervised visual language understanding and localization system, enabling the entire system to adapt to diverse pseudo data while retaining the generalization capability of the multimodal base large model CLIP. Furthermore, through specific source reliability and cross-source reliability, the deep distribution feature information of visual localization data can be mined, thereby enabling the model to achieve stronger generalization capabilities. Furthermore, through a greedy multi-source self-adaptive sample selection algorithm, paired data of images and pseudo labels can be screened to retrain the visual localization model. In the sample selection process at each step of self-adaptive learning, an optimal balance between reliability and diversity can be found. This embodiment can adapt the multimodal base large model CLIP based on multi-source pseudo labels to solve the problem of unsupervised visual localization.

[0178] Exemplary devices

[0179] Based on the above embodiments, the present invention further provides a multimodal visual language understanding and positioning device, comprising:

[0180] A pseudo-label source generation module is used to process unlabeled image data and generate multiple pseudo-label sources;

[0181] a visual language understanding and positioning module, configured to select several sources from the plurality of pseudo-label sources as specific sources, and respectively learn visual language understanding and positioning models for the several specific sources;

[0182] A multi-source step-by-step adaptive algorithm module is used to gradually select pseudo data sources using the average number of entities in each expression text according to a multi-source step-by-step adaptive algorithm;

[0183] The calculation and learning module is used to calculate the sample instance-level reliability of each paired data of image and pseudo-label from the specific source reliability and cross-source reliability, and learn the visual language understanding and positioning model based on the calculation results.

[0184] Based on the above embodiment, the present invention further provides a terminal, whose principle block diagram can be shown as follows: Figure 8 shown.

[0185] The terminal includes: a processor, memory, interface, display screen and communication module connected via a system bus; wherein the processor of the terminal is used to provide computing and control capabilities; the memory of the terminal includes a storage medium and an internal memory; the storage medium stores an operating system and computer programs; the internal memory provides an environment for the operation of the operating system and computer programs in the storage medium; the interface is used to connect to external devices, such as mobile terminals and computers; the display screen is used to display corresponding information; and the communication module is used to communicate with a cloud server or a mobile terminal.

[0186] When the computer program is executed by a processor, it is used to implement the operation of a multimodal visual language understanding and positioning method.

[0187] It will be understood by those skilled in the art that Figure 8 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the solution of the present invention, and does not constitute a limitation on the terminal to which the solution of the present invention is applied. The specific terminal may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0188] In one embodiment, a terminal is provided, which includes: a processor and a memory, wherein the memory stores a multimodal visual language understanding and positioning program, and when the multimodal visual language understanding and positioning program is executed by the processor, it is used to implement the operation of the above multimodal visual language understanding and positioning method.

[0189] In one embodiment, a storage medium is provided, wherein the storage medium stores a multimodal visual language understanding and positioning program, and when the multimodal visual language understanding and positioning program is executed by a processor, it is used to implement the operation of the above multimodal visual language understanding and positioning method.

[0190] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware through a computer program. The computer program can be stored in a non-volatile storage medium. When executed, the computer program can include the processes in the above-described method embodiments. Any reference to memory, storage, database, or other media used in the various embodiments provided herein may include non-volatile and / or volatile memory.

[0191] In summary, the present invention provides a multimodal visual language understanding and positioning method, device, terminal and medium, the method comprising: constructing a visual language understanding and positioning model; processing unlabeled image data to generate multiple pseudo-label sources; learning the visual language understanding and positioning model of each source respectively; according to a multi-source synchronous adaptive algorithm, gradually selecting all pseudo data sources from easy to difficult according to the average number of entities in the expression text of each source; according to the multi-source synchronous adaptive algorithm, performing sample instance-level reliability calculation on each paired data of the image and the pseudo-label, obtaining the optimal sample subset through reliability evaluator selection, reliability threshold selection and greedy algorithm, and learning the optimal visual language understanding and positioning model based on the optimal sample subset; the present invention proposes an unsupervised visual language understanding and positioning method, which can adapt the multimodal basic large model CLIP based on multi-source pseudo-labels to solve the problem of unsupervised visual positioning.

[0192] It should be understood that the application of the present invention is not limited to the above examples. For those skilled in the art, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A multimodal visual language understanding and localization method, characterized in that: include: Build a visual language understanding and positioning model; Process unlabeled image data to generate multiple pseudo-label sources; Learning a visual language understanding and localization model for each pseudo-label source separately, and using the learned model as a reliability evaluator; wherein the reliability evaluator is used to evaluate and obtain the specific source reliability and cross-source reliability of the sample; According to the multi-source self-adaptive algorithm, the plurality of pseudo label sources are sorted according to the average number of entities in the expression text of each pseudo label source, and all pseudo data sources are gradually selected from easy to difficult; According to the multi-source adaptive algorithm, in each currently selected pseudo data source, sample instance-level reliability calculation is performed on each paired data of the image and pseudo label from the perspective of specific source reliability and cross-source reliability, and the optimal reliability evaluator and the optimal reliability threshold are selected in sequence by fixing the reliability threshold and fixing the reliability evaluator; The current source pseudo sample data is filtered out in a greedy manner and added to the total sample subset, gradually expanding and obtaining the optimal sample subset. The optimal visual language understanding and positioning model is learned based on the optimal sample subset. The processing of unlabeled image data to generate multiple pseudo-label sources includes: Processing the unlabeled image data through an object detector to generate a template expression pseudo label source; Processing the unlabeled image data through a scene graph generator to generate a relational expression pseudo label source; The unlabeled image data is processed by an image title generator to generate a title expression pseudo label source.

2. The multimodal visual language understanding and positioning method according to claim 1, characterized in that: The processing of the unlabeled image data previously includes: Defining an unlabeled image dataset and Pseudo triplet data from different sources ; in, , Indicates the The set of pseudo-expressions of the source, Represents a collection of bounding boxes; Define the test dataset and learning goal models; Wherein, the learning target model is: Based on Model; represents the loss function.

3. The multimodal visual language understanding and positioning method according to claim 1, characterized in that: The processing of the unlabeled image data by the target detector to generate a template expression pseudo label source includes: Generate several templates through the target detector and the attribute detector; A vocabulary expression of the template expression pseudo-label source is generated according to several templates.

4. The multimodal visual language understanding and positioning method according to claim 1, characterized in that: The step of processing the unlabeled image data by the scene graph generator to generate a relational expression pseudo label source includes: The scene graph generator uses the scene graph relationship as the expression text and the bounding box of the subject in the scene graph as the bounding box of the expression text to generate the relationship expression pseudo label source.

5. The multimodal visual language understanding and positioning method according to claim 1, characterized in that: The processing of the unlabeled image data by the image title generator to generate a title expression pseudo-label source includes: Parsing the generated pseudo-sentence subject through a language parser, and pairing the subject with the detection result of the image; According to the pairing result, the bounding box detected by the detector is paired with the corresponding pseudo sentence to obtain the title expression pseudo label source.

6. The multimodal visual language understanding and positioning method according to claim 1, characterized in that: The visual language understanding and positioning model includes: an image encoder, a text encoder, and a visual language cross-modal fusion positioning module.

7. The multimodal visual language understanding and positioning method according to claim 6, characterized in that: The visual language understanding and localization model for each pseudo-label source is learned separately, including: Extracting and concatenating the layer features of the image encoder, and fusing the layer features into the feature size of the original single layer according to linear projection; Input the language tag, visual tag, category tag and region tag into the visual language cross-modal fusion positioning module: in, is the language tag, For the visual mark, The category label generated by the image encoder, Mark the area.

8. The multimodal visual language understanding and positioning method according to claim 1, characterized in that: The method greedily filters out the current source pseudo sample data and adds it to the total sample subset, gradually expands and obtains the optimal sample subset, and learns the optimal visual language understanding and positioning model based on the optimal sample subset, including: A greedy sample selection algorithm based on binary search is used to solve the problem and obtain a set of final pseudo samples. ; Based on pseudo samples Learning the final visual localization model : in, are pseudo samples selected by the greedy algorithm.

9. A multimodal visual language understanding and positioning device, used to implement the multimodal visual language understanding and positioning method according to any one of claims 1 to 8, characterized in that: include: A pseudo-label source generation module is used to process unlabeled image data and generate multiple pseudo-label sources; a visual language understanding and positioning module, configured to select several sources from the plurality of pseudo-label sources as specific sources, and respectively learn visual language understanding and positioning models for the several specific sources; A multi-source step-by-step adaptive algorithm module is used to gradually select pseudo data sources using the average number of entities in each expression text according to a multi-source step-by-step adaptive algorithm; The calculation and learning module is used to calculate the sample instance-level reliability of each paired data of image and pseudo-label from the specific source reliability and cross-source reliability, and learn the visual language understanding and positioning model based on the calculation results.

10. A terminal, characterized in that: include: A processor and a memory, wherein the memory stores a multimodal visual language understanding and positioning program, and when the multimodal visual language understanding and positioning program is executed by the processor, it is used to implement the operation of the multimodal visual language understanding and positioning method as described in any one of claims 1-8.

11. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, which stores a multimodal visual language understanding and positioning program. When the multimodal visual language understanding and positioning program is executed by a processor, it is used to implement the operation of the multimodal visual language understanding and positioning method as described in any one of claims 1 to 8.

Images