Historical cue calibration method and system for persistent vision-language retrieval and localization
Through the in-modal correlation estimation and intermodal consistency alignment of the historical prompt calibration method, the problems of task unfamiliarity and intermodal heterogeneity in continuous learning are solved, and the performance of the model in cross-modal tasks is improved. and learning efficiency, achieving better semantic mapping and alignment.
Patent Information
- Application Number
- CN202411248829.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-06
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-09-06
AI Technical Summary
Existing continuous learning techniques have problems with task unfamiliarity and inter-modal heterogeneity in multi-task and cross-modal tasks, resulting in poor adaptability and low learning efficiency in new tasks, and difficulty between visual and linguistic modalities. Establish consistent semantic mapping and alignment relationships.
Using the historical prompt calibration method, through correlation estimation within the modal and consistency alignment between modals, encourage the correlation of prompts to be with the current task, enhance the consistency of two modal prompts in the same task, and use historical prompts to perform cross-modal comparison. Learn, optimize cross-modal feature alignment.
The performance of the model in cross-modal tasks is improved, and the retention of task-specific information and cross-modal alignment consistency is significantly improved, and the performance is better than existing methods, especially in the overall performance of complex and diverse tasks.
Smart Images

Figure CN119046491A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of creative generative artificial intelligence, and in particular relates to a method and system for historical cue calibration for continuous visual-linguistic retrieval and localization. Background Art
[0002] In the field of machine learning, continuous learning is a crucial concept that enables models to adapt to non-stationary data distributions. However, existing continuous learning techniques have exposed some significant deficiencies when dealing with multi-task and cross-modal tasks, which are mainly reflected in the unfamiliarity of cross-tasks and heterogeneity between modalities. When a model faces a new task, it is often difficult to effectively transfer and apply the knowledge accumulated in previous tasks to the current task, especially when there are significant differences between the new task and the historical task. This unfamiliarity makes it impossible for the model to quickly adapt to the new task, resulting in low learning efficiency. In addition, in cross-modal tasks, due to the large differences in the representation of visual modalities (such as images or videos) and language modalities (such as text) in the feature space, it is difficult for existing models to establish consistent semantic mapping and alignment relationships between different modalities. This heterogeneity between modalities further exacerbates the challenges of the model in dealing with cross-modal tasks, making the model perform poorly in fusing and understanding multimodal information, seriously affecting the overall performance of the model in complex and diverse tasks. Summary of the invention
[0003] To solve the problems in the prior art, the present invention provides a method and system for calibrating historical prompts for continuous visual-language retrieval and localization. To alleviate the problem of task unfamiliarity, the present invention designs intra-modal relevance estimation to encourage prompts to be more closely associated with the current task and utilize prompts from past tasks; to bridge the gap between modalities, the present invention proposes inter-modal consistency alignment to enhance the consistency between two modal prompts in the same task.
[0004] In order to achieve the above object, the technical solution adopted by the present invention is as follows:
[0005] In a first aspect, the present invention discloses a method for calibration of history cues for continuous visual-linguistic retrieval and localization, comprising the following steps:
[0006] 1) Obtain a publicly available object detection dataset containing visual-language data, and set a retrieval or positioning task for the data based on the content information of the data in the dataset;
[0007] 2) constructing a retrieval and localization model for retrieving or localizing visual-linguistic data, the model comprising an encoder, an IRE module, an ICA module, and a head network;
[0008] 3) Preprocess the data with tasks in step 1) to obtain preprocessed data, input the preprocessed data into the encoder, IRE module and ICA module, the encoder outputs feature representation to the IRE module, ICA module and head network, the IRE module calculates the IRE loss, the ICA module calculates the ICA loss through contrast learning, the head network processes the feature representation, outputs the retrieval or positioning results, and calculates the basic loss;
[0009] 4) Update the parameters in the model based on IRE loss, ICA loss and base loss to obtain the trained model;
[0010] 5) Retrieve or locate visual-linguistic data to be retrieved or located using the pre-trained model; the visual-linguistic data includes visual data and language data, the visual data includes video or image, and the language data includes text.
[0011] In a second aspect, the present invention discloses a history prompt calibration system for continuous visual-linguistic retrieval and localization, comprising:
[0012] The data set acquisition and preprocessing module is used to acquire publicly available target detection data sets containing visual-language data, and to set retrieval or positioning tasks for the data according to the content information of the data in the data set;
[0013] A retrieval and localization model building module, which is used to build a retrieval and localization model for retrieving or localizing visual-language data, wherein the model includes an encoder, an IRE module, an ICA module, and a head network;
[0014] The model pre-training module is used to pre-process the data with the task to obtain the pre-processed data, input the pre-processed data into the encoder, IRE module and ICA module, the encoder outputs the feature representation to the IRE module, ICA module and head network, the IRE module calculates the IRE loss, the ICA module calculates the ICA loss through contrastive learning, the head network processes the feature representation, outputs the retrieval or positioning results, and calculates the basic loss;
[0015] Update the parameters in the model based on IRE loss, ICA loss and base loss to obtain the trained model;
[0016] The history hint calibration module is used to retrieve or locate the visual-language data to be retrieved or located using the pre-trained model; for image-text retrieval and video-text retrieval, the head network outputs a similarity score; for referential expression understanding, the head network outputs the upper-left and lower-right coordinates of the candidate box; for referential expression segmentation, the head network outputs a matrix consistent with the image size.
[0017] Compared with the prior art, the present invention has the following beneficial effects:
[0018] Through intra-modality relevance estimation, cues from past tasks are leveraged to make the cues more relevant to the current task. The technique involves developing a relevance estimator on top of the Transformer model to identify whether the input cue is relevant to the current task, where positive samples are learnable cues from the current task and negative samples are randomly drawn from frozen cues from past tasks. This goal helps improve the quality of cues because it explicitly requires that each task-specific group of cues be distinguished from the other groups, allowing them to retain a lot of task-specific information. Through inter-modality consistency alignment, the consistency of the two modal cues in the same task is enhanced by comparing the cues from the current task with those from historical tasks. We can further enhance cross-modal alignment by leveraging historical cues for comparison in a cross-modal manner. In summary, History Cue Calibration (HPC) demonstrates its superior performance over existing state-of-the-art methods in two visual-language retrieval tasks and two visual-language localization tasks. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Figure 1 It is the framework of the history prompt calibration system of the present invention. DETAILED DESCRIPTION
[0020] The present invention is further described and illustrated below in conjunction with specific embodiments. The embodiments are merely exemplary of the present disclosure and do not define the scope of limitation. The technical features of each embodiment of the present invention may be combined accordingly without conflicting with each other.
[0021] We design an intramodal relevance estimation task to help prompts encode more task-specific information by distinguishing previous prompts from historical tasks, which is achieved through a relevance estimator built on top of a pre-trained encoder. We develop a cross-modal consistency alignment training method to enhance cross-modal alignment by comparing features of one modality with features of another modality, where the comparison samples are generated by prompts from the current task and previous tasks.
[0022] First, the present invention provides a history cue calibration method for continuous visual-linguistic retrieval and localization, comprising the following steps:
[0023] 1) Obtain a suitable dataset for object detection that is publicly available and contains visual-linguistic data. The data will be extracted and prepared from the dataset, and the data includes both visual modality information and language modality information.
[0024] Object Detection Dataset D k Contains N kFor each data in the data set, a content information retrieval or positioning task T is set for the data. k , get data with tasks in, For tasks with T k The i-th data, For tasks with T k The language modality information of the i-th data, For tasks with T k The visual modality information of the i-th data, For tasks with T k The label of the i-th data, all tasks T k Constitute a task set T, T = {T1, T2, ..., T k ,...,T K}, k∈(1,K), K is the total number of tasks.
[0025] In a specific embodiment of the present invention, the visual-linguistic retrieval is image-text retrieval or video-text retrieval, and the visual-linguistic localization is referential expression understanding or referential expression segmentation.
[0026] For Tags In image-text retrieval tasks and video-text retrieval, All belong to the set {0,1}, 0 means there is no correspondence between the image (or video) and the text, and 1 means there is a correspondence between the image (or video) and the text; in the referential expression understanding task, Indicates the coordinates of the upper left corner (upper left vertex) and lower right corner (lower right vertex) of the candidate box formed by the information corresponding to the text in the image; for referential expression segmentation, Represents a matrix of the same size as the image. The values in the matrix are 0 or 1. Both the candidate box and the matrix are used to represent the position of the target object in the image.
[0027] 2) constructing a retrieval and localization model for retrieving or localizing visual-linguistic data, the model comprising an encoder, an IRE module, an ICA module, and a head network;
[0028] For image-text retrieval, the head network is a multi-layer perceptron; for video-text retrieval, the head network is an average pooling layer; for referential expression understanding and referential expression segmentation, the head network is a Transformer model, which is composed of a Transformer encoder, two Transformer decoders and a multi-layer perceptron connected in sequence;
[0029] The encoder includes a speech encoder φq (·) and a visual encoder φ v (·), both the language encoder and the visual encoder are Transformer models.
[0030] 3) Preprocess the data with tasks in step 1) to obtain preprocessed data, input the preprocessed data into the encoder, IRE module and ICA module, the encoder outputs feature representation to the IRE module, ICA module and head network, the IRE module calculates the IRE loss, the ICA module calculates the ICA loss through contrast learning, the head network processes the feature representation, outputs the retrieval or positioning results, and calculates the basic loss;
[0031] Specifically, the preprocessing of the data with tasks in step 1) includes:
[0032] In the task T k Data Insert a prompt and a special marker to get the preprocessed data in, is the preprocessed language modality information, is the preprocessed visual modality information, P k For tasks with T k Data Tips, P k = {P v,k ,P q,k}, P k ∈R L×D , L is the length of the prompt, L∈R, D is the embedding dimension of the prompt, D∈R, P v,k For tasks with T k The visual modality information prompt of the i-th data, P q,k For tasks with T k The language modality information prompt of the i-th data; [REL] is a special tag, which is a two-dimensional tensor; at the same time, a task-level prompt pool P is constructed, P = {P1, P2, ..., P k}, and input the task-level prompt pool P into the IRE module and the ICA module;
[0033] The preprocessed data Input to the encoder, which includes a language encoder φ q (·) and a visual encoder φ v (·), language encoder and visual encoder are both Transformer models; encoder output feature representation is the language feature representation output by the language encoder, is the visual feature representation output by the visual encoder,
[0034] Specifically, the IRE module calculates the IRE loss, including:
[0035] The IRE module receives the pre-processed data Task-level prompt pool P and feature representation The IRE module includes a linear layer, which represents the language features received at the current moment as language-related pairs, represents the visual features received at the current moment as visual-related pairs, and represents the language-related pairs as visual-related pairs. Input to the linear layer to get the logits of language related pairs; Input to the linear layer to obtain the logits of visually related pairs;
[0036] Randomly select the historical language modality information prompt P from the task-level prompt pool q,t Replace the prompt P in the preprocessed language modal information q,k , forming the first language feature representation, inputting the first language feature representation into the encoder for encoding, obtaining the second language feature representation, using the second language feature representation as a language-irrelevant pair, inputting the special mark of the language-irrelevant pair into the linear layer, obtaining the logits of the language-irrelevant pair, where t∈(1, k);
[0037] Randomly select the historical visual modality information prompt P from the task-level prompt pool v,a Replace the hint P in the preprocessed visual modality information v,k , forming a first visual feature representation, inputting the first visual feature representation into the encoder for encoding, obtaining a second visual feature representation, using the second visual feature representation as a visually irrelevant pair, inputting the special mark of the visually irrelevant pair into the linear layer, obtaining the logits of the visually irrelevant pair, where a∈(1, k);
[0038] IRE loss L IRE The IRE loss L is performed by binary cross entropy loss. IRE for:
[0039] L IrE =L IREv +L IREq
[0040] Among them, L IREv is the visual IRE loss; L IREq for language IRE loss;
[0041]
[0042] in, are the logits of visually related pairs; are the logits of visually unrelated pairs; are the logits of language related pairs; are the logits for language-unrelated pairs.
[0043] Specifically, the ICA module calculates the ICA loss through contrastive learning, including:
[0044] The ICA module receives the preprocessed data Task-level prompt pool P and feature representation
[0045] The present invention constructs negative instances by integrating historical prompts, that is, each historical visual modality information prompt in the task-level prompt pool replaces the prompt P in the preprocessed visual modality information in turn. v,k , forming l-1 first visual modality information Input l-1 first visual modality information into the encoder for encoding respectively, and obtain l-1 encoded visual feature representations (i.e. (k-1) negative instances);
[0046] Each historical language modality information prompt in the task-level prompt pool replaces the prompt P in the preprocessed language modality information in turn. q,k , forming k-1 first language modal information Input k-1 first language modal information into the encoder for encoding, and obtain k-1 encoded language feature representations (i.e. (k-1) negative instances);
[0047] Finally, 2×(l-1) negative instances are generated, and the negative instances are and Use contrastive learning to regularize the learning of the current cue.
[0048] This paper develops a contrastive alignment loss to bring each modality feature v closer k and q k distance between them, and also separate them from all other negative instances and On the language side, ICA loss of language modality information It can be expressed as:
[0049]
[0050] On the visual side, ICA loss of visual modality information It can be expressed as:
[0051]
[0052] Among them, τ is the temperature parameter, which is set to 0.1;
[0053] ICA Loss L ICA for
[0054] Specifically, the head network represents the feature Process and output the search or positioning results. k , H(*) is a function of the head network; when performing visual-linguistic retrieval, the output of the head network is a similarity score; when performing visual-linguistic localization, for referential expression understanding, the head network outputs the coordinates of the upper left vertex and the lower right vertex of the candidate box, the upper left vertex coordinates are the x-axis coordinate value of the upper left vertex and the y-axis coordinate value of the upper left vertex, and the lower right vertex coordinates are the x-axis coordinate value of the lower right vertex and the y-axis coordinate value of the lower right vertex; for referential expression segmentation, the head network outputs a matrix consistent with the image size, and the values in the matrix are greater than or equal to 0 and less than or equal to 1; both the candidate box and the matrix are used to represent the position of the target object in the image.
[0055] For image-text retrieval and video-text retrieval, the base loss L is calculated based on the cross entropy loss function. base ; For referential expression understanding, the basic loss L is calculated based on the mean absolute error function base ; For referential expression segmentation, the basic loss L is calculated based on the Dice loss function base .
[0056] 4) Update the parameters in the model based on IRE loss, ICA loss and base loss to obtain the trained model;
[0057] Specifically, based on the IRE loss L IRE 、ICA loss L ICA and the base loss L base Calculate the total loss L HPC , the total loss L HPC for:
[0058] L HPC =λ1L base +λ2L IRE +λ3L IcA
[0059] Among them, λ1, λ2 and λ3 are all loss parameters; λ1, λ2 and λ3 are set to 1.0, 0.1 and 0.1 respectively to balance the three losses.
[0060] When the current task is the initial task T1, L IRE =L ICA =0, based on the basic loss L base Update the parameters of the prompt and head network; when the current task is the second task T2, update the parameters of the linear layer, [REL], the prompt and the head network. n When n>2, update the parameters of the linear layer, the prompt and the head network.
[0061] This step is crucial for learning effective cue representations that contain rich task-specific information, because relevance estimation is also subject to the catastrophic forgetting problem, and continuous updating of labels or linear layers will lead to severe forgetting of previous relevance knowledge. Instead, the labels are fixed to retain the initial relevance information, and the linear layers are updated slowly to avoid rapid forgetting. In this process, the cue parameters of the current task can be gradually distinguished from the cues of the previous task, thereby becoming more task-specific. The training process of inter-modal consistency alignment is similar to that of intra-modal relevance estimation, starting from the second task because it requires the preparation of historical tasks. By using the same modality features with different task cues, the contrast samples constructed by the present invention can more effectively enhance the consistency of cross-modal features and ensure that this enhancement comes from the cues corresponding to the task, thereby calibrating the overall optimization direction of the cues.
[0062] It should be noted that the HPC paradigm of the present invention is only incorporated during the training process to ensure that it does not affect the inference speed or operating efficiency. After training, the model will have different outputs for different tasks. In the retrieval task, a similarity score is usually obtained to measure the degree of match between the input image (or video) and the text description. In the localization task, the output of the model is usually the position coordinates used to represent the location of the target object in the image.
[0063] 5) Using the pre-trained model to retrieve or locate the visual-language data to be retrieved or located.
[0064] Specifically, the visual-language data to be retrieved or located is input into the trained encoder, the encoder outputs feature representation to the trained head network, and the head network outputs the retrieval or positioning result. When performing visual-language retrieval, the output of the head network is a similarity score; when performing visual-language positioning, the output of the head network is a position coordinate, which is used to represent the position of the target object in the image.
[0065] Based on the visual feature representation and language feature representation generated by the encoder, the ICA module ensures feature alignment and consistency between different modalities by comparing the prompt features of the current task with the prompt features of the historical tasks. The ICA loss is calculated through contrastive learning, which will be used to optimize the cross-modal feature alignment, making the features of the visual and language modalities more consistent in the current task.
[0066] The downstream task head (head network) is a set of lightweight neural network layers that process feature representations according to the requirements of the current task (such as classification, retrieval, and positioning) and output the final task results. Based on the difference between the output of the head network and the true label, the basic loss (such as contrast loss, L1 loss, or Dice loss) is calculated. This loss will also be used to adjust the parameters of the model.
[0067] After all loss functions (including IRE loss, ICA loss, and base loss) are aggregated, each trainable parameter of the model is updated through back propagation. These parameters include special tags, parameters of linear layers, encoder parameters (if not frozen), and parameters of the head network. Finally, the model training is completed to achieve the function of historical prompt calibration.
[0068] The present invention has been fully experimentally verified on two cross-modal generation tasks.
[0069] In a specific embodiment of the present invention, an Adam optimizer and a cosine scheduler are used to set different training cycles, batch sizes, and learning rates.
[0070] Specifically, the present invention conducted experiments on four mainstream vision-language tasks, including two retrieval tasks and two localization tasks. For image-text retrieval, the present invention selected the MS-COCO dataset for evaluation, and divided it into 12 tasks according to content categories (such as people, vehicles, animals, etc.); for video-text retrieval, the present invention selected the MSR-VTT dataset for evaluation, and divided it into 10 tasks according to content categories (such as news, movies, sports, etc.); for referential expression understanding, the present invention selected the RefCOCO dataset for evaluation, and divided it into 12 tasks like MS-COCO; for referential expression segmentation, which is similar to the understanding task, the present invention selected the RefCOCO dataset for evaluation, and adopted the same task division method to divide it into 12 tasks.
[0071] The first two tasks focus on global-level alignment, while the latter two tasks are more challenging because they involve local-level positioning. To generate continuous learning tasks for image-text retrieval, the present invention uses the “category” annotations in the original dataset to define 12 tasks, as shown in Table 1. To generate continuous learning tasks for video-text retrieval, the present invention uses the “category” annotations in the original dataset to define 10 tasks, as shown in Table 2. The present invention evaluates referential expression understanding and segmentation through the box-level and mask-level annotations of this dataset. Similar to MS-COCO, the same division in Table 1 is adopted.
[0072] Table 1 Task division based on Coco dataset
[0073]
[0074] Table 2 Task division in the MSR-VTT dataset
[0075]
[0076] In order to objectively evaluate the performance of the present invention, the present invention adopts two widely used indicators, namely, average precision and forgetting rate. Specifically, for the two retrieval tasks, the present invention reports R@K (Recall at K) as the precision indicator, which calculates the percentage of correct results in the top K retrieval results of the query sample in the test sample. For the two localization tasks, the present invention calculates the precision (mIoU) by calculating the mean of the intersection over union (IoU) of each test sample, where IoU is the ratio of the intersection area to the union area.
[0077] For model selection, we follow a prompt-based process and select two powerful and widely used Transformer models as encoders: CLIP (ViT-B / 16) for retrieval tasks and GLIP-T for localization tasks. In CLIP, the prompt length is set to 16 and 12 for language and vision encoders, respectively. In GLIP, the prompt length is set to 12 and 20 for language and vision encoders, respectively. To train our model, we use the Adam optimizer with a cosine scheduler.
[0078] The proposed HPC strategy is compared with the following state-of-the-art continuous learning methods, which can be divided into three categories: (1) Replay-based methods. This includes the "replay head" strategy, which fine-tunes the sequence based on the frozen pre-trained model; the "replay all" strategy, which performs fine-tuning of the pre-trained model weights at the same time. (2) Regularization-based methods, including EWC and LwF. (3) Prompt-based methods, including DyTox, L2P, S-Prompts, DualPrompt, MaPLe, and DCP. Specifically, L2P and S-Prompts use a single encoder to extract features, while DualPrompt, MaPLe, and DCP consider the interaction between modalities in the feature extraction process.
[0079] According to the steps described in the specific implementation manner, the experimental results obtained are shown in Tables 3 to 6.
[0080] Table 3 Performance evaluation on image-text retrieval tasks
[0081]
[0082] Table 4 Performance evaluation on video-text retrieval task
[0083]
[0084]
[0085] Tables 3 and 4 show the results of two retrieval tasks. The bold numbers in Tables 3 and 4 represent the best results of the method without examples; the underlined numbers represent the second best results of the method without examples; the upper limit represents supervised fine-tuning on independent and identically distributed data of all tasks, which is usually regarded as the upper limit performance. The results show that the HPC method proposed in this paper significantly outperforms other sample-free methods such as S-Prompts and DualPrompt. HPC is slightly better than MaPLe and DCP, and uses fewer additional parameters, and the degree of forgetting of the HPC method is significantly lower than that of other methods. The performance of the playback-based method is limited. The present invention believes that the reason is that the limited data in each task limits retrieval learning because it requires a large number of contrast samples, which leads to model bias. This can explain why freezing the pre-trained model and adjusting only the head can lead to better performance. In contrast, the method of the present invention only fine-tunes the prompt parameters and avoids wasting the knowledge of the pre-trained model. The reason for the poor performance of other prompt-based methods may be that they rely on the weak transfer ability of prompt learning, especially between heterogeneous tasks and modalities.
[0086] Table 5 Performance evaluation of two vision-language localization tasks
[0087]
[0088]
[0089] Table 5 summarizes the results of two challenging localization tasks, and the bold numbers in Table 5 represent the best results of the method without examples; the underlined numbers represent the second best results of the method without examples; the upper limit represents supervised fine-tuning on independent and identically distributed data of all tasks, which is generally regarded as the upper limit performance. The HPC of the present invention still exceeds most sample-free methods. Unlike the retrieval task, the playback-based method is effective in the localization task, and tuning all model parameters can bring further improvement. However, these methods still require a large amount of memory space to store class-specific samples, and require a large number of parameters for tuning. In addition, the method of the present invention brings more significant improvements in the segmentation task than in the understanding task. The reason should be that the segmentation task is more complex and a complex downstream head is designed, which indirectly poses a greater challenge to continuous learning. This fact once again verifies the superiority of the method of the present invention.
[0090] Table 6 Ablation study of history hint calibration method
[0091]
[0092] We conducted an ablation study on the historical cue calibration method, and the results are shown in Table 6. First, we developed two overall ablations: HPC with intra-modality correlation estimation removed (w / o. IRE) and HPC with inter-modality consistency alignment removed (w / o. ICA). The results show that both calibration tasks can help improve the performance of the model in continuous tasks, showing that our method can effectively calibrate cues for pre-trained Transformers.
[0093] In addition, we further explore the detailed design by generating two ablations: HPC IRE without estimator (HPC(IRE w / o.estimator)) and HPC ICA without contrast (HPC(ICA w / o.contrastive)). HPC(IRE w / o.estimator) uses contrastive loss to expand the gap between different task prompt groups instead of using our estimator on the Transformer model. HPC(ICA w / o.contrastive) directly enhances cross-modal alignment without contrastive history cues. The results show that the design of our invention is effective because the estimator is able to leverage the discriminative power of the pre-trained model, while the contrastive loss is able to better achieve cross-modal alignment by leveraging historical information.
[0094] The present invention proposes Historical Prompt Calibration (HPC), and in two visual-language retrieval tasks and two visual-language localization tasks, HPC shows superior performance beyond the existing state-of-the-art methods. The above examples are only specific embodiments of the present invention. Obviously, the present invention is not limited to the above embodiments, and there may be many variations. All variations that can be directly derived or associated with the contents disclosed by a person of ordinary skill in the art from the present invention should be considered to be within the scope of protection of the present invention.
[0095] In a specific embodiment of the present invention, the present invention provides a history prompt calibration system for continuous visual-linguistic retrieval and localization, comprising:
[0096] The data set acquisition and preprocessing module is used to acquire publicly available target detection data sets containing visual-language data, and to set retrieval or positioning tasks for the data according to the content information of the data in the data set;
[0097] A retrieval and localization model building module, which is used to build a retrieval and localization model for retrieving or localizing visual-language data, wherein the model includes an encoder, an IRE module, an ICA module, and a head network;
[0098] The model pre-training module is used to pre-process the data with the task to obtain the pre-processed data, input the pre-processed data into the encoder, IRE module and ICA module, the encoder outputs the feature representation to the IRE module, ICA module and head network, the IRE module calculates the IRE loss, the ICA module calculates the ICA loss through contrastive learning, the head network processes the feature representation, outputs the retrieval or positioning results, and calculates the basic loss;
[0099] Update the parameters in the model based on IRE loss, ICA loss and base loss to obtain the trained model;
[0100] The history hint calibration module is used to retrieve or locate the visual-language data to be retrieved or located using the pre-trained model; for image-text retrieval and video-text retrieval, the head network outputs a similarity score; for referential expression understanding, the head network outputs the upper-left and lower-right coordinates of the candidate box; for referential expression segmentation, the head network outputs a matrix consistent with the image size.
[0101] The above-mentioned embodiments only express several implementation modes of the present invention, and the description is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. For those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.
Claims
1. A method for calibration of history cues for continuous visual-linguistic retrieval and localization, characterized in that The following steps are involved: 1) Obtain a publicly available object detection dataset containing visual-language data, and set a retrieval or positioning task for the data based on the content information of the data in the dataset; 2) constructing a retrieval and localization model for retrieving or localizing visual-linguistic data, the model comprising an encoder, an IRE module, an ICA module, and a head network; 3) Preprocess the data with tasks in step 1) to obtain preprocessed data, input the preprocessed data into the encoder, IRE module and ICA module, the encoder outputs feature representation to the IRE module, ICA module and head network, the IRE module calculates the IRE loss, the ICA module calculates the ICA loss through contrast learning, the head network processes the feature representation, outputs the retrieval or positioning results, and calculates the basic loss; 4) Update the parameters in the model based on IRE loss, ICA loss and base loss to obtain the trained model; 5) Using the pre-trained model to retrieve or locate the visual-language data to be retrieved or located.
2. The method according to claim 1, characterized in that Visual-language retrieval is image-text retrieval or video-text retrieval, and visual-language localization is referential expression understanding or referential expression segmentation.
3. The method according to claim 2, characterized in that In step 1), the target detection dataset D k Contains N k For each data in the data set, a content information retrieval or positioning task T is set for the data. k , get data with tasks in, For tasks with T k The i-th data, For tasks with T k The language modality information of the i-th data, For tasks with T k The visual modality information of the i-th data, For tasks with T k The label of the i-th data, all tasks T k Constitute a task set T, T = {T1, T2, ..., T k ,...,T K }, k∈(1,K), K is the total number of tasks.
4. The method according to claim 3, characterized in that: In step 3), the data with tasks in step 1) is preprocessed, including: In the task T k Data Insert a prompt and a special marker to get the preprocessed data in, is the preprocessed language modality information, is the preprocessed visual modality information, P k For tasks with T k Data Tips, P k = {P v,k ,P q,k }, P k ∈R L×D , L is the length of the prompt, L∈R, D is the embedding dimension of the prompt, D∈R, P v,k For tasks with T k The visual modality information prompt of the i-th data, P q,k For tasks with T k The language modality information prompt of the i-th data; [REL] is a special tag, which is a two-dimensional tensor; at the same time, a task-level prompt pool P is constructed, P = {P1, P2, ..., P k }, and input the task-level prompt pool P into the IRE module and the ICA module; The preprocessed data Input to the encoder, which includes a language encoder φ q (~) and a visual encoder φ v (·), the language encoder and the visual encoder are both Transformer models; the encoder output feature representation is the language feature representation output by the language encoder, is the visual feature representation output by the visual encoder, 5. The method according to claim 4, characterized in that In step 3), the IRE module calculates the IRE loss, including: The IRE module receives the pre-processed data Task-level prompt pool P and feature representation The IRE module includes a linear layer, which represents the language features received at the current moment as language-related pairs, represents the visual features received at the current moment as visual-related pairs, and represents the language-related pairs as visual-related pairs. Input to the linear layer to get the logits of language related pairs; Input to the linear layer to obtain the logits of visually related pairs; Randomly select the historical language modality information prompt P from the task-level prompt pool q,t Replace the prompt P in the preprocessed language modal information q,k , forming the first language feature representation, inputting the first language feature representation into the encoder for encoding, obtaining the second language feature representation, using the second language feature representation as a language-irrelevant pair, inputting the special mark of the language-irrelevant pair into the linear layer, obtaining the logits of the language-irrelevant pair, where t∈(1, k); Randomly select the historical visual modality information prompt P from the task-level prompt pool v,a Replace the hint P in the preprocessed visual modality information v,k , forming a first visual feature representation, inputting the first visual feature representation into the encoder for encoding, obtaining a second visual feature representation, using the second visual feature representation as a visually irrelevant pair, inputting the special mark of the visually irrelevant pair into the linear layer, obtaining the logits of the visually irrelevant pair, where a∈(1, k); The IRE loss L IRE for: L IRE =L IREv +L IREq Among them, L IREv is the visual IRE loss; L IREq for language IRE loss; in, are the logits of visually related pairs; are the logits of visually unrelated pairs; are the logits of language related pairs; are the logits for language-unrelated pairs.
6. The method according to claim 5, characterized in that In step 3), the ICA module calculates the ICA loss by contrastive learning, including: The ICA module receives the preprocessed data Task-level prompt pool P and feature representation Each historical visual modality information prompt in the task-level prompt pool replaces the prompt P in the preprocessed visual modality information in turn. v,k , forming k-1 first visual modality information Input k-1 first visual modality information into the encoder for encoding respectively, and obtain k-1 encoded visual feature representations Each historical language modality information prompt in the task-level prompt pool replaces the prompt P in the preprocessed language modality information in turn. q,k , forming k-1 first language modal information Input k-1 first language modal information into the encoder for encoding, and obtain k-1 encoded language feature representations The ICA loss L ICA for is the ICA loss of language modality information, is the ICA loss of visual modality information, Among them, τ is the temperature parameter.
7. The method according to claim 6, characterized in that In step 2), for image-text retrieval, the head network is a multi-layer perceptron; for video-text retrieval, the head network is an average pooling layer; for referential expression understanding and referential expression segmentation, the head network is a Transformer model, and the Transformer model is composed of a Transformer encoder, two Transformer decoders and a multi-layer perceptron connected in sequence; When performing visual-linguistic retrieval, the output of the head network is a similarity score; when performing visual-linguistic localization, for referential expression understanding, the head network outputs the upper-left and lower-right coordinates of the candidate box, and for referential expression segmentation, the head network outputs a matrix of the same size as the image.
8. The method according to claim 7, characterized in that In step 3), the head network represents the feature Process and output the search or positioning results. k , H(*) is the function of the head network; For image-text retrieval and video-text retrieval, the base loss L is calculated based on the cross entropy loss function. base ; For referential expression understanding, the basic loss L is calculated based on the mean absolute error function base ; For referential expression segmentation, the basic loss L is calculated based on the Dice loss function base .
9. The method according to claim 8, characterized in that The step 4) comprises: Based on IRE loss L IRE 、ICA loss L ICA and the base loss L base Calculate the total loss L HPC , the total loss L HPC for: L HPC =λ1L basr +λ2L IRE +λ3L ICA Among them, λ1, λ2 and λ3 are loss parameters; When the current task is the initial task T1, L IRE =L ICA =0, based on the basic loss L base Update the parameters of the prompt and head network; when the current task is the second task T2, update the parameters of the linear layer, [REL], the prompt and the head network. n When n>2, update the parameters of the linear layer, the prompt and the head network.
10. A history-cue calibration system for continuous visual-linguistic retrieval and localization, characterized in that include: The data set acquisition and preprocessing module is used to acquire publicly available target detection data sets containing visual-language data, and to set retrieval or positioning tasks for the data according to the content information of the data in the data set; A retrieval and localization model building module, which is used to build a retrieval and localization model for retrieving or localizing visual-language data, wherein the model includes an encoder, an IRE module, an ICA module, and a head network; The model pre-training module is used to pre-process the data with the task to obtain the pre-processed data, input the pre-processed data into the encoder, IRE module and ICA module, the encoder outputs the feature representation to the IRE module, ICA module and head network, the IRE module calculates the IRE loss, the ICA module calculates the ICA loss through contrastive learning, the head network processes the feature representation, outputs the retrieval or positioning results, and calculates the basic loss; Update the parameters in the model based on IRE loss, ICA loss and base loss to obtain the trained model; The history hint calibration module is used to retrieve or locate the visual-language data to be retrieved or located using the pre-trained model; for image-text retrieval and video-text retrieval, the head network outputs a similarity score; for referential expression understanding, the head network outputs the upper-left and lower-right coordinates of the candidate box; for referential expression segmentation, the head network outputs a matrix consistent with the image size.
Citation Information
Patent Citations
Cross-modal retrieval confrontation and defense method based on prompt learning
CN115658954A
Prompt learning method for modal interaction enhancement of visual language model
CN116503683A
Apparatus and method for task-adaptive neural network retrieval based on meta-contrastive learning
US20220366240A1
Systems and methods for vision-and-language representation learning
US20220391755A1
