Cyclic converter and history-aware based visual dialog method, device and equipment

Abstract

The application discloses a visual dialogue method, device and equipment based on a cycle converter and history perception, and the method comprises the following steps: acquiring a target image, a title corresponding to the target image and a dialogue history of the target image; the dialogue history comprises multiple dialogue rounds, each dialogue round comprises a question and an answer corresponding to the question; image features of the target image are extracted; the title and each dialogue round are encoded round by round to obtain dialogue round features; the target image and the title are matched with each dialogue round according to the image features and the dialogue round features, so that each candidate dialogue round matched with the target image and the title is obtained; and a target dialogue round with a correct answer in each candidate dialogue round is determined. The application can improve the time dependence and accuracy of visual dialogue, and can be widely applied in the field of artificial intelligence neural network models.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence neural network models, and in particular to a visual dialogue method, device and equipment based on a recurrent transformer and history perception. BACKGROUND

[0002] Visual dialogue tasks require dialogue agents to answer a series of image-based questions. Visual dialogue not only needs to understand image semantics according to text, but also needs to reason through dialogue history. Existing technologies mostly focus on developing various attention mechanisms to establish interaction models between entities including images, captions, questions and dialogue history. In order to obtain the powerful representation capability of pre-trained visual language transformers, existing visual dialogue models such as VisDial-BERT, VD-BERT, ICMU and UTC adapt various pre-trained visual language models to achieve new performance records. VisDial-BERT is a BERT-based model pre-trained on visual language datasets and then fine-tuned on the VisDial dataset. VD-BERT uses a single-stream transformer encoder to capture all interaction relationships between images and multi-turn dialogues. UTC proposes a contrastive learning-based framework to unify and facilitate the discrimination and generation tasks in visual dialogue. ICMU strengthens cross-modal understanding through four-way contrastive learning. AlignVD pre-trains visual and dialogue encoders through unsupervised and pseudo-supervised visual-language alignment losses.

[0003] Inspired by the success of large-scale pre-trained language models such as BERT in various downstream natural language processing tasks, there are currently many studies on extending BERT to improve the performance of downstream visual-language tasks by pre-training models on large-scale image-text pairs, such as ViLBERT, VL-BERT and BLIP, etc. However, these methods mostly use general large-scale images and text pairs scraped from the web, lacking alignment capability between images and dialogue history. In addition, these methods are not designed for visual dialogue, so it is not ideal to apply them to visual dialogue tasks to capture temporal dependencies. Previous methods such as BLIP learn from the entire dialogue roughly, only concatenating all utterances in the dialogue history as dialogue history and interacting with visual information directly through cross-attention or multi-modal transformers. This means that these works hope to extract enough information to answer the current question through a one-time multi-modal interaction. However, existing technologies ignore the sequential nature of multi-turn dialogue systems and the inherent temporal dependencies in dialogue history, which is a clear feature of multi-turn dialogue, and the temporal dependencies of multi-modal information learned by agents to accurately predict responses are crucial. Therefore, existing technologies are prone to overlook detailed information in the dialogue history and cannot give accurate answers.

[0004] The existing pre-trained visual dialogue model has the following problems in the visual dialogue task: the time dependence of the visual dialogue is low; and the coordinated semantics between specific features of different modalities are difficult, resulting in that the visual dialogue is difficult to meet the multi-modal interaction and understanding. SUMMARY

[0005] Therefore, the embodiment of the present application provides a visual dialogue method and device based on a cycle transformer and historical perception to improve the time dependence and accuracy of the visual dialogue.

[0006] An aspect of the embodiment of the present application provides a visual dialogue method based on a cycle transformer and historical perception, comprising:

[0007] obtaining a target image, a title corresponding to the target image, and a dialogue history of the target image; the dialogue history comprises a plurality of dialogue turns, and each dialogue turn comprises a question and an answer corresponding to the question;

[0008] extracting image features of the target image;

[0009] encoding the title and each dialogue turn round by round to obtain dialogue turn features;

[0010] matching the target image and the title with each dialogue turn according to the image features and the dialogue turn features to obtain each candidate dialogue turn matched with the target image and the title;

[0011] determining a target dialogue turn with a correct answer in each candidate dialogue turn.

[0012] Optionally, the extracting the image features of the target image comprises:

[0013] adjusting the size of the target image and dividing the target image with the adjusted size into a plurality of tokens;

[0014] inputting the plurality of tokens into a pre-set image encoder to obtain the image features.

[0015] Optionally, the encoding the title and each dialogue turn round by round to obtain dialogue turn features comprises:

[0016] labeling all words in the title and all words in each dialogue turn to obtain a labeled sequence;

[0017] adding a code representing the start of encoding at the beginning of the labeled sequence and adding a code representing the end of encoding at the end of the labeled sequence;

[0018] encoding the mark sequence round by round to obtain the dialogue turn features.

[0019] Optionally, the encoding the mark sequence round by round to obtain the dialogue turn features comprises:

[0020] inputting the mark sequence into a pre-configured recurrent dialogue encoder to obtain the dialogue turn features;

[0021] The expression of the recurrent dialogue encoder is:

[0022]

[0023] wherein H t represents the dialogue turn features, is the input of the recurrent dialogue encoder, is the input of the cross-attention module in the recurrent dialogue encoder, wherein F I is the image feature, is the dialogue history feature from the 0th round to the t-1th round.

[0024] Optionally, the matching the target image and the title with each dialogue turn according to the image feature and the dialogue turn features to obtain each candidate dialogue turn matched with the target image and the title comprises:

[0025] determining a first similarity of each dialogue turn feature to the image feature and a second similarity of the image feature to each dialogue turn feature;

[0026] performing weighted summation on the first similarity and the second similarity to obtain a contrastive loss similarity;

[0027] matching the target image and the title with each dialogue turn according to the contrastive loss similarity to obtain each candidate dialogue turn matched with the target image and the title.

[0028] Optionally, the determining a first similarity of each dialogue turn feature to the image feature and a second similarity of the image feature to each dialogue turn feature comprises:

[0029] determining a first similarity of each dialogue turn feature to the image feature according to a first expression;

[0030] The first expression is:

[0031]

[0032] wherein H represents a first marked feature in the dialog turn feature, I represents a first marked feature in the image feature, and τ is a temperature parameter, is a set of image features that do not match H;

[0033] determining a second similarity of the image feature to each of the dialog turn features according to a second expression;

[0034] The second expression is:

[0035]

[0036] wherein, is a set of dialog turn features that do not match I.

[0037] Optionally, the determining of the target dialog turn that answers correctly in each of the candidate dialog turns comprises:

[0038] adding a preset loss function to the contrast loss similarity to obtain a total loss function;

[0039] determining the target dialog turn that answers correctly in each of the candidate dialog turns according to the total loss function;

[0040] The expression of the preset loss function is:

[0041]

[0042] wherein, represents a label of a dialog question and answer in the i-th turn of the sample, the value of the label is 1 if the answer is correct, otherwise it is 0, and the sample i includes a ground truth question and answer pair of visual dialog and a randomly sampled negative question and answer pair; represents a probability that the i-th turn of the sample is predicted as a correct answer, and N represents the total number of the sample i.

[0043] Another aspect of the embodiment of the present application also provides a visual dialog device based on a cycle converter and historical perception, comprising:

[0044] A first unit is configured to acquire a target image, a title corresponding to the target image, and a dialog history of the target image, wherein the dialog history comprises a plurality of dialog turns, and each dialog turn comprises a question and an answer corresponding to the question.

[0045] A second unit is configured to extract an image feature of the target image.

[0046] A third unit is configured to perform turn-by-turn encoding on the title and each of the dialog turns to obtain a dialog turn feature.

[0047] a fourth unit configured to match each of the dialogue turns with the target image and the title according to the image feature and the dialogue turn feature, to obtain each candidate dialogue turn matched with the target image and the title;

[0048] a fifth unit configured to determine a target dialogue turn with correct answers in each of the candidate dialogue turns.

[0049] Another aspect of the embodiment of the present application further provides an electronic device comprising a processor and a memory;

[0050] The memory is configured to store a program.

[0051] The processor executes the program to implement the visual dialogue method based on the cycle converter and the history perception.

[0052] Another aspect of the embodiment of the present application further provides a computer readable storage medium, the storage medium stores a program, and the program is executed by a processor to implement the visual dialogue method based on the cycle converter and the history perception.

[0053] The embodiment of the present application further discloses a computer program product or a computer program, which comprises computer instructions stored in a computer readable storage medium. The processor of the electronic device can read the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the electronic device to perform the foregoing method.

[0054] The present application extracts the image feature of the target image, and then encodes the title and each dialogue turn to obtain the dialogue turn feature. Then, each candidate dialogue turn matched with the target image and the title is matched according to the image feature and the dialogue turn feature, and a target dialogue turn with correct answers is determined therefrom. The dialogue history, i.e., the title and each dialogue turn, is encoded successively to better capture the dependency in time, and then different modalities are aligned in the fine-tuning stage of the visual dialogue, i.e., the candidate dialogue turns are matched, and thus the accuracy of the visual dialogue can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort.

[0056] Figure 1 A flowchart of a visual dialogue method based on a cycle converter and history perception provided by the embodiment of the present application is shown.

[0057] Figure 2 An architecture example diagram of a RecFormer provided for an embodiment of the present application is shown in FIG. 1.

[0058] Figure 3 A specific example diagram of a visual dialogue provided for an embodiment of the present application is shown in FIG. 2.

[0059] Figure 4 A comparative example diagram of a visual dialogue provided for an embodiment of the present application is shown in FIG. 3.

[0060] Figure 5 A structure block diagram of a visual dialogue device based on a recurrent transformer and history perception provided for an embodiment of the present application is shown in FIG. 4. DETAILED DESCRIPTION

[0061] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0062] It should be noted that although the functional modules are divided in the device schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a manner different from the module division in the device or the order in the flowchart.

[0063] The terms "first", "second", and the like in the specification and claims and the above-described drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. Also, the terms "include", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, method, article or device including the element.

[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0065] In order to overcome the defect that the time dependence of the visual dialogue in the prior art is low, the present application can be established on a pre-trained visual-language converter, the time dependence of the dialogue is better captured by encoding the dialogue history one by one, and the different modalities are adjusted in the fine-tuning stage, and then the agent can pay more attention to important information, which is crucial for distinguishing whether the answer is consistent with the current question according to the dialogue history, the image and the title.

[0066] In order to overcome the defect that the coordination semantics between the specific features of different modalities in the prior art is difficult, resulting in that the visual dialogue is difficult to meet the multi-modal interaction and understanding, the present application proposes a recurrent multi-modal converter RecFormer on the basis of the visual language pre-training, the time dependence is better captured by encoding the dialogue history one by one, and the history-aware contrastive learning is proposed to align the different modalities in the fine-tuning stage of the visual dialogue. The recurrent multi-modal converter RecFormer pays more attention to the fine-grained dialogue history of each round, so it can capture the dialogue history more related to the current question.

[0067] In order to solve the above problems, the present application proposes a recurrent multi-modal converter (Recurrent Transformer, named RecFormer), which captures the time dependence between the corpora by encoding the dialogue corpus and interacting with the visual information round by round. Specifically, the present application provides a recurrent function for the pre-trained converter to maintain the cross-modal history encoding for the dialogue agent. Therefore, the dialogue agent can make better predictions by considering the time dependence. In addition, the present application also proposes a history-aware contrastive learning as an auxiliary task to keep the image features and dialogue history features consistent to improve the understanding of the visual dialogue.

[0068] Next, the present application is specifically described. Referring to Figure 1 The embodiment of the present application provides a visual dialogue method based on a recurrent converter and history awareness, which comprises steps S100 to S140, and specifically as follows:

[0069] S100: acquiring a target image, a title corresponding to the target image and a dialogue history of the target image; the dialogue history comprises a plurality of dialogue rounds, and each dialogue round comprises a question and an answer corresponding to the question.

[0070] Specifically, the visual dialogue can be a series of dialogues for a given target image and a given title of the target image. The historical dialogue is to ask a plurality of questions according to the target image and the title, then answer the questions in the visual dialogue, and then get the answers. Optionally, each dialogue round in the embodiment of the present application can include a question and an answer.

[0071] For the convenience of description, the target image is referred to as image hereinafter.

[0072] Next, the RecFormer proposed by the present application is described, which is a recurrent multimodal transformer with history-aware contrastive learning for visual dialogue.

[0073] First, the task definition of visual dialogue is introduced: given an image I, a caption C, a dialogue history H and a follow-up question Q, the visual dialogue task requires the dialogue agent to give an answer A to the question Q in natural language. To achieve this goal, the present application learns a neural visual dialogue model by ranking candidate answers, as follows: given an image I and a dialogue history H = (C, (Q1, A1), (Q2, A2), …, (Q t-1 ,A t-1 )), a follow-up question Q t , and a list of 100 candidate answers , the neural visual dialogue model is required to return the ranking of A t . The present application can obtain a neural visual dialogue model, the input of which is “given an image I and a dialogue history H = (C, (Q1, A1), (Q2, A2), …, (Q t-1 ,A t-1 )), a follow-up question Q t , and a list of 100 candidate answers ”, and the output is “the neural visual dialogue model is required to return the ranking of A t ”.

[0074] Referring to Figure 2 , the present application provides an example diagram of the architecture of the RecFormer. The RecFormer can include two parts: an image encoder and a recurrent dialogue encoder. The image encoder is responsible for extracting image features. Unlike most existing technologies, which treat all dialogue history as long text and encode it once, the dialogue encoder of the present application encodes the dialogue history and interacts with the image features and the dialogue features of the previous round through a cross-attention mechanism in a round-by-round manner. The multimodal dialogue features will be passed to a simple linear layer to distinguish whether the current dialogue round matches the dialogue history, image and caption. In addition, a history-aware multimodal contrastive loss is deployed, which adjusts the different modal features involved by calculating the interaction between the image, caption and dialogue round. This loss makes the positive samples, i.e. the pairs of image, caption and dialogue round, closer to each other, and pushes away the distance between the samples and the negative samples.

[0075] Next, the present application is further described in detail.

[0076] S110: Extract image features of the target image.

[0077] Further, step S110 can include:

[0078] S111: Adjust the size of the target image and divide the target image after adjusting the size into a plurality of tokens;

[0079] S112: Input a plurality of tokens into a pre-configured image encoder to obtain the image features.

[0080] Specifically, the embodiment of the present application can use a 12-layer visual transformer ViT-B / 16 as an image encoder, and use the weights pre-trained on the guided dataset from BLIP to initialize it. Optionally, the image I can be adjusted to 384x384, and then divided into a plurality of 16x16 patches as tokens. The above tokens are input into the image encoder to extract the image features Where n is the number of 16x16 patches labeled in the image I (cls is a label code, English is classification, indicating classification). Therefore, the image encoder can be represented as:

[0081] F I =ViT(I)

[0082] The image features can be used to form the components of the subsequent cross-attention module input, and the input of the image features in the contrastive learning loss function.

[0083] S120: Encode the title and each of the dialogue turns round by round to obtain dialogue turn features.

[0084] Further, step S120 can include:

[0085] S121: Label all the words in the title and all the words in each of the dialogue turns to obtain a labeled sequence;

[0086] S122: Add a code representing the beginning of encoding at the beginning of the labeled sequence, and add a code representing the end of encoding at the end of the labeled sequence;

[0087] S123: Encode the labeled sequence round by round to obtain the dialogue turn features.

[0088] Further, step S123 can include:

[0089] Input the labeled sequence into a pre-configured recurrent dialogue encoder to obtain the dialogue turn features;

[0090] The expression of the recurrent dialogue encoder is:

[0091]

[0092] where H t is the input of the cross-attention module in the recurrent dialogue encoder, where F I is the image feature, is the dialogue history feature from the 0th round to the (t-1)th round.

[0093] Specifically, in order to better capture the time dependence in visual dialogue and generate the final multi-modal dialogue feature is used to encode the current question-answer pair H t and the ground truth dialogue history (including the title) H = (H0, H1,..., H t-1 ), the embodiment of the present application can deploy a BERT-based uncased model, which is initialized with the pre-trained weights of BLIP, to encode the current question-answer pair H t and interact with the context (I, H) through the cross-attention mechanism. For each question-answer pair, all the words are marked using the Word Piece tokenizer to obtain a token sequence. Then, a [Encode] token (a token code, representing a specific encoding task) is added at the beginning of the input sequence, and a [SEP] token (a token code, English for separation, representing separation) is appended to the input sequence to mark the end of the input sequence. Specifically, for a multi-turn dialogue, the recurrent dialogue encoder of the embodiment of the present application encodes in a turn-by-turn manner to capture the temporal dependence. Therefore, the recurrent dialogue encoder can be modeled in a recurrent manner as follows:

[0094]

[0095] where H t is the input of the cross-attention module in the recurrent dialogue encoder, where F I is the image feature, is the dialogue history feature from the 0th round to the (t-1)th round. is obtained by concatenating the above multiple features to form the dialogue feature can also be used as the input of the dialogue feature in the subsequent contrastive loss function, and the dialogue feature

[0096] ​Therefore, the RecFormer in the embodiments of the present application can encode and interact with the multi-modal information in a round-by-round manner, and pay more attention to the introduced information to understand and reason the current dialogue round, thereby better capturing the temporal dependency.

[0097] S130: matching each dialogue round with the target image and the title according to the image feature and the dialogue round feature, to obtain each candidate dialogue round matched with the target image and the title.

[0098] Further, step S130 can include:

[0099] S131, determining a first similarity of each dialogue round feature to the image feature, and determining a second similarity of the image feature to each dialogue round feature;

[0100] S132, performing weighted summation on the first similarity and the second similarity to obtain a contrast loss similarity;

[0101] S133, matching each dialogue round with the target image and the title according to the contrast loss similarity, to obtain each candidate dialogue round matched with the target image and the title.

[0102] Further, step S131 can include:

[0103] S1310, determining a first similarity of each dialogue round feature to the image feature according to a first expression;

[0104] The first expression is:

[0105]

[0106] wherein H represents a feature of a first token in a single dialogue round feature, I represents a feature of a first token in the image feature, and τ is a temperature parameter, is a set of image features that do not match H;

[0107] S1311, determining a second similarity of the image feature to each dialogue round feature according to a second expression;

[0108] The second expression is:

[0109]

[0110] wherein, is a set of dialogue round features that do not match I.

[0111] In particular, semantic alignment between different modalities is important for a dialogue agent to consistently understand the relationship between different modalities and infer the underlying objects. However, most of the existing pre-trained encoders are only pre-trained based on general image-caption pairs, lacking the consideration of consistency with dialogue history, while contrastive learning is an effective target to improve visual and language understanding. Therefore, in order to unify the semantics of different modalities in visual dialogue, the embodiment of the present application proposes a history-aware multi-modal contrastive learning as an auxiliary task, which unifies the transformer-based features from different modalities by encouraging positive image-caption-dialogue triplet to have more similar representations than negative triplets.

[0112] Optionally, the embodiment of the present application considers the contrastive loss between dialogue histories by learning a similarity function s(m1, m2) = g1(m1) T g2(m2) to enforce history-aware contrastive learning, so that the parallel multi-modal pairs have higher similarity scores. g1 and g2 are two linear transformations that map the embeddings m1 and m2 to normalized low-dimensional (256-d) representations, where m1 and m2 are two representations from two different encoders. Here, for a dialogue turn-image pair, m1 and m2 are the first token in the dialogue feature and image feature and

[0113] Therefore, the history-aware contrastive learning between dialogue turns and images can be expressed as: for each dialogue turn and image, the embodiment of the present application can calculate the similarity of a single dialogue turn to an image as softmax normalized (for the sake of brevity and easy understanding of the formula, H in the following formula represents the first token in the single dialogue turn feature I represents the first token in the image feature ):

[0114]

[0115] where τ is a temperature parameter. is a set of negative examples that do not match H.

[0116] Similarly, the similarity of an image to a dialogue turn can be calculated as:

[0117]

[0118] where τ is a temperature parameter. is a set of negative examples that do not match I.

[0119] Therefore, the single dialogue turn-image contrastive loss L hcl can be expressed as:

[0120]

[0121] where λ is a loss weight that balances the main loss and the contrastive loss of the history-aware.

[0122] S140: determining a target dialogue turn that answers correctly in each of the candidate dialogue turns.

[0123] Further, S140 can include:

[0124] S141, adding the preset loss function and the contrastive loss similarity to obtain a total loss function;

[0125] S142, determining a target dialogue turn that answers correctly in each of the candidate dialogue turns according to the total loss function;

[0126] The expression of the preset loss function is:

[0127]

[0128] wherein, indicates the label of the tth dialogue question and answer in the sample i, if the answer is correct, the value of the label is 1, otherwise it is 0, the sample i includes a ground truth question and answer pair of a visual dialogue and a randomly sampled negative question and answer pair; indicates the probability that the tth dialogue question and answer in the sample i is predicted as a correct answer, and N indicates the total number of the sample i.

[0129] Specifically, in order to train the RecFormer of the embodiment of the application to solve the defects existing in the prior art, the embodiment of the application can deploy a main loss L ce (i.e. the preset loss function), which can be a cross-entropy loss function, for distinguishing whether the answer to a question is true or false under the condition of the entire dialogue history, image and title. In addition, the embodiment of the application can also deploy a history-aware contrastive loss L hcl (i.e. the contrastive loss similarity described above) to perform history-aware multi-modal alignment to achieve consistent understanding and reasoning.

[0130] wherein, the main loss function L ce indicates:

[0131]

[0132] wherein, for a visual dialogue example, indicates the label of the tth dialogue question and answer in the sample i (including the ground truth question and answer pair of the visual dialogue example and the randomly sampled negative question and answer pair), and is 1 if it is a positive class, that is, the answer is correct, otherwise the answer is wrong is 0. denotes the probability that the t-th round of dialog question in sample i (including the ground truth question-answer pair of this visual dialog example and the randomly sampled negative question-answer pair) is the correct answer, which is obtained by the dialog round features of sample i obtained in step S120 is obtained by a linear layer. N denotes the total number of samples. Therefore, the total training target (i.e., the total loss function) of the embodiment of the present application can be expressed as follows:

[0133] L = L ce + L hcl

[0134] The visual dialog method based on the recurrent converter and the history perception of the present application provides a contrastive learning of a recurrent multi-modal converter and history perception for visual dialog. The recurrent multi-modal converter better captures the temporal dependency by sequentially encoding the dialog history, and the contrastive learning of the history perception aligns different modalities in the fine-tuning stage of the visual dialog.

[0135] In order to verify the effectiveness of the present application, the following comparative tests are performed by the embodiment of the present application:

[0136] The RecFormer of the method of the present application is evaluated on two datasets: VisDial v0.9 (Visual Dialog) and VisDial v1.0.

[0137] The VisDial v0.9 dataset is collected based on MS-COCO images and captions. In a two-player chat game, one player tries to understand an unseen image and asks questions based on the previous dialog, while the other player watches the image and answers in free form. During the entire chat process, each picture lasts for 10 rounds. Therefore, the VisDial v0.9 dataset contains 83k training dialogues and 40k validation dialogues.

[0138] The VisDial v1.0 dataset is published by expanding the combination of the training set dialogues and the validation set of VisDial v0.9. The new validation and test parts are collected on Flickr pictures. In total, the VisDial v1.0 dataset contains 123k training dialogues, 2k validation dialogues and 8k test dialogues. Unlike the validation split in VisDial v0.9, the dialogues in the test split of VisDial v1.0 have a random length within 10 rounds.

[0139] The ranking indicators are used by the embodiment of the present application to evaluate the performance of the present application on the VisDial v0.9 and v1.0 datasets. These indicators include:

[0140] 1. MRR (Mean Reciprocal Rank), evaluates the inverse rank of the ground truth answer option;

[0141] 2. Recall@K (K e {1, 5, 10}), evaluates the percentage of the ground truth answer in the top-k answers;

[0142] 3. MR (Mean Rank), evaluates the average rank of the ground truth answer option;

[0143] 4. NDCG (Normalized Discounted Cumulative Gain), considers multiple correct answers and penalizes lower-ranked but correct answer options.

[0144] Performance of the method and the baseline on the validation set of the VisDial v0.9 dataset. ↑ indicates the higher the better, and ↓ indicates the lower the better. Table 1 is as follows:

[0145]

[0146]

[0147] Performance of the method and the baseline on the validation set of the VisDial v1.0 dataset. ↑ indicates the higher the better, and ↓ indicates the lower the better. Table 2 is as follows:

[0148]

[0149] Performance of the method and the baseline on the standard test (test-std) set of the VisDial v1.0 dataset. ↑ indicates the higher the better, and ↓ indicates the lower the better. Table 3 is as follows:

[0150]

[0151] Among them, AlignVD pre-trains a lot of additional pseudo-label data, including bounding boxes, which are crucial to performance improvement and are not very fair compared to other models. In addition, it can be seen that the present application can perform very well.

[0152] Ablation study of different components of RecFormer on the VisDial v1.0 validation set in the embodiment of the present application. Table 4 is as follows:

[0153]

[0154] To verify the effect of different components of the RecFormer of the embodiment of the present application, the embodiment of the present application performs experiments on VisDial v1.0, and the results of the VisDial v1.0 validation set are shown in Table 0. From the table, it can be observed that the recurrent dialogue encoder and the history-aware contrastive loss of the embodiment of the present application are effective for visual dialogue. Among the two key parts, the improvement of the recurrent dialogue encoder to the model is greater, which shows that the recurrent mechanism of the embodiment of the present application can effectively capture the temporal dependency implied in the visual dialogue. In addition, the RecFormer trained with the history-aware contrastive loss can outperform the model without the history-aware contrastive loss, which shows the effectiveness of the history-aware contrastive learning proposed by the embodiment of the present application in adjusting the semantics between different modal-specific features to achieve better multi-modal interaction and understanding. In summary, the recurrent dialogue encoder and the history-aware contrastive loss are effective for improving visual dialogue.

[0155] Next, the application process of the present application will be described with specific examples.

[0156] Referring to Figure 3 , the embodiment of the present application provides a specific example diagram of visual dialogue.

[0157] Specifically, the attention weight visualization of the recurrent multi-modal transformer RecFormer of the method of the present application in each dialogue turn. Figure 2 In the figure, the left side of the first row shows the label, and the rest of the text is the content of each round of dialogue turn, Q represents the question of each round, A represents the answer, and "RecFormer: Yes." in the lower right corner is the answer predicted by the visual dialogue model of the present application.

[0158] Referring to Figure 4 , the embodiment of the present application provides a comparison example diagram of visual dialogue.

[0159] Specifically, compared with Figure 4 (a) most existing methods (such as ICMU, UTC, VD-BERT) and Figure 4 (b) BLIP, the RecFormer of the present application, that is, Figure 4 (c) learns dialogue and other modal information in a recurrent manner by sharing dialogue encoders and passing previous encodings. The RecFormer (c) of the present application learns dialogue and other modal information in a recurrent manner by sharing dialogue encoders and passing previous encodings, so the present application can capture temporal dependencies in a more fine-grained manner.

[0160] Referring to Figure 5 , the embodiment of the present application provides a visual dialogue device based on a recurrent transformer and history awareness, comprising:

[0161] a first unit configured to acquire a target image, a title corresponding to the target image, and a dialogue history of the target image, the dialogue history including a plurality of dialogue turns, each of the dialogue turns including a question and an answer corresponding to the question;

[0162] a second unit configured to extract an image feature of the target image;

[0163] a third unit configured to encode the title and each of the dialogue turns round by round to obtain dialogue turn features;

[0164] a fourth unit configured to match the target image and the title with each of the dialogue turns according to the image feature and the dialogue turn features to obtain each candidate dialogue turn matched with the target image and the title;

[0165] a fifth unit configured to determine a target dialogue turn with a correct answer in each of the candidate dialogue turns.

[0166] The specific implementation of the visual dialogue device is basically the same as that of the above-mentioned visual dialogue method, and will not be repeated here.

[0167] The embodiment of the present application also provides an electronic device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned visual dialogue method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0168] Specifically, the electronic device can be a user terminal or a server.

[0169] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the above-mentioned visual dialogue method.

[0170] The memory is a non-transitory computer readable storage medium, which can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0171] The embodiments of the present application also disclose a computer program product or computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of an electronic device can read the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the electronic device to perform the method shown in the method. Figure 1

[0172] In some alternative embodiments, the functions / operations mentioned in the block diagrams can not occur in the order mentioned in the operation diagrams. For example, depending on the functions / operations involved, two blocks shown in succession can actually be executed substantially concurrently or the blocks can sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flow diagrams of the present application are only examples. The steps presented and described in the flow diagrams are provided by way of example in order to provide a more thorough understanding of the technology. The disclosed methods are not limited to the operations and logical flows presented in the specification. Alternative embodiments are contemplated in which the order of various steps is changed and in which sub-steps of a larger step are performed in parallel with each other.

[0173] Furthermore, although the present application is described in the context of functional modules, it is to be understood that one or more of the functions and / or features described can be integrated in a single physical device and / or software module, or one or more functions and / or features can be implemented in separate physical devices or software modules. It is also to be understood that detailed discussion of the actual implementation of each module is unnecessary to an understanding of the present application. Rather, the actual implementation is within the routine of an engineer's knowledge given the property, functionality and internal relationships of the various functional modules disclosed in the devices herein. Therefore, the present application is not limited to the specific details of the embodiments described herein, but is encompassed by the appended claims as well as their equivalents. In particular, it is not intended that the application be limited to the specific embodiments disclosed. The application can be practiced with the exact details thereof as understood by one skilled in the art. It is therefore contemplated that the application shall also cover any and all modifications of the application within the scope of the claims.

[0174] ​If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the technical solutions that essentially contribute to the prior art or the parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0175] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered a list of executable instructions for implementing logic functions, and can be specifically embodied in any computer-readable medium for use by an instruction execution system, apparatus, or device, such as a computer-based system, a system including a processor, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions, or in conjunction with these instructions execution systems, apparatuses, or devices. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport programs for use by an instruction execution system, apparatus, or device, or in conjunction with these instruction execution systems, apparatuses, or devices.

[0176] More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection having one or more wires (electrical devices), a portable computer diskette (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium can even be paper or other suitable medium on which the program can be printed, as the program can be electronically obtained, for example, by optical scanning of the paper or other medium, followed by editing, interpreting, or otherwise processing, if necessary, in other suitable ways to be electronically obtained, and then stored in the computer memory.

[0177] It should be understood that aspects of the application can be implemented in hardware, software, firmware or a combination thereof. In the above embodiments, various steps or methods can be implemented in software or firmware which is stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and in another embodiment, any of the following technologies, known in the art, can be used: a combination of discrete logic circuits having logic gates for implementing logic functions upon an application of data signals, application specific integrated circuits having logic gates, field programmable gate arrays (FPGA), or other components, in combination or as the case can be.

[0178] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0179] Although embodiments of the present application have been shown and described, it would be appreciated by those skilled in the art that changes, modifications, alternatives and variations to these embodiments could be made without departing from the principles and spirit of the application, the scope of which is defined by the claims and their equivalents.

[0180] The above is a specific description of the preferred embodiments of the present application, but the present application is not limited to the described embodiments, and those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present application, and these equivalent modifications or replacements are included in the scope defined by the claims of the present application.

Claims

1. A cycle-converter and history-aware based visual dialogue method, characterized in that, The method comprises the following steps: obtaining a target image, a title corresponding to the target image, and a dialogue history of the target image; the dialogue history comprises a plurality of dialogue turns, and each dialogue turn comprises a question and an answer corresponding to the question; extracting an image feature of the target image; performing turn-by-turn encoding on the title and each dialogue turn to obtain a dialogue turn feature; matching the target image and the title with each dialogue turn according to the image feature and the dialogue turn feature to obtain each candidate dialogue turn matched with the target image and the title; determining a target dialogue turn with a correct answer in each candidate dialogue turn; the step of performing turn-by-turn encoding on the title and each dialogue turn to obtain a dialogue turn feature comprises the following steps: labeling all words in the title and all words in each dialogue turn to obtain a label sequence; adding a code representing the start of encoding at the beginning of the label sequence and adding a code representing the end of encoding at the end of the label sequence; performing turn-by-turn encoding on the label sequence to obtain the dialogue turn feature; the step of performing turn-by-turn encoding on the label sequence to obtain the dialogue turn feature comprises the following steps: inputting the label sequence into a pre-configured recurrent dialogue encoder to obtain the dialogue turn feature; the expression of the recurrent dialogue encoder is: wherein, denotes the dialogue turn feature, is the input of the recurrent dialogue encoder, is the input of the cross-attention module in the recurrent dialogue encoder, wherein is the image feature, is the dialogue history feature from the 0th turn to the t-1th turn.

2. The method of claim 1, wherein, the step of extracting the image feature of the target image comprises the following steps: adjusting the size of the target image and dividing the target image with the adjusted size into a plurality of tokens; inputting the plurality of tokens into a pre-set image encoder to obtain the image feature.

3. The method of claim 1, wherein, the step of matching the target image and the title with each dialogue turn according to the image feature and the dialogue turn feature to obtain each candidate dialogue turn matched with the target image and the title comprises the following steps: determining a first similarity of each dialogue turn feature to the image feature and a second similarity of the image feature to each dialogue turn feature; performing weighted summation on the first similarity and the second similarity to obtain a contrastive loss similarity; matching the target image and the title with each dialogue turn according to the contrastive loss similarity to obtain each candidate dialogue turn matched with the target image and the title.

4. The method of claim 3, wherein, the step of determining the first similarity of each dialogue turn feature to the image feature and the second similarity of the image feature to each dialogue turn feature comprises the following steps: determining the first similarity of each dialogue turn feature to the image feature according to a first expression; the first expression is: wherein H denotes a first labeled feature of the individual dialog turn feature, I denotes a first labeled feature of the image feature, and τ is a temperature parameter, is a set of features associated with non-matching image features; determining the second similarity of the image feature to each dialogue turn feature according to a second expression; the second expression is: wherein, is a set of dialog turn features that do not match.

5. The method of claim 3, wherein, the step of determining a target dialogue turn with a correct answer in each candidate dialogue turn comprises the following steps: adding a preset loss function to the contrastive loss similarity to obtain a total loss function; determining a target dialogue turn with a correct answer in each candidate dialogue turn according to the total loss function; the expression of the preset loss function is: wherein, denotes a sample in the tth round of dialogue question answering, and the value of the label is 1 if the answer is correct, otherwise 0, the sample includes ground truth question-answer pairs of visual dialogues and randomly sampled negative question-answer pairs; denotes the sample in the tth round of dialogue question answering, and the value of the label is 1 if the answer is correct, otherwise 0, the sample total number.

6. A cycle converter and history-aware based visual dialog apparatus, characterized by, the method comprises the following steps: The first unit is configured to obtain a target image, a title corresponding to the target image, and a dialogue history of the target image; The dialogue history includes a plurality of dialogue turns, and each dialogue turn includes a question and an answer corresponding to the question; The second unit is configured to extract an image feature of the target image; The third unit is configured to encode the title and each dialogue turn round by round to obtain dialogue turn features; The fourth unit is configured to match the target image and the title with each dialogue turn according to the image feature and the dialogue turn feature to obtain each candidate dialogue turn matched with the target image and the title; The fifth unit is configured to determine a target dialogue turn with a correct answer in each candidate dialogue turn; Wherein: The encoding of the title and each dialogue turn round by round to obtain dialogue turn features includes: Labeling all words in the title and all words in each dialogue turn to obtain a label sequence; Adding a code representing the start of encoding at the beginning of the label sequence and adding a code representing the end of encoding at the end of the label sequence; Encoding the label sequence round by round to obtain the dialogue turn features; The encoding of the label sequence round by round to obtain the dialogue turn features includes: Inputting the label sequence into a pre-configured recurrent dialogue encoder to obtain the dialogue turn features; The expression of the recurrent dialogue encoder is: wherein, denotes the dialogue turn feature, is an input of the recurrent dialogue encoder, is an input of a cross-attention module in the recurrent dialogue encoder, wherein is the image feature, is a dialogue history feature from the 0th turn to the t-1th turn.

7. An electronic device, comprising: The processor and the memory are included; The memory is configured to store a program; The processor executes the program to implement the visual dialogue method based on the recurrent converter and the history perception according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The storage medium stores a program, and the program is executed by the processor to implement the visual dialogue method based on the recurrent converter and the history perception according to any one of claims 1 to 5.

Images