Visual dialogue method and device based on external knowledge enhancement and semantic consistency

Abstract

The invention relates to a visual dialogue method and device based on external knowledge enhancement and semantic consistency, and the method comprises the steps: constructing a multi-modal scene graph through extracting the incidence relation between intra-domain visual objects and the incidence relation between intra-domain text vocabularies according to an obtained image, a historical dialogue text and a current question text, constructing a common sense knowledge base by extracting common sense knowledge triples; based on the common knowledge base, expanding the multi-modal scene graph and carrying out relation reinforcement learning to obtain characteristics of associated reinforcement nodes; obtaining instance-level representation by using a pre-trained large language model, obtaining text common sense-level representation based on the characteristics of the association enhancement node, and performing characteristic fusion for the instance-level representation and the common sense-level representation to obtain global fusion characteristics; and based on the fusion feature, obtaining an answer of the visual dialogue through reasoning. According to the method, semantic association can be carefully mined, so that the accuracy, richness and comprehensiveness of multiple rounds of visual dialogues are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of visual reasoning technology, and in particular to a visual dialogue method and device based on external knowledge enhancement and semantic consistency. Background Art

[0002] Multimodal data covers a wide range of types, sources and complex relational structures. How to quickly and efficiently use these diverse data to serve humanity has become a hot research topic. At present, the application of multimodal data is still in its infancy, and related theories and technologies are constantly evolving. Improving the ability to automatically perceive and understand multimodal data and exploring the semantic connections between different modalities is a challenge with important application prospects and research significance. Among them, visual dialogue, as a key cross-modal understanding task, enables artificial intelligence agents to conduct multiple rounds of questions and answers based on visual information and historical dialogue content. It is one of the core technologies for realizing intelligent human-computer interaction. The potential applications of this technology include intelligent customer service, entertainment games, smart homes, medical care, and intelligent manufacturing, showing broad development prospects.

[0003] In visual dialogue tasks, existing methods have obvious deficiencies in fine-grained modeling of visual and textual information, and have difficulty in making accurate responses in complex scenarios. On the one hand, most current methods overemphasize the modeling of complex associations between texts, while ignoring the diverse relationships between visual objects. This bias limits the model's ability to fully utilize visual details when answering questions, resulting in its limitations in understanding and parsing visual information. On the other hand, most methods find it difficult to effectively maintain semantic consistency within and between modalities in the process of extracting fine-grained semantic relationships and transmitting multimodal information. This deficiency not only affects the model's reasoning ability, but also restricts its performance in practical applications. Therefore, it is particularly important to develop a fine-grained modeling method that can comprehensively consider the relationship between visual objects and text associations while ensuring semantic consistency. This will help improve the overall performance of the visual dialogue system, enabling it to more accurately understand user intent in complex scenarios and provide more accurate and rich answers. Summary of the invention

[0004] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a visual dialogue method and device based on external knowledge enhancement and semantic consistency, so as to solve or partially solve the problem of lack of consideration of the diverse relationships between visual objects and difficulty in effectively maintaining semantic consistency within and between modalities.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] One aspect of the present invention provides a visual dialogue method based on external knowledge enhancement and semantic consistency, comprising the following steps:

[0007] Based on the acquired images, historical conversation texts, and current question texts, a multimodal scene graph is constructed by extracting the associations between visual objects in the domain and the associations between text words in the domain, and a common sense knowledge base is constructed by extracting common sense knowledge triplets.

[0008] Based on the common sense knowledge base, the multimodal scene graph is expanded and relationship enhancement learning is performed to obtain features of associated enhanced nodes;

[0009] Based on the image, historical conversation text, current question text and candidate answer text, a pre-trained large language model is used to obtain an instance-level representation, based on the features of the associated enhanced nodes, a text common-sense-level representation is obtained, and feature fusion is performed on the instance-level representation and the common-sense-level representation to obtain a global fusion feature;

[0010] Based on the fused features, the answer to the visual dialogue is obtained through reasoning.

[0011] As a preferred technical solution, the multimodal scene graph includes a visual sub-scene graph and a text sub-scene graph, and the process of constructing the visual scene graph and the text sub-scene graph includes the following steps:

[0012] Based on the acquired image, extracting a visual subscene graph including object nodes and edges, wherein each object node includes a text label;

[0013] For the historical dialogue text and the current question text, meaningless words are filtered out, and word entities are used as nodes and text relations are used as edges to obtain a text sub-scene graph.

[0014] As a preferred technical solution, the process of constructing the common sense knowledge base includes the following steps:

[0015] Extract <subject, predicate, object> tuples from the pre-acquired data set, retain the <subject, object> pairs with the highest co-occurrence frequency, and form a common sense knowledge base.

[0016] As a preferred technical solution, based on the common sense knowledge base, the multimodal scene graph is expanded and relationship enhancement learning is performed to obtain local explicit structured features, which includes the following steps:

[0017] For each sub-scene graph in the multimodal scene graph, a query is performed based on the nodes of the sub-scene graph, and by calculating the relevance scores with the nodes in the common sense knowledge base, the nodes with high scores and their associations are integrated into the visual scene graph to achieve the expansion of the multimodal scene graph;

[0018] For each sub-scene graph in the expanded multimodal scene graph, the pre-trained CNN network model is used to extract node features, and the pre-trained BERT model is used to extract edge features, and the expanded multimodal scene graph is encoded;

[0019] For each sub-scene graph in the encoded multimodal scene graph, an adjacency matrix is ​​constructed to represent the frequency of different types of relationships between node pairs;

[0020] Based on the adjacency matrix, a graph convolutional neural network is used to extract the features of the associated enhanced nodes to achieve the understanding and fusion of multimodal features.

[0021] As a preferred technical solution, based on the image, historical conversation text, current question text and candidate answer text, the process of obtaining implicit features using a pre-trained large language model includes the following steps:

[0022] The image title text of the current dialogue round, the historical dialogue text, the current question text, and the texts of multiple candidate answers are concatenated to obtain a text sequence;

[0023] Use the pre-trained CNN network model to encode the image of the current dialogue round to obtain an image object sequence;

[0024] Based on the text sequence and the image object sequence, an instance-level implicit visual-text representation is extracted using a pre-trained large language model.

[0025] As a preferred technical solution, the process of obtaining the common sense level representation of the text based on the features of the associated enhanced nodes includes the following steps:

[0026] Based on the instance-level representation and the features of the association enhancement node, the significance score of each association enhancement node is calculated by the following formula:

[0027]

[0028] Based on the significance score, the common sense representation of the text is obtained using the following formula:

[0029]

[0030] in, and Respectively represent the associated enhancement nodes and The corresponding significance score, and is a learnable parameter matrix, λ is a hyperparameter that controls the smoothness of the softmax function, V h 、T h They are the visual hidden state and text hidden state in instance-level representation, respectively.T Indicates transpose.

[0031] As a preferred technical solution, the fusion of the instance-level representation and the common-sense-level representation is achieved through weighted summation.

[0032] As a preferred technical solution, the process of obtaining the answer to the visual dialogue through reasoning based on the fusion features includes the following steps:

[0033] Based on the fusion features, the discriminant model and the generative model are trained with the goal of minimizing the loss function value, wherein the loss function includes one or more of a masked language modeling loss, a masked region modeling loss, a next sentence prediction loss, a KL divergence loss, and a bidirectional triple ranking loss;

[0034] Based on the discriminant model, the candidate answers are ranked by calculating the semantic consistency between the candidate answers and the current question and the historical dialogue, and the optimal answer is obtained by reasoning;

[0035] Based on the generative model, accurate and coherent answers are generated according to the understanding of the dialogue context and visual scene.

[0036] Another aspect of the present invention provides an electronic device, comprising: one or more processors and a memory, wherein the memory stores one or more programs, and the one or more programs include instructions for executing the aforementioned visual dialogue method based on external knowledge enhancement and semantic consistency.

[0037] Another aspect of the present invention provides a visual dialogue device based on external knowledge enhancement and semantic consistency, comprising:

[0038] The knowledge graph construction module is used to construct a multimodal scene graph based on the acquired images, historical conversation texts, and current question texts by extracting the associations between visual objects in the domain and the associations between text words in the domain, and to construct a common sense knowledge base by extracting common sense knowledge triples;

[0039] A scene graph enhancement module, used to expand the multimodal scene graph and perform relationship enhancement learning based on the common sense knowledge base to obtain features of associated enhanced nodes;

[0040] A bilateral knowledge fusion and reasoning strategy module is used to obtain instance-level representations based on the image, historical conversation text, current question text, and candidate answer text using a pre-trained large language model, obtain text common sense-level representations based on the features of the associated enhancement nodes, and perform feature fusion on the instance-level representations and the common sense-level representations to obtain global fusion features;

[0041] The answer reasoning module is used to obtain the answer to the visual dialogue through reasoning based on the fused features.

[0042] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0043] (1) Fine-grained balanced modeling: The present invention improves the model's ability to utilize visual details by comprehensively considering the relationship between visual objects and text associations, thereby having obvious advantages in understanding and parsing visual information. It can also effectively and balancedly model visual and text information at a fine-grained level, thus overcoming the shortcomings of existing methods in this regard.

[0044] (2) Semantic consistency maintenance: In the process of extracting fine-grained semantic relationships and transmitting multimodal information, the present invention can better maintain semantic consistency within and between modalities. This feature not only improves the reasoning ability of the model, but also enhances its practical application performance in complex scenarios.

[0045] (3) Improved responsiveness in complex scenarios: With the help of external knowledge, the design of the present invention enables it to better understand user intent in complex scenarios. This capability enables the visual dialogue system to perform better in handling challenging tasks and provide more accurate and rich answers. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a flowchart of a visual dialogue method based on external knowledge enhancement and semantic consistency in an embodiment;

[0047] Figure 2 A schematic diagram of a visual dialogue framework in an embodiment;

[0048] Figure 3 is a schematic diagram of a visual dialogue device based on external knowledge enhancement and semantic consistency in an embodiment;

[0049] Figure 4 Schematic diagram of an electronic device in an embodiment. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0051] Example 1

[0052] In view of the problems existing in the aforementioned prior art, this embodiment provides a visual dialogue method based on external knowledge enhancement and semantic consistency, which improves the ability of human-computer interaction through balanced fine-grained modeling and multi-level semantic consistency reasoning.

[0053] See also Figure 1 , the method includes the following steps S1-S4, each step will be described below.

[0054] Step S1, constructing a multimodal scene graph and a common sense knowledge base: constructing a multimodal scene graph and a common sense knowledge base to obtain fine-grained associations and common sense knowledge triples between visual objects in the domain and between text words in the domain.

[0055] like Figure 2 As shown, the knowledge graph construction module includes constructing a visual scene graph, a text scene graph and a common sense knowledge base, and respectively obtaining fine-grained associations and common sense knowledge triples between visual objects in the domain and between text words in the domain. Specifically, step S1 may include steps S101-S103.

[0056] Step S101, given an image I, extract a visual scene graph (VSG) using the scene graph generator Neural Motifs To show the structured image context, is a collection of objects, Represents the set of VSG edges. In addition, each object node has a word label (i.e., text label), which provides additional linguistic details in addition to visual features.

[0057] Step S102, based on historical conversations and current questions, taking into account the rich vocabulary and meaningless terms, remove infrequently appearing words (such as "is", "a", etc.). ot The words are considered as graph nodes and the text scene graph (TSG) is obtained using in and Represent word entity set and text relation set respectively.

[0058] Step S103, in order to reduce the differences in semantics and relationships between the visual graph and the text graph, useful knowledge is extracted from the VisualGenome dataset to build a common sense knowledge base (CKB). Specifically, first select the tuples in the VisualGenome dataset<subject,predicate,object> (i.e. <subject, predicate, object> tuple), and then retain the k pairs with the highest co-occurrence frequency<subject,object> These retained pairs constitute the knowledge base in and They are the subject, predicate and object sets.

[0059] Step S2, expand the multimodal scene graph and perform relationship enhancement learning to obtain the features of the associated enhanced nodes: for the multimodal scene graph and the common sense knowledge base, by expanding the concept nodes and common sense relationships, explicit structured features with local semantic consistency are obtained

[0060] like Figure 2 As shown, the scene graph enhancement module uses external knowledge to expand the concept nodes and common sense relationships of the initial multimodal scene graph. Specifically, step S2 may include steps S201-S204.

[0061] Step S201, given The query node in CKB is calculated, and the relevance score of the query node in CKB is calculated. The top-300 relevant nodes with the highest scores are added to the extended visual scene graph (E-VSG), and the edges corresponding to the relevant nodes are added to E-VSG. Using the same method, given For the query node in , we take the top-300 related nodes with the highest scores and their edges and add them to the extended visual scene graph (E-TSG).

[0062] Step S202: Encode the nodes and edges of E-VSG and E-TSG. For E-VSG, node features are extracted by the Faster R-CNN model pre-trained on the VisualGenome dataset, and edge features are obtained by the pre-trained BERT model. In order to achieve a comprehensive representation of the nodes, the features of the edges associated with each node are concatenated to the node itself, and finally the node feature representation is obtained. Among them, d is the dimension of node features, m oυ is the number of nodes in the E-VSG. Similarly, for ot The E-TSG of nodes uses the BERT model to extract node features including edge features, which is recorded as

[0063] Step S203, based on m oυ The E-VSG with m nodes and ot The E-TSG of nodes is constructed by two directed adjacency matrices and For each pair of nodes, the frequency of different types of relationships between them is calculated. The counts are then normalized to values ​​between 0 and 1 and used as element values ​​in the adjacency matrix. This step ensures that the adjacency matrix captures the existence and diversity of relationships.

[0064] Step S204, using stacked GCN to capture semantic relations and learn local features of relation enhancement, that is, features of associated enhanced nodes, the specific operation is shown in the following formula:

[0065]

[0066] Among them, l m is the number of stacked GCN layers, represents the learnable weight matrix, is a learnable bias matrix, and σ(·) represents the LeakyReLU activation function. The initial node representation is The calculation formula of the normalized adjacency matrix is ​​given as Where D represents the degree matrix. The output of the final GCN layer is is regarded as the global representation of E-VSG, i.e. Each visual node is embedded as Using a similar approach, we can derive the global representation of E-TSG in Represents the embedding representation of each text node.

[0067] Step S3, extract instance-level representations and common-sense-level representations, and obtain global fusion features through feature fusion: use the pre-trained visual language large model to obtain implicitly aligned in-domain features, and fuse the implicitly aligned features with explicit structured features through bilateral knowledge fusion and reasoning strategies to obtain fusion features with global consistency.

[0068] The bilateral knowledge fusion and reasoning strategy integrates the instance-level visual-text features extracted by the pre-trained visual language model and the common-sense-level visual-text feature representation to provide the model with globally consistent semantic information. It is divided into instance-level feature extraction, common-sense-level feature extraction, and knowledge fusion and reasoning.

[0069] Specifically, step S3 may include steps S301-S303.

[0070] Step S301: Instance-level feature extraction. For a given image-question pair with a dialogue turn of r, the image title Cap, the historical dialogue H r 、Current QuestionQ r and candidate answer A r are concatenated to form the following text sequence:

[0071] T={[ClS]Cap[SEP|Q1|SEP]A1,...,Q r [SEP]A r}

[0072] The [CLS] token indicates the start of a conversation sequence, and the [SEP] token separates each question-answer pair. To encode an image I, the object-level feature vector is extracted by the Faster R-CNN model pre-trained on the Visual Genome dataset. Each object feature υ i The dimension is 1024, n υ =36 represents the number of targets detected. Then, for υ i The mean pooling operation is performed to obtain the image object sequence starting with the [IMG] tag. After that, the pre-trained two-stream ViLBERT model is applied to jointly model the visual and text features. The visual hidden state V h and the text hidden state T h The acquisition process is as follows:

[0073] V h ,T h =ViLBERT(V, T)

[0074] Step S302: Common sense level feature extraction. In order to obtain relevant external knowledge, the instance-level visual and textual representation (i.e., V h and T h ) is used as input to query related concept nodes from E-VSG and E-TSG. Mathematically, the importance scores of different semantic nodes are first calculated to measure the importance of each associated enhancement node, as shown below:

[0075]

[0076] in, and Respectively represent the semantic nodes (i.e., associated enhancement nodes), and The corresponding significance score, and is a learnable parameter matrix, and λ is a hyperparameter that controls the smoothness of the softmax function. Next, the visual common sense level representation V c and the common sense representation of the text T c The calculation formula is as follows:

[0077]

[0078] Step S303: Knowledge fusion and reasoning. Instance-level representation V h (or T h ) and common sense representation V c (or V h ) are fused together by the following simple weighted sum operation:

[0079] V f =βV h +(1-β)V c

[0080] Tf =βT h +(1-β)T c

[0081] Among them, β is a parameter used to adjust the balance between different representation types, V f and T f Represent the fused visual representation and textual representation respectively.

[0082] Step S4, based on the fused features, the answer to the visual dialogue is obtained through reasoning.

[0083] For the answer reasoning module, three traditional loss functions are used, namely, masked language modeling (MLM) loss function, masked region modeling (MRM) loss function and next sentence prediction (NSP) loss function. In addition, the semantic consistency at the local and global levels is also deeply studied, and the Kullback-Leibler (KL) divergence loss function and the bidirectional triple ranking loss function are used to ensure that the model can rank relevant answers in a globally consistent manner.

[0084] Specifically, step S4 may include steps S401-S405.

[0085] S401, Masked Language Modeling (MLM) loss function and Masked Region Modeling (MRM) loss function. In this application, 10% of the text area and 15% of the visual area will be randomly masked by a special marker [MASK]. The model needs to restore the masked markers based on the surrounding markers and cross-modal clues under the constraints of the following formula:

[0086]

[0087] Among them, t m and m Represent blocked words and area markers, T \m and V \m Represent unblocked words and region tags, S tr Represents the training set.

[0088] S402, Next Sentence Prediction (NSP) loss function. NSP loss aims to predict whether a candidate answer is correct, and it plays a key role in capturing the interdependencies between dialogues. The formula of NSP loss is as follows:

[0089]

[0090] where y∈{0,1} is the binary label of the true value, is a binary classifier that estimates the probability from the dot product of the [CLS] and [IMG] token representations.

[0091] S403, Kullback-Leibler (KL) divergence loss function. The purpose of using KL divergence as a loss function is to minimize the difference between the attention distributions of E-VSG and E-TSG, thereby promoting semantic consistency between the text and visual representations of the same scene content. The calculation formula of the KL loss function is:

[0092]

[0093] S404, Bidirectional triple ranking loss function. In order to globally align visual and textual representations, the model adopts the widely adopted bidirectional triple ranking loss function. Given two embedding vectors of the visual scene graph and the textual scene graph, the global semantic relevance is calculated as:

[0094]

[0095] where ||·|| represents the Euclidean norm of the vector. Indicates V f and T f The bidirectional triplet ranking loss between , can be defined as:

[0096]

[0097] in,

[0098]

[0099] Where γ is a predefined difference parameter, (V f , T f ) are the matched visual-text positive pairs, and their corresponding negative pairs are represented as and

[0100] S405, the total loss function of the model is calculated using the following formula:

[0101]

[0102] Among them, τ1 and τ2 represent weight parameters.

[0103] In order to verify the performance of this method, the following experiments were designed.

[0104] The proposed method SCVD+ is validated on the visual dialogue benchmark datasets VisDial v0.9 and VisDial v1.0, and the performance of the proposed method is evaluated under both discriminative and generative settings.

[0105] The visual dialogue model is trained using the benchmark dataset. The specific process of visual dialogue model training includes:

[0106] Step 1: Given an image, a question, a historical conversation, and 100 candidate answers;

[0107] Step 2: Build visual scene graph, text scene graph and common sense knowledge base;

[0108] Step 3: Based on the common sense knowledge base in Step 2, expand the concept nodes and common sense relationships of the visual scene graph and text scene graph;

[0109] Step 4: Use the pre-trained visual language model to extract visual-text features of implicitly aligned instance-level features;

[0110] Step 5: Encode the visual scene graph and text scene graph expanded in Step 3, and measure the importance of each associated node in the scene graph to obtain common sense-level visual-text features;

[0111] Step 6: Adopt bilateral knowledge fusion and reasoning strategy to fuse instance-level features and common sense-level features in Step 4 and Step 5, and infer the correct answer;

[0112] Step 7: Repeat Step 1-Step 6 multiple times to complete multi-round dialogue answer reasoning for all images in the dataset.

[0113] When evaluating the performance of the visual dialogue model, indicators including mean rank (Mean), mean reciprocal rank (MRR), recall at different thresholds k (Recall@k, k is 1, 5, 10) and normalized discounted cumulative gain (NDCG) are used. Mean rank and mean reciprocal rank are calculated based on a single correct answer, while recall and NDCG consider multiple semantically related correct answers. Among these indicators, the lower the mean rank, the better the performance, while the higher the other indicators, the better. In particular, NDCG is an evaluation indicator dedicated to the VisDial v1.0 dataset, which is used to measure the generalization performance of the dialogue model; MRR is used to evaluate the accuracy of the model's answer reasoning. The test results of this method on the VisDial v0.9 and v1.0 datasets are shown in Tables 1 and 2, respectively.

[0114] Table 1 Performance of different models on the VisDial v0.9 dataset

[0115]

[0116] Table 2 Performance of different models on the VisDial v1.0 dataset

[0117]

[0118] In summary, this method has the following beneficial effects:

[0119] (1) This method utilizes visual and textual information through scene graphs, achieving a balanced emphasis on visual and textual information. With the assistance of external knowledge, it reduces conceptual confusion in multimodal scene graphs, enabling the visual dialogue system to understand and generate information more accurately.

[0120] (2) This method provides a two-level knowledge fusion and reasoning strategy that combines the advantages of BERT-based models in capturing implicit multimodal knowledge in large-scale pre-training with the ability to explicitly encode common sense knowledge from extended scene graphs, thereby improving the performance of handling complex common sense-based problems.

[0121] (3) This method designs a loss function to ensure local and global semantic consistency in the process of integrating in-domain and out-of-domain knowledge, reduces the impact of multimodal representation differences, and enhances the performance of the model in joint reasoning of multimodal information.

[0122] (4) Through extensive experiments on the VisDial v0.9 and VisDial v1.0 public datasets, our method demonstrates significantly superior performance, indicating the effectiveness and reliability of our method in practical applications.

[0123] Example 2

[0124] Based on Example 1, see Figure 2 and Figure 3 , this embodiment provides a visual dialogue device based on external knowledge enhancement and semantic consistency, including:

[0125] The knowledge graph construction module is used to construct a multimodal scene graph based on the acquired images, historical conversation texts, and current question texts by extracting the associations between visual objects in the domain and the associations between text words in the domain, and to construct a common sense knowledge base by extracting common sense knowledge triples;

[0126] A scene graph enhancement module, used to expand the multimodal scene graph and perform relationship enhancement learning based on the common sense knowledge base to obtain features of associated enhanced nodes;

[0127] A bilateral knowledge fusion and reasoning strategy module is used to obtain instance-level representations based on the image, historical conversation text, current question text, and candidate answer text using a pre-trained large language model, obtain text common sense-level representations based on the features of the associated enhancement nodes, and perform feature fusion on the instance-level representations and the common sense-level representations to obtain global fusion features;

[0128] The answer reasoning module is used to obtain the answer to the visual dialogue through reasoning based on the fused features.

[0129] Example 3

[0130] Based on Example 1, this embodiment provides an electronic device, including: one or more processors and a memory, wherein the memory stores one or more programs, and the one or more programs include instructions for executing the visual dialogue method based on external knowledge enhancement and semantic consistency as described in Example 1.

[0131] like Figure 2 As mentioned above, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 Of course, in addition to the software implementation, the present invention does not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0132] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0133] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0134] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A visual dialogue method based on external knowledge enhancement and semantic consistency, characterized in that: The steps include: Based on the acquired images, historical conversation texts, and current question texts, a multimodal scene graph is constructed by extracting the associations between visual objects in the domain and the associations between text words in the domain, and a common sense knowledge base is constructed by extracting common sense knowledge triplets. Based on the common sense knowledge base, the multimodal scene graph is expanded and relationship enhancement learning is performed to obtain features of associated enhancement nodes; Based on the image, historical conversation text, current question text and candidate answer text, a pre-trained large language model is used to obtain an instance-level representation, based on the features of the associated enhanced nodes, a text common-sense-level representation is obtained, and feature fusion is performed on the instance-level representation and the common-sense-level representation to obtain a global fusion feature; Based on the fused features, the answer to the visual dialogue is obtained through reasoning.

2. The visual dialogue method based on external knowledge enhancement and semantic consistency according to claim 1, characterized in that: The multimodal scene graph includes a visual sub-scene graph and a text sub-scene graph. The process of constructing the visual scene graph and the text sub-scene graph includes the following steps: Based on the acquired image, extracting a visual subscene graph including object nodes and edges, wherein each object node includes a text label; For the historical dialogue text and the current question text, meaningless words are filtered out, and word entities are used as nodes and text relations are used as edges to obtain a text sub-scene graph.

3. The visual dialogue method based on external knowledge enhancement and semantic consistency according to claim 1, characterized in that: The construction process of the common sense knowledge base includes the following steps: Extract <subject, predicate, object> tuples from the pre-acquired data set, retain the <subject, object> pairs with the highest co-occurrence frequency, and form a common sense knowledge base.

4. The visual dialogue method based on external knowledge enhancement and semantic consistency according to claim 1, characterized in that: Based on the common sense knowledge base, the multimodal scene graph is expanded and relationship enhancement learning is performed to obtain local explicit structured features, which includes the following steps: For each sub-scene graph in the multimodal scene graph, a query is performed based on the nodes of the sub-scene graph, and by calculating the relevance scores with the nodes in the common sense knowledge base, the nodes with high scores and their associations are integrated into the visual scene graph, thereby realizing the expansion of the multimodal scene graph; For each sub-scene graph in the expanded multimodal scene graph, the pre-trained CNN network model is used to extract node features, and the pre-trained BERT model is used to extract edge features, and the expanded multimodal scene graph is encoded; For each sub-scene graph in the encoded multimodal scene graph, an adjacency matrix is ​​constructed to represent the frequency of different types of relationships between node pairs; Based on the adjacency matrix, a graph convolutional neural network is used to extract the features of the associated enhanced nodes to achieve the understanding and fusion of multimodal features.

5. The visual dialogue method based on external knowledge enhancement and semantic consistency according to claim 1, characterized in that: Based on the image, historical conversation text, current question text and candidate answer text, the process of obtaining implicit features using a pre-trained large language model includes the following steps: The image title text of the current dialogue round, the historical dialogue text, the current question text, and the texts of multiple candidate answers are concatenated to obtain a text sequence; Use the pre-trained CNN network model to encode the image of the current dialogue round to obtain an image object sequence; Based on the text sequence and the image object sequence, an instance-level implicit visual-text representation is extracted using a pre-trained large language model.

6. The visual dialogue method based on external knowledge enhancement and semantic consistency according to claim 1, characterized in that: The process of obtaining the common sense level representation of the text based on the features of the associated enhanced nodes comprises the following steps: Based on the instance-level representation and the features of the association enhancement node, the significance score of each association enhancement node is calculated by the following formula: Based on the significance score, the common sense representation of the text is obtained using the following formula: in, and Respectively represent the associated enhancement nodes and The corresponding significance score, and is a learnable parameter matrix, is a hyperparameter that controls the smoothness of the softmax function, and is the visual hidden state and text hidden state in the instance-level representation, respectively. T Indicates transpose.

7. The visual dialogue method based on external knowledge enhancement and semantic consistency according to claim 1, characterized in that: The fusion of the instance-level representation and the common-sense-level representation is achieved through weighted summation.

8. The visual dialogue method based on external knowledge enhancement and semantic consistency according to claim 1, characterized in that: Based on the fusion features, the process of obtaining the answer to the visual dialogue through reasoning includes the following steps: Based on the fusion features, the discriminant model and the generative model are trained with the goal of minimizing the loss function value, wherein the loss function includes one or more of a masked language modeling loss, a masked region modeling loss, a next sentence prediction loss, a KL divergence loss, and a bidirectional triple ranking loss; Based on the discriminant model, the candidate answers are ranked by calculating the semantic consistency between the candidate answers and the current question and the historical dialogue, and the optimal answer is obtained by reasoning; Based on the generative model, accurate and coherent answers are generated according to the understanding of the dialogue context and visual scene.

9. An electronic device, characterized in that: include: One or more processors and a memory, wherein the memory stores one or more programs, wherein the one or more programs include instructions for executing the visual dialogue method based on external knowledge enhancement and semantic consistency as described in any one of claims 1-8.

10. A visual dialogue device based on external knowledge enhancement and semantic consistency, characterized in that: include: The knowledge graph construction module is used to construct a multimodal scene graph based on the acquired images, historical conversation texts, and current question texts by extracting the associations between visual objects in the domain and the associations between text words in the domain, and to construct a common sense knowledge base by extracting common sense knowledge triples; A scene graph enhancement module, used to expand the multimodal scene graph and perform relationship enhancement learning based on the common sense knowledge base to obtain features of associated enhanced nodes; A bilateral knowledge fusion and reasoning strategy module is used to obtain instance-level representations based on the image, historical conversation text, current question text, and candidate answer text using a pre-trained large language model, obtain text common sense-level representations based on the features of the associated enhancement nodes, and perform feature fusion on the instance-level representations and the common sense-level representations to obtain global fusion features; The answer reasoning module is used to obtain the answer to the visual dialogue through reasoning based on the fused features.

Images