Visual dialogue methods and devices based on external knowledge enhancement and semantic consistency

Abstract

This invention relates to a visual dialogue method and device based on external knowledge enhancement and semantic consistency. The method includes: constructing a multimodal scene graph by extracting the relationships between visual objects and words in the domain based on acquired images, historical dialogue text, and current question text; constructing a common sense knowledge base by extracting common sense knowledge triples; expanding the multimodal scene graph based on the common sense knowledge base and performing relationship enhancement learning to obtain features of the association enhancement nodes; obtaining instance-level representations using a pre-trained large language model; obtaining text common sense-level representations based on the features of the association enhancement nodes; fusing the instance-level and common sense-level representations to obtain global fused features; and obtaining the answer to the visual dialogue through reasoning based on the fused features. This invention can meticulously mine semantic relationships, significantly improving the accuracy, richness, and comprehensiveness of multi-turn visual dialogues.

Description

Technical Field

[0001] This invention relates to the field of visual reasoning technology, and in particular to a visual dialogue method and device based on external knowledge enhancement and semantic consistency. Background Technology

[0002] Multimodal data encompasses rich types, diverse sources, and complex relational structures. How to quickly and efficiently utilize this diverse data to serve humanity has become a hot research topic. Currently, the application of multimodal data is still in its early stages, and related theories and technologies are constantly evolving. Improving the automatic perception and understanding of multimodal data and exploring the semantic connections between different modalities is a challenge with significant application prospects and research significance. Among these, visual dialogue, as a key cross-modal understanding task, enables AI agents to conduct multi-turn question-and-answer sessions based on visual information and historical dialogue content, making it one of the core technologies for achieving intelligent human-computer interaction. The potential applications of this technology include intelligent customer service, entertainment games, smart homes, healthcare, and intelligent manufacturing, demonstrating broad development prospects.

[0003] In visual dialogue tasks, existing methods have significant shortcomings in fine-grained modeling of visual and textual information, and struggle to provide accurate responses in complex scenarios. On one hand, current methods largely overemphasize modeling complex relationships between textual elements while neglecting the diverse relationships between visual objects. This bias limits the model's ability to fully utilize visual details when answering questions, leading to limitations in understanding and parsing visual information. On the other hand, most methods struggle to effectively maintain semantic consistency within and between modalities when extracting fine-grained semantic relationships and transmitting multimodal information. This deficiency not only affects the model's reasoning ability but also restricts its performance in practical applications. Therefore, developing a fine-grained modeling method that comprehensively considers relationships between visual objects and textual associations while ensuring semantic consistency is crucial. This will help improve the overall performance of visual dialogue systems, enabling them to more accurately understand user intent and provide more accurate and richer responses in complex scenarios. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art by providing a visual dialogue method and device based on external knowledge enhancement and semantic consistency, so as to solve or partially solve the problems of lack of consideration for the diverse relationships between visual objects and difficulty in effectively maintaining semantic consistency within and between modalities.

[0005] The objective of this invention can be achieved through the following technical solutions:

[0006] One aspect of the present invention provides a visual dialogue method based on external knowledge enhancement and semantic consistency, comprising the following steps:

[0007] Based on the acquired images, historical dialogue texts, and current question texts, a multimodal scene graph is constructed by extracting the relationships between visual objects and words in the domain, and a common sense knowledge base is constructed by extracting common sense knowledge triples.

[0008] Based on the aforementioned common sense knowledge base, the multimodal scene graph is expanded and relation enhancement learning is performed to obtain the features of the associated enhancement nodes;

[0009] Based on the image, historical dialogue text, current question text, and candidate answer text, an instance-level representation is obtained using a pre-trained large language model. Based on the features of the associated enhancement nodes, a common-sense-level representation of the text is obtained. Feature fusion is performed on the instance-level representation and the common-sense-level representation to obtain a global fused feature.

[0010] Based on the fusion features, the answer to the visual dialogue is obtained through reasoning.

[0011] As a preferred technical solution, the multimodal scene graph includes a visual sub-scene graph and a text sub-scene graph. The process of constructing the visual scene graph and the text sub-scene graph includes the following steps:

[0012] Based on the acquired image, a visual sub-scene graph including object nodes and edges is extracted, wherein each object node includes a text label;

[0013] For the historical dialogue text and the current question text, meaningless words are filtered out, and word entities are used as nodes and text relationships are used as edges to obtain a text sub-scene graph.

[0014] As a preferred technical solution, the construction process of the common sense knowledge base includes the following steps:

[0015] Extract <subject, predicate, object> tuples from the pre-acquired dataset, retain the <subject, object> pairs with the highest co-occurrence frequency, and construct a common sense knowledge base.

[0016] As a preferred technical solution, the process of expanding the multimodal scene graph and performing relation enhancement learning based on the common sense knowledge base to obtain local explicit structured features includes the following steps:

[0017] For each sub-scene graph in the multimodal scene graph, queries are performed on the nodes of the sub-scene graph. By calculating the relevance score with the nodes in the common sense knowledge base, nodes with high scores and their associations are integrated into the visual scene graph, thereby expanding the multimodal scene graph.

[0018] For each sub-scene graph in the expanded multimodal scene graph, node features are extracted using a pre-trained CNN network model, and edge features are extracted using a pre-trained BERT model to encode the expanded multimodal scene graph.

[0019] For each sub-scene graph in the encoded multimodal scene graph, construct an adjacency matrix representing the frequency of different types of relationships between node pairs;

[0020] Based on the adjacency matrix, features of associated enhanced nodes are extracted using a graph convolutional neural network, enabling the understanding and fusion of multimodal features.

[0021] As a preferred technical solution, the process of obtaining implicit features using a pre-trained large language model based on the image, historical dialogue text, current question text, and candidate answer text includes the following steps:

[0022] The text sequence is obtained by concatenating the image title text of the current dialogue round, the text of the historical dialogue, the text of the current question, and the text of multiple candidate answers;

[0023] Using a pre-trained CNN network model, the images of the current dialogue round are encoded to obtain a sequence of image objects;

[0024] Based on the text sequence and image object sequence, an instance-level implicit visual-text representation is extracted using a pre-trained large language model.

[0025] As a preferred technical solution, the process of obtaining a commonsense-level representation of text based on the features of the associated enhanced nodes includes the following steps:

[0026] Based on the instance-level representation and the features of the association enhancement nodes, the salience score of each association enhancement node is calculated using the following formula:

[0027]

[0028] Based on the saliency score, the commonsense-level representation of the text is obtained using the following formula:

[0029]

[0030] in, and Representing the associated enhanced nodes respectively and The corresponding significance score, and It is a learnable parameter matrix, where λ is a hyperparameter controlling the smoothness of the soft maximum function, and V h T h These are the visual hidden state and the text hidden state in the instance-level representation, respectively, with superscripts.T This indicates transpose.

[0031] As a preferred technical solution, the fusion of the instance-level representation and the common sense-level representation is achieved by weighted summation.

[0032] As a preferred technical solution, the process of obtaining the answer to the visual dialogue through reasoning based on the fusion features includes the following steps:

[0033] Based on the fusion features, a discriminative model and a generative model are trained with the goal of minimizing the loss function value. The loss function includes one or more of the following: masked language modeling loss, masked region modeling loss, next sentence prediction loss, KL divergence loss, and bidirectional triple ranking loss.

[0034] Based on a discriminative model, the candidate answers are ranked by calculating the semantic consistency between the candidate answers and the current question and historical dialogues, and the optimal answer is obtained through reasoning.

[0035] Based on generative models, accurate and coherent answers are generated according to the understanding of dialogue context and visual scene.

[0036] In another aspect, an electronic device is provided, comprising: one or more processors and a memory, wherein the memory stores one or more programs, the one or more programs including instructions for executing the aforementioned visual dialogue method based on external knowledge enhancement and semantic consistency.

[0037] Another aspect of the present invention provides a visual dialogue device based on external knowledge enhancement and semantic consistency, comprising:

[0038] The knowledge graph construction module is used to construct a multimodal scene graph by extracting the relationships between visual objects and text words within the domain based on the acquired images, historical dialogue text, and current question text, and to construct a common sense knowledge base by extracting common sense knowledge triples.

[0039] The scene graph enhancement module is used to expand the multimodal scene graph based on the common sense knowledge base and perform relationship enhancement learning to obtain the features of the associated enhancement nodes;

[0040] The bilateral knowledge fusion and reasoning strategy module is used to obtain instance-level representations based on the image, historical dialogue text, current question text, and candidate answer text using a pre-trained large language model, obtain text common sense-level representations based on the features of the association enhancement nodes, and perform feature fusion on the instance-level representations and the common sense-level representations to obtain global fused features.

[0041] The answer reasoning module is used to obtain the answer to the visual dialogue based on the fused features.

[0042] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0043] (1) Fine-grained balanced modeling: By comprehensively considering the relationship between visual objects and the text association, this invention improves the model's ability to utilize visual details, thus having a significant advantage in understanding and parsing visual information. It can also effectively model visual and text information in a balanced manner at a fine-grained level, solving the shortcomings of existing methods in this regard.

[0044] (2) Semantic consistency maintenance: In the process of extracting fine-grained semantic relationships and transmitting multimodal information, this invention can better maintain semantic consistency within and between modalities. This feature not only improves the model's reasoning ability, but also enhances its practical application performance in complex scenarios.

[0045] (3) Improved responsiveness in complex scenarios: With the assistance of external knowledge, the design of this invention enables it to better understand user intent in complex scenarios. This capability allows the visual dialogue system to perform better when handling challenging tasks, providing more accurate and richer responses. Attached Figure Description

[0046] Figure 1 This is a flowchart of the visual dialogue method based on external knowledge enhancement and semantic consistency in the embodiment.

[0047] Figure 2 This is a schematic diagram of the visual dialogue framework in the embodiment;

[0048] Figure 3 This is a schematic diagram of a visual dialogue device based on external knowledge enhancement and semantic consistency in the embodiment.

[0049] Figure 4 This is a schematic diagram of the electronic device in the embodiment. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0051] Example 1

[0052] To address the problems existing in the aforementioned prior art, this embodiment provides a visual dialogue method based on external knowledge enhancement and semantic consistency, which improves human-computer interaction capabilities through balanced fine-grained modeling and multi-level semantic consistency reasoning.

[0053] See Figure 1 This method includes the following steps S1-S4, and each step will be explained below.

[0054] Step S1, Construct a multimodal scene graph and common sense knowledge base: Construct a multimodal scene graph and common sense knowledge base to obtain fine-grained associations between visual objects in the domain, between text words in the domain, and common sense knowledge triples.

[0055] like Figure 2 As shown, the knowledge graph construction module includes constructing a visual scene graph, a text scene graph, and a common sense knowledge base, respectively obtaining fine-grained associations between visual objects within the domain, between text words within the domain, and common sense knowledge triples. Specifically, step S1 may include steps S101-S103.

[0056] Step S101: Given image I, extract the visual scene graph (VSG) using the scene graph generator Neural Motifs. To display a structured image context, in which It is a collection of objects. This represents the set of VSG edges. Additionally, each object node has a word label (i.e., a text label), providing extra linguistic detail beyond visual features.

[0057] Step S102: Based on historical dialogues and the current question, and considering a rich vocabulary and meaningless terms, remove infrequently occurring words (such as "is", "a", etc.). Then, n ot Each word is treated as a graph node, and the Text Scene Graph (TSG) is obtained using [method name missing]. in and These represent the set of word entities and the set of text relations, respectively.

[0058] Step S103: To reduce the semantic and relational differences between visual and textual graphs, useful knowledge is extracted from the VisualGenome dataset to construct a commonsense knowledge base (CKB). Specifically, tuples from the VisualGenome dataset are first selected.<subject,predicate,object> (i.e., <subject, predicate, object> tuples), then retain the k pairs with the highest co-occurrence frequency.<subject,object> These preserved pairs constitute the knowledge base. in and These are the subject, predicate, and object sets, respectively.

[0059] Step S2 involves expanding the multimodal scene graph and performing relation enhancement learning to obtain features for the associated enhanced nodes: For the multimodal scene graph and the commonsense knowledge base, explicit structured features with local semantic consistency are obtained by expanding concept nodes and commonsense relationships.

[0060] like Figure 2 As shown, the scene graph enhancement module uses external knowledge to expand the initial multimodal scene graph with concept nodes and common sense relationships. Specifically, step S2 may include steps S201-S204.

[0061] Step S201, given For each query node in the CKB (Children's Block), a relevance score is calculated. The top-300 nodes with the highest scores are added to the Extended Visual Scene Graph (E-VSG), and the edges corresponding to these nodes are added to the E-VSG. Using the same method, given... For each query node, the top-300 most-scoring related nodes and their edges are added to the extended visual scene graph (E-TSG).

[0062] Step S202: Encode the nodes and edges of E-VSG and E-TSG. For E-VSG, node features are extracted using a Faster R-CNN model pre-trained on the VisualGenome dataset, and edge features are obtained using a pre-trained BERT model. To achieve a comprehensive representation of the nodes, the features of the edges associated with each node are concatenated to the node itself, ultimately obtaining the node feature representation. Where d is the dimension of the node feature, m ov This is the number of nodes in the E-VSG. Similarly, for a node with m... ot The E-TSG of each node is used to extract node features containing edge features using the BERT model, denoted as .

[0063] Step S203, based on having m ov E-VSG with m nodes and ot For each node's E-TSG, establish two directed adjacency matrices. and For each pair of nodes, the frequency of different types of relationships between them is calculated. The counts are then normalized to values ​​between 0 and 1, which are used as element values ​​in the adjacency matrix. This step ensures that the adjacency matrix captures the existence and diversity of relationships.

[0064] Step S204 involves using stacked GCNs to capture semantic relationships and learn local features for relationship enhancement, i.e., features of associated enhancement nodes. The specific operation is shown in the following formula:

[0065]

[0066] Among them, l m It is the number of stacked GCN layers. Represents the learnable weight matrix. It is a learnable bias matrix, where σ(·) represents the LeakyReLU activation function. The initial nodes are represented by... The formula for calculating the normalized adjacency matrix is ​​given below. Where D represents the degree matrix. The final output of the GCN layer. Considered a global representation of E-VSG, i.e. Each visual node is embedded as Using a similar method, a global representation of the E-TSG can be obtained. in An embedded representation of each text node.

[0067] Step S3: Extract instance-level representations and common-sense-level representations, and obtain global fused features through feature fusion: Using a pre-trained visual language large model, obtain implicitly aligned intra-domain features, and fuse the implicitly aligned features with explicit structured features through bilateral knowledge fusion and inference strategies to obtain fused features with global consistency.

[0068] The bilateral knowledge fusion and reasoning strategy integrates instance-level visual-text features and common-sense-level visual-text feature representations extracted by a pre-trained visual language large model, providing the model with globally consistent semantic information. It is divided into instance-level feature extraction, common-sense-level feature extraction, and knowledge fusion and reasoning.

[0069] Specifically, step S3 may include steps S301-S303.

[0070] Step S301, instance-level feature extraction. For a given image-question pair with dialogue turn r, extract the image title (Cap), historical dialogue (H). r Current problem Q r And candidate answer A r When connected, they form the following text sequence:

[0071] T={[CLS]Cap[SEP]Q1[SEP]A1,...,Q r [SEP]A r}

[0072] Here, [CLS] marks the start of the dialogue sequence, and [SEP] marks separate each question-and-answer pair. To encode image I, object-level feature vectors... The features v of each object are extracted by a Faster R-CNN model pre-trained on the Visual Genome dataset. i The dimension is 1024, n v =36 represents the number of targets detected. Then, for v i Mean pooling is performed to obtain a sequence of image objects starting with the [IMG] tag. Then, a pre-trained two-stream ViLBERT model is applied to jointly model the visual and textual features. The visual hidden state V... h and text hidden state T h The acquisition process is as follows:

[0073] V h T h =ViLBERT(V, T)

[0074] Step S302, Commonsense-level Feature Extraction. To acquire relevant external knowledge, instance-level visual and textual representations (i.e., V) are extracted. h and T h Using as input, relevant concept nodes are queried from E-VSG and E-TSG. Mathematically, importance scores for different semantic nodes are first calculated to measure the importance of each association-enhancing node, as shown below:

[0075]

[0076] in, and These represent semantic nodes (i.e., association enhancement nodes), and The corresponding significance score, and It is a learnable parameter matrix, where λ is a hyperparameter controlling the smoothness of the soft maximum function. Next, the visual commonsense-level representation V... c and common sense level representation of text T c The calculation formula is as follows:

[0077]

[0078] Step S303, Knowledge Fusion and Reasoning. Instance-level Representation V h (or T) h ) and common sense level representation V c (or V) h They are combined using the following simple weighted summation operation:

[0079] V f =βV h +(1-β)V c

[0080] Tf =βT h +(1-β)T c

[0081] Where β is a parameter used to adjust the balance between different representation types, V f and T f These represent the fused visual and textual representations, respectively.

[0082] Step S4: Based on the fused features, the answer to the visual dialogue is obtained through reasoning.

[0083] For the answer reasoning module, three traditional loss functions are employed: Masked Language Modeling (MLM), Masked Region Modeling (MRM), and Next Sentence Prediction (NSP). In addition, semantic consistency at both local and global levels is investigated, using the Kullback-Leibler (KL) divergence loss function and a bidirectional triple ranking loss function to ensure the model can rank relevant answers in a globally consistent manner.

[0084] Specifically, step S4 can be extended to steps S401-S405.

[0085] S401, Masked Language Modeling (MLM) loss function and Masked Region Modeling (MRM) loss function. In this application, 10% of the text region and 15% of the visual region are randomly masked by special labels [MASK]. The model needs to recover the masked labels based on the surrounding labels and cross-modal cues, under the constraints of the following formula:

[0086]

[0087] Among them, t m and v m T represents the masked word and the region marker, respectively. \m and V \m S represents unblocked words and region markers, respectively. tr This represents the training set.

[0088] S402, Next Sentence Prediction (NSP) Loss Function. The NSP loss aims to predict whether a candidate answer is correct, playing a crucial role in capturing the interdependencies between dialogues. The formula for the NSP loss is as follows:

[0089]

[0090] Where y∈{0,1} is the binary label of the truth value. It is a binary classifier used to estimate probabilities based on the dot product represented by [CLS] and [IMG] labels.

[0091] S403, Kullback-Leibler (KL) divergence loss function. The purpose of using KL divergence as a loss function is to minimize the difference between the E-VSG and E-TSG attention distributions, thereby promoting semantic consistency between textual and visual representations of the same scene content. The formula for calculating the KL loss function is:

[0092]

[0093] S404, Bidirectional Triple Ranking Loss Function. To achieve global alignment between visual and textual representations, the model employs the widely adopted bidirectional triplet ranking loss function. Given two embedding vectors from the visual scene graph and the textual scene graph, the formula for calculating global semantic relevance is:

[0094]

[0095] Where ||·|| denotes the Euclidean norm of the vector. Let V represents f and T f The ranking loss of the two-way triplet can be defined as:

[0096]

[0097] in,

[0098]

[0099] Where γ is a predefined difference parameter, (V f T f ) represents the matched visual-text positive pair, and its corresponding negative pairs are represented as follows: and

[0100] S405, the total loss function of the model is calculated using the following formula:

[0101]

[0102] Where τ1 and τ2 represent weight parameters.

[0103] To verify the performance of this method, the following experiments were designed.

[0104] Our method, SCVD+, is validated on the visual dialogue benchmark datasets VisDial v0.9 and VisDial v1.0. The performance of our method is evaluated under both discriminative and generative settings.

[0105] The visual dialogue model is trained using a benchmark dataset. The specific process of training the visual dialogue model includes:

[0106] Step 1: Given an image, a question, a historical dialogue, and 100 candidate answers;

[0107] Step 2: Construct visual scene diagrams, text scene diagrams, and common knowledge bases;

[0108] Step 3: Based on the common sense knowledge base in Step 2, expand the concept nodes and common sense relationships of the visual scene diagram and text scene diagram;

[0109] Step 4: Utilize a pre-trained visual language model to extract visual-text features of implicitly aligned instance-level features;

[0110] Step 5: Encode the expanded visual scene graph and text scene graph from Step 3, and measure the importance of each associated node in the scene graph to obtain common sense-level visual-text features;

[0111] Step 6: Employ a bilateral knowledge fusion and reasoning strategy to integrate instance-level features and common-sense features from Step 4 and Step 5, and deduce the correct answer;

[0112] Step 7: Repeat steps 1-6 multiple times to complete the multi-turn dialogue answer reasoning for all images in the dataset.

[0113] When evaluating the performance of the visual dialogue model, metrics including Mean, Mean Reciprocal Rank (MRR), Recall@k (k = 1, 5, 10) at different thresholds k, and Normalized Discounted Cumulative Gain (NDCG) were used. Mean and Mean Reciprocal Rank were calculated based on a single correct answer, while Recall and NDCG considered multiple semantically related correct answers. Among these metrics, a lower Mean Rank indicates better performance, while higher values ​​are generally considered better for the others. Specifically, NDCG is a dedicated evaluation metric for the VisDial v1.0 dataset, used to measure the generalization performance of the dialogue model; MRR is used to evaluate the accuracy of the model's answer reasoning. The test results of this method on the VisDial v0.9 and v1.0 datasets are shown in Tables 1 and 2, respectively.

[0114] Table 1 shows the performance of different models on the VisDial v0.9 dataset.

[0115]

[0116] Table 2 shows the performance of different models on the VisDial v1.0 dataset.

[0117]

[0118] In summary, this method has the following beneficial effects:

[0119] (1) This method utilizes visual and textual information through scene graphs to achieve a balanced emphasis on visual and textual information. With the assistance of external knowledge, it reduces conceptual confusion in multimodal scene graphs, enabling visual dialogue systems to understand and generate information more accurately.

[0120] (2) This method provides a two-level knowledge fusion and reasoning strategy that combines the advantages of BERT-based models in capturing implicit multimodal knowledge in large-scale pre-training with the ability to explicitly encode common sense knowledge from extended scene graphs, thereby improving the performance of handling complex common sense-based problems.

[0121] (3) This method designs a loss function to ensure local and global semantic consistency in the process of integrating knowledge within and outside the domain, reduces the impact of differences in multimodal representations, and enhances the performance of the model in joint reasoning of multimodal information.

[0122] (4) Through extensive experiments on the VisDial v0.9 and VisDial v1.0 public datasets, this method demonstrates significant superior performance, indicating its effectiveness and reliability in practical applications.

[0123] Example 2

[0124] Based on Example 1, see Figure 2 and Figure 3 This embodiment provides a visual dialogue device based on external knowledge enhancement and semantic consistency, including:

[0125] The knowledge graph construction module is used to construct a multimodal scene graph by extracting the relationships between visual objects and text words within the domain based on the acquired images, historical dialogue text, and current question text, and to construct a common sense knowledge base by extracting common sense knowledge triples.

[0126] The scene graph enhancement module is used to expand the multimodal scene graph based on the common sense knowledge base and perform relationship enhancement learning to obtain the features of the associated enhancement nodes;

[0127] The bilateral knowledge fusion and reasoning strategy module is used to obtain instance-level representations based on the image, historical dialogue text, current question text, and candidate answer text using a pre-trained large language model, obtain text common sense-level representations based on the features of the association enhancement nodes, and perform feature fusion on the instance-level representations and the common sense-level representations to obtain global fused features.

[0128] The answer reasoning module is used to obtain the answer to the visual dialogue based on the fused features.

[0129] Example 3

[0130] Based on Embodiment 1, this embodiment provides an electronic device, including: one or more processors and a memory, wherein the memory stores one or more programs, the one or more programs including instructions for executing the visual dialogue method based on external knowledge enhancement and semantic consistency as described in Embodiment 1.

[0131] like Figure 2 At the hardware level, the electronic device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for the business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to achieve the above-mentioned functions. Figure 1 The visual dialogue method described above. Of course, in addition to software implementation, this invention does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0132] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0133] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0134] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A visual dialogue method based on external knowledge enhancement and semantic consistency, characterized in that, Includes the following steps: Based on the acquired images, historical dialogue texts, and current question texts, a multimodal scene graph is constructed by extracting the relationships between visual objects and words in the domain, and a common sense knowledge base is constructed by extracting common sense knowledge triples. Based on the aforementioned common sense knowledge base, the multimodal scene graph is expanded and relation enhancement learning is performed to obtain local explicit structured features, which serve as features for association enhancement nodes. Based on the image, historical dialogue text, current question text, and candidate answer text, implicit features are obtained using a pre-trained large language model as instance-level representations. Based on the features of the associated enhancement nodes, common sense-level representations of the text are obtained. Feature fusion is performed on the instance-level representations and the common sense-level representations to obtain global fused features. Based on the fused features, the answer to the visual dialogue is obtained through reasoning. The process of expanding the multimodal scene graph and performing relation enhancement learning based on the common sense knowledge base to obtain local explicit structured features includes the following steps: For each sub-scene graph in the multimodal scene graph, queries are performed on the nodes of the sub-scene graph. By calculating the relevance score with the nodes in the common sense knowledge base, nodes with high scores and their associations are integrated into the visual scene graph, thereby expanding the multimodal scene graph. For each sub-scene graph in the expanded multimodal scene graph, node features are extracted using a pre-trained CNN network model, and edge features are extracted using a pre-trained BERT model to encode the expanded multimodal scene graph. For each sub-scene graph in the encoded multimodal scene graph, construct an adjacency matrix representing the frequency of different types of relationships between node pairs; Based on the adjacency matrix, features of associated enhanced nodes are extracted using a graph convolutional neural network, enabling the understanding and fusion of multimodal features.

2. The visual dialogue method based on external knowledge enhancement and semantic consistency according to claim 1, characterized in that, The multimodal scene graph includes a visual sub-scene graph and a text sub-scene graph. The process of constructing the visual scene graph and the text sub-scene graph includes the following steps: Based on the acquired image, a visual sub-scene graph including object nodes and edges is extracted, wherein each object node includes a text label; For the historical dialogue text and the current question text, meaningless words are filtered out, and word entities are used as nodes and text relationships are used as edges to obtain a text sub-scene graph.

3. The visual dialogue method based on external knowledge enhancement and semantic consistency according to claim 1, characterized in that, The construction process of the aforementioned common sense knowledge base includes the following steps: Extract <subject, predicate, object> tuples from the pre-acquired dataset, retain the <subject, object> pairs with the highest co-occurrence frequency, and construct a common sense knowledge base.

4. The visual dialogue method based on external knowledge enhancement and semantic consistency according to claim 1, characterized in that, The process of obtaining implicit features using a pre-trained large language model based on the image, historical dialogue text, current question text, and candidate answer text includes the following steps: The text sequence is obtained by concatenating the image title text of the current dialogue round, the text of the historical dialogue, the text of the current question, and the text of multiple candidate answers; Using a pre-trained CNN network model, the images of the current dialogue round are encoded to obtain a sequence of image objects; Based on the text sequence and image object sequence, an instance-level implicit visual-text representation is extracted using a pre-trained large language model.

5. A visual dialogue method based on external knowledge enhancement and semantic consistency according to claim 1, characterized in that, The process of obtaining a commonsense-level representation of text based on the features of the associated enhanced nodes includes the following steps: Based on the instance-level representation and the features of the association enhancement nodes, the salience score of each association enhancement node is calculated using the following formula: Based on the saliency score, the commonsense-level representation of the text is obtained using the following formula: in, and Representing the associated enhanced nodes respectively and The corresponding significance score, and It is a learnable parameter matrix, where λ is a hyperparameter controlling the smoothness of the soft maximum function, and V h T h These are the visual hidden state and text hidden state in the instance-level representation, respectively, with superscripts. T Indicates transpose, m ov m is the number of nodes in the visual sub-scene graph. ot It represents the number of nodes in the text sub-scene graph.

6. The visual dialogue method based on external knowledge enhancement and semantic consistency according to claim 1, characterized in that, The instance-level representation and the common-sense-level representation are fused by weighted summation.

7. A visual dialogue method based on external knowledge enhancement and semantic consistency according to claim 1, characterized in that, Based on the fused features, the process of obtaining the answer to the visual dialogue through reasoning includes the following steps: Based on the fusion features, a discriminative model and a generative model are trained with the goal of minimizing the loss function value. The loss function includes one or more of the following: masked language modeling loss, masked region modeling loss, next sentence prediction loss, KL divergence loss, and bidirectional triple ranking loss. Based on a discriminative model, the candidate answers are ranked by calculating the semantic consistency between the candidate answers and the current question and historical dialogues, and the optimal answer is obtained through reasoning. Based on generative models, accurate and coherent answers are generated according to the understanding of dialogue context and visual scene.

8. An electronic device, characterized in that, include: One or more processors and a memory, wherein the memory stores one or more programs, the one or more programs including instructions for executing the visual dialogue method based on external knowledge enhancement and semantic consistency as described in any one of claims 1-7.

9. A visual dialogue device based on external knowledge enhancement and semantic consistency, characterized in that, The system for implementing the visual dialogue method based on external knowledge enhancement and semantic consistency as described in any one of claims 1-7 includes: The knowledge graph construction module is used to construct a multimodal scene graph by extracting the relationships between visual objects and text words within the domain based on the acquired images, historical dialogue text, and current question text, and to construct a common sense knowledge base by extracting common sense knowledge triples. The scene graph enhancement module is used to expand the multimodal scene graph based on the common sense knowledge base and perform relationship enhancement learning to obtain the features of the associated enhancement nodes; The bilateral knowledge fusion and reasoning strategy module is used to obtain instance-level representations based on the image, historical dialogue text, current question text, and candidate answer text using a pre-trained large language model, obtain text common sense-level representations based on the features of the association enhancement nodes, and perform feature fusion on the instance-level representations and the common sense-level representations to obtain global fused features. The answer reasoning module is used to obtain the answer to the visual dialogue based on the fused features.

Images