A semantically rich dialogue generation method integrating visual context

Abstract

This paper discloses a method for generating semantically rich dialogues that integrates visual context. Challenging audiovisual scene perception datasets are collected for model training. The overall model, based on the Transformer, designs and implements a multi-step cross-modal attention mechanism, capturing heterogeneous semantic associations between different modalities in spatiotemporal dimensions in a fine-grained manner. Multimodal feature representations are then jointly constructed into a spatiotemporal graph structure, which is then used for cross-modal learning and reasoning using a graph convolutional network. Finally, the method decodes and generates rich and accurate dialogue responses that are consistent with the current context. By integrating multimodal data and conducting cross-modal interactions, the present invention captures multi-angle, fine-grained, progressive feature interactions and semantic associations between modalities, achieving visual-language cross-modal semantic alignment, improving the model's semantic understanding and reasoning capabilities, and ultimately generating informative and high-quality responses.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning technology, and specifically relates to a method for generating semantically rich dialogue by integrating visual context. Background Art

[0002] Amidst the rapid development of artificial intelligence, achieving natural interaction between machines and humans remains a challenging task. One key reason is that existing dialogue systems are mostly based on unimodal text. While text is an important means for humans to express information and communicate with each other, the real world is inherently multimodal. In addition to text, people continuously consume multimodal information such as video, audio, and images, and develop more personalized and diverse processing and interaction methods tailored to specific scenarios. Therefore, audiovisual and visual learning are essential capabilities for intelligent systems to achieve natural interaction. Dialogue systems based on multimodal data (text, images, video, etc.) will be a key path to achieving truly human-like dialogue systems. By understanding visual context, dialogue systems can generate informative and high-quality responses, enabling more natural conversational interactions with humans. In the future, multimodal dialogue systems will be an integral component of numerous complex AI systems and applications. For example, navigation systems for the blind can help visually impaired users understand their surroundings and social media content through dialogue, improving their quality of life.

[0003] Compared to text- or voice-based dialogue systems, visual context-integrated human-computer dialogue systems aim to understand and reason based on the given visual context input (video content) and the natural language textual dialogue content (historical dialogues, video summaries, current questions), ultimately generating a natural language response corresponding to the question. This entire process requires not only processing complex visual scene information but also considering the linguistic information in the historical context and fully integrating the information from both modalities to generate accurate and fluent dialogue content that aligns with the current visual scene. The challenge lies in integrating these key technologies with different focuses to help dialogue systems acquire contextual sensitivity and cognitive reasoning capabilities, thereby enabling a deeper and more comprehensive understanding of the real world and achieving more harmonious human-computer interaction. Summary of the Invention

[0004] In order to overcome the shortcomings of the existing technology, the present invention provides a method for generating rich semantic dialogues that integrates visual contexts. Challenging audio-visual scene perception datasets are collected to train the model. The overall model designs and implements a multi-step cross-modal attention mechanism based on the Transformer, which captures heterogeneous semantic associations between different modalities in the spatiotemporal dimension in a fine-grained manner. The multimodal feature representations are then jointly constructed into a spatiotemporal graph structure and a graph convolutional network is used for cross-modal learning and reasoning. Finally, the multimodal feature representations are decoded to generate rich and accurate dialogue responses that are consistent with the current context. The present invention captures multi-angle, fine-grained progressive feature interactions and semantic associations between modalities through the fusion and cross-modal interaction of multimodal data, achieves visual-language cross-modal semantic alignment, improves the semantic understanding and reasoning capabilities of the model, and ultimately generates information-rich and high-quality responses.

[0005] The technical solution adopted by the present invention to solve the technical problem includes the following steps:

[0006] Step 1: Collect open-source datasets related to video conversations, preprocess the data, and divide it into training, validation, and test sets;

[0007] Step 2: Data preprocessing;

[0008] There are four types of multimodal feature representations in video content: static video features, dynamic video features, current question features, and text features that combine historical conversations and summary information. Position encoding is added during feature extraction to obtain the final model input representation as follows:

[0009]

[0010]

[0011] Where PE(pos,2i) represents the value of the 2i-th dimension of the pos-th word in the sentence sequence, and PE(pos,2i+1) represents the value of the 2i+1-th dimension of the pos-th word in the sentence sequence;

[0012] Step 3: Model construction;

[0013] (1) First, a dialogue system based on a codec architecture is constructed. By using a pre-trained model to extract features from video frames, static and dynamic video semantic information is obtained. This information is then encoded separately from the dialogue text content, and fine-grained intra-modal contextual semantic information is modeled.

[0014] (2) Using a cross-modal multi-step attention mechanism to fuse multimodal data and perform cross-modal interaction, it captures multi-angle fine-grained progressive feature interactions and inter-modal semantic associations, and achieves visual-linguistic cross-modal semantic alignment;

[0015] (3) Multimodal feature representations are jointly constructed into a spatiotemporal graph structure. After performing graph reasoning based on dynamic spatiotemporal scenarios to obtain cross-modal fusion features, they are decoded to generate dialogue responses.

[0016] Step 4: Coding phase;

[0017] For encoding, four standard Transformer encoders are used to semantically encode input features from different modalities, including video static features, video dynamic features, historical conversation and video summary features, and current question features. The encoders used for historical conversation and video summary features and current question features, both belonging to the text modality, share weights.

[0018] First, the word vectors in the sentence sequence are updated according to the context through the multi-head attention module as follows:

[0019] MultiHead(Q,K,V)=Concat(head1,head2,...head h )W O

[0020] head i =Attention(QW i Q , KW i K , VW i V )

[0021]

[0022] Where Q, K, and V are represented by three different weight matrices W i Q 、W i K 、W i V Multiplying with the model input vector, the dimensions of the three weight matrices are all d k , head i Represents an attention head in the multi-head attention mechanism;

[0023] Then the output of the encoding phase is obtained through the feedforward neural network layer as follows:

[0024] FFN(Z)=max(0,Z,W1+b1)W2+b2

[0025] Where Z represents the output of the multi-head attention layer, W1 and W2 represent the learning weights of the feedforward neural network; b1 and b2 represent the biases;

[0026] The multi-head attention layer and feedforward neural network layer in the encoding phase are followed by residual connections and layer normalization, as follows:

[0027] SubLayer output =LayerNorm(x+(SubLayer(x))

[0028] Where SubLayer refers to the multi-head attention layer or feedforward neural network layer, and x represents the input;

[0029] Step 5: Multi-step cross-modal attention mechanism;

[0030] The attention mechanism is extended from single-step to multi-step. In different attention calculations, the query vector is derived from one of the modal type data in turn, and the other three modal type data are used to calculate the key vector and value vector respectively, realizing the cross attention between one feature dimension and the other three dimensions. For each attention mechanism calculation, a strategy of repeated interaction is adopted, with three groups of six cross-modal attention calculations. The specific update and operation formulas of the multi-step cross-attention calculation are as follows:

[0031] a (t) =softmax(qLM (t-1)T )

[0032] M (t) ←(a (t) ⊙M (t-1) )W (t) +b (t)

[0033] Among them, a (t) Indicates M (t-1) The attention weight, T is the number of steps, ← represents the update operation, W (t) and b (t) is a learnable parameter of the linear FC layer; q represents the query vector, and L represents the remaining modal vectors;

[0034] Similarly, the interaction between the current modality and the remaining modalities is calculated, all the calculated interactions are concatenated and mapped to the same dimension through the linear layer, and finally added to the current modality feature to obtain the updated feature representation of the current modality after fusion with other modalities;

[0035] Step 6: Cross-modal reasoning;

[0036] Graph convolutional networks (GCNs) are used for cross-modal reasoning. In the previous feature representation and fusion steps, cross-embedded features of the language and visual modalities are obtained. An undirected heterogeneous graph is constructed for this feature, i.e., a heterogeneous input matrix containing all visual and language vectors. Graph-based alignment is performed on the heterogeneous graph to obtain a cross-modal aligned adjacency matrix weighted by semantic similarity. GCNs are then used to perform relational reasoning on the graph. Specifically, node values are updated through linear transformations of adjacent nodes and the node itself in the graph. The weights of the edges in the graph are specified by the aligned adjacency matrix. A layer of GCN is represented as follows:

[0037] X (f) =A (f) X (f-1) W (f)

[0038] where X (f) is the hidden feature of the fth layer of GCN, A (f) By X (f-1) The adjacency matrix calculated by the dot product similarity of the node features, W (f) is a learnable weight matrix;

[0039] Step 7: Decoding stage;

[0040] The input of the decoding part passes through a multi-head masked attention mechanism and an encoder-decoder attention mechanism with the same structure, and finally passes through a feedforward neural network layer to generate the final reply sequence. In the decoding stage, each sublayer is also followed by a residual connection and layer normalization process. The model parameters are learned by minimizing the negative log-likelihood loss of the generated sequence, resulting in a multi-round dialogue content generation model as follows:

[0041]

[0042] where t1, ..., t i Represents the i-th word in the generated sentence sequence, D KL represents KL divergence, p(t i |t1,...,t i-1 , x) represents the probability of generating a word at the current moment, and n represents the maximum generation length.

[0043] Preferably, the data set is an Audio Visual Scene-Aware Dialog data set.

[0044] The beneficial effects of the present invention are as follows:

[0045] This paper models a visual-context fusion dialogue system based on the Transformer architecture. It designs a multi-step cross-modal attention mechanism to fuse and interact with spatiotemporal features of video, audio, and text modal data across modalities, improving the system's scene perception and ability to locate key information. GCN is then used for cross-modal reasoning to obtain a fine-grained multimodal graph representation for subsequent decoding, enabling the system to generate information-rich, semantically accurate, and natural dialogue content. Experimental results demonstrate that the proposed video dialogue model achieves superior performance across various evaluation metrics, effectively capturing the core of the question and fully utilizing and combining multimodal contextual information to correlate and locate relevant content in other modalities. Ultimately, it generates information-rich, accurate, and context-appropriate dialogue to answer questions. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a system architecture diagram of the overall model of the present invention.

[0047] Figure 2 This is a multimodal dialogue flow chart of an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The present invention will be further described below with reference to the accompanying drawings and examples.

[0049] This paper proposes a method for generating semantically rich dialogues by integrating visual context. This method is based on the following principles: integrating visual context information in a pervasive environment helps the dialogue system gain a deeper understanding of the dialogue context, further enabling harmonious and natural human-computer interaction: (1) Visual context contains rich semantic association information for dialogues; (2) it enhances the intelligent system's ability to interact naturally with others, visually, and auditorily; and (3) through the fusion of multimodal data and cross-modal interaction, it captures multi-angle, fine-grained, progressive feature interactions and inter-modal semantic associations, achieving visual-linguistic cross-modal semantic alignment, improving the model's semantic understanding and reasoning capabilities, and ultimately generating informative and high-quality responses.

[0050] like Figure 1 As shown, a method for generating semantically rich dialogue by integrating visual context includes the following steps:

[0051] Step 1: Collect open-source datasets related to video conversations, preprocess the data, and divide it into training sets, validation sets, and test sets to support subsequent model training.

[0052] Step 2: Data Preprocessing. Since video content changes dynamically in both time and space, different time frames have different spatial features (appearance features), and adjacent time frames have spatiotemporal motion features (motion features). Furthermore, there are semantic features between question input words (word features), and coreference and semantic dependency features between multiple rounds of conversation history (history features). Therefore, there are four types of multimodal feature representations: static video features, dynamic video features, current question features, and text features that combine historical conversations and summary information. Corresponding methods and models are required for feature extraction, and position encoding is added, as follows:

[0053]

[0054]

[0055] Where PE(pos,2i) represents the value of the 2i-th dimension of the pos-th word in the sentence sequence, and PE(pos,2i+1) represents the value of the 2i+1-th dimension of the pos-th word in the sentence sequence; the input representation of the final model is obtained.

[0056] Step 3: Model construction. (1) First, a dialogue system based on the encoder-decoder architecture is constructed. By using a pre-trained model to extract features from the frame information in the video, static and dynamic video semantic information is obtained, which is then encoded separately with the dialogue text content, and fine-grained intra-modal contextual semantic information is modeled. (2) A multi-step attention mechanism based on cross-modality is further used to fuse multimodal data and interact across modalities, capturing multi-angle fine-grained progressive feature interactions and semantic associations between modalities, achieving visual-language cross-modal semantic alignment, and improving the model's semantic understanding and reasoning capabilities. (3) Finally, the multimodal feature representation is jointly constructed into a spatiotemporal graph structure. After performing graph reasoning based on dynamic spatiotemporal scenes to obtain cross-modal fusion features, the system decodes and generates rich, diverse, and context-appropriate dialogue responses, achieving the goal of natural interaction with humans. The model achieves comparable results to the current state-of-the-art models on audio-visual scene perception datasets.

[0057] Step 4: Encoding. Four standard Transformer encoders are used to semantically encode input features from different modalities, including video static features, video dynamic features, historical conversation and video summary features, and current question features. The encoders used for two types of features belonging to the same text modality share weights.

[0058] First, the word vectors in the sentence sequence are updated according to the context through the multi-head attention module as follows:

[0059] MultiHead(Q,K,V)=Concat(head1,head2,...head h )W O

[0060] head i =Attention(QW i Q , KW i K , VW i V )

[0061]

[0062] Among them, Q, K, and V are obtained by multiplying three different weight matrices with the model input vector. i Represents an attention head in a multi-head attention mechanism.

[0063] Then the output of the encoding phase is obtained through the feedforward neural network layer as follows:

[0064] FFN(Z)=max(0,Z,W1+b1)W2+b2

[0065] Where Z represents the output content of the multi-head attention layer.

[0066] The multi-head attention layer and feedforward neural network layer in the encoding phase are followed by residual connections and layer normalization, as follows:

[0067] SubLayer output =LayerNorm(x+(SubLayer(x))

[0068] SubLayer refers to the multi-head attention layer or feedforward neural network layer.

[0069] Step 5: Multi-step cross-modal attention mechanism. In addition to understanding the semantics of the video content and the question itself through self-attention, it is also necessary to explore the relationships between different modalities to establish cross-modal semantic associations. However, single-step attention can only extract partially relevant information and lacks the ability to understand complex semantics and infer answers. Therefore, this paper expands the attention mechanism from a single-step to a multi-step one, proposing a multi-step cross-modal attention mechanism. Leveraging multi-perspective and progressive attention, this model can capture more fine-grained relationships between different modalities. By focusing on content associations between sequences from different sources, it achieves cross-modal attention. Specifically, in each attention calculation, the query vector is derived from one modal data, and the remaining three modal data are used to calculate the key vector and value vector, respectively. This achieves cross-attention between one feature dimension and the remaining three dimensions. "Multi-step" refers to the fact that for each attention mechanism calculation, a two-step iterative interaction strategy is employed, resulting in three sets of six cross-modal attention calculations to capture deep connections between the cross-modal data, focusing on relevant information between different modalities and helping subsequent decoding generate accurate and informative answers. The specific update and operation formulas for the multi-step cross-attention calculation are as follows.

[0070] a (t) =softmax(qLM (t-1)T )

[0071] M (t) ←(a (t) ⊙M (t-1) )W (t) +b (t)

[0072] Among them, a (t) Indicates M (t-1) The attention weight, T is the number of steps, ← represents the update operation, W (t) and b (t) M is the learnable parameter of the linear FC layer. (t) It is the dense interaction between the two modalities after attention calculation. Similarly, the interaction between the current modality and the remaining modalities can be calculated. All the calculated interactions are concatenated and mapped to the same dimension through the linear layer. Finally, they are added to the current modality features to obtain the current modality feature representation after fusion and update with other modalities.

[0073] Step 6: Cross-modal reasoning. In order to capture the deep dynamic scene change information in the video, the present invention uses a graph convolutional network (GCN) to perform cross-modal reasoning. In the previous feature representation and fusion steps, the cross-embedded features of the language modality and the visual modality are obtained. Subsequently, an undirected heterogeneous graph is constructed on this basis, that is, a heterogeneous input matrix containing all visual and language vectors. Graph-based alignment is performed in the heterogeneous graph to obtain a cross-modal aligned adjacency matrix weighted by semantic similarity, and GCN is further used to perform relational reasoning on the graph. Specifically, the node values are updated by linear transformation of adjacent nodes and the nodes themselves in the graph. The weights of the edges in the graph are specified by the aligned adjacency matrix. In order to merge the input signals of the graph, the present invention represents a layer of GCN as follows:

[0074] X (f) =A (f) X (f-1) W (f)

[0075] where X (f) is the hidden feature of the fth layer of GCN. A (f) By X (f-1) The adjacency matrix is calculated by the dot product similarity of the node features in W. (f) is a learnable weight matrix.

[0076] Step 7: Decoding. After graph reasoning, the most relevant information for the current question is obtained for subsequent decoding. The decoding input passes through a multi-head masked attention mechanism and an encoder-decoder attention mechanism with the same structure, and finally passes through a feedforward neural network layer to generate the final response sequence. Each sublayer in the decoding phase is also followed by a residual connection and layer normalization process. The model parameters are learned by minimizing the negative log-likelihood loss of the generated sequence, resulting in a multi-round dialogue content generation model as follows:

[0077]

[0078] where t1, ..., t i Represent the i-th word in the generated sentence sequence. Specific embodiment:

[0080] 1. If Figure 2As shown, we collected open-source datasets related to video conversations, preprocessed the data, and divided it into training, validation, and test sets to support subsequent model training. Taking the Audio Visual Scene-Aware Dialog (AVSD) dataset as an example, we divided it into a training set of 7,985 rounds, a validation set of 1,863 rounds, and a test set of 1,968 rounds. Each round of conversation contains 10 question-answer pairs, for a total of 11,816 rounds.

[0081] 2. Data preprocessing: For the video and text content contained in the AVSD dataset, feature extraction is performed to obtain four types of feature vectors: video static features, video dynamic features, current question features, and text features spliced from historical dialogues and summary information. During processing, each video is first divided into 40 segments, each segment contains an average of 18 frames, and one frame is sampled from each segment. For each video, 40 frames are sampled and input into the pre-trained ResNet model to extract 40 frame-level feature representations. After the features are extracted, they are spliced (with a dimension of 40*2048) as the final video static feature vector V f Video dynamic features V c It is obtained based on the features of 40 static frames using the pre-trained action model C3D (dimension is 40*2048). f and V c The linear layer is then used to unify the dimensions (40*512) and add position encoding information.

[0082] The present invention makes unified treatment for the two types of features of text modality. Before that, a vocabulary is first established for all text corpora and ID numbers are performed, in which words with a frequency lower than a certain value are filtered out. For the richness of data, the present invention constructs multiple rounds of dialogue pairs for a set of dialogue data. For example, the first round of dialogue is itself without historical information; the historical information of the second round of dialogue is the first round of dialogue, and so on. The different rounds constructed by this method still belong to the same group of dialogues, so the corresponding video information is the same. The summary information is then spliced with the multi-round dialogue history, and a word ID sequence is obtained after mapping in the vocabulary. The embedding layer is further used to obtain a 512-dimensional word embedding to form the historical feature H. The same operation is performed on the question to obtain the word-level question embedding Q. l Finally, the same is true for H and Q l The positional encoding information is calculated and added together to form the final model input representation.

[0083]

[0084]

[0085] H={h1, h2, ..., h l-1}

[0086] Q l ={q1, q2, ..., q l}

[0087] 3. In the encoding phase, the model encoding part uses four standard transformer model encoders to semantically encode the four types of input features. First, the four types of feature vectors are updated according to the context through the multi-head attention module, as follows:

[0088] MultiHead(Q,K,V)=Concat(head1,head2,...head h )W O

[0089] head i =Attention(QW i Q , KW i K , VW i V )

[0090]

[0091] Among them, Q, K, and V are obtained by multiplying three different weight matrices with the model input vector. i Represents an attention head in a multi-head attention mechanism.

[0092] Then the output of the encoding phase is obtained through the feedforward neural network layer as follows:

[0093] FFN(Z)=max(0,Z,W1+b1)W2+b2

[0094] Where Z represents the output content of the multi-head attention layer.

[0095] The multi-head attention layer and feedforward neural network layer in the encoding phase are followed by residual connections and layer normalization, as follows:

[0096] SubLayer output =LayerNorm(x+(SubLayer(x))

[0097] SubLayer refers to the multi-head attention layer or feedforward neural network layer.

[0098] 4. Multi-step cross-modal attention mechanism. After completing the self-attention and update of the four types of features, a multi-step cross-modal attention mechanism is used to focus on the content association between sequences from different sources to achieve cross-modal mutual attention, as follows:

[0099] a (t) =softmax(qLM (t-1)T )

[0100] M (t) ←(a (t) ⊙M (t-1) )W (t) +b (t)

[0101] Among them, a (t) Indicates M (t-1) The attention weight, T is the number of steps, ← represents the update operation, W (t) and b (t) M is the learnable parameter of the linear FC layer. (t) It is the dense interaction between the two modalities after attention calculation. Similarly, the interaction between the current modality and the remaining modalities can be calculated. All the calculated interactions are concatenated and mapped to the same dimension through the linear layer. Finally, they are added to the current modality features to obtain the current modality feature representation after fusion and update with other modalities.

[0102] 5. Cross-modal reasoning. In the previous feature representation and fusion steps, four types of cross-embedded features of language modality and visual modality are obtained. Subsequently, an undirected heterogeneous graph is constructed based on this feature, that is, a heterogeneous input matrix containing all visual and language vectors. Graph-based alignment is performed in the heterogeneous graph to obtain a cross-modal aligned adjacency matrix weighted by semantic similarity. GCN is further used to perform relational reasoning on the graph. Specifically, the node values are updated through linear transformations of adjacent nodes and the nodes themselves in the graph. The weights of the edges in the graph are specified by the aligned adjacency matrix. In order to merge the input signals of the graph, the present invention represents a layer of GCN as follows:

[0103] X (f) =A (f) X (f-1) W (f)

[0104] where X (f) is the hidden feature of the fth layer of GCN. A (f) By X (f-1) The adjacency matrix is calculated by the dot product similarity of the node features in W. (f) is a learnable weight matrix.

[0105] 6. Decoding phase. The decoding part of the model is similar to the encoding part. The input of the decoding phase also first undergoes word embedding and position encoding to obtain an input vector representation. The input vector is updated through a multi-head masked attention mechanism, and then passes through the encoding-decoding attention mechanism of the same structure together with the final feature representation obtained by cross-modal graph reasoning, and finally passes through the feedforward neural network layer to obtain the output of the decoding phase. In the decoding phase, each sublayer is also followed by a residual connection and layer normalization process. The parameters of the model are learned by minimizing the negative log-likelihood function loss of the generated sequence to obtain a multi-round dialogue content generation model, as follows:

[0106]

[0107] where t1,…,t i Represent the i-th word in the generated sentence sequence.

Claims

1. A semantically rich dialogue generation method integrating visual context, characterized by: The following steps are involved: Step 1: Collect open-source datasets related to video conversations, preprocess the data, and divide it into training, validation, and test sets; Step 2: Data preprocessing; There are four types of multimodal feature representations in video content: static video features, dynamic video features, current question features, and text features that combine historical conversations and summary information. Position encoding is added during feature extraction to obtain the final model input representation as follows: Where PE(pos, 2i) represents the value of the 2i-th dimension of the pos-th word in the sentence sequence, and PE(pos, 2i+1) represents the value of the 2i+1-th dimension of the pos-th word in the sentence sequence; Step 3: Model construction; (1) First, a dialogue system based on a codec architecture is constructed. By using a pre-trained model to extract features from video frames, static and dynamic video semantic information is obtained. This information is then encoded separately from the dialogue text content, and fine-grained intra-modal contextual semantic information is modeled. (2) Using a cross-modal multi-step attention mechanism to fuse multimodal data and perform cross-modal interaction, it captures multi-angle fine-grained progressive feature interactions and inter-modal semantic associations, and achieves visual-linguistic cross-modal semantic alignment; (3) Multimodal feature representations are jointly constructed into a spatiotemporal graph structure. After performing graph reasoning based on dynamic spatiotemporal scenarios to obtain cross-modal fusion features, they are decoded to generate dialogue responses. Step 4: Coding phase; For encoding, four standard Transformer encoders are used to semantically encode input features from different modalities, including video static features, video dynamic features, historical conversation and video summary features, and current question features. The encoders used for historical conversation and video summary features and current question features, both belonging to the text modality, share weights. First, the word vectors in the sentence sequence are updated according to the context through the multi-head attention module as follows: MultiHead(Q,K,V)=Concat(head1,head2,...head h )W O head i =Attention(QW i Q ,KW i K ,VW i V ) Where Q, K, and V are represented by three different weight matrices W i Q 、W i K 、W i V Multiplying with the model input vector, the dimensions of the three weight matrices are all d k , head i Represents an attention head in the multi-head attention mechanism; Then the output of the encoding phase is obtained through the feedforward neural network layer as follows: FFN(Z)=max(0,Z,W1+b1)W2+b2 Where Z represents the output of the multi-head attention layer, W1 and W2 represent the learning weights of the feedforward neural network; b1 and b2 represent the biases; The multi-head attention layer and feedforward neural network layer in the encoding phase are followed by residual connections and layer normalization, as follows: SubLayer output =LayerNorm(x+(SubLayer(x)) Where SubLayer refers to the multi-head attention layer or feedforward neural network layer, and x represents the input; Step 5: Multi-step cross-modal attention mechanism; The attention mechanism is extended from single-step to multi-step. In different attention calculations, the query vector is derived from one of the modal type data in turn, and the other three modal type data are used to calculate the key vector and value vector respectively, realizing the cross attention between one feature dimension and the other three dimensions. For each attention mechanism calculation, a strategy of repeated interaction is adopted, with three groups of six cross-modal attention calculations. The specific update and operation formulas of the multi-step cross-attention calculation are as follows: a (t) =softmax(qLM (t-1)T ) M (t) ←(a (t) ⊙M (t-1) )W (t) +b (t) Among them, a (t) Indicates M (t-1) The attention weight, T is the number of steps, ← represents the update operation, W (t) and b (t) is a learnable parameter of the linear FC layer; q represents the query vector, and L represents the remaining modal vectors; Similarly, the interaction between the current modality and the remaining modalities is calculated, all the calculated interactions are concatenated and mapped to the same dimension through the linear layer, and finally added to the current modality feature to obtain the updated feature representation of the current modality after fusion with other modalities; Step 6: Cross-modal reasoning; Graph convolutional networks (GCNs) are used for cross-modal reasoning. In the previous feature representation and fusion steps, cross-embedded features of the language and visual modalities are obtained. An undirected heterogeneous graph is constructed for this feature, i.e., a heterogeneous input matrix containing all visual and language vectors. Graph-based alignment is performed on the heterogeneous graph to obtain a cross-modal aligned adjacency matrix weighted by semantic similarity. GCNs are then used to perform relational reasoning on the graph. Specifically, node values are updated through linear transformations of adjacent nodes and the node itself in the graph. The weights of the edges in the graph are specified by the aligned adjacency matrix. A layer of GCN is represented as follows: X (f) =A (f) X (f-1) W (f) where X (f) is the hidden feature of the fth layer of GCN, A (f) By X (f-1) The adjacency matrix calculated by the dot product similarity of the node features, W (f) is a learnable weight matrix; Step 7: Decoding stage; The input of the decoding part passes through a multi-head masked attention mechanism and an encoder-decoder attention mechanism with the same structure, and finally passes through a feedforward neural network layer to generate the final reply sequence. In the decoding stage, each sublayer is also followed by a residual connection and layer normalization process. The model parameters are learned by minimizing the negative log-likelihood loss of the generated sequence, resulting in a multi-round dialogue content generation model as follows: where t1, ..., t i Represents the i-th word in the generated sentence sequence, D KL represents KL divergence, p(t i |t1,...,t i-1 , x) represents the probability of generating a word at the current moment, and n represents the maximum generation length.

2. The method for generating semantically rich dialogue integrating visual context according to claim 1, characterized in that: The dataset is an Audio Visual Scene-Aware Dialog dataset.

Images