Multi-modal dialogue state tracking method
Through the multimodal dialogue state tracking method, multimodal information is used to predict dialogue state and intention, the problem of insufficient information utilization is solved, the performance and user experience of the dialogue system are improved, and the accurate understanding of complex scenarios is achieved .
Patent Information
- Application Number
- CN202411102291.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-12
- Publication Date
- 2024-11-26
AI Technical Summary
When using multimodal information to track dialogue states, the information utilization is insufficient and the prior knowledge is lacking, resulting in insufficient accuracy and reliability of dialogue state prediction, especially in complex scenarios such as metacosmic environments. User intentions and situations.
Using multimodal dialogue state tracking method, the prompt vector is generated by encoding the scene and dialogue context, the text and image mode diagrams are initialized, the mode diagrams are fused, and the dialogue state and intention are predicted through slice filtering and mode diagrams are fused, combining pre-trained models and volumes. The neural network performs information processing.
It improves the comprehensiveness and accuracy of dialogue status tracking, reduces information complexity, enhances the overall performance and user experience of the dialogue system, and can more accurately understand user intentions and situations.
Smart Images

Figure CN119025647A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data mining in natural language processing, and in particular to a multimodal dialogue state tracking method. Background Art
[0002] Task-based dialogue is a human-computer interaction method that uses dialogue to complete specific tasks or solve specific problems. Users communicate with computer systems through natural language and ask specific requirements or questions to the system. The system will understand the user's intentions and answer the user's questions, provide relevant information, or perform corresponding operations in an appropriate manner to complete the tasks specified by the user. Dialogue state tracking plays a vital role in task-based dialogue systems. It is responsible for capturing and updating key information such as user intentions, preferences, and task-related details from the interaction between the user and the system in real time. This information is crucial for the system because it helps the system understand the user's goals and make decisions and generate responses accordingly. Effective dialogue state tracking not only improves the response accuracy of the task-based dialogue system, but also enhances the user experience, allowing the system to help users complete specific tasks more smoothly and efficiently.
[0003] In traditional dialogue state tracking methods, the system usually relies on predefined slot values (S l ots) and intents (Acts) to parse the user's language input. Slot values represent specific information that needs to be filled in the dialogue, such as time, place, and quantity, while intents represent the user's goals or behaviors, such as booking tickets and inquiries. The task of dialogue state tracking is to extract this information from the user's input and update the dialogue state.
[0004] With the development of technology, the application scenarios of task-based dialogue have become more complex and diverse. For example, in addition to input and domain complexity, it also faces multiple challenges such as context understanding, user behavior analysis, support for multiple interaction modes, cross-cultural and cross-language adaptation, emotion and psychological state processing, and security and privacy protection. These factors increase the complexity of dialogue function design and require dialogue systems to be highly flexible and adaptable. For example, in a modally rich environment such as the metaverse, this challenge is particularly prominent. Multimodal dialogue models need to handle diverse inputs that are not limited to text and voice, such as images, positions in virtual space, relative relationships with other objects, and even the user's line of sight and gestures to respond, while considering how to integrate this information to more accurately understand user intentions and situations. Dialogue models need to be able to understand rich data from different modalities. This places higher demands on the system's understanding and data processing capabilities. In complex scenarios, task-based dialogue becomes a highly complex form of interaction. Therefore, it needs to use multimodal data for deeper context analysis to provide users with a more personalized and rich interactive experience. Summary of the invention
[0005] Based on natural language processing, the present invention proposes a multimodal dialogue state tracking method to address the above shortcomings, which aims to solve the technical problems of insufficient utilization and fusion of multimodal information and lack of prior knowledge when using text and image multimodal information to track and predict dialogue states.
[0006] To achieve the above object, the present invention provides the following technical solution: a multimodal dialogue state tracking method, comprising the following steps:
[0007] Step 1: Given the scene and dialogue context, first encode, then generate a prompt vector, then generate a slice score map, and finally perform slice screening;
[0008] Step 2: First, initialize the text pattern graph and the image pattern graph, then initialize the fusion pattern graph, then count the node frequency relationship of the text and image, and add adjacent edges of the pattern graph according to the probability;
[0009] Step 3: After filtering and slicing the text, the text is encoded as a context and then passed through the pattern graph to obtain a text pattern graph output, a visual pattern graph output, and a fusion pattern graph output;
[0010] Step 4: Fuse the outputs of each pattern graph to predict the dialogue state and dialogue intent.
[0011] Preferably, the step 1 comprises the following steps:
[0012] Step 1.1, after encoding the scene image and the dialogue context through the image encoder and the pre-trained language model respectively, generate a scene image vector and a dialogue context vector respectively;
[0013] Step 1.2, after comprehensive processing of the scene image vector and the dialogue context vector, a prompt vector is generated;
[0014] Step 1.3: After combining the prompt vector with the original scene image vector, a convolution operation is performed to generate a score map;
[0015] Step 1.4: Filter the scene slices according to the score map.
[0016] Preferably, the step 2 comprises the following steps:
[0017] Step 2.1: Initialize the text pattern graph and encode the Wikipedia description of the object using the pre-trained model;
[0018] Step 2.2, initializing the image mode map, and encoding the image of the object using the image encoder;
[0019] Step 2.3, initializing the fusion pattern graph, according to the initial values of the nodes of the text pattern graph and the image pattern graph, splicing and outputting the initialization values of the nodes of the fusion pattern graph through the fully connected neural network;
[0020] Step 2.4: Probability statistics of slot names and slot values, count the frequency of co-occurrence, co-reference and co-update of objects mentioned in the context, and set the threshold for adding adjacent edges;
[0021] Step 2.5: When the frequency of co-occurrence, co-reference and co-update of two nodes exceeds a threshold, an adjacent edge is added to the text-image fusion pattern graph.
[0022] Preferably, the step 3 comprises the following steps:
[0023] Step 3.1, encoding the filtered slices through an image encoder;
[0024] Step 3.2: Encode the conversation context using a pre-trained language model.
[0025] Step 3.3: Take the context as a special node, the slice code and the conversation context code as the node initial value, and perform graph attention calculation with other nodes to obtain the output results of each pattern graph.
[0026] Preferably, the step 4 comprises the following steps:
[0027] Step 4.1, the text mode map output, the image mode map output, and the fusion mode map output are fused through a convolutional neural network;
[0028] Step 4.2: After fusing the output vectors of each pattern graph, the dialogue state classifier of each slot is used to predict the dialogue state;
[0029] Step 4.3: After fusing the output vectors of each pattern graph, the intent classifier is used to predict the dialogue action.
[0030] Compared with the prior art, the present invention has the following beneficial effects:
[0031] 1. The application of pattern graph makes the dialogue state tracking more comprehensive and accurate, and improves the overall performance of the dialogue system. Secondly, the correlation and characteristics between various modal information are fully considered in the construction process of the pattern graph, making the prediction of the dialogue state more accurate and reliable.
[0032] 2. Traditional methods often face the challenges of information overload and noise interference when processing multimodal information, resulting in poor tracking of dialogue status. However, this method uses scene slicing and filtering to segment and filter key scenes in the dialogue process, thereby reducing information complexity and improving the accuracy and reliability of dialogue status tracking. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Figure 1 It is the overall flow chart of the present invention;
[0034] Figure 2 Screening flow chart for the scenario of the present invention;
[0035] Figure 3 Initialize the flow chart for the mode diagram of the present invention;
[0036] Figure 4 The hidden vector calculation flow chart of the present invention;
[0037] Figure 5 This is a flow chart of the conversation state prediction of the present invention. DETAILED DESCRIPTION
[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0039] The present invention will be further described below in conjunction with the accompanying drawings:
[0040] Example 1
[0041] The instance data dialogue data contains multiple annotation fields that record the speech of the system and the user, where the slot value refers to the commodity attribute and the object number points to the specific item. The scene information file records the location and attributes of the commodity in the virtual shopping environment. Each object is equipped with identification information to match the items in the dialogue with the entities in the visual image. The specific explanation is as follows:
[0042]
[0043] like Figure 1 As shown, it is the overall flow chart of the whole method. The present invention includes four major steps: scene screening, pattern initialization, hidden vector calculation and dialogue state prediction. Each major step contains 3-4 small steps. The scene screening process is to encode the scene and dialogue, the purpose is to generate prompt vectors and slice score graphs and filter slices, and the pattern initialization is to initialize the text pattern graph, image pattern graph and initialize the fusion pattern graph, count the frequency relationship of text and image nodes, and add the adjacent edges of the pattern graph. The hidden vector calculation is to filter the text context after the slice encoding, and obtain the text pattern graph output, visual pattern graph output and fusion pattern graph output through the pattern graph. The dialogue state prediction is to fuse the outputs of each pattern graph to predict the dialogue state and dialogue intention.
[0044] like Figure 2 As shown in the figure, the scene and dialogue context are first encoded, the prompt text is generated and the scene image is encoded into an image vector. Then the image prompt vector is generated, which is achieved through text vector mapping and multi-head attention calculation. Subsequently, the slice score map is generated, the multi-head attention is regularized, the score map is calculated and the probability of the slice is predicted. Finally, the predicted slice probability is screened and the final slice result is obtained as follows:
[0045] 1.1 After encoding the scene image and the dialogue context through the image encoder and the pre-trained language model respectively, the scene image vector and the dialogue context vector are generated respectively. The specific steps are as follows:
[0046] 1.1.1 Prompt text generation, the prompt text is "Please focus on the visual elements (distance, direction, color and material) involved in the conversation according to the conversation <conversation context>, and predict the value of <slot name> in the current round";
[0047] 1.1.2 After encoding the prompt text using Bert, the text embedding vector after the prompt is obtained;
[0048] 1.1.3 Encode the scene image using RestNet, take the feature map of the fourth stage of the model, and calculate the global average feature map of this stage.
[0049] 1.2 After comprehensive processing of the scene image vector and the dialogue context vector, a prompt vector is generated. The specific steps are as follows:
[0050] 1.2.1 Text vector mapping: the text embedding vector after the prompt is processed through the fully connected layer to obtain the dialogue text mapping vector;
[0051] 1.2.2 The multi-head attention mechanism is used to combine the text vector and the image vector to generate a prompt vector related to the current conversation. The conversation text mapping vector is used as the key and value, and the feature map of the fourth stage is used as the query to obtain the scene encoding vector after the prompt after multi-head attention processing.
[0052] 1.3 After combining the prompt vector with the original scene image vector, a convolution operation is performed to generate a score map:
[0053] 1.3.1 Regularization: perform L2 regularization on the scene encoding vector after prompting;
[0054] 1.3.2 Score map calculation: the regularized scene prompt encoding and the dialogue text mapping vector are calculated using the Einstein summation convention to obtain the score map;
[0055] 1.3.3 Probability prediction, the probability of the slice is calculated based on the score map using a fully connected neural network and ReLU activation function.
[0056] 1.4 Filter the scene slices according to the score map.
[0057] like Figure 3 As shown, it is a flow chart of the pattern graph initialization process of the present invention. When processing the object description, the Wikipedia description of the object is first encoded using a pre-trained model to form a text pattern graph initialization. Then, the picture of the object is encoded by an image encoder to generate an image pattern graph initialization. Next, based on the initial values of the nodes of the text and image pattern graphs, the initialization values of the fusion pattern graph nodes are output through a fully connected neural network. On this basis, the probabilities of the slot names and slot values are counted, and the co-occurrence, co-reference, and co-update of objects in the context are considered to set a threshold for adding adjacent edges. Finally, if the co-occurrence and co-reference update frequency of the two nodes exceeds the threshold, an adjacent edge is added to the text image fusion pattern graph. It is specifically shown as follows:
[0058] 2.1 Text pattern graph initialization, using the pre-trained model to encode the Wikipedia description of the object.
[0059] 2.1.1 Term search, search for Wikipedia explanations of object attributes (type, color, material and brand);
[0060] 2.1.2 Encoding of entries: Encode the Wikipedia explanation using the Bert pre-trained model as the initial value of the text pattern graph node;
[0061] 2.1.3 Add adjacent edges and add bidirectional edges based on the hierarchical relationship between domains, attributes and attribute values.
[0062] 2.2. Image mode map initialization, use the image encoder to encode the object's picture, the specific steps are as follows:
[0063] 2.2.1 Image cropping: crop the object image using the image border annotation, and classify and summarize it according to attributes and attribute values;
[0064] 2.2.2 Image coding: Encode the classified and summarized object images using an image encoder. For multiple images, take the average of the image codes as the classified and summarized code value;
[0065] 2.2.3 Add adjacent edges and add bidirectional edges based on the hierarchical relationship between domains, attributes and attribute values.
[0066] 2.3 Initialization of the fusion pattern graph: According to the initial values of the nodes of the text pattern graph and the image pattern graph, the initialization values of the nodes of the fusion pattern graph are output through the fully connected neural network after splicing. The specific steps are as follows:
[0067] 2.3.1 After combining the term encoding and the image mean encoding, the fully connected upstream neural network is used for processing, and then the fully connected downstream neural network is used for processing to obtain the initial value of the fusion pattern graph;
[0068] 2.3.2 Add adjacent edges and add bidirectional edges based on the hierarchical relationship between domains, attributes and attribute values.
[0069] 2.4 Probability statistics of slot names and slot values. Count the frequency of co-occurrence, co-reference, and co-update of objects mentioned in the context, and set the threshold for adding adjacent edges. The specific steps are as follows:
[0070] 2.4.1 Co-reference frequency statistics: analyze the co-reference of object attributes mentioned in the context and record the co-reference frequency between each pair of attributes;
[0071] 2.4.2 Co-occurrence frequency statistics: observe the co-occurrence of object attributes in the context and record the frequency of object attributes appearing in the same round of dialogue;
[0072] 2.4.3 Same as update frequency statistics, analyze the update of context object attributes and record the frequency of simultaneous attribute updates.
[0073] 2.5 For the case where the frequency of co-occurrence, co-reference and co-update of two nodes exceeds the threshold of 0.35 times / dialogue, an adjacent edge is added to the text-image fusion pattern graph.
[0074] like Figure 4 As shown in FIG. 1 , it is a flowchart of the hidden vector calculation process of the present invention. First, the image encoder is used to encode the filtered slices, and then the conversation context is encoded using the pre-trained language model. Next, the context is used as a special node, and the slice encoding and the conversation context encoding are used as the initial values of the node. Finally, the graph attention calculation is used to interact with other nodes to obtain the output results of each pattern graph. The specific steps are as follows:
[0075] 3.1 Screening slice encoding, the screened slices are encoded by the image encoder; the specific steps are as follows:
[0076] 3.1.1 Slice encoding, the slices that are selected as the retained ones are encoded using the ResNET encoder;
[0077] 3.1.2 Slice information encoding: The length and width pixel dimensions of the scene image, the slice size (32*32, 64*64, 128*128), and the slice index (slice length direction index, slice width direction index) are integrated into a vector, and the slice information encoding is obtained after being processed by a fully connected neural network;
[0078] 3.1.3 Coding combination, after connecting the slice code and the slice information code, the regularized vector is converted to the comprehensive code of the slice. If multiple slices are retained, the average value of the comprehensive codes of the slices is taken.
[0079] 3.2 Conversation context encoding,The conversation context is encoded using a pre-trained language model. The specific steps are as follows:
[0080] 3.2.1 Prompt text combination, the prompt text is "Please focus on the visual elements (distance, direction, color and material) involved in the conversation according to the conversation <conversation context>, and predict the value of <slot name> in the current round";
[0081] 3.2.2 After encoding the prompt text using Bert, the text embedding vector after the prompt is obtained;
[0082] 3.3 Pattern graph calculation, taking the context as a special node, the slice code and the conversation context code as the node initial value, and performing graph attention calculation with other nodes to obtain the output results of each pattern graph.
[0083] 3.3.1 Multi-head attention mechanism update. In the calculation process of each attention head, the weight factor of the neighboring one-hop node to the current node is first calculated, and then the neighboring node values are added according to the weights and regularized;
[0084] 3.3.2 Multi-head attention feature aggregation, connecting multiple attention heads and inputting them into the fully connected neural network;
[0085] 3.3.3 Multi-head attention network calculation, the latent vector output of the previous layer of virtual nodes is used as the latent vector input of the next layer of virtual nodes, and the latent vector output of the last layer of virtual nodes is used as the final calculation result of the single-modal pattern graph;
[0086] 3.3.4 The feature vector calculation of the virtual node is based on the output of the image pattern map or the text pattern map, which is used as the input of the text pattern map or the image pattern map. After calculation, it is used as the input of the fusion pattern map, and the calculation results of the pattern map of the text-image fusion layer order and the calculation results of the combined pattern map of the image-text fusion layer order are taken as the hidden vector of the dialogue state.
[0087] like Figure 5 As shown in FIG. 1 , it is a flow chart of the dialog state prediction process of the present invention. First, the output results of the pattern graph are preliminarily fused through the dialog state hidden vector. Then, the hybrid convolution technique is used to further fuse the results to better capture information of different scales and features. Then, the system predicts the user's intention or dialog state based on the current dialog content and context. The system predicts the next action to be taken based on the predicted dialog state and historical dialog information. The specific steps are as follows:
[0088] 4.1 Fusion of pattern graph input results. After fusing the pattern graph output results of different layers, they are passed through a convolutional neural network. The specific calculation steps are as follows:
[0089] 4.1.1 The pattern hidden vector output of the text-image fusion layer sequence and the pattern hidden vector output of the image-text fusion layer sequence are calculated using the Einstein summation convention to obtain the input of the mixed convolution;
[0090] 4.1.2 Input the calculation results of the previous step into the convolutional neural network for further calculation to obtain the final fusion calculation results.
[0091] 4.2 Dialogue state prediction: The final fusion calculation results are applied to the classifiers of each slot value to predict the dialogue state slot value.
[0092] 4.3 Conversation intention prediction: The final calculation results will be integrated with the action prediction classifier to perform conversation action prediction.
[0093] For multimodal dialogue state tracking, the present invention introduces the concept of a multi-layer multimodal mixed mode graph by fusing text and visual information, and combines it with the encoding process of the pre-trained model to enable the model to effectively process dialogue and visual context, thereby improving the prediction performance.
[0094] The present invention also studies the context-driven visual scene slice screening technology, proposes a slice-text level score map, and combines the dialogue context for accurate scene analysis and processing. It can accurately screen image slices, provide performance gains for dialogue state prediction, and effectively improve the accuracy of the model.
[0095] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.
[0096] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A multimodal dialogue state tracking method, characterized in that: The following steps are involved: Step 1: Given the scene and dialogue context, first encode, then generate a prompt vector, then generate a slice score map, and finally perform slice screening; Step 2: First, initialize the text pattern graph and the image pattern graph, then initialize the fusion pattern graph, then count the node frequency relationship of the text and image, and add adjacent edges of the pattern graph according to the probability; Step 3: After filtering and slicing the text, the text is encoded as a context and then passed through the pattern graph to obtain a text pattern graph output, a visual pattern graph output, and a fusion pattern graph output; Step 4: Fuse the outputs of each pattern graph to predict the dialogue state and dialogue intent.
2. A multimodal dialogue state tracking method according to claim 1, characterized in that: The step 1 comprises the following steps: Step 1.1, after encoding the scene image and the dialogue context through the image encoder and the pre-trained language model respectively, generate a scene image vector and a dialogue context vector respectively; Step 1.2, after comprehensive processing of the scene image vector and the dialogue context vector, a prompt vector is generated; Step 1.3: After combining the prompt vector with the original scene image vector, a convolution operation is performed to generate a score map; Step 1.4: Filter the scene slices according to the score map.
3. The multimodal dialogue state tracking method according to claim 1, characterized in that: The step 2 comprises the following steps: Step 2.1: Initialize the text pattern graph and encode the Wikipedia description of the object using the pre-trained model; Step 2.2, initializing the image mode map, and encoding the image of the object using the image encoder; Step 2.3, initializing the fusion pattern graph, according to the initial values of the nodes of the text pattern graph and the image pattern graph, splicing and outputting the initialization values of the nodes of the fusion pattern graph through the fully connected neural network; Step 2.4: Probability statistics of slot names and slot values, count the frequency of co-occurrence, co-reference and co-update of objects mentioned in the context, and set the threshold for adding adjacent edges; Step 2.5: When the frequency of co-occurrence, co-reference and co-update of two nodes exceeds a threshold, an adjacent edge is added to the text-image fusion pattern graph.
4. The multimodal dialogue state tracking method according to claim 1, characterized in that: The step 3 comprises the following steps: Step 3.1, encoding the filtered slices through an image encoder; Step 3.2: Encode the conversation context using a pre-trained language model. Step 3.3: Take the context as a special node, the slice code and the conversation context code as the node initial value, and perform graph attention calculation with other nodes to obtain the output results of each pattern graph.
5. A multimodal dialogue state tracking method according to claim 4, characterized in that: In the step 3.3, the pattern image output includes text pattern image output, image pattern image output, and fusion pattern image output.
6. A multimodal dialogue state tracking method according to claim 1, characterized in that: The step 4 comprises the following steps: Step 4.1, the text mode map output, the image mode map output, and the fusion mode map output are fused through a convolutional neural network; Step 4.2: After fusing the output vectors of each pattern graph, the dialogue state classifier of each slot is used to predict the dialogue state; Step 4.3: After fusing the output vectors of each pattern graph, the intent classifier is used to predict the dialogue action.
Citation Information
Cited By
System and method of objective-driven intelligent navigation
US12572374B1
