Visual dialogue generation method based on human-like visual perception and language memory network

Abstract

The application provides a visual dialogue generation method based on human-like visual perception and language memory network, which comprises preprocessing of visual dialogue Chinese text input and construction of a word table; image feature representation of the visual dialogue and feature representation of dialogue text; construction of a visual context perception graph; iterative updating of the visual context perception graph to obtain human-like global and local visual feature vectors; obtaining of a historical dialogue context language memory feature vector under the guidance of a current question; multi-modal semantic fusion processing; decoding and prediction of an answer feature sequence; parameter optimization of a visual dialogue generation network model based on human-like visual perception and language memory network; and prediction of a candidate answer. The application constructs a visual context perception graph on a visual dialogue, uses a graph convolutional neural network to obtain global and local visual feature representations, and fully understands image content; and based on learning of historical dialogue context language memory, more fine-grained text information is captured to infer deep semantic information implied by the dialogue.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically a visual dialogue generation scheme based on human-like visual perception and language memory networks. Background Technology

[0002] Visual dialogue aims to explore how an intelligent agent can respond appropriately to a range of questions using natural language, relying on dialogue history and the understanding of given images. Most visual language tasks focus more on entity-level or region-level visual features. In contrast, visual dialogue is more challenging because it requires the agent to have a sufficiently comprehensive understanding of images, going beyond entity and region levels, to handle the diversity and complexity of dialogue content. Therefore, it requires the agent to have a comprehensive understanding of images from multiple perspectives and to comprehend the semantic interaction details within a deep dialogue history. How to understand images from multiple perspectives and grasp the details of complex linguistic content, just like humans, has become a crucial challenge in visual dialogue. Visual dialogue is currently a hot research topic in computer vision, with a wide range of applications, including: AI chatbots, and helping visually impaired individuals quickly grasp information about their surroundings.

[0003] With the development of deep learning in recent years, visual dialogue technology has also made great progress, but the following challenges and problems still exist:

[0004] First, existing methods have limited processing of image information and insufficient image understanding.

[0005] For example, Jiang 2020 introduced external knowledge, namely the scene graph of the image, to characterize object-relation visual features in visual dialogue tasks, jointly depicting the image from both visual and semantic perspectives. Chen 2021 argued that accurately locating question-related visual objects in images is crucial; therefore, he proposed understanding visual objects in the visual dialogue history by minimizing the distance between the prior and posterior distributions of visual objects. Chen 2021 also proposed a relation-aware graph convolutional network to extract spatial relationships between entities in images to improve the agent's understanding of images. Although the above methods have gradually strengthened the exploration of visual context features, they mainly come from object-local features and still ignore global visual context information that reflects the visual environment of the entire image.

[0006] Second, the lack of learning of fine-grained textual context semantic features.

[0007] For example, Gan (2019) introduced the concept of iterative reasoning and proposed a Recurrent DualAttention Network, which updates the semantic representation of the question in each round of dialogue, enabling the model to understand multimodal context with fine granularity. Schwartz (2019) attempted to construct a graph structure based on question-answer pairs in the dialogue and then used a general attention mechanism to capture the details and nuances of the dialogue content, thereby helping the model predict the answer. Kang (2019) proposed two attention modules, REFER and FIND, to address ambiguous referential meanings in the question. Although the above works have focused on learning textual content, they are still at the coarse-grained learning stage, neglecting the importance of fine-grained linguistic information. Summary of the Invention

[0008] This invention addresses the shortcomings of existing technologies. Specifically, it proposes a visual dialogue generation scheme based on human-like visual perception and language memory networks. This aims to provide agents with comprehensive visual feature understanding information and more fine-grained textual language memory information, thereby improving the agent's understanding of images and text to predict the appropriateness and accuracy of the agent's generated answers in visual dialogue.

[0009] To achieve the above objectives, the technical solution adopted by this invention is a visual dialogue generation method based on human-like visual perception and language memory networks, comprising the following steps:

[0010] Step 1: Preprocessing of text input and construction of vocabulary in visual dialogue;

[0011] Step 2: Image feature representation of visual dialogue and feature representation of dialogue text;

[0012] Step 3: Construct a visual context-aware graph;

[0013] Step 4: Iteratively update the visual context-aware map to obtain human-like global and local visual feature vectors;

[0014] Step 5: Obtain the historical dialogue context language memory feature vector under the guidance of the current question;

[0015] Step 6: Multimodal semantic fusion processing;

[0016] Step 7: Decode the feature sequence of the predicted answer;

[0017] Step 8: Parameter optimization of the visual dialogue generation network model based on human-like visual perception and language memory networks;

[0018] Step 9: Predict candidate answers, including minimizing the loss function as the model trains. When the loss is minimized, the answer sequence corresponding to the minimum loss is taken as the candidate answer sequence predicted by the model.

[0019] Moreover, the implementation of step 1 includes the following sub-steps: Step 1.1: Obtain a visual dialogue dataset, which contains sentence text and images; perform word segmentation on all the sentence text contained in the visual dialogue dataset to obtain the segmented sentence sequence.

[0020] Step 1.2: Obtain all words from the segmented sentence sequence, and select all words whose word frequency is greater than a set threshold. Construct a word index table corresponding to all words in the dataset, denoted as Voc. Then, perform one-hot encoding on each word in the index table Voc to obtain a one-hot vector table.

[0021] Step 1.3: Randomly initialize a word embedding matrix W. e By using the word embedding matrix W e The one-hot vector of each word is mapped to the corresponding word vector to obtain the vector representation of each word.

[0022] Furthermore, step 2 is implemented by including the following sub-steps:

[0023] Step 2.1: Obtain any image I from the visual dialogue dataset, and use the object detection feature extractor to obtain the visual features of image I;

[0024] Step 2.2: Obtain the dialogue history text corresponding to any image from the visual dialogue dataset, denoted as U, the current question Q, and the actual answer label, denoted as A. gt ;

[0025] Step 2.3: Use a bidirectional long short-term memory network to process the r-th round of dialogue u in the dialogue history text U from Step 2.2. r Semantic encoding yields the hidden layer state sequence as follows: Using the feature vector h of the last hidden state r,L As the rth round of dialogue u r Sentence-level feature representation h r This yields the sentence-level features h corresponding to the total dialogue history U. u ;

[0026] Step 2.4: Use a bidirectional long short-term memory network to semantically encode the current problem Q from step 2.2 and extract the context feature vector of the current problem, denoted as , and take the state feature h of the last hidden layer of the Bi-LSTM. Q,L h, as the sentence-level feature vector of the current problemq .

[0027] Furthermore, step 3 is implemented by utilizing the visual features V of image I from step 2.1 and the sentence-level feature vector h of the current problem. q As input for constructing a visual context-aware graph, during the graph construction process, the visual features V of image I are compared with h. q Attention alignment is performed to obtain image features containing information about the current problem. The feature representations of the graph nodes and the edge weights between nodes are then represented as V. * and

[0028] Moreover, step 4 is implemented by using a multi-layer graph convolutional neural network to update the visual context perception map established in step 3 to obtain the global and local visual feature vectors of the human-like image.

[0029] Furthermore, step 5 is implemented by using the dialogue history feature vector h obtained in step 2.3 for the r-th round. r With the current problem feature vector h q To construct a language memory feature vector representation, firstly, the dialogue history feature vectors from round 0 to round r are concatenated to obtain the total dialogue history features. Then with h q Perform feature fusion to obtain topic feature vector representation. Finally, the dialogue history of all rounds is iteratively updated and learned to obtain the final language memory feature T. m .

[0030] Furthermore, step 6 is implemented by using the sentence-level feature vector h of the current problem obtained in step 2.1. q The visual context feature vector obtained in step 4 and the language memory feature vector T obtained in step 5 m We perform feature fusion processing between visual and textual modalities to obtain the final high-level fused visual-language feature vectors. * .

[0031] Furthermore, step 7 is implemented by selecting candidate answer A. * Similar to problem encoding, Bi-LSTM uses the last hidden state vector as A. * Sentence-level feature representation of each answer; let p t It contains the probability values ​​of 100 candidate answers, and then takes p. t The value with the highest probability and its order are used as the final predicted answer value.

[0032] Furthermore, step 8 is implemented by using the predicted answer p calculated in step 7. t With the real answer A gt The loss cost function F between:

[0033]

[0034] In the formula, y t，i p represents the one-hot encoded vector representation of the true answer label of the i-th candidate answer corresponding to the t-th round question. t,i Let represent the probability value of the i-th candidate answer corresponding to the question in round t.

[0035] On the other hand, the present invention provides a visual dialogue generation system based on a human-like visual perception and language memory network, which is used to implement the visual dialogue generation method based on a human-like visual perception and language memory network as described above.

[0036] Compared with existing technologies, the beneficial effects of this invention are reflected in:

[0037] 1. This invention encodes image information from two perspectives, namely the global perspective and the local perspective. It captures not only entity-level visual features but also image-level visual features, thereby enriching the representation of image information.

[0038] 2. This invention designs an iterative learning strategy for language understanding. This strategy decomposes the entire dialogue history into rounds, and then iteratively captures the semantic interaction information between each round of dialogue history and the question as the dialogue progresses. In this way, the model can capture subtle details of the text, enhancing its memory and understanding of language content.

[0039] 3. This invention enhances the model's learning ability through deep learning and feature extraction of visual and textual semantics, resulting in a significant improvement in the agent's ability to predict answers and more accurate prediction results.

[0040] The present invention is simple and convenient to implement, highly practical, and solves the problems of low practicality and inconvenience in actual application of related technologies. It can improve user experience and has significant market value. Attached Figure Description

[0041] Figure 1 This is a flowchart of a method according to an embodiment of the present invention. Detailed Implementation

[0042] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0043] This invention constructs a visual context-aware graph for visual dialogue, utilizing a graph convolutional neural network to acquire global and local visual feature representations to fully understand image content. Furthermore, learning based on historical dialogue contextual language memory captures finer-grained textual information to infer the deep semantic information implicit in the dialogue. The combination and complementarity of these two approaches improve the reasonableness and accuracy of the agent's predicted answers to questions.

[0044] See Figure 1 The visual dialogue generation method based on human-like visual perception and language memory networks proposed in this embodiment is carried out in the following steps:

[0045] Step 1: Preprocessing of text input and construction of vocabulary in visual dialogue;

[0046] Furthermore, in the embodiments, step 1 is preferably implemented using the following sub-steps:

[0047] Step 1.1: Obtain the visual dialogue dataset, which contains sentence text and images. Perform word segmentation on all the sentence text contained in the visual dialogue dataset to obtain the segmented sentence sequence;

[0048] Step 1.2: Obtain all words from the segmented sentence sequence, and select all words whose frequency is greater than a set threshold. Construct a word index table (Voc) corresponding to all words in the dataset. Then, perform one-hot encoding on each word in the index table Voc to obtain a one-hot vector denoted as O = [o1, o2, ..., o2]. i ,…,o s ], where o i This represents the one-hot encoded vector corresponding to the i-th word in the index table Voc, where S is the total number of words in the index table Voc. Here, the value of the index variable i is 1, 2, ..., S.

[0049] In practice, users can preset the threshold value as needed.

[0050] Step 1.3: Randomly initialize a word embedding matrix W. e , Where d w R represents the dimension of each word vector, and R denotes the real number field; this is achieved by using the word embedding matrix W. e The one-hot vector of each word is mapped to the corresponding word vector to obtain the vector representation of each word.

[0051] Step 2: Image feature representation of visual dialogue and feature representation of dialogue text;

[0052] Furthermore, in the embodiments, step 2 is preferably implemented using the following sub-steps:

[0053] Step 2.1: Obtain any image I from the visual dialogue dataset, and use the object detection feature extractor to obtain the visual features of image I. Where v i Let represent the i-th target object feature detected in image I, n represent the number of target objects (where the index variable i takes values ​​of 1, 2, ..., n), z represent the total number of target objects detected in visual feature V, and d v The dimension of each visual object's features;

[0054] Step 2.2: Obtain the dialogue history text corresponding to any image from the visual dialogue dataset, denoted as U = [u1, u2, ..., u...]. r ,…,u R The current problem Q = [w] Q,1 ,w Q，2 ,…,w Q,i ,…w Q,L The actual answer is labeled A. gt Where R represents the total number of dialogue segments in the dialogue history U, u r This represents the content of the r-th round of dialogue, where the index variable r takes values ​​of 1, 2, ..., R, L represents the sentence length of the current question, and w Q,i Let i represent the word vector representation of the i-th word in question Q, where the index variable i takes the value 1, 2, ..., L;

[0055] Step 2.3: Use a bidirectional long short-term memory network (Bi-LSTM) to process the r-th round of dialogue u in the dialogue history text U from Step 2.2. r =[w r,1 ,w r,2 ,…,w r,i ,…,w r,L Semantic encoding is performed to obtain the hidden layer state sequence representation as follows: Here, we use the feature vector h of the last hidden state. r,L As the rth round of dialogue u r Sentence-level feature representation h r , Therefore, the sentence-level features corresponding to the total dialogue history U are denoted as h. u =[h1,h2,…,h r ,…h R ], Where, d h It is the dimension of the hidden layer state vector;

[0056] Step 2.4: Use a bidirectional long short-term memory network (Bi-LSTM) to solve the current problem Q = [w] in Step 2.2.Q,1 ,w Q,2 ,…,w Q,i ,…w Q,L Semantic encoding is performed to extract the contextual feature vector of the current problem, denoted as h. Q =[h Q,1 ,h Q,2 ,…,h Q,i ,…,h Q,L Here, the index variable i takes values ​​of 1, 2, ..., L, and here we take the state feature h of the last hidden layer of the Bi-LSTM. Q,L h, as the sentence-level feature vector of the current problem q ,

[0057] Step 3: Construct a visual context-aware graph;

[0058] Furthermore, in the embodiments, step 3 is preferably implemented in the following specific manner:

[0059] Using the visual features V = [v1, v2, ..., v] of image I from step 2.1 i ,…,v n ] and the sentence-level feature vector h of the current problem q As input for constructing the visual context-aware graph, the index variable i here takes values ​​of 1, 2, ..., n. During graph construction, the visual features V of image I are compared with h... q Attention alignment is performed to obtain image features V containing information about the current problem. * ,

[0060] Among them, f u The (·) function represents attention alignment;

[0061] In equation (1), The attention distribution weights representing the visual features of the image to the current problem are mainly calculated as follows:

[0062]

[0063] Where Softmax(·) represents the activation function, i.e., the normalization exponential function;

[0064] In equation (2), The feature representation representing the association between the visual features of the image and the current problem mainly involves the following calculations:

[0065]

[0066] In equation (3), and b1∈R z×1f represents the trainable parameters. v (·), f q (·) and f a (·) denotes a nonlinear mapping function. * indicates dot product, the superscript T indicates transpose, z in the preceding text represents the total number of target objects detected in visual feature V, and d v The dimension of each visual object feature.

[0067] After calculation using equations (1)-(3), the feature representations of the constructed graph nodes and the edge weight representations between nodes are obtained as follows: and This represents the i-th image feature node. express The weights between the node and all other visual feature nodes, where the index variable i takes the values ​​1, 2, ..., n;

[0068] Step 4: Iteratively update the visual context-aware map to obtain human-like global and local visual feature vectors;

[0069] Furthermore, in the embodiments, step 4 is preferably implemented in the following specific manner:

[0070] The visual context perception map established in step 3 is updated using a multi-layer graph convolutional neural network to obtain the global and local visual feature vectors of the human-like image; the main calculation is shown in equation (4);

[0071]

[0072] In equation (4), f g (·) represents a nonlinear mapping function; α∈[0,1] is a fine-tuned parameter; K represents the number of layers in the graph convolutional network, where the index variable k takes the values ​​1, 2, ... K; These are the visual context features extracted through context-aware graph learning.

[0073] Step 5: Obtain the historical dialogue context language memory feature vector under the guidance of the current question;

[0074] Furthermore, in the embodiments, step 5 is preferably implemented in the following specific manner:

[0075] Using the dialogue history feature vector h obtained in step 2.3 for the r-th round r With the current problem feature vector h q To construct the language memory feature vector representation; first, concatenate the dialogue history feature vectors from round 0 to round r, denoted as [h0|h1|…|h r [To obtain the overall characteristics of the dialogue history] Then with h q Perform feature fusion to obtain topic feature vector representation. Finally, the dialogue history of all rounds is iteratively updated and learned to obtain the final language memory feature T. m The main calculations are shown in formulas (5)-(9) below;

[0076]

[0077]

[0078]

[0079]

[0080]

[0081] In equations (5)-(9), and These are trainable parameters; f q ′ (h q ) indicates that f m ′ (·) denotes a nonlinear mapping function, and σ(·) denotes a Sigmoid activation function; This represents the semantic interaction features captured in the r-th round; This represents the language memory feature vector, as mentioned earlier in d. h It is the dimension of the hidden layer state vector, Cat ite (·) indicates a splicing operation.

[0082] Step 6: Multimodal semantic fusion processing;

[0083] Furthermore, in the embodiments, step 6 is preferably implemented in the following specific manner:

[0084] Using the sentence-level feature vector h of the current problem obtained in step 2.1 q The visual context feature vector obtained in step 4 and the language memory feature vector T obtained in step 5 m We perform feature fusion processing between visual and textual modalities to obtain the final high-level fused visual-language feature vectors. * ,

[0085]

[0086] In equation (10), s represents the primary multimodal fusion feature. For the parameters used in model training, ReLU represents the (·) activation function, Layernorm(·) represents the layer normalization operation, and we have:

[0087]

[0088] In equation (11), and They represent the values ​​for h respectively. q , and T m The problem after mutual attention processing, the feature vector representation of visual and linguistic memory; and we have:

[0089]

[0090] In equation (12), a v Indicates attention weights. The visual feature representation includes the characteristics of attention, which are related to problem-solving and language memory, and includes:

[0091]

[0092] In equation (13), b5∈R z×1 The parameters represent those used in model training; and we have:

[0093]

[0094]

[0095] In equation (15), the function Att X (Y) represents the attention computation from vector X to vector Y; Represents the parameters used in model training; for and The calculation is the same as the operation of the above formulas (15)-(12), that is, formula (15) should be:

[0096] and and The calculation process and The calculation process is the same; the final result is... This is a high-level fused visual language feature vector.

[0097] Step 7: Decode the feature sequence of the predicted answer;

[0098] Furthermore, in the embodiments, step 7 is preferably implemented in the following specific manner:

[0099] Candidate answer A *Similar to problem encoding, Bi-LSTM uses the last hidden state vector as A. * Sentence-level feature representation of each answer; Indicates the candidate answers to the question. The vector representation of , where 100 represents 100 candidate answers, and the index variable i takes values ​​of 1, 2, ..., 100; and we have:

[0100]

[0101] p t =Softmax(score) (17)

[0102] In equation (16), p represents the model training parameters; t It contains the probability values ​​of each of the 100 candidate answers, p. t ∈R 1×100 Then take p t The value with the highest probability and its order are used as the final predicted answer value; Sum(·) represents the summation operation on the last dimension of the feature.

[0103] Step 8: Parameter optimization of the visual dialogue generation network model based on human-like visual perception and language memory networks;

[0104] Furthermore, in the embodiments, step 8 is preferably implemented in the following specific manner:

[0105] The predicted answer p is calculated using equation (17) in step 7. t With the real answer A gt The loss cost function F between:

[0106]

[0107] In equation (18), y t,i p represents the one-hot encoded vector representation of the true answer label of the i-th candidate answer corresponding to the t-th round question. t,i Let represent the probability value of the i-th candidate answer corresponding to the question in round t.

[0108] Step 9: Predict candidate answers;

[0109] The loss function in equation (18) above minimizes the loss as the model trains and learns. When the loss is minimized, y is taken. t The i-th answer sequence that minimizes the corresponding loss is the candidate answer sequence predicted by the model.

[0110] In specific implementation, the method proposed in the technical solution of this invention can be automatically executed by those skilled in the art using computer software technology. System devices for implementing the method, such as computer-readable storage media storing the corresponding computer program of the technical solution of this invention and computer equipment including the computer program running the corresponding computer program, should also be within the protection scope of this invention.

[0111] In some possible embodiments, a visual dialogue generation system based on a human-like visual perception and language memory network is provided, including a processor and a memory. The memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a visual dialogue generation method based on a human-like visual perception and language memory network as described above.

[0112] In some possible embodiments, a visual dialogue generation system based on a human-like visual perception and language memory network is provided, including a readable storage medium on which a computer program is stored. When the computer program is executed, it implements a visual dialogue generation method based on a human-like visual perception and language memory network as described above.

[0113] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A visual dialogue generation method based on human-like visual perception and language memory networks, characterized in that: Includes the following steps, Step 1: Preprocessing of text input and vocabulary construction in visual dialogue; Step 2: Image feature representation of visual dialogue and feature representation of dialogue text. Obtain the visual features of image I and the sentence-level feature vector of the current question. And sentence-level feature representation of each round of historical dialogue. ; Step 3: Construct a visual context-aware map, including utilizing the visual features of image I from step 2.

1. and the sentence-level feature vector of the current problem As input to construct a visual context-aware map, the visual features of the image are calculated. Attention distribution weights for the current problem Based on attention distribution weights Visual features of image I and Attention alignment is performed to obtain image features containing information about the current problem. The feature representation of the graph nodes and the edge weight representation between the nodes are respectively... and ; During the graph construction process, the visual features of image I are... and Attention alignment is performed to obtain image features containing information about the current problem. , in, The function represents attention alignment; in, This represents the activation function, i.e., the normalized exponential function; This represents the correlation between the visual features of the image and the current problem. In the formula, and Indicates trainable parameters, , and Represents a nonlinear mapping function. This indicates the dot product operation; the superscript T indicates transpose. Step 4: Iteratively update the visual context-aware map to obtain the global and local visual feature vectors of the human-like image. This includes updating the visual context-aware map established in Step 3 using a multi-layer graph convolutional neural network to obtain the global and local visual feature vectors of the human-like image. The update method is as follows: In the formula, Represents a nonlinear mapping function; It is a finely tuned parameter; This indicates the number of layers in the graph convolutional network, where the index variable k takes values ​​of 1, 2, ..., K. These are the visual context features extracted through context-aware graph learning; Step 5: Obtain the language memory feature vector of the historical dialogue context under the guidance of the current question, including sentence-level feature representations of each round of historical dialogue. Sentence-level feature vectors of the current problem By integrating and iteratively updating the learning process, the final language memory features are obtained. The iterative update learning includes: In the formula, and These are trainable parameters; , Represents a nonlinear mapping function. This represents the Sigmoid activation function; This represents the semantic interaction features captured in the r-th round; This represents the language memory feature vector. It is the dimension of the hidden layer state vector. Indicates a splicing operation; Step 6: Multimodal semantic fusion processing, including utilizing the sentence-level feature vector of the current problem obtained in Step 2.

1. The visual context feature vector obtained in step 4 and the language memory feature vector obtained in step 5 We perform feature fusion processing between visual and textual modalities to obtain the final high-level fused visual-language feature vector. The multimodal semantic fusion processing includes: Using the sentence-level feature vector of the current problem obtained in step 2.1 The visual context feature vector obtained in step 4 and the language memory feature vector obtained in step 5 We perform feature fusion processing between visual and textual modalities to obtain the final high-level fused visual-language feature vector. ; In the formula, This represents the initial multimodal fusion feature. , , , The parameters for model training, express Activation function The presentation layer standardizes operations and includes: In the formula, , ,and They represent respectively for , ,and The problem after mutual attention processing, the feature vector representation of visual and linguistic memory; and we have: In the formula, Indicates attention weights. The visual feature representation includes the characteristics of attention, which are related to problem-solving and language memory, and includes: In the formula, , , , The parameters represent the training parameters of the model; and we have: In the formula, the function Represents from vector to vector Attention calculation; , Represents the parameters used in model training; for and The calculations and the above are related to The operations are the same, respectively: and The final result This is a high-level fused visual language feature vector; Step 7: Decode the feature sequence of the predicted answer; Step 8: Parameter optimization of the visual dialogue generation network model based on human-like visual perception and language memory networks; Step 9: Predict candidate answers, including minimizing the loss function as the model trains. When the loss is minimized, the answer sequence corresponding to the minimum loss is taken as the candidate answer sequence predicted by the model.

2. The visual dialogue generation method based on human-like visual perception and language memory networks according to claim 1, characterized in that: Step 1 is implemented by including the following sub-steps: Step 1.1: Obtain the visual dialogue dataset, which contains sentence text and images; perform word segmentation on all the sentence text contained in the visual dialogue dataset to obtain the segmented sentence sequence; Step 1.2: Obtain all words from the segmented sentence sequence, and select all words whose word frequency is greater than a set threshold. Construct a word index table corresponding to all words in the dataset, denoted as Voc. Then, perform one-hot encoding on each word in the index table Voc to obtain a one-hot vector table. Step 1.3: Randomly initialize a word embedding matrix. By using word embedding matrix The one-hot vector of each word is mapped to the corresponding word vector to obtain the vector representation of each word.

3. The visual dialogue generation method based on human-like visual perception and language memory networks according to claim 2, characterized in that: Step 2 is implemented by including the following sub-steps: Step 2.1: Obtain any image I from the visual dialogue dataset, and use the object detection feature extractor to obtain the visual features of image I; Step 2.2: Obtain the dialogue history text corresponding to any image from the visual dialogue dataset, denoted as . Current problem And the real answer label is recorded as ; Step 2.3: Use a bidirectional long short-term memory network to process the r-th round of dialogue in the dialogue history text U from Step 2.

2. Semantic encoding yields the hidden layer state sequence as follows: The last hidden state feature vector is used. As the rth round of dialogue Sentence-level feature representation This yields the sentence-level features corresponding to the total dialogue history U. ; Step 2.4: Use a bidirectional long short-term memory network to address the current problem in step 2.

2. Semantic encoding is performed to extract the context feature vector of the current problem, denoted as , and the state feature of the last hidden layer of Bi-LSTM is taken. As the sentence-level feature vector of the current problem .

4. The visual dialogue generation method based on human-like visual perception and language memory networks according to claim 3, characterized in that: In step 3, The feature representation of the graph nodes and the edge weight representation between nodes are respectively... and , This represents the i-th image feature node. express The weights between the node and all other visual feature nodes.

5. The visual dialogue generation method based on human-like visual perception and language memory networks according to claim 4, characterized in that: Step 4 is implemented by using a multi-layer graph convolutional neural network to update the visual context perception map established in step 3 to obtain the global and local visual feature vectors of the human-like image.

6. The visual dialogue generation method based on human-like visual perception and language memory networks according to claim 5, characterized in that: Step 5 is implemented by first concatenating the dialogue history feature vectors from round 0 to round r to obtain the total dialogue history features. Then and Perform feature fusion to obtain topic feature vector representation. Finally, the dialogue history of all rounds is iteratively updated to learn and obtain the final language memory features. .

7. The visual dialogue generation method based on human-like visual perception and language memory networks according to claim 6, characterized in that: Step 6 is implemented by using the sentence-level feature vector of the current problem obtained in step 2.

1. The visual context feature vector obtained in step 4 and the language memory feature vector obtained in step 5 We perform feature fusion processing between visual and textual modalities to obtain the final high-level fused visual-language feature vector. .

8. The visual dialogue generation method based on human-like visual perception and language memory networks according to claim 7, characterized in that: Step 7 is implemented by selecting candidate answers. Similar to problem encoding, Bi-LSTM uses the last hidden state vector as... Sentence-level feature representation of each answer; set up It contains the probability values ​​of 100 candidate answers, and then takes... The value with the highest probability and its order are used as the final predicted answer value.

9. The visual dialogue generation method based on human-like visual perception and language memory networks according to claim 8, characterized in that: Step 8 is implemented by using the predicted answer calculated in step 7. With the real answer The loss cost function F between: In the formula, Indicates the first The first round of the problem The one-hot encoded vector representation of the true answer label of the candidate answer. Indicates the first The first round of the problem The probability value of the candidate answer.

10. A visual dialogue generation system based on human-like visual perception and language memory networks, characterized in that: This method is used to implement a visual dialogue generation method based on a human-like visual perception and language memory network as described in any one of claims 1-9.

Images