System and method for a dialog response generation system

Abstract

A computer-implemented method for training a dialogue response generation system and a dialogue response generation system are provided. The method includes: arranging a first multi-mode encoder-decoder for dialogue response generation or video description, the first multi-mode encoder-decoder having a first input and a first output, wherein the first multi-mode encoder-decoder has been pre-trained by training an audio-visual dataset with a video description sentence; arranging a second multi-mode encoder-decoder for dialogue response generation, the second multi-mode encoder-decoder having a second input and a second output; providing a first audio-visual dataset with a first corresponding video description sentence to the first input of the first multi-mode encoder-decoder, wherein the first encoder-decoder generates a first output value based on the first audio-visual dataset with the first corresponding description sentence; providing the first audio-visual dataset not including the first corresponding video description sentence to the second multi-mode encoder-decoder. In this case, the second multi-mode encoder-decoder generates a second output value based on the first audio-visual dataset without the first corresponding video description sentence.

Description

Technical Field

[0001] The present invention generally relates to a system and method for training a dialog response generation system, and more particularly to a training system and training method for a dialog response generation system and a dialog response generation system trained thereby. Background Art

[0002] Human-machine interfaces that can handle spoken conversations have revolutionized the way we interact with smartphone digital assistants, car navigation systems, voice-controlled smart speakers, and human-facing robots. Going forward, such systems will need the ability to adapt to other input modalities, including vision, to generate adequate responses in different user contexts or handle new situations that were not available during training. However, current state-of-the-art dialogue systems lack effective models for multimodal sensory inputs (e.g., vision, sound, and text) required to handle such dynamic scenarios, and thus may fail to generate appropriate responses in conversations.

[0003] In order to interact with humans about the environmental information around the user, the system needs to understand the context of the environment and the natural language input of the user. This scene-aware dialogue method is essential for human-computer interfaces for real-world applications. In order to respond to human behavior, machines need to understand the scene using multimodal information consisting of any type of physical signals (features) such as audio and video. The semantic representation of multimodal information that describes the scene in natural language is the most effective way to help generate system responses. Therefore, it is necessary to develop methods to improve the quality of dialogue response generation through multimodal scene understanding.

[0004] Recently, a new dialogue task using multimodal information processing, called Audio-Visual Scene-Aware Dialogue (AVSD), has been proposed. AVSD focuses on response sentence generation for dialogue systems, aiming to answer user questions about a provided video, where the system can use the audio-visual information in the video as well as the dialogue history up to the user's last question. Optionally, a manual video description sentence explaining the video clip can also be used as input to the system. Recently, the 7th Dialogue System Technical Challenge (7 th The approach proposed in the Dialog System Technology Challenge (DSTC7) for the AVSD task showed that the multimodal fusion of acoustic, visual, and textual information can effectively improve the quality of responses. Furthermore, it was found that the best performance can be achieved when applying text features extracted from “manual” video description sentences. However, such manual video description sentences are not available in the real world, which poses challenges during deployment.

[0005] In order to improve the performance of response generation without using manual video description sentences at the inference stage, a new method is needed to transfer the performance gain obtained by applying manual video description sentences at training time to generate more accurate responses. Summary of the invention

[0006] According to some embodiments of the present invention, a computer-implemented method for training a dialogue response generation system and a dialogue response generation system may be provided. The method may include: arranging a first multi-mode encoder-decoder for dialogue response generation or video description, the first multi-mode encoder-decoder having a first input and a first output, wherein the first multi-mode encoder-decoder has been pre-trained by training an audio-visual data set with a training video description sentence; arranging a second multi-mode encoder-decoder for dialogue response generation, the second multi-mode encoder-decoder having a second input and a second output; providing a first audio-visual data set with a first corresponding video description sentence to the first input of the first multi-mode encoder-decoder, wherein the first encoder-decoder generates a first output value based on the first audio-visual data set with the first corresponding description sentence; providing the first audio-visual data set not including the first corresponding video description sentence to the second multi-mode encoder-decoder. In this case, the second multi-mode encoder-decoder generates a second output value based on the first audio-visual data set without the first corresponding video description sentence.

[0007] In some cases, the automatic video description sentence output from the first multimodal encoder-decoder can be input into the second multimodal encoder-decoder for dialogue response generation. In addition, the video description features (which are context vectors extracted from the first multimodal encoder-decoder for automatic video description) can be embedded into the second multimodal encoder-decoder for dialogue response generation to consider the semantic representation of multimodal information, thereby describing the scene using natural language.

[0008] Furthermore, in some cases, when a first multimodal encoder-decoder for dialogue response generation (teacher network) is trained using manual video description sentences, a second multimodal encoder-decoder (student network) can be trained to transfer the performance gains obtained by the teacher network for dialogue response generation to the student network.

[0009] In addition, the context vector output from the first multi-modal encoder-decoder for video description can be embedded in the second multi-modal encoder-decoder for dialogue response generation, wherein the automatic video description sentence obtained from the first multi-modal encoder-decoder can be used instead of the manual description sentence. In this case, the above embodiments can be combined to use the output of the automatic video description network and the intermediate representation of the output to generate more accurate dialogue responses based on audio-visual scene understanding.

[0010] The presently disclosed embodiments will be further explained with reference to the accompanying drawings. The drawings shown are not necessarily drawn to scale, with emphasis instead generally being placed upon illustrating the principles of the presently disclosed embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] [ Figure 1 ]

[0012] Figure 1 is a block diagram illustrating a multi-modal fusion system according to some embodiments of the present disclosure.

[0013] [ Figure 2A ]

[0014] Figure 2A is a block diagram illustrating an AVSD system using a multi-modal fusion method according to an embodiment of the present disclosure.

[0015] [ Figure 2B ]

[0016] Figure 2B is a block diagram illustrating a student-teacher learning system for an AVSD system according to some embodiments of the present invention.

[0017] [ Figure 3 ]

[0018] Figure 3 is a diagram illustrating how to train an AVSD system using an automatic video description encoder-decoder according to an embodiment of the present invention.

[0019] [ Figure 4 ]

[0020] Figure 4 Statistics of a video scene-aware dialogue dataset according to some embodiments of the present invention are shown.

[0021] [ Figure 5A ]

[0022] Figure 5A Evaluation results of an AVSD trial reasoning set with a single reference according to an embodiment of the present invention are shown.

[0023] [ Figure 5B ]

[0024] Figure 5B Evaluation results for an AVSD formal reasoning set with six references per response according to an embodiment of the present invention are shown. DETAILED DESCRIPTION

[0025] Although the above-labeled drawings illustrate the presently disclosed embodiments, other embodiments may be conceived as noted in the discussion. The present disclosure presents illustrative embodiments by way of representation and not limitation. Those skilled in the art may devise numerous other modifications and embodiments that fall within the scope and spirit of the principles of the presently disclosed embodiments.

[0026] The following description provides only exemplary embodiments and is not intended to limit the scope, applicability or configuration of the present disclosure. Rather, the following description of exemplary embodiments will provide a feasible description for implementing one or more exemplary embodiments to those skilled in the art. Various changes that can be made in the functions and arrangements of elements are contemplated without departing from the spirit and scope of the disclosed subject matter as set forth in the appended claims.

[0027] Figure 1 is a block diagram illustrating a multi-modal fusion system according to some embodiments of the present invention.

[0028] The present disclosure is based on a multi-modal "fusion" system 200 that generates a context vector 220 from input data including multiple modes 211. In some cases, the multi-modal fusion system 200 receives input features including text features 201, image (video) features 202, sound features 203, and motion features extracted from video features 202 and generates a dialog system response 231 related to the input features 211, such as Figure 2A The text input 201 may include manual video description 209 or automatic video description 391 , user input such as questions 208 and conversation history 207 .

[0029] Figure 3 is a diagram showing how to use an automatic video description encoder-decoder to train an AVSD system according to an embodiment of the present invention. The figure shows a first multimodal encoder-decoder 350 for video description and a second multimodal encoder-decoder 300 for dialogue response generation. In this case, the input is multimodal features 303 and the output is natural language 341, 391.

[0030] Some embodiments of the present disclosure are based on generating context vectors for automatic video description 380, audio-visual fusion 330, and context vectors 335 for dialogue system responses, wherein the audio-visual context vectors 330 from input data including "multiple modalities" 303 are combined with context vectors of questions 331, dialogue history 332, and embedded context vectors of automatic video description 380, such as Figure 3In some cases, the pattern may be text features 331 and 332 , 333 , video features (image features) 301 , sound features 302 , and motion features extracted from video features 301 .

[0031] The present disclosure is based on a multi-modal "fusion" system 210, which includes Figure 2A Input data of multiple modes 211 shown generates context vector 220. In some cases, multimodal fusion system 210 receives input features including text features 201, image (video) features 202, sound features 203, and motion features extracted from video features 202, and generates a dialogue system response 231 related to the input features 211.

[0032] Some embodiments of the present disclosure are based on generating a context vector 333 obtained from a first multimodal encoder-decoder 350 for automatic video description, wherein the automatic video description sentences 391 are input as text features 333 to a second multimodal encoder-decoder 300 for dialogue response generation, rather than being input as manual video description sentences 201 associated with an automatic video dataset.

[0033] Furthermore, the context vector output 380 from the encoder of the first multi-modal encoder-decoder 350 for video description may be embedded into the context vector of the dialogue response sentence 335 input to the decoder of the second multi-modal encoder-decoder 300 for dialogue response generation.

[0034] In addition, some embodiments of the present invention may provide a system or method that can address how to compensate for the performance gain obtained by applying manual video description sentences in the reasoning stage (which is missing in the reasoning stage but available in the training stage) to improve the system response quality without using manual video description sentences.

[0035] To transfer the performance gains obtained by applying manual video description sentences at training time to the inference stage, we can Figure 2BThe student-teacher learning method 290 shown is used to train the AVSD system. The teacher model 250 for dialogue response generation based on the first multimodal encoder-decoder is first trained with manual video description sentences, and then the student model 210 based on the second multimodal encoder-decoder for dialogue response generation is trained without manual video description to simulate the teacher's output 281. The student model 210 is used in the inference phase. The framework can be extended to joint student-teacher learning, in which the two models are trained simultaneously, not only to reduce their own loss functions, but also to have hidden representations 230 and 270 with similar context vectors to each other. In this learning, since the context vector of the teacher model 270 is close to the context vector of the student model 230, the teacher model 250 is updated to be more easily simulated by the student model 210. The new system using student-teacher learning 290 achieves better performance without using manual video description sentences. In addition, it is competitive with those systems trained with manual video description sentences.

[0036] In addition, other embodiments may be based on a pair of first multi-mode encoder-decoder 210 and second multi-mode encoder-decoder 250, respectively, for example Figure 2B Dialogue response generation is shown, wherein one multimodal encoder-decoder is named teacher network 250 trained by inputting manual video description sentences 209, and another multimodal encoder-decoder is named student network 210 is trained without manual video description sentences. The second multimodal encoder-decoder 210 trained without manual video description sentences 209 is applied to infer dialogue response generation.

[0037] Training Methods

[0038] According to some embodiments of the present disclosure, a computer-implemented method for training a dialogue response generation system includes the following steps: arranging a first multi-mode encoder-decoder 250, 350 for video description or dialogue response generation, the first multi-mode encoder-decoder having a first input and a first output, wherein the first multi-mode encoder-decoder has been pre-trained by training an audio-visual dataset using a video description sentence 209; arranging a second multi-mode encoder-decoder 300, 210 for dialogue response generation, the second multi-mode encoder-decoder 300, 210 having a second input and a second output; providing the first audio-visual dataset having a first corresponding video description sentence 209 to the second multi-mode encoder-decoder 300, 210; A first input end of a multi-mode encoder-decoder 350, 250, wherein the first encoder-decoder generates a first output value based on a first audio-visual data set having a first corresponding video description sentence 209; providing the first audio-visual data set not including the first corresponding video description sentence 209 to a second multi-mode encoder-decoder 210 for dialogue response generation, wherein the second multi-mode encoder-decoder generates a second output value based on the first audio-visual data set without the first corresponding video description sentence 209, wherein an optimizer module updates network parameters of the second multi-mode encoder-decoder until an error between the first output value and the second output value is reduced to a predetermined range, wherein the error is calculated based on a loss function.

[0039] Training System

[0040] In addition, another embodiment of the present invention can provide a system (training system) for training a dialogue response generation system. Figure 1The training system includes a memory 140 and one or more storage devices 130 for storing instructions of a computer-implemented method; one or more processors 120, which are associated with the operable memory 140 and one or more storage devices 130, and when executed by the one or more processors 120, cause the one or more processors 120 to perform operations including the following steps: arranging a first multi-mode encoder-decoder 210 for video description or dialogue response generation, the first multi-mode encoder-decoder 210 having a first input terminal and a first output terminal via 110, wherein the first multi-mode encoder-decoder 210 has been pre-trained by training an audio-visual data set 195 using a training video description sentence 195; arranging a second multi-mode encoder-decoder 210 for dialogue response generation, the second multi-mode encoder-decoder 210 having There is a second input terminal and a second output terminal via 110; a first audio-visual data set 195 having a first corresponding description sentence 195 is provided to a first input terminal of a first multi-mode encoder-decoder 210, wherein the first multi-mode encoder-decoder 210 generates a first output value based on the first audio-visual data set 195 having the first corresponding description sentence 195; the first audio-visual data set 195 not including the first corresponding description sentence 195 is provided to a second multi-mode encoder-decoder 210, wherein the second multi-mode encoder-decoder 210 generates a second output value based on the first audio-visual data set 195 without the first corresponding description sentence, wherein the optimizer module updates a second network parameter of the second multi-mode encoder-decoder 210 until the error between the first output value and the second output value is reduced to a predetermined range, wherein the error is calculated based on a loss function.

[0041] Reasoning System

[0042] Furthermore, according to some embodiments of the present invention, it is possible to provide Figure 1 In this case, the dialog response generation system includes a memory 140 storing instructions of a multi-mode encoder-decoder 210 and one or more storage devices 130, wherein the multi-mode encoder-decoder 210 has been stored in 130 ( Figure 1The method is not shown in the figure; one or more processors 120 are associated with one or more storage devices 130 and memory 140. When executed by the one or more processors 120, the memory 140 and the one or more storage devices 130 are operable to cause the one or more processors 120 to perform operations including the following steps: receiving a first input vector and a second input vector according to a first sequential interval and a second sequential interval; extracting a first feature vector and a second feature vector from the first input terminal and the second input terminal using a first feature extractor and a second feature extractor stored in 130, respectively; extracting a first feature vector and a second feature vector from the first feature vector and the second feature vector and a prestep context vector of the sequence generator ... vector) estimate a first set of weights and a second set of weights; calculate a first context vector from the first set of weights and the first feature vector, and calculate a second context vector from the second set of weights and the second feature vector; convert the first context vector into a first pattern context vector having a predetermined dimension, and convert the second context vector into a second pattern context vector having a predetermined dimension; estimate a pattern attention weight group from the precondition context vector and the first context vector and the second context vector or the first pattern context vector and the second pattern context vector; generate a weighted context vector having a predetermined dimension from the pattern attention weight group and the first pattern context vector and the second pattern context vector; and generate a predicted word using a sequence generator for generating a word sequence from the weighted context vectors.

[0043] Specific details are given in the following description to provide a thorough understanding of the embodiments. However, it will be appreciated by those skilled in the art that the embodiments may be practiced without these specific details. For example, the systems, processes and other elements in the disclosed subject matter may be shown as components in block diagram form to avoid confusing the embodiments with unnecessary details. In other cases, well-known processes, structures and techniques may be shown without unnecessary details to avoid confusing the embodiments. In addition, the same reference numerals and marks in the various figures indicate the same elements.

[0044] In addition, each embodiment can be described as a process, which is depicted as a flow chart, a flow chart, a data flow diagram, a structure diagram or a block diagram. Although a flow chart can describe an operation as a sequential process, many operations can be performed in parallel or simultaneously. In addition, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but there may be additional steps that are not discussed or included in the figure. In addition, not all operations in any specifically described process will occur in all embodiments. The process can correspond to a method, a function, a process, a subroutine, a subprogram, etc. When the process corresponds to a function, the termination of the function can correspond to the function returning to the calling function or the main function.

[0045] In addition, embodiments of the disclosed subject matter may be implemented at least in part manually or automatically. Manual or automatic implementation may be performed or at least assisted by the use of a machine, hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof. When implemented in software, firmware, middleware, or microcode, program code or code segments that perform the necessary tasks may be stored in a machine-readable medium. A processor may perform the necessary tasks.

[0046] Figure 1 A block diagram of a dialog response generation system 100 according to some embodiments of the present invention is shown. The system 100 may include: a human-machine interface (HMI) 110 having an input / output (I / O) interface, which may be connected to a keyboard 111 and a pointing device / medium 112; a microphone 113; a receiver 114; a transmitter 115; a 3D sensor 116; a global positioning system (GPS) 117; one or more I / O interfaces 118; a processor 120; a storage device 130; a memory 140; a network interface controller 150 (NIC) that may be connected to a network 190 including a local area network and the Internet (not shown); a display interface 160 that is connected to a display device 165; an imaging interface 170 that may be connected to an imaging device 175, wherein the imaging device 175 includes a camera that may obtain pictures and moving images (video features) with / without acoustic features; and a printer interface 180 that may be connected to a printing device 185. The HMI with I / O interface 110 may include an analog / digital converter and a digital / analog converter. The HMI with I / O interface 110 includes a wireless communication interface, which can communicate with other 3D point cloud display systems or other computers via a wireless Internet connection or a wireless local area network, thereby enabling the construction of multiple 3D point clouds. The system 100 may include a power supply 190. The power supply 190 may be a battery that can be charged from an external power source (not shown) via the I / O interface 118. Depending on the application, the power supply 190 may optionally be located outside the system 100.

[0047] The HMI and I / O interface 110 and the I / O interface 118 may be adapted to connect to another display device (not shown) including a computer monitor, camera, television, projector, or mobile device, etc. The interfaces 110 and 118 are configured to receive video features and sound features, and these features may be provided via the network 190 .

[0048] The system 100 may receive an electronic text / image document 195 including voice data via a network 190 connected to the NIC 150. The storage device 130 includes a sequence generation model 131, a feature extraction model 132, and a multi-mode encoder-decoder 200, wherein algorithms of the sequence generation model 131, the feature extraction model 132, and the multi-mode encoder-decoder 200 are stored in the storage 130 as program code data. The algorithms of the models 131-132 and 200 may be stored to a computer-readable recording medium (not shown) so that the processor 120 may execute the algorithms of the models 131-132 and the multi-mode encoder-decoder 200 by loading the algorithms from the medium. In addition, the pointing device / medium 112 may include a module for reading and executing a program stored on a computer-readable recording medium.

[0049] To begin executing the algorithms of the models 131-132 and the multi-modal encoder-decoder 200, instructions may be sent to the system 100 using the keyboard 111, the pointing device / medium 112, or via a wireless network or network 190 connected to other computers (not shown). The algorithms of the models 131-132 and 200 may be initiated in response to receiving acoustic features or video features via the display interface 160 or the network 190 using a pre-installed conventional speech recognition program (not shown) stored in the memory 130. For example, the system 100 includes an on / off switch (not shown) to allow a user to start / stop operating the system 100.

[0050] The HMI and I / O interface 110 may include an analog-to-digital (A / D) converter, a digital-to-analog (D / A) converter, and a wireless signal antenna for connecting to a network 190. In addition, one or more I / O interfaces 118 may be connected to a cable television (TV) network, an optical fiber network, or a conventional television (TV) antenna that receives TV signals and multi-mode information signals. The signals received via the interface 118 may be converted into digital image and audio signals, which may be processed in accordance with the algorithms of the models 131-132 and 200 in conjunction with the processor 120 and the memory 140, thereby generating and displaying a video script on the display device 165 using picture frames of digital images, while outputting the acoustic sound of the TV signal via the speaker 19. The speaker may be included in the system 100, or an external speaker may be connected via the interface 110 or the I / O interface 118.

[0051] The processor 120 may be a plurality of processors including one or more graphics processing units (GPUs). The memory 130 may include a speech recognition algorithm (not shown) that may recognize a speech signal obtained via the microphone 113 .

[0052] The multi-mode encoder-decoder system module 200, the sequence generation model 131 and the feature extraction model 132 can be formed by a neural network.

[0053] Some embodiments of the present invention are based on the recognition that student-teacher learning can be a transfer learning technique, where knowledge in a teacher model is transferred to a student model. This can be used for model compression, where a small model is trained to mimic the output of a large model with higher prediction accuracy. Student-teacher learning can bring the performance of a small model closer to that of a large model, while retaining the benefits of a small model's reduced computational cost and memory consumption.

[0054] Student-teacher learning can also be used to compensate for missing information in the input. In this case, the teacher model is trained to predict the target label using additional information, but the student model is trained to simulate the teacher's output without that information. For example, in automatic speech recognition (ASR), the teacher model is trained with augmented speech obtained through a microphone array, while the student model is trained to simulate the teacher's output for the same speech, but using only noisy speech recorded in a single channel. With this approach, the student model can improve performance without a microphone array at inference time. This technique has also been used for domain adaptation between children and adult speech. The proposed AVSD system adopts this approach to compensate for missing video descriptions. The student model can generate better responses without description features. We further extend this framework to joint teacher-student learning, aiming to improve the teacher model into a better teacher for the student model.

[0055] Figure 2A is a block diagram showing the architecture of an audio-visual scene-aware dialogue system (architecture) based on a computer-implemented multimodal attention model (method) 200 according to an embodiment of the present disclosure.

[0056] The system generates a context vector 220 from input data including multiple modes 211. In some cases, the multimodal fusion system 200 receives input features including text features 201, image (video) features 202, sound features 203, and motion features extracted from video features 202, and generates a dialogue system response 231 related to the input features 211. The text input 201 may include a manual video description 209 or an automatic video description 391, user input such as questions 208, and a dialogue history 207.

[0057] This figure shows an example of the architecture of the proposed AVSD system according to an embodiment of the present invention. The model (method) 200 adopts attention-based encoder-decoders 210 and 230, which enable the network to emphasize features from a specific time frame according to the current context, so that the next word can be generated more accurately. The efficacy of attention models has been demonstrated in many tasks such as machine translation and video description.

[0058] The attention-based encoder-decoder is designed to process sequence-to-sequence mapping using a recurrent neural network (RNN). Let X and Y be the input sequence and output sequence, respectively. The model computes the posterior probability distribution P(Y|X). For the AVSD task, X includes all input information, such as the user's question 208, audio-visual features 202 and 203, and conversation context (conversation history) 207. Y is the system response 231 to be generated, which answers the user's question 208. The most likely hypothesis for Y is obtained as follows:

[0059]

[0060] Among them, V * represents a sequence of zero or more words in the system vocabulary V, where each word y i is a word in the response.

[0061] Let X = {X 1 , ..., X K} is a set of input sequences, where X k is the kth input sequence, which can represent the user's question 208, a sequence of feature vectors extracted from the target video clips 202 and 203, or a conversation history 207 including all previous questions and answers in a conversation about a video clip. To generate the system response Y, each input sequence in X is first encoded into a better representation using the corresponding encoder.

[0062] If X k is the user's question 208, then the statement Q = w Q，1 , ..., w Q，N Encode using word embedding and BLSTM layer. If X k is the video feature sequence It can be extracted from a sequence of images from a video clip using a pre-trained CNN such as VGG-16, C3D, or I3D, which were originally trained for image or video classification tasks. In the case of C3D and I3D, multiple images are fed to the network simultaneously to capture dynamic features in the video. Sound features can also be extracted in a similar manner using a pre-trained CNN such as SoundNet or VGGish. Each sequence of feature vectors is encoded into an appropriate representation using a single projection layer. To reduce the dimension. If X k is the conversation history, which can be a series of question-answer pairs H=H that appeared before the current question in the conversation 1 , ..., H J H is encoded using a hierarchical LSTM encoder, where each question-answer pair is first encoded into a fixed-dimensional vector H using a sentence embedding LSTM j , and then an additional BLSTM layer is used to further embed the sentence embedding sequence.

[0063] Decoder 230 starts with a sentence marker <sos>Starts predicting the next word iteratively until it predicts the end-of-sentence token <eos>Given a decoder state s i-1 , decoder network λ D The probability distribution of the next word is inferred to be

[0064]

[0065] And according to Generate the word y with the highest probability i .

[0066] Using the decoder’s LSTM network, the decoder state is updated as

[0067] s i =LSTM(s i-1 , [y′ i , g i ];λ D ), (5)

[0068] Among them, y′ i Yes i The word embedding vector, g i is a context vector 220 that includes input information related to the previous decoder state. D Represents a decoder parameter set.

[0069] The context vector 220 is obtained through a hierarchical attention mechanism, which first aggregates the frame-level hidden vectors of each input sequence into a per-mode context vector c k，i , and then the context vector c 1，i , ..., c K，i Fusion into a single context vector g i The attention mechanism is implemented by applying attention weights to the hidden activation vectors across the entire input sequence. These weights enable the network to emphasize the features at the time steps that are most important for predicting the next output word.

[0070] Let α k，i，t is the attention weight between the i-th output word and the t-th input feature vector from the k-th pattern. For the t-th output, a vector representing the relevant context of the input sequence is obtained as a weighted sum of the hidden unit activation vectors:

[0071]

[0072] Among them, h k，t is the t-th output vector of the k-th encoder. The attention weights are calculated as follows:

[0073]

[0074] and

[0075]

[0076] Among them, W Ak and V Ak is a matrix, w Ak and b Ak is a vector, and e k，i，t is a scalar.

[0077] The model also utilizes a multimodal attention mechanism. In order to fuse multimodal information, the attention mechanism is extended from temporal attention (attention over time) to attention on the mode. The following equation shows the method of performing attention-based feature fusion:

[0078]

[0079] in,

[0080]

[0081] And c k，i is the context vector obtained using the kth input modality. A similar mechanism for temporal attention is applied to obtain the multimodal attention weights β k，i . 0

[0082]

[0083] in,

[0084]

[0085] Here, the multimodal attention weights are given by the decoder state S i and the context vector c for each pattern k，i And the temporal attention weight of each mode is determined. W B and V Bk is a matrix, w B and b Bk is a vector, and v k，n is a scalar. These weights can change depending on the decoder state and the context vector from each encoder. This enables the decoder network to pay attention to a different set of features and / or patterns when predicting each subsequent word in the description.

[0086] Student-Teacher Learning (Please use Figure 2B The label numbers in the

[0087] Figure 2B is a block diagram illustrating student-teacher learning of an AVSD system according to some embodiments of the present invention. The AVSD system includes a student network 210 and a teacher network 250. The figure depicts the concept of student-teacher learning of the AVSD system. The goal of this step is to obtain a student model 210 that does not use video description text, which is trained to simulate a teacher model 250 that has been trained using video description text. Therefore, the student model 210 can be used to generate system responses without relying on description text, while hoping to achieve performance similar to that of the teacher model 250.

[0088] Following the best system in the DSTC7-AVSD track, we insert the description text 209 at the beginning of each question. This means that in each round of dialogue about the target video clip, the same description is always fed to the encoder together with the new question. The student network 210 is trained to reduce the cross entropy loss by using the output of the teacher network 250 as a soft target to make the output distribution of the student model 210 closer to the output distribution of the teacher model 250.

[0089] In this disclosure, we study three loss functions for student-teacher learning290. The first is the cross-entropy loss with a soft target:

[0090]

[0091] in, represents the probability distribution of the i-th word obtained by the teacher network 250, and and is the state and context vector 270 generated by the teacher network 250 for the training sample (X, Y). Here, P(y|s i-1 , g i ) is the posterior distribution from the current student network 210 (which is being trained), which is predicted without the description text 209.

[0092] The second loss function further incorporates the context vector similarity as

[0093] L' ST (X, Y) = L ST (X,Y)+λ c L MSE (X, Y) (14)

[0094] in, where MSE(·,·) represents the mean square error between the two context vectors 230 and 270, and λ c represents the scaling factor. Our goal is to compensate for the missing input features at the context vector level, which is expected to more aggressively exploit other modes.

[0095] The last loss function we consider is joint student-teacher learning. The parameters of the teacher network 250 are usually kept fixed throughout the training phase. However, in the joint training method, we update not only the student network 210 but also the teacher network 250. The loss function is calculated as

[0096]

[0097] in, is the standard cross entropy of the hard target Y, which is only used for the teacher network 250 in the back-propagation process. Similarly, For student network 210 only, Used in both networks.

[0098] Figure 3 is a block diagram illustrating some embodiments of the present disclosure based on generating a context vector 333 obtained from a first multimodal encoder-decoder 350 for automatic video description, wherein the automatic video description sentence 391 is input to a second multimodal encoder-decoder for dialogue response generation 300 as a text feature 333 input rather than being input as a manual video description sentence 201 associated with an automatic video dataset.

[0099] Furthermore, the context vector output 380 from the encoder of the first multi-modal encoder-decoder 350 for video description may be embedded into the context vector of the dialogue response sentence 335 input to the decoder of the second multi-modal encoder-decoder 300 for dialogue response generation.

[0100] Figure 4 Statistics of a video scene-aware conversation dataset according to some embodiments of the present invention are shown. The AVSD dataset is a collection of text-based conversations about short videos. The video clips originally come from the Charades dataset, an untrimmed multi-action dataset containing 11,848 videos, 7985 for training, 1863 for validation, and 2000 for inference. It has 157 action categories with several fine-grained actions. In addition, the dataset provides 27,847 text descriptions for the videos; each video is described using 1-3 sentences. For each video in the Charades dataset, the AVSD dataset contains a text conversation between two people discussing the video.

[0101] AVSD System

[0102] Figure 2A 200, which indicates how to train the AVSD system according to an embodiment of the present invention. The question encoder has a word embedding layer (200 dimensions) and two BLSTM layers (256 dimensions in each direction). The audio-visual features consisting of I3D-rgb (2048 dimensions), I3D-stream (2048 dimensions) and VGGish (128 dimensions) are extracted from the video frames using a pre-trained deep CNN. Then, those feature sequences are fed to a multi-mode encoder with a single projection layer to convert them into 512, 512 and 64 dimensional vectors, respectively. The history encoder has a word embedding layer (200 dimensions) and two LSTM layers for question-answer pair embedding (256 dimensions) and a 1-layer BLSTM for embedding history (256 dimensions in each direction). We use the ADAM optimizer for training, in which, if the verification perplexity does not decrease after each epoch, the learning rate is halved and training continues for up to 20 epochs. The vocabulary size is 3910, of which we only keep words that appear at least four times in the training set.

[0103] Figure 5A Evaluation results on the AVSD experimental reasoning set with a single reference are shown according to an embodiment of the present invention. The quality of the system response is measured using objective scores such as BLEU, METEOR, ROUGE-L, and CIDEr, which are based on the degree of overlap of words with the reference. A baseline system provided by the DSTC7-AVSD track organizers is also evaluated, which is a simple LSTM-based encoder-decoder that uses the same audio-visual features as ours. We also show results for the best system on AVSD. This system has a similar architecture to ours, but only has two encoders: one for the question and the other for the video features obtained via a 3D ResNet. This network is also pre-trained using the How2 dataset, while our model is trained only on the AVSD dataset.

[0104] While our system outperforms the best AVSD system when trained and inferenced with manual video description sentences ("man.man." in the second column), performance drops significantly when no description is fed to the network at inference time ("man.-"). When we provide automatic descriptions instead of manual descriptions ("man.auto"), using a video description model trained on the same AVSD dataset, the improvement is limited. The model trained without descriptions ("--") performs slightly better than the other conditions.

[0105] Next, we apply the loss L ST The trained model provides significant gains on all objective metrics (e.g., BLEU4: 0.105 → 0.113, METEOR: 0.135 → 0.143), which are closer to the metrics obtained using manual video description sentences (e.g., BLEU4: 0.117, METEOR: 0.150). We also apply a loss function L that takes into account the similarity of the context vectors. ST , but the response quality is almost the same as L ST Finally, we have L JST Joint student-teacher learning was performed and further improvements were obtained on most objective metrics (e.g., BLEU4: 0.113 → 0.115, METEOR: 0.143 → 0.144).

[0106] Figure 5B Evaluation results for the AVSD formal reasoning set are shown, with six references per response. Similar to FIG6A , the system according to the present invention outperforms other systems including the best system of DSTC7. The student-teacher framework also provides significant gains for the formal reasoning set.

[0107] As described above, some embodiments according to the present invention may provide a computer-implemented method for compensating for the lack of video description features available during training at the time of inference. The present invention may provide a student-teacher learning framework for audio-visual scene perception dialogue (AVSD). It was found that the AVSD system according to the present invention can achieve better performance than previous methods, which is more competitive than systems trained with manual video description sentences, and is also better than the best DSTC7-AVSD system. The trained model can answer questions about the context of the video by fusing text information, audio information, and visual information related to the video, and generate high-quality responses without relying on manual video description sentences. In addition, another embodiment of the present invention may provide a joint student-teacher learning method that can achieve further gains in most objective indicators.

[0108] In some embodiments of the present disclosure, when the above-mentioned multimodal fusion model is installed in a computer system, video scripts can be effectively generated with less computing power, so using a multimodal fusion model method or system can reduce central processing unit usage and power consumption.

[0109] In addition, an effective method for executing a multi-modal fusion model is provided according to an embodiment of the present disclosure, so that the use of a method and system utilizing a multi-modal fusion model can reduce central processing unit (CPU) usage, power consumption, and / or network bandwidth usage.

[0110] The above-mentioned embodiments of the present disclosure may be implemented in any of a variety of ways. For example, the embodiments may be implemented using hardware, software, or a combination thereof. When implemented in software, the software code may be executed on any suitable processor or set of processors, whether provided in a single computer or distributed in multiple computers. Such processors may be implemented as integrated circuits, with one or more processors in an integrated circuit assembly. However, the processor may be implemented using circuits of any suitable format.

[0111] In addition, the various methods or processes outlined herein may be encoded as software that can be executed on one or more processors using any of a variety of operating systems or platforms. In addition, such software may be written using any of a variety of suitable programming languages ​​and / or programming or scripting tools, and may also be compiled into executable machine language code or intermediate code that is executed on a framework or virtual machine. Typically, the functionality of the program modules may be combined or distributed as desired in various implementations.

[0112] In addition, the embodiments of the present disclosure can be implemented as methods, and examples thereof have been provided. The actions performed as a part of the method can be sorted in any suitable manner. Therefore, it is possible to construct an embodiment in which the actions are performed in a sequence different from that shown, which can include performing some actions simultaneously, even if they are shown as sequential actions in illustrative embodiments. In addition, the use of sequential terms such as first, second, etc. in the claims to modify the claim elements does not itself imply any priority, priority or order of a claim element, or the time sequence of the actions of the execution method, but is only used as a label to distinguish a claim element with a specific name from another element with the same name (but for ordinal terms) to distinguish the claim elements.

[0113] Although the present disclosure has been described with reference to certain preferred embodiments, it should be understood that various other changes and modifications may be made within the spirit and scope of the present disclosure. Therefore, the aspects of the appended claims are to cover all such changes and modifications that fall within the true spirit and scope of the present disclosure.< / eos> < / sos>

Claims

1. A computer-implemented method for training a dialog response generation system, the computer-implemented method comprising the following steps: pre-training a first multimodal encoder-decoder by training an audio-visual dataset with video training descriptions; arranging said first multi-mode encoder-decoder having a first input and a first output; arranging a second multi-mode encoder-decoder having a second input and a second output; providing a first audio-visual data set having a first corresponding descriptive sentence to the first input of the first multimodal encoder-decoder, wherein the first attention-based encoder-decoder generates a first output value based on the first audio-visual data set having the first corresponding descriptive sentence; A first audio-visual data set that does not include the first corresponding descriptive sentence is provided to the second multi-mode encoder-decoder, wherein the second multi-mode encoder-decoder generates a second output value based on the first audio-visual data set without the first corresponding descriptive sentence, wherein an optimizer module updates a second network parameter of the second multi-mode encoder-decoder until an error between the first output value and the second output value is reduced to a predetermined range, wherein the error is calculated based on a loss function.

2. The computer-implemented method of claim 1 , wherein: The loss function is a cross entropy loss function.

3. A computer-implemented method according to claim 2, wherein the loss function incorporates the mean square error between the context vector of the first multi-mode encoder-decoder and the context vector of the second multi-mode encoder-decoder.

4. The computer-implemented method of claim 1 , wherein: The first parameters of the first multi-mode encoder-decoder are not updated.

5. The computer-implemented method of claim 1 , wherein: The optimizer module updates first parameters of the first multi-mode encoder-decoder based on a cross entropy loss function.

6. The computer-implemented method of claim 1 , wherein: The optimizer module updates second network parameters of the second multi-mode encoder-decoder using a back-propagation method.

7. The computer-implemented method of claim 1 , further comprising: The second audiovisual data set is provided to the first input of the first multimode encoder-decoder to generate a third audiovisual data set, wherein the generated third audiovisual data set is further provided to the second multimode encoder-decoder to further update the second network parameters.

8. A system for training a dialog response generation system, the system comprising: a memory and one or more storage devices storing instructions to be executed by one or more processors; The one or more processors, the one or more processors are connected to the memory and the one or more storage devices, and when executed by the one or more processors, the memory and the one or more storage devices are operable to cause the one or more processors to perform operations including the following steps: pre-training a first multimodal encoder-decoder by training an audio-visual dataset with video training descriptions; arranging said first multi-mode encoder-decoder having a first input and a first output; arranging a second multi-mode encoder-decoder having a second input and a second output; providing a first audio-visual data set having a first corresponding descriptive sentence to a first input of the first multimodal encoder-decoder, wherein the first attention-based encoder-decoder generates a first output value based on the first audio-visual data set having the first corresponding descriptive sentence; A first audio-visual data set that does not include the first corresponding descriptive sentence is provided to the second multi-mode encoder-decoder, wherein the second multi-mode encoder-decoder generates a second output value based on the first audio-visual data set without the first corresponding descriptive sentence, wherein an optimizer module updates a second network parameter of the second multi-mode encoder-decoder until an error between the first output value and the second output value is reduced to a predetermined range, wherein the error is calculated based on a loss function.

9. The system according to claim 8, wherein: The loss function is a cross entropy loss function.

10. The system of claim 9, wherein the loss function incorporates a mean square error between the context vector of the first multi-mode encoder-decoder and the context vector of the second multi-mode encoder-decoder.

11. The system according to claim 8, wherein: The first parameters of the first multi-mode encoder-decoder are not updated.

12. The system according to claim 8, wherein: The optimizer module updates first parameters of the first multi-mode encoder-decoder based on a cross entropy loss function.

13. The system according to claim 8, wherein: The optimizer module updates second network parameters of the second multi-mode encoder-decoder using a back-propagation method.

14. The system according to claim 8, further comprising: The second audiovisual data set is provided to the first input of the first multi-mode encoder-decoder to generate a third audiovisual data set, wherein the generated third audiovisual data set is further provided to the second multi-mode encoder-decoder to further update the second network parameters.

15. A dialogue response generation system, the dialogue response generation system being trained by the method according to claim 1, the dialogue response generation system comprising: a memory and one or more storage devices storing instructions for a multi-mode encoder-decoder; one or more processors, the one or more processors being connected to the memory and the one or more storage devices, which when executed by the one or more processors are operable to cause the one or more processors to perform operations comprising the following steps: receiving a first input vector and a second input vector according to a first sequential interval and a second sequential interval; extracting a first feature vector and a second feature vector from the first input terminal and the second input terminal using a first feature extractor and a second feature extractor, respectively; estimating a first set of weights and a second set of weights from a precondition context vector of a sequence generator, the first feature vector, and the second feature vector, respectively; computing a first context vector from the first set of weights and the first feature vector, and computing a second context vector from the second set of weights and the second feature vector; converting the first context vector into a first mode context vector having a predetermined dimension, and converting the second context vector into a second mode context vector having the predetermined dimension; estimating a set of mode attention weights from the precondition context vector and the first context vector and the second context vector or the first mode context vector and the second mode context vector; generating a weighted context vector having the predetermined dimension from the mode attention weight group and the first mode context vector and the second mode context vector; as well as Predicted terms are generated using the sequence generator for generating a term sequence from the weighted context vector.

16. The dialog response generation system according to claim 15, wherein: The loss function is a cross entropy loss function.

17. The dialog response generation system of claim 16, wherein the loss function incorporates a mean square error between context vectors of the first multi-modal encoder-decoder and the second multi-modal encoder-decoder.

18. The dialog response generation system according to claim 15, wherein: The first parameters of the first multi-mode encoder-decoder are not updated.

19. The dialog response generation system according to claim 15, wherein: The optimizer module updates first parameters of the first multi-mode encoder-decoder based on a cross entropy loss function.

20. The dialog response generation system according to claim 15, wherein: The optimizer module updates second network parameters of the second multi-mode encoder-decoder using a back-propagation method.

Images