Explanatory text generation device, method, and program

Abstract

To generate a sentence which explains the procedure of a work from a video obtained by imaging the work.SOLUTION: An acquisition unit 30 acquires, for a work including a plurality of steps, a material feature quantity representing each material used in the work, and a video feature quantity extracted from each clip, which is a video of each step in which the work is captured. An updating unit 40 identifies an action for a material included in the clip on the basis of the video feature quantity of each clip, and updates the material feature quantity of the identified material in accordance with the identified action. A generation unit 50 generates a sentence explaining a work procedure for each step on the basis of the updated material feature quantity, the specified action, and the video feature quantity.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an explanatory text generation device, an explanatory text generation method, and an explanatory text generation program. [Background technology]

[0002] Conventionally, technologies have been proposed that use models such as neural networks to understand information about tasks such as cooking recipes and assembling parts. For example, a model has been proposed that simulates changes in entities due to actions from task information described in sentences (text data) and understands the sentences (see Non-Patent Document 1). Also, a technology has been proposed that, when given a video with multiple chronologically ordered event segments, generates a coherent paragraph that explains the entire video using multiple sentences that explain the content of each segment (see Non-Patent Document 2). [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Antoine Bosselut, Omer Levy, Ari Holtzman, Corin Ennis, Dieter Fox, Yejin Choi, "SIMULATING ACTION DYNAMICS WITH NEURAL PROCESS NETWORKS", ICLR2018. [Non-patent document 2] Jie Lei, Liwei Wang, Yelong Shen, Dong Yu, Tamara L. Berg, Mohit Bansal, "MART: Memory-Augmented Recurrent Transformer for Coherent Video Paragraph Captioning", arXiv:2005.05402v1 [cs.CL] 11 May 2020. Summary of the Invention [Problem to be solved by the invention]

[0004] However, the technology described in Non-Patent Document 1 is premised on understanding sentences from text data, and therefore cannot be applied to a technology for generating sentences explaining work procedures from videos of the work. Also, the technology described in Non-Patent Document 2 generates sentences from videos, but does not target videos of the work, and therefore is difficult to apply to a technology for generating sentences explaining work procedures.

[0005] The present invention has been made in consideration of the above points, and aims to generate text explaining the steps of a task by using a video captured by a computer as input, through calculations within the computer. [Means for solving the problem]

[0006] In order to achieve the above object, the explanatory text generation device according to the first aspect of the present disclosure includes an acquisition unit that acquires, for a task including multiple steps, material features representing each of the materials used in the task and video features extracted from each of the videos of the task for each step that have been filmed; an update unit that identifies actions for the materials included in the videos for each step based on the video features of each of the videos for each step and updates the material features of the identified materials according to the identified actions; and a generation unit that generates a sentence explaining the procedure of the task for each step based on the updated material features, the identified actions, and the video features.

[0007] The updating unit may set the material feature amount to be updated as a material feature amount updated for a video of a previous process in chronological order of the processes in the work.

[0008] The update unit may perform at least one of adding, deleting, and merging material feature quantities on the updated material feature quantities.

[0009] The update unit may also update the ingredient features using a first model that has been pre-trained to identify actions from video features and update the ingredient features based on the identified actions, and the generation unit may generate the sentences using a second model that has been pre-trained to generate sentences that explain the work procedures for each process based on the ingredient features, actions, and video features.

[0010] In addition, the explanatory text generation device according to the first aspect may be configured to include a training unit that trains the first model and the second model using a material list, a video for each process, and correct sentences corresponding to the material list and the video for each process as training data.

[0011] The training unit may also train the first model and the second model so as to minimize a total loss including a first loss based on a comparison between the sentence generated by the generation unit and the correct sentence, and a second loss based on a comparison between the action and ingredient features identified by the update unit and the correct action and ingredient included in the video for each process.

[0012] The training unit may also acquire the correct action and material by performing a linguistic analysis on the correct sentence.

[0013] The training unit may also train the first model, the second model, and the third model so as to minimize the total loss, which further includes a third loss based on a comparison between the output of a third model pre-trained to estimate material features and actions from the sentences generated by the generation unit and the correct actions and materials.

[0014] In addition, an explanatory text generation method according to a second aspect of the present disclosure is a method in which an acquisition unit acquires, for a task including multiple steps, material features representing each of the materials used in the task and video features extracted from each of the videos of the task for each step, an update unit identifies actions for the materials included in the videos for each step based on the video features of each of the videos for each step, and updates the material features of the identified materials according to the identified actions, and a generation unit generates a sentence explaining the procedure of the task for each step based on the updated material features, the identified actions, and the video features.

[0015] In addition, an explanatory text generation program according to a third aspect of the present disclosure is a program for causing a computer to function as an acquisition unit that acquires, for a task including multiple steps, material features representing each of the materials used in the task and video features extracted from each of the videos of the task for each step, based on the video features of each of the videos for each step, and an update unit that identifies actions for the materials included in the videos for each step and updates the material features of the identified materials according to the identified actions, and a generation unit that generates text explaining the procedure of the task for each step based on the updated material features, the identified actions, and the video features. [Effects of the Invention]

[0016] According to the explanatory text generation device, method, and program of the present invention, it is possible to generate text that explains the steps of a task from a video of the task. [Brief explanation of the drawings]

[0017] [Figure 1] FIG. 1 is a diagram for explaining an overview of the present embodiment. [Figure 2] FIG. 2 is a block diagram showing a hardware configuration of the description generation device. [Figure 3] FIG. 2 is a block diagram illustrating an example of a functional configuration of the description generation device. [Figure 4]FIG. 2 is a diagram for explaining an outline of each functional configuration and the flow of data between the functional configurations. [Figure 5] FIG. 10 is a diagram for explaining acquisition of a material vector. [Figure 6] FIG. 10 is a diagram for explaining acquisition of a motion vector. [Figure 7] FIG. 10 is a diagram for explaining a state estimator of the update unit. [Figure 8] FIG. 10 is a diagram illustrating a description generator of the generation unit. [Figure 9] FIG. 10 is a diagram for explaining a re-estimator in the training section. [Figure 10] FIG. 10 is a diagram for explaining calculation of loss. [Figure 11] 10 is a flowchart showing the flow of a training process. [Figure 12] 10 is a flowchart showing the flow of a generation process. [Figure 13] FIG. 10 is a diagram illustrating the results of a performance comparison between the method of the present disclosure and a reference method. [Figure 14] FIG. 10 is a diagram showing an example of the result of generating an explanatory text from a recipe video. [Figure 15] FIG. 10 is a diagram showing an example of plotting material vectors in a vector space. [Figure 16] FIG. 10 is a diagram illustrating an example of addition and subtraction of material vectors. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of the present invention will be described below with reference to the drawings. Note that the same reference numerals are used to designate the same or equivalent components and parts in each drawing. Also, the dimensions and proportions of the drawings are exaggerated for the sake of explanation and may differ from the actual proportions.

[0019] First, an overview of the explanation generation device according to this embodiment will be described.

[0020] The explanatory text generation device according to this embodiment generates text explaining the steps of a task from a video of a task that includes multiple steps, a series of video segments divided into individual steps, and a materials list listing the materials used in the task. Hereinafter, the video segments for each step will be referred to as "clips," the series of clips as "clip strings," and the text explaining the steps of the task as "explanation." The explanatory text generation device according to this embodiment uses a network model that represents the process of the task as state changes to train a model that can obtain features representing the intermediate states of materials without labels representing those states.

[0021] A specific description will be given with reference to FIG. 1. In the example of FIG. 1, the work includes steps 1 to 3, and the ingredient list includes "butter," "eggs," and "cheese." The explanation generation device identifies actions for the ingredients from each clip. In the example of FIG. 1, the action "add" for the ingredient "butter" is identified from the clip of step 1. Furthermore, the action "cracked" and the action "stirred" for the ingredient "eggs" are identified from the clip of step 2. Furthermore, the action "add" and the action "stirred" for the ingredients "butter," "eggs," and "cheese" are identified from the clip of step 3.

[0022] The description generator updates the feature of the ingredient based on the identified action, thereby generating a feature representing the intermediate state of the ingredient. For example, the description generator updates the feature of the ingredient "butter" based on the action "add," thereby generating a feature representing an intermediate state such as "added butter." The description generator then generates a description for each process based on the feature of each clip, the action identified from each clip, and the updated feature of the ingredient.

[0023] The description generation device according to this embodiment will be described in detail below. In the following detailed description, when describing a specific example, an example will be used in which the work includes steps 1 to 3 and the ingredient list includes "butter," "eggs," and "cheese," as in the example of Figure 1. Note that the ingredient list may also include image data of the ingredients and IDs representing the ingredients.

[0024] Fig. 2 is a block diagram showing the hardware configuration of the explanatory text generation device 10 according to this embodiment. As shown in Fig. 2, the explanatory text generation device 10 includes a CPU (Central Processing Unit) 12, a memory 14, a storage device 16, an input device 18, an output device 20, a storage medium reader 22, and a communication I / F (Interface) 24. Each component is connected to each other via a bus 26 so as to be able to communicate with each other.

[0025] The storage device 16 stores an explanation generation program for executing the training process and generation process described below. The CPU 12 is a central processing unit that executes various programs and controls each component. That is, the CPU 12 reads the program from the storage device 16 and executes the program using the memory 14 as a work area. The CPU 12 controls each component and performs various arithmetic operations in accordance with the program stored in the storage device 16.

[0026] The memory 14 is made up of RAM (Random Access Memory) and serves as a working area to temporarily store programs and data. The storage device 16 is made up of ROM (Read Only Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), etc., and stores various programs including the operating system and various data.

[0027] The input device 18 is a device for performing various inputs, such as a keyboard or a mouse. The output device 20 is a device for outputting various types of information, such as a display or a printer. A touch panel display may be used as the output device 20 to function as the input device 18. The storage medium reader 22 reads data stored in various storage media, such as a CD (Compact Disc)-ROM, a DVD (Digital Versatile Disc)-ROM, a Blu-ray disc, or a USB (Universal Serial Bus) memory, and writes data to the storage media. The communication I / F 24 is an interface for communicating with other devices, and uses standards such as Ethernet (registered trademark), FDDI, or Wi-Fi (registered trademark).

[0028] Next, the functional configuration of the explanation generation device 10 according to this embodiment will be described.

[0029] Fig. 3 is a block diagram showing an example of the functional configuration of the explanatory sentence generation device 10. As shown in Fig. 3, the explanatory sentence generation device 10 includes, as its functional configuration, an acquisition unit 30, an update unit 40, a generation unit 50, and a training unit 60. Each functional configuration is realized by the CPU 12 reading out an explanatory sentence generation program stored in the storage device 16, expanding it in the memory 14, and executing it.

[0030] Here, we will explain an overview of each functional configuration and the data flow between the functional configurations, and then we will explain each functional configuration in detail with reference to Figure 4. In Figure 4, solid arrows represent data flows both during training and when generating explanations, dashed arrows represent data flows that occur only during training, and thick double-headed arrows represent parts related to loss calculation during training.

[0031] Acquisition unit 30 includes ingredient encoder 31 and video encoder 32. Acquisition unit 30 inputs an ingredient list, which is text data, to ingredient encoder 31 and acquires ingredient features representing each ingredient listed in the ingredient list. Acquisition unit 30 also inputs each clip included in the clip sequence to video encoder 32 and acquires video features extracted from each clip. FIG. 4 shows an example in which video feature 1 is acquired from the clip of process 1, video feature 2 is acquired from the clip of process 2, and video feature 3 is acquired from the clip of process 3. Acquisition unit 30 passes the acquired ingredient features and video features to update unit 40.

[0032] The update unit 40 includes state estimators 41A, 41B, and 41C, the number of which corresponds to the number of steps included in the work. In the example of FIG. 4, the state estimator 41A corresponds to step 1, the state estimator 41B corresponds to step 2, and the state estimator 41C corresponds to step 3. Hereinafter, when the state estimators 41A, 41B, and 41C are not distinguished from one another, they will be simply referred to as "state estimator 41." The state estimator 41 identifies actions for ingredients included in each clip based on the video features of the clip, and updates the material features of the identified ingredients according to the identified actions. The update unit 40 acquires the actions identified by each state estimator 41 and the updated state features, and passes them to the generation unit 50 together with the video features.

[0033] The material feature to be updated in the state estimator 41 is the material feature updated by the state estimator 41 corresponding to the previous clip in the clip sequence, i.e., the clip of the previous process in the chronological order of the processes in the work. That is, the material feature and video feature 1 passed from the acquisition unit 30 are input to the state estimator 41A. The material feature and video feature 2 updated by the state estimator 41A are input to the state estimator 41B. The material feature and video feature 3 updated by the state estimator 41B are input to the state estimator 41C.

[0034] The generation unit 50 includes explanatory sentence generators 51A, 51B, and 51C, the number of which corresponds to the number of steps included in the work. In the example of FIG. 4, explanatory sentence generator 51A corresponds to step 1, explanatory sentence generator 51B corresponds to step 2, and explanatory sentence generator 51C corresponds to step 3. Hereinafter, when the explanatory sentence generators 51A, 51B, and 51C are not to be distinguished from one another, they will be simply referred to as "explanatory sentence generator 51." The explanatory sentence generator 51 generates explanatory sentences for each step based on the updated material features, identified actions, and video features passed from the update unit 40. The generation unit 50 combines the explanatory sentences generated by each explanatory sentence generator 51 and outputs them as explanatory sentences describing the entire work.

[0035] The training unit 60 calculates a loss based on a comparison between the correct sentences represented by the ingredient list and clip sequence and the explanatory text output from the generation unit 50. The training unit 60 also acquires actions for the ingredients included in each clip, i.e., ingredient labels representing the correct ingredients and action labels representing the correct actions. The training unit 60 calculates a loss based on a comparison between the acquired ingredient labels and action labels and the actions and updated ingredient features acquired by the update unit 40.

[0036] Furthermore, the training unit 60 includes re-estimators 61A, 61B, and 61C, the number of which corresponds to the number of steps included in the work. In the example of FIG. 4, the re-estimator 61A corresponds to step 1, the re-estimator 61B corresponds to step 2, and the re-estimator 61C corresponds to step 3. Hereinafter, when the re-estimators 61A, 61B, and 61C are not distinguished from one another, they will be simply referred to as the "re-estimator 61." The re-estimator 61 estimates the actions for the materials in each step based on the explanations and material lists generated by the explanation generator 51 corresponding to the step. The training unit 60 calculates losses based on comparisons of the estimated actions for the materials with the material labels and action labels. The training unit 60 trains the state estimator 41, the explanation generator 51, and the re-estimator 61 to minimize the total loss, which is the integration of the calculated losses.

[0037] Each functional configuration will be described in more detail below. Note that the specific examples of each functional configuration described below are merely examples, and the method of realizing each functional configuration is not limited to the specific examples below.

[0038] The acquisition unit 30 acquires the ingredient list G=(g 1 ,···,g m ,···,g M ), and clip sequence V = (v 1 ,···,v n ,···,v N ) to get g m is the word representing the mth ingredient in the ingredient list G, and M is the total number of ingredients in the ingredient list. n is the n-th clip in the clip sequence, and N is the total number of steps included in the work. The acquisition unit 30 inputs the material list G to the material encoder 31 as shown in Figure 5, and inputs the clip sequence V to the video encoder 32 as shown in Figure 6.

[0039] The ingredient encoder 31 is an encoder configured with a neural network that is pre-trained to extract ingredient features representing the characteristics of each ingredient from the ingredient list. For example, the ingredient encoder 31 may be a concatenated neural network of multi-layer perceptrons (MLPs) equipped with word embedding and a ReLU activation function, such as GloVe (global vectors). As shown in Figure 5, the ingredient encoder 31 extracts a word g representing each ingredient included in the ingredient list G. m is vectorized by word embedding and positional encoding is added to obtain the initial material vector E 0 =(e1 0 ,···,e m 0 ,···,e M 0 In the example of this embodiment, E 0 =(e1 0 ,e2 0 ,e3 0 ) The material vector is an example of a material feature.

[0040] The video encoder 32 is an encoder configured with a neural network such as a transformer, which is trained in advance to extract video features representing the content characteristics of each clip from a clip sequence. n Each clip contains n is composed of consecutive frames. That is, the clip sequence V is hierarchical. Therefore, in order to effectively encode the clip sequence V, the video encoder 32 may be configured as a two-stage transformer suitable for encoding sequence data. In this case, the first-stage transformer extracts vectors corresponding to CLS tokens of BERT (Bidirectional Encoder Representations from Transformers) to encode each clip v n into a feature vector. The subsequent transformer is then trained on the entire sequence. As shown in Figure 6, the video encoder 32 encodes the video vector H = (h1,...,h n ,···,h N ) is output. In the example of this embodiment, H=(h1, h2, h3). Note that the motion vector is an example of a motion feature.

[0041] The update unit 40 updates the material vector E 0 and the motion vector H, the state change of the material in each process is repeatedly estimated. Specifically, as shown in FIG. 7, the update unit 40 sends the clip v to the state estimator 41, which executes the processes of (1) action selection, (2) material selection, and (3) update, in the n-th step. n Video Vector H n and the material vector E n-1 The details of each process of the state estimator 41 will be described later. After the n-th step, the state estimator 41 calculates the state estimation vector u n The state estimation vector u n is the video vector h n , the selected action ￣f n , and the material vector ￣e nThe notation "￣X" is expressed as "￣ (bar)" above "X" in the figures and formulas. The state estimator 41 calculates the final clip v N The above process is repeated until the process of (1) is completed.

[0042] The following describes in detail the processes of (1) operation selection, (2) material selection, and (3) update of the state estimator 41 in the n-th step.

[0043] First, (1) Action Selection will be explained. The state estimator 41 selects a motion vector h n Given a predefined behavior embedding F, we can generate a clip v n The action embedding F is a vector of predefined action words. For example, the actions "crack" and "stir" are used to select the action to be performed by the clip v. n If it is running on f crack and f stir Therefore, in order to select multiple actions, the state estimator 41 calculates the motion vector h n Then, for example, by MLP, we extract the clip v n The probability of an action being performed in w p Next, the state estimator 41 calculates the motion vector f indicating the selected motion, as shown in the following equations (2) and (3). n The probability of action w p and the motion embedding F. In equations (1) to (3), MLP(·) represents a two-layer MLP with a sigmoid function, and w p is the attention distribution of possible actions among the actions contained in the action embedding F.

[0044]

number

[0045] Next, (2) Material Selection will be explained. The state estimator 41 calculates the probability of operation w p and motion vector h n Based on the material vector E n-1 From Clip V n By selecting the materials included in n For example, in the example of Figure 1, in step 3 (n=3), the raw material "cheese" and the manipulated ingredients "butter" and "eggs" need to be selected. In order to be able to select a combination of raw materials and manipulated ingredients in this way, the state estimator 41 has two attention modules: clip attention and recurrent attention.

[0046] Clip attention is a video vector h n and the probability of action w p Based on this, the material vector e m n-1 Attention weight d for m In equations (4) and (5), W1 and W2 are linear and bilinear mappings, and b1 and b2 are biases.

[0047]

number

[0048] Recurrent attention uses information from both the current and previous clips, and the output of Clip attention, to generate a material vector E n-1 Of the clips v n The material probability associated with the operation being performed in a n In equations (6) and (7), W3 is a linear mapping, b3 is a bias, and c∈R 3 is the selection distribution, a m n-1 is the previous clip v n-1 Material vector e inm n Attention weight of a m n is the final distribution of each material vector, and 0 is the zero vector that represents no material selection.

[0049]

number

[0050] Next, the state estimator 41 calculates the material vector e indicating the selected material as shown in the following equation (8). n The final distribution of the material vector a m n and material vector e m n-1 It is calculated as a weighted sum of

[0051]

number

[0052] Next, (3) Update will be explained. The state estimator 41 generates an updated material vector ̂e that represents the change in the state of the material based on the selected operation and material vector. m In the drawings and equations, the notation "^X" is expressed by a "^ (hat)" above "X". Specifically, the state estimator 41 calculates the selected operation ^f as shown in the following equation (9). n and the material vector ￣e n Using the bilinear transformation of n In equation (9), W4 is a bilinear map and b4 is a bias.

[0053]

number

[0054] Next, the state estimator 41 calculates the probability a n Based on this, the motion proposal vector ln and the current material vector e m n-1 and the updated material vector ^e m The state estimator 41 calculates the updated material vector ^e m Material Vector E m n and forwards it to the next (n+1)th process.

[0055]

number

[0056] As shown in FIG. 8, the generation unit 50 generates the state estimation vector u output from the state estimator 41. n is input to the corresponding n-th explanation generator 51. The explanation generator 51 calculates the state estimation vector u n From step-by-step instructions n Repeatedly generate the explanation Y=(y 1 ,···,y n ,···,y N The exposition generator 51 may be a decoder configured with a pre-trained neural network, such as a Transformer.

[0057] Specifically, the description generator 51 has a copy mechanism to refer to the ingredient list G and to prompt the generation of words representing ingredients to be included in the description. When generating the k-th word of the description of the n-th process, the description generator 51 uses the updated ingredient vector e m n ∈E n is given, the output o of the explanation generator 51 is obtained by the copy mechanism as shown in the following equation (11). n,k and material vector e m n Using the bilinear dot product with n,k m In equation (11), W c represents a bilinear map.

[0058]

number

[0059] Next, the explanation generator 51 selects whether to select an ingredient from the ingredient list or generate a word from a vocabulary prepared in advance, as shown in the following formula (12). n,k (0≦g n,k ≦1). In equation (12), [·] is the link function, σ(·) is the sigmoid function, and W g is a linear map, b g represents the bias.

[0060]

number

[0061] Next, the explanation generator 51 generates a copy gate g n,k Based on this, the final predicted word probability P n,k (w) is calculated as the weighted sum of the probability of copying from the material list and the probability of generating from the vocabulary. n,k VOC (w) is the probability of the kth word w in the nth sentence of the vocabulary, |g m | represents the number of words in the mth ingredient in the ingredient list.

[0062]

number

[0063] The description generator 51 calculates the predicted word probabilities P n,k By generating words sequentially based on (w), the explanation text y for each process is n and generate explanatory text Y for all steps.

[0064] As shown in Fig. 9, the re-estimator 61 included in the training unit 60 includes two sub-modules: a sentence encoder 62 and an estimator 63. The sentence encoder 62 converts the explanatory sentence Y generated by the generation unit 50 into a sentence vector S = (s1,...,s n ,···,s N ) to the sentence generator 51. Specifically, the sentence encoder 62 first samples the description sentence while maintaining differentiable chains, for example, by applying a straight version of Gumbel softmax resampling. The sentence encoder 62 converts the sampled description sentence into a feature vector by calculating the average vector of the embedded words at each step. The word embeddings are shared between the description generator 51 and the sentence encoder 62. Next, the sentence encoder 62 converts the feature vector into a sentence vector S corresponding to each step, for example, using BiLSTM.

[0065] The estimator 63 has the same configuration as the state estimator 41, and is configured to calculate the sentence vector S and the initial material vector E 0 The state change of the material is estimated again based on the

[0066] In addition, the training unit 60 trains the state estimator 41, the explanation generator 51, and the re-estimator 61 by using a sentence generation loss L sent , state estimation loss L v_sim , and the re-estimated loss L t_sim Calculate.

[0067] Sentence generation loss L sent is the loss for the explanation generator 51. Specifically, the training unit 60 calculates the correct sentence Y′=(y′ 1 ,···,y' n ,···,y' N ) is acquired. The combination of the material list G, the clip sequence V, and the correct sentence Y' is training data, and the training unit 60 acquires a plurality of training data. Then, for all the training data, the training unit 60 calculates the error between the explanatory sentence Y, which is the output for the input (V, G), and the correct sentence Y', for example, |yn -y' n The sum of N negative log-likelihoods of | is the loss L sent Calculate as follows.

[0068] State estimation loss L v_sim is a loss related to the state estimator 41 and is composed of an ingredient selection loss and an action selection loss. The training unit 60 performs linguistic analysis on the correct sentence, acquires words representing ingredients contained in the correct sentence as ingredient labels, and acquires words whose part of speech is a verb contained in the correct sentence as action labels. For example, from the correct sentence "crack the eggs and stir," "eggs" is acquired as the ingredient label, and "crack" and "stir" are acquired as the action labels. The training unit 60 converts the ingredient labels into a vector containing the same number of elements as the ingredient labels acquired from the correct sentences for all steps, and assigns a value of 1 to the elements corresponding to the ingredient labels acquired from the correct sentences for each step and a value of 0 to the other elements. Similarly, the training unit 60 converts the action labels into a vector containing the same number of elements as the action labels acquired from the correct sentences for all steps, and assigns a value of 1 to the elements corresponding to the action labels acquired from the correct sentences for each step and a value of 0 to the other elements. By obtaining ingredient labels and action labels from the correct sentences for each process, the burden of manually preparing ingredient labels and action labels can be reduced.

[0069] The training unit 60 calculates the probability a of the material calculated by the state estimator 41. n and the material label, for example, with probability a n The training unit 60 calculates the sum of negative log-likelihoods of the difference between the values ​​of the elements of the corresponding material labels and the probability w of the action calculated by the state estimator 41 as the loss of material selection. p and the error between the action label, e.g., probability w pThe sum of the negative log likelihoods of the difference between the element values ​​of the corresponding action labels is calculated as the loss of action selection. Note that, since there is a large imbalance in the ratio of positive actions (actions with a value of 1) to negative actions (actions with a value of 0) for the action labels, an asymmetric loss, which is a weighted negative log likelihood, may be used to eliminate the imbalance. The training unit 60 calculates the sum of the loss of material selection and the loss of action selection as the state estimation loss L v_sim Calculate as follows.

[0070] Reestimation loss L t_sim is the loss for the re-estimator 61. The training unit 60 calculates the state estimation loss L v_sim Similarly, the re-estimator 61 estimates the state change of the material by using the selected action and material, and the re-estimation loss L t_sim Calculate.

[0071] The training unit 60 calculates the sentence generation loss L sent , state estimation loss L v_sim , and the re-estimated loss L t_sim The sum of the total loss L total (=L sent +L v_sim +L t_sim ) and the training unit 60 calculates the total loss L total The state estimator 41, the explanation generator 51, and the re-estimator 61 are trained by repeatedly updating the parameters of the state estimator 41, the explanation generator 51, and the re-estimator 61 so as to minimize the total loss L. The training termination condition may be, for example, when the number of iterations of parameter update reaches a predetermined number, or when the total loss L total When the value is less than the specified value, the total loss L total and the total loss calculated this time L total The detection may be made when the difference between the two is equal to or less than a predetermined value.

[0072] Next, the operation of the explanation generation device 10 according to this embodiment will be described.

[0073] 11 is a flowchart showing the flow of training processing executed by the CPU 12 of the explanatory sentence generation device 10. The CPU 12 reads out the explanatory sentence generation program from the storage device 16, expands it in the memory 14, and executes it, whereby the CPU 12 functions as each functional component of the explanatory sentence generation device 10, and the training processing shown in FIG.

[0074] In step S10, the acquisition unit 30 acquires the ingredient list G and clip sequence V input to the explanatory sentence generation device 10, and the training unit 60 acquires the correct answer sentence Y' for the ingredient list G and clip sequence V input to the explanatory sentence generation device 10. Next, in step S12, the acquisition unit 30 inputs the ingredient list G to the ingredient encoder 31, and the initial ingredient vector E 0 , and inputs the clip sequence V to the video encoder 32 to obtain a video vector H.

[0075] Next, in step S14, the update unit 40 updates the initial material vector E 0 and the motion vector H are input to the state estimator 41. The state estimator 41 calculates the motion vectors of the clip v from the predefined motion embedding F. n Select the action to be performed by the action vector ￣f n The state estimator 41 also calculates the material vector E n-1 From Clip V n Select the materials included in the list and use the material vector ￣e n Then, the state estimator 41 calculates the state estimation vector u n =(h n ,￣f n ,￣e n ) is output.

[0076] Next, in step S16, the state estimator 41 generates an updated material vector ̂e representing the state change of the material based on the selected operation and material vector. m Then, the state estimator 41 calculates the updated material vector ^e m Material Vector E m n and forwards it to the next (n+1)th process.

[0077] Next, in step S18, the generator 50 generates the state estimation vector u output from the state estimator 41. n is input to the corresponding n-th explanation generator 51. The explanation generator 51 calculates the state estimation vector u n From step-by-step instructions n is generated repeatedly to generate the explanation Y for all the processes.

[0078] Next, in step S20, the training unit 60 uses the re-estimator 61 to generate the explanatory text Y and the initial material vector E 0 From the motion vector ￣f n and material vector E n-1 Then, the training unit 60 re-estimates the sentence generation loss L sent , state estimation loss L v_sim , and the re-estimated loss L t_sim Calculate the total loss L total (=L sent +L v_sim +L t_sim ) is calculated. Then, the training unit 60 calculates the total loss L total The parameters of the state estimator 41, the explanation generator 51, and the re-estimator 61 are updated so as to minimize

[0079] Next, in step S22, the training unit 60 determines whether or not the termination condition of the training is satisfied. If the termination condition is not satisfied, the process returns to step S14, and if the termination condition is satisfied, the process proceeds to step S24. In step S24, the training unit 60 outputs the parameters of the state estimator 41, the explanation generator 51, and the re-estimator 61 when the termination condition is satisfied, and the training process ends.

[0080] Fig. 12 is a flowchart showing the flow of the generation process executed by the CPU 12 of the explanatory sentence generation device 10. The CPU 12 reads out the explanatory sentence generation program from the storage device 16, expands it in the memory 14, and executes it, whereby the CPU 12 functions as each functional component of the explanatory sentence generation device 10, and the generation process shown in Fig. 12 is executed. Note that before the start of the generation process, the parameters output in the training process are set in the state estimator 41, the explanatory sentence generator 51, and the re-estimator 61, respectively.

[0081] In step S30, the acquisition unit 30 acquires the ingredient list G and clip sequence V for which an explanation is to be generated, which have been input to the explanation generation device 10. Thereafter, steps S12 to S18 are executed in the same manner as in the training process, and an explanation Y corresponding to the ingredient list G and clip sequence V acquired in step S30 is generated and output, and the generation process ends.

[0082] As described above, the explanatory text generation device according to this embodiment acquires, for a task including multiple steps, material features representing each material used in the task and video features extracted from each video of the task, each of which is filmed for that step. The explanatory text generation device also identifies actions for the material included in the video of each step based on the video features of each step, and updates the material features of the identified material according to the identified actions. In this case, the material features to be updated are those updated for the video of the previous step in the chronological order of the tasks. The explanatory text generation device then generates text describing the work content for each step based on the updated material features, the identified actions, and the video features. To accurately generate text describing the steps of a task from a video of the task, it is essential to track changes in the state of the materials in the chronological order of the video of the task. By updating the material features as described above, the explanatory text generation device according to this embodiment can generate text describing the steps of the task from a video of the task.

[0083] In the above embodiment, the re-estimator re-estimates the state change of the material from the generated explanatory text and uses the results to train the model. However, this configuration is not essential, and the configuration may not include the re-estimator. In this case, the training unit calculates the total loss by L total =L sent +L v_sim This can be done as follows.

[0084] Here, with reference to FIG. 13, the results of a performance comparison between the method of the present disclosure and the reference method will be described. In FIG. 13, the four "Baseline" methods and "Ours"'s "Video only (V)" and "V+Ingredients (VI)" are reference methods, and "Ours"'s "VI+Visual simulator (VIV)" and "VIV+Textual re-simulator (VIVT)" are methods of the present disclosure. VIVT of the method of the present disclosure is the method of the above embodiment, and VIV is a case where the re-estimation results are not used. V of the reference method is a method that uses only the clip sequence of the present embodiment, and VI is a method that uses an ingredient list in addition to V. In other words, V and VI do not estimate changes in ingredient state, i.e., do not update ingredient features. Also, in FIG. 13, B1, B4, M, C, and RL are word overlap evaluation scores, where B = BLEU, M = METEOR, C = CIDEr-D, and RL = ROUGE-L. Furthermore, the "Baseline" method does not originally use an ingredient list, but in consideration of fairness when comparing with the method of the present disclosure, it includes a method that was modified from a conventional method to use an ingredient list. In Figure 13, the method with a check mark in the "I" box indicates that it uses an ingredient list. Also, in Figure 13, the bold numbers indicate the highest score in the corresponding word overlap evaluation.

[0085] As shown in Figure 13, the VIV and VIVT methods of the present disclosure significantly outperform the reference methods in all evaluations. Furthermore, a comparison between VI and VIV shows that VIV outperforms VI, and that VIVT further improves the performance of VIV. This indicates that both estimation and re-estimation of state changes are effective in generating sentences that accurately describe procedures.

[0086] 14 shows an example of the results of generating explanatory text from a recipe video using the method of the present disclosure and the reference method. In FIG. 14, single-underlined words indicate ingredients that match ingredients included in the ground truth. Words underlined with a wavy line indicate ingredients that do not match ingredients included in the ground truth, or ingredients that are included in the ground truth but not in the generated explanatory text.

[0087] As shown in FIG. 14, the reference method "MART-I" fails to generate accurate descriptions, for example, overlooking "eggs" and "milk" in steps 1 and 2. A similar trend is observed in VI, where these methods over-generate words indicating ingredients listed in the ingredients list. VIV, the method of the present disclosure, overcomes these issues, for example, "batter" in step 3. Furthermore, VIVT, a re-estimation method, is able to generate words indicating ingredients overlooked by VIV, for example, "baking soda" and "pepper" in step 1 and "water" in step 2.

[0088] Furthermore, the explanatory text generation device according to this embodiment acquires updated material vectors based on actions performed on materials during the process of generating explanatory text. This allows the device to not only generate explanatory texts but also simulate changes in the state of materials after training. Furthermore, annotation of intermediate states of materials can be performed automatically without the need for correct answer data, which would be extremely laborious if done manually.

[0089] Furthermore, to explain the effect of obtaining updated material vectors, Fig. 15 shows an example of plotting material vectors in vector space. Note that in Fig. 15, for ease of explanation, multidimensional material vectors are projected into two-dimensional space. Material vectors represented by light plots (material vectors located roughly on the right side of the vector space) are original material vectors, and material vectors represented by dark plots (material vectors located roughly on the left side) are updated material vectors. As shown in Fig. 15, it can be seen that the original material vectors and the updated material vectors are clearly divided into two major clusters in the vector space.

[0090] We also investigated the trajectory of ingredients using the top two closest ingredient vectors obtained from the updated ingredient vectors (enlarged portion of Figure 15). Regardless of the differences in the predefined recipe categories for the ingredients included in each clip, ingredient vectors with similar states belong to the same cluster in the vector space. For example, ingredient vectors for "eggs" updated with the action "beat" are found near ingredient vectors for "eggs" updated with actions similar to "beat," such as "mix" and "stir." The same trend is observed for "flour."

[0091] The above-mentioned ingredient vectors are expressed as vectors that allow addition and subtraction, as shown in FIG. 16. For example, as shown in FIG. 16(a), the state transition of ingredients is calculated as v(cut potatoes) = v(potatoes) + v(cut tomatoes) - v(tomatoes). Here, v represents a map to a vector space. Note that FIG. 16 shows the case where the ingredient vector of the first term is the original ingredient vector (the ingredient vector that has not been updated), but the ingredient vector of this term may also be an updated ingredient vector. For example, a calculation such as v(added chopped shallot) = v(chopped shallot) + v(add egg) - v(egg) is also possible.

[0092] The technology of the present disclosure is effective for improving the searchability of video searches using text, for example.

[0093] In the above embodiment, the material vector E m Material vector ^e assigned to m Although the case where the number of material vectors is the same has been described, this is not limiting. m The material vector ^e assigned to m At least one of adding, deleting, and merging material vectors may be performed on the vectors. Such processing may be realized by applying a memory network, for example.

[0094] Furthermore, in the above embodiment, in the explanation of the specific example, the video is a recipe video (a video explaining the steps of cooking) and the ingredients are the ingredients used in the dish, etc., but the scope of application of the technology of the present disclosure is not limited to the above example. For example, it can also be applied to videos of factory work or biochemical experiments. In the former case, for example, parts used in the work correspond to the materials in the above embodiment, and assembling the parts corresponds to the actions in the above embodiment. In the latter case, for example, medicines or specimens correspond to the ingredients, and adding, stirring, etc. correspond to the actions.

[0095] In the above embodiment, a description has been given of a case where an explanation generation device having both a training function and a generation function is realized by a single computer, but this is not limiting. The training device and the generation device may be realized by different computers. In this case, a password output from the training device may be set in the state estimator, explanation generator, and re-estimator of the generation device.

[0096] In addition, the explanatory text generation process executed by the CPU after reading the software (program) in the above embodiment may be executed by various processors other than the CPU. Examples of such processors include programmable logic devices (PLDs) such as field-programmable gate arrays (FPGAs), whose circuit configuration can be changed after fabrication, and application-specific integrated circuits (ASICs), which are dedicated electrical circuits that are processors with circuit configurations specifically designed to execute specific processes. The explanatory text generation process may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (e.g., multiple FPGAs, or a combination of a CPU and an FPGA). The hardware structure of these various processors is, more specifically, an electrical circuit that combines circuit elements such as semiconductor devices.

[0097] In the above embodiment, the description is given of a case in which the description generation program is pre-stored (installed) in a storage device, but this is not limiting. The program may be provided in a form stored in a storage medium such as a CD-ROM, a DVD-ROM, a Blu-ray disc, or a USB memory. The program may also be downloaded from an external device via a network. [Explanation of symbols]

[0098] 10. Explanatory text generator 12 CPU 14 Memory 16 Storage device 18 Input Devices 20 Output Devices 22 Storage medium reader 24 Communication I / F 26 Bus 30 Acquisition Department 31 Material Encoder 32 Video Encoder 40 Update section 41, 41A, 41B, 41C State Estimator 50 Generation part 51, 51A, 51B, 51C Description Generator 60 Training Department 61, 61A, 61B, 61C Re-estimator 62 Sentence Encoder 63 Estimator

Claims

1. an acquisition unit that acquires, for a job including a plurality of steps, material features obtained by vectorizing words representing each of materials used in the job, and video features that are extracted from each of the videos of the job for each of the steps and represent content features of each of the videos for each of the steps; an updating unit that identifies an action for an ingredient included in the video for each process based on the video feature of each of the process steps acquired by the acquiring unit, using a first model that is pre-trained to identify an action from the video feature and update the ingredient feature based on the identified action, and updates the ingredient feature of the identified ingredient according to the identified action; a generation unit that generates a sentence describing the work procedure for each of the processes based on the material features updated by the update unit, the identified actions, and the video features acquired by the acquisition unit, using a second model that has been trained in advance to generate a sentence describing the work procedure for each of the processes based on the material features, the actions, and video features; An explanatory text generation device including:

2. The description generation device according to claim 1 , wherein the updating unit sets the material feature to be updated as the material feature updated for a video of a previous process in chronological order of the processes in the work.

3. The explanation generation device according to claim 1 or 2, wherein the update unit performs at least one of adding, deleting, and merging ingredient features with respect to the updated ingredient features.

4. The explanatory text generation device according to any one of claims 1 to 3, further comprising a training unit that trains the first model and the second model using a material list, a video for each process, and a correct sentence corresponding to the material list and the video for each process as training data.

5. The explanatory text generation device according to claim 4, wherein the training unit trains the first model and the second model so as to minimize a total loss including a first loss based on a comparison between the sentence generated by the generation unit and the correct sentence, and a second loss based on a comparison between the action and material features identified by the update unit and the correct action and material included in the video for each process.

6. The explanation generation device according to claim 5 , wherein the training unit acquires the correct action and material by performing linguistic analysis on the correct sentence.

7. The explanatory text generation device according to claim 5 and claim 6, wherein the training unit trains the first model, the second model, and the third model so as to minimize the total loss, which further includes a third loss based on a comparison between the output of a third model pre-trained to estimate material features and actions from the sentence generated by the generation unit and the correct actions and materials.

8. 1. A computer-implemented method for generating explanatory text, comprising: an acquisition unit implemented in the computer acquires, for a task including a plurality of steps, material features obtained by vectorizing words representing each of materials used in the task, and video features extracted from each of the videos of the task for each of the steps, representing content features of each of the videos; an updating unit implemented in the computer identifies an action from video features and uses a first model that has been trained in advance to update material features based on the identified action, to identify an action for a material included in the video for each process based on the video features of each of the videos for each process acquired by the acquiring unit, and updates the material features of the identified material in accordance with the identified action; A generation unit implemented in the computer generates a sentence explaining the work procedure for each of the processes based on the material features updated by the update unit, the identified actions, and the video features acquired by the acquisition unit, using a second model that has been trained in advance to generate a sentence explaining the work procedure for each of the processes based on the material features, the actions, and the video features. How to generate descriptions.

9. Computer, an acquisition unit that acquires, for a job including a plurality of steps, material features obtained by vectorizing words representing each of materials used in the job, and video features that are extracted from each of videos of the job for each of the steps and represent content features of each of the videos for each of the steps; an updating unit that identifies an action for an ingredient included in the video for each process based on the video feature of each of the process steps acquired by the acquiring unit, using a first model that is pre-trained to identify an action from the video feature and update the ingredient feature based on the identified action, and updates the ingredient feature of the identified ingredient according to the identified action; and a generation unit that generates a sentence explaining the work procedure for each of the processes based on the material features updated by the update unit, the identified actions, and the video features acquired by the acquisition unit, using a second model that has been trained in advance to generate a sentence explaining the work procedure for each of the processes based on the material features, the actions, and the video features; A description generation program to function as a

Images