Training data generation device, training data generation method, and program

Abstract

This training data generation device generates a training data set to train a summarization model for generating a summary text for a video, and is provided with a training data generation unit that generates at least one further training data set from; a first text, which is a text extracted from an image in the video; a second text, which is a text extracted from an audio in the video; and an original training data set having a correct summary text of the video.

Description

[Technical field]

[0001] The present invention relates to a technique for generating training data used to train a summary model for generating a summary text of a video from the video. [Background technology]

[0002] In recent years, online meetings have become more common, and many videos of presentations at these meetings have been made available on the Internet.

[0003] Generally, a presentation video is long, and to understand the content, one must watch the video for a long time. Therefore, there is a demand to be able to understand the content of a presentation video in a short period of time. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension Summary of the Invention [Problem to be solved by the invention]

[0005] In order to quickly grasp the contents of a presentation video, it is conceivable to generate text that represents a summary of the presentation video (summary text) using a neural network model (called a summary model).

[0006] However, in the case of presentation videos, the amount of correct answer data (training data) used to train the summary model was small, and it was not possible to generate a summary model with sufficient accuracy using only the correct answer data that was collected. This issue is not limited to presentation videos, but can occur with any video for which summaries are to be generated.

[0007] The present invention has been made in consideration of the above points, and has an object to provide a technique that makes it possible to generate training data for training a summary model that generates summary text from video. [Means for solving the problem]

[0008] According to the disclosed technology, there is provided a training data generation device for generating a training data set for training a summary model that generates a summary text for a video, the device comprising: a training data generation unit for generating at least one further training data set from an original training data set having a first text, the first text being text extracted from an image in the video, a second text being text extracted from an audio in the video, and a ground-truth summary text of the video; There is provided a training data generating device comprising: Effect of the Invention

[0009] The disclosed technology provides a technology that makes it possible to generate training data for training a summary model that generates summary text from video. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram showing a basic process flow for creating a summary text from a presentation video. [Diagram 2] 1 is a configuration diagram of a summary generation device 100. FIG. [Diagram 3] 2 is a flowchart illustrating the operation of the summary generation device 100. [Figure 4] FIG. 2 is a configuration diagram of a summary model learning device 200. [Diagram 5] FIG. 1 illustrates a configuration for pre-training a summary model. [Figure 6] 10 is a flowchart for explaining the operation of the summary model learning device 200. [Figure 7] FIG. 13 is a diagram showing an example of input to a summary model and output from the summary model in pre-learning. [Figure 8] FIG. 11 is a diagram for explaining a process of extracting an image from a moving image. [Figure 9] FIG. 1 is a diagram for explaining extraction of text from an image. [Figure 10] FIG. 1 is a diagram for explaining extraction of text from speech. [Figure 11] FIG. 2 is a diagram showing an example of input to a summary model and output from the summary model in learning. [Figure 12] FIG. 4 is a diagram showing the configuration of a data extension unit 400. [Figure 13] 13 is a flowchart for explaining the operation of the data expansion unit 400. [Figure 14] FIG. 13 is a diagram illustrating an example of data division. [Figure 15] FIG. 13 is a diagram for explaining learning using a divided learning data set. [Figure 16] FIG. 2 illustrates an example of a hardware configuration of the apparatus. [Figure 17] FIG. 13 is a diagram showing the effect of learning paper data in advance. [Figure 18] FIG. 13 is a diagram showing the effect of pre-learning slide summaries. [Figure 19] FIG. 13 is a diagram showing the effect of training the original training data set together with a training data set obtained by division. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention (the present embodiment) will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment.

[0012] Both the summary generation device 100 and the summary model learning device 200 described below provide certain improvements over conventional techniques, such as generating summaries from papers, and represent an advancement in the field of technology related to generating summaries from videos.

[0013] The data augmentation unit 400 (training data generation device 400) described below provides certain improvements over conventional techniques such as manually generating summaries, and represents an advancement in the technical field related to techniques for learning summary models for generating summary text of videos.

[0014] In the following, a presentation video is used as a video for which a summary is to be generated, but this is just an example. The technology according to the present invention is not limited to presentation videos and can be applied to any video in general.

[0015] (Outline of the embodiment) In recent years, online conferences have become more common, and many videos of presentations at conferences and other events have been made public. Generally, presentation videos are long, so there is a demand for the content to be understood in a short time. In order to understand the content of a presentation video in a short time, it is desirable to be able to generate a summary of the presentation video.

[0016] In this embodiment, a technique for generating a summary text corresponding to a presentation video will be described.

[0017] <Example of presentation video> As an example, as disclosed in "https: / / slideslive.com / 38928967 / predicting-depression-in-screening-interviews-from-latent-categorization-of-interview-prompts" (searched February 27, 2022) and "https: / / videolectures.net / " (searched February 27, 2022), a typical presentation video consists of an image of the slides containing the content of the presentation, an image of the presenter, and the presenter's audio. Note that in many cases, the image of the presenter is not displayed.

[0018] <Basic process for creating summary text from a presentation video> The basic process flow for creating a summary text from a presentation video will be described with reference to Figure 1. In the following description, for convenience, the presentation video may be referred to as a "video" and the summary text may be referred to as a "summary."

[0019] First, from the video to be summarized, (A) presentation slides, (B) images extracted from the video, and (C) audio are prepared as input data to the summary generating unit 130.

[0020] It is assumed that the presentation slides in (A) are separate files from the video. Although a summary can be generated with at least one of the three input data (A), (B), and (C), in order to generate a more accurate summary, it is preferable to have three input data (A), (B), and (C), or two input data (A) and (C), or two input data (B) and (C).

[0021] Next, the input data converted into text by image recognition / voice recognition is input to summary generation unit 130, which outputs the summary text. Summary generation unit 130 is a functional unit included in summary generation device 100, which will be described later.

[0022] <About summary generation technology> In this embodiment, the summary generator 130 uses a neural network model (called a summary model) to generate a summary from text.

[0023] Any summarization model may be used as long as it inputs text and outputs summary text. In this embodiment, as an example, a model based on BART disclosed in Non-Patent Document 1 is used.

[0024] BART is a model that consists of an encoder and a decoder. By using a pre-trained model, when text is input to the encoder, a summary text is output from the decoder.

[0025] <About the assignment> Although there have been technologies that input text and output a summary, there are no technologies that output a summary from multimodal input data. In other words, there has been no technology in the past that can appropriately generate a summary text from a video that includes audio and images (slide images, etc.), such as a presentation video.

[0026] If the above-mentioned problem is divided into more specific problems from the viewpoint of the embodiment, it can be divided into the following problems 1 to 3.

[0027] Challenge 1: The cost of creating training data containing correct summary text to be used when training a summarization model to generate summaries for videos is high.

[0028] Issue 2: There is no summary generation technology that uses a summary model to extract audio and images from video and output summary text using these as input.

[0029] Issue 3: Even if it were possible to collect correct summary text from an external server, etc. to use when training a summary model to generate summaries for videos, the amount of such text would be small, resulting in a small amount of training data and making it impossible to generate an accurate summary model.

[0030] Below, we will explain the configuration and operation of a summary generation device 100 that generates a summary from a presentation video, and a summary model learning device 200 that generates (learns) a summary model used in the summary generation device 100. The technology described below solves the above problems 1 to 3.

[0031] (Configuration and Operation of Summary Generation Device 100) Fig. 2 shows a configuration diagram of summary generation device 100 in this embodiment. As shown in Fig. 2, summary generation device 100 has an image processing unit 110, an audio processing unit 120, a summary generation unit 130, and a summary model DB (database) 140. Learned summary models are stored in the summary model DB 140. Note that the DB in this specification may also be called a memory unit or a storage unit.

[0032] The flow of operations of the summary generating device 100 shown in FIG. 2 will be described with reference to the flowchart of FIG.

[0033] Audio information and image information are extracted from a video for which a summary is to be created, and in S101, the image information is input to image processing unit 110, and the audio information is input to audio processing unit 120. Note that in the example of Fig. 2, it is assumed that the functional unit for extracting audio information and image information (particularly image information) from a video is external to summary generation device 100, but the functional unit may be provided inside summary generation device 100.

[0034] In S102, the image processing unit 110 extracts text from the image using an image recognition technique. The image processing unit 110 may extract, in addition to the text, auxiliary information associated with the text (such as the color of the text in the slide).

[0035] In S103, the voice processing unit 120 extracts text from the voice using a voice recognition technique. Note that the order of the processes of S102 and S103 may be reversed, or S102 and S103 may be executed simultaneously.

[0036] The text extracted in S102 and the text extracted in S103 are input to the summary generation unit 130. In S104, the summary generation unit 130 generates a summary from the text extracted in S102 and the text extracted in S103 using the summary model read from the summary model DB 140. As will be described in the section on learning the summary model, in addition to the text, information in which any one, any multiple, or all of character arrangement features, image features, and audio features are added may be used as an input to the summary model. The actual substance of the "summary model" is data consisting of functions and weight parameters that constitute a neural network. In S104, the summary generation unit 130 outputs the generated summary.

[0037] As described above, by using both audio and image information obtained from a video, a high-quality summary can be generated.

[0038] The processing in the functional units for extracting audio information and image information from video, the image processing unit 110, and the audio processing unit 120, is the same as the processing in the learning data input unit 220, the image processing unit 230, and the audio processing unit 240 of the summary model learning device 200 described below, so these processing details will be explained in the explanation of the summary model learning device 220.

[0039] The summary generation device 100 of this embodiment solves the above-mentioned problem 2 and realizes a summary generation technology that uses a summary model that extracts audio and images from video and outputs summary text using these as input. Note that the summary model is trained by a summary model training device 200 described below.

[0040] (Configuration and operation of summary model learning device) Fig. 4 shows an example of the configuration of summary model learning device 200 in this embodiment. As shown in Fig. 4, summary model learning device 200 has a data acquisition unit 210, a learning data input unit 220, an image processing unit 230, an audio processing unit 240, a summary model learning unit 250, a data extension unit 400, a model setting unit 270, a summary model DB 280 that stores pre-trained summary models, and a summary model DB 290 that stores summary models currently being trained.

[0041] In this embodiment, when training the summary model, a summary model is created by learning a large amount of summaries of papers that are considered to have a high similarity in content to the presentation, and the summary model is fine-tuned using a small amount of summary data of the presentation. This makes it possible to achieve high accuracy even with a small amount of correct summary data for the presentation video.

[0042] Note that performing pre-learning as described above is one of the methods for solving Problem 3. Problem 3 can also be solved by using further learning data generated by the data extension unit 400 described later without performing pre-learning. Performing pre-learning and using further learning data generated by the data extension unit 400 described later may be combined.

[0043] 4 shows a configuration in the case where the above-mentioned pre-learning is performed, but learning may be performed using the learning data generated by the data extension unit 400 without performing pre-learning. Also, learning may be performed using the learning data generated by the data extension unit 400 for a summary model that has been pre-learned.

[0044] The configuration for pre-learning is shown in Fig. 5. As shown in Fig. 5, the configuration for pre-learning includes summary model pre-learning section 310 and summary model DB 320 that stores summary models undergoing pre-learning.

[0045] A summary model pre-training device (a device separate from summary model training device 200) may be configured having summary model pre-training section 310 and summary model DB 320, or summary model pre-training section 310 and summary model DB 320 may be included within summary model training device 200.

[0046] The flow of operations of the summary model learning device 200 and the summary model pre-learning unit 310 will be described with reference to the flowchart of Fig. 6. The detailed processing will be described later.

[0047] S201 and S202 are processes in the configuration for pre-learning shown in Fig. 5. In S201, pre-learning data is input to summary model pre-learning unit 310. The pre-learning data is, for example, the text of a paper related to the presentation and a summary of that paper (correct answer data).

[0048] In S202, summary model pre-learning unit 310 uses input data to learn (pre-learn) a summary model. The pre-learned summary model is stored in summary model DB 280 in summary model learning device 200.

[0049] 4. In the input process of S203, access information (e.g., URLs where papers and presentation videos are published) is input to the data acquisition unit 210. The data acquisition unit 210 uses the access information to acquire learning data, for example, from a server on the network, and inputs the learning data to the learning data input unit 220. The learning data is, for example, a presentation video related to a paper and a correct summary text corresponding to the video. In S203, the learning data input unit 220 further performs a process of separating the presentation video into image information and audio information, inputs the image information to the image processing unit 230, inputs the audio information to the audio processing unit 240, and inputs the correct summary to the summary model learning unit 250.

[0050] The image information input by the learning data input unit 220 to the image processing unit 230 may be a slide image or the like that is a separate file from the presentation video, or may be a slide image or the like extracted from the presentation video. In either case, the image may be expressed as an "image in the video" or an "image related to the video". In either case, text can be extracted from the "image in the video" or the "image related to the video" by image recognition processing.

[0051] In the following description, it is assumed that the image information input to the image processing unit 230 is a slide image or the like extracted from a presentation video.

[0052] In S204, the image processing unit 230 extracts text from the image using an image recognition technique. In addition to the text, the image processing unit 230 may extract accompanying auxiliary information (such as the color of the text in the slide), character arrangement features, image features, and the like.

[0053] In S205, the voice processing unit 120 extracts text from the voice using a voice recognition technique. The voice processing unit 120 may extract voice features in addition to the text. Note that the order of the processes of S204 and S205 may be reversed, or S204 and S205 may be executed simultaneously.

[0054] The text extracted in S204 and the text extracted in S205 are input to the summary model training unit 250. The correct summary is also input to the summary model training unit 250.

[0055] Here, the model setting section 270 reads out a pre-trained summary model from the summary model DB 280, and the pre-trained summary model is stored in the summary model DB 290. The following learning (fine tuning) is performed using the parameters in this pre-trained summary model as initial values.

[0056] In S206, the summary model training unit 250 uses the summary model read from the summary model DB 290 to generate a summary from the text extracted in S204 and the text extracted in S205, and trains the summary model (updates parameters) so as to minimize the error between the generated summary and the correct summary.

[0057] When the learning is completed, summary model learning section 250 stores the learned summary model in summary model DB 140 of summary generation device 100.

[0058] In the above example, the pre-learning is performed to fine-tune the pre-learned learning model, but as described above, the pre-learning is not essential. The process may be started from S203 in Fig. 6 without performing the pre-learning. When the pre-learning is not performed, the initial values ​​of the parameters of the summary model may be random values ​​or values ​​other than random values.

[0059] The process contents of each step from S201 to S207 will be described in more detail below.

[0060] (S201, S202: Pre-learning) A detailed example of pre-learning executed by the summary model pre-learning unit 310 shown in Fig. 5 will be described. In the pre-learning, a summary model is trained using text from a field related to the field of the presentation video to be summarized (called related field text) and its correct summary. The related field text is, for example, a paper text (text of the body of the paper), text of slides, etc.

[0061] An example of input to and output from the summarization model when a research paper text is used as the related field text is shown in Fig. 7. As described above, the summarization model according to this embodiment is a model consisting of an encoder and a decoder.

[0062] As shown in Figure 7, the text of the paper is input to the encoder, and summary text is output from the decoder. A summary model is trained to minimize the error between the output summary text and the correct summary text. When slide text is used as input, the processing is the same as when paper text is used.

[0063] When inputting text to the encoder, the token sequence of the text is first converted into a fixed-dimension vector, and then converted into summary text through the encoder-decoder.

[0064] An example of the input paper text is shown below.

[0065] 「We assume familiarity with basic notions of graph theory (see, for instance, 1]) and with elementary notions of polyhedral combinatorics (see, for instance, 6]).", "Our graphs will be undirected and simple (no loops and no multiple edges).", "As usual, K n denotes the complete graph with n vertices; K n;m denotes the complete bipartite graph with n + m vertices and n m edges.", "Let G be a graph; G is connected if for every pair of distinct vertices there exists a path in G joining them; G is twoconnected if for every vertex v of G, the graph G ?", "v is connected; G is planar if it can be embedded in the plane.", "A subgraph H of a G is spanning if the vertex sets of H and G are the same.", "Subdivision of an edge uv of G consists of removing edge uv, and adding a new vertex w and the two edges uw and vw; w is called subdivision vertex.", "If G and H are two graphs, we say that G contains a subdivision of H, if H arises by subdivision of the edges of some subgraph of G.As usual, (u) denotes the set of all edges that are incident in the vertex u.", "In automatic graph drawing the following problem arises: nd in a complete graph with weights on its edges a two-connected planar spanning subgraph with weight as Partially supported by DFG-Grant JU204 / 7-1 Forschungsschwerpunkt \" E ziente Algorithmen f ur diskrete Probleme und ihre Anw…". An example of the output (or summary text, which is the correct answer data) for the above input is shown below.

[0066] "The problem of finding a two-connected planar spanning subgraph of maximum weight in a complete edge-weighted graph is important in automatic graph drawing.", "We investigate the problem from a polyhedral point of view." On presentation video sites, slide files may be available as separate files from the video. Slide files often contain both the slide data (slide text) and a summary of the slide (summary text). In such cases, a summary model can be pre-trained by using the slide text as input to the encoder-decoder and the summary text as the correct answer.

[0067] An example of input slide text is shown below.

[0068] 「[["ssn"], ["MASTERS", "IN", "AUTOMOTIVE"], ["ENGINEERING"], ["Karthiek", "Nagaraj"], ["PRESENTED", "AT", "IRIS", ",", "DEPARTMENT", "OF", "MECHANICAL", "ENGINEERING"], ["SSN"], ["WHY", "AUTOMOBILE", "ENGINEERING", "?"], ["Its", "scope", "is", "irrefutable", "and", "job", "prospects", "are", "very", "strong", "in", "any", "part", "of", "the", "world", ".", "Also", "the", "prospect", "of", "returning", "to", "India", "to", "work", "is", "bright", "as", "the", "indian", "automotive", "industry", "is", "making", "tremendous", "progress", "."], [">", "It", "is", "a", "stream", "which", "blends", "passion", "for", "vehicles", "and", "technical", "knowledge", ",", "thus", "making", "it", "all", "the", "more", "interesting", "."], ["It", "is", "an", "interdisciplinary", "field", "which", "encompasses", "mechanical", "engineering", ",", "electrical", "and", "electronics", "engineering", "and", "software", "engineering", ".", "This", "again", "adds", "to", "the", "interest", "factor", "."], ["A", "multitude", "of", "research", "options", "are", "on", "offer", ",", "especially", "in", "hybrid", "powertrains", "and", "fuel", "cells", "."], ["PRESENTED", "AT", "IRIS", ",", "DEPARTMENT", "OF", "MECHANICAL", "ENGINEERING"], ["2"], ["SSN"], ["KEY", "AREAS", "OF", "AUTOMOTIVE", "ENGINEERING"], ["Vehicle", "Propulsion", "~", "Internal", "combustion", "engines"], ["Powertrain", "dynamics", "and", "control"], ["Vehicle", "dynamics", "~", "Handling", "response"], ["~", "Advanced", "transmission"], ["systems"], ["~", "Hybrid", "propulsion", "systems"], ["~", "Terrain", "modelling"], ["~", "Fuel", "cells"], ["~", "Drivetrain", "control", "systems"], ["~", "NVH", "modelling"], ["Automotive", "body", "structures", "~", "Material", "selection"], ["Automotive", "safety", "~", "Active", "and", "passive", "safety"], ["systems"], ["~", "Crash", "worthiness"], ["~", "Human", "factor", "engineering"], ["and",". An example of the output (or a slide summary, which is the correct answer data) for the above input is shown below.

[0069] "A guide to Masters in Automotive Engineering at International Destinations" (S203: Input process of summary model learning device 200) Next, a detailed example of the processing by the data acquisition unit 210 and the processing by the learning data input unit 220 in the summary model learning device 200 shown in FIG. 4 will be described.

[0070] The data acquisition unit 210 accesses, for example, a presentation video site on the Internet and acquires the presentation video and a summary of the correct answer corresponding to the video from the site. An example of a site from which such videos and summaries can be acquired is "https: / / aclanthology.org / " (searched on February 27, 2022).

[0071] As described above, by obtaining presentation videos and their summaries from a server on the network, learning data can be created without having to manually create summaries, thereby solving the aforementioned problem 1.

[0072] The learning data input unit 220 performs processing to separate the presentation moving image acquired by the data acquisition unit 210 into image information and audio information, and inputs the image information to the image processing unit 230 and the audio information to the audio processing unit 240 .

[0073] Although the image information is not limited to a specific image, it is assumed here that the image information is a slide image in a video presentation.

[0074] An example of a process of extracting an image from a presentation video by the learning data input unit 220 will be described with reference to FIG.

[0075] S203(1-1): The learning data input unit 220 extracts images from the presentation video every k seconds, where k is a predetermined real number greater than 0. The upper part of Fig. 8 shows six images extracted every k seconds.

[0076] S203(1-2): The learning data input unit 220 compares the images extracted in S203(1-1) in order for each time, and if the similarity between the t-th image and the t-1-th image is equal to or greater than a threshold, it determines that these images are the same image. Note that any method may be used to determine the similarity between images. FIG. 8 shows an example of the similarity between each pair of six images.

[0077] S203(1-3): The learning data input unit 220 repeats S203(1-1) and S203(1-2) to extract a set of different images. In Fig. 8, image 1, image 4, and image 6 are shown as a set of different images when the threshold value is 25. The obtained set of images is input to the image processing unit 230.

[0078] (S204: Image processing) Next, a detailed example of image processing executed by the image processing unit 230 will be described. The image processing unit 230 performs OCR (Optical Character Recognition) processing on the set of different images input from the learning data input unit 220, and acquires text, character color, character size, character position information, and the like from each image in the set of different images, as shown in Fig. 9. Note that the acquired information may be only text.

[0079] (S205: Audio processing) Next, a detailed example of the voice processing executed by the voice processing unit 240 will be described. As shown in Fig. 10, the voice processing unit 240 performs voice recognition processing on the voice input from the learning data input unit 220, and obtains text of the voice recognition result.

[0080] (S206: Learning process) Next, a detailed example of the learning process executed by the summary model learning unit 250 will be described. The summary model learning unit 250 combines the text obtained by the image processing unit 230 and the text obtained by the voice processing unit 240, and inputs the combined text to the summary model. The summary model learning unit 250 learns the summary model so that the error between the summary text output from the summary model and the correct summary text is minimized. For input to the summary model, information obtained by adding character arrangement features, image features, character size, color information, etc. obtained by the image processing unit 230 to the combined text may be used. Information obtained by adding voice features obtained by the voice processing unit 240 to the combined text may also be used.

[0081] The initial state of the above summary model is the summary model pre-trained in S202. However, as described above, it is possible not to perform pre-learning, so the initial state of the above summary model does not have to be the summary model pre-trained in S202. When pre-learning is not performed, learning may be performed using further learning data generated by the data extension unit 400 described later.

[0082] An example of input to and output from the summary model is shown in Figure 11. As described above, the summary model according to this embodiment is a model consisting of an encoder and a decoder.

[0083] As shown in Fig. 11, the text combined by [SEP], character size, and color information are input to the encoder, and summary text is output from the decoder. A summary model is trained so that the error between the output summary text and the correct summary text is minimized.

[0084] When inputting text to the encoder, the text token sequence is first converted into a fixed dimension vector of d dimensions, and then converted into a summary text through the encoder-decoder. Also, the input does not need to include character size and color information.

[0085] The text obtained by the voice processing unit 240 may be called ASR (Automatic Speech Recognition) text, and the text obtained by the image processing unit 240 may be called OCR text.

[0086] An example of ASR text is shown below.

[0087] “So to put in context to put my presentation in the context, I will, I would like to begin with the word decision support or decision-making. And first ask the question who, or what is making decisions and obviously we get two branches here. One is that we have a human decision maker who makes a decision and all of us are decision makers and then we are also talking about the decision systems. So robots.” An example of OCR text is shown below. The example below is an example of text obtained from a slide image disclosed in "http: / / videolectures.net / site / normal_dl / tag=1005123 / icml2015_schmidt_time_framework_01.pdf" (retrieved February 26, 2022).

[0088] “Structured sparsity sparsity is widely used in signal processing, machine learning, and statistics (compressive sensing, sparse linear regression, etc.) Examples of sparsity….” Below is an example of the summary text (or its correct answer) that is output when ASR text and OCR text are combined and input into the summarization model.

[0089] 「Decision Support is a discipline concerned with human decision making: it aims to provide methods and tools that support, rather than replace, people in making difficult decisions. One of the widely used decision-support approaches relies on decision models, which are developed in the decision process and used to evaluate and analyse decision alternatives. In this lecture, we shall present the method DEX (Decision EXpert), which was heavily influenced by ideas from Artificial Intelligence. DEX is a hierarchical, qualitative, rule-based, multi-criteria modelling method, suitable particularly for solving classification decision problems. DEX combines traditional approaches with those from expert systems and machine learning. DEX is supported by the software called DEXi and has been used in hundreds of real-world decision-making studies. The presentation will be illustrated by recent applications in the areas of electric energy production, food safety and health care.」 (Configuration and Operation of Data Expansion Unit 400) Below, we will explain one of the technologies for solving Problem 3, which is a technology for automatically generating an additional training dataset.

[0090] Fig. 12 shows the configuration of the data extension unit 400 in the summary model learning device 200 shown in Fig. 4. As shown in Fig. 12, the data extension unit 400 has a learning data generation unit 410, an important sentence extraction unit 420, and a task information attachment unit 430. Note that the data extension unit 400 may be a functional unit within the summary model learning device 200, or may be a separate device external to the summary model learning device 200. When the data extension unit 400 is within the summary model learning device 200, the summary model learning device 200 may be called the learning data generation device 400. When the data extension unit 400 is a separate device external to the summary model learning device 200, the separate device may be called the learning data generation device 400.

[0091] The operation flow of the data expansion unit 400 (training data generation device 400) shown in Fig. 12 will be described with reference to the flowchart in Fig. 13. In S301, ASR text obtained by speech processing, OCR text obtained by image processing, and the correct summary text corresponding to these are input to the training data generation unit 410.

[0092] In S302, the data division unit 410 performs a learning data generation process (which may be called a data division process) on the input data. In S302, an important sentence extraction process is also performed by the important sentence extraction unit 420. Note that the important sentence extraction unit 420 may be included in the learning data generation unit 410.

[0093] In S303, task information assigning unit 430 assigns task information to the generated learning data set, and in S304 outputs the learning data set with the task information assigned. The output data is input to summary model learning unit 250 and used for learning the summary model. The processing of each of the above steps will be described in more detail below.

[0094] (S301: Input, S302: Data division) Data consisting of "OCR text, ASR text, and correct summary text" for one presentation video is input as one set to the learning data generation unit 410. A data set for learning is called a learning data set.

[0095] Based on the above input data, the training data generation unit 410 generates the following five training data sets as shown in FIG. 14. Note that (1) is the original training data set. Since each training data set represents a task, the training data set may be called a task. Note that the following five are examples, and in addition to the original training data set, at least one further training data set may be generated. In addition to the following, (6) OCR text, OCR key sentence, (7) ASR text, and ASR key sentence may be generated.

[0096] (1) OCR text, ASR text, and correct answer summary text (2) OCR text, summary text of correct answers (3) ASR text and summary of the correct answer (4) OCR text, ASR key text (5) ASR text, OCR important text Both the ASR important sentences and the OCR important sentences are examples of pseudo-correct answer information. Both the ASR important sentences and the OCR important sentences are created by the important sentence extraction unit 420. An example of a method for creating these important sentences will be described below.

[0097] Regarding the ASR important sentences, the important sentence extraction unit 420 extracts the ASR important sentences by matching the summary text with the ASR text. For example, the important sentence extraction unit 420 extracts a part of the ASR text that has a high similarity to the summary text as the ASR important sentence.

[0098] Regarding the OCR important sentences, the important sentence extraction unit 420 extracts the OCR important sentences by matching the summary text with the OCR text. For example, the important sentence extraction unit 420 extracts, from the OCR text, a portion that has a high similarity to the summary text as the OCR important sentence.

[0099] Any method can be applied as the matching method for extracting ASR / OCR important sentences, but it is also possible to use the method used to create data for extractive summarization, such as the method described in Fine-tune BERT for Extractive Summarization (https: / / arxiv.org / pdf / 1903.10318v2.pdf, retrieved February 27, 2022).

[0100] (S303: Assign task information) The task information assigning unit 430 assigns identification information (which may be called a label) for identifying a task to each learning data set generated by the learning data generating unit 410. The identification information is a special token. In the above examples (1) to (5), for example, identification information such as [task0] is assigned as follows.

[0101] (1) [task0] OCR text, ASR text, correct summary text (2) [task1] OCR text, correct summary text (3) [task2] ASR text, summary text of correct answer (4) [task3] OCR text, ASR key text (5) [task 4] ASR text, OCR important text (S304: Output, (and learning)) Each task (each learning data set) to which identification information has been assigned in S303 is output to the summary model learning unit 250.

[0102] Summary model training unit 250 trains the summary model using each training data set with identification information. The training method for each training data set is the same as the training method in S206 described above. However, here, as shown in FIG. 15, text with the above identification information is used for input to the decoder. FIG. 15 shows an example of training for task (2) out of the above five tasks. Such training is performed for each of (1) to (5).

[0103] This makes it possible to increase the amount of training data and generate a more accurate summary model.

[0104] (Hardware configuration example) The summary generation device 100, the summary model learning device 200, and the training data generation device 400 can all be realized, for example, by causing a computer to execute a program. This computer may be a physical computer or a virtual machine on the cloud. Hereinafter, the summary generation device 100, the summary model learning device 200, and the training data generation device 400 will be collectively referred to as the "devices."

[0105] That is, the device can be realized by executing a program corresponding to the processing performed by the device, using hardware resources such as a CPU and memory built into a computer. The program can be recorded on a computer-readable recording medium (such as a portable memory) and stored or distributed. The program can also be provided via a network such as the Internet or email.

[0106] Fig. 16 is a diagram showing an example of the hardware configuration of the computer. The computer in Fig. 16 includes a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, and the like, all of which are interconnected via a bus BS.

[0107] The program for realizing the processing on the computer is provided by a recording medium 1001 such as a CD-ROM or a memory card. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001, but may be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files, data, etc.

[0108] The memory device 1003 reads out and stores the program from the auxiliary storage device 1002 when an instruction to start the program is received. The CPU 1004 realizes functions related to the light touch maintenance device 100 in accordance with the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network, etc. The display device 1006 displays a GUI (Graphical User Interface) or the like according to a program. The input device 1007 is composed of a keyboard and mouse, buttons, a touch panel, etc., and is used to input various operation instructions. The output device 1008 outputs the results of calculations.

[0109] (Effects of the embodiment) As described above, the technology according to the present embodiment makes it possible to appropriately generate summary text from videos that include audio and images, such as presentation videos. It also makes it possible to automatically generate additional training data for training a summary model that generates summary text from videos.

[0110] In particular, in this embodiment, the accuracy of the summary model can be improved by performing pre-learning or data expansion (generation of additional learning data by data division).

[0111] Below, we explain the effects based on the experimental results when pre-learning was performed and when data division was performed. In the following, ROUGE-1, ROUGE-2, and ROUGE-L are used as evaluation indices, and are abbreviated as R1, R2, and RL, respectively.

[0112] Figure 17 shows the effect of pre-learning paper data. For comparison, "ASR+OCR" shows the evaluation results when no paper data was pre-learned. "+Paper Abstracts (300,000)" and "+Paper Abstracts (500,000)" show the evaluation results when 300,000 and 500,000 paper summaries were pre-learned, respectively. As shown in Figure 17, it can be seen that accuracy is improved by pre-learning paper data.

[0113] FIG. 18 shows the effect of pre-learning slide summaries. For comparison, "ASR+OCR(4096)" shows the evaluation result when slide summaries are not pre-learned. "+slideshare" shows the evaluation result when slide summaries are pre-learned. As shown in FIG. 18, it can be seen that accuracy is improved by pre-learning slide summaries.

[0114] FIG. 19 is a diagram showing the effect of training the original learning dataset together with a further learning dataset obtained by splitting. For comparison, "ASR+OCR(4096)" shows the evaluation result when only the original learning dataset is trained. "ASR+OCR(4096)+extend" shows the evaluation result when the original learning dataset is trained together with a further learning dataset obtained by splitting. As shown in FIG. 19, it can be seen that accuracy is improved by training the original learning dataset together with the learning dataset obtained by splitting.

[0115] (Additional Note) The following supplementary items are further disclosed regarding the above-described embodiment. (Additional note 1) A training data generation device for generating a training data set for training a summary model for generating a summary text for a video, comprising: Memory, at least one processor coupled to the memory; Including, The processor, Generate at least one further training data set from the original training data set, the first text being text extracted from images in the video, the second text being text extracted from audio in the video, and a ground-truth summary text of the video. Training data generation device. (Additional note 2) The processor generates, as the further training data set, a training data set that includes the first text and does not include the second text, or a training data set that includes the second text and does not include the first text. Item 1. A training data generating device according to claim 1. (Additional note 3) The processor generates, as the further training data set, a training data set including one of the first text and the second text, and an important sentence obtained by matching one of the first text and the second text with the correct summary text. Item 1. A training data generating device according to claim 1. (Additional note 4) The processor assigns identification information to the further training data set to identify a task performed by the further training data set. Item 1. A training data generating device according to claim 1. (Additional note 5) 1. A computer-implemented training data generation method for use as a training data generation device for generating a training data set for training a summary model for generating summary text for a video, comprising: A training data generation step of generating at least one further training data set from an original training data set having a first text, which is a text extracted from an image in the video, a second text, which is a text extracted from an audio in the video, and a correct summary text of the video. A learning data generation method comprising: (Additional note 6) A non-transitory storage medium storing a program executable by a computer to perform a training data generation process for generating a training data set for training a summary model that generates a summary text for a video, the non-transitory storage medium comprising: The learning data generation process includes: Generate at least one further training data set from the original training data set, the first text being text extracted from images in the video, the second text being text extracted from audio in the video, and a ground-truth summary text of the video. Non-transitory storage medium.

[0116] Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications and variations are possible within the scope of the gist of the present invention described in the claims. [Explanation of symbols]

[0117] 100 Summary Generator 110 Image processing section 120 Audio Processing Unit 130 Summary generator 140 Summary Model DB 200 Summary model learning device 210 Data Acquisition Section 220 Learning Data Input Unit 230 Image Processing Unit 240 Audio Processing Unit 250 Summary Model Learning Unit 270 Model Setting Section 280 Summary Model DB 290 Summary Model DB 310 Summary Model Pre-Learning Section 320 Summary Model DB 400 Data Extension 410 Learning Data Generation Unit 420 Important sentence extraction part 430 Task information assignment unit 1000 Drive Device 1001 Recording media 1002 Auxiliary storage device 1003 Memory device 1004 CPU 1005 Interface device 1006 Display device 1007 Input Device 1008 Output device

Claims

1. A training data generation device for generating a training data set for training a summary model for generating a summary text for a video, comprising: a training data generation unit for generating at least one further training data set from an original training data set having a first text, the first text being text extracted from an image in the video, a second text being text extracted from an audio in the video, and a ground-truth summary text of the video; A learning data generating device comprising:

2. The training data generation unit generates, as the further training data set, a training data set that includes the first text and does not include the second text, or a training data set that includes the second text and does not include the first text. The training data generating device according to claim 1 .

3. The training data generation unit generates, as the further training data set, a training data set including one of the first text and the second text, and important sentences obtained by matching one of the first text and the second text with the correct summary text. The training data generating device according to claim 1 .

4. a task information assigning unit that assigns, to the further learning data set, identification information for identifying a task performed by the further learning data set; The training data generating device according to claim 1 , further comprising:

5. 1. A computer-implemented training data generation method for use as a training data generation device for generating a training data set for training a summary model for generating summary text for a video, comprising: A training data generation step of generating at least one further training data set from an original training data set having a first text, which is a text extracted from an image in the video, a second text, which is a text extracted from an audio in the video, and a correct summary text of the video. A learning data generation method comprising:

6. A program for causing a computer to function as each unit in the training data generating device according to any one of claims 1 to 4.

Images