Generation system, learning system, generation method, learning method, and program

Abstract

To provide a machine-reading system in consideration of visual information.SOLUTION: A learning device according to an embodiment is provided with: a generation unit for, with data including a visual region and first information related to the data as inputs, generating second information corresponding to the first information from information representing a feature of the region using model parameters of a machine learning model; and a learning unit for learning the model parameters based on the second information and third information representing a correct answer for the second information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a learning device, a generation device, an output device, a learning method, a generation method, a program, and a method for manufacturing a trained machine learning model. [Background technology]

[0002] If "machine reading comprehension," which generates answers to questions based on a set of given documents, could be accurately performed using artificial intelligence, it could be applied to a wide range of services such as question answering and intelligent agent dialogue. There are extractive and generative types of machine reading comprehension, and one prior art technique for generative machine reading comprehension is known, for example, from the technique disclosed in Non-Patent Document 1. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Kyosuke Nishida, Itsumi Saito, Kosuke Nishida, Kazutoshi Shinoda, Atsushi Otsuka, Hisako Asano, Junji Tomita: Multi-style Generative Reading Comprehension. ACL (1) 2019: 2273-2284 Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional machine reading comprehension technologies only handle text and are unable to handle visual information such as the position and size of text within a document. As a result, when understanding documents with multiple text layouts (such as HTML (HyperText Markup Language) documents and PDF (Portable Document Format) documents) using machine reading comprehension, all information other than the text content is lost.

[0005] An embodiment of the present invention has been made in view of the above points, and aims to realize machine reading comprehension that takes visual information into consideration. [Means for solving the problem]

[0006] In order to achieve the above object, a learning device according to one embodiment is characterized by having a generation unit that receives data including a visual area and first information related to the data as input, and generates second information corresponding to the first information from information representing the characteristics of the area using model parameters of a machine learning model, and a learning unit that learns the model parameters based on the second information and third information representing a correct answer to the second information. [Effects of the Invention]

[0007] It is possible to realize machine reading comprehension that takes visual information into account. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating an example of an overall configuration (during learning) of a question answering device according to a first embodiment. [Figure 2] 10 is a flowchart illustrating an example of a learning process according to the first embodiment. [Figure 3] 10 is a flowchart illustrating an example of a model parameter update process according to the first embodiment. [Figure 4] FIG. 10 is a diagram illustrating an example of extraction of a feature region. [Figure 5] 10 is a flowchart illustrating an example of a language understanding process with visual effects according to the first embodiment. [Figure 6] 10 is a flowchart illustrating an example of an answer text generation probability calculation process according to the first embodiment. [Figure 7] 1 is a diagram illustrating an example of the overall configuration (at the time of inference) of a question answering device according to a first embodiment. [Figure 8] 10 is a flowchart illustrating an example of an inference process according to the first embodiment. [Figure 9] 10 is a flowchart illustrating an example of an answer text generation process according to the first embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of the overall configuration (during learning) of a question answering device according to a second embodiment. [Figure 11] 10 is a flowchart illustrating an example of a model parameter update process according to the second embodiment. [Figure 12] 10 is a flowchart illustrating an example of a language understanding process with visual effects according to a second embodiment. [Figure 13] 10 is a flowchart illustrating an example of an answer text generation probability calculation process according to the second embodiment. [Figure 14] FIG. 10 is a diagram illustrating an example of the overall configuration (at the time of inference) of a question answering device according to a second embodiment. [Figure 15] 10 is a flowchart illustrating an example of an inference process according to the second embodiment. [Figure 16] 10 is a flowchart illustrating an example of an answer text generation process according to the second embodiment. [Figure 17] FIG. 2 is a diagram illustrating an example of a hardware configuration of a question answering device according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] An embodiment of the present invention will be described below.

[0010] First embodiment In this embodiment, a question answering device 10 is described that, when given an image containing text and question text related to this image, can generate answer text taking into consideration visual information in the image (e.g., the position and size of the text in the image). Furthermore, the question answering device 10 according to this embodiment can generate answer text taking into consideration not only the position and size of the text in the image, but also visual information such as graphs and photographs included in the image (in other words, auxiliary information that helps understand the text).

[0011] As described above, it is assumed that an image containing text is provided to the question answering device 10. However, this is not limited to this, and the present embodiment can be similarly applied to any data containing text. Therefore, for example, the present embodiment can be similarly applied to any data containing text, regardless of the format, such as HTML or PDF. Examples of data containing text include HTML documents (web pages) containing text, PDF documents containing text, landscape images containing explanatory text, document data, and the like.

[0012] Here, the question answering device 10 according to this embodiment realizes machine reading comprehension using a neural network model. Therefore, the question answering device 10 according to this embodiment has a learning period in which it learns parameters of this neural network model (hereinafter also referred to as "model parameters"), and an inference period in which it performs machine reading comprehension using a neural network model using the trained model parameters. Therefore, hereinafter, the learning period and the inference period of the question answering device 10 will be described.

[0013] [Study time] First, the learning process will be described. During learning, a set of training data (training data set) including an image containing text, a question text related to this image, and a correct answer text indicating the correct answer to this question text is input to question answering device 10.

[0014] <Overall Configuration of Question Answering Device 10 During Learning> The overall configuration of the question answering device 10 during learning will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the overall configuration (during learning) of the question answering device according to the first embodiment.

[0015] As shown in FIG. 1, the question answering device 10 during learning includes a feature region extraction unit 101, a text recognition unit 102, a text analysis unit 103, a language understanding unit with visual effects 104, an answer text generation unit 105, a parameter learning unit 106, and a parameter storage unit 107.

[0016] The characteristic region extraction unit 101 extracts characteristic regions from an input image. The text recognition unit 102 performs text recognition on characteristic regions that contain text, out of the characteristic regions extracted by the characteristic region extraction unit 101, and outputs the text. The text analysis unit 103 divides the text output by the text recognition unit 102 and the input question text into token sequences. The text analysis unit 103 also divides the correct answer text into token sequences.

[0017] The visual effect language understanding unit 104 is realized by a neural network, and encodes the token sequence obtained by the text analysis unit 103 using the model parameters during training stored in the parameter storage unit 107. This results in an encoded sequence that takes visual information into consideration. In other words, language understanding that also takes the visual effects in the image into consideration is achieved.

[0018] The answer text generation unit 105 is realized by a neural network, and uses the model parameters under training stored in the parameter storage unit 107 to calculate a probability distribution representing the probability of generating an answer text from the coded sequence obtained by the language understanding unit with visual effects 104.

[0019] The parameter learning unit 106 uses the loss between the answer text generated by the answer text generation unit 105 and the input correct answer text to update the model parameters under learning stored in the parameter storage unit 107. In this way, the model parameters are learned.

[0020] The parameter storage unit 107 stores model parameters under learning (i.e., model parameters to be learned) of the neural network model that realizes the visual effect-added language understanding unit 104 and the answer text generation unit 105. Note that the model parameters under learning refer to model parameters that have not yet been learned.

[0021] <Learning process> Next, the learning process according to this embodiment will be described with reference to Fig. 2. Fig. 2 is a flowchart showing an example of the learning process according to the first embodiment. Note that, hereinafter, as an example, a case where model parameters during learning are learned by stochastic gradient descent will be described. However, the model parameters during learning may also be learned by an optimization method other than stochastic gradient descent.

[0022] First, the parameter learning unit 106 sets a variable n e is initialized to 1 (step S101).

[0023] Next, the parameter learning unit 106 learns the input training data set in a maximum of N b The training data is divided into mini-batches containing N training data (step S102). b is a preset value and can be set to any value. For example, N b =60, etc.

[0024] Next, the question answering apparatus 10 executes a model parameter update process for each mini-batch (step S103). The model parameter update process will be described in detail later.

[0025] Next, the parameter learning unit 106 e >N e It is determined whether the value is -1 (step S104). e is the number of epochs that is set in advance, and can be set to any value. For example, N e =15, etc.

[0026] In step S104 above, n e >N e If it is determined to be -1, parameter learning unit 106 ends the learning process, thereby completing the learning of the model parameters stored in parameter storage unit 107.

[0027] On the other hand, in step S104 above, n e >Ne If it is determined that the value is not -1, the parameter learning unit 106 e (step S105), and the process returns to step S102. e Steps S102 and S103 are repeatedly executed.

[0028] <Model parameter update process> Next, details of the model parameter update process in step S103 above will be described with reference to Fig. 3. Fig. 3 is a flowchart showing an example of the model parameter update process according to the first embodiment. Note that hereinafter, the model parameter update process for a certain mini-batch will be described.

[0029] First, the parameter learning unit 106 reads one piece of training data in the mini-batch (step S201).

[0030] Next, the feature region extraction unit 101 extracts K feature regions from the images included in the loaded training data (step S202). A feature region is a region based on visual features, and in this embodiment, is represented by a rectangular region. The k-th feature region is represented by an image token i having position information (7 dimensions in total) including the top left coordinate, bottom right coordinate, width, height, and area, a rectangular image representation (D dimensions), and a region type (C types). k However, any information may be used as the position information as long as it can identify the position of the feature region (for example, information on at least one of width, height, and area may be omitted, or the top-right and bottom-left coordinates may be used instead of the top-left and bottom-right coordinates, or center coordinates may be used). Also, information on either the rectangular image representation or the region type may be omitted. For example, if the feature region is a polygon (polygonal region), the rectangular region surrounding this polygon may be newly defined as the feature region.

[0031] In this embodiment, nine region types are handled: "Image," "Data (Diagram)," "Paragraph / Body," "Subdata," "Heading / Title," "Caption," "Subtitle / Author," "List," and "Other Text." Region types other than "Image" and "Data (Diagram)" are considered to contain text. However, these region types are merely examples, and other region types may be set. For example, a region type "Image Information" that combines "Image" and "Data (Diagram)" may be set, or a region type "Text Information" that combines "Paragraph / Body," "Subdata," "Heading / Title," "Caption," "Subtitle / Author," "List," and "Other Text" may be set. Thus, it is sufficient to set at least two region types: a region type indicating that a feature region does not contain text, and a region type indicating that a feature region contains text.

[0032] An example of feature region extraction by the feature region extraction unit 101 is shown in Fig. 4. The example shown in Fig. 4 shows a case where five feature regions, feature region 1100, feature region 1200, feature region 1300, feature region 1400, and feature region 1500, are extracted from image 1000 containing text. In addition, in the example shown in Fig. 4, the region type of feature region 1100 is "image," the region type of feature region 1200 is "paragraph / body text," the region type of feature region 1300 is "heading / title," the region type of feature region 1400 is "list," and the region type of feature region 1500 is "list."

[0033] For example, Faster R-CNN, as described in Reference 1 "Shaoqing Ren, Kaiming He, Ross B. Girshick, Jian Sun: Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. NIPS 2015: 91-99," can be used to extract such feature regions. However, other methods (e.g., object identification technology) can also be used as long as they are capable of extracting regions based on visual features. In addition to this, feature regions may be manually extracted from an input image (i.e., image tokens are created by manually setting, for example, the upper left coordinate, the lower right coordinate, the region type, etc.).

[0034] Next, the text recognition unit 102 performs text recognition on the feature regions extracted in step S202 and having a region type indicating that text is included, and outputs the text (step S203). Note that this text recognition may be performed using the method described in, for example, Reference 2 "Google: Tesseract Manual. 2018. Internet" (http: / / www.google.com / tesseract / ).<URL:https: / / github.com / tesseract-ocr / tesseract / blob / master / doc / tesseract.1.asc> It is possible to use Tesseract, etc., which is described in

[0035] Next, the text analysis unit 103 divides the text output in step S203 into a text token sequence (step S204). Hereinafter, it is assumed that the text is included in a certain k-th feature region, and the text token sequence obtained by dividing this text is called

[0036]

number

[0037] By using the above-mentioned Byte-level BPE, text is divided into a sequence of subword tokens. However, instead of subword tokens, a sequence of words separated by spaces or the like may be used as a text token sequence.

[0038] Next, in the same manner as in step S204, the text analysis unit 103 analyzes the question text contained in the read training data as a question token sequence (x1 q ,x2 q ,···,x J q ) (step S205), where J is the number of tokens in the question text. The question token sequence is a token sequence of subwords.

[0039] Next, the question answering device 10 executes a language understanding process with visual effects to obtain an encoded sequence in which visual information is taken into consideration (step S206). Details of the language understanding process with visual effects will now be described with reference to Fig. 5. Fig. 5 is a flowchart showing an example of the language understanding process with visual effects according to the first embodiment.

[0040] First, the visual effect language understanding unit 104 uses the image token, the text token sequence, and the question token sequence to generate the following input token sequence:

[0041]

number

[0042]

number

[0043] Hereinafter, the length of the input token sequence will be referred to as L. This L is generally adjusted to a predetermined length (for example, L = 512, etc.). If the length of the input token sequence exceeds L, the longest text included in each feature region can be deleted, or each text can be deleted equally, so that the length L of the input token sequence becomes the predetermined length. On the other hand, if the length L of the input token sequence is less than the predetermined length, it can be padded with a special token.

[0044] Next, the visual effect-added language understanding unit 104 sets the first token in the input token sequence as the processing target (step S302).

[0045] Next, the visual effects-added language understanding unit 104 determines whether the token set as the processing target is a text token (step S303). Here, a text token refers to a token included in the question token sequence, a token included in the text token sequence, or a special token such as [CLS], [SEP], or [EOS] (i.e., a subword token).

[0046] If it is determined in step S303 above that the token to be processed is a text token, the language understanding unit with visual effects 104 encodes the token to be processed (step S304). In this embodiment, the language understanding unit with visual effects 104 is assumed to be realized by a neural network model including BERT (Bidirectional Encoder Representations from Transformers), and the language understanding unit with visual effects 104 encodes the token to be processed as follows.

[0047] h=LayerNorm(TokenEmb(x)+PositionEmb(x)+SegmentEmb(x)) x represents the target token (i.e., subword token), and h represents the target token after encoding. For more information about BERT, see, for example, Reference 4, "Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language."

[0048] TokenEmb is a process of converting subword tokens into corresponding G-dimensional vectors using a neural network model. In this embodiment, the embedding vectors (G=1024) trained according to Reference 5, "Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, Luke Zettlemoyer: BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension, arXiv, 2019," are used as the initial values ​​of the model parameters of the neural network model and as the model parameters to be trained. Note that parameters of a pre-trained language model other than those in Reference 5 may also be used as the training target.

[0049] PositionEmb is a process that uses a neural network model to convert a target token into a G-dimensional vector according to its position in an input token sequence. In this embodiment, the method described in Reference 6, "Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017," is used.

[0050] SegmentEmb is a process of converting a target token in an input token sequence into a G-dimensional vector according to its segment. In this embodiment, the segment is not distinguished, and the converted vector is treated as a G-dimensional zero vector. A segment is information for distinguishing text input to BERT. In this embodiment, an image token i k Since SegmentEmb plays the role of a segment, SegmentEmb does not distinguish between segments. Note that SegmentEmb is also used in this embodiment because BERT requires SegmentEmb, but if BERT is not used, SegmentEmb is not necessary.

[0051] LayerNorm takes a G-dimensional vector as input and outputs a G-dimensional vector using the normalization method described in Reference 7, "Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E. Hinton: Layer Normalization. Arxiv, 2016."

[0052] On the other hand, if it is determined in step S303 above that the token to be processed is not a text token (i.e., the token to be processed is an image token), the visual effects-added language understanding unit 104 encodes the token to be processed as follows (step S305):

[0053] h=LayerNorm(ImgfEmb(i)+LocationEmb(i)+SegmentEmb(i)) i represents the token to be processed (i.e., image token), and h represents the token to be processed after encoding. Also, SegmentEmb and LayerNorm are as described in step S304 above.

[0054] ImgfEmb is a process for converting a rectangular image representation included in an image token from D-dimensional to G-dimensional using a feedforward network model consisting of a fully connected layer. In this embodiment, a feedforward network model consisting of one fully connected layer is used, and the model parameters of this feedforward network model are used as the model parameters to be learned.

[0055] LocationEmb is a process for converting location information contained in image tokens from 7 dimensions to D dimensions using a feedforward network model consisting of a fully connected layer. In this embodiment, a feedforward network model consisting of one fully connected layer is used, and the model parameters of this feedforward network model are used as the model parameters to be learned.

[0056] Following step S304 or step S305, the visual effect-added language understanding unit 104 determines whether the token to be processed is the final token in the input token sequence (step S306).

[0057] If it is determined in step S306 above that the token to be processed is not the final token, the visual effect-added language understanding unit 104 sets the token to be processed that is next to the current token to be processed in the input token sequence (step S307), and returns to step S303 above. As a result, each token in the input token sequence is coded, and the coded sequence H=(h1, h2, . . . , h L ) is obtained. Note that h r is the encoding of the rth token (r=1, 2, . . . , L) in the input token sequence.

[0058] On the other hand, if it is determined in step S306 above that the token to be processed is the final token, the visual effects-added language understanding unit 104 converts the coded sequence H of the input token sequence into H' using an M-layer TransformerEncoder (step S308). That is, the visual effects-added language understanding unit 104 sets H' = TransformerEncoder(H). For details about TransformerEncoder, see, for example, Reference 5 above. In this embodiment, M = 12, and a TransformerEncoder trained in Reference 5 above is used as the initial value, and the parameters of this TransformerEncoder are used as the model parameters to be trained.

[0059] Returning to Fig. 3, following step S206, the question answering device 10 executes a process for calculating the probability of generation of an answer text to calculate a probability distribution representing the probability of generation of the answer text (step S207). Details of the process for calculating the probability of generation of an answer text will now be described with reference to Fig. 6. Fig. 6 is a flowchart showing an example of the process for calculating the probability of generation of an answer text according to the first embodiment.

[0060] First, similarly to step S204, the text analysis unit 103 converts the correct answer text included in the read training data into a correct answer token sequence

[0061]

number

[0062] Next, the visual effect-added language understanding unit 104 uses the correct answer token sequence obtained in step S401 to generate a correct output token sequence.

[0063]

number

[0064] Next, the visual effect language understanding unit 104 outputs the correct output token sequence Y * The first token in the list is set as the processing target (the t-1th token to be processed) (step S403).

[0065] Next, the visual effect-added language understanding unit 104 encodes the processing target token as follows (step S404), similarly to step S304 in FIG.

[0066] h y* =LayerNorm(TokenEmb(y*)+PositionEmb(y*)+SegmentEmb(y*)) y* is the target token (i.e., subword token), h y* represents the token to be processed after encoding.

[0067] Hereafter, we will denote the coded sequence representing the coding results of the first t-1 tokens as H y* =(h1 y* ,h2 y* ,···,h t-1 y* ) is expressed as

[0068] Next, the visual effect language understanding unit 104 compares the coded sequence H' obtained by the visual effect language understanding process with the coded sequence H obtained in step S404. y* =(h1 y* ,h2 y* ,···,h t-1 y* ) is converted by the M-layer Transformer Decoder (step S405). t y =TransformerDecoder(Hy* ,H').

[0069] For details about the TransformerDecoder, see, for example, Reference 5. In this embodiment, M=12, and a TransformerDecoder trained according to Reference 5 is used as the initial value, and the parameters of this TransformerDecoder are used as the model parameters to be trained. Note that parameters of a pre-trained language model other than that of Reference 5 may also be used as the training target.

[0070] Next, the answer text generation unit 105 calculates the probability distribution p(y t |y <t *) is calculated (step S406). t The probability distribution of p(y t |y <t *)=softmax(Wh t y +b), where W∈R V×G , b∈R V are the model parameters to be learned. Note that the vocabulary size V can be set to any value, for example, 50257.

[0071] Next, the visual effect language understanding unit 104 finds the correct word y t * It is determined whether or not t=L is the final word (step S407). T The correct word is the special word [EOS] that indicates the end of the sentence or the yth word. t * is a word that indicates the end of a sentence, and t <L T If so, the answer text generation unit 105 T Pad up to with a special word.

[0072] In step S407 above, the correct word y t *If it is determined that t is not the final word, the visual effect-added language understanding unit 104 adds 1 to t (step S408) and returns to step S404, whereby the above steps S404 to S406 are repeatedly executed.

[0073] Returning to FIG. 3, following step S207, the parameter learning unit 106 learns the probability distribution p(y t |y <t *) to calculate the loss (step S208). The parameter learning unit 106 may calculate the loss, for example, as follows:

[0074]

number

[0075] Next, the parameter learning unit 106 determines whether all the training data in the mini-batch has been read (step S209).

[0076] If it is determined in step S209 that there is training data in the mini-batch that has not yet been read, the parameter learning unit 106 reads one piece of training data that has not yet been read (step S210) and returns to step S202. As a result, steps S202 to S208 are repeatedly executed for each piece of training data included in the mini-batch.

[0077] On the other hand, if it is determined in step S209 that all training data in the mini-batch has been read, the parameter learning unit 106 updates the model parameters using the loss calculated for each training data in step S208 (step S211). That is, the model parameters are updated using a known optimization method so that the loss is minimized.

[0078] As described above, in the question answering device 10 according to this embodiment, when an image including text and a question text related to this image are given, the model parameters are trained so that an answer text that takes into account the visual information in the image is generated. In other words, the model parameters are trained so that machine reading comprehension that takes into account the visual information is possible.

[0079] [At the time of inference] Next, the inference process will be described. Test data including an image containing text and a question text related to this image is input to the question answering device 10 during the inference process.

[0080] <Overall Configuration of Question Answering Device 10 During Inference> The overall configuration of the question answering device 10 during inference will be described with reference to Fig. 7. Fig. 7 is a diagram showing an example of the overall configuration (during inference) of the question answering device according to the first embodiment.

[0081] 7, the question answering device 10 during inference includes a feature region extraction unit 101, a text recognition unit 102, a text analysis unit 103, a language understanding unit with visual effects 104, an answer text generation unit 105, and a parameter storage unit 107. Of these units, the feature region extraction unit 101, the text recognition unit 102, and the text analysis unit 103 are the same as those during learning. On the other hand, the language understanding unit with visual effects 104 and the answer text generation unit 105 use trained model parameters stored in the parameter storage unit 107. Furthermore, the answer text generation unit 105 generates an answer text using a probability distribution calculated from the coded sequence obtained by the language understanding unit with visual effects 104.

[0082] <Inference processing> Next, the inference processing according to this embodiment will be described with reference to Fig. 8. Fig. 8 is a flowchart showing an example of the inference processing according to the first embodiment. In the following, it is assumed that test data given to the question answering device 10 has been read.

[0083] First, similarly to step S202 in FIG. 3, the characteristic region extraction unit 101 extracts K characteristic regions from an image included in the loaded test data (step S501).

[0084] Next, similar to step S203 in FIG. 3, the text recognition unit 102 performs text recognition on the feature regions extracted in step S501 above that have a region type indicating that text is included, and outputs the text (step S502).

[0085] Next, the text analysis unit 103 divides the text output in step S502 into a text token sequence, similar to step S204 in FIG. 3 (step S503).

[0086] Next, the text analysis unit 103 divides the question text included in the read test data into a series of question tokens (step S504), similar to step S205 in FIG.

[0087] Next, the question answering device 10 executes a language understanding process with visual effects to obtain an encoded sequence that takes visual information into consideration (step S505). The language understanding process with visual effects is the same as step S206 in Fig. 3, and therefore its description will be omitted. Hereinafter, the description will be continued assuming that an encoded sequence H' has been obtained.

[0088] Next, the question answering device 10 executes an answer text generation process to generate an answer text (step S506). Details of the answer text generation process will now be described with reference to Fig. 9. Fig. 9 is a flowchart showing an example of the answer text generation process according to the first embodiment.

[0089] First, the response text generation unit 105 sets the first token of the output token sequence to [CLS] (step S601). At this point, the only token included in the output token sequence is [CLS].

[0090] Next, the visual effect-added language understanding unit 104 sets t=2 and sets the first token of the output token sequence as the token to be processed (the t-1th token to be processed) (step S602).

[0091] Next, the visual effect-added language understanding unit 104 encodes the processing target token as follows (step S603), similarly to step S404 in FIG.

[0092] h y =LayerNorm(TokenEmb(y)+PositionEmb(y)+SegmentEmb(y)) y is the target token (i.e., subword token), h y represents the token to be processed after encoding.

[0093] Hereafter, we will denote the coded sequence representing the coding results of the first t-1 tokens as H y =(h1 y ,h2 y ,···,h t-1 y ) is expressed as

[0094] Next, similarly to step S405 in FIG. 6, the visual effect-added language understanding unit 104 compares the coded sequence H' obtained by the visual effect-added language understanding process with the coded sequence H obtained in step S603. y and are transformed by the M-layer Transformer Decoder (step S604). t y =TransformerDecoder(H y ,H'). This gives H y '= (h1 y ,h2 y,···,h t-1 y ,h t y ) is obtained.

[0095] Next, the answer text generation unit 105 calculates the probability distribution p(y t |y <t ) is calculated (step S605). t The probability distribution of p(y t |y <t )=softmax(Wh t y +b), where W∈R V×G , b∈R V are the trained model parameters.

[0096] Next, the answer text generation unit 105 calculates the probability distribution p(y t |y <t ) and generates the t-th word (step S606). The answer text generation unit 105 may generate the word with the highest probability as the t-th word, or may generate the t-th word by sampling according to the probability distribution.

[0097] Next, the response text generation unit 105 concatenates the t-th word generated in step S606 above to the end of the output token sequence (step S607).

[0098] Next, the visual effects-added language understanding unit 104 determines whether the t-th word generated in step S606 is the final word (step S608). The final word is the special word [EOS] that indicates the end of a sentence.

[0099] If it is determined in step S608 above that the t-th word is not the final word, the visual effect-added language understanding unit 104 adds 1 to t (step S609) and returns to step S603. As a result, steps S603 to S607 above are repeatedly executed to obtain a word sequence.

[0100] As described above, when an image containing text and a question text related to this image are given, the question answering device 10 according to this embodiment can generate an answer text (word sequence) that takes into account the visual information in the image.

[0101] Second embodiment In this embodiment, a case will be described in which an answer text is generated taking into consideration whether or not the characteristic regions extracted by the characteristic region extraction unit 101 are information necessary for answering the question.

[0102] In this embodiment, differences from the first embodiment will be mainly described, and descriptions of components that are the same as those in the first embodiment will be omitted.

[0103] [Study time] First, the learning process will be described. The training data input to the question answering device 10 during learning includes an image containing text, a question text, a correct answer, and a set of correct feature regions. The set of correct feature regions is a set of feature regions that are necessary to obtain a correct answer, among the feature regions extracted from the image.

[0104] <Overall Configuration of Question Answering Device 10 During Learning> The overall configuration of the question answering device 10 during learning will be described with reference to Fig. 10. Fig. 10 is a diagram showing an example of the overall configuration (during learning) of the question answering device 10 according to the second embodiment.

[0105] 10 , the question answering apparatus 10 during learning includes a feature region extraction unit 101, a text recognition unit 102, a text analysis unit 103, a language understanding unit with visual effects 104, an answer text generation unit 105, a parameter learning unit 106, a related feature region determination unit 108, and a parameter storage unit 107. The second embodiment differs from the first embodiment mainly in that the question answering apparatus 10 includes the related feature region determination unit 108.

[0106] The associated feature region determination unit 108 is realized by a neural network, and calculates the probability indicating whether the feature region extracted by the feature region extraction unit 101 is information necessary to answer the question, using the model parameters under training stored in the parameter storage unit 107. Therefore, the model parameters under training stored in the parameter storage unit 107 also include the model parameters under training of the neural network model that realizes the associated feature region determination unit 108.

[0107] Furthermore, the parameter learning unit 106 calculates a loss using the probability calculated by the related feature region determination unit 108 and the correct feature region set, and updates the model parameters under learning stored in the parameter storage unit 107 .

[0108] <Learning process> Next, the learning process according to this embodiment will be described. The overall flow of the learning process may be the same as the learning process described in FIG. 2, so hereinafter, the details of the model parameter update process in step S103 in FIG. 2 will be described. However, if the number of epochs N e and the maximum number of training data included in a mini-batch, N b may be different from that in the first embodiment. For example, the maximum number of training data N included in a mini-batch b , N b =32, etc.

[0109] <Model parameter update process> Details of the model parameter update process in step S103 in Fig. 2 will be described with reference to Fig. 11. Fig. 11 is a flowchart showing an example of the model parameter update process according to the second embodiment. Note that hereinafter, the model parameter update process for a certain mini-batch will be described.

[0110] First, the parameter learning unit 106 reads one piece of training data in the mini-batch (step S701).

[0111] Next, the feature region extraction unit 101 extracts K feature regions from the images included in the loaded training data (step S702). In this embodiment, as in the first embodiment, feature regions are represented by rectangular regions, and the kth feature region has position information (four dimensions in total) including the top left coordinate and the bottom right coordinate, a rectangular image representation (D dimensions), and a region type (C types). However, any information may be used as the position information as long as it can identify the position of the feature region, and either the rectangular image representation or the region type information may be omitted. Additionally, as in the first embodiment, for example, if the feature region is a polygon (polygonal region), the rectangular region surrounding this polygon may be newly defined as the feature region.

[0112] The same nine types of region types as in the first embodiment are also handled. However, these nine types of region types are merely examples, and it goes without saying that other region types may be set. In this embodiment, as in the first embodiment, it is sufficient to set at least two types of region types: one indicating that the characteristic region does not contain text, and one indicating that the characteristic region contains text.

[0113] Note that, similarly to the first embodiment, the feature region may be extracted using, for example, Faster R-CNN described in the above-mentioned Reference 1. In addition, in this embodiment, for example, D=2048.

[0114] Next, the text recognition unit 102 performs text recognition on feature regions of a region type indicating that text is included among the feature regions extracted in step S702 above, and outputs a word region sequence consisting of word regions that contain words resulting from the text recognition (step S703). Hereinafter, each word region is assumed to be a rectangular region, and to have position information including the top left and bottom right coordinates of the word region (four dimensions in total), and the word obtained by text recognition. As in the first embodiment, text recognition may be performed using, for example, Tesseract, as described in Reference 2 above. Note that a word region is a subregion of a feature region that contains a word resulting from text recognition.

[0115] Next, for each word region obtained in step S703, the characteristic region extraction unit 101 outputs a rectangular image representation (D dimensions) of the word region (step S704). Note that this rectangular image representation may be output in the same manner as when the rectangular image representation of the feature region was obtained in step S702. As a result, each word region has position information including the top left coordinates and bottom right coordinates of the word region (four dimensions in total), the word obtained by text recognition, and the rectangular image representation (D dimensions) of the word region.

[0116] Next, the text analysis unit 103 divides the word region sequence obtained in step S704 into a sub-word token sequence (step S705). Hereinafter, the sub-word token sequence obtained by dividing the word region sequence obtained from a certain k-th feature region is referred to as

[0117]

number

[0118] When a word contained in one word area is divided into a plurality of subwords, the word area of ​​each subword is the same as the word area of ​​the word before division.

[0119] Next, the text analysis unit 103 analyzes the question text contained in the read training data as a subword token sequence (x1 q ,x2 q ,···,x J q ) (step S706), where J is the number of subword tokens in the question text.

[0120] Next, the question answering device 10 executes a language understanding process with visual effects to obtain an encoded sequence in which visual information is taken into consideration (step S707). Details of the language understanding process with visual effects will now be described with reference to Fig. 12. Fig. 12 is a flowchart showing an example of the language understanding process with visual effects according to the second embodiment.

[0121] First, the visual effect-added language understanding unit 104 uses the subword token sequence of the word region sequence and the subword token sequence of the question text to generate the following input token sequence:

[0122]

number

[0123] Note that if the kth feature region does not contain text, the length of the subword token sequence of the word region sequence obtained from the kth feature region is 0 (i.e., L k =0).

[0124] Hereinafter, as in the first embodiment, the length of the input token sequence is set to L. If the length of the input token sequence exceeds L, the length L of the input token sequence is adjusted to a predetermined length by deleting the longest piece of text included in each feature region, deleting each piece of text evenly, etc. On the other hand, if the length L of the input token sequence is less than the predetermined length, it is sufficient to pad it with a special token.

[0125] Next, the visual effect-added language understanding unit 104 encodes each token (sub-word token) in the input token sequence (step S802). In this embodiment, the visual effect-added language understanding unit 104 encodes each token x as follows:

[0126] h=LayerNorm(TokenEmb(x)+PositionEmb(x)+SegmentEmb(x)+ROIEmb(x)+LocationEmb(x)) TokenEmb is a process for converting subword tokens (including special tokens) into corresponding G-dimensional vectors. In this embodiment, as in the first embodiment, the embedding vectors (G=1024) pre-trained according to the above-mentioned Reference 5 are used as the initial values ​​and model parameters to be trained. Note that parameters of pre-trained language models other than those in Reference 5 may also be used as training targets. However, untrained special tokens are initialized with random numbers following a normal distribution N(0,0.02).

[0127] PositionEmb is a process for converting a subword token into a G-dimensional vector according to its position in the input token sequence. In this embodiment, the embedding vector (G=1024) learned according to the above-mentioned Reference 5 is used as the initial value and the model parameter to be learned. However, as in the first embodiment, conversion into a G-dimensional vector may also be performed using the method described in the above-mentioned Reference 6.

[0128] SegmentEmb is a process that converts subword tokens into G-dimensional vectors according to the segment to which they belong. In this embodiment, a total of 10 types of segments are used: nine types of domain types and one question. An embedding vector (G=1024) is prepared for each segment, and initialized with random numbers following a normal distribution N(0,0.02) to use as the model parameters to be trained.

[0129] ROIEmb is a process of converting a rectangular image representation corresponding to a subword token into a G-dimensional vector. The rectangular image representation is a D-dimensional vector obtained by inputting a rectangular region in an input image into a neural network that implements the feature region extraction unit 101. If a subword token corresponds to a region token i, k If k is the rectangular image representation of the kth feature region, the subword token is the document token x j k If the subword token is a document token x, then the rectangular image representation of the i-th word region obtained from the k-th feature region is j k If D is , the output of ROIEmb is assumed to be a G-dimensional zero vector. In this embodiment, D=2048, and ROIEmb is converted into a G-dimensional (G=1024) vector by a feedforward network composed of fully connected layers. In this embodiment, the feedforward network is composed of one fully connected layer, and its parameters are initialized with random numbers following a normal distribution N(0,0.02) and used as the model parameters to be trained.

[0130] LocationEmb is a process that converts the location information of a region (feature region or word region) corresponding to a subword token (either a region token or a document token) from four dimensions to a G-dimensional (G=1024) vector using a feedforward network consisting of fully connected layers. LocationEmb normalizes the x-coordinate of the location information of the region by the width of the input image and the y-coordinate of the location information of the region by the height of the image, before inputting it to the feedforward network. In this embodiment, the feedforward network is composed of one fully connected layer, and its parameters are initialized with random numbers following a normal distribution N(0,0.02) and used as the model parameters to be trained. Note that if the subword token is other than a region token or a document token, the output of LocationEmb is a G-dimensional zero vector.

[0131] As in the first embodiment, LayerNorm takes a G-dimensional vector as input and outputs a G-dimensional vector using the normalization method described in Reference 7 above.

[0132] As a result, the rth subword token in the input token sequence is coded as h r Then, the coded sequence H=(h1,h2,...,h L ) is obtained. Note that each h r is a G-dimensional vector, so H is a vector sequence.

[0133] Next, the visual effects-added language understanding unit 104 converts the coded sequence H obtained in step S802 above into a vector sequence H' using a TransformerEncoder with M layers (step S803). That is, the visual effects-added language understanding unit 104 sets H' = TransformerEncoder(H). For details about TransformerEncoder, see, for example, Reference 5 above. In this embodiment, M = 12, and a TransformerEncoder trained in Reference 5 above is used as the initial value, and the parameters of this TransformerEncoder are used as the model parameters to be trained.

[0134] Next, the related feature region determination unit 108 calculates the probability indicating whether the feature region is a region necessary for generating an answer (step S804). That is, if the element of H' corresponding to a sub-word token x (either a region token or a document token) in the input token sequence is h', the related feature region determination unit 108 calculates the probability that the feature region corresponding to the sub-word token x is necessary for generating a correct answer as follows:

[0135] p=sigmoid(w1 τ h'+b1) where w1∈R G , b1∈R are the model parameters to be trained, and τ represents the transpose.

[0136] Next, the related feature region determining unit 108 converts the vector sequence H' into a vector sequence H'' using the probability obtained in step S804 (step S805). That is, the related feature region determining unit 108 converts the vector sequence H' into a vector sequence H'' using the probability obtained in step S804 (step S805). r ''=h r 'a r The vector sequence H' is converted to the vector sequence H'' by r '' is the r-th element of the vector series H'', h r ' is the r-th element of the vector sequence H'. Also, a r is the weight, and its value is the weight of the rth subword token in the input token sequence when the rth subword token is kor document token x j k If p k、 In other cases, it is set to 1.0. k is the region token i k is the probability calculated in step S804 above.

[0137] Returning to Fig. 11, following step S707, the question answering device 10 executes a process for calculating the probability of generation of an answer text to calculate a probability distribution representing the probability of generation of the answer text (step S708). Here, the details of the process for calculating the probability of generation of an answer text will be described with reference to Fig. 13. Fig. 13 is a flowchart showing an example of the process for calculating the probability of generation of an answer text according to the second embodiment.

[0138] First, similarly to step S401 in FIG. 6, the text analysis unit 103 converts the correct answer text included in the read training data into a correct answer token sequence

[0139]

number

[0140] Next, the visual effect-added language understanding unit 104 sets an index indicating the number of repetitions as t and initializes t to 0 (step S902). The following describes the processing at the tth repetition.

[0141] The visual effect language understanding unit 104 calculates the decoder input token sequence y <t is created (step S903).

[0142] y <t=([CLS],y1 * ,···,y t-1 * )=(y0,y1,...,y t-1 ) However, when t=0, y <t = ([CLS]). Also, the final step, t = L T +1 is y t =[EOS].

[0143] Next, the visual effect language understanding unit 104 calculates the decoder input token sequence y <t Each subword token y included in is encoded as follows (step S904):

[0144] h y =LayerNorm(TokenEmb(y)+PositionEmb(y)) This results in the subword token y t The encoded version is h t y Then, the coded sequence H y =(h0 y ,h1 y ,···,h t-1 y ) is obtained.

[0145] Next, the visual effect language understanding unit 104 converts the coded sequence H obtained in step S904 into y The M-layer Transformer Decoder y ' (step S905). That is, the visual effect language understanding unit 104 converts H y '=TransformerDecoder(H y ,H''). This means that H y '=(h0 y ',h1 y ',···,h t-1 y ') is obtained.

[0146] For details about the TransformerDecoder, see, for example, Reference 5. In this embodiment, M=12, and a TransformerDecoder trained according to Reference 5 is used as the initial value, and the parameters of this TransformerDecoder are used as the model parameters to be trained. Note that parameters of a pre-trained language model other than that of Reference 5 may also be used as the training target.

[0147] Next, the answer text generation unit 105 calculates the probability distribution p(y t |y <t ) is calculated (step S906). t The probability distribution of p(y t |y <t )=softmax(Wh t-1 y '+b), where W∈R V×G , b∈R V are the model parameters to be learned. Note that the vocabulary size V can be set to any value, for example, 50257.

[0148] Next, the visual effect language understanding unit 104 calculates t=L T It is determined whether the value is +1 (step S907).

[0149] In step S907 above, t=L T If it is determined that t is not +1, the visual effect-added language understanding unit 104 adds 1 to t (step S908) and returns to step S903. As a result, the above steps S903 to S906 are performed for t=0, 1, . . . , L T Repeated for +1.

[0150] Returning to FIG. 11, following step S708, the parameter learning unit 106 learns the probability distribution p(y t |y <t) and the set of correct feature regions included in the read training data, and calculates the loss (step S709). The parameter learning unit 106 may calculate the loss, for example, as follows:

[0151]

number

[0152] The subsequent steps S710 to S712 are the same as steps S209 to S211 in FIG. 3, and therefore the description thereof will be omitted.

[0153] As described above, in the question answering device 10 according to this embodiment, when an image including text, a question text related to this image, and a set of correct feature regions are given, the model parameters are trained so that an answer text that takes into account the visual information in this image is generated. In other words, the model parameters are trained so that machine reading comprehension that takes into account the visual information is possible.

[0154] [At the time of inference] Next, the inference process will be described. Test data including an image containing text and a question text related to this image is input to the question answering device 10 during the inference process.

[0155] <Overall Configuration of Question Answering Device 10 During Inference> The overall configuration of the question answering apparatus 10 during inference will be described with reference to Fig. 14. Fig. 14 is a diagram showing an example of the overall configuration (during inference) of the question answering apparatus 10 according to the second embodiment.

[0156] As shown in FIG. 14 , the question answering device 10 during inference includes a feature region extraction unit 101, a text recognition unit 102, a text analysis unit 103, a language understanding unit with visual effects 104, an answer text generation unit 105, a related feature region determination unit 108, and a parameter storage unit 107. Of these units, the feature region extraction unit 101, the text recognition unit 102, and the text analysis unit 103 are the same as those during learning. Meanwhile, the language understanding unit with visual effects 104, the answer text generation unit 105, and the related feature region determination unit 108 use trained model parameters stored in the parameter storage unit 107. Furthermore, the answer text generation unit 105 generates an answer text using a probability distribution calculated from the coded sequence obtained by the language understanding unit with visual effects 104. The related feature region determination unit 108 may output a score (related feature region score) calculated or determined from the probability indicating whether the feature region extracted by the feature region extraction unit 101 is information necessary for answering the question.

[0157] <Inference processing> Next, the inference processing according to this embodiment will be described with reference to Fig. 15. Fig. 15 is a flowchart showing an example of the inference processing according to the second embodiment. In the following, it is assumed that test data given to the question answering device 10 has been read.

[0158] First, similarly to step S702 in FIG. 11, the characteristic region extraction unit 101 extracts K characteristic regions from an image included in the loaded test data (step S1001).

[0159] Next, similar to step S703 in FIG. 11, the text recognition unit 102 performs text recognition on the feature regions extracted in step S1001 above that have a region type indicating that text is included, and outputs a word region sequence (step S1002).

[0160] Next, the feature region extraction unit 101 outputs a rectangular image representation (D dimensions) of each word region obtained in step S1002 (step S1003), similar to step S704 in Fig. 11. This results in a sequence of word regions each having position information (four dimensions in total) including the top left coordinate and bottom right coordinate, the word obtained by text recognition, and a rectangular image representation (D dimensions).

[0161] Next, the text analysis unit 103 divides the word region sequence obtained in step S1003 above into a sub-word token sequence (step S1004), similar to step S705 in FIG.

[0162] Next, similarly to step S706 in FIG. 11, the text analysis unit 103 analyzes the question text included in the read test data as a sub-word token sequence (x1 q ,x2 q ,···,x J q ) (step S1005).

[0163] Next, the question answering apparatus 10 executes a language understanding process with visual effects to obtain an encoded sequence that takes visual information into consideration (step S1006). The language understanding process with visual effects is similar to step S707 in Fig. 11, and therefore a description thereof will be omitted. In the following description, it is assumed that the vector sequence H'' has been obtained.

[0164] Next, the question answering device 10 executes an answer text generation process to generate an answer text (step S1007). Details of the answer text generation process will now be described with reference to Fig. 16. Fig. 16 is a flowchart showing an example of the answer text generation process according to the second embodiment.

[0165] First, the visual effect-added language understanding unit 104 sets an index indicating the number of repetitions as t and initializes t to 0 (step S1101). Hereinafter, the processing at the tth repetition will be described.

[0166] The language understanding unit with visual effects 104 converts the decoder input token sequence into y <t That is, the visual effect language understanding unit 104 initializes the decoder input token sequence y <t Let be a series containing only [CLS].

[0167] In the following, the processing at the tth iteration will be explained.

[0168] The visual effect-added language understanding unit 104 generates the decoder input token sequence y <t Each subword token y included in is encoded as follows (step S1103):

[0169] h y =LayerNorm(TokenEmb(y)+PositionEmb(y)) This results in the subword token y t The encoded version is h t y Then, the coded sequence H y =(h0 y ,h1 y ,···,h t-1 y ) is obtained.

[0170] Next, the visual effect-added language understanding unit 104 performs the same processing as in step S905 of FIG. 13 on the coded sequence H obtained in step S1103 above. y The M-layer Transformer Decoder y ' (step S1104). That is, the visual effect language understanding unit 104 converts H y '=TransformerDecoder(H y ,H''). This means that H y '=(h0 y ',h1 y ',···,h t-1 y ') is obtained.

[0171] Next, the answer text generation unit 105 calculates the probability distribution p(y t |y <t ) is calculated (step S1105). t The probability distribution of p(y t |y <t )=softmax(Wh t-1 y '+b), where W∈R V×G , b∈R V are the trained model parameters.

[0172] Next, the answer text generation unit 105 calculates the probability distribution p(y t |y <t ) and generates the t-th word (step S1106). The answer text generation unit 105 may generate the word with the highest probability as the t-th word, or may generate the t-th word by sampling according to the probability distribution.

[0173] Next, the answer text generation unit 105 converts the t-th word generated in step S1106 into the decoder input token sequence y <t (step S1107).

[0174] Next, the visual effect-added language understanding unit 104 determines whether the t-th word generated in the above step S1106 is the final word (step S1108).

[0175] If it is determined in step S1108 above that the t-th word is not the final word, the visual effect-added language understanding unit 104 adds 1 to t (step S1109) and returns to step S1103. As a result, steps S1103 to S1107 above are repeatedly executed to obtain a word sequence.

[0176] As described above, when an image containing text and a question text related to this image are given, the question answering device 10 according to this embodiment can generate an answer text (word sequence) that takes into account the visual information in the image.

[0177] [Evaluation of this embodiment] Next, an evaluation of whether or not a feature region is information necessary for answering a question will be described.

[0178] To evaluate this embodiment, a performance comparison was performed with a baseline. The models used in this embodiment were a model using BART as a pre-trained model, as described in Reference 5 above, and a model using T5 as a pre-trained model, as described in Reference 8, "Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; and Liu, PJ 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. J. Mach. Learn. Res. 21(140): 1-67." Hereinafter, a model using BART will be referred to as "LayoutBART," and a model using T5 will be referred to as "LayoutT5." In particular, models using LARGE as BERT will be referred to as "LayoutBART." LARGE ", "LayoutT5 LARGE " is written as ".

[0179] Furthermore, as a baseline, we adopted a model called M4C, which is described in Reference 9, "Hu, R.; Singh, A.; Darrell, T.; and Rohrbach, M. 2020. Iterative Answer Prediction with Pointer-Augmented Multi-modal Transformers for TextVQA. In CVPR, 9992-10002." M4C is a model that generates an answer to a question using a question, feature regions, and OCR tokens (corresponding to document tokens in this embodiment) as input, and has been confirmed to achieve high performance.

[0180] The five evaluation metrics used were BLEU, METEOR, ROUGE-L, CIDEr, and BERTscore. After training the model using a training dataset prepared in advance for experiments, the above four evaluation metrics were calculated using test data. The results are shown in Table 1 below.

[0181] [Table 1] As shown in Table 1 above, LayoutBART and LayoutT5 achieve higher performance than M4C in all evaluation indices. Also, as shown in Table 1 above, it can be seen that using LARGE as BERT rather than BASE can achieve higher performance. From the above, it can be seen that the method of this embodiment can achieve higher performance than conventional methods in the task of generating answer text when an image containing text and question text are given.

[0182] <Hardware configuration> Finally, the hardware configuration of the question answering device 10 according to the first and second embodiments will be described with reference to Fig. 17. Fig. 17 is a diagram illustrating an example of the hardware configuration of the question answering device 10 according to an embodiment.

[0183] 17, the question answering device 10 according to one embodiment is realized by a general computer or computer system, and includes an input device 201, a display device 202, an external I / F 203, a communication I / F 204, a processor 205, and a memory device 206. Each of these pieces of hardware is connected to each other via a bus 207 so as to be able to communicate with each other.

[0184] The input device 201 is, for example, a keyboard, a mouse, a touch panel, etc. The display device 202 is, for example, a display, etc. Note that the question answering device 10 does not necessarily have to have at least one of the input device 201 and the display device 202.

[0185] The external I / F 203 is an interface with an external device. The external device may be a recording medium 203a. The question answering apparatus 10 can read from and write to the recording medium 203a via the external I / F 203. The recording medium 203a may store one or more programs that realize each functional unit of the question answering apparatus 10 (the feature region extraction unit 101, the text recognition unit 102, the text analysis unit 103, the language understanding unit with visual effects 104, the answer text generation unit 105, the parameter learning unit 106, and the related feature region determination unit 108).

[0186] The recording medium 203a may be, for example, a CD (Compact Disc), a DVD (Digital Versatile Disk), an SD memory card (Secure Digital memory card), or a USB (Universal Serial Bus) memory card.

[0187] The communication I / F 204 is an interface for connecting the question answering apparatus 10 to a communication network. Note that one or more programs for realizing each functional unit of the question answering apparatus 10 may be acquired (downloaded) from a predetermined server device or the like via the communication I / F 204.

[0188] The processor 205 is, for example, a computing device such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc. Each functional unit of the question answering device 10 is realized by, for example, processing that is executed by the processor 205 in accordance with one or more programs stored in the memory device 206.

[0189] The memory device 206 is, for example, any of various storage devices such as a hard disk drive (HDD), a solid state drive (SSD), a random access memory (RAM), a read only memory (ROM), a flash memory, etc. The parameter storage unit 107 of the question answering apparatus 10 can be realized using, for example, the memory device 206. Note that the parameter storage unit 107 may also be realized using a storage device (for example, a database server, etc.) connected to the question answering apparatus 10 via a communication network.

[0190] The question answering device 10 according to the first and second embodiments can realize the above-described learning process and inference process by having the hardware configuration shown in Fig. 17. Note that the hardware configuration shown in Fig. 17 is an example, and the question answering device 10 may have other hardware configurations. For example, the question answering device 10 may have multiple processors 205 or multiple memory devices 206.

[0191] The following additional notes are provided regarding the above-described embodiments.

[0192] (Appendix 1) Memory and at least one processor coupled to said memory; Including, Using data including a visual region and first information related to the data as input, generate second information corresponding to the first information from information representing features of the region using model parameters of a machine learning model; a learning device that learns the model parameters based on the second information and third information that represents a correct answer to the second information. (Appendix 2) The processor: The learning device according to claim 1, wherein the learning device creates features of the first information and information representing the characteristics of the region, and generates the second information from the features. (Appendix 3) 3. The learning device according to claim 1, wherein the area includes at least an image or a diagram. (Appendix 4) 4. The learning device according to claim 1, wherein the first information is text information representing content related to the data. (Appendix 5) Memory and at least one processor coupled to said memory; Including, Using data including a visual region and first information related to the data as input, calculate a degree of association between the region and second information corresponding to the first information using model parameters of a machine learning model; A learning device that learns the model parameters based on the relevance and information indicating a correct answer to the relevance. (Appendix 6) Memory and at least one processor coupled to said memory; Including, A generation device that receives data including a visual region and first information related to the data as input, and generates second information corresponding to the first information from information representing the characteristics of the region using model parameters of a trained machine learning model. (Appendix 7) Memory and at least one processor coupled to said memory; Including, An output device that receives as input data including a visual area and first information related to the data, and outputs a predetermined evaluation value for the area based on the degree of association between the area and second information corresponding to the first information, using model parameters of a trained machine learning model. (Appendix 8) A non-transitory storage medium storing a program executable by a computer to perform a learning process, The learning process includes: Using data including a visual region and first information related to the data as input, generate second information corresponding to the first information from information representing features of the region using model parameters of a machine learning model; A non-transitory storage medium that learns the model parameters based on the second information and third information that represents a correct answer to the second information. (Appendix 9) A non-transitory storage medium storing a program executable by a computer to perform a generation process, The generation process includes: A non-transitory storage medium that receives data containing a visual region and first information related to the data as input, and generates second information corresponding to the first information from information representing the characteristics of the region using model parameters of a trained machine learning model.

[0193] The present invention is not limited to the above-described specifically disclosed embodiments, and various modifications, changes, and combinations with known technologies are possible without departing from the scope of the claims.

[0194] This application is based on PCT / JP2020 / 008390, filed in Japan on February 28, 2020, the entire contents of which are incorporated herein by reference. [Explanation of symbols]

[0195] 10 Question Answering Device 101 Feature region extraction unit 102 Text Recognition Unit 103 Text Analysis Section 104 Language Understanding Unit with Visual Effects 105 Answer text generation unit 106 Parameter Learning Unit 107 Parameter storage section 108 Related feature region determination unit

Claims

1. a generation unit that receives data including a visual region and first information related to the data as input, and generates second information corresponding to the first information from information representing features of the region using model parameters of a machine learning model; a learning unit that learns the model parameters based on the second information and third information that indicates a correct answer to the second information; A learning device comprising:

2. The generation unit The learning device according to claim 1 , wherein feature quantities are created for the information representing the features of the region and the first information, and the second information is generated from the feature quantities.

3. The learning device according to claim 1 or 2, wherein the area includes at least an image or a diagram.

4. The learning device according to claim 1 , wherein the first information is text information that represents content related to the data.

5. a calculation unit that receives data including a visual region and first information related to the data as input, and calculates a degree of association between the region and second information corresponding to the first information using model parameters of a machine learning model; a learning unit that learns the model parameters based on the relevance and information indicating a correct answer to the relevance; A learning device comprising:

6. a generation unit that receives data including a visual region and first information related to the data as input, and generates second information corresponding to the first information from information representing features of the region using model parameters of a trained machine learning model; A generating device comprising:

7. an output unit that receives data including a visual region and first information related to the data as input, and outputs a predetermined evaluation value for the region based on the degree of association between the region and second information corresponding to the first information, using model parameters of a trained machine learning model; An output device comprising:

8. a generation step of receiving data including a visual region and first information related to the data as input, and generating second information corresponding to the first information from information representing features of the region using model parameters of a machine learning model; a learning procedure for learning the model parameters based on the second information and third information representing a correct answer to the second information; A learning method characterized in that the above is executed by a computer.

9. a generation step of receiving data including a visual region and first information related to the data as input, and generating second information corresponding to the first information from information representing features of the region using model parameters of a trained machine learning model; A generating method characterized by being executed by a computer.

10. a generation unit that receives data including a visual region and first information related to the data as input, and generates second information corresponding to the first information from information representing features of the region using model parameters of a machine learning model; a learning unit that learns the model parameters based on the second information and third information that indicates a correct answer to the second information; A program that causes a computer to function as a

11. a generation unit that receives data including a visual region and first information related to the data as input, and generates second information corresponding to the first information from information representing features of the region using model parameters of a trained machine learning model; A program that causes a computer to function as a

12. a generation step of receiving data including a visual region and first information related to the data as input, and generating second information corresponding to the first information from information representing features of the region using model parameters of a machine learning model; a learning procedure for learning the model parameters based on the second information and third information representing a correct answer to the second information; A method for producing a trained machine learning model, characterized in that a computer executes the above.

Images