Generation system, learning system, generation method, learning method, and program

Abstract

To provide a machine-reading system in consideration of visual information.SOLUTION: A learning device according to an embodiment is provided with: a generation unit for, with data including a visual region and first information related to the data as inputs, generating second information corresponding to the first information from information representing a feature of the region using model parameters of a machine learning model; and a learning unit for learning the model parameters based on the second information and third information representing a correct answer for the second information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention Generation system, learning system, generation method, learning method, and program Regarding. [Background technology]

[0002] If artificial intelligence can accurately perform "machine reading comprehension," which generates answers to questions based on a given set of documents, it can be applied to a wide range of services such as question answering and intelligent agent dialogue. There are two types of machine reading comprehension: extractive and generative. As a conventional technique for generative machine reading comprehension, for example, the technique disclosed in Non-Patent Document 1 is known. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Kyosuke Nishida, Itsumi Saito, Kosuke Nishida, Kazutoshi Shinoda, Atsushi Otsuka, Hisako Asano, Junji Tomita: Multi-style Generative Reading Comprehension. ACL (1) 2019: 2273-2284 [Overview of the project] [Problems that the invention aims to solve]

[0004] However, conventional machine reading technologies only dealt with text and could not handle visual information such as the position and size of text within a document. Therefore, when machine reading was used to understand documents with multiple texts laid out (for example, HTML (HyperText Markup Language) documents or PDF (Portable Document Format) documents), all information other than the text content was lost.

[0005] One embodiment of the present invention has been made in view of the above points, and aims to realize machine reading comprehension that takes visual information into consideration. [Means for solving the problem]

[0006] To achieve the above objective, a learning device according to one embodiment is characterized by comprising: a generation unit that takes data including a visual region and first information related to the data as input and uses model parameters of a machine learning model to generate second information corresponding to the first information from information representing the features of the region; and a learning unit that learns the model parameters based on the second information and third information representing the correct answer for the second information. [Effects of the Invention]

[0007] This enables machine reading comprehension that takes visual information into account. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows an example of the overall configuration (during learning) of the question answering device according to the first embodiment. [Figure 2] This is a flowchart showing an example of the learning process according to the first embodiment. [Figure 3] This is a flowchart showing an example of the model parameter update process according to the first embodiment. [Figure 4] This is a diagram illustrating an example of feature region extraction. [Figure 5] This is a flowchart showing an example of language understanding processing with visual effects according to the first embodiment. [Figure 6] This is a flowchart showing an example of the process for calculating the probability of generating response text according to the first embodiment. [Figure 7] This figure shows an example of the overall configuration (during inference) of a question answering device according to the first embodiment. [Figure 8] This is a flowchart showing an example of the inference process according to the first embodiment. [Figure 9] This is a flowchart showing an example of the response text generation process according to the first embodiment. [Figure 10] This figure shows an example of the overall configuration (during learning) of the question answering device according to the second embodiment. [Figure 11] A flowchart showing an example of the model parameter update process according to the second embodiment. [Figure 12] This is a flowchart showing an example of language understanding processing with visual effects according to the second embodiment. [Figure 13] This flowchart shows an example of the process for calculating the probability of generating response text according to the second embodiment. [Figure 14] This figure shows an example of the overall configuration (during inference) of a question answering device according to the second embodiment. [Figure 15] This is a flowchart showing an example of the inference process according to the second embodiment. [Figure 16] This is a flowchart showing an example of the response text generation process according to the second embodiment. [Figure 17] This figure shows an example of the hardware configuration of a question answering device according to one embodiment. [Modes for carrying out the invention]

[0009] The following describes one embodiment of the present invention.

[0010] • First embodiment This embodiment describes a question answering device 10 that, given an image containing text and a question text related to this image, can generate an answer text that takes into account the visual information in the image (for example, the position and size of the text in the image). Furthermore, the question answering device 10 according to this embodiment can generate an answer text that takes into account not only the position and size of the text in the image, but also other visual information such as graphs and photographs included in the image (in other words, auxiliary information that helps in understanding the text).

[0011] As mentioned above, the question answering device 10 is assumed to be provided with an image containing text, but it is not limited to this, and this embodiment can be similarly applied to any data containing text. Therefore, it can be similarly applied to any data containing text, regardless of format such as HTML or PDF. Examples of data containing text include HTML documents (web pages) containing text, PDF documents containing text, landscape images containing explanatory text, document data, etc.

[0012] In this embodiment, the question answering device 10 achieves machine reading comprehension using a neural network model. Therefore, the question answering device 10 in this embodiment has a learning phase in which it learns the parameters of this neural network model (hereinafter also referred to as "model parameters"), and an inference phase in which it performs machine reading comprehension using the neural network model with the learned model parameters. Accordingly, the learning phase and inference phase of the question answering device 10 will be described below.

[0013] [During learning] First, let's explain the training process. During training, the question answering device 10 receives a set of training data (training dataset) that includes images containing text, question text related to these images, and correct answer text representing the correct answer to these question texts.

[0014] <Overall configuration of the question answering device 10 during learning> The overall configuration of the question answering device 10 during learning will be explained with reference to Figure 1. Figure 1 is a diagram showing an example of the overall configuration (during learning) of the question answering device according to the first embodiment.

[0015] As shown in Figure 1, the question answering device 10 during learning includes a feature region extraction unit 101, a text recognition unit 102, a text analysis unit 103, a language understanding unit with visual effects 104, a response text generation unit 105, a parameter learning unit 106, and a parameter storage unit 107.

[0016] The feature region extraction unit 101 extracts feature regions from the input image. The text recognition unit 102 performs text recognition on the feature regions containing text among the feature regions extracted by the feature region extraction unit 101 and outputs the text. The text analysis unit 103 divides the text output by the text recognition unit 102 and the input question text into token sequences. The text analysis unit 103 also divides the correct answer text into token sequences.

[0017] The visual effects-enhanced language understanding unit 104 is implemented using a neural network and encodes the token sequence obtained by the text analysis unit 103 using the model parameters being trained and stored in the parameter storage unit 107. This results in an encoded sequence that takes visual information into account. In other words, language understanding that also takes into account the visual effects in images is achieved.

[0018] The response text generation unit 105 is implemented using a neural network and uses the model parameters being trained, stored in the parameter storage unit 107, to calculate a probability distribution representing the probability of generating the response text from the coded sequence obtained by the language understanding unit 104 with visual effects.

[0019] The parameter learning unit 106 updates the model parameters being learned, which are stored in the parameter storage unit 107, using the loss between the answer text generated by the answer text generation unit 105 and the input correct answer text. This allows the model parameters to be learned.

[0020] The parameter storage unit 107 stores the model parameters being trained (i.e., the model parameters to be trained) of the neural network model that implements the visual effects language understanding unit 104 and the response text generation unit 105. Note that "model parameters being trained" refers to model parameters that have not yet been trained.

[0021] <Learning Process> Next, the learning process according to this embodiment will be described while referring to FIG. 2. FIG. 2 is a flowchart showing an example of the learning process according to the first embodiment. Hereinafter, as an example, the case where the in-training model parameters are learned by stochastic gradient descent will be described. However, the in-training model parameters may be learned by other optimization methods other than stochastic gradient descent.

[0022] First, the parameter learning unit 106 initializes a variable n representing the number of epochs to 1 (step S101). e (step S101).

[0023] Next, the parameter learning unit 106 divides the input training data set into mini-batches each containing a maximum of N b training data (step S102). N b is a preset value and can be set to any value. For example, it is conceivable to set N b = 60 or the like.

[0024] Next, the question-and-answer device 10 executes model parameter update processing for each mini-batch (step S103). Details of the model parameter update processing will be described later.

[0025] Next, the parameter learning unit 106 determines whether n e > N e - 1 (step S104). N e is the preset number of epochs and can be set to any value. For example, it is conceivable to set N e = 15 or the like.

[0026] In step S104 above, if it is determined that n e > N e - 1, the parameter learning unit 106 ends the learning process. As a result, the learning of the model parameters stored in the parameter storage unit 107 is completed.

[0027] On the other hand, in step S104 above, if n e > Ne If it is determined that it is not -1, the parameter learning unit 106 will determine n e Add 1 to this (step S105) and return to step S102 above. This gives the number of epochs N. e Steps S102 and S103 described above are repeated for minutes.

[0028] ≪Model parameter update process≫ Next, the details of the model parameter update process in step S103 described above will be explained with reference to Figure 3. Figure 3 is a flowchart of an example of the model parameter update process according to the first embodiment. In the following, the model parameter update process for a certain minibatch will be described.

[0029] First, the parameter learning unit 106 reads one training data item from the minibatch (step S201).

[0030] Next, the feature region extraction unit 101 extracts K feature regions from the images included in the loaded training data (step S202). A feature region is a region based on visual features, and in this embodiment, it is represented as a rectangular region. The k-th feature region is an image token i that has positional information (7 dimensions in total) including the top-left coordinate, bottom-right coordinate, width, height, and area, a rectangular image representation (D dimensions), and a region type (C types). k It shall be represented as follows. However, any information can be used for positional information as long as it can identify the position of the feature region (for example, at least one piece of information such as width, height, and area may be omitted, or the top-right and bottom-left coordinates may be used instead of the top-left and bottom-right coordinates, or the center coordinate may be used). Also, information on either the rectangular image representation or the region type may be omitted. For example, if the feature region is a polygon (polygonal region), the rectangular region surrounding this polygon may be used as the feature region.

[0031] In this embodiment, nine types of region types are used, for example, "image," "data (chart)," "paragraph / body text," "subdata," "heading / title," "caption," "subtitle / author name," "list," and "other text." Furthermore, all region types except "image" and "data (chart)" include text. However, these region types are merely examples, and other region types may be set. For example, a region type "image information" may be set that combines "image" and "data (chart)," or a region type "text information" may be set that combines "paragraph / body text," "subdata," "heading / title," "caption," "subtitle / author name," "list," and "other text." Thus, it is sufficient to set at least two types of region types: region types that indicate that the feature region does not contain text, and region types that indicate that the feature region contains text.

[0032] Figure 4 shows an example of feature region extraction by the feature region extraction unit 101. In the example shown in Figure 4, five feature regions, feature region 1100, feature region 1200, feature region 1300, feature region 1400, and feature region 1500, are extracted from an image 1000 containing text. In the example shown in Figure 4, the region type of feature region 1100 is "image", the region type of feature region 1200 is "paragraph / body text", the region type of feature region 1300 is "heading / title", the region type of feature region 1400 is "list", and the region type of feature region 1500 is "list".

[0033] For extracting such feature regions, it is possible to use methods such as Faster R-CNN, as described in Reference 1, "Shaoqing Ren, Kaiming He, Ross B. Girshick, Jian Sun: Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. NIPS 2015: 91-99". However, other methods (e.g., object recognition techniques) that can extract regions based on visual features can also be used. In addition to the above, feature regions may be extracted manually from the input image (i.e., image tokens with manually defined top-left and bottom-right coordinates, region type, etc., are created).

[0034] Next, the text recognition unit 102 performs text recognition on the feature regions of the region type that indicate the presence of text, from among the feature regions extracted in step S202, and outputs the text (step S203). Note that this text recognition is based on, for example, Reference 2 "Google: Tesseract Manual. 2018. Internet<URL:https: / / github.com / tesseract-ocr / tesseract / blob / master / doc / tesseract.1.asc> It is possible to use Tesseract, etc., as described in [the document].

[0035] Next, the text analysis unit 103 divides the text output in step S203 into a sequence of text tokens (step S204). From here on, assuming that the text is contained in a certain k-th feature region, the sequence of text tokens obtained by dividing this text is...

[0036]

number

[0037] While the above Byte-level BPE divides the text into subword token sequences, a sequence of words separated by spaces, for example, may be used as the text token sequence instead of subword tokens.

[0038] Next, the text analysis unit 103, similar to step S204 above, analyzes the question text contained in the loaded training data into a question token sequence (x1 q , x2 q ,···,x J q Divide into (step S205). J is the number of tokens in the question text. Note that the question token sequence is the token sequence of the subwords.

[0039] Next, the question answering device 10 performs language understanding processing with visual effects to obtain an encoded sequence that takes visual information into account (step S206). The details of the language understanding processing with visual effects will now be explained with reference to Figure 5. Figure 5 is a flowchart showing an example of the language understanding processing with visual effects according to the first embodiment.

[0040] First, the visual effects language understanding unit 104 uses image tokens, text token sequences, and question token sequences to determine the following input token sequence

[0041]

number

[0042]

number

[0043] Hereafter, the length of the input token sequence will be denoted as L. It is common practice to adjust L to a predetermined length (e.g., L = 512). If the input token sequence length exceeds L, the length of the input token sequence L is adjusted to the predetermined length by removing the longest text from each feature region, or by removing all texts equally. On the other hand, if the input token sequence length L is insufficient, it can be padded with special tokens.

[0044] Next, the visual effects language understanding unit 104 sets the first token in the input token sequence as the target for processing (step S302).

[0045] Next, the visual effects language understanding unit 104 determines whether the token set as the target for processing is a text token (step S303). Here, text tokens refer to tokens included in the question token series, tokens included in the text token series, and special tokens such as [CLS], [SEP], and [EOS] (i.e., subword tokens).

[0046] If the token to be processed is determined to be a text token in step S303 above, the visual effects language understanding unit 104 encodes the token to be processed (step S304). Here, in this embodiment, the visual effects language understanding unit 104 is implemented by a neural network model including BERT (Bidirectional Encoder Representations from Transformers), and the visual effects language understanding unit 104 encodes the token to be processed as follows.

[0047] h=LayerNorm(TokenEmb(x)+PositionEmb(x)+SegmentEmb(x)) x represents the token to be processed (i.e., the subword token), and h represents the token to be processed after encoding. For more information on BERT, please refer to, for example, Reference 4, "Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language".

[0048] TokenEmb is a process that converts subword tokens into corresponding G-dimensional vectors using a neural network model. In this embodiment, the embedding vectors (G=1024) trained according to Reference 5, "Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, Luke Zettlemoyer: BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension, arXiv, 2019," are used as the initial values ​​for the model parameters of the neural network model and are the model parameters to be trained. Alternatively, parameters from a pre-trained language model other than those from Reference 5 may be used as the training targets.

[0049] PositionEmb is a process that uses a neural network model to convert the position of the token to be processed in the input token sequence into a G-dimensional vector. In this embodiment, the method described in reference 6, "Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017." is used.

[0050] SegmentEmb is a process that converts the token to be processed in the input token sequence into a G-dimensional vector according to its segment. In this embodiment, the converted vector is treated as a G-dimensional zero vector without distinguishing segments. A segment is information used to distinguish text input to BERT. In this embodiment, image token i k Since the segments function as segments, SegmentEmb does not distinguish between segments. Note that SegmentEmb is required for BERT, and therefore is used in this embodiment; however, if BERT is not used, SegmentEmb is not necessary.

[0051] LayerNorm takes a G-dimensional vector as input and outputs a G-dimensional vector using the normalization method described in reference 7, "Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E. Hinton: Layer Normalization. Arxiv, 2016."

[0052] On the other hand, if it is determined in step S303 above that the token to be processed is not a text token (i.e., the token to be processed is an image token), the visual effects language understanding unit 104 encodes the token to be processed as follows (step S305).

[0053] h=LayerNorm(ImgfEmb(i)+LocationEmb(i)+SegmentEmb(i)) i represents the token to be processed (i.e., the image token), and h represents the token to be processed after encoding. Furthermore, SegmentEmb and LayerNorm are as described in step S304 above.

[0054] ImgfEmb is a process that transforms the rectangular image representation contained in an image token from D-dimensional to G-dimensional using a feedforward network model composed of fully connected layers. In this embodiment, a feedforward network model consisting of one fully connected layer is used, and the model parameters of this feedforward network model are used as the model parameters to be trained.

[0055] LocationEmb is a process that transforms location information contained in image tokens from 7 dimensions to D dimensions using a feedforward network model composed of fully connected layers. In this embodiment, a feedforward network model consisting of one fully connected layer is used, and the model parameters of this feedforward network model are used as the model parameters to be trained.

[0056] Following step S304 or step S305 described above, the visual effects language understanding unit 104 determines whether the token to be processed is the last token in the input token sequence (step S306).

[0057] If it is determined in step S306 that the token to be processed is not the last token, the visual effects language understanding unit 104 sets the next token in the input token sequence after the current token to be processed as the target of processing (step S307), and returns to step S303. As a result, each token in the input token sequence is encoded, and the encoded sequence H=(h1,h2,···,h L ) is obtained. Note that h r This is the encoded r-th token (r=1,2,···,L) in the input token sequence.

[0058] On the other hand, if the token to be processed in step S306 is determined to be the final token, the visual language understanding unit 104 converts the encoded sequence H of the input token sequence to H' using the M-layer TransformerEncoder (step S308). That is, the visual language understanding unit 104 sets H' = TransformerEncoder(H). For more information on the TransformerEncoder, please refer to, for example, Reference 5 mentioned above. In this embodiment, M=12, and the TransformerEncoder trained according to Reference 5 is used as the initial value, and the parameters of this TransformerEncoder are used as the model parameters to be trained.

[0059] Returning to Figure 3, following step S206, the question answering device 10 performs a process to calculate the probability of generating the answer text and calculates a probability distribution representing the probability of generating the answer text (step S207). Here, the details of the process to calculate the probability of generating the answer text will be explained with reference to Figure 6. Figure 6 is a flowchart showing an example of the process to calculate the probability of generating the answer text according to the first embodiment.

[0060] First, the text analysis unit 103, similar to step S204 above, analyzes the correct answer text contained in the loaded training data into a series of correct answer tokens.

[0061]

number

[0062] Next, the visual language understanding unit 104 uses the correct answer token sequence obtained in step S401 above to produce the correct output token sequence.

[0063]

number

[0064] Next, the visual effects language understanding unit 104, with t=2, outputs the correct output token sequence Y * The first token in the sequence is set as the token to be processed (the (t-1)th token to be processed) (step S403).

[0065] Next, the visual effects language understanding unit 104 encodes the token to be processed as follows (step S404), similar to step S304 in Figure 5.

[0066] h y* =LayerNorm(TokenEmb(y*)+PositionEmb(y*)+SegmentEmb(y*)) y* is the token to be processed (i.e., the subword token), h y* This represents the token to be processed after encoding.

[0067] Hereafter, the encoding sequence representing the encoding result of the first t-1 tokens to be processed is H y* =(h1 y* ,h2 y* ,···,h t-1 y* ) is expressed as.

[0068] Next, the visual effects language understanding unit 104 uses the coded sequence H' obtained in the visual effects language understanding process and the coded sequence H obtained in step S404 above. y* =(h1 y* ,h2 y* ,···,h t-1 y* ) is converted by the TransformerDecoder of the M layer (step S405). That is, the language understanding unit with visual effects 104 converts h t y =TransformerDecoder(Hy* Let's call it H'.

[0069] For more information on TransformerDecoder, please refer to, for example, Reference 5 mentioned above. In this embodiment, M=12, a TransformerDecoder trained according to Reference 5 is used as the initial value, and the parameters of this TransformerDecoder are used as the model parameters to be trained. Alternatively, parameters of a pre-trained language model other than those in Reference 5 may be used as the training targets.

[0070] Next, the response text generation unit 105 generates the probability distribution p(y) of the t-th word. t |y <t Calculate the word y in the pre-set output vocabulary (vocabulary size V) (step S406). t The probability distribution of is p(y t |y <t *)=softmax(Wh t y It is calculated by (+b). Here, W∈R V×G , b∈R V These are the model parameters to be learned. Note that the vocabulary size V can be set to any value, but for example, 50257 is a possible value.

[0071] Next, the language comprehension unit with visual effects 104 determines the correct word y t * Determine whether t = L T This refers to the special word [EOS] which indicates the nth word or the end of a sentence. The correct word is y. t * The word that indicates the end of the sentence, and t <L T If so, the answer text generation unit 105 will generate the text from t+1 to L T Padding up to that point with special words.

[0072] In step S407 above, the correct word is y t *If it is determined that t is not the final word, the visual effects language understanding unit 104 adds 1 to t (step S408) and returns to step S404. As a result, steps S404 to S406 described above are repeatedly executed.

[0073] Return to Figure 3. Following step S207, the parameter learning unit 106 uses the probability distribution p(y) calculated in step S406 of Figure 6. t |y <t The loss is calculated using *) (step S208). The parameter learning unit 106 can calculate the loss as follows, for example.

[0074]

number

[0075] Next, the parameter learning unit 106 determines whether or not it has read all the training data in the minibatch (step S209).

[0076] If step S209 determines that there is training data in the minibatch that has not yet been loaded, the parameter learning unit 106 loads one training data item that has not yet been loaded (step S210) and returns to step S202. As a result, steps S202 to S208 are repeatedly executed for each training data item included in the minibatch.

[0077] On the other hand, when it is determined that all the training data in the mini-batch has been read in step S209 above, the parameter learning unit 106 updates the model parameters using the loss Loss calculated in step S208 above for each training data (step S211). That is, the model parameters are updated by a known optimization method so that the loss Loss is minimized.

[0078] As described above, in the question-and-answer device 10 according to the present embodiment, when an image including text and a question text related to this image are given, the model parameters are learned so that an answer text considering the visual information in this image is generated. That is, the model parameters are learned so that machine reading comprehension considering visual information is possible.

[0079] [During inference] Next, the situation during inference will be described. Test data including an image including text and a question text related to this image is input to the question-and-answer device 10 during inference.

[0080] [Overall configuration of the question-and-answer device 10 during inference] The overall configuration of the question-and-answer device 10 during inference will be described while referring to FIG. 7. FIG. 7 is a diagram showing an example of the overall configuration (during inference) of the question-and-answer device according to the first embodiment.

[0081] As shown in FIG. 7, the question-and-answer device 10 during inference includes a feature region extraction unit 101, a text recognition unit 102, a text analysis unit 103, a vision-augmented language understanding unit 104, an answer text generation unit 105, and a parameter storage unit 107. Among these units, the feature region extraction unit 101, the text recognition unit 102, and the text analysis unit 103 are the same as during learning. On the other hand, the vision-augmented language understanding unit 104 and the answer text generation unit 105 use the learned model parameters stored in the parameter storage unit 107. Also, the answer text generation unit 105 generates an answer text using the probability distribution calculated from the encoded sequence obtained by the vision-augmented language understanding unit 104.

[0082] <Inference Processing> Next, the inference process according to this embodiment will be described with reference to Figure 8. Figure 8 is a flowchart showing an example of the inference process according to the first embodiment. Hereafter, it will be assumed that the test data provided to the question answering device 10 has been read.

[0083] First, the feature region extraction unit 101 extracts K feature regions from the image contained in the loaded test data, similar to step S202 in Figure 3 (step S501).

[0084] Next, the text recognition unit 102 performs text recognition on the feature regions of the region type that indicate the presence of text, from among the feature regions extracted in step S501, similar to step S203 in Figure 3, and outputs the text (step S502).

[0085] Next, the text analysis unit 103 divides the text output in step S502 into a sequence of text tokens, similar to step S204 in Figure 3 (step S503).

[0086] Next, the text analysis unit 103 divides the question text contained in the loaded test data into a sequence of question tokens, similar to step S205 in Figure 3 (step S504).

[0087] Next, the question answering device 10 performs language understanding processing with visual effects to obtain an encoded sequence that takes visual information into account (step S505). Since the language processing with visual effects is the same as step S206 in Figure 3, its explanation is omitted. From here on, the explanation will continue assuming that the encoded sequence H' has been obtained.

[0088] Next, the question answering device 10 executes a response text generation process to generate response text (step S506). The details of the response text generation process will now be explained with reference to Figure 9. Figure 9 is a flowchart showing an example of the response text generation process according to the first embodiment.

[0089] First, the response text generation unit 105 sets the first token of the output token sequence to [CLS] (step S601). At this point, the only token included in the output token sequence is [CLS].

[0090] Next, the visual effects language understanding unit 104 sets the first token of the output token sequence as the processing target (the (t-1)th processing target token) when t=2 (step S602).

[0091] Next, the visual effects language understanding unit 104 encodes the token to be processed as follows, similar to step S404 in Figure 6 (step S603).

[0092] h y =LayerNorm(TokenEmb(y)+PositionEmb(y)+SegmentEmb(y)) y is the token to be processed (i.e., the subword token), h y This represents the token to be processed after encoding.

[0093] Hereafter, the encoding sequence representing the encoding result of the first t-1 tokens to be processed is H y =(h1 y ,h2 y ,···,h t-1 y ) is expressed as.

[0094] Next, the visual effects language understanding unit 104, similar to step S405 in Figure 6, uses the coded sequence H' obtained in the visual effects language understanding process and the coded sequence H obtained in step S603 above. y The M-layer TransformerDecoder converts the two (step S604). That is, the visual effects language understanding unit 104 converts the two. t y =TransformerDecoder(H y Let ,H'). This means H y '= (h1 y ,h2 y, ···, h t-1 y , h t y ) is obtained.

[0095] Next, the response text generation unit 105 calculates the probability distribution p(y t |y <t ) for the t-th word (step S605). The probability distribution of the word y t in the preset output vocabulary (vocabulary size V) is calculated as p(y t |y <t ) = softmax(Wh t y + b). Here, W ∈ R V×G , b ∈ R V are learned model parameters.

[0096] Next, the response text generation unit 105 generates the t-th word based on the probability distribution p(y t |y <t ) calculated in step S605 above (step S606). The response text generation unit 105 may generate the word with the maximum probability as the t-th word, or may generate the t-th word by sampling according to the probability distribution.

[0097] Next, the response text generation unit 105 concatenates the t-th word generated in step S606 above to the end of the output token sequence (step S607).

[0098] Next, the visual effect-enhanced language understanding unit 104 determines whether the t-th word generated in step S606 above is the final word (step S608). The final word refers to the special word [EOS] indicating the end of the sentence.

[0099] If it is determined in step S608 above that the t-th word is not the final word, the visual effect-enhanced language understanding unit 104 adds 1 to t (step S609) and returns to step S603. As a result, steps S603 to S607 above are repeatedly executed, and a word sequence is obtained.

[0100] As described above, the question answering device 10 according to this embodiment can generate a response text (word sequence) that takes into account the visual information in the image when given an image containing text and a question text related to the image.

[0101] • Second embodiment In this embodiment, we will describe a case in which the response text is generated while also considering whether the feature region extracted by the feature region extraction unit 101 is information necessary to answer the question.

[0102] In this embodiment, we will mainly describe the differences from the first embodiment, and will omit the explanation of components that are the same as those in the first embodiment.

[0103] [During learning] First, let's explain the learning process. The training data input to the question answering device 10 during the learning process includes an image containing text, the question text, and the correct answer, as well as a set of correct feature regions. The set of correct feature regions is the set of feature regions extracted from the image that are necessary to obtain the correct answer.

[0104] <Overall configuration of the question answering device 10 during learning> The overall configuration of the question answering device 10 during learning will be explained with reference to Figure 10. Figure 10 is a diagram showing an example of the overall configuration (during learning) of the question answering device 10 according to the second embodiment.

[0105] As shown in Figure 10, the question answering device 10 during learning includes a feature region extraction unit 101, a text recognition unit 102, a text analysis unit 103, a language understanding unit with visual effects 104, a response text generation unit 105, a parameter learning unit 106, a related feature region determination unit 108, and a parameter storage unit 107. The second embodiment differs from the first embodiment mainly in that the question answering device 10 includes a related feature region determination unit 108.

[0106] The related feature region determination unit 108 is implemented using a neural network and uses the model parameters being trained stored in the parameter storage unit 107 to calculate the probability that the feature region extracted by the feature region extraction unit 101 is information necessary to answer the question. Therefore, the model parameters being trained stored in the parameter storage unit 107 also include the model parameters being trained for the neural network model that implements the related feature region determination unit 108.

[0107] Furthermore, the parameter learning unit 106 calculates the loss using the probability calculated by the related feature region determination unit 108 and the set of correct feature regions, and updates the model parameters being learned stored in the parameter storage unit 107.

[0108] <Learning Process> Next, the learning process according to this embodiment will be described. The overall flow of the learning process is the same as the learning process described in Figure 2, so the details of the model parameter update process in step S103 of Figure 2 will be described below. However, the number of epochs N e and the maximum number of training data points N included in a mini-batch b This may differ from the first embodiment. For example, the maximum number of training data points N included in a minibatch. b N b You can also use values ​​like =32.

[0109] ≪Model parameter update process≫ The details of the model parameter update process in step S103 of Figure 2 will be explained with reference to Figure 11. Figure 11 is a flowchart of an example of the model parameter update process according to the second embodiment. In the following, the model parameter update process for a certain minibatch will be described.

[0110] First, the parameter learning unit 106 reads one training data item from the minibatch (step S701).

[0111] Next, the feature region extraction unit 101 extracts K feature regions from the images included in the loaded training data (step S702). In this embodiment, as in the first embodiment, feature regions are represented as rectangular regions, and the k-th feature region has positional information including the top-left and bottom-right coordinates (4 dimensions in total), a rectangular image representation (D dimensions), and a region type (C types). However, any information can be used for the positional information as long as it can identify the position of the feature region, and information on the rectangular image representation or region type may be omitted. In addition, as in the first embodiment, for example, if a feature region is a polygon (polygonal region), the rectangular region surrounding this polygon may be used as the feature region.

[0112] Furthermore, the same nine types of region types as in the first embodiment will be used. However, these nine types of region types are merely examples, and it goes without saying that other region types may be set. In this embodiment as well, as in the first embodiment, it is sufficient to set at least two types of region types: one that indicates that the feature region does not contain text, and another that indicates that the feature region contains text.

[0113] For feature region extraction, as in the first embodiment, for example, Faster R-CNN described in Reference 1 above may be used. In this embodiment, for example, D=2048 is used.

[0114] Next, the text recognition unit 102 performs text recognition on the feature regions of the region type that indicate the presence of text, selected from the feature regions extracted in step S702, and outputs a sequence of word regions consisting of word regions that contain the words resulting from the text recognition (step S703). From here on, each word region is assumed to be a rectangular region and to have positional information (4 dimensions in total) including the top-left and bottom-right coordinates of the word region, and the word obtained by text recognition. Text recognition can be performed using, for example, Tesseract as described in Reference 2 above, as in the first embodiment. Note that a word region is a subregion of a feature region that contains the word resulting from the text recognition.

[0115] Next, the feature region extraction unit 101 outputs a rectangular image representation (D-dimensional) of each word region obtained in step S703 (step S704). This rectangular image representation can be output in the same way as when the rectangular image representation of the feature region was obtained in step S702. As a result, each word region will have positional information (4 dimensions in total) including the top-left and bottom-right coordinates of the word region, the word obtained by text recognition, and a rectangular image representation (D-dimensional) of the word region.

[0116] Next, the text analysis unit 103 divides the word region sequence obtained in step S704 into subword token sequences (step S705). From here on, the subword token sequences obtained by dividing the word region sequence from a certain k-th feature region are used.

[0117]

number

[0118] Furthermore, if a word contained within a single word region is divided into multiple subwords, the word region of each subword will be considered the same as the word region of the original word.

[0119] Next, the text analysis unit 103 analyzes the question text contained in the loaded training data into a subword token sequence (x1 q , x2 q ,···,x J q Divide it into (step S706). J is the number of subword tokens in the question text.

[0120] Next, the question answering device 10 performs language understanding processing with visual effects to obtain an encoded sequence that takes visual information into account (step S707). The details of the language understanding processing with visual effects will now be explained with reference to Figure 12. Figure 12 is a flowchart showing an example of the language understanding processing with visual effects according to the second embodiment.

[0121] First, the visual effects language understanding unit 104 uses the subword token sequence of the word region sequence and the subword token sequence of the question text to determine the following input token sequence

[0122]

number

[0123] Note that if the k-th feature region does not contain text, the length of the subword token sequence of the word region sequence obtained from the k-th feature region is 0 (i.e., L). k = 0)

[0124] Hereafter, as in the first embodiment, the length of the input token sequence will be L. If the length of the input token sequence exceeds L, the length of the input token sequence L will be reduced to a predetermined length by removing the longest text from each feature region, or by removing all texts equally. On the other hand, if the length of the input token sequence L is less than the predetermined length, it can be padded with special tokens.

[0125] Next, the visual language understanding unit 104 encodes each token (subword token) in the input token sequence (step S802). In this embodiment, the visual language understanding unit 104 encodes each token x as follows.

[0126] h=LayerNorm(TokenEmb(x)+PositionEmb(x)+SegmentEmb(x)+ROIEmb(x)+LocationEmb(x)) TokenEmb is a process that converts subword tokens (including special tokens) into corresponding G-dimensional vectors. In this embodiment, as in the first embodiment, the embedding vector (G=1024) pre-trained according to Reference 5 is used as the initial value and as the model parameters to be trained. Parameters of pre-trained language models other than those in Reference 5 may also be used as the training targets. However, special tokens that have not been trained are initialized with random numbers following a normal distribution N(0,0.02).

[0127] PositionEmb is a process that converts the input token sequence into a G-dimensional vector based on the position of the subword tokens. In this embodiment, the embedding vector (G=1024) learned according to Reference 5 is used as the initial value and as the model parameter to be learned. However, as in the first embodiment, the conversion to a G-dimensional vector may be done using the method described in Reference 6.

[0128] SegmentEmb is a process that converts subword tokens into G-dimensional vectors according to the segment to which they belong. In this embodiment, a total of 10 types of segments are used: 9 types of region types and 10 types of questions. Then, an embedding vector (G=1024) is prepared for each segment, initialized with random numbers following a normal distribution N(0,0.02), and used as the model parameters to be trained.

[0129] ROIEmb is a process that converts a rectangular image representation corresponding to a subword token into a G-dimensional vector. The rectangular image representation is a D-dimensional vector obtained by inputting a certain rectangular region in the input image into a neural network that realizes the feature region extraction unit 101. The subword token corresponds to region token i k If this is the case, it refers to the rectangular image representation of the k-th feature region, and the subword token is the document token x j k If this is the case, it refers to the rectangular image representation of the i-th word region obtained from the k-th feature region. On the other hand, if the subword token is document token x j k If this is the case, the output of ROIEmb is assumed to be a G-dimensional zero vector. In this embodiment, with D=2048, ROIEmb is converted into a G-dimensional (G=1024) vector by a feedforward network consisting of fully connected layers. In this embodiment, the feedforward network consists of one fully connected layer, and its parameters are initialized with random numbers following a normal distribution N(0,0.02) to be used as the model parameters to be trained.

[0130] LocationEmb is a process that converts the location information of a region (feature region or word region) corresponding to a subword token (however, either a region token or a document token) into a G-dimensional (G=1024) vector using a feedforward network composed of fully connected layers. LocationEmb normalizes the x-coordinate of the location information of the region by dividing it by the width of the input image, and the y-coordinate of the location information of the region by dividing it by the height of the image, before inputting it into the feedforward network. In this embodiment, the feedforward network consists of one fully connected layer, and its parameters are initialized with random numbers following a normal distribution N(0,0.02) to be used as the model parameters to be trained. Note that if the subword token is anything other than a region token or a document token, the output of LocationEmb is assumed to be a G-dimensional zero vector.

[0131] Similar to the first embodiment, LayerNorm takes a G-dimensional vector as input and outputs a G-dimensional vector using the normalization method described in Reference 7 above.

[0132] Therefore, the encoded r-th subword token in the input token sequence is h r If expressed as follows, the coded sequence H=(h1,h2,···,h L ) is obtained. Note that each h r Since is a G-dimensional vector, H is a vector sequence.

[0133] Next, the visual language understanding unit 104 converts the encoded sequence H obtained in step S802 into a vector sequence H' using the M-layer TransformerEncoder (step S803). That is, the visual language understanding unit 104 sets H' = TransformerEncoder(H). For more information on the TransformerEncoder, please refer to, for example, Reference 5 mentioned above. In this embodiment, M=12, and the TransformerEncoder trained according to Reference 5 is used as the initial value, and the parameters of this TransformerEncoder are used as the model parameters to be trained.

[0134] Next, the related feature region determination unit 108 calculates a probability indicating whether or not a feature region is a region necessary for answer generation (step S804). That is, if h' is the element of H' corresponding to a subword token x in the input token sequence (however, either a region token or a document token), the related feature region determination unit 108 calculates the probability that the feature region corresponding to the subword token x is necessary for the correct answer as follows.

[0135] p=sigmoid(w1 τ h'+b1) Here, w1∈R G b1∈R are the model parameters to be trained, and τ represents the transpose.

[0136] Next, the related feature region determination unit 108 converts the vector sequence H' to the vector sequence H'' using the probability obtained in step S804 (step S805). That is, the related feature region determination unit 108 converts h r ''=h r 'a r This converts the vector sequence H' to the vector sequence H''. Here, h r '' is the r-th element of the vector sequence H'', h r ' is the r-th element of the vector sequence H'. Also, a r is the weight, and its value is determined by the value of the r-th subword token in the input token sequence relative to the region token i kor document token x j k If that is the case, then p k、 Otherwise, use 1.0. k is area token i k This is the probability calculated in step S804 above.

[0137] Returning to Figure 11, following step S707, the question answering device 10 performs a process to calculate the probability of generating the answer text and calculates a probability distribution representing the probability of generating the answer text (step S708). Here, the details of the process to calculate the probability of generating the answer text will be explained with reference to Figure 13. Figure 13 is a flowchart of an example of the process to calculate the probability of generating the answer text according to the second embodiment.

[0138] First, the text analysis unit 103, similar to step S401 in Figure 6, analyzes the correct answer text contained in the loaded training data into a series of correct answer tokens.

[0139]

number

[0140] Next, the visual effects language understanding unit 104 initializes t to 0, with t being the index representing the number of repetitions (step S902). The processing at a certain t-th repetition will be described below.

[0141] The language understanding unit with visual effects 104 processes the decoder input token sequence y as follows: <t Create (step S903).

[0142] y <t=([CLS],y1 * ,···,y t-1 * )=(y0,y1,···,y t-1 ) However, when t=0, y <t Let =([CLS]). Also, the final step is t=L T When it's +1, y t Let =[EOS]

[0143] Next, the language understanding unit with visual effects 104 processes the decoder input token sequence y <t Each subword token y contained in is encoded as follows (step S904).

[0144] h y =LayerNorm(TokenEmb(y)+PositionEmb(y)) This results in the subword token y t The encoded form of h t y Therefore, the coded sequence H y =( h0 y ,h1 y ,···,h t-1 y ) can be obtained.

[0145] Next, the visual effects language understanding unit 104 processes the coded sequence H obtained in step S904 above. y H y Convert to ' (step S905). That is, the visual effects language understanding unit 104 converts to H y '=TransformerDecoder(H y Let H''). This means that H y '=(h0 y ',h1 y ',···,h t-1 y ') is obtained.

[0146] For more information on TransformerDecoder, please refer to, for example, Reference 5 mentioned above. In this embodiment, M=12, a TransformerDecoder trained according to Reference 5 is used as the initial value, and the parameters of this TransformerDecoder are used as the model parameters to be trained. Alternatively, parameters of a pre-trained language model other than those in Reference 5 may be used as the training targets.

[0147] Next, the response text generation unit 105 generates the probability distribution p(y) of the t-th word. t |y <t ) is calculated (step S906). The word y in the pre-set output vocabulary (vocabulary size V) t The probability distribution of is p(y t |y <t ) = softmax(Wh t-1 y It is calculated by '+b). Here, W∈R V×G , b∈R V These are the model parameters to be learned. Note that the vocabulary size V can be set to any value, but for example, 50257 is a possible value.

[0148] Next, the language understanding unit with visual effects 104 determines t=L T Determine whether it is +1 or not (step S907).

[0149] In step S907 above, t=L T If it is determined that the value is not +1, the visual effects language understanding unit 104 adds 1 to t (step S908) and returns to step S903. As a result, steps S903 to S906 above are t=0,1,···,L T This process is repeated for each +1.

[0150] Return to Figure 11. Following step S708, the parameter learning unit 106 uses the probability distribution p(y) calculated in step S906 of Figure 13. t |y <t) and the set of correct feature regions included in the loaded training data are used to calculate the loss (step S709). The parameter learning unit 106 can calculate the loss, for example, as follows.

[0151]

number

[0152] The following steps S710 to S712 are the same as steps S209 to S211 in Figure 3, so their explanation will be omitted.

[0153] As described above, in the question answering device 10 according to this embodiment, when an image containing text, a question text related to this image, and a set of correct answer feature regions are given, the model parameters are learned so that a response text that takes into account the visual information in the image is generated. In other words, the model parameters are learned so that machine reading comprehension that takes visual information into account is possible.

[0154] [At the time of inference] Next, we will explain the inference process. During the inference process, the question answering device 10 receives test data that includes an image containing text and a question text related to this image.

[0155] <Overall configuration of the question answering device 10 during inference> The overall configuration of the question answering device 10 during inference will be explained with reference to Figure 14. Figure 14 is a diagram showing an example of the overall configuration (during inference) of the question answering device 10 according to the second embodiment.

[0156] As shown in Figure 14, the question answering device 10 during inference includes a feature region extraction unit 101, a text recognition unit 102, a text analysis unit 103, a language understanding unit with visual effects 104, a response text generation unit 105, a related feature region determination unit 108, and a parameter storage unit 107. Of these units, the feature region extraction unit 101, the text recognition unit 102, and the text analysis unit 103 are the same as during training. On the other hand, the language understanding unit with visual effects 104, the response text generation unit 105, and the related feature region determination unit 108 use trained model parameters stored in the parameter storage unit 107. The response text generation unit 105 generates the response text using a probability distribution calculated from the coded sequence obtained by the language understanding unit with visual effects 104. The related feature region determination unit 108 may output a score (related feature region score) calculated or determined from the probability of whether the feature region extracted by the feature region extraction unit 101 is information necessary to answer the question.

[0157] <Inference Processing> Next, the inference process according to this embodiment will be described with reference to Figure 15. Figure 15 is a flowchart showing an example of the inference process according to the second embodiment. Hereafter, it will be assumed that the test data provided to the question answering device 10 has been read.

[0158] First, the feature region extraction unit 101 extracts K feature regions from the image contained in the loaded test data, similar to step S702 in Figure 11 (step S1001).

[0159] Next, the text recognition unit 102 performs text recognition on the feature regions of the region type that indicate the presence of text, from among the feature regions extracted in step S1001, similar to step S703 in Figure 11, and outputs a word region sequence (step S1002).

[0160] Next, the feature region extraction unit 101 outputs a rectangular image representation (D-dimensional) of each word region obtained in step S1002, similar to step S704 in Figure 11 (step S1003). This provides a sequence of word regions having positional information (4 dimensions in total) including the top-left and bottom-right coordinates, the words obtained by text recognition, and the rectangular image representation (D-dimensional).

[0161] Next, the text analysis unit 103 divides the word region sequence obtained in step S1003 into subword token sequences, similar to step S705 in Figure 11 (step S1004).

[0162] Next, the text analysis unit 103, similar to step S706 in Figure 11, analyzes the question text contained in the loaded test data into a subword token sequence (x1 q , x2 q ,···,x J q Divide into (step S1005).

[0163] Next, the question answering device 10 performs language understanding processing with visual effects to obtain an encoded sequence that takes visual information into account (step S1006). Since the language understanding processing with visual effects is the same as step S707 in Figure 11, its explanation is omitted. From here on, the explanation will continue assuming that the vector sequence H'' has been obtained.

[0164] Next, the question answering device 10 executes a response text generation process to generate response text (step S1007). The details of the response text generation process will now be explained with reference to Figure 16. Figure 16 is a flowchart showing an example of the response text generation process according to the second embodiment.

[0165] First, the visual effects language understanding unit 104 initializes the index t, which represents the number of repetitions, to 0 (step S1101). The following describes the processing at a certain t-th repetition.

[0166] The language understanding unit with visual effects 104 processes the decoder input token sequence y <t =([CLS]) is initialized (step S1102). That is, the language understanding unit with visual effects 104 initializes the decoder input token sequence y at t=0. <t This is defined as a series that contains only [CLS].

[0167] The following section will explain the processing that occurs during a certain t-th iteration.

[0168] The language understanding unit with visual effects 104, similar to step S904 in Figure 13, performs the decoder input token sequence y <t Each subword token y contained in is encoded as follows (step S1103).

[0169] h y =LayerNorm(TokenEmb(y)+PositionEmb(y)) This results in the subword token y t The encoded form of h t y Therefore, the coded sequence H y =( h0 y ,h1 y ,···,h t-1 y ) can be obtained.

[0170] Next, the visual effects language understanding unit 104, similar to step S905 in Figure 13, performs the coding sequence H obtained in step S1103 above. y H y Convert to ' (step S1104). That is, the visual effects language understanding unit 104 converts to H y '=TransformerDecoder(H y Let H''). This means that H y '=(h0 y ',h1 y ',···,h t-1 y ') is obtained.

[0171] Next, the response text generation unit 105 generates the t-th word using a probability distribution p(y) similar to step S906 in Figure 13. t |y <t ) is calculated (step S1105). The word y in the pre-set output vocabulary (vocabulary size V) t The probability distribution of is p(y t |y <t ) = softmax(Wh t-1 y It is calculated by '+b). Here, W∈R V×G , b∈R V These are the trained model parameters.

[0172] Next, the response text generation unit 105 generates the probability distribution p(y) calculated in step S1105 above. t |y <t Based on this, the t-th word is generated (step S1106). The response text generation unit 105 may generate the word with the highest probability as the t-th word, or it may generate the t-th word by sampling according to a probability distribution.

[0173] Next, the response text generation unit 105 uses the t-th word generated in step S1106 above as the decoder input token sequence y <t It is appended to the end (step S1107).

[0174] Next, the visual effects language understanding unit 104 determines whether the t-th word generated in step S1106 is the final word (step S1108).

[0175] If step S1108 determines that the t-th word is not the last word, the visual effects language understanding unit 104 adds 1 to t (step S1109) and returns to step S1103. As a result, steps S1103 to S1107 are repeatedly executed to obtain a word sequence.

[0176] As described above, the question answering device 10 according to this embodiment can generate a response text (word sequence) that takes into account the visual information in the image when given an image containing text and a question text related to the image.

[0177] [Evaluation of this embodiment] Next, we will discuss the evaluation of considering whether a feature region contains information necessary to answer a question.

[0178] To evaluate this embodiment, a performance comparison was performed with a baseline. The models used in this embodiment were a model using BART as a pre-trained model, as described in Reference 5, and a model using T5 as a pre-trained model, as described in Reference 8, "Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; and Liu, PJ 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. J. Mach. Learn. Res. 21(140): 1-67." Hereinafter, the model using BART will be referred to as "LayoutBART," and the model using T5 will be referred to as "LayoutT5." Furthermore, models using LARGE as BERT will be specifically referred to as "LayoutBART." LARGE "LayoutT5" LARGE It is written as "".

[0179] Furthermore, the baseline adopted is a model called M4C, described in Reference 9, "Hu, R.; Singh, A.; Darrell, T.; and Rohrbach, M. 2020. Iterative Answer Prediction with Pointer-Augmented Multi-modal Transformers for TextVQA. In CVPR, 9992-10002." M4C is a model that generates answers to a question using the question text, feature regions, and OCR tokens (corresponding to document tokens in this embodiment) as input, and has been confirmed to be capable of achieving high performance.

[0180] Five evaluation metrics were used: BLEU, METEOR, ROUGE-L, CIDEr, and BERTscore. The model was trained using a pre-prepared experimental training dataset, and then the four evaluation metrics were calculated using the test data. The results are shown in Table 1 below.

[0181] [Table 1] As shown in Table 1 above, LayoutBART and LayoutT5 achieve higher performance than M4C in all evaluation metrics. Furthermore, as shown in Table 1 above, using LARGE as BERT achieves higher performance than BASE. In conclusion, the method of this embodiment can achieve higher performance than conventional methods in the task of generating response text when given an image containing text and a question text.

[0182] <Hardware Configuration> Finally, the hardware configuration of the question answering device 10 according to the first and second embodiments will be described with reference to Figure 17. Figure 17 is a diagram showing an example of the hardware configuration of the question answering device 10 according to one embodiment.

[0183] As shown in FIG. 17, the question-and-answer device 10 according to one embodiment is realized by a general computer or computer system, and includes an input device 201, a display device 202, an external I / F 203, a communication I / F 204, a processor 205, and a memory device 206. These hardware components are communicably connected to each other via a bus 207.

[0184] The input device 201 is, for example, a keyboard, a mouse, a touch panel, or the like. The display device 202 is, for example, a display or the like. Note that the question-and-answer device 10 may not have at least one of the input device 201 and the display device 202.

[0185] The external I / F 203 is an interface with an external device. Examples of the external device include a recording medium 203a. The question-and-answer device 10 can read from and write to the recording medium 203a via the external I / F 203. The recording medium 203a may store one or more programs that implement each functional unit (feature area extraction unit 101, text recognition unit 102, text analysis unit 103, visual effect-added language understanding unit 104, answer text generation unit 105, parameter learning unit 106, and related feature area determination unit 108) of the question-and-answer device 10.

[0186] Examples of the recording medium 203a include a CD (Compact Disc), a DVD (Digital Versatile Disk), an SD memory card (Secure Digital memory card), a USB (Universal Serial Bus) memory card, and the like.

[0187] The communication I / F 204 is an interface for connecting the question-and-answer device 10 to a communication network. One or more programs that implement each functional unit of the question-and-answer device 10 may be acquired (downloaded) from a predetermined server device or the like via the communication I / F 204.

[0188] The processor 205 is, for example, a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), among other types of computing devices. Each functional unit of the question answering device 10 is realized, for example, by one or more programs stored in the memory device 206 causing the processor 205 to execute.

[0189] The memory device 206 is, for example, one of various storage devices such as an HDD (Hard Disk Drive), SSD (Solid State Drive), RAM (Random Access Memory), ROM (Read Only Memory), or flash memory. The parameter storage unit 107 of the question answering device 10 can be implemented using, for example, the memory device 206. Alternatively, the parameter storage unit 107 may be implemented using a storage device (for example, a database server) connected to the question answering device 10 via a communication network.

[0190] The question answering device 10 according to the first and second embodiments can realize the learning and inference processes described above by having the hardware configuration shown in Figure 17. Note that the hardware configuration shown in Figure 17 is just an example, and the question answering device 10 may have other hardware configurations. For example, the question answering device 10 may have multiple processors 205 or multiple memory devices 206.

[0191] The following additional information is disclosed regarding the embodiments described above.

[0192] (Note 1) Memory and At least one processor connected to the memory, Includes, Using data containing a visual region and first information related to the data as input, a machine learning model's model parameters are used to generate second information corresponding to the first information from information representing the features of the region. A learning device that learns the model parameters based on the second piece of information and a third piece of information representing the correct answer to the second piece of information. (Note 2) The aforementioned processor, A learning device as described in Appendix 1, which creates feature quantities from information representing the characteristics of the aforementioned region and the first information, and generates second information from the feature quantities. (Note 3) The learning device according to Appendix 1 or 2, wherein the region includes at least an image or a diagram. (Note 4) The learning device described in any one of the appendices 1 to 3, wherein the first information is text information representing content related to the data. (Note 5) Memory and At least one processor connected to the memory, Includes, Using data containing a visual region and first information related to the data as input, the degree of association between the region and second information corresponding to the first information is calculated using the model parameters of a machine learning model. A learning device that learns the model parameters based on the aforementioned relevance and information representing the correct answer for the aforementioned relevance. (Note 6) Memory and At least one processor connected to the memory, Includes, A generation device that takes data containing a visual region and first information related to the data as input, and uses model parameters of a trained machine learning model to generate second information corresponding to the first information from information representing the features of the region. (Note 7) Memory and At least one processor connected to the memory, Includes, An output device that takes as input data including a visual area and first information related to the data, and outputs a predetermined evaluation value based on the degree of association between the area and second information corresponding to the first information, using model parameters of a learned machine learning model. (Appendix 8) A non-transitory storage medium storing a program executable by a computer to execute learning processing, The learning processing includes: Taking as input data including a visual area and first information related to the data, and generating second information corresponding to the first information from information representing the characteristics of the area, using model parameters of a machine learning model. A non-transitory storage medium that learns the model parameters based on the second information and third information representing the correct answer of the second information. (Appendix 9) A non-transitory storage medium storing a program executable by a computer to execute generation processing, The generation processing includes: Taking as input data including a visual area and first information related to the data, and generating second information corresponding to the first information from information representing the characteristics of the area, using model parameters of a learned machine learning model.

[0193] The present invention is not limited to the specifically disclosed above embodiments, and various modifications, changes, combinations with known technologies, etc. are possible without departing from the description of the claims.

[0194] This application is based on a basic application PCT / JP2020 / 008390 filed in Japan on February 28, 2020, the entire contents of which are incorporated herein by reference.

Explanation of Reference Numerals

[0195] 10 Question-and-Answer Device 101 Feature Region Extraction Unit 102 Text Recognition Unit 103 Text Analysis Department 104 Visual Effects-Equipped Language Comprehension Unit 105 Answer Text Generation Unit 106 Parameter Learning Unit 107 Parameter Storage Unit 108 Related Feature Region Determination Unit

Claims

1. A generation unit that, upon receiving data containing text and first information related to the data, generates second information using the model parameters of a trained machine learning model. It has, The generating unit is A generation system characterized by, when data containing the aforementioned text is input, generating image tokens which are feature quantities containing visual information from the data, and generating the second information based on a token sequence containing the generated image tokens.

2. A generation unit that, when data containing text and first information related to the data are input, generates second information using the model parameters of a trained machine learning model, It has, The generating unit is A generation system characterized by, when data containing the aforementioned text is input, generating at least two tokens of different types from the data, and generating the second information based on a token sequence including the generated tokens.

3. The generating unit is If the aforementioned data contains text, a text token is generated. An image token is generated for the aforementioned data, The generation system according to claim 2, characterized in that it generates the second information based on a token sequence including the generated image tokens.

4. The generation system according to claim 1 or 2, characterized in that the second information is information based on evaluation values ​​relating to the region included in the data.

5. The generation system according to claim 1 or 2, characterized in that the second information is information based on the degree to which each of the multiple regions included in the data contributes to generating a response to the first information.

6. The generation system according to claim 4, characterized in that the evaluation value is generated based on the degree of correlation with the second information corresponding to the first information.

7. The generation system according to claim 1, wherein the data containing the text is an image.

8. The generating unit is The generation system according to claim 1, which extracts the image token from the region extracted from the aforementioned data.

9. A generation unit that, when data containing text and first information related to the said data are input, generates second information using the model parameters of a machine learning model, A learning unit that learns the model parameters based on the second information and the third information corresponding to the second information, It has, The generating unit is A learning system characterized in that, when data containing the aforementioned text is input, it generates image tokens, which are feature quantities containing visual information, from the data, and generates the second information based on a token sequence containing the generated image tokens.

10. A generation unit that, when data containing text and first information related to the data are input, generates second information using model parameters of a machine learning model, A learning unit that learns the model parameters based on the second information and the third information corresponding to the second information, It has, The generating unit is A learning system characterized by, when data containing the aforementioned text is input, generating at least two tokens of different types from the data, and generating the second information based on a token sequence including the generated tokens.

11. A generation procedure that, when data containing text and first information related to the said data are input, generates second information using the model parameters of a trained machine learning model. The computer executes this, The aforementioned generation procedure is: A generation method characterized by, when data containing the aforementioned text is input, generating image tokens which are feature quantities containing visual information from the data, and generating the second information based on a token sequence including the generated image tokens.

12. A generation procedure for generating second information using model parameters of a trained machine learning model when data containing text and first information related to the data are input, The computer executes this, The aforementioned generation procedure is: A generation method characterized by, when data containing the aforementioned text is input, generating at least two tokens of different types from the data, and generating the second information based on a token sequence including the generated tokens.

13. A generation procedure for generating second information using model parameters of a machine learning model when data containing text and first information related to said data are input, A learning procedure for learning the model parameters based on the second piece of information and a third piece of information corresponding to the second piece of information, The computer executes this, The aforementioned generation procedure is: A learning method characterized by, when data containing the aforementioned text is input, generating image tokens which are feature quantities containing visual information from the data, and generating the second information based on a token sequence containing the generated image tokens.

14. A generation procedure for generating second information using model parameters of a machine learning model when data containing text and first information related to the data are input, A learning procedure for learning the model parameters based on the second piece of information and a third piece of information corresponding to the second piece of information, The computer executes this, The aforementioned generation procedure is: A learning method characterized by, when data containing the aforementioned text is input, generating at least two tokens of different types from the data, and generating the second information based on a token sequence including the generated tokens.

15. A program for causing a computer to perform the generation method described in claim 11 or 12, or the learning method described in claim 13 or 14.

Images