Answer generation device, machine learning method, and program

Abstract

To generate more accurate answer information by considering a relationship of arrangement between an object in a document image and a text.SOLUTION: An answer generation apparatus 3a generates answer information to a question sentence based on the content of a document image. The answer generation apparatus 3b performs learning on a relationship of arrangement between an object in the document image and a text, by an arrangement relationship prediction unit 37 and a parameter learning unit 38b, in addition to machine learning for generating answer information. Accordingly, answer information can be generated in consideration of the arrangement relationship.SELECTED DRAWING: Figure 10

Description

[Technical Field]

[0001] The present disclosure relates to a technology that receives question text indicating a question sentence and a document image including the text as input and generates answer information to the question based on the content of the document image. [Background technology]

[0002] There are question-answering technologies that use question text indicating a question and a document image containing the text as input, and generate answer text to the question based on the contents of the document image (Non-Patent Document 1, Non-Patent Document 2). In these conventional systems, training data is a triplet of {document image, question, correct answer information}. The conventional systems first extract text and document regions (title, paragraph, caption, etc.) from the document image, and acquire visual information such as text, coordinates of the text region in the document, and icons. Next, the conventional systems use the information acquired from the document image and the given question as input, and train to generate an answer. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Powalski Rafal, Borchmann Lukasz, Jurkiewicz Dawid, Dwojak Tomasz, Pietruszka Michal, Palka Gabriela: Going Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformer. ICDAR21 [Non-patent document 2] Ryota Tanaka, Kyosuke Nishida, Sen Yoshida: VisualMRC: Machine Reading Comprehension on Document Images. AAAI21 Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional techniques do not learn how objects and text are arranged within document images, and there remain challenges in understanding document images that include not only text but also objects.

[0005] The present invention has been made in view of the above points, and has an object to generate more accurate answer information by taking into consideration the positional relationship between objects and text in a document image. [Means for solving the problem]

[0006] In order to solve the above problem, the invention of claim 1 is an answer generation device that generates answer information to a question sentence based on the content of a document image, the answer generation device including an encoding unit that outputs encoded features based on text information in the document image, object information indicating an object in the document image, and the question sentence, a decoding unit that outputs output probability distribution information of the answer information based on the encoded features, and a decoding unit that derives first rectangular selection area information indicating a rectangular area of ​​a predetermined document text selected from rectangular area information indicating rectangular areas of a plurality of document texts in the document image, and a decoding unit that indicates a rectangular area of ​​a predetermined object selected from rectangular area information indicating rectangular areas of objects in the document image. a correct layout relation derivation unit that derives second rectangular selection area information that indicates the layout relationship of the specified document text with respect to the specified object, and derives correct layout relation information that indicates the layout relationship of the specified document text with respect to the specified object; a layout relation prediction unit that predicts the layout relationship between the specified document text related to the first rectangular selection area information and the specified object related to the second rectangular selection area information based on the encoded features, and outputs predicted layout relation distribution information; and a parameter learning unit that learns model parameters of a neural network based on output probability distribution information of the answer information, correct answer information, the predicted layout relation distribution information, and the correct layout relation information. [Effects of the Invention]

[0007] As described above, according to the present invention, the performance of understanding the positional relationship between an object and text is improved, and more accurate answer information can be generated. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating the overall configuration of a communication system according to an embodiment of the present invention. [Figure 2] 1 is a diagram illustrating an electrical hardware configuration of an answer generation device according to an embodiment of the present invention. [Figure 3] FIG. 2 is a diagram illustrating an electrical hardware configuration of a communication terminal according to the present embodiment. [Figure 4] FIG. 2 is a functional configuration diagram of the answer generation device in a learning phase according to the first embodiment. [Figure 5] 10A and 10B are diagrams illustrating examples of a question sentence, a document image, and correct answer information. [Figure 6] FIG. 3 is a detailed configuration diagram of a stream conversion unit according to the first embodiment. [Figure 7] FIG. 2 is a functional configuration diagram of the answer generation device in the inference phase according to the first embodiment. [Figure 8] 5 is a flowchart showing the processing or operation executed by the answer generation device in the learning phase according to the first embodiment. [Figure 9] 5 is a flowchart showing the processing or operation executed by the answer generation device in the inference phase according to the first embodiment. [Figure 10] FIG. 10 is a functional configuration diagram of the answer generation device in the learning phase according to the second embodiment. [Figure 11] FIG. 10 is a conceptual diagram of the process of a correct layout relationship deriving unit according to the second embodiment. [Figure 12] FIG. 10 is a detailed configuration diagram of a stream conversion unit according to the second embodiment. [Figure 13] FIG. 10 is a functional configuration diagram of the answer generation device in the inference phase according to the second embodiment. [Figure 14]10 is a flowchart showing the processing or operation executed by the answer generation device in the learning phase according to the second embodiment. [Figure 15] 10 is a flowchart showing the processing or operation executed by the answer generation device in the learning phase according to the second embodiment. [Figure 16] 4 is a table showing experimental results according to the first and second embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0010] First embodiment First, a first embodiment of the present invention will be described with reference to Figures 1 to 9. Note that the answer generation device 3a according to the first embodiment provides a specific improvement over the conventional technology that receives question text indicating a question sentence and a document image including the text as input and generates answer information to a question based on the contents of the document image, as described above, and represents an advancement in the technical field related to generation of answer information using a neural network.

[0011] [System configuration of the embodiment] First, the overall configuration of the communication system of this embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing the overall configuration of the communication system according to this embodiment.

[0012] As shown in Fig. 1, a communication system 1 of this embodiment is constructed by an answer generation device 3 and a communication terminal 5. The communication terminal 5 is managed and used by a user. The user refers to the output result of the answer generation device and determines how to respond thereafter.

[0013] The answer generation device 3 and the communication terminal 5 can communicate with each other via a communication network 100 such as the Internet. The communication network 100 may be connected wirelessly or by wire.

[0014] The answer generation device 3 is configured by one or more computers. When the answer generation device 3 is configured by multiple computers, it may be referred to as an "answer generation device" or an "answer generation system."

[0015] As shown in Fig. 5, even if the text image does not directly contain correct answer information relating to numerical information (e.g., "40%), the answer generation device 3 automatically generates arithmetic operation process information indicating the process of arithmetic operation using a template such as "100%-*%=" (* is a variable), thereby improving the ability to understand numerical information. Note that, for example, all numerical values ​​extracted from the text image are substituted for "*" in the template.

[0016] The communication terminal 5 is a computer, and a laptop computer is shown as an example in Fig. 1. In Fig. 1, a user operates the communication terminal 5. Note that the answer generation device 3 may perform processing independently without using the communication terminal 5.

[0017] [Hardware configuration] <Hardware configuration of the answer generation device> Next, the electrical hardware configuration of the answer generation device 3 will be described with reference to Fig. 2. Fig. 2 is a diagram showing the electrical hardware configuration of the answer generation device.

[0018] As shown in FIG. 2, the answer generation device 3 is a computer that includes a processor such as a CPU (Central Processing Unit) 301, a ROM (Read Only Memory) 302, a RAM (Random Access Memory) 303, an SSD (Solid State Drive) 304, an external device connection I / F (Interface) 305, a network I / F 306, a media I / F 309, and a bus line 310.

[0019] Of these, the CPU 301 controls the overall operation of the answer generation device 3. The ROM 302 stores programs such as an IPL (Initial Program Loader) used to drive the CPU 301. The RAM 303 is used as a work area for the CPU 301.

[0020] The SSD 304 reads or writes various data under the control of the CPU 301. Note that instead of the SSD 304, a hard disk drive (HDD) may be used.

[0021] The external device connection I / F 305 is an interface for connecting various external devices, such as a display, a speaker, a keyboard, a mouse, a USB (Universal Serial Bus) memory, and a printer.

[0022] The network I / F 306 is an interface for performing data communication via the communication network 100 .

[0023] The media I / F 309 controls reading and writing (storing) of data from and to a recording medium 309m such as a flash memory, etc. The recording medium 309m includes a DVD (Digital Versatile Disc) and a Blu-ray Disc (registered trademark).

[0024] The bus line 310 is an address bus, a data bus, etc. for electrically connecting the components such as the CPU 301 shown in FIG.

[0025] <Hardware configuration of communication terminal> Next, the electrical hardware configuration of the communication terminal 5 will be described with reference to Fig. 3. Fig. 3 is a diagram showing the electrical hardware configuration of the communication terminal.

[0026] As shown in FIG. 3, the communication terminal 5 is a computer and includes a CPU 501, a ROM 502, a RAM 503, an SSD 504, an external device connection I / F (Interface) 505, a network I / F 506, a display 507, a pointing device 508, a media I / F 509, and a bus line 510.

[0027] Of these, the CPU 501 controls the overall operation of the communication terminal 5. The ROM 502 stores programs such as IPL used to drive the CPU 501. The RAM 503 is used as a work area for the CPU 501.

[0028] The SSD 504 reads or writes various data under the control of the CPU 501. Note that instead of the SSD 504, an HDD (Hard Disk Drive) may be used.

[0029] The external device connection I / F 505 is an interface for connecting various external devices, such as a display, a speaker, a keyboard, a mouse, a USB memory, and a printer.

[0030] The network I / F 506 is an interface for performing data communication via the communication network 100 .

[0031] The display 507 is a type of display means such as a liquid crystal display or organic EL (Electro Luminescence) display that displays various images.

[0032] The pointing device 508 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. If the user uses a keyboard, the function of the pointing device 508 may be turned off.

[0033] The media I / F 509 controls reading and writing (storing) of data from and to a recording medium 509m such as a flash memory, etc. The recording medium 509m includes DVDs, Blu-ray Discs (registered trademarks), etc.

[0034] The bus line 510 is an address bus, a data bus, etc. for electrically connecting the components such as the CPU 501 shown in FIG.

[0035] [Functional configuration of the answer generation device] The functional configuration of the answer generation device 3a according to this embodiment in the learning phase and inference (prediction) phase will be described. Note that Fig. 4 shows the functional configuration of the learning phase, and Fig. 7 shows the functional configuration of the inference phase, but the answer generation device 3a may have all the units in Figs. 4 and 7 collectively.

[0036] <Learning phase functional configuration> First, each function of the answer generation device 3a in the learning phase will be described with reference to Fig. 4. Fig. 4 is a functional configuration diagram of the answer generation device in the learning phase according to the first embodiment. Note that the answer generation device 3a is an example of the answer generation device 3.

[0037] As shown in Fig. 4, the answer generation device 3a has an input unit 30, a text extraction unit 31a, an object extraction unit 32a, a data extension unit 33, an encoding unit 35a, a decoding unit 36a, and a parameter learning unit 38a. Each of these units has a function realized by an instruction from the CPU 301 in Fig. 2 based on a program. A machine learning model 41a is built in the RAM 303 or the SSD 304. The processing of the encoding unit and the decoding unit is based on the model parameters of a neural network.

[0038] The input unit 30 inputs learning data (data on the question sentence, correct answer information, and document image) to the answer generation device 3a via the communication terminal 7. Alternatively, the input unit 30 inputs learning data directly to the answer generation device 3a. FIG. 5 is a diagram showing an example of a question sentence, a document image, and correct answer information. For example, as shown in FIG. 5, the question sentence is "What is the percentage of cases where diabetes was prevented?" The document image is the image on the right side of FIG. 5. The correct answer information is "40%."

[0039] The text extraction unit 31a receives the document image input by the input unit 30 and outputs text information in the document image (information on the document text sequence and the document text rectangular area).

[0040] Therefore, the text extraction unit 31a detects text areas included in the document image, recognizes the text in the detected areas using OCR (Optical Character Recognition) or the like, and outputs text information from the document image. The text information from the document image includes at least a document text sequence and a document text rectangular area. The document text sequence output here is sorted left-to-right, top-to-down according to the text coordinates. The document text rectangular area is represented by, for example, the top-left and bottom-right coordinates of a rectangular area in the document image that indicates "60%." Any processing can be used as long as it can output a text sequence and a text rectangular area from the document image. In this embodiment, Tesseract, which is described in Reference 1, is used as an example of processing. <Reference 1> Google: Tesseract Manual. 2018.(https: / / github.com / tesseract-ocr / tesseract / blob / master / doc / tesseract.1.asc) The object extraction unit 32a receives the document image input by the input unit 30 and outputs object information in the document image (object region feature information, object region semantic label, and object rectangular region information).

[0041] Therefore, the object extraction unit 32a extracts object information contained in the document image and outputs it as object information in the document image. The object information in the document image includes area features v of the object (graph, icon, etc.). obj The object region information includes at least an object region semantic label, which is a label indicating the meaning of the object region, and object rectangular region information indicating the coordinates of the object region. Any processing can be used for these processes as long as the object region feature information, object region semantic label, and object rectangular region information can be acquired from the document image. In this embodiment, Faster-RCNN in Reference 2 is used as an example of the processing. <Reference 2>Shaoqing Ren, Kaiming He, Ross B. Girshick, Jian Sun: Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. NIPS 2015: 91-99 In this embodiment, the region feature v obj is a 2048-dimensional vector sequence, the object region semantic labels are 391 types of semantic labels (such as graph or dog) defined in Reference 3, and rectangular region information is expressed by the top left and bottom right coordinates of the region on the image information. <Reference 3> Spandan Mada, Zoya Bylinskii, Mathew Tanik, Adria Recasens, Kimberli Zhong, Sami Alesheikh, Hanspeter Pfister, Aude Oliva, Fredo Durand: Synthetically Trained Icon Proposals for Parsing and Summarizing Inforgraphics. arXiv:1807.10441 The data extension unit 33 receives the correct answer information input by the input unit 30 and the document text sequence extracted by the text extraction unit 31a, and outputs arithmetic operation process information indicating the arithmetic operation process.

[0042] Therefore, the data extension unit 33 outputs an arithmetic operation process (for example, "100%-60%=") using the correct answer information and the text in the document image.

[0043] The process of the data extension unit 33 is as follows.

[0044] S111: The data extension unit 33 extracts a predetermined number of pieces of numerical data from the document text sequence. In this embodiment, as an example, seven pieces of data are extracted. Numerical values ​​written in letters (one, two, etc.) are converted into numerical data by the data extension unit 33. In the example of FIG. 5, the numerical data extracted are "1.7, 1, 10, 4.3, 3.6, 6, 60."

[0045] S112: The data extension unit 33 stores in advance templates indicating three different types of arithmetic operation processes, for example, "A+B," "AB," and "100-A." Templates indicating arithmetic operation processes other than these can also be used. The data extension unit 33 takes arbitrary numerical values ​​A and B from the numerical data extracted in S111 and substitutes them into the template indicating the arithmetic operation process. In the example of FIG. 5, when A=60 and B=6, 66, 54, and 40 are output as the substitution results.

[0046] S113: The data extension unit 33 performs calculations on the predetermined template substituted in S112 above, and compares the calculation result (substitution result) with the correct answer information. If the comparison shows that the calculation result and the correct answer information have the same text (value), the template substituted in S112 above is identified as an arithmetic operation process, and arithmetic operation process information indicating this arithmetic operation process is generated and output. In the example of FIG. 5, since "40%", which is the substitution result in S2 above, matches the answer correct information, the arithmetic operation process information is "100%-60%".

[0047] The encoding unit 35a and decoding unit 36a constitute the sequence transformation unit 20a. FIG. 6 is a detailed block diagram of the sequence transformation unit according to the first embodiment. The sequence transformation unit 20a uses neural network model parameters to generate output sequence information based on input information. The model configuration employs a general encoder-decoder model format that generates a sequence using a sequence as input.

[0048] 6, the encoding unit 35a has a text information in document image receiving unit 351, an object information in document image receiving unit 352, and a question sentence receiving unit 353. These constitute the input layer of the neural network.

[0049] The text information in document image receiving unit 351 receives text information in the document image (text sequence, rectangular area information of the text). The object information in document image receiving unit 352 receives object information in the document image (object area features, object area semantic labels, rectangular area information of the object area). The question sentence receiving unit 353 receives a question sentence.

[0050] Furthermore, the encoding unit 35a includes a plurality of transformer layers 1 (Transformer) 3551 to 355L (Transformer). L A plurality of transformer layers 1 (Transformer) 3551 to 355L (Transformer) L The layer receives the above information from the input layer and performs the calculations described below in sequence.

[0051] The decoding unit 36a includes a plurality of transformer layers 1 (Transformer) 3651 to 365L (Transformer). L It has a plurality of transformer layers 1 (Transformer) 3651 to L (Transformer) 365 L is the transformer layer L (Transformer) 355 of the encoding unit 35a. LThe encoding unit 35a receives the encoded features from the input data and performs the calculations described below. The encoded features represent vectors of a predetermined dimension that are obtained by converting the information on the document image and question sentence input to the encoding unit 35a based on the model parameters of the neural network.

[0052] In addition, a plurality of transformer layers 1 (Transformer) 3551 to 355L (Transformer) L , and a plurality of transformer layers 1 (Transformer) 3651 to L (Transformer) 365 L is building the intermediate layer of a neural network.

[0053] Furthermore, the decoding unit 36a has an answer information output unit 357 and an arithmetic operation process information output unit 358. These constitute the output layer of the neural network. The answer information output unit 357 outputs output probability distribution information of the answer information obtained by the intermediate layer. The arithmetic operation process information output unit 358 outputs output probability distribution information of the arithmetic operation process obtained by the intermediate layer.

[0054] As described above, the sequence conversion unit 20a inputs the question sentence, text information in the document image, object information in the document image, answer information, and arithmetic operation process information, and outputs output sequence information. The answer information uses correct answer information in the learning phase, and answer information inferred by the text generation unit 39 in the inference phase. The arithmetic operation process information is arithmetic operation process information output from the data extension unit 33 in the learning phase, and arithmetic operation process information inferred by the text generation unit 39 in the inference phase. The output sequence information indicates output probability distribution information of the answer information and output probability distribution information of the arithmetic operation process. The encoding unit 35a and decoding unit 36a will be described in more detail below.

[0055] The encoding unit 35a receives the question, the text information in the document image extracted by the text extraction unit 31a, and the object information in the document image extracted by the object extraction unit 32a, and outputs encoded features. Specifically, the encoding unit 35a receives the question, the document text sequence, the document text rectangular region, the object rectangular region information, the object region semantic label, and the object rectangular region information.

[0056] Therefore, the encoding unit 35a encodes the text, objects, and questions in the document image. In this case, it is sufficient to perform general processing when inputting both image and language features and outputting the combined features. An example of the processing is as follows.

[0057] S121: The encoding unit 35a encodes the question sentence, the document text sequence, and the object region semantic label into a token sequence w, which is a predetermined processing unit. ocr , w q , w obj In this embodiment, as an example of the tokenization process, the Byte-level BPE shown in Reference 4 is used. Note that instead of this step, the encoding unit 35a may divide the token sequence w into a token sequence generated in advance by an external unit, for example, the text extraction unit 31a and the object extraction unit 32a, ocr , w q , w obj may be acquired.

[0058] Furthermore, the processing unit (token) may be anything obtained by dividing (tokenizing) a text into predetermined units. The token sequence resulting from tokenization is a sequence consisting of one or more tokens. The predetermined unit (token) may be, for example, a word or a subword. <Reference 4>Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever: Language models are unsupervised multitask learners. Technical report, OpenAI, 2019. S122: The encoding unit 35a converts the subword token sequence w ocr ,w q ,w obj and object region feature v obj The following input token sequence is prepared using:

[0059]

number

[0060] S123: The encoding unit 35a defines the k-th input embedding sequence in the sequence as follows.

[0061]

number

[0062]

number

[0063]

number

[0064]

number

[0065]

number

[0066]

number

[0067] S124: The encoding unit 35a encodes the input embedding sequence

[0068]

number

[0069]

number

[0070] Therefore, the decoding unit 36a outputs the probability distribution of the answer and the arithmetic operation process based on the encoded feature obtained by the encoding unit 35a. The process is as follows.

[0071] S131: In the learning phase, the decoding unit 36a acquires the correct answer information and the token sequence of the arithmetic operation process output by the parameter learning unit 38a for each output step, and creates an output sequence by combining the respective token sequences. Furthermore, in the inference phase, the decoding unit 36a combines the tokens recursively output for each output step in the text generation unit 39 to create an output sequence. Furthermore, the decoding unit 36a assigns a start symbol and a terminal symbol to the sequence in the learning phase, and only assigns the start symbol to the sequence in the inference phase. In this embodiment, as an example, the start symbol is assigned a [CLS] symbol, and the terminal symbol is assigned an [EOS] symbol. Furthermore, in this embodiment, the Byte-level BPE shown in Reference 3 is used as an example of tokenization processing.

[0072] S132: The decoding unit 36a converts the output sequence into the following embedding, similar to the encoding unit 35a.

[0073]

number

[0074]

number

[0075] where:

[0076]

number

[0077]

number

[0078]

number

[0079]

number

[0080] S133: The decoding unit 36a converts the representation of the output token into

[0081]

number

[0082]

number

[0083] The parameter learning unit 38a inputs output sequence information (output probability distribution information of answer information, output probability distribution information of arithmetic operation process) and correct answer information (correct answer information, arithmetic operation process information), and outputs parameter update information, tokenized correct answer information, and tokenized arithmetic operation process information. However, the correct answer information may be tokenized information.

[0084] Therefore, the parameter learning unit 38a performs tokenization processing on the correct answer information (correct answer information, arithmetic operation process information). However, the correct answer information may be input as tokenized information. Furthermore, the arithmetic operation process information may be information that has been tokenized by another processing unit (for example, the data extension unit 33). In this embodiment, the Byte-level BPE shown in Reference 3 is used as an example of tokenization processing.

[0085] The parameter learning unit 38a calculates the following loss based on the output sequence information and the correct answer information, and updates the model parameters of the machine learning model 41a so as to minimize it.

[0086]

number

[0087]

number

[0088] This completes the description of the functional configuration of the learning phase.

[0089] <Functional configuration of the inference phase> Next, the functions of the answer generation device 3a in the inference phase will be described. Figure 7 is a functional configuration diagram of the answer generation device in the inference phase.

[0090] As shown in Fig. 7, the answer generation device 3a includes an input unit 30, a text extraction unit 31a, an object extraction unit 32a, an encoding unit 35a, a decoding unit 36a, a text generation unit 39b, and an output unit 40a. These units each have a function realized by an instruction from the CPU 301 in Fig. 2 based on a program. A trained machine learning model 41b is stored in the RAM 303 or the SSD 304. Note that functional components similar to those in the learning phase are denoted by the same reference numerals, and their description will be omitted.

[0091] The text generation unit 39 receives the output sequence information from the decoding unit 36a and outputs answer information and arithmetic operation process information. The output sequence information in the first embodiment is output probability distribution information of the answer information and output probability distribution information of the arithmetic operation process.

[0092] Therefore, the text generation unit 39 recursively generates answer information and arithmetic operation process information based on the output probability distribution information of the answer information and the output probability distribution information of the arithmetic operation process included in the output sequence information. The text generation unit 39 selects the word with the maximum probability distribution for each output process, or generates words by sampling according to the probability distribution, and ends word generation when the end-of-sentence token [EOS] is generated.

[0093] The output unit 40a outputs the inference result from the answer generation device 3a based on the answer information and arithmetic operation process information output by the text generation unit 39. Examples of output include displaying the result on a display connected to the external device connection I / F 305 in Fig. 2, or transmitting the result to the communication terminal 5 or the like via the network I / F 306.

[0094] [Processing or operation of the answer generation device] Next, the processing or operation of the answer generation device 3a in the learning phase and inference phase will be described with reference to FIGS.

[0095] <Processing or operation in the learning phase> FIG. 8 is a flowchart showing the machine learning method executed by the answer generation device in the learning phase.

[0096] S11: The input unit 30 inputs the learning data (correct answer information, document image, question text) from the communication terminal 7 or directly.

[0097] S12: The text extraction unit 31a extracts text information from the document image, and the object extraction unit 32a extracts object information from the document information.

[0098] S13: The data extension unit 33 generates arithmetic operation process information.

[0099] S14: The encoding unit 35a generates an encoded feature.

[0100] S15: The decoding unit 36a generates answer information and arithmetic operation process information.

[0101] S16: The parameter learning unit 38a calculates the loss and updates the parameters.

[0102] S17: The parameter learning unit 38a determines whether there is unprocessed data. If there is unprocessed data (S17; YES), the process returns to step S11. On the other hand, if there is no unprocessed data (S17; NO), the learning phase process ends.

[0103] This completes the description of the processing or operation in the learning phase.

[0104] <Processing or operation in the inference phase> FIG. 9 is a flowchart showing an answer generation method executed by the answer generation device in the inference phase.

[0105] S21: The input unit 30a inputs input data (document image, question text) from the communication terminal 7 or directly.

[0106] S22: The text extraction unit 31a extracts text information from the document image, and the object extraction unit 32a extracts object information from the document information.

[0107] S23: The encoding unit 35a generates an encoded feature.

[0108] S24: The text generation unit 39 generates answer information and arithmetic operation process information.

[0109] S25: The text generator 39 determines whether a sentence-end symbol has been output. If a sentence-end symbol has not been output (S25; NO), the process returns to step S24. On the other hand, if a sentence-end symbol has been output (S25; YES), the process proceeds to step S26.

[0110] S26: The output unit 40 outputs information on the inference result including the answer information and the arithmetic operation process information.

[0111] This completes the description of the processing or operation of the inference phase.

[0112] [Experimental conditions and results of the first embodiment] Next, the experimental conditions and results of this embodiment will be described.

[0113] <Experimental conditions> (Experimental settings) The batch size was set to 64 for both pre-training and fine-tuning, and the answer generator 3a was trained for 30 epochs. Optimization was performed using Adam (Reference 8), with a learning rate of 3e-5. obj ,w obj ,w ocr ,w task The maximum lengths of the BERT-{base,large} trained ... <Reference 8>Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015. (Evaluation indicators) We use the ANLS (Reference 9) (average edit distance between predicted sentences and the set of ground truth sentences) adopted in the ICDAR 2021 Competition. ANUM denotes the ANLS when limited to examples requiring arithmetic operations (17.4% of the development data). <Reference 9>Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomezi Bigorda, Marcal Rusinol, CV Jawahar, Ernest Valveny, and Dimosthenis Karatzas. Scene text visual question answering.In ICCV, pp. 4290-4300, 2019. <Experimental Results> Fig. 19 is a table showing the experimental results according to the first and second embodiments. As shown in Fig. 19, the values ​​of ANLS and ANUM were reduced by using the concept of arithmetic operation process compared to the conventional case where the concept was not used, as shown in the second row "BERT w / o ADA" and the fifth row "IG-BERT w / o SRP + ADA" of the table.

[0114] [Effects of the first embodiment] As described above, according to the first embodiment, in order for the answer generation device 3a to generate arithmetic operation processes required for answer information, the sequence conversion unit 20a uses the machine learning model 41a to perform learning in the learning phase to generate arithmetic operation processes required for answer information. This improves the answer generation device 3a's ability to understand numerical data included in document images, making it possible to generate answer information involving arithmetic operations with higher performance than conventional techniques.

[0115] The data extension unit also prepares multiple templates and automatically generates arithmetic operation process information by using a predetermined template, which reduces the cost of creating training data compared to manually creating correct answer information for arithmetic operation processes.

[0116] Furthermore, understanding the positional relationship between objects (icons, graphs, etc.) and text (e.g., 60%) included in a document image is one of the very important factors for a system to understand a document image and perform question answering. The answer generation device 3a of this embodiment can understand visually expressed documents (infographics, PDF documents, HTML documents, etc.) that exist in large numbers in the real world as knowledge sources and perform question answering involving arithmetic operations.

[0117] Second embodiment Next, a second embodiment of the present invention will be described with reference to FIGS. 10 to 15. Compared to conventional techniques that use question text indicating a question sentence and a document image containing the text as input and generate answer information to a question based on the contents of the document image, an answer generation device 3b according to the second embodiment of the present invention provides specific improvements and represents an advancement in the technical field related to generating answer information using a neural network. The overall configuration and hardware configuration of the communication system according to this embodiment are similar to those of the first embodiment, and therefore will not be described here. Furthermore, while FIG. 10 shows the functional configuration of the learning phase and FIG. 15 shows the functional configuration of the inference phase, the answer generation device 3b may also include the components of FIGS. 10 and 13 collectively.

[0118] [Functional configuration of the answer generation device] The functional configuration of the answer generation device 3b according to this embodiment in the learning phase and inference (prediction) phase will be described.

[0119] <Learning phase functional configuration> Each function of the answer generation device 3b in the learning phase will be described with reference to Fig. 10. Fig. 10 is a functional configuration diagram of the answer generation device in the learning phase according to the second embodiment. Note that the answer generation device 3b is an example of the answer generation device 3.

[0120] As shown in FIG. 4, the answer generation device 3b has an input unit 30, a text extraction unit 31b, an object extraction unit 32b, a correct layout relation derivation unit 34, an encoding unit 35b, a decoding unit 36b, a layout relation prediction unit 37, and a parameter learning unit 38b. Each of these units is a function realized by an instruction from the CPU 301 in FIG. 2 based on a program. In addition, the RAM 303 or the SSD 304 stores a machine learning model 42a and a machine learning model (W srp ) 43 is constructed. Note that the input unit 30, the text extraction unit 31b, the object extraction unit 32b, and the encoding unit 35b perform the same processes as the input unit 30, the text extraction unit 31a, the object extraction unit 32a, and the encoding unit 35b in the first embodiment, respectively, and therefore descriptions thereof will be omitted.

[0121] The input unit 30 inputs learning data (data on the question sentence, correct answer information, and document image) to the answer generation device 3b via the communication terminal 7. Alternatively, the input unit 30 inputs learning data directly to the answer generation device 3a. FIG. 11 is a conceptual diagram of the processing of the correct layout relation derivation unit according to the second embodiment. For example, as shown in FIG. 11, the question sentence is "What number is written above the woman's icon?" The document image is the same as that in FIG. 5. The correct answer information is "3.6%."

[0122] The correct layout relationship derivation unit 34 inputs rectangular area information of the document text output from the text extraction unit 31b and rectangular area information of the object output from the object extraction unit 32b, and outputs rectangular selection area information of the document text (an example of first rectangular selection area information), rectangular selection area information of the object (an example of second rectangular selection area information), and correct layout relationship information.

[0123] Therefore, the correct layout relation derivation unit 34 randomly selects one rectangular area information of a predetermined object and one rectangular area information of a predetermined document text from the rectangular area information of the object and one rectangular area information of the document text, and defines and outputs a correct layout relation consisting of M classes (any number as long as M=1 or more). In this embodiment, the layout relation of M=12 defined in Reference 7 (see the lower right of FIG. 11) is used. Here, an example is shown in which, regarding the layout relations of dashed-line rectangular areas with respect to solid-line rectangular areas, a layout relation classification task of 11 classes and one class in which the rectangular areas are too far apart are solved, for a total of 12 classes. <Reference 7> Ting Yao, Yingwei Pan, Yehao Li, Tao Mei: Exploring visual relationship for image captioning. ECCV18 Furthermore, the encoding unit 35b and the decoding unit 36b constitute a sequence transformation unit 20b. FIG. 12 is a detailed block diagram of the sequence transformation unit according to the second embodiment. The sequence transformation unit 20b uses neural network model parameters to generate output sequence information based on input information. The model configuration employs a general encoder-decoder model format for generating a sequence using an input sequence.

[0124] The stream conversion unit 20b does not have the arithmetic operation process information output unit 358 of the stream conversion unit 20a in the first embodiment, but does have the other components. Therefore, the same components are given the same reference numerals and their description will be omitted.

[0125] The decoding unit 36b basically performs the same processing as the decoding unit 36a, but since the second embodiment does not have the data extension unit 33, there is no input of arithmetic operation process information and no output of output probability distribution information of the arithmetic operation process.

[0126] The layout relationship prediction unit 37 inputs the rectangular selection area information of the document text and the rectangular selection area information of the object derived by the correct layout relationship derivation unit 34, as well as the encoded features output from the encoding unit 35b, and outputs predicted layout relationship distribution information.

[0127] Therefore, the layout relationship prediction unit 37 predicts the layout relationship between the object and the text in the image document based on the encoded feature amount, thereby making it possible to generate an answer that takes the layout relationship into consideration. The layout relationship prediction unit 37 performs the following process.

[0128] S211: The layout relation prediction unit 37 calculates feature values ​​e corresponding to the rectangular selection area information of the document text and the rectangular selection area information of the object output by the correct layout relation derivation unit 34 from the encoded feature values ​​corresponding to the object and the text (a vector of a predetermined dimension obtained by converting the information of the document image and the question sentence input to the encoding unit 35a based on the model parameters of the neural network). obj and e text to acquire.

[0129] S212: The layout relationship prediction unit 37 calculates e obj and e text Based on this, predicted layout relationship distribution information is calculated and output as follows:

[0130]

number

[0131] Therefore, the parameter learning unit 38b performs tokenization processing on the correct answer information. However, the correct answer information may be input after being tokenized. In this embodiment, the Byte-level BPE shown in Reference 3 is used as an example of the tokenization processing.

[0132] The parameter learning unit 38b calculates the following loss based on the output sequence information and the correct answer information, and updates the model parameters of the machine learning model 42a so as to minimize it.

[0133]

number

[0134]

number

[0135]

number

[0136]

number

[0137] This completes the description of the functional configuration of the learning phase.

[0138] <Functional configuration of the inference phase> Next, the functions of the answer generation device 3b in the inference phase will be described. Fig. 13 is a functional configuration diagram of the answer generation device in the inference phase.

[0139] As shown in Fig. 7, the answer generation device 3b includes an input unit 30, a text extraction unit 31b, an object extraction unit 32b, an encoding unit 35b, a decoding unit 36b, a text generation unit 39b, and an output unit 40b. Each of these units has a function realized by an instruction from the CPU 301 in Fig. 2 based on a program. A trained machine learning model 42b is stored in the RAM 303 or the SSD 304. Note that functional components similar to those in the learning phase are denoted by the same reference numerals, and their description will be omitted.

[0140] The text generating unit 39b receives the output sequence information from the decoding unit 36a and outputs answer information. The output sequence information in the second embodiment is output probability distribution information of the answer information.

[0141] Therefore, the text generator 39b recursively generates answer information based on the output sequence information. The text generator 39b selects the word with the maximum probability distribution for each output step, or generates words by sampling according to the probability distribution, and ends word generation when an end-of-sentence token [EOS] is generated.

[0142] The output unit 40b outputs the inference result from the answer generation device 3b based on the answer information output by the text generation unit 39. Examples of output include displaying the result on a display connected to the external device connection I / F 305 in Fig. 2, transmitting the result to the communication terminal 5 or the like via the network I / F 306, and the like.

[0143] [Processing or operation of the answer generation device] Next, the processing or operation of the answer generation device 3b in the learning phase and inference phase will be described with reference to FIGS.

[0144] <Processing or operation in the learning phase> FIG. 14 is a flowchart showing the machine learning method executed by the answer generation device in the learning phase.

[0145] S31: The input unit 30 inputs learning data (correct answer information, document image, question text) from the communication terminal 7 or directly.

[0146] S32: The text extraction unit 31b extracts text information from the document image, and the object extraction unit 32b extracts object information from the document information.

[0147] S33: The correct layout relation deriving unit 34 derives the correct layout relation.

[0148] S34: The encoding unit 35b generates an encoded feature.

[0149] S35: The decoding unit 36b predicts the answer information and outputs output probability distribution information of the answer information, and the layout relationship prediction unit 37 predicts the layout relationship information and outputs predicted layout relationship distribution information.

[0150] S36: The parameter learning unit 38b calculates the loss and updates the parameters.

[0151] S37: The parameter learning unit 38b determines whether there is unprocessed data. If there is unprocessed data (S37; YES), the process returns to step S31. On the other hand, if there is no unprocessed data (S37; NO), the process of the learning phase ends.

[0152] This completes the description of the processing or operation in the learning phase.

[0153] <Processing or operation in the inference phase> FIG. 15 is a flowchart showing an answer generation method executed by the answer generation device in the inference phase.

[0154] S41: The input unit 30 inputs input data (document image, question text) from the communication terminal 7 or directly.

[0155] S42: The text extraction unit 31b extracts text information from the document image, and the object extraction unit 32a extracts object information from the document information.

[0156] S43: The encoding unit 35b generates an encoded feature.

[0157] S44: The text generation unit 39 generates answer information.

[0158] S45: The text generator 39 determines whether a sentence-end symbol has been output. If a sentence-end symbol has not been output (S45; NO), the process returns to step S44. On the other hand, if a sentence-end symbol has been output (S45; YES), the process proceeds to step S46.

[0159] S46: The output unit 40b outputs information on the inference result including the answer information.

[0160] This completes the description of the processing or operation of the inference phase.

[0161] [Experimental conditions and results] Next, the experimental conditions and results of the second embodiment will be described. Note that the experimental conditions of the second embodiment are the same as those of the first embodiment, and therefore will not be described here.

[0162] <Experimental Results> As shown in FIG. 19, as shown in the fourth row of the table, "IG-BERT w / o SRP," by using the concept of layout relationship, the values ​​of ANLS and ANUM were reduced compared to the conventional case where the concept was not used.

[0163] [Effects of the second embodiment] As described above, according to the second embodiment, in the answer generation device 3b, in addition to machine learning for generating answer information, the layout relationship prediction unit 37 and the parameter learning unit 38b perform learning on the layout relationship between objects and text in a document image. This makes it possible to generate answer information that takes the layout relationship into consideration. Therefore, the answer generation device 3b's ability to understand the layout relationship between objects and text is improved, making it possible to understand document images and respond to questions with higher performance than conventional techniques.

[0164] As a result, the answer generation device 3b can understand the many visually expressed documents (infographics, PDF documents, HTML documents, etc.) that exist in the real world as knowledge sources and perform question and answering.

[0165] Supplementary Information The present invention is not limited to the above-described embodiment, and may have the following configurations or processes (operations).

[0166] (1) The answer generating devices 3 a and 3 b can be realized by a computer and a program, but this program can also be recorded on a (non-transitory) recording medium or provided via the communication network 100 .

[0167] (2) In the above embodiment, a notebook computer is shown as an example of the communication terminal 5, but the present invention is not limited to this and may be, for example, a desktop computer, a tablet terminal, a smartphone, a smart watch, a car navigation device, a refrigerator, a microwave oven, etc. (3) Each CPU 301, 501 may be a single CPU or multiple CPUs.

[0168] ●Additional notes The above-described embodiment can also be expressed as the following invention.

[0169] [Additional Note 1] An answer generation device having a processor that generates answer information to a question based on the content of a document image, The processor: an encoding process for outputting encoded features based on text information in the document image, object information indicating objects in the document image, and the question; a decoding process for outputting output probability distribution information of the answer information based on the encoded feature; a correct layout relationship derivation process that derives first rectangular selection area information indicating a rectangular area of ​​a predetermined document text selected from rectangular area information indicating rectangular areas of a plurality of document texts in the document image, derives second rectangular selection area information indicating a rectangular area of ​​a predetermined object selected from rectangular area information indicating rectangular areas of one or more objects in the document image, and derives correct layout relationship information indicating a layout relationship of the predetermined document text with respect to the predetermined object; a layout relationship prediction unit that predicts a layout relationship between the predetermined document text related to the first rectangular selection area information and the predetermined object related to the second rectangular selection area information based on the encoded feature, and outputs predicted layout relationship distribution information; a parameter learning process for learning model parameters of a neural network based on output probability distribution information of the answer information and correct answer information (so that the output probability distribution information of the answer information approaches the correct answer information), and based on the predicted layout relationship distribution information and the correct layout relationship information (so that the predicted layout relationship distribution information approaches the correct layout relationship information); An answer generation device that executes the above.

[0170] [Additional note 2] The answer generation device described in Appendix 1, wherein the correct layout relationship derivation process includes a process of deriving the correct layout relationship information by predicting to which of a plurality of classes representing predetermined layout relationships the layout relationship between the specified document text related to the first rectangular selection area information and the specified object related to the second rectangular selection area information corresponds.

[0171] [Additional note 3] The response generation device according to claim 1 or 2, the encoding process and the decoding process include a process of outputting output probability distribution information of the answer information based on the document image and the question sentence by using the model parameters previously learned by the parameter learning unit, the processor executes a text generation process to generate the answer information based on output probability distribution information of the answer information. Answer generator.

[0172] [Additional Note 4] A machine learning method executed by a processor that generates answer information to a question based on the content of a document image, The processor: outputting an encoded feature, which is a vector indicating a region in the document image corresponding to the question, based on text information in the document image, object information indicating an object in the document image, and the question; outputting output probability distribution information of the answer information based on the encoded feature; deriving first rectangular selection area information indicating a rectangular area of ​​a predetermined document text selected from rectangular area information indicating rectangular areas of a plurality of document texts in the document image; deriving second rectangular selection area information indicating a rectangular area of ​​a predetermined object selected from rectangular area information indicating rectangular areas of one or more objects in the document image; and deriving correct layout relationship information indicating a layout relationship of the predetermined document text with respect to the predetermined object; predicting a layout relationship between the predetermined document text related to the first rectangular selection area information and the predetermined object related to the second rectangular selection area information based on the encoded feature, thereby outputting predicted layout relationship distribution information; performing machine learning of model parameters of a neural network based on output probability distribution information of the answer information, correct answer information, the predicted layout relationship distribution information, and the correct layout relationship information; Machine learning methods.

[0173] [Additional Note 5] A non-transitory recording medium having a program recorded thereon for causing a computer to realize the answer generation device according to any one of appendixes 1 to 3. [Explanation of symbols]

[0174] 1. Communication Systems 3 Answer generator 3a Answer generator 3b Answer generator 5. Communication terminals 30 Input section 31a, 31b Text extraction section 32a,32b Object extraction part 33 Data Extension 34 Correct placement relationship derivation part 35a, 35b Encoding section 36a, 36b Decoder 37 Layout Prediction Unit 38a, 38b Parameter learning section 39a, 39b Text generation section 40a, 44b Output section 20a Series conversion unit 20b Sequence conversion unit 41a Machine Learning Models 41b Trained machine learning model 42a Machine Learning Models 42b Trained Machine Learning Model

Claims

1. An answer generation device that generates answer information to a question based on the content of a document image, an encoding unit that outputs encoded features based on text information in the document image, object information indicating an object in the document image, and the question; a decoding unit that outputs output probability distribution information of the answer information based on the encoded feature; a correct layout relation derivation unit that derives first rectangular selection area information indicating a rectangular area of ​​a predetermined document text selected from rectangular area information indicating rectangular areas of a plurality of document texts in the document image, derives second rectangular selection area information indicating a rectangular area of ​​a predetermined object selected from rectangular area information indicating rectangular areas of objects in the document image, and derives correct layout relation information indicating a layout relation of the predetermined document text with respect to the predetermined object; a layout relationship prediction unit that predicts a layout relationship between the predetermined document text related to the first rectangular selection area information and the predetermined object related to the second rectangular selection area information based on the encoded feature, and outputs predicted layout relationship distribution information; a parameter learning unit that learns model parameters of a neural network based on output probability distribution information of the answer information, correct answer information, the predicted layout relationship distribution information, and the correct layout relationship information; An answer generation device having the above.

2. 2. The answer generation device according to claim 1, wherein the correct layout relationship derivation unit derives the correct layout relationship information by predicting to which of a plurality of classes representing a predetermined layout relationship the layout relationship between the specified document text related to the first rectangular selection area information and the specified object related to the second rectangular selection area information corresponds.

3. 3. The response generation device according to claim 1, the encoding unit and the decoding unit include a process of outputting output probability distribution information of the answer information based on the document image and the question sentence, using the model parameters previously learned by the parameter learning unit; a text generation unit that generates the answer information based on output probability distribution information of the answer information; Answer generator.

4. A machine learning method executed by an answer generation device that generates answer information to a question based on the content of a document image, The answer generation device outputting encoded features based on text information in the document image, object information indicating an object in the document image, and the question; outputting output probability distribution information of the answer information based on the encoded feature; deriving first rectangular selection area information indicating a rectangular area of ​​a predetermined document text selected from rectangular area information indicating rectangular areas of a plurality of document texts in the document image; deriving second rectangular selection area information indicating a rectangular area of ​​a predetermined object selected from rectangular area information indicating rectangular areas of objects in the document image; and deriving correct layout relationship information indicating a layout relationship of the predetermined document text with respect to the predetermined object; predicting a layout relationship between the predetermined document text related to the first rectangular selection area information and the predetermined object related to the second rectangular selection area information based on the encoded feature, and outputting predicted layout relationship distribution information; learning model parameters of a neural network based on output probability distribution information of the answer information, correct answer information, the predicted layout relationship distribution information, and the correct layout relationship information; Machine learning methods.

5. A program for causing a computer to implement the answer generation device according to claim 1.

Images