Inference device, learning device, inference method, and program

Abstract

This inference device comprises a feature generation unit that generates a feature on the basis of a trained model from a first text and visual information related to a text and extracted from data including the text, and that outputs the feature in order to input the feature to a generation unit. The generation unit outputs a second text on the basis of the feature and the first text.

Description

Inference device, learning device, inference method, and program

[0001] The present invention relates to a technique for outputting text based on a document image.

[0002] There are known techniques for executing tasks targeting document images in which text and images are arranged in various positions. Examples of such tasks include a question-answering task in which a document image is used as a knowledge source to generate answer text to a question, and an information extraction task in which specific information is extracted from a document image.

[0003] One known conventional technique for executing a task targeting natural images, which are images of natural scenery, etc., is the technique disclosed in Non-Patent Document 1. The technique disclosed in Non-Patent Document 1 uses a pre-trained large language model (LLM).

[0004] Junnan Li, Dongxu Li, Silvio Savarese, Steven Hoi. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. ICML23

[0005] However, in the conventional technology disclosed in Non-Patent Document 1, when a document image is input, it is not possible to use features that explicitly take into account visual information, such as layout information related to the text, in the document image. Note that a document image is an example of "data containing text."

[0006] The present invention has been made in consideration of the above points, and aims to provide a technique for generating features from data including text that are explicitly based on visual information related to the data.

[0007] According to the disclosed technology, there is provided an inference device comprising: a feature generation unit that generates features from visual information about the text extracted from data including text and a first text based on a trained model, and outputs the features to be input to a generation unit; and the generation unit outputs a second text based on the features and the first text.

[0008] The disclosed technology provides a technique for generating features from data that includes text that are explicitly based on visual information about the data.

[0009] FIG. 1 is a diagram illustrating task example 1. FIG. 2 is a diagram illustrating task example 3. FIG. 4 is a diagram illustrating task example 5. FIG. 1 is a diagram illustrating an example configuration of a learning device 100. A flowchart illustrating an example operation of the learning device 100. FIG. 2 is a diagram illustrating an example internal configuration of a generation unit 140. FIG. 3 is a diagram illustrating parameters to be updated and parameters not to be updated. FIG. 3 is a diagram illustrating an example configuration of an inference device 200. FIG. 4 is a diagram illustrating parameters to be updated and parameters not to be updated. A flowchart illustrating an example operation of the inference device 200. FIG. 4 is another example configuration of the learning device 100. FIG. 5 is another example configuration of the inference device 200. FIG. 6 is a diagram illustrating an example hardware configuration of an device. FIG. 7 is a diagram illustrating an example configuration of a system.

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment.

[0011] With regard to the tasks targeting document images mentioned above, conventional techniques typically prepare a dataset for the target task (e.g., question answering) for each document image type (e.g., invoice, etc.), and then use the dataset for training to build a model that can be adapted to a specific document type and task. However, there are a wide variety of document images and tasks (user needs), and it is difficult to achieve all of these with a single model using conventional techniques.

[0012] Furthermore, as mentioned above, the technique disclosed in Non-Patent Document 1 does not allow LLM to use features that explicitly take into account visual information about the text in document images, and therefore the accuracy of task execution is insufficient.

[0013] In contrast, in this embodiment, a mechanism is provided for inputting features that explicitly take into account visual information about text in document images into the LLM, thereby improving task execution accuracy. In other words, each device according to this embodiment provides specific improvements over conventional task execution techniques such as those disclosed in Non-Patent Document 1, and represents an advancement in the technical field of document image understanding.

[0014] The technology according to the present embodiment can be applied to any language. In addition, although the following description is given assuming that learning device 100 and inference device 200 are separate devices, learning device 100 and inference device 200 may be the same device.

[0015] (Definition of Terms) In the present embodiment, a "document image" is an example of "data containing text." Examples of a "document image" include, but are not limited to, presentation slides, papers, PDF documents, and HTML documents.

[0016] A "document image" is assumed to include text (character string) information and image information. However, the data format of the "document image" may be an image format, an image and text format, or a text format.

[0017] The "image format" mentioned above refers to image data in which text information is embedded as image information. For example, the non-searchable PDF format falls into this category. The "image and text format" refers to image data in which text information is embedded as is. For example, the searchable PDF format falls into this category. The "text format" refers to data that contains only text, no figures or photos, but is treated as image data.

[0018] Examples of "data including text" that are not included in the general meaning of "document image" include landscape images including explanatory text, television screens such as news, etc. The technology according to this embodiment is applicable to "data including text" in general. Note that in this embodiment, "document image" also includes landscape images including explanatory text, television screens such as news, etc.

[0019] Furthermore, "visual information" in "data including text" refers to information that can be visually recognized in "data including text." "Visual information" may include features converted from visually recognized information.

[0020] Examples of "visual information" include the position of a text area, the size of a text area, the color of the text, the shape of the text, the color, type, shape, and size of the text font, the position of non-text areas, the size of non-text areas, the color of non-text areas, and the shape of non-text areas. Examples of non-text areas include graphs, photographs, landscapes, and backgrounds. Note that even non-text areas are related to the text. Other examples of "visual information" include depth information within an image and line information within an image. Depth information is, for example, information indicating that a certain area in an image is located further back than another area, and may also be referred to as context. Line information is, for example, information about lines used to separate tables or underline text. Note that "visual information" includes information such as the color and shape of text (visually recognizable information), but does not include linguistic information (semantic information) contained in the text strings.

[0021] In this specification, "visual information," "information indicating an area," "area information," and "position information" are used broadly, and "information indicating an area" is included in "visual information." In other words, "information indicating an area" is an example of "visual information." "Information indicating an area" and "area information" are synonymous. Furthermore, "position information" related to an area is included in "information indicating an area." In other words, "position information" related to an area is an example of "information indicating an area."

[0022] The "instruction sentence" is an example of the "first text." The "instruction sentence" may also be called the "instruction text." The "answer sentence" is an example of the "second text." The "answer sentence" may also be called the "answer text." Furthermore, the text portion of the "data including text" is an example of the "third text."

[0023] The type of "instruction statement" used in this embodiment is not limited to a specific one. Examples of "instruction statements" include instructions, queries, questions, requests, and text for specifying output. However, "instruction statements" can be any text that has some meaning, and do not have to be text that expresses an "instruction" or a "question."

[0024] In this embodiment, both "token" and "token sequence" may be replaced with "text" or "word." Also, "word" may be replaced with "text."

[0025] (Example of a Task) Learning device 100 and inference device 200 according to this embodiment will be described in detail below, but before that, to facilitate understanding of the operation of these devices, examples of tasks executed by learning device 100 / inference device 200 will be described. The examples described below may be considered to be tasks executed by learning device 100 in the learning phase, or may be considered to be tasks executed by inference device 200 in the inference phase, but here they will be described assuming that they are tasks executed by inference device 200.

[0026] <Example 1> Example 1 is shown in Figure 1. Example 1 uses data from WTQ (https: / / github.com / ppasupat / WikiTableQuestions). In Example 1, when the document image shown in (a) and the directive shown in (b) are input to inference device 200, inference device 200 outputs the numerical value (example text) shown in (c). Note that the portion of the document image in Figure 1 that is framed in bold has been added to make the contents of the document image easier to understand, and this framed portion does not exist in the actual task.

[0027] As shown in Figure 1, the document image contains a multi-year forecast for Operations and Maintenance at the University of Minnesota, and the directive queries for the forecast for 2010-2011. In response, reasoning apparatus 200 outputs accurate numerical values.

[0028] The same task as in Example 1 was performed using ChatGPT (registered trademark), an existing large-scale language model (LLM), and BLIP-2 disclosed in Non-Patent Document 1, but no appropriate answer was obtained.

[0029] <Example 2> Figure 2 shows Example 2. Example 2 uses data from Screen2Words (https: / / github.com / google-research-datasets / screen2words). In Example 2, when the document image shown in (a) and the directive shown in (b) are input to inference device 200, inference device 200 outputs the text shown in (c).

[0030] As shown in Figure 2, the document image contains an image of a guitar being played, a list of music genres, and an image of a search page. The directive shown in (b) is a query, limited in number of words, about what the UI image is. In response, the inference device 200 outputs the correct content for the document image.

[0031] As in Example 1, in Example 2, it was confirmed that even when performing the same task as in Example 2 using ChatGPT (registered trademark) and BLIP-2, appropriate answers could not be obtained.

[0032] <Example 3> Figure 3 shows Example 3. Example 3 uses data from FUNSD (Form Understanding in Noisy Scanned Documents) (https: / / guillaumejaume.github.io / FUNSD).

[0033] In Example 3, when the document image shown in (a) and the directive shown in (b) are input to inference device 200, inference device 200 outputs the text shown in (c).

[0034] As shown in Figure 3, the document image shows an image titled "CASE FORM," and the instruction indicates answer options and asks about the category of "CASE FORM." In response, reasoning device 200 outputs the correct content.

[0035] <Example 4> Figure 4 shows Example 4. Example 4 uses data from SlideVQA (https: / / github.com / nttmdlab-nlp / SlideVQA).

[0036] In Example 4, when the two document images shown in (a) and the directive shown in (b) are input to inference device 200, inference device 200 outputs the text shown in (c).

[0037] As shown in Figure 4, the document image on the left displays a graph of the number of journalists by region, and the document image on the right displays percentage values ​​such as "Competition media" for each region. The instruction text includes a question and an answer style. In response, reasoning device 200 outputs the correct content.

[0038] Example 5 Example 5 is shown in Figure 5. Example 5 uses data from SciCap (Scientific Figures Dataset) (https: / / github.com / tingyaohsu / SciCap).

[0039] In Example 5, when the document image shown in (a) and the directive shown in (b) are input to inference device 200, inference device 200 outputs the text shown in (c).

[0040] As shown in Figure 5, the document image shows a graph related to "Corruption Gaussian noise." The directive is a statement requesting an explanation of the image. In response, the inference device 200 outputs the correct content.

[0041] The configurations and operations of the learning device 100 and the inference device 200 will be described below.

[0042] (Configuration example of learning device 100) Fig. 6 shows a configuration example of the learning device 100. As shown in Fig. 6, the learning device 100 has a text extraction unit 110, an image feature extraction unit 120, a feature conversion unit 130, a generation unit 140, and a parameter learning unit 150. In addition, the generation unit 140 has an encoding unit 141 and a decoding unit 142.

[0043] The image feature extraction unit 120, feature conversion unit 130, and generation unit 140 are functional units realized by a neural network model, and parameters of the model are stored in a model DB 160. The learning device 100 (specifically, a computer) reads the parameters from the model DB 160 and executes the operation of each functional unit using the parameters. Note that functional units other than the "image feature extraction unit 120, feature conversion unit 130, and generation unit 140" may also be realized by a neural network model.

[0044] Furthermore, although the model parameters are updated by the parameter learning unit 150 of the learning device 100, there are also parameters that are not updated and remain fixed. Details of this will be described later.

[0045] (Example of Operation of Learning Device 100) Fig. 7 is a flowchart showing an example of operation of the learning device 100. Below, the operation of each unit constituting the learning device 100 will be described in detail along the steps of the flowchart in Fig. 7 .

[0046] <S101 (Step 101): Input of Training Data> In S101, a training data set is input to the training device 100. The training data set includes multiple training data (which may also be called instances). Each training data set has one or multiple "instruction sentences, document images, and correct answer sentences." Here, it is assumed that the process of executing S102 to S105 using one training data set is performed for each training data set included in the training data set.

[0047] 6, a document image is input to each of a text extraction unit 110 and an image feature extraction unit 120, and a directive sentence is input to each of a feature conversion unit 130 and a generation unit 140. In addition, a correct answer sentence is input to a parameter learning unit 150.

[0048] The type of training dataset (i.e., the type of task for learning) is not limited to a specific type, but it is possible to use training datasets for tasks such as reading comprehension, summarizing, information extraction, Q&A, dialogue, captioning, and region illustration. Furthermore, it is also possible to use training datasets of the types shown in Figures 1 to 5 (Examples 1 to 5).

[0049] <S102: Information Extraction> In S102, the text extraction unit 110 extracts text information from the input document image and outputs the text information to the feature conversion unit 130. The image feature extraction unit 120 extracts document image features from the input document image and outputs the document image features to the feature conversion unit 130. The operations of the text extraction unit 110 and the image feature extraction unit 120 will be described in more detail below.

[0050] <S102: Operational Details of Text Extraction Unit 110> The text extraction unit 110 receives a document image as input and outputs text information in the document image. The text information in the document image includes a document text sequence (which may also be referred to as text) included in the input document image, and information indicating a document text rectangular area in which the document text sequence exists (e.g., top left coordinate, bottom right coordinate).

[0051] Specifically, the text extraction unit 110 detects regions of one or more document text sequences included in the input document image, recognizes the document text sequences in the detected regions, and outputs text information in the document image including this information (text and regions). If there are multiple document text sequences, the text information in the document image includes the multiple document text sequences and their region information.

[0052] In this embodiment, the region of the document text sequence is a rectangular region, and the rectangular region is defined by the upper left coordinate (x １ , y １ ) and the bottom right coordinate (x ２ , y ２ However, this expression is just an example, and other expressions may be used.

[0053] It should be noted that the information indicating an area is not limited to "top left coordinate, bottom right coordinate" as in the present embodiment. For example, it may be "top right coordinate, bottom left coordinate". Basically, the information indicating an area includes position information. However, the information indicating an area may not include position information and may only have size (area) information. Furthermore, the information indicating an area may not include size (area) information and may only have position information. Furthermore, the information indicating an area may include visual information corresponding to the area.

[0054] Furthermore, the "top left coordinate, bottom right coordinate" is information that assumes that the shape of the area is rectangular, but the shape of the area may be assumed to be a shape other than rectangular, and parameters indicating that shape (e.g., the center and radius of a circle) may be used as information indicating the area.

[0055] A more detailed example of using "top left coordinates, bottom right coordinates" as information indicating an area will be described below.

[0056] A document text sequence is a set of M words, where {s ｉ} Ｍ ｉ＝１ It is expressed as s ｉ is the i-th word. In other words, the text extraction unit 110 extracts {s ｉ} Ｍ ｉ＝１ The text extraction unit 110 also extracts a set of rectangular regions for each word {(x ｉ １ , y ｉ １ , x ｉ ２ , y ｉ ２ ) Ｍ ｉ＝１ Extract and output (x ｉ １ , y ｉ １ ), (x ｉ ２ , y ｉ ２ ) are respectively s ｉ are the top left and bottom right coordinates of the

[0057] When outputting multiple document text sequences, the text extraction unit 110 rearranges the document text sequences in the order of "left to right" and "top to bottom" according to their coordinates in the document image. However, such rearrangement may not be performed.

[0058] The text extraction unit 110 may use any method that can output a document text sequence and information indicating the region of the document text sequence from a document image. For example, OCR (Optical Character Recognition) may be used. Specifically, Google Vision API (https: / / cloud.google.com / vision) or the like may be used.

[0059] <S102: Operation Details of Image Feature Extraction Unit 120> The image feature extraction unit 120 receives a document image as input and outputs document image features. More specifically, the image feature extraction unit 120 has an image encoder, and the image encoder extracts a vector z ｖｉｓ Extract and output.

[0060] Any image encoder can be used as long as it can acquire image features from document images. For example, a pre-trained model called CLIP (Contrastive Language-Image Pre-training) disclosed in "Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. ICML21" can be used as the image encoder.

[0061] <S103: Feature Conversion> In S103, the feature conversion unit 130 receives the instruction sentence, the text information in the document image output from the text extraction unit 110, and the document image features output from the image feature extraction unit 120 as input, converts the input information into document image conversion features, and outputs the document image conversion features. The output document image conversion features are input to the generation unit 140. The document image conversion features output from the feature conversion unit 130 are features that can be understood by the generation unit 140 (LLM). As described above, the text information in the document image includes the text itself and information about rectangular areas, which is visual information.

[0062] The document image transformation features are ｄｏｃ The feature conversion unit 130 converts h ｄｏｃ is calculated by the following steps S1 to S3.

[0063] S1) In S1, the feature conversion unit 130 converts the text information and the instruction sentence in the document image into an embedding sequence. Here, the text information in the document image is {s ｉ} Ｍ ｉ＝１ and {(x ｉ １ , y ｉ １ , x ｉ ２ , y ｉ ２ ) Ｍ ｉ＝１ is input to the feature transform unit 130.

[0064] The feature conversion unit 130 converts the i-th word s ｉ Embedding z ｉ ｏｃｒ is calculated using the following formula: ｓ , W ｘ , W ｙ , W ｈ , W ｗ are the learnable weights (specifically, matrices), respectively.

[0065] If a document text sequence consists of M words, the above calculation is performed for each of the M words to obtain an embedding sequence z corresponding to the document text sequence (text information in a document image). ｏｃｒ can be obtained.

[0066] Similarly, the feature transformer 130 uses the learnable weights W ｓ Using this, the directive is embedded into the sequence z ｉｎｓ For example, convert the words that make up the instruction into ins ｉ Then, the feature conversion unit 130 converts W ｓ (ins ｉ ) and calculate the W ｓ (ins ｉ ) are concatenated into z ｉｎｓ Let's say.

[0067] z ｏｃｒ and Z ｉｎｓ are vectors. ｏｃｒ and Z ｉｎｓ An input token sequence (vector) is constructed by vector concatenation of and.

[0068] S2) The feature transformation unit 130 has a cross-attention layer and a self-attention layer. The feature transformation unit 130 combines a trainable parameter sequence and a document image feature z ｖｉｓ The self-attention layer interacts with the learnable parameter sequence and the input token sequence (z ｏｃｒ and Z ｉｎｓ In S2, a trainable parameter sequence is obtained through these interactions.

[0069] More specifically, the cross-attention layer includes a trainable parameter sequence and document image features zz ｖｉｓ is input, and the converted "learnable parameter sequence" is output. The self-attention layer receives the learnable parameter sequence and the input token sequence (z ｏｃｒ and Z ｉｎｓ The input is a concatenation of the inputs, and the converted "trainable parameter sequence" is output.

[0070] In other words, using the input learnable parameter sequence as a pivot, the cross-attention layer interacts with document image features, and the self-attention layer interacts with the input token sequence, resulting in a "learnable parameter sequence" through these interactions.

[0071] As a variation, the learnable parameter sequence interacted with document image features in the cross-attention layer may be further interacted with an input token sequence in the self-attention layer to output a transformed "learnable parameter sequence." Also, the learnable parameter sequence interacted with an input token sequence in the self-attention layer may be further interacted with document image features in the cross-attention layer to output a transformed "learnable parameter sequence."

[0072] As a model having the above-mentioned cross-attention layer and self-attention layer, for example, the pre-trained BERT disclosed in "Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL19" can be used.

[0073] The trainable parameter sequence is a D×L dimensional vector, where D=1024 and L=32 in one example.

[0074] S3) The feature transformation unit 130 further includes a feedforward network. In S3, the feature transformation unit 130 inputs the trainable parameter sequence obtained in S2 into the feedforward network, and outputs the document image transformation feature h ｄｏｃ The feature conversion unit 130 converts the obtained document image conversion feature h ｄｏｃ Output.

[0075] Document image conversion feature h ｄｏｃThe number of dimensions of the learnable parameter sequence is suitable for input to the generation unit 140 (specifically, the encoding unit 141) described in S104. Even if the number of dimensions of the learnable parameter sequence is not suitable for input to the encoding unit 141, the learnable parameter sequence can be passed through a feedforward network to generate the document image conversion feature h ｄｏｃ The number of dimensions can be set to a number of dimensions suitable for input to the encoding unit 141.

[0076] Note that the above method for generating document image conversion features is merely an example. Any method may be used as long as it can generate features that reflect "text information in the document image, document image features, and instruction sentences." The features are input to the generation unit 140.

[0077] <S104: Generation process> In S104, the generation unit 140 receives as input the instruction sentence, the document image conversion feature, the text information in the document image (specifically, the document text sequence) output from the text extraction unit 110, and the tokenized answer sentence output from the parameter learning unit 150, and outputs a probability distribution of the output sequence.

[0078] The generation unit 140 including the encoding unit 141 and the decoding unit 142 can be realized using a pre-trained LLM. As the LLM, for example, a well-known model using Transformer can be used.

[0079] Specifically, a model called FlanT5 can be used as the pre-trained LLM. However, using FlanT5 is merely an example. A pre-trained LLM other than FlanT5 may also be used.

[0080] An example of the internal configuration of the generation unit 140 when a model using a Transformer such as FlanT5 is used as the LLM is shown in Fig. 8. The configuration of the generation unit 140 is the same during training and during inference, and Fig. 8 shows not only the functional units used during training and inference, but also the text generation unit 6 used only during inference.

[0081] 8, the encoding unit 141 in the generation unit 140 includes a document image conversion feature receiving unit 1, a directive sentence receiving unit 2, a document image text information receiving unit 3, and multiple conversion layers 4. The decoding unit 142 in the generation unit 140 includes multiple conversion layers 5 and a text generation unit 6. A Transformer is used as each conversion layer.

[0082] The operations of the encoding unit 141 and the decoding unit 142 that make up the generating unit 140 will be described below.

[0083] <S104: Operation of encoding unit 141> In encoding unit 141, document image conversion features are input by document image conversion feature receiving unit 1, instruction statements are input by instruction statement receiving unit 2, and a document text sequence is input by document image text information receiving unit 3.

[0084] The transformer layer 4 converts the instruction sentence, the document image transformation feature, and the document text sequence into encoded features and outputs the encoded features. The encoded features are, for example, K-dimensional vectors, where K=1024 in one embodiment.

[0085] <S105: Operation of Decoding Unit 142 (Partly Operation of Parameter Learning Unit 150)> The decoding unit 142 receives the encoding feature output from the encoding unit 141 and the tokenized answer sentence output from the parameter learning unit 150 as input, and outputs an output sequence probability distribution. More specifically, during learning, the encoding unit 142 executes the following processes S1 to S3.

[0086] S1) In S1, the decoding unit 142 acquires the token sequence of the correct answer sentence output by the parameter learning unit 150 for each output step, and creates an output sequence (correct answer sentence) by combining each token sequence.

[0087] That is, during learning, the parameter learning unit 150 performs a tokenization process on the correct answer sentence, and inputs the resulting token sequence to the decoding unit 142. However, the tokenized sentence may also be input to the parameter learning unit 150.

[0088] During learning, the decoding unit 142 assigns a start symbol and a terminal symbol to the input token sequence. In this embodiment, as an example, the start symbol is <s> and the terminal symbol is < / s>.

[0089] The parameter learning unit 150 may use any method for tokenizing processing, but for example, the Byte-level BPE disclosed in "Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever: Language models are unsupervised multitask learners. Technical report, OpenAI, 2019." may be used.

[0090] S2) In S2, the decoding unit 142 inputs the encoded features and the output sequence to the transformer layer 5, and the transformer layer 5 generates a representation of the output token H=[h １ , ..., h Ｔ ], where T is the length of the output token sequence, and in one example, T=128.

[0091] S3) In S3, the decoding unit 142 decodes the representation of the output token H=[h １ , ..., h Ｔ ] is linearly transformed and then the softmax function is used to obtain the probability distribution p(y ｔ │y ＜ｔ ) is calculated, where t satisfies 0≦t≦T.

[0092] <S105: Parameter Update> The parameter learning unit 150 receives as input the correct answer sentence (correct answer sentence) and the probability distribution of the output sequence output from the decoding unit 142, and outputs parameter update information and a tokenized correct answer sentence. As described above, the tokenized correct answer sentence may be input as input to the parameter learning unit 150. Furthermore, the tokenized correct answer sentence output from the parameter learning unit 150 is input to the decoding unit 141, as described above.

[0093] The parameter learning unit 150 calculates the loss L using the following formula based on the probability distribution of the output sequence and the tokenized correct sentence, and updates the parameters of the neural network model in the learning device 100 so that the loss L is minimized.

[0094] In the above formula, y ＊ ｔ is the t-th token in the correct sentence. In other words, the above formula updates the parameters so that the probability of outputting the correct sentence is maximized.

[0095] In this embodiment, among the "image feature extraction unit 120, feature conversion unit 130, and generation unit 140" configured using a neural network, only the parameters of the feature conversion unit 130 are updated, and the parameters of the other functional units, "image feature extraction unit 120, generation unit 140", are fixed and not updated. Note that the parameters of the parameter update unit 150 may also be subject to update.

[0096] This is shown in Figure 9. As shown in Figure 9, among the parameters stored in the model DB 160, the parameters of the model corresponding to the feature conversion unit 130 are parameter 2 that are updated by learning. For parameters that are not updated by learning, parameters that have been learned in advance can be used. Note that, technically, any parameter can be updated as long as all parameters are not fixed, but in this embodiment, the learning cost is reduced by updating the parameters of only the feature conversion unit 130. The same applies to Figure 11, which will be described later.

[0097] <S106: Determination> If there is unprocessed learning data in the learning device 100, the process returns to S101 and the next learning data is used to perform the processes of S102 to S105. If there is no unprocessed learning data, the process ends.

[0098] (Configuration Example of Inference Apparatus 200) Next, a description will be given of a configuration example and an operation example of the inference apparatus 200. First, a configuration example of the inference apparatus 200 will be described.

[0099] Fig. 10 shows an example configuration of an inference device 200. As shown in Fig. 10, the inference device 200 has a text extraction unit 110, an image feature extraction unit 120, a feature conversion unit 130, and a generation unit 140. The generation unit 140 has an encoding unit 141, a decoding unit 142, and a text generation unit 143.

[0100] The text extraction unit 110, image feature extraction unit 120, and feature conversion unit 130 in the inference device 200 are the same as the text extraction unit 110, image feature extraction unit 120, and feature conversion unit 130 in the learning device 100, respectively.

[0101] Furthermore, the generation unit 140 is also basically the same between the learning device 100 and the inference device 200, but in the inference process, in addition to the encoding unit 141 and decoding unit 142, a text generation unit 143 is used.

[0102] As shown in FIG. 10, a model DB 160 is provided, and the model DB 160 stores learned parameters.

[0103] As described above, among the "image feature extraction unit 120, feature conversion unit 130, and generation unit 140," the parameters of the "image feature extraction unit 120 and generation unit 140" are parameters that have not been updated by learning in the learning device 100, while the parameters of the "feature conversion unit 130" are parameters that have been updated by learning in the learning device 100. This is shown in FIG. 11.

[0104] (Example of operation of inference device 200) Figure 12 is a flowchart showing an example of operation of the inference device 200. Below, the operation of the inference device 200 will be explained according to the procedure of the flowchart in Figure 12. Note that the "text extraction unit 110, image feature extraction unit 120, feature conversion unit 130, encoding unit 141, and decoding unit 142" in the inference device 100 are the same as those in the learning device 100, so only an outline of the operation of each unit will be described.

[0105] <S201: Input of instruction sentence and document image> In S201, an instruction sentence and a document image are input to the inference device 200. As shown in Fig. 10 , the document image is input to each of the text extraction unit 110 and the image feature extraction unit 120. In addition, the instruction sentence is input to each of the generation unit 140 and the feature conversion unit 130.

[0106] <S202: Information Extraction> In S202, the text extraction unit 110 extracts text information in the document image from the input document image, and outputs the text in the document image to the feature conversion unit 130. In addition, the image feature extraction unit 120 extracts document image features from the input document image, and outputs the document image features to the feature conversion unit 130.

[0107] <S203: Feature Conversion> In S203, the feature conversion unit 130 receives the instruction sentence, the text information in the document image output from the text extraction unit 110, and the document image features output from the image feature extraction unit 120, converts the received information into document-image conversion features, and outputs the document-image conversion features. The output document-image conversion features are input to the generation unit 140.

[0108] As in the case of learning, any method may be used as long as it can generate features that reflect “text information in a document image, document image features, and instruction sentences.” The features are input to the generation unit 140.

[0109] <S204, S205: Generation of Answer Sentence> In S204, the generation unit 140 receives the instruction sentence, the document image conversion feature, and the text information in the document image (specifically, the document text sequence) as input, and outputs the answer sentence. More specifically, the generation unit 140 executes the following process. The generation unit 140 is assumed to have the configuration shown in FIG. 8.

[0110] In the encoding unit 141, a document image conversion feature receiving unit 1 inputs a document image conversion feature, a directive receiving unit 2 inputs a directive, and a document text sequence is input by a document image text information receiving unit 3.

[0111] The transformer layer 4 transforms the "instruction sentence, document image transformation feature, and document text sequence" into encoded features and outputs the encoded features.

[0112] Next, the decoding unit 142 and the text generating unit 143 execute the following steps S1 to S4.

[0113] S1) In S1, the decoding unit 142 receives tokens recursively output for each output step from the text generation unit 143 and combines the tokens to obtain an output sequence. The decoding unit 142 also assigns only the start symbol of the sequence (e.g., <s>) to the output sequence.

[0114] S2) In S2, the decoding unit 142 inputs the encoded features and the output sequence to the transformer layer 5, and the transformer layer 5 generates a representation of the output token H=[h １ , ..., h Ｔ ]. As mentioned above, T is the length of the output token sequence, and in one example, T=128.

[0115] S3) In S3, the decoding unit 142 decodes the representation of the output token H=[h １ , ..., h Ｔ ] is linearly transformed and then the softmax function is used to obtain the probability distribution p(y ｔ │y ＜ｔ ) is calculated, where t satisfies 0≦t≦T.

[0116] S4) In S4, the text generation unit 143 receives the probability distribution of the output sequence output from the decoding unit 142, generates a response sentence based on the probability distribution, and outputs it.

[0117] Specifically, the text generator 143 recursively generates a token sequence of the answer sentence based on the probability distribution of the output sequence. More specifically, the text generator 143 selects a word that maximizes the probability distribution for each output step, or generates words by sampling according to the probability distribution, and ends generation when a sentence-end symbol is generated (Yes in S205 of FIG. 12 ).

[0118] (When Multiple Document Images are Input) As shown in task example 4 (FIG. 4), inference device 200 can also handle the case where multiple document images are input along with an instruction statement.

[0119] When multiple document images are input, the image feature extraction unit 120, text extraction unit 110, and feature conversion unit 130 (excluding the feedforward network) perform the same processing as described above for each document image. Then, the features (trainable parameter series) for the multiple document images output from the feature conversion unit 130 (excluding the feedforward network) are averaged by mean-pooling processing in the feedforward network. In this way, the averaged document image conversion features obtained from the multiple document images are input to the generation unit 140.

[0120] Furthermore, the document text sequences obtained by the text extraction unit 110 from each of the multiple document images are concatenated for each of the multiple document images and input to the generation unit 140. The operation of the generation unit 140 is the same as that described above as the operation of the inference device 200.

[0121] The same applies to the operation during learning in learning device 100: when learning data containing multiple document images is input as learning data, image feature extraction unit 120, text extraction unit 110, and feature conversion unit 130 in learning device 100 perform the same operations as image feature extraction unit 120, text extraction unit 110, and feature conversion unit 130 in the above-mentioned inference device 200. Subsequent operations are the same as those of learning device 100 described above.

[0122] (Experimental Results) An experiment was conducted using an inference device 200 (herein referred to as the proposed method) equipped with a feature transformation unit 130 that uses model parameters learned by a learning device 100. For comparison, an experiment was also conducted using fine-tuned BLIP-2 (herein referred to as fine-tuned BLIP-2).

[0123] Experiments on various tasks using data not used in training showed that the average F1 score was 43.1 for the proposed method and 39.3 for fine-tuned BLIP-2. Furthermore, the average scores for evaluation indices other than F1 were 43.0 for the proposed method and 37.0 for fine-tuned BLIP-2. This demonstrates that the technology according to this embodiment improves task processing accuracy compared to conventional technologies.

[0124] (Variations in usage) Regarding the usage of the inference device 200, for example, when using a document image such as a receipt, a photo of the receipt can be taken immediately upon receiving the receipt and the image can be uploaded to the cloud (inference device 200).

[0125] Furthermore, for example, when using a task such as extracting only the total amount from a receipt, the user may be prevented from having to input an instruction each time. For example, when an image is received from a user who has set an instruction in advance for the inference device 200, the inference device 200 generates a response using the set instruction and the received image.

[0126] (Other Configuration Examples) The learning device 100 may be configured as shown in Fig. 13. As shown in Fig. 13, the learning device 100 includes an extraction unit 210, a feature generation unit 220, and a learning unit 230. In the example of Fig. 13, the generation unit 240 is provided outside the learning device 100, but the generation unit 240 may also be provided inside the learning device 100. Furthermore, the model DB may be provided inside or outside the learning device 100. Furthermore, the extraction unit 210 may also be provided outside the learning device 100.

[0127] Note that even when the generation unit 240 is provided outside the learning device 100, a system including the learning device 100 and the generation unit 240 is referred to as a "learning device." Furthermore, when the generation unit 240 is provided outside the learning device 100, the generation unit 240 is realized, for example, as a server connected to the learning device 100 via a network. The generation unit 240 provided outside the learning device 100 may also be referred to as a "generation device."

[0128] The extraction unit 210 extracts visual information related to text from data containing the text (for example, a document image).

[0129] The feature generation unit 220 generates features (e.g., document-image conversion features) from the visual information and the first text (e.g., instructional sentence), and inputs the features to the generation unit 240. The extraction unit 210 may input the text to the generation unit 240.

[0130] The learning unit 230 learns model parameters of the neural network that constitutes the feature generation unit 220 using information output from the generation unit 240 to which the features and the first text are input, and the data and a second text that is a correct answer to the first text.

[0131] Note that the "text extraction unit 110 and image feature extraction unit 120" shown in Fig. 6 are examples of the extraction unit 210. The feature conversion unit 130 shown in Fig. 6 is an example of the feature generation unit 220. The generation unit 140 shown in Fig. 6 is an example of the generation unit 240. The parameter learning unit 150 shown in Fig. 6 is an example of the learning unit 130.

[0132] Inference device 200 may be configured as shown in Fig. 14. As shown in Fig. 14, inference device 200 includes extraction unit 210 and feature generation unit 220. In the example of Fig. 14, generation unit 240 is provided outside inference device 200, but generation unit 240 may also be provided inside inference device 200. Furthermore, model DB may be provided inside or outside inference device 200. Furthermore, extraction unit 210 may also be provided outside inference device 200.

[0133] Note that even when generation unit 240 is provided outside inference device 200, a system including inference device 200 and generation unit 240 is referred to as an "inference device." Furthermore, when generation unit 240 is provided outside inference device 200, generation unit 240 is realized, for example, as a server connected to inference device 200 via a network. Generation unit 240 provided outside inference device 200 may also be referred to as a "generation device."

[0134] The extraction unit 210 extracts visual information related to text from data containing the text (for example, a document image).

[0135] The feature generation unit 220 generates features from the visual information and a first text (e.g., an instructional sentence) and inputs the features to the generation unit 240. The extraction unit 210 may input the text to the generation unit 240. The generation unit 240, to which the features and the first text have been input, outputs a second text generated based on the features and the first text.

[0136] 9 is an example of the extraction unit 210. The feature conversion unit 130 shown in FIG. 9 is an example of the feature generation unit 220. The generation unit 140 shown in FIG. 9 is an example of the generation unit 240.

[0137] (Hardware Configuration Example) Any of the devices described in this embodiment (learning device 100, inference device 200) can be realized, for example, by causing a computer to execute a program. This computer may be a physical computer or a virtual machine on the cloud.

[0138] That is, the device can be realized by executing a program corresponding to the processing performed by the device using hardware resources such as a CPU and memory built into a computer. The program can be recorded on a computer-readable recording medium (such as a portable memory) and stored or distributed. The program can also be provided via a network such as the Internet or email.

[0139] Fig. 15 is a diagram showing an example of the hardware configuration of the computer. The computer in Fig. 15 includes a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, and the like, all of which are interconnected by a bus BS. The computer may further include a GPU.

[0140] The program that realizes the processing on the computer is provided by a recording medium 1001, such as a CD-ROM or a memory card. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001, but may be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files, data, etc.

[0141] The memory device 1003 reads and stores a program from the auxiliary storage device 1002 when an instruction to start the program is received. The CPU 1004 realizes functions related to the device in accordance with the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network, etc. The display device 1006 displays a GUI (Graphical User Interface) or the like according to the program. The input device 1007 is composed of a keyboard, mouse, buttons, a touch panel, etc., and is used to input various operation instructions. The output device 1008 outputs the results of calculations.

[0142] (Regarding the System) In this embodiment, the system shown in FIG. 16 may be used. This system includes a user device 501, a server 502, a database 503, and a network 504. Note that server 502 is, for example, inference device 200. Server 502 may also be learning device 100. Server 502 may also be a device that includes both learning device 100 and inference device 200. User device 501, server 502, and database 503 may all be the computers shown in FIG. 15.

[0143] The user device 501 is a terminal through which a user interacts with the system, and includes devices such as a smartphone, tablet, personal computer, etc. The user device 501 receives input from the user and displays output from the system via an interface including an input unit, a display unit, and a communication unit.

[0144] The server 502 is the core processing device of the system, receiving requests from the user device 501 and executing the corresponding processing. The server 502 includes at least a processor, memory, and storage, and provides various functions by executing program code. The server 502 may be composed of multiple physical or virtual servers that function as web servers, application servers, and database servers.

[0145] The network 504 is a communication infrastructure that connects the user device 501 and the server 502, and includes the Internet, a local network (LAN), a wide area network (WAN), and combinations thereof. The network 504 provides both wired and wireless communication means and enables data transmission and reception.

[0146] The database 503 is responsible for storing data in this system. The database 503 can be realized using any database technology, such as a relational database, an object database, or a NoSQL database. For example, the database 503 stores trained models trained by the learning device 100. For example, the server 502 serving as the inference device 200 obtains and uses the models from the database 503.

[0147] An example of the operation flow in this system is as follows:

[0148] In S501, a user starts an operation via a user device 501 and sends a request to a server 502 via a network 504. This request includes, for example, a file of explanatory text with illustrations and an instruction (question, etc.) from the user.

[0149] In S502, the server 502 receives the request and performs the necessary processing. For example, the server 502 as the inference device 200 generates an answer to the question and an explanation of the diagram in response to the request.

[0150] In S503, the server 502 notifies the user device 501 of the results of the above processing.

[0151] In S504, the user device 501 displays the results received from the server 502 to the user.

[0152] When the server 502 is the learning device 100, for example, learning data is stored in the database 503. In this case, when a learning instruction is sent from the user device 501 to the server 502, the server 502 reads the learning data from the database 503 and uses the learning data to learn a model. The learned model is stored in the database 503, for example.

[0153] (Effects of the Embodiment) As described above, the technology described in the present embodiment makes it possible to accurately acquire information about data including text from the data.

[0154] More specifically, when a document image is used as the data, the document image can be understood as a knowledge source, and a response to a user's instruction can be obtained with high accuracy.

[0155] Furthermore, in the technology according to the present embodiment, the parameters of the generation unit 140 (specifically, the LLM) are fixed, and the desired function can be achieved by training only the feature conversion unit 130, thereby shortening the training time and reducing the training cost.

[0156] The following additional notes are provided regarding the above-described embodiments.

[0157] <Additional Notes> (Additional Item 1) An inference device that generates features of data including text based on a first text and the data to input to a sentence generation model that outputs a second text, the inference device comprising: a memory; and at least one processor connected to the memory, wherein the processor generates the features from visual information related to the text and the first text, which is visual information extracted from the data, based on a trained model, and outputs the features to be input to the sentence generation model. (Additional Item 2) The inference device according to Additional Item 1, wherein the visual information includes region information indicating a region of the text in the data and image features in the data, and the processor generates the features from the visual information, third text extracted from the data, and the first text. (Additional Item 3) The inference device according to Additional Item 2, wherein the processor generates the features using information weighted on the text and information weighted on the region information. (Supplementary Item 4) A learning device including: a memory; and at least one processor connected to the memory, wherein the processor performs: a feature generation process of generating features from visual information related to the text, the visual information being extracted from data including text, and a first text, and outputting the features for input to a sentence generation model; and a learning process of learning model parameters of a neural network that performs the feature generation process, using information output from the sentence generation model to which the features and the first text have been input, and the data and a second text that is a correct answer for the first text. (Supplementary Item 5) An inference method executed by an inference device, comprising: a step of generating features from visual information related to the text, the visual information being extracted from data including text, and the first text, based on a trained model, and outputting the features for input to a sentence generation model, wherein the sentence generation model outputs second text based on the features and the first text. (Supplementary Item 6) A non-transitory storage medium storing a program for causing a computer to function as the inference device according to any one of Supplementary Items 1 to 3.(Supplementary Item 7) A non-transitory storage medium storing a program for causing a computer to function as the learning device according to Supplementary Item 4.

[0158] Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims.

[0159] This patent application claims priority based on Japanese Patent Application No. 2024-036356 filed on March 8, 2024, and Japanese Patent Application No. 2024-091632 filed on June 5, 2024. The entire contents of Japanese Patent Application No. 2024-036356 and Japanese Patent Application No. 2024-091632 are incorporated herein by reference.

[0160] 100 Learning device 110 Text extraction unit 120 Image feature extraction unit 130 Feature conversion unit 140 Generation unit 141 Encoding unit 1 Document image conversion feature reception unit 2 Instruction sentence reception unit 3 Text information reception unit in document image 4 Conversion layer 142 Decoding unit 5 Conversion layer 6, 143 Text generation unit 150 Parameter learning unit 160 Model DB 200 Inference device 210 Extraction unit 220 Feature generation unit 230 Learning unit 240 Generation unit 501 User device 502 Server 503 Database 504 Network 1000 Drive device 1001 Recording medium 1002 Auxiliary storage device 1003 Memory device 1004 CPU 1005 Interface device 1006 Display device 1007 Input device 1008 Output device

Claims

1. An inference device comprising: visual information extracted from data including text, the visual information relating to the text, and a first text, which generates features based on a trained model, and outputs the features to be input to a generation unit, the generation unit outputting a second text based on the features and the first text.

2. The inference device described in claim 1, wherein the visual information includes area information indicating an area of ​​the text in the data and image features in the data, and the feature generation unit generates the features based on the trained model from the visual information, third text extracted from the data, and the first text.

3. The inference device according to claim 2, wherein the feature generation unit generates the features using weighted information on the text and weighted information on the region information.

4. A learning device comprising: a feature generation unit that generates features from visual information about the text and a first text, the visual information being extracted from data including text, and outputs the features to be input to a generation unit; and a learning unit that learns model parameters of a neural network that constitutes the feature generation unit using information output from the generation unit to which the features and the first text have been input, and the data and a second text that is a correct answer to the first text.

5. An inference method executed by an inference device, the inference method comprising: generating features from visual information about the text extracted from data including text and a first text; and outputting the features to be input to a generation unit; and the generation unit outputting a second text based on the features and the first text.

6. A program for causing a computer to function as the feature generation unit in the inference device according to any one of claims 1 to 3.

7. A program for causing a computer to function as the feature generation unit and the learning unit in the learning device according to claim 4.

Images