Information processing system, information processing method, and program

The information processing system addresses the challenge of extracting language features from complex images by using neural networks to combine and enhance context understanding, improving feature extraction accuracy in images with unclear reading orders and non-consecutive contexts.

JP2024157784AInactive Publication Date: 2024-11-08KK TOSHIBA +1
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2023072361
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-04-26
Publication Date
2024-11-08
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Conventional technologies face challenges in maintaining the quality of language features extracted from images containing multiple sentences with unclear reading orders or non-consecutive contexts due to character recognition issues.

Method used

An information processing system that includes an input section, token conversion, estimation, auxiliary token string generation, combined token string generation, and feature vector conversion using neural networks to enhance context understanding and improve feature extraction accuracy.

Benefits of technology

The system effectively improves the quality of language features extracted from images with complex sentence structures by combining target and auxiliary token strings, enhancing accuracy even when reading orders are ambiguous or contexts are not consecutive.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024157784000001_ABST
    Figure 2024157784000001_ABST
Patent Text Reader

Abstract

To improve quality of a language feature obtained from a character recognition result.SOLUTION: An information processing system of an embodiment includes an input unit, a token converting unit, an estimation unit, an auxiliary token string generation unit, a combined token string generation unit, and a feature vector conversion unit. The input unit accepts input of an input text and position information. The token converting unit converts the input text to a token string group on the basis of a lexical dictionary which determines division units of the token. The estimation unit estimates structured information indicating a structure of the token string group on the basis of the position information. The auxiliary token string generation unit generates an auxiliary token string for each target token string contained in the token string group on the basis of the structured information. The combined token string generation unit generates a combined token string by combining the target token string and the auxiliary token string. The feature vector conversion unit converts the combined token string to a feature vector indicating context information of the target token string by using a neural network.SELECTED DRAWING: Figure 2
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] An embodiment of the present invention relates to an information processing system, an information processing method, and a program. [Background technology]

[0002] There is a conventional technology that automatically extracts specified items from an image containing text (e.g., a form image) based on linguistic features obtained from the results of character recognition, without predefining the positions of the items to be read. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2022-096748 [Non-patent literature]

[0004] [Non-Patent Document 1] Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,[online],[Retrieved April 10, 2022], Internet〈URL:https: / / arxiv.org / pdf / 1810.04805.pdf〉 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the conventional technology has a problem that the quality of the language features obtained from the character recognition results is degraded. For example, when an image contains multiple sentences whose reading order is not self-evident or multiple sentences whose contexts are not consecutive, the quality of the language features obtained from the character recognition results is degraded. [Means for solving the problem]

[0006] An information processing system according to an embodiment includes an input unit, a token conversion unit, an estimation unit, an auxiliary token sequence generation unit, a combined token sequence generation unit, and a feature vector conversion unit. The input unit receives an input of an input text including at least one sentence included in an image and position information of the sentence. The token conversion unit converts the input text into a token sequence group based on a vocabulary dictionary in which a unit of token division is defined. The estimation unit estimates structured information indicating a structure of the token sequence group based on the position information. The auxiliary token sequence generation unit generates an auxiliary token sequence for estimating context information of the target token sequence based on the structured information for each target token sequence included in the token sequence group. The combined token sequence generation unit generates a combined token sequence by combining the target token sequence and the auxiliary token sequence. The feature vector conversion unit converts the combined token sequence into a feature vector indicating context information of the target token sequence using a neural network including at least one neural network layer. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram illustrating an example of extraction of designated items by the information processing system according to the embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of a configuration of an information processing system according to the embodiment. [Diagram 3] FIG. 3 is a diagram illustrating an example of an image processed by the information processing system according to the embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of a neural network language model used in the auxiliary token sequence generating unit of the embodiment. [Diagram 5] FIG. 5 is a diagram illustrating an example of processing by the feature vector conversion unit and the classification unit according to the embodiment. [Figure 6] FIG. 6 is a diagram for explaining an example of extraction of language features according to the embodiment. [Figure 7] FIG. 7 is a flowchart illustrating an example of an information processing method according to the embodiment. [Figure 8] FIG. 8 is a diagram for explaining an example of extraction of language features according to the embodiment. [Figure 9] FIG. 9 is a diagram illustrating an example of a hardware configuration of the information processing system according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, embodiments of an information processing system, an information processing method, and a program will be described in detail with reference to the accompanying drawings.

[0009] The information processing system of the embodiment automatically extracts the specified item from the character recognition result without predefining the position of the item to be read.

[0010] 1 is a diagram showing an example of extraction of designated items by the information processing system according to the embodiment, in which a symbol string of an invoice number and a company name are extracted from an image of a document representing a form.

[0011] In the information processing system of the embodiment, the item determination is defined as a class classification problem for each word (token), and the specified item is extracted by classifying the language features extracted using a DNN (Deep Neural Network) language model using a classifier. For example, BERT (Bidirectional Encoder Representations from Transformers) is used as the DNN language model. For example, LSTM (Long short-term memory) is used as the classifier.

[0012] [Example of functional configuration] 2 is a diagram illustrating an example of a configuration of an information processing system 10 according to an embodiment. The information processing system 10 according to the embodiment includes an input unit 1, a token conversion unit 2, an estimation unit 3, an auxiliary token sequence generation unit 4, a combined token sequence generation unit 5, a feature vector conversion unit 6, and a classification unit 7.

[0013] The information processing system 10 of the embodiment includes an input text storage unit 101, a position information storage unit 102, a token sequence group storage unit 103, a structured information storage unit 104, an auxiliary token sequence storage unit 105, and a combined token sequence storage unit 106.

[0014] The input text memory unit 101, the positional information memory unit 102, the token sequence group memory unit 103, the structured information memory unit 104, the auxiliary token sequence memory unit 105 and the combined token sequence memory unit 106 are realized, for example, by at least one memory device provided in the information processing system 10.

[0015] The input text storage unit 101 stores input text including at least one sentence included in an image. The position information storage unit 102 stores position information (for example, coordinate information) of at least one sentence included in an image.

[0016] The input unit 1 accepts input of an input text including at least one sentence included in an image and position information (for example, coordinate information) of the sentence. The sentence included in the input text is, for example, a character string, a number, a symbol, and the like.

[0017] Fig. 3 is a diagram showing an example of an image processed by the information processing system 10 according to the embodiment. The example in Fig. 3 shows a case where sentences are included in 17 places in the image. In this case, the input unit 1 accepts input of an input text including 17 sentences and position information of the 17 sentences.

[0018] The input unit 1 may further have a function of receiving an image and identifying the input text and position information contained in the image.

[0019] Returning to Fig. 2, the token conversion unit 2 divides each sentence into token units. For example, a "token conversion unit" is divided into a token sequence such as "token, conversion, unit". The token conversion unit 2 stores each token sequence converted from each sentence in the token sequence group storage unit 103.

[0020] The estimation unit 3 sets each token sequence obtained by the token conversion unit 2 as a target token sequence to be processed. For one target token sequence, the estimation unit 3 estimates structured information indicating candidates for a paired token sequence using position information. For example, the estimation unit 3 divides a sentence included in an image into a plurality of blocks using a learning or rule-based method. For example, in the example of FIG. 3, the sentence included in an image is divided into three blocks, such as 1-3, 4-5, and 6-17. The estimation unit 3 creates pairs of token sequences within each block.

[0021] Alternatively, the estimation unit 3 may treat the entire image as one block and create pairs of a target token sequence and all token sequences other than the target token sequence.

[0022] The estimation unit 3 stores the estimated structured information in the structured information storage unit 104 .

[0023] The auxiliary token sequence generation unit 4 generates an auxiliary token sequence that estimates context information of a target token sequence based on the structured information for each target token sequence included in the token sequence group. Specifically, the auxiliary token sequence generation unit 4 extracts a token sequence that has a semantic connection with the target token sequence as an auxiliary token sequence. Specifically, for each pair created by the estimation unit 3, a neural network language model is constructed (or fine-tuned) that determines whether or not the two token sequences have a semantic connection.

[0024] The neural network language model used for the determination is, for example, Word2vec, CNN, BERT, etc. The neural network language model used for the determination may be the same as the language model used in the feature vector conversion unit 6 described below, or may be a different model from the language model used in the feature vector conversion unit 6. When the language model used in the feature vector conversion unit 6 is the same model, the generation of the auxiliary token sequence, the generation of the combined token sequence, and the generation of the feature vector may be performed simultaneously (see FIG. 4).

[0025] 4 is a diagram showing an example of a neural network language model used in the auxiliary token sequence generation unit 4 of the embodiment. [CLS] is a token indicating the start of an auxiliary token sequence. tok1 to tokA indicate candidates for an auxiliary token sequence. [SEP] is a token indicating the end of an auxiliary token sequence. Tok1 to TokA indicate a target token sequence. Also, att indicates an attention network layer.

[0026] 4, the output corresponding to [CLS] indicates an auxiliary token sequence, which is used as the output of the auxiliary token sequence generation unit 4. Moreover, the outputs corresponding to Tok1 to TokA indicate feature vectors, which are used as the output of the feature vector conversion unit 6, which will be described later.

[0027] Returning to FIG. 2, the auxiliary token sequence generator 4 stores the generated auxiliary token sequence in the auxiliary token sequence storage unit 105.

[0028] The combined token sequence generator 5 generates a combined token sequence by combining a target token sequence and an auxiliary token sequence. An auxiliary token sequence that is determined to have a semantic connection with the target token sequence is combined with the target token sequence by the combined token sequence generator 5.

[0029] When there are multiple auxiliary token sequences determined to have a semantic connection with the target token sequence, the combined token sequence generation unit 5 selects one auxiliary token sequence from the multiple auxiliary token sequences using a predetermined method and combines it with the target token sequence. For example, the combined token sequence generation unit 5 selects the auxiliary token sequence with the highest output from the auxiliary token sequence generation unit 4 in the neural network language model of Fig. 4 described above (the auxiliary token sequence with the deepest semantic connection) and combines it with the target token sequence.

[0030] Furthermore, for example, when there are a plurality of auxiliary token sequences that are determined to have a semantic connection with the target token sequence, the combined token sequence generation unit 5 may combine the plurality of auxiliary token sequences with the target token sequence.

[0031] The combined token string generation unit 5 may insert a delimiter token between the auxiliary token string and the target token string. Furthermore, when multiple auxiliary token strings are combined with the target token string, the combined token string generation unit 5 may insert a delimiter token between the multiple auxiliary token strings.

[0032] The combined token sequence generating unit 5 stores the generated combined token sequence in the combined token sequence storage unit 106 .

[0033] The feature vector conversion unit 6 converts each combined token string combined by the combined token string generation unit 5 into a feature vector using a neural network language model. For the neural network language model, for example, a model (such as Word2vec, CNN, and BERT) in which conversion rules from token strings to feature vectors are learned using external data is used.

[0034] In the example of the neural network language model in FIG. 4 described above, the model is fine-tuned so as to output the feature vector portion of the target token sequence from the combined token sequence, or to output the feature vector portion of the target token sequence from the combined token sequence.

[0035] The classification unit 7 uses LSTM to classify the language features based on the language features indicated by the feature vector converted by the feature vector conversion unit 6. Each class indicates, for example, an invoice number or a company name. From the classification results by the classification unit 7, the invoice number and the company name contained in the image are identified (see FIG. 1 above).

[0036] 5 is a diagram showing an example of processing by the feature vector conversion unit 6 and the classification unit 7 according to the embodiment. A 1,T A 2,…,T A n][A1,A2,…,An], combined token sequence [T B 1,T B 2,…,T B n][B1,B2,…,Bn] and the combined token sequence [T c 1,Tc 2,…,T c n][C1,C2,…,Cn] is a combined token sequence generated from the sentences written at three different positions in the image.

[0037] [T A 1,T A 2,…,T A n] denotes the auxiliary token sequence of the target token sequence [A1, A2, ..., An]. B 1,T B 2,…,T B n] denotes the auxiliary token sequence of the target token sequence [B1, B2, ..., Bn]. c 1,T c 2,…,T c n] denotes the auxiliary token sequence of the target token sequence [C1, C2, ..., Cn].

[0038] The feature vector conversion unit 6 converts the combined token sequence [T A 1,T A 2,…,T A n][A1, A2, ..., An], the feature vector conversion unit 6 outputs a feature vector indicating the linguistic features of the target token sequence [A1, A2, ..., An]. Similarly, the feature vector conversion unit 6 converts the combined token sequence [T B 1,T B 2,…,T B n][B1, B2, ..., Bn], the feature vector conversion unit 6 outputs a feature vector indicating the linguistic features of the target token sequence [B1, B2, ..., Bn]. c 1,T c 2,…,T c n][C1, C2, …, Cn], output a feature vector indicating the linguistic features of the target token sequence [C1, C2, …, Cn].

[0039] The classification unit 7 performs class classification of the language features of the target token sequences [A1, A2, ..., An], [B1, B2, ..., Bn], and [C1, C2, ..., Cn] using LSTM. In the classification unit 7 of Fig. 5, the language features of each target token sequence output by the feature vector conversion unit 6 are combined and then input to the classification unit 7, but classification may be performed using each language feature without combining them.

[0040] Fig. 6 is a diagram for explaining an example of extraction of language features in the embodiment. The example in Fig. 6 shows a case where language features of sentences (including numbers and symbols) included in a form are extracted. For normal texts such as articles, papers, and novels, the sentences can be input consecutively into a feature extractor (such as BERT) to obtain features, but forms have the following problems compared to normal texts. (1) The reading order is not self-evident (2) The context is not continuous

[0041] For example, if sentences with incorrect reading order or contextually unrelated sentences are input together into the feature extractor, the quality of the linguistic features will be degraded. For example, sentence 201 (a number in the example of Figure 6) is related to sentences 202 and 203, but not to the adjacent numbers.

[0042] According to the embodiment of the information processing system 10, by using a combined token string that combines a target token string and an auxiliary token string, it is possible to prevent a decrease in the accuracy of extracting language features, even when extracting language features from a document as shown in Figure 6.

[0043] 7 is a flowchart showing an example of an information processing method according to the embodiment, which will be described by taking as an example a case where linguistic features are extracted from the form shown in FIG.

[0044] First, the input unit 1 accepts an input of an input text including a plurality of sentences contained in a form and position information of the plurality of sentences (step S1).

[0045] Next, the token conversion unit 2 converts the input text into a token sequence group based on a vocabulary dictionary in which a division unit of tokens is defined (step S2). For example, the sentence 202 is converted into the tokens [“this month”, “of”, “unit price”] based on a predetermined vocabulary dictionary.

[0046] Next, the estimation unit 3 estimates structured information indicating the structure of a group of token sequences based on the positional information of the multiple sentences (step S3). In the example of Fig. 6, for example, structured information indicating that sentence 201 is a sentence in one cell of a table included in the form is estimated. Also, for example, structured information indicating that sentences 202 and 203 are sentences explaining items in a table included in the form is estimated.

[0047] Next, the auxiliary token sequence generation unit 4 generates an auxiliary token sequence that estimates context information of the target token sequence based on the structured information estimated in step S3 for each target token sequence included in the token sequence group converted in step S2 (step S4).

[0048] Next, the combined token sequence generator 5 generates a combined token sequence by combining the target token sequence and the auxiliary token sequence generated in step S4 (step S5).

[0049] Next, the feature vector conversion unit 6 converts the combined token sequence generated in step S5 into a feature vector representing context information of the target token sequence, using a neural network including at least one neural network layer (step S6).

[0050] Fig. 8 is a diagram for explaining an example of linguistic feature extraction in an embodiment. The example in Fig. 8 shows a case where linguistic features of sentence 201 are extracted from the form in Fig. 6. ["Orange", "This month's unit price", [T], "200"] indicates a combined token string. ["200"] indicates a target token string. In the example of sentence 201, the target token string contains one token. ["Orange", "This month's unit price"] indicates an auxiliary token string of the target token string, and corresponds to sentences 202 and 203 in Fig. 6. [T] indicates a separator token.

[0051] As shown in Figure 8, when determining the linguistic features of a target token sequence, by adding an auxiliary token sequence that complements the context, it is possible to prevent a decrease in the accuracy of linguistic feature extraction, even when extracting linguistic features from a form, as shown in Figure 6.

[0052] Note that the auxiliary token sequence does not necessarily have to be an actual token sequence in the document, and metadata converted from the actual token sequence according to a predetermined rule may be included in the auxiliary token sequence (see variant example 1 of the embodiment described below).

[0053] 7, the classification unit 7 classifies the language features of the sentence 201 using LSTM from the language features indicated by the feature vector converted in step S6 (step S7). In the example of the sentence 201 in Fig. 6, for example, it is classified into a class (context information) indicating that it is numerical information on the unit price of mandarin oranges this month.

[0054] As described above, in the information processing system 10 of the embodiment, the input unit 1 accepts input of an input text including at least one sentence included in an image and position information of the sentence. The token conversion unit 2 converts the input text into a token sequence group based on a vocabulary dictionary in which a division unit of tokens is defined. The estimation unit 3 estimates structured information indicating a structure of the token sequence group based on the position information. The auxiliary token sequence generation unit 4 generates an auxiliary token sequence for estimating context information of the target token sequence based on the structured information for each target token sequence included in the token sequence group. The combined token sequence generation unit 5 generates a combined token sequence by combining the target token sequence and the auxiliary token sequence. Then, the feature vector conversion unit 6 converts the combined token sequence into a feature vector representing context information of the target token sequence using a neural network including at least one neural network layer.

[0055] As a result, the information processing system 10 according to the embodiment can improve the quality of language features obtained from the result of character recognition. For example, even if an image contains multiple sentences whose reading order is not self-evident or multiple sentences whose contexts are not consecutive, the quality of language features obtained from the result of character recognition can be improved.

[0056] (First Modification of the Embodiment) Next, a first modification of the embodiment will be described. In the description of the first modification, the same description as in the embodiment will be omitted, and only the differences from the embodiment will be described.

[0057] For example, the image to be processed may include multiple sentences having a document structure, in which case the target token sequence is part of the multiple sentences having the document structure.

[0058] The estimation unit 3 may analyze the document structure from each target token sequence and estimate structured information indicating the document structure of a sentence (e.g., a character string) corresponding to each target token sequence. For example, if the document has a chapter-section-paragraph structure, the estimation unit 3 analyzes whether a delimited sentence corresponds to a chapter, section, or paragraph, or whether it does not correspond to any of them.

[0059] For example, the analysis method may be a machine learning-based analysis method that uses a predetermined language model to output labels that estimate the document structure. Also, for example, the analysis method may be rule-based (e.g., based on the number at the beginning of a sentence, the size of the characters, the length of the indentation, etc.).

[0060] The auxiliary token sequence generation unit 4 generates an auxiliary token sequence that assigns label tokens to a target token sequence based on structuring information indicating a document structure. For example, the auxiliary token sequence generation unit 4 converts the document structure of the target token sequence into label tokens that identify the document structure according to a predetermined conversion rule, and generates an auxiliary token sequence including the label tokens.

[0061] Specifically, if the document has a chapter / section / paragraph structure, the auxiliary token sequence generation unit 4 generates an auxiliary token sequence including one of four types of label tokens: chapter, section, paragraph, or not applicable. For example, if the document corresponds to a chapter, section, or paragraph, the auxiliary token sequence generation unit 4 may also assign a number (chapter number, section number, paragraph number). Alternatively, if the document has headings with aligned indents, the auxiliary token sequence generation unit 4 may assign different label tokens depending on the length of the indent.

[0062] The feature vector conversion unit 6 converts the target token sequence, out of the combined token sequence, into a feature vector based on the auxiliary token sequence. In the first modification, since the auxiliary token sequence includes a label token, metadata indicating the label token is converted into a feature vector.

[0063] Specifically, in the case of a document with a chapter-section-paragraph structure, for example, the feature vector is a one-dimensional vector where chapters are 4, sections are 3, paragraphs are 2, and other sentences are 1. In another example, the feature vector is a four-dimensional one-hot vector of chapters, sections, paragraphs, and other sentences. In another example, the feature vector may be a three-dimensional vector where the values ​​of each dimension indicate the chapter number, for example, for section 6.3, the first dimension is 6, the second dimension is 3, and the third dimension is 0. In another example, in the case of a document with headings with aligned indentations, the feature vector may be a vector corresponding to the length of the indentation.

[0064] (Modification 2 of the embodiment) Next, a second modification of the embodiment will be described. In the description of the second modification, the same description as in the embodiment will be omitted, and only the differences from the embodiment will be described. The target token sequence and the auxiliary token sequence of the embodiment may be generated in a key-value format consisting of a value, which is a value having some meaning, and a key, which is information for identifying the value.

[0065] For example, the auxiliary token sequence generating unit 4 may generate a target token sequence as at least one piece of value information, and generate the auxiliary token sequence as a key that explains the at least one piece of value information. Conversely, the auxiliary token sequence generating unit 4 may generate an auxiliary token sequence as at least one piece of value information, and generate a target token sequence as a key that explains the at least one piece of value information.

[0066] (Modification 3 of the embodiment) Next, a third modification of the embodiment will be described. In the description of the third modification, the same description as in the embodiment will be omitted, and only the differences from the embodiment will be described. In addition to the examples of the embodiment described above, examples of the target token sequence and the auxiliary token sequence include, for example, the following specific examples.

[0067] For example, the target token sequence may be all or a part of data in at least one cell included in a table. In this case, the auxiliary token sequence generator 4 generates an auxiliary token sequence including all or a part of at least one heading given to the table. Conversely, the target token sequence may be all or a part of at least one heading given to the table. In this case, the auxiliary token sequence generator 4 generates an auxiliary token sequence including all or a part of data in at least one cell included in the table.

[0068] Also, for example, the target token sequence may be a part of a sentence. In this case, the auxiliary token sequence generation unit 4 generates an auxiliary token sequence including all or a part of token sequences belonging to the same sentence as the target token sequence.

[0069] For example, the target token sequence may be all or part of a sentence citing a figure or table. In this case, the auxiliary token sequence generation unit 4 generates an auxiliary token sequence including all or part of a title given to the figure or table. Conversely, the target token sequence may be all or part of a title given to a figure or table. In this case, the auxiliary token sequence generation unit 4 generates an auxiliary token sequence including all or part of a sentence citing a figure or table.

[0070] For example, the target token sequence may be all or part of the title of a form or a chapter. In this case, the auxiliary token sequence generation unit 4 generates an auxiliary token sequence including all or part of the data corresponding to the form or chapter. Conversely, the target token sequence may be all or part of the data corresponding to the form or chapter. In this case, the auxiliary token sequence generation unit 4 generates an auxiliary token sequence including all or part of the title of the form or chapter.

[0071] Finally, an example of the hardware configuration of the information processing system 10 according to the embodiment will be described.

[0072] [Example of hardware configuration] 9 is a diagram illustrating an example of a hardware configuration of the information processing system 10 according to the embodiment. The information processing system 10 according to the embodiment includes a processor 301, a main storage device 302, an auxiliary storage device 303, a display device 304, an input device 305, and a communication device 306. The processor 301, the main storage device 302, the auxiliary storage device 303, the display device 304, the input device 305, and the communication device 306 are connected via a bus 310.

[0073] Note that the information processing system 10 may not include some of the above configurations. For example, when the information processing system 10 can use the input function and display function of an external device, the information processing system 10 may not include the display device 304 and the input device 305.

[0074] The processor 301 executes a program read from the auxiliary storage device 303 to the main storage device 302. The main storage device 302 is a memory such as a read only memory (ROM) and a random access memory (RAM). The auxiliary storage device 303 is a hard disk drive (HDD), a memory card, or the like.

[0075] The display device 304 is, for example, a liquid crystal display. The input device 305 is an interface for operating the information processing system 10. The display device 304 and the input device 305 may be realized by a touch panel or the like having a display function and an input function. The communication device 306 is an interface for communicating with other devices.

[0076] For example, the program executed by the information processing system 10 is provided as a computer program product in the form of a file in an installable or executable format and recorded on a computer-readable storage medium such as a memory card, a hard disk, a CD-RW, a CD-ROM, a CD-R, a DVD-RAM, or a DVD-R.

[0077] Also, for example, the programs executed by the information processing system 10 may be configured to be stored on a computer connected to a network such as the Internet, and provided by being downloaded via the network.

[0078] Also, for example, the programs executed by the information processing system 10 may be provided via a network such as the Internet without being downloaded. Specifically, for example, the programs may be configured as an ASP (Application Service Provider) type cloud service.

[0079] Also, for example, the program for the information processing system 10 may be configured to be provided by being pre-installed in a ROM or the like.

[0080] The programs executed by the information processing system 10 are modularized to include functions that can be realized by the programs among the above-mentioned functional configurations. As for each function, as actual hardware, the processor 301 reads the program from a storage medium and executes it, and the above-mentioned functional blocks are loaded onto the main storage device 302. In other words, the above-mentioned functional blocks are generated on the main storage device 302.

[0081] Note that some or all of the above-described functions may be realized by hardware such as an integrated circuit (IC) instead of by software.

[0082] Furthermore, each function may be realized using a plurality of processors 301. In this case, each processor 301 may realize one of the functions, or may realize two or more of the functions.

[0083] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0084] 1 Input section 2 Token conversion section 3 Estimation part 4. Auxiliary token sequence generator 5. Combined token sequence generator 6 Feature Vector Conversion Unit 7 Classification section 10. Information Processing Systems 101 input text storage unit, 102 Location information storage unit, 103 token sequence group storage unit, 104 Structured information storage unit, 105 Auxiliary token string storage unit 106 Combined token string storage unit 301 Processor 302 Main storage 303 Auxiliary storage device 304 Display device 305 Input Device 306 Communication Equipment 310 Bus

Claims

1. an input unit that receives an input of an input text including at least one sentence included in an image and position information of the sentence; a token conversion unit that converts the input text into a group of token sequences based on a vocabulary dictionary in which a unit of token division is defined; an estimation unit that estimates structured information indicating a structure of the token sequence group based on the position information; an auxiliary token sequence generation unit that generates, for each target token sequence included in the token sequence group, an auxiliary token sequence that estimates context information of the target token sequence based on the structured information; a combined token sequence generator for generating a combined token sequence by combining the target token sequence and the auxiliary token sequence; a feature vector conversion unit that converts the combined token sequence into a feature vector that represents context information of the target token sequence using a neural network including at least one neural network layer; An information processing system comprising:

2. the image includes a plurality of sentences having a document structure; the target token sequence is a portion of a plurality of sentences having the document structure; the auxiliary token sequence generation unit converts a document structure of the target token sequence into label tokens that identify the document structure according to a predetermined conversion rule, and generates an auxiliary token sequence including the label tokens. The information processing system according to claim 1 .

3. the auxiliary token sequence generator generates the target token sequence as at least one piece of value information, and generates the auxiliary token sequence as a key explaining the at least one piece of value information; 3. The information processing system according to claim 1 or 2.

4. the auxiliary token sequence generator generates the auxiliary token sequence as at least one piece of value information, and generates the target token sequence as a key explaining the at least one piece of value information; 3. The information processing system according to claim 1 or 2.

5. the target token string is all or a portion of data in at least one cell of a table; The auxiliary token sequence generator generates an auxiliary token sequence including all or a part of at least one heading given to the table.

3. The information processing system according to claim 1 or 2.

6. the target token string is all or a portion of at least one heading of a table; The auxiliary token sequence generator generates an auxiliary token sequence including all or a part of data in at least one cell included in the table.

3. The information processing system according to claim 1 or 2.

7. the target token sequence is a portion of a sentence, the auxiliary token sequence generation unit generates an auxiliary token sequence including all or a part of a token sequence belonging to the same sentence as the target token sequence; 3. The information processing system according to claim 1 or 2.

8. the target token string is a whole or part of a sentence citing a figure or a table; The auxiliary token sequence generation unit generates an auxiliary token sequence including all or a part of a title given to the figure or table.

3. The information processing system according to claim 1 or 2.

9. the target token string is a whole or a part of a title given to a figure or a table, the auxiliary token sequence generation unit generates an auxiliary token sequence including all or a part of a sentence citing the figure or table.

3. The information processing system according to claim 1 or 2.

10. the target token string is a whole or a part of a document or chapter title, the auxiliary token sequence generation unit generates an auxiliary token sequence including all or a part of the data corresponding to the form or chapter.

3. The information processing system according to claim 1 or 2.

11. the target token sequence is all or a part of data corresponding to a form or a chapter; The auxiliary token sequence generating unit generates an auxiliary token sequence including all or a part of the title of the document or chapter.

3. The information processing system according to claim 1 or 2.

12. An information processing system receives an input of an input text including at least one sentence included in an image and position information of the sentence; a token step in which the information processing system converts the input text into a group of token sequences based on a vocabulary dictionary in which a division unit of tokens is defined; a step of estimating, by the information processing system, structured information indicating a structure of the token sequence group based on the position information; generating, for each target token sequence included in the token sequence group, an auxiliary token sequence that estimates context information of the target token sequence based on the structured information by the information processing system; generating a combined token sequence by combining the target token sequence and the auxiliary token sequence; the information processing system converting the combined token sequence into a feature vector representing context information of the target token sequence using a neural network including at least one neural network layer; An information processing method comprising:

13. Computer, an input unit that receives an input of an input text including at least one sentence included in an image and position information of the sentence; a token conversion unit that converts the input text into a group of token sequences based on a vocabulary dictionary in which a unit of token division is defined; an estimation unit that estimates structured information indicating a structure of the token sequence group based on the position information; an auxiliary token sequence generation unit that generates, for each target token sequence included in the token sequence group, an auxiliary token sequence that estimates context information of the target token sequence based on the structured information; a combined token sequence generator for generating a combined token sequence by combining the target token sequence and the auxiliary token sequence; a feature vector conversion unit that converts the combined token sequence into a feature vector representing context information of the target token sequence using a neural network including at least one neural network layer; A program to function as a

Citation Information

Patent Citations

  • Language model processing system and language model processing method

    JP2022096748A