Information processing apparatus, information processing method, and recording medium

Abstract

To provide an information processor or the like capable of improving the quality of generated text data.SOLUTION: The information processing device includes a memory that stores a program, and a processor that executes the program, and the processor executes the program to acquire document file 200, generate, based on document file 200, a plurality of first sentences each including a character string group and positional information about a position of each of the plurality of first sentences in document file 200, and estimate a first order indicating an order of the plurality of first sentences based on a context of each of the plurality of first sentences. A second ordering sequence indicating the ordering sequence of the plurality of first sentences is estimated on the basis of position information of each of the plurality of first sentences, and the plurality of first sentences are rearranged and / or connected on the basis of the first ordering sequence and the second ordering sequence to generate a second sentence.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a recording medium. [Background technology]

[0002] In today's world, a wide variety of information is stored electronically. This information is saved on computers in a format that can be handled by word processing software (for example, document files). In other words, the vast majority of information is stored as text within computers. For this reason, the technology to extract and format the contents of document files as text data is an important fundamental technology in both academia and industry.

[0003] As a process for forming text data, basic preprocessing generally involves extracting characters from a document file, correcting errors in the extracted characters, and rearranging and concatenating the characters. For example, Patent Document 1 discloses a technology for extracting character string data located near pre-registered items in a form as data associated with the items. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2021-144393 Summary of the Invention [Problem to be solved by the invention]

[0005] However, it is desirable to improve the quality of the generated text data.

[0006] Therefore, the present disclosure provides an information processing device, an information processing method, and a recording medium that are capable of improving the quality of generated text data. [Means for solving the problem]

[0007] An information processing device according to one embodiment of the present disclosure is an information processing device comprising a memory for storing a program and a processor for executing the program, wherein when the processor executes the program, the processor acquires a document file, generates a plurality of first sentences, each of which is composed of a group of character strings, and location information regarding the location of each of the plurality of first sentences in the document file, based on the document file, estimates a first ordering order indicating the ordering order of the plurality of first sentences based on the context of each of the plurality of first sentences, estimates a second ordering order indicating the ordering order of the plurality of first sentences based on the location information of each of the plurality of first sentences, and generates second sentences by rearranging and / or concatenating the plurality of first sentences based on the first ordering order and the second ordering order.

[0008] An information processing method according to one embodiment of the present disclosure acquires a document file, generates a plurality of first sentences, each of which is composed of a group of character strings, and location information regarding the location of each of the plurality of first sentences in the document file based on the document file, estimates a first ordering order indicating the ordering order of the plurality of first sentences based on the context of each of the plurality of first sentences, estimates a second ordering order indicating the ordering order of the plurality of first sentences based on the location information of each of the plurality of first sentences, and generates second sentences by rearranging and / or concatenating the plurality of first sentences based on the first ordering order and the second ordering order.

[0009] An information processing device according to one embodiment of the present disclosure is an information processing device comprising a memory for storing a program and a processor for executing the program, wherein by executing the program, the processor acquires a document file, generates a plurality of first sentences based on the document file, each of which is composed of a group of character strings, generates a second sentence by rearranging and / or concatenating the plurality of first sentences, and further acquires connector information indicating the relationship between two or more words and / or sentences in the document file based on the document file, and generates a single first sentence by rearranging and / or concatenating the two or more words and / or sentences based on the connector information, wherein the plurality of first sentences includes the single first sentence.

[0010] An information processing method according to one embodiment of the present disclosure includes acquiring a document file, generating a plurality of first sentences based on the document file, each of which is composed of a group of character strings, generating a second sentence by rearranging and / or concatenating the plurality of first sentences, and further acquiring connector information based on the document file that indicates the relationship between two or more words and / or sentences in the document file, and generating a single first sentence by rearranging and / or concatenating the two or more words and / or sentences based on the connector information, wherein the plurality of first sentences includes the single first sentence.

[0011] A recording medium according to one aspect of the present disclosure is a recording medium on which a program for causing a computer to execute the above-described information processing method is recorded. [Effects of the Invention]

[0012] According to one aspect of the present disclosure, it is possible to realize an information processing device or the like that is capable of improving the quality of generated text data. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a block diagram showing the functional configuration of a text content extraction device according to an embodiment. [Figure 2]FIG. 2 is a flowchart showing the operation of the text content extraction device according to the embodiment. [Figure 3] FIG. 3 is a first diagram for explaining generation of short sentence group data in the case of a standard layout according to the embodiment. [Figure 4] FIG. 4 is a second diagram for explaining generation of short sentence group data in the case of a standard layout according to the embodiment. [Figure 5] FIG. 5 is a diagram showing an example of short sentence group data generated in the case of a standard layout according to the embodiment. [Figure 6] FIG. 6 is a first diagram for explaining generation of short sentence group data in the case of an unfixed layout according to the embodiment. [Figure 7] FIG. 7 is a second diagram for explaining generation of short sentence group data in the case of an unfixed layout according to the embodiment. [Figure 8] FIG. 8 is a third diagram for explaining generation of short sentence group data in the case of an unfixed layout according to the embodiment. [Figure 9] FIG. 9 is a diagram showing an example of small sentence group data generated in the case of an unfixed layout according to the embodiment. [Figure 10] FIG. 10 is a diagram for explaining how to determine the order of the short sentences according to the embodiment. [Figure 11] FIG. 11 is a diagram showing an example of the arrangement of short sentences for explaining the process of estimating a positional order plan for each short sentence using position information according to the embodiment. [Figure 12] FIG. 12 is a diagram for explaining the movement of a human line of sight. [Figure 13] FIG. 13 is a diagram schematically illustrating a first example of single-page data of a document file in an unfixed format. [Figure 14] FIG. 14 is a diagram schematically illustrating a second example of single-page data of a document file in an unfixed format. [Figure 15] FIG. 15 is a diagram schematically illustrating a third example of single-page data of a document file in an unfixed format. [Figure 16] FIG. 16 is a diagram illustrating a fourth example of single-page data of a document file in an unfixed format. [Figure 17] FIG. 17 is a diagram illustrating a fifth example of single-page data of a document file in an unfixed format. DETAILED DESCRIPTION OF THE INVENTION

[0014] (Background to this disclosure) Before describing this disclosure, we will explain how this disclosure came about. This disclosure relates to a technology that can improve the quality of generated text data. For example, this disclosure relates to a technology that extracts, reorders, and outputs character string information when processing an electronic document file (document file) that contains such information. Reordering means rearranging multiple sentences in an appropriate order and concatenating them to form one long, natural-sounding sentence.

[0015] As mentioned in the "Background Art" above, the technology to extract and format the contents of document files as text data is an important fundamental technology in both academia and industry. Text data refers to data consisting of a series of character strings, such as, but not limited to, "AIUEO 123" or "This is a pen. I am Ben."

[0016] In recent years, advances in hardware and acceleration technologies have led to remarkable technological advances in fields where big data (large amounts of electronic data) undergoes preprocessing before undergoing large-scale processing to obtain new insights and results. The same can be said for textual information, and Large Language Models (LLMs), particularly those typified by Generative Pre-trained Transformers (GPTs), which are trained using large amounts of text data, have made it possible to output fluent sentences and handle ambiguous tasks such as summarizing and genre classification. LLMs are machine learning models capable of outputting sentences that are likely to follow an input sentence. By adjusting numerous internal parameters using vast amounts of training data, they are able to output highly natural-sounding sentences.

[0017] Existing publicly known technologies for extracting and formatting text data include OCR (Optical Character Recognition), which recognizes image data scanned from paper documents and extracts characters, and layout analysis technology, which automatically recognizes the structure of a document.

[0018] In particular, in the reordering process, for documents such as reports that have rectangularly divided areas, that is, documents that can be expected to have a standard structure (predetermined format), existing technology can be used to reorder the documents taking into account their structure, and natural text data can be generated from the extracted characters.

[0019] It is desirable to generate natural-looking text data even for documents with an atypical structure. However, the technology of Patent Document 1 targets documents with a typical structure for which a certain format can be assumed. For documents with an atypical structure for which a certain format cannot be assumed, it is difficult to reorder the sentences, making it difficult to maintain high quality of the generated text data. In other words, with existing technologies, it is difficult to extract natural-looking text data indicating the contents of documents with an atypical structure (so-called graphical documents). Examples of documents with an atypical structure include documents in which it is not predetermined what is written where (which position or which area on a page), such as presentation materials (e.g., Figures 13 to 15, described below) and materials used for technical explanations (e.g., Figure 16, described below), but these are not limited to these.

[0020] It is also desirable to improve the quality of generated text data even for documents with a fixed structure.

[0021] Therefore, the inventors of the present application have conducted extensive research into information processing devices and the like that can improve the quality of generated text data, and have devised the information processing device and the like described below.

[0022] An information processing device according to a first aspect of the present disclosure is an information processing device comprising a memory for storing a program and a processor for executing the program, wherein by executing the program, the processor acquires a document file, generates a plurality of first sentences, each of which is composed of a group of character strings, and location information regarding the location of each of the plurality of first sentences in the document file, based on the document file, estimates a first ordering order indicating the ordering order of the plurality of first sentences based on the context of each of the plurality of first sentences, estimates a second ordering order indicating the ordering order of the plurality of first sentences based on the location information of each of the plurality of first sentences, and generates second sentences by rearranging and / or concatenating the plurality of first sentences based on the first ordering order and the second ordering order.

[0023] As a result, the first sentences are rearranged and / or linked using both the positions and contexts of the first sentences, which improves the quality of the second sentences compared to when rearrangement and / or linking is performed using only one of the positions and contexts, thereby improving the quality of text data generated based on the second sentences.

[0024] Also, for example, an information processing device according to the second aspect may be an information processing device according to the first aspect, and may obtain connector information indicating the relationship between two or more words and / or sentences in the document file based on the document file, and generate a single first sentence by rearranging and / or linking the two or more words and / or sentences based on the connector information.

[0025] This allows the quality of the second sentence to be further improved because the connector information reveals the relationship between two or more words and / or sentences, thereby further improving the quality of the text data generated from the second sentence.

[0026] Also, for example, an information processing device according to a third aspect may be an information processing device according to the second aspect, wherein the document file is image data, and the information processing device may perform at least one of extracting the plurality of first sentences and obtaining the connector information based on the pixel values ​​of each pixel of the image data.

[0027] This makes it possible to improve the quality of generated text data by using information obtainable from image data when the document file is image data.

[0028] Also, for example, an information processing device according to a fourth aspect may be an information processing device according to the second or third aspect, wherein the document file includes one or more objects, and the plurality of first sentences may be extracted based on at least one of the position and size of the one or more objects.

[0029] As a result, when a document file includes one or more objects, the quality of the generated text data can be improved by using information obtainable from the one or more objects.

[0030] Furthermore, for example, an information processing device according to a fifth aspect may be an information processing device according to any one of the first to fourth aspects, and may estimate the first sorting order by inputting the plurality of first sentences into a language model that has been trained to output information indicating the sorting order of a plurality of character string groups when the plurality of character string groups are input.

[0031] This allows us to estimate the ordering order taking the context into account using a language model.

[0032] Also, for example, an information processing device according to a sixth aspect may be an information processing device according to any one of the first to fifth aspects, and the document file may include information indicating at least one of defect information of a product and defect information of a manufacturing process of the product.

[0033] This makes it possible to improve the quality of text data generated from information indicating at least one of product defect information and product manufacturing process defect information.

[0034] Also, for example, an information processing device according to a seventh aspect may be an information processing device according to any one of the first to sixth aspects, and may recognize the layout of the document file, and if the recognized layout is an atypical layout, may generate the second sentences by rearranging and / or concatenating the plurality of first sentences based on the first sorting order and the second sorting order.

[0035] This makes it possible to effectively improve the quality of text data generated from document files with non-standard layouts, for which it is difficult to maintain high quality of the text data.

[0036] Also, for example, an information processing device according to an eighth aspect may be the information processing device according to the seventh aspect, wherein the document file of the non-standard layout includes at least one of a presentation material, a technical paper, and a magazine.

[0037] This makes it possible to effectively improve the quality of text data generated from non-standard document files including at least one of presentation materials, technical papers, and magazines.

[0038] Also, for example, an information processing device according to a ninth aspect may be an information processing device according to any one of the first to eighth aspects, wherein the document file includes a plurality of pages, the second sentence is generated for each of the plurality of pages, and a third sentence is generated based on the plurality of second sentences.

[0039] This makes it possible to improve the quality of the third sentence, and therefore the quality of the text data generated from the third sentence.

[0040] Also, for example, an information processing device according to a tenth aspect may be the information processing device according to any one of the first to ninth aspects, and may transmit at least one of the second sentence and the third sentence to the outside.

[0041] This makes it possible to notify at least one of the second sentence and the third sentence to the outside.

[0042] In addition, an information processing method according to an eleventh aspect of the present disclosure acquires a document file, generates a plurality of first sentences, each of which is composed of a group of character strings, and location information regarding the location of each of the plurality of first sentences in the document file, based on the document file, estimates a first ordering order indicating the ordering order of the plurality of first sentences based on the context of each of the plurality of first sentences, estimates a second ordering order indicating the ordering order of the plurality of first sentences based on the location information of each of the plurality of first sentences, and generates second sentences by rearranging and / or concatenating the plurality of first sentences based on the first ordering order and the second ordering order.

[0043] This provides the same effects as the above-described information processing device.

[0044] In addition, an information processing device according to a twelfth aspect of the present disclosure is an information processing device comprising a memory for storing a program and a processor for executing the program, wherein by executing the program, the processor acquires a document file, generates a plurality of first sentences based on the document file, each of which is composed of a group of character strings, generates a second sentence by rearranging and / or concatenating the plurality of first sentences, and further acquires connector information indicating the relationship between two or more words and / or sentences in the document file based on the document file, and generates one first sentence by rearranging and / or concatenating the two or more words and / or sentences based on the connector information, wherein the plurality of first sentences includes the one first sentence.

[0045] As a result, the connector information allows the relationship between two or more words and / or sentences to be understood, and the quality of the second sentence can be improved compared to when the connector information is not used. For example, even if the document file is created in an undefined format, the quality of the second sentence can be improved. Therefore, it is possible to improve the quality of text data generated based on the second sentence.

[0046] In addition, an information processing method according to a thirteenth aspect of the present disclosure acquires a document file, generates a plurality of first sentences based on the document file, each of which is composed of a group of character strings, generates a second sentence by rearranging and / or concatenating the plurality of first sentences, and further acquires connector information based on the document file that indicates the relationship between two or more words and / or sentences in the document file, and generates a single first sentence by rearranging and / or concatenating the two or more words and / or sentences based on the connector information, wherein the plurality of first sentences includes the single first sentence.

[0047] This provides the same effects as the above-described information processing device.

[0048] A recording medium according to one aspect of the present disclosure is a recording medium on which a program for causing a computer to execute the above-described information processing method is recorded.

[0049] This provides the same effects as the above-described information processing device.

[0050] These general or specific aspects may be realized as a system, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or as any combination of the system, method, integrated circuit, computer program, or recording medium. The program may be pre-stored in the recording medium, or may be supplied to the recording medium via a wide area communication network including the Internet.

[0051] Hereinafter, the embodiments will be specifically described with reference to the drawings.

[0052] The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, components, component placement and connection configurations, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components not described in independent claims are described as optional components.

[0053] Furthermore, each figure is a schematic diagram and is not necessarily an exact illustration. Therefore, for example, the scales of the figures do not necessarily match. Furthermore, in each figure, substantially the same components are given the same reference numerals, and redundant explanations are omitted or simplified.

[0054] Furthermore, in this specification, terms indicating the shape of elements such as rectangle and circle, as well as numerical values ​​and numerical ranges, are not expressions that express only the strict meaning, but are expressions that mean that they include a substantially equivalent range, for example, a difference of about several percent (or about 10%).

[0055] Furthermore, in this specification, ordinal numbers such as "first" and "second" do not refer to the number or order of components unless otherwise specified, but are used for the purpose of avoiding confusion and distinguishing between components of the same type.

[0056] (Embodiment) The text content extraction device according to this embodiment will be described below with reference to FIG.

[0057] [1. Structure of text content extraction device] First, the configuration of a text content extraction device according to this embodiment will be described with reference to FIG. 1. FIG. 1 is a block diagram showing the functional configuration of a text content extraction device 100 according to this embodiment. Note that FIG. 1 shows an exemplary functional configuration of the text content extraction device 100, and the functional configuration of the text content extraction device 100 is not limited to that shown in FIG. 1. The text content extraction device 100 is an information processing device configured to extract and arrange characters in tables or text boxes of a document file and output text data indicating the written content of the document. A text box represents an object that contains character string information and displays the character string information. A text box can be placed in virtually any location. An object represents graphic data such as a graph, table, or arrow in a document file. An object generally includes information such as shape, color, position, and size.

[0058] As shown in FIG. 1, the text content extraction device 100 includes a page recognition and selection unit 101, a layout recognition unit 102, a rectangular layout processing unit 103, a non-rectangular layout processing unit 107, a small sentence concatenation processing unit 111, a large sentence concatenation processing unit 115, and a communication unit 118.

[0059] The text content extraction device 100 also includes, as its hardware configuration, a non-volatile memory in which a program is stored, a volatile memory which is a temporary storage area for executing the program, an input / output port, a communication interface, a processor which executes the program, etc. The memory is a ROM (Read Only Memory) and a RAM (Random Access Memory), etc., and can store the program to be executed by the processor. The page recognition and selection unit 101, layout recognition unit 102, rectangular layout processing unit 103, non-rectangular layout processing unit 107, small sentence concatenation processing unit 111, large sentence concatenation processing unit 115, and communication unit 118 are realized by a processor which executes a program stored in memory, etc.

[0060] The text content extraction device 100 may be realized by a desktop PC (Personal Computer), a mobile terminal such as a smartphone or tablet, a dedicated computer, a server (e.g., a cloud server), or a combination thereof.

[0061] The page recognition and selection unit 101 counts the number of pages in the document file 200 input to the text content extraction device 100, and also designates a new page to be subjected to text data extraction processing, and outputs that data (single-page data 201 shown in FIG. 1) to the layout recognition unit 102. Furthermore, if there is no page to be designated, for example, if the text data extraction processing for the document file 200 has been completed, the page recognition and selection unit 101 outputs the large document data 204 stored in the storage unit 117 to the communication unit 118.

[0062] The page recognition and selection unit 101 may specify pages that have not yet been subjected to text data extraction processing as targets for text data extraction processing, for example, in ascending order of page number among the multiple pages contained in the document file 200. The page recognition and selection unit 101 may also obtain at least one of the number of pages in the document file 200 and the pages in the document file 200 that are targets for text data extraction processing from an external device. The text data extraction processing refers to the processing of steps S30 to S120 shown in FIG. 2, which will be described later.

[0063] The document file 200 may be a presentation material, a material used for technical explanations, or a material containing information indicating at least one of product defect information and defect information in the product manufacturing process.

[0064] The layout recognition unit 102 determines whether the single-page data 201, which is a page from which text data in the document file 200 is extracted, has a layout that can be divided into rectangles. In other words, the layout recognition unit 102 determines whether the recognized layout is standard (for example, a standard format) or non-standard (for example, a non-standard format). If the layout is standard, the layout recognition unit 102 outputs the single-page data 201 to the rectangle layout processing unit 103, and if the layout is non-standard, the layout recognition unit 102 outputs the single-page data 201 to the non-rectangle layout processing unit 107.

[0065] The rectangular layout processing unit 103 is a processing unit that generates short sentence group data 202 based on single-page data 201 whose layout has been determined to be standard, and has a division configuration recognition unit 104, a minimum unit pair acquisition unit 105, and a block text processing unit 106.

[0066] The division configuration recognition unit 104 acquires information about the position and size of each rectangle when the layout of the single page data 201 to be extracted is divided into the smallest possible rectangles.

[0067] Based on the information acquired by the division configuration recognition unit 104, the minimum unit pair acquisition unit 105 acquires two (or three or more) adjacent rectangular sections of the minimum unit as a pair.

[0068] The block text processing unit 106 connects the words and / or sentences acquired from each pair of blocks acquired by the minimum unit pair acquisition unit 105 while adding particles, etc., and outputs the result as short sentence group data 202. A word is the smallest linguistic unit that has meaning and function, and a sentence includes two or more words.

[0069] The non-rectangular layout processing unit 107 is a processing unit that generates short sentence group data 202 based on single-page data 201 whose layout is determined to be atypical, and has a text box recognition unit 108, an in-table text processing unit 109, and a connector processing unit 110.

[0070] The text box recognition unit 108 acquires information about the positions and sizes of text boxes, tables, objects, etc. included in the single page data 201 based on the single page data 201 that is the extraction target.

[0071] The table text processing unit 109 connects words, sentences, etc. included in the table acquired by the text box recognition unit 108 while adding particles, etc. for each row, to generate short sentence data for the table.

[0072] The connector processing unit 110 acquires connector information indicating the relationship between at least two words and / or sentences based on the objects acquired by the text box recognition unit 108. For example, the connector processing unit 110 acquires information about the positions of the start and end points of connectors included in the page to be extracted as connector information, and connects words, sentences, etc. included in text boxes existing at each position to generate short sentences. A connector refers to an object, such as an arrow or a line, that connects two or more points (for example, two or more text boxes) among multiple objects.

[0073] The short sentence group data 202 output from the non-rectangular layout processing unit 107 includes short sentences generated by the in-table text processing unit 109 and the connector processing unit 110 .

[0074] The short sentence concatenation processing unit 111 is a processing unit that determines the order in which to connect a group of short sentences, and generates medium sentence data 203 by rearranging and / or concatenating each short sentence of the group of short sentences in the determined order, and has a position-based reordering order estimation unit 112, a context-based reordering order estimation unit 113, and a reordering and concatenation processing unit 114.

[0075] The position-based sorting order estimation unit 112 estimates the sorting order of each short sentence based on information about the position and size of each short sentence included in the short sentence group data 202. The sorting order is the order in which the short sentences are arranged. The sorting order can also be said to be the order in which the short sentences are rearranged.

[0076] The context-based sorting order estimation unit 113 estimates the sorting order based on the content (for example, context) of each short sentence included in the short sentence group data 202.

[0077] The reordering and linking processor 114 determines the final reordering order of each short sentence by taking into consideration both the positional relationship of each sentence and the content of the sentence, which is the order in which humans read sentences included in a graphical document. Specifically, the reordering and linking processor 114 determines the final reordering order based on the reordering orders estimated by the position-based reordering order estimation unit 112 and the context-based reordering order estimation unit 113, links the short sentences in that order, and outputs the result as medium sentence data 203. The medium sentence data 203 includes, for example, each of the short sentences for one page of the document file 200. The medium sentence data 203 is an example of a second sentence.

[0078] The large sentence concatenation processing unit 115 is a processing unit that generates and stores large sentence data 204 from medium sentence data 203, and includes a text data concatenation unit 116 and a storage unit 117.

[0079] The text data concatenation unit 116 concatenates the medium sentence data 203 output from the small sentence concatenation processing unit 111 to the large sentence data 204 stored in the memory unit 117, and outputs the result as new large sentence data 204. Furthermore, if the large sentence data 204 does not exist in the memory unit 117, the text data concatenation unit 116 outputs the medium sentence data 203 to the memory unit 117 as the large sentence data 204.

[0080] The large sentence data 204 includes, for example, each of the small sentences for a plurality of pages of the document file 200. The large sentence data 204 has, for example, a larger amount of data than the medium sentence data 203. The large sentence data 204 is an example of a third sentence.

[0081] The storage unit 117 stores the large text data 204 output from the text data linking unit 116. The storage unit 117 is realized by a memory (for example, a semiconductor memory) or a HDD (Hard Disk Drive), but is not limited to these.

[0082] The communication unit 118 transmits the large-sized sentence data 204 output from the storage unit 117 to an external device via communication as text data 205. The communication unit 118 only needs to transmit at least one of the large-sized sentence data 204 and the medium-sized sentence data 203 to the external device. The communication unit 118 may be configured to include a communication circuit (or a communication module).

[0083] [2. Operation of text content extraction device] Next, the operation of the text content extraction device 100 configured as above will be described with reference to Figures 2 to 12. Figure 2 is a flowchart showing the operation (information processing method) of the text content extraction device 100 according to this embodiment.

[0084] 2, first, the page recognition and selection unit 101 reads the document file 200 (S10). The page recognition and selection unit 101 may obtain the target document file 200 from, for example, a device that manages multiple document files.

[0085] Next, the page recognition and selection unit 101 selects one page from which to generate intermediate text data 203 (S20). The page recognition and selection unit 101 counts the number of pages in the input document file 200 and newly specifies a page to be subjected to text data extraction processing. For example, the page recognition and selection unit 101 may select the youngest page among pages that have not yet undergone text data extraction processing as the page to be subjected to text data extraction processing next. The page recognition and selection unit 101 extracts the selected page (single-page data 201) from the document file 200 and outputs it to the layout recognition unit 102.

[0086] The layout recognition unit 102 determines whether the single-page data 201 has a standard layout or a non-standard layout based on the single-page data 201 (S30). In other words, the layout recognition unit 102 determines whether the single-page data 201 has a rectangular form layout. Step S30 corresponds to recognizing the layout of the document file 200.

[0087] The layout recognition unit 102 may determine whether the ratio of the largest frame in the single-page data 201 to the paper surface is equal to or greater than a predetermined value, and if it is equal to or greater than the predetermined value, determine that the layout is rectangular, and if it is less than the predetermined value, determine that the layout is non-rectangular. The ratio is calculated, for example, by (area enclosed by the frame / area of ​​the paper surface)×100, and the predetermined value is, for example, but not limited to, 70%.

[0088] Furthermore, if identification information (e.g., format identification information) indicating whether the layout is rectangular or non-rectangular is included in the single-page data 201, the layout recognition unit 102 may determine whether the layout is rectangular or non-rectangular based on the identification information. For example, the document file 200 may include information indicating whether each page has a rectangular layout or a non-rectangular layout.

[0089] Furthermore, if the file name of the single-page data 201 differs depending on whether the single-page data 201 has a rectangular layout, the layout recognition unit 102 may determine whether the single-page data 201 has a rectangular layout based on the file name of the single-page data 201. Furthermore, if the layout recognition unit 102 stores reference frame data indicating the position and size of the frame of the rectangular layout, it may determine whether the single-page data 201 has a rectangular layout by comparing the position and size of the frame included in the single-page data 201 with the reference frame data (for example, determining whether they match).

[0090] In addition, the layout recognition unit 102 may determine whether or not the layout is rectangular based on the output of a machine learning model obtained by inputting the single-page data 201 into a machine learning model that has been trained to input single-page data and output whether the single-page data has a rectangular layout or a non-rectangular layout.

[0091] Next, if the layout recognition unit 102 determines that the layout is a standard format (YES in S30), the division configuration recognition unit 104 recognizes and acquires a rectangular division configuration from the frame lines and the like of the single-page data 201 (S40).

[0092] 3 is a first diagram for explaining generation of short sentence group data 202 in the case of a standard layout according to this embodiment. In FIG. 3, for identification purposes, the single page data 201 is denoted by the reference numeral 201a.

[0093] Fig. 3(a) shows an example of single-page data 201a in the case of a standard layout. Each frame shown in Fig. 3(a) is a standardized frame associated with a report.

[0094] 3B shows each rectangle obtained when the layout of the single-page data 201a is divided into the smallest possible rectangles by the division configuration recognition unit 104. In other words, the division configuration recognition unit 104 recognizes the division into the smallest units, as shown in FIG. 3B. For example, in step S40, the division configuration recognition unit 104 divides each of a plurality of standardly determined frames (eight frames in the case of FIG. 3A) into one unit. In step S40, information about the position and size of each rectangle divided into the smallest units is acquired.

[0095] 2 again, next, the minimum unit pair acquisition unit 105 classifies pairs of item names and description texts from the division configuration obtained by the division configuration recognition unit 104 (S50). The minimum unit pair acquisition unit 105 classifies, for example, two (or three or more) minimum unit rectangular sections that are adjacent to each other on either the top, bottom, left, or right as a pair.

[0096] (c) of Fig. 3 shows each pair classified by the minimum unit pair acquisition unit 105. In (c) of Fig. 3, four pairs have been generated. Here, "Case content" is the item name, and "screw licking" is the description text corresponding to "Case content." Also, "Cause" is the item name, and "inappropriate force distribution" is the description text corresponding to "Cause."

[0097] 2 again, next, the block text processing unit 106 generates a short sentence by concatenating the item name and description text of the pair while adding particles and objects (S60). The block text processing unit 106 generates a short sentence by, for example, extracting words and / or sentences in each pair block and concatenating them by adding particles, etc.

[0098] Fig. 4 is a second diagram for explaining the generation of short sentence group data 202 in the case of a fixed layout according to this embodiment. For convenience, supplemented parts are underlined in Fig. 4. Note that particles and the like that are actually supplemented are not underlined.

[0099] Figure 4(a) shows data extracted from each pair section, including words and / or sentences. The pairs are arranged so that they are close to each other. Note that no fixed frames are extracted.

[0100] Figure 4(b) shows short sentences generated by adding particles and concatenating them. In Figure 4(b), four short sentences are generated.

[0101] The block text processing unit 106 also generates short sentence group data 202 by adding position and size information of paired blocks to each short sentence, and outputs the generated short sentence group data 202 to the short sentence concatenation processing unit 111 .

[0102] 5 is a diagram showing an example of short sentence group data 202a generated in the case of a standard layout according to the present embodiment. In FIG. 5, for identification purposes, the short sentence group data 202 is denoted by the code 202a.

[0103] As shown in FIG. 5, the short sentence group data 202a is data in which short sentences are associated with information indicating the position of the short sentence on the paper (the position on the single-page data 201). The short sentence group data 202a includes all of the short sentences generated in step S60 and information indicating their positions (position information). The information indicating the position indicates the range on the paper in which the short sentence is located, and indicates, for example, the coordinates of the four corners of the range. The information indicating the position includes the position in the X-axis direction on the paper (X-position left and X-axis position right) and the position in the Y-axis direction on the paper (Y-position left and Y-axis position right). The X-axis and Y-axis are axes that are perpendicular to each other when viewed from the front of the paper.

[0104] For example, "The case study is about licking a screw." indicates that the sentence is located in the area surrounded by four points in the single-page data 201a: X position 15 and Y position 10, X position 15 and Y position 40, X position 85 and Y position 10, and X position 85 and Y position 40. The same applies to other sentences.

[0105] In this way, the short sentence group data 202a includes short sentences and information indicating the range on the paper surface where the short sentences are arranged. It can also be said that the rectangle layout processing unit 103 generates a plurality of short sentences, each consisting of a group of character strings, and position information relating to the position of each of the plurality of short sentences in the document file 200, based on the document file 200. The coordinates of the four corners can be obtained from the position information of each rectangle partitioned into the smallest units obtained in step S40.

[0106] Each of the four sentences (short sentences) shown in FIG. 5 is an example of a first sentence.

[0107] 2 again, if the layout recognition unit 102 determines that the layout is non-standard (NO in S30), the text box recognition unit 108 acquires position and size information of the table and text boxes (S70) based on the single-page data 201. The text box recognition unit 108 recognizes all text boxes, including tables, to acquire where characters are positioned in the single-page data 201.

[0108] 6 is a first diagram for explaining generation of short sentence group data 202 in the case of an atypical layout according to this embodiment. In FIG. 6, for identification purposes, the single page data 201 is denoted by the reference numeral 201b.

[0109] FIG. 6(a) shows an example of single-page data 201b (non-standard layout document) in the case of a non-standard layout. The frames shown in FIG. 6(a) form tables arranged arbitrarily. Note that the single-page data 201b is not limited to including tables. A table is also an example of an object. The single-page data 201b (or document file 200) may be data including one or more objects.

[0110] For example, if the document file 200 (here, single-page data 201b) is image data, the text box recognition unit 108 may recognize text boxes based on the pixel value (e.g., RGB value) of each pixel. For example, if the pixel value of a pixel is equal to or greater than a predetermined value, the text box recognition unit 108 determines that the pixel is part of a character, and if the pixel value is less than the predetermined value, the text box recognition unit 108 determines that the pixel is not part of a character (e.g., a blank portion), and sets a text box surrounding the sentence when adjacent characters are closer than a threshold value as a single sentence. The text box recognition unit 108 also extracts tables as they are. The determination of whether a text box is a table may be made, for example, by determining whether multiple frames are connected. The text boxes for each sentence may be set so that they do not overlap each other except at their boundaries.

[0111] FIG. 6(b) shows the results of text box recognition. FIG. 6(b) shows an example in which the following sentences are determined to be a single sentence: "Lightweight and rigid" (hereinafter also referred to as sentence (i)), "Among materials that meet the rigidity criteria, the lightest material was selected" (hereinafter also referred to as sentence (ii)), "The results of the confirmation experiment are shown in the table below," and "Lightweight aluminum was selected because it exceeded the stiffness target value of 100 N / mm." Between sentences (i) and (ii) on the paper, there is an area where the pixel value is less than a predetermined value (e.g., a blank line), so separate text boxes are set for the two sentences. Furthermore, between sentence (ii) and "The results of the confirmation experiment are shown in the table below," there is also an area where the pixel value is less than a predetermined value (e.g., a blank line), so separate text boxes are set for the two sentences.

[0112] Furthermore, for example, when the document file 200 (here, the single-page data 201b) is image data, the text box recognition unit 108 may recognize objects such as arrows. The text box recognition unit 108 may identify objects and their positions included in the single-page data 201b based on the arrangement of pixel values, etc. Information indicating an object is an example of connector information.

[0113] The text box recognition unit 108 may set text boxes based on the output of a machine learning model obtained by inputting the single-page data 201b to a machine learning model trained to receive single-page data as input and output text boxes of the single-page data. The text box recognition unit 108 may also detect objects based on the output of a machine learning model obtained by inputting the single-page data 201b to a machine learning model trained to receive single-page data as input and output objects (e.g., object types) of the single-page data.

[0114] Referring again to FIG. 2, next, the table text processor 109 adds particles to each row of the table and connects the description text row by row (S80), generating a short sentence showing the contents of the table (S90).

[0115] Fig. 7 is a second diagram for explaining the generation of short sentence group data 202 in the case of an atypical layout according to this embodiment. Fig. 7 shows an example in which particles and the like are added to the words and phrases in the table to form a sentence. For convenience, the added parts are underlined in Fig. 7. Note that the particles and the like that are actually added are not underlined.

[0116] As shown in FIG. 7, the table text processor 109 concatenates descriptive text while supplementing words written in the table with particles on a row-by-row basis. In FIG. 7, two sentences are generated as one short sentence. It can also be said that the table text processor 109 extracts one or more (e.g., multiple) short sentences based on at least one of the position and size of the table. In the example of FIG. 7, "The weight [g] of aluminum is 30, and its rigidity [N / mm] is 130. The weight [g] of steel is 70, and its rigidity [N / mm] is 200." is generated as one short sentence. Note that any known method may be used to concatenate descriptive text from a table.

[0117] Referring back to FIG. 2, next, the connector processing unit 110 concatenates the texts in the different text boxes connected by connectors to generate a short sentence (S100).

[0118] Fig. 8 is a third diagram for explaining the generation of short sentence group data 202 in the case of an atypical layout according to this embodiment. Fig. 8(a) is a diagram showing words and / or sentences connected by connectors with dashed lines. Fig. 8(b) shows the data after connector processing.

[0119] As shown in Fig. 8(a), the connector processing unit 110 identifies texts connected by arrows (an example of connectors), which are an example of objects shown in the single-page data 201b shown in Fig. 6(a). In the example of Fig. 8(a), sentences (i) and (ii) are identified and connected by a dashed line.

[0120] In this way, the connector processing unit 110 identifies the relationship between sentence (i) and sentence (ii) based on the arrow. It can also be said that the connector processing unit 110 extracts words and / or sentences connected by connectors from the single-page data 201b.

[0121] As shown in (b) of Figure 8, the connector processing unit 110 generates one short sentence by concatenating words and / or sentences connected by connectors based on connector information indicating the connectors. The connector processing unit 110 adds the underlined comma (,) after "necessary" to generate one short sentence indicated as "Both lightness and hardness are required, and the lightest material that exceeds the hardness standard will be selected." It can also be said that the connector processing unit 110 extracts sentences (i) and (ii) as one short sentence.

[0122] 8(b), four short sentences are generated. Note that the connector processing unit 110 may rearrange the words and / or sentences connected by the connectors, for example, based on the direction of the arrow, or may rearrange them before joining them.

[0123] The connector processing unit 110 also assigns position and size information to each short sentence to generate short sentence group data 202 , and outputs the generated short sentence group data 202 to the short sentence concatenation processing unit 111 .

[0124] 9 is a diagram showing an example of short sentence group data 202b generated in the case of an atypical layout according to the present embodiment. In FIG. 9, for identification purposes, the code of the short sentence group data 202 is 202b.

[0125] As shown in FIG. 9, the short sentence group data 202b is data in which short sentences are associated with information (position information) indicating the position of the short sentence on the paper (position on the single-page data 201b). The short sentence group data 202b includes all of the short sentences generated in steps S70 to S100 and information indicating their positions. The information indicating the position indicates the range on the paper in which the short sentence is arranged, and indicates, for example, the coordinates of the four corners of the range. The information indicating the position includes the position in the X-axis direction on the paper (X-position left and X-axis position right) and the position in the Y-axis direction on the paper (Y-position left and Y-axis position right). The X-axis and Y-axis are axes that are perpendicular to each other when viewed from the front of the paper.

[0126] For example, "Both lightness and hardness are required. The lightest material that exceeds the hardness standard will be selected." indicates that the text is placed in the area surrounded by four points in the single-page data 201b: X position 10 and Y position 5, X position 10 and Y position 30, X position 90 and Y position 5, and X position 90 and Y position 30. The coordinates of these four points are calculated based on the coordinates of the text box for sentence (i) and the coordinates of the text box for sentence (ii). The same applies to other sentences.

[0127] In this way, the short sentence group data 202b includes short sentences and information indicating the area on the paper where the short sentences are arranged. It can also be said that the non-rectangular layout processing unit 107 generates a plurality of short sentences, each consisting of a group of character strings, and position information regarding the position of each of the plurality of short sentences in the document file 200, based on the document file 200. The coordinates of the four corners can be obtained from the position and size information of the text box obtained in step S70. Furthermore, the coordinates of the four corners of a short sentence formed by text in a table, such as "The weight [g] of aluminum is 30, and its stiffness [N / mm] is 130. The weight [g] of steel is 70, and its stiffness [N / mm] is 200," can be calculated based on the coordinates of the outer frame of the table, for example.

[0128] The multiple short sentences for which the medium sentence data 203 is generated include short sentences generated based on connector information.

[0129] Each of the four sentences (short sentences) shown in FIG. 9 is an example of a first sentence.

[0130] Referring again to FIG. 2, the short sentence concatenation processing unit 111 generates a middle sentence by rearranging and concatenating each short sentence based on the text content and position information of each short sentence (S110). The position-based reordering order estimation unit 112 lists possible reordering orders according to rules, such as from top to bottom or left to right, based on the position information of the short sentence group data 202b. It can also be said that the position-based reordering order estimation unit 112 estimates a positional reordering order indicating the arrangement order of a plurality of short sentences based on the position information of each of the plurality of short sentences. The positional reordering order is an example of a second reordering order. The position-based reordering order estimation unit 112 estimates a positional reordering order using only the positional information of the positional information and the text content. The estimation of a positional reordering order will be described later with reference to FIGS. 11 and 12.

[0131] Fig. 10 is a diagram for explaining how to determine the order of each short sentence according to this embodiment. Fig. 10(a) shows a position-based ordering plan estimated by the position-based ordering order estimation unit 112 using position information. Fig. 10(a) shows an example in which three position ordering plans (No. 1 to 3) are estimated. Note that the number of estimated position ordering plans is not limited to three, and may be one or more.

[0132] The context-based sorting order estimation unit 113 then lists possible sorting orders as contextual sorting proposals based on the content of each short sentence. The context-based sorting order estimation unit 113 generates contextual sorting proposals using only the text content out of the location information and the text content.

[0133] The context-based sorting order estimation unit 113 inputs each of the short sentences included in the short sentence group data 202b into a language model trained to output information indicating the context-based sorting order of a group of multiple character strings (e.g., multiple short sentences) when the group of multiple character strings is input. The language model acquires information indicating the sorting order of each of the short sentences, which is the output of the language model, and estimates a proposed context sorting order based on the acquired information indicating the sorting order. The context-based sorting order estimation unit 113 can also be said to estimate a proposed context sorting order indicating the sorting order of the multiple short sentences based on the context of each of the multiple short sentences. The proposed context sorting order is an example of a first sorting order. It is sufficient for the language model to estimate one or more sorting orders, and may be capable of outputting, for example, multiple sorting order estimation results. The information indicating the context-based sorting order of the group of multiple character strings is correct answer data of the machine learning model, and may include, for example, an order rearranged by a human based on the content of each short sentence.

[0134] FIG. 10(b) shows an example in which three context ordering plans (No. 1 to 3) are estimated.

[0135] As the language model, for example, LLM or Language Models (LM) may be used. As the LLM, for example, GPT is exemplified, but is not limited to this. The number of estimated context ordering proposals is not limited to three, and may be one or more.

[0136] The reordering and linking processing unit 114 then rearranges and / or links the multiple short sentences based on the positional ordering proposal and the contextual ordering proposal to generate medium sentence data 203. The reordering and linking processing unit 114 determines an appropriate proposal for the ordering order, for example by selecting the positional ordering proposal that matches most closely with each contextual ordering proposal, and links the sentences in that order to generate medium sentence data 203. The medium sentence data 203 is the result of extraction from the single-page data 201 that is the target of the text data extraction process, and is output to the large sentence linking processing unit 115.

[0137] As shown in (c) of FIG. 10, the reordering / linking processor 114 may calculate the number of matching locations between the positional ordering and the contextual ordering as the number of matches, and determine the reordering order to be adopted based on the calculated number of matches. The number of matches is an example of the degree of match. The reordering / linking processor 114 calculates the number of matches between the positional ordering proposal No. 1 and each of the contextual ordering proposals No. 1 to 3, and calculates the sum of the three calculated numbers of matches as the total score. The reordering / linking processor 114 may determine the positional ordering with the highest total score or the total score equal to or greater than a threshold as the final reordering order.

[0138] In this way, the reordering and linking processor 114 reorders the text data 205 by introducing information on the rearrangement of sentences based on context using a language model such as LLM. By using both the positional relationship and information on the ordering of sentences based on context, the reordering and linking processor 114 can improve the accuracy of reordering even for a collection of sentences that is difficult to reorder based on positional relationship alone.

[0139] Here, the process of generating a position reordering plan will be described with reference to Fig. 11 and Fig. 12. Fig. 11 is a diagram showing an example of the arrangement of short sentences to explain the process of estimating a position reordering plan for each short sentence using position information according to this embodiment. Fig. 11 shows an example of the arrangement of short sentences to explain the generation of a position reordering plan.

[0140] 11, the process of generating a position reordering proposal will be described using an example in which seven short sentences, sentences 1 to 7, are arranged as single-page data 201. The position and size of each of sentences 1 to 7 can be calculated based on the position information (X position, Y position) included in the corresponding short sentence group data 202.

[0141] It is said that humans have unconsciously determined patterns of gaze movement. Known gaze movement patterns include, for example, F-type, N-type, and Z-type. Figure 12 is a diagram for explaining human gaze movement.

[0142] As shown in Figure 12(a), in the F-pattern, the gaze moves along the shape of the letter "F." Specifically, in the F-pattern, the gaze moves from the top left ⇒ top right ⇒ slightly down, from left to right ⇒ back to the left edge. The F-pattern is also known as F's law. When information is not evenly distributed on a page, humans may unconsciously move their gaze according to F's law. For example, when there is a lot of text and you want to fully understand the content, humans may move their gaze according to F's law.

[0143] As shown in Figure 12(b), in the N-pattern, the eyes move along the shape of the letter "N." Specifically, in the N-pattern, the eyes move vertically starting from the right, for example, from the top of the first column to the bottom, and then from the top of the second column to the left to the bottom. The N-pattern is also known as the N rule. When a layout is vertical (for example, vertical writing), humans may unconsciously move their eyes based on the N rule.

[0144] As shown in Figure 12(c), in the Z pattern, the gaze moves along the shape of the letter "Z." Specifically, in the Z pattern, the gaze moves from top left ⇒ top right ⇒ bottom left ⇒ bottom right. The Z pattern is also known as the Z law. When information is not evenly distributed, humans may unconsciously move their gaze according to the Z law. For example, when there are many images lined up and people want to grasp the overall content, their gaze may move according to the Z law.

[0145] Referring again to FIG. 11, the position-based sorting order estimation unit 112 estimates the order of sentences 1 to 7 shown in FIG. 11 using at least one of F-type, N-type, and Z-type. The position-based sorting order estimation unit 112 lists the order of sentences 1 to 7 shown in FIG. 11 using, for example, at least one of F-type, N-type, and Z-type. Note that the numbers in " " below indicate one of sentences 1 to 7. "1" indicates sentence 1, and "2" indicates sentence 2. The same applies to "3" and onwards.

[0146] The position-based sorting order estimation unit 112 lists possible F-type sorting order candidates (positional sorting proposals), for example. The position-based sorting order estimation unit 112 lists possible sorting orders such as "1234567" and "1234756." Note that "1234567" indicates the sorting order of sentence 1, sentence 2, sentence 3, sentence 4, sentence 5, sentence 6, and sentence 7.

[0147] The position-based sorting order estimation unit 112 lists, for example, N-type possible sorting order candidates (position sorting proposals). The position-based sorting order estimation unit 112 lists, for example, sorting order candidates such as "1562347" and "1523647".

[0148] The position-based sorting order estimation unit 112 lists possible Z-shaped sorting order candidates (positional sorting proposals), for example. The position-based sorting order estimation unit 112 lists possible sorting orders such as "1253467" and "1235467".

[0149] The location-based ordering estimation unit 112 estimates, for example, a plurality of location ordering proposals.

[0150] The reordering / concatenation processing unit 114, for example, determines whether or not there is a position ordering plan among the multiple position ordering plans that matches the context ordering plan estimated by the context-based ordering order estimation unit 113, and if there is a matching position ordering plan, determines the context ordering plan as the final ordering order.

[0151] Furthermore, when there is no position ordering plan that matches the context ordering plan among the multiple position ordering plans, the reordering / connecting processing unit 114 determines a final ordering order based on the degree of match between each of the multiple position ordering plans and the context ordering plan. If the context ordering plan is "1234567" and the position ordering plan is "1234756," four out of seven positions (specifically, the four positions of "1234") match, so the reordering / connecting processing unit 114 calculates the degree of match as 4. In this manner, the degree of match may be the number of matching points in the ordering plan. The reordering / connecting processing unit 114 may determine the position ordering plan that has a degree of match equal to or greater than a predetermined threshold among the degrees of match between the multiple position ordering plans as the final ordering order, or may determine the position ordering plan that has the highest degree of match among the multiple position ordering plans.

[0152] If there are two or more position ordering plans with a degree of matching equal to or greater than a preset threshold among the plurality of position ordering plans, the reordering / connecting processing unit 114 may allow the user to select which of the two or more position ordering plans to adopt. Also, if there is no position ordering plan with a degree of matching equal to or greater than a preset threshold among the plurality of position ordering plans, the reordering / connecting processing unit 114 may determine the context ordering plan as the final ordering order.

[0153] In this way, the short sentence concatenation processing unit 111 according to this embodiment determines the order of each short sentence by taking into account both the content and the position of the description, just like a human being, and therefore can perform natural reordering that would be difficult if only one of the content and the position were used.

[0154] 2 again, next, the large sentence concatenation processing unit 115 concatenates the medium sentence (medium sentence data 203) to the large sentence (large sentence data 204) to generate a new large sentence, and if the original large sentence does not exist, stores the medium sentence as the large sentence (S120). The text data concatenation unit 116 reads the large sentence data 204 already stored in the memory unit 117, concatenates the newly acquired medium sentence data 203 to the end of the read large sentence data 204, and stores the concatenated data in the memory unit 117 as new large sentence data 204. It can also be said that the text data concatenation unit 116 updates the large sentence data 204 already stored in the memory unit 117 based on the newly acquired medium sentence data 203. In other words, the large sentence data 204 is data generated based on multiple medium sentence data 203. The large sentence data 204 may be generated by rearranging and / or concatenating multiple medium sentence data 203. Furthermore, if large-sized sentence data 204 does not exist in storage unit 117, text data linking unit 116 stores medium-sized sentence data 203 acquired immediately before in storage unit 117 as large-sized sentence data 204 as is.

[0155] Then, the text data concatenation unit 116 notifies the page recognition and selection unit 101 that the large sentence concatenation process has ended, which enables the page recognition and selection unit 101 to execute the process of generating medium sentence data 203 for the next page.

[0156] When the page recognition and selection unit 101 receives a notification from the text data linking unit 116, it determines whether medium-length sentences (medium-length sentence data 203) have been extracted from all pages of the document file 200 (S130). The page recognition and selection unit 101 determines whether there are any pages that have not been selected as targets for text data extraction processing. If there are any pages that have not been selected as targets for text data extraction processing (NO in S130), the page recognition and selection unit 101 returns to step S20, selects one of the pages, and repeats the above processing (steps S30 to S120 shown in FIG. 2). If there are no pages that have not been selected as targets for text data extraction processing (YES in S130), the page recognition and selection unit 101 proceeds to step S140. For example, when the page recognition and selection unit 101 receives a notification from the text data linking unit 116 after extraction processing has been performed on all pages, it outputs the large-length sentence data 204 stored in the storage unit 117 to the communication unit 118.

[0157] Next, the communication unit 118 outputs the acquired large text (large text data 204) as the extraction result (text data 205) from the document file 200 (S140).

[0158] In this way, the text content extraction device 100 generates the intermediate text data 203 for each of the multiple pages of the document file 200, and then executes step S140.

[0159] As described above, the text content extraction device 100 can sort sentences in a graphical document with a quality close to human perception, and by linking them together, it is possible to extract natural text data 205 that indicates the content of the document.

[0160] [3. Examples of document files in non-standard formats] Next, specific examples of document files in non-standard formats will be described with reference to Figs. 13 to 17. Figs. 13 to 17 are diagrams that schematically show examples of single-page data 201 in document files 200 in non-standard formats. In Figs. 13 to 17, characters are simplified with symbols such as "·" and "◯", with the exception of some characters. Note that the single-page data 201 shown in Figs. 13 to 17 is an example, and the single-page data 201 is not limited to the contents of Figs. 13 to 17.

[0161] 13 to 15 schematically show presentation materials and distribution materials corresponding to the presentation materials. (a) of Fig. 13, (a) of Fig. 14, and (a) of Fig. 15 show the presentation materials, and (b) of Fig. 13, (b) of Fig. 14, and (b) of Fig. 15 show the distribution materials. The presentation materials and distribution materials are examples of document files 200.

[0162] Presentation materials are used to support the content of a presentation in an easier-to-understand way, and are materials that complement the main points that the speaker wants to convey. Presentation materials are often created using illustrations and other materials created by the creator at their own discretion. Handouts are materials used to convey the content of a presentation without causing any misunderstandings. For example, handouts are materials that reduce or summarize the illustrations in the presentation materials, and use the remaining space to write explanations of the illustrations and key points in bullet points.

[0163] As shown in (a) of Fig. 13, the presentation material includes, for example, a flow chart as an illustration. The arrows and x (symbols) shown in (a) of Fig. 13 are examples of connector information.

[0164] As shown in (b) of Figure 13, the handout includes, for example, an explanatory text for the illustration in addition to the illustration shown in (a) of Figure 13. The explanatory text is created by a person.

[0165] As shown in (a) of Fig. 14, the presentation material includes, for example, a graph as an illustration. The speech bubble shown in (a) of Fig. 14 is an example of connector information.

[0166] As shown in (b) of Figure 14, the handout includes, for example, an explanatory text for the illustration in addition to the illustration shown in (a) of Figure 14. The explanatory text is created by a person.

[0167] As shown in FIG. 15(a), the presentation materials include, for example, graphs and tables as illustrations.

[0168] As shown in (b) of Figure 15, the handout includes, for example, a graph summarizing the graphs and tables shown in (a) of Figure 15, as well as an explanatory text for the graph. The pentagon facing the pie chart shown on the right side of (b) of Figure 15 is an example of connector information indicating that the text within the pentagon is related to the pie chart. The explanatory text is created by a person.

[0169] Even for such presentation materials and handouts, the text content extraction device 100 of the present disclosure can improve the quality of the generated text data.

[0170] FIG. 16 shows an example of a technical paper. Technical papers may contain additional information such as drawings as shown in FIG. 1. Technical papers may also contain various additional information other than drawings, such as photographs, mathematical formulas, tables, and graphs. The position and size of such additional information are not fixed and are determined arbitrarily by the author. Even for technical papers containing such additional information, the text content extraction device 100 of the present disclosure can improve the quality of the generated text data.

[0171] Note that a character string specifying a drawing, such as "Figure 1," may be used as connector information. For example, the connector processing unit 110 may generate one short sentence by determining that the description following "Figure 1" in the character string is related to the drawing "Figure 1." Also, a short sentence may be created for each paragraph, for example. In other words, the text content extraction device 100 may extract one paragraph as one short sentence. A technical paper is an example of a document file 200.

[0172] FIG. 1 shows a schematic diagram of an example of a magazine. Magazines may contain additional information such as photographs. Magazines may also contain various additional information other than photographs, such as tables and graphs. The position and size of such additional information are not fixed and are determined arbitrarily by the creator. Even for magazines containing such additional information, the text content extraction device 100 of the present disclosure can improve the quality of the generated text data.

[0173] (Other embodiments) The text content extraction device 100 (an example of an information processing device) according to one or more aspects has been described above based on the embodiments, but the present disclosure is not limited to these embodiments. As long as it does not deviate from the spirit of the present disclosure, various modifications conceivable by a person skilled in the art to the present embodiments and embodiments constructed by combining components of different embodiments may also be included in the present disclosure.

[0174] For example, the text content extraction device 100 according to the above embodiment may determine that the single-page data 201, which contains a mixture of fixed and non-fixed parts on one page, has a non-fixed layout or a mixed layout. The layout recognition unit 102 may determine which of three or more layout types the data belongs to.

[0175] Furthermore, the text content extraction device 100 according to the above embodiment may not include the connector processing unit 110, or may not include one of the position-based reordering order estimation unit 112 and the context-based reordering order estimation unit 113. For example, when one of the position-based reordering order estimation unit 112 and the context-based reordering order estimation unit 113 is not included, the text content extraction device 100 may include the connector processing unit 110. Furthermore, when both the position-based reordering order estimation unit 112 and the context-based reordering order estimation unit 113 are included, the text content extraction device 100 may not include the connector processing unit 110, and may further include only one of the rectangular layout processing unit 103 and the non-rectangular layout processing unit 107.

[0176] Furthermore, the fixed layout in the above embodiment may include objects such as connectors. In this case, the connector processing unit 110 may rearrange and / or connect the short sentences included in the fixed layout based on the connectors.

[0177] Furthermore, in the above embodiment, an example was described in which the shape of the frame in the standard layout is rectangular, but the shape of the frame is not limited to being rectangular, and may be, for example, a circle, an oval, a square, or a polygon other than a rectangle.

[0178] Furthermore, in the above embodiment, an example has been described in which the short sentence concatenation processing unit 111 determines the order of each short sentence using both position and context, regardless of whether the layout of the single-page data 201 is a standard layout or an atypical layout, but this is not limited to this. For example, if the layout of the single-page data 201 is atypical, the short sentence concatenation processing unit 111 may determine the order of each short sentence using both position and context, and if the layout of the single-page data 201 is standard, the short sentence concatenation processing unit 111 may determine the order of each short sentence using either position or context.

[0179] In the above embodiment, the context-based sorting order estimation unit 113 estimates a context order proposal using a language model, but the method of estimating a context order proposal is not limited to this. The context-based sorting order estimation unit 113 may estimate a context order proposal using any other known method that takes into account the contents of multiple short sentences.

[0180] Furthermore, in the above embodiment, an example of generating a sub-text for each page has been described, but this is not limited to this. For example, a page may be divided into two or more areas, and a sub-text may be generated for each divided area, or a sub-text may be generated for each of multiple pages.

[0181] In the above embodiments, each component may be configured with dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0182] The order in which the steps in the flowchart are executed is merely an example for specifically explaining the present disclosure, and an order other than the above may be used. Also, some of the steps may be executed simultaneously (in parallel) with other steps, or some of the steps may not be executed.

[0183] The division of functional blocks in the block diagram is an example, and multiple functional blocks may be realized as a single functional block, one functional block may be divided into multiple blocks, or some functions may be moved to another functional block.Furthermore, the functions of multiple functional blocks having similar functions may be processed in parallel or in time-sharing by a single piece of hardware or software.

[0184] Furthermore, the text content extraction device 100 according to the above-described embodiments and the like may be realized as a single device or may be realized by multiple devices. When the text content extraction device 100 is realized by multiple devices, the components of the text content extraction device 100 may be distributed in any manner among the multiple devices. When the text content extraction device 100 is realized by multiple devices, the communication method between the multiple devices is not particularly limited, and may be wireless communication or wired communication. Furthermore, wireless communication and wired communication may be combined between the devices.

[0185] Furthermore, each component described in the above embodiments may be implemented as software or, typically, as an LSI, an integrated circuit. These components may be integrated individually on a single chip, or some or all of them may be integrated on a single chip. While LSI is used here, it may also be referred to as an IC, system LSI, super LSI, or ultra LSI depending on the level of integration. Furthermore, the integration method is not limited to LSI; it may be implemented using a dedicated circuit (a general-purpose circuit that executes a dedicated program) or a general-purpose processor. It is also possible to use a field programmable gate array (FPGA), which can be programmed after LSI fabrication, or a reconfigurable processor, which allows the connection or settings of circuit cells within an LSI to be reconfigured. Furthermore, if an integrated circuit technology that can replace LSI emerges due to advances in semiconductor technology or a derivative technology, that technology may naturally be used to integrate the components.

[0186] A system LSI is an ultra-multifunctional LSI manufactured by integrating multiple processing units on a single chip, and is specifically a computer system consisting of a microprocessor, ROM, RAM, etc. The ROM stores computer programs. The system LSI achieves its functions when the microprocessor operates in accordance with the computer programs.

[0187] Furthermore, one aspect of the present disclosure may be a computer program that causes a computer to execute each of the characteristic steps included in the information processing method shown in FIG.

[0188] Furthermore, for example, the program may be a program to be executed by a computer. Another aspect of the present disclosure may be a computer-readable non-transitory recording medium on which such a program is recorded. For example, such a program may be recorded on a recording medium and distributed or circulated. For example, the distributed program may be installed in a device having another processor, and the program may be executed by the processor, thereby causing the device to perform each of the above processes. [Industrial Applicability]

[0189] The present disclosure is useful for an information processing device or the like that generates text data from a document file. [Explanation of symbols]

[0190] 100 Text content extraction device (information processing device) 101 Page Recognition and Selection 102 Layout Recognition Unit 103 Rectangle Layout Processing Unit 104 Split configuration recognition unit 105 Minimum Unit Pair Acquisition Unit 106 Block text processing section 107 Non-rectangular layout processing unit 108 Text Box Recognition Unit 109 Text processing section in table 110 Connector processing section 111 Short sentence concatenation processing unit 112 Position-based ordering estimation unit 113 Context-based ordering estimation unit 114 Reordering and Concatenation Processing Unit 115 Large Sentence Concatenation Processing Unit 116 Text data concatenation 117 Storage section 118 Communications Department 200 document files 201, 201a, 201b Single page data 202, 202a, 202b Short text data 203 Chinese text data 204 Large Text Data 205 Text Data

Claims

1. An information processing device comprising a memory for storing a program and a processor for executing the program, When the processor executes the program, the processor Get the document file, generating a plurality of first sentences, each of which is composed of a group of character strings, and position information relating to a position of each of the plurality of first sentences in the document file, based on the document file; estimating a first sorting order indicating an ordering order of the plurality of first sentences based on a context of each of the plurality of first sentences; estimating a second sorting order indicating an sorting order of the plurality of first sentences based on the position information of each of the plurality of first sentences; generating a second sentence by rearranging and / or concatenating the first sentences based on the first sorting order and the second sorting order; Information processing device.

2. acquiring connector information indicating a relationship between two or more words and / or sentences in the document file based on the document file; generating a first sentence by rearranging and / or connecting the two or more words and / or sentences based on the connector information; The information processing device according to claim 1 .

3. the document file is image data, extracting the plurality of first sentences based on pixel values ​​of the pixels of the image data and / or acquiring the connector information; The information processing device according to claim 2 .

4. the document file includes one or more objects; Extracting the plurality of first sentences based on at least one of the position and the size of the one or more objects. The information processing device according to claim 2 .

5. The first sorting order is estimated by inputting the plurality of first sentences into a language model that has been trained so that, when a group of multiple character strings is input, information indicating the sorting order of the group of multiple character strings is output. The information processing device according to claim 1 .

6. The document file includes information indicating at least one of defect information of the product and defect information of the manufacturing process of the product. The information processing device according to claim 1 .

7. Recognizing the layout of the document file; If the recognized layout is an atypical layout, the second sentence is generated by rearranging and / or concatenating the first sentences based on the first sorting order and the second sorting order. The information processing device according to claim 1 .

8. The document file with the non-standard layout includes at least one of a presentation material, a technical paper, and a magazine. The information processing device according to claim 7 .

9. the document file includes a plurality of pages; generating the second sentence for each of the plurality of pages; generating a third sentence based on the plurality of second sentences; The information processing device according to claim 1 .

10. Transmitting at least one of the second sentence and the third sentence to the outside. The information processing device according to claim 9 .

11. Get the document file, generating a plurality of first sentences, each of which is composed of a group of character strings, and position information relating to a position of each of the plurality of first sentences in the document file, based on the document file; estimating a first sorting order indicating an ordering order of the plurality of first sentences based on a context of each of the plurality of first sentences; estimating a second sorting order indicating an sorting order of the plurality of first sentences based on the position information of each of the plurality of first sentences; generating a second sentence by rearranging and / or concatenating the first sentences based on the first sorting order and the second sorting order; Information processing methods.

12. An information processing device comprising a memory for storing a program and a processor for executing the program, When the processor executes the program, the processor Get the document file, generating a plurality of first sentences, each of which is composed of a group of character strings, based on the document file; generating a second sentence by rearranging and / or concatenating the plurality of first sentences; Further, connector information indicating a relationship between two or more words and / or sentences in the document file is acquired based on the document file; generating a first sentence by rearranging and / or concatenating the two or more words and / or sentences based on the connector information; The plurality of first sentences includes the one first sentence. Information processing device.

13. Get the document file, generating a plurality of first sentences, each of which is composed of a group of character strings, based on the document file; generating a second sentence by rearranging and / or concatenating the plurality of first sentences; Further, connector information indicating a relationship between two or more words and / or sentences in the document file is acquired based on the document file; generating a first sentence by rearranging and / or concatenating the two or more words and / or sentences based on the connector information; The plurality of first sentences includes the one first sentence. Information processing methods.

14. A recording medium on which a program for causing a computer to execute the information processing method according to claim 11 or 13 is recorded.

Images