Structuralization device, structuralization method, and structuralization program

Abstract

To make the structure of document data highly accurate.SOLUTION: A structuralization device can access a processing module pool for storing a plurality of processing modules that can execute processing based on features related to a layout in document data, and a template data pool for storing template data obtained by regulating two or more processing modules combined according to dependency relations among the plurality of processing modules, acquires document data to be structuralized, extracts specific template data from the template data pool on the basis of a result of a selection input of features related to the layout of the document data to be structuralized, and outputs first structuralization data obtained by structuralizing the document data to be structuralized with the features related to the layout by executing two or more processing modules constituting the extracted specific template data according to the dependency relations of the two or more specific processing modules.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a structuring device, a structuring method, and a structuring program for structuring a structured object. [Background technology]

[0002] By structuring unstructured documents, information extraction and information retrieval from documents can be performed with high accuracy, increasing the usefulness of the information obtained. For example, academic literature search services such as Google Scholar and Semantic Scholar have extracted abstracts from academic papers and used them in search systems. In addition, while large amounts of text data are required to train large-scale language models, using structured data makes it possible to build high-performance models with smaller amounts of data.

[0003] Patent Document 1 is a related technology for structuring unstructured documents. Patent Document 1 states, "A deep learning-based method for extracting structured information from a negative document, the method being performed by at least one processor of a computing device." Patent Document 1 also states, "The deep learning-based method for extracting structured information from an unstructured document includes the steps of receiving an input image, and converting, from the input image, a token sequence indicating the structure of the input image into structured information by an encoder-decoder model based on deep learning."

[0004] Patent Document 2 also states that "a structured document is defined that includes a hierarchy of structural elements constructed by analyzing an unstructured document." Patent Document 2 also states that "to define a structured document, basic elements of the unstructured document are used to identify various geometric attributes of the unstructured document. The identified geometric attributes and other attributes of the basic elements are used to define related basic elements (e.g., words, paragraphs, and bond graphs), structural elements such as tables, guides, and paragraphs, and to define the reading flow of the basic elements and structural elements."

[0005] Patent Document 3 also states, "A method for converting content information from an unstructured data format to a structured data format." Patent Document 3 also states, "A conversion module converts content information from an unstructured data format to a structured data format according to rules."

[0006] Furthermore, Patent Document 4 states that "conversion from unstructured documents to structured documents is performed using rules created directly from preset logical structure definitions, making it easy to create structured documents that conform to the logical structure of each document."

[0007] There have been many attempts to structure unstructured document data. For example, the technology described in Patent Document 1 discloses a means for extracting structured information from unstructured documents end-to-end using deep learning. The technology described in Patent Document 2 uses a rule-based approach to structure unstructured documents by implementing a predefined processing flow. The technology described in Patent Document 3 provides a method for converting input data from an unstructured data format to a structured data format, and makes it possible to change the granularity of the information to be displayed depending on the type of display client. The technology described in Patent Document 4 discloses a means for converting unstructured documents into structured data by processing according to a predefined pattern. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] Patent Publication No. 2023-080045 [Patent Document 2] JP 2016-006661 A [Patent Document 3] Special Publication No. 2017-529622 [Patent Document 4] Japanese Patent Application Publication No. 09-069101 Summary of the Invention [Problem to be solved by the invention]

[0009] In the above-mentioned conventional technology, it is difficult to perform structuring processing according to differences in the layout of the input document, so structuring cannot always be performed with low noise. As an example, structuring from a single-column document was assumed, but if the input document is actually a double-column document, noise will occur because structuring of sentences across columns was not assumed.

[0010] An object of the present invention is to improve the accuracy of structuring document data. [Means for solving the problem]

[0011] A structuring device according to one aspect of the invention disclosed in this application has a processor that executes a program and a storage device that stores the program, and is capable of accessing a processing module pool that stores a plurality of processing modules capable of executing processing based on layout-related features in document data, and a template data pool that stores template data in which two or more processing modules combined in accordance with dependency relationships between the plurality of processing modules are defined, wherein the processor executes an acquisition process to acquire document data to be structured, an extraction process to extract specific template data from the template data pool based on a result of a selection input of layout-related features of the document data to be structured acquired by the acquisition process, and a structuring process to output first structured data in which the document data to be structured is structured using the layout-related features by executing two or more specific processing modules that constitute the specific template data extracted by the extraction process in accordance with the dependency relationships between the two or more specific processing modules. Effect of the Invention

[0012] According to the representative embodiment of the present invention, it is possible to structure document data with high accuracy. Problems, configurations and effects other than those described above will become apparent from the following description of the embodiment. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a block diagram of an example of a functional configuration of a structuring device according to a first embodiment. [Diagram 2] FIG. 2 is an explanatory diagram showing an example of document data to be structurized shown in FIG. [Diagram 3] FIG. 3 is a flowchart illustrating an example of a data load processing procedure performed by the data load module. [Figure 4] FIG. 4 is a flowchart illustrating an example of a line extraction process procedure performed by the line extraction module. [Diagram 5] FIG. 5 is a flowchart illustrating an example of a procedure for a footnote extraction process performed by the footnote extraction module. [Figure 6] FIG. 6 is a flowchart illustrating an example of a diagram extraction process procedure performed by the diagram extraction module. [Figure 7] FIG. 7 is a flowchart illustrating an example of a procedure for a caption extraction process performed by the caption extraction module. [Figure 8] FIG. 8 is a flowchart illustrating an example of a mathematical formula extraction process performed by the mathematical formula extraction module. [Figure 9] FIG. 9 is a flowchart illustrating an example of a procedure for the auxiliary information extraction process performed by the auxiliary information extraction module. [Figure 10] FIG. 10 is a flowchart showing an example of a paragraph extraction process procedure performed by the paragraph extraction module. [Figure 11] FIG. 11 is a flowchart illustrating an example of a chapter / section structure detection process performed by the chapter / section structure detection module. [Figure 12] FIG. 12 is a flowchart illustrating an example of a column combining process performed by the column combining module. [Figure 13]FIG. 13 is a flowchart illustrating an example of a page merging process procedure performed by the page merging module. [Figure 14] FIG. 14 is a flowchart illustrating an example of an output process procedure by the output module. [Figure 15] FIG. 15 is a flowchart showing a detailed example of the chapter / section structure creating process (step S1403) by the output module. [Figure 16] FIG. 16 is an explanatory diagram illustrating an example of structured data. [Figure 17] FIG. 17 is an explanatory diagram showing an example of dependency data in which dependencies are defined. [Figure 18] FIG. 18 is an explanatory diagram showing an example of the template data. [Figure 19] FIG. 19 is an explanatory diagram illustrating an example of the setting data. [Figure 20] FIG. 20 is a flowchart illustrating an example of a procedure for extracting template data. [Figure 21] FIG. 21 is an explanatory diagram of Example 1 of the setting execution screen of the structuring device according to the first embodiment. [Figure 22] FIG. 22 is an explanatory diagram of Example 2 of the setting execution screen of the structuring device according to the first embodiment. [Figure 23] FIG. 23 is a flowchart of an example of a procedure for checking a dependency relationship between processing modules by the structuring device according to the first embodiment. [Figure 24] FIG. 24 is a block diagram of an example of a hardware configuration of the structuring device according to the first embodiment. [Diagram 25] FIG. 25 is a block diagram of a first example of a functional configuration of a structuring device according to the second embodiment. As illustrated in FIG. [Figure 26] FIG. 26 is a block diagram of a second example of a functional configuration of the structuring device according to the second embodiment. As illustrated in FIG. [Figure 27] FIG. 27 is an explanatory diagram of Example 1 of the setting execution screen according to the second embodiment. [Figure 28] FIG. 28 is an explanatory diagram of Example 2 of the setting execution screen according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS EXAMPLES

[0014] <Functional configuration of the structuring device> 1 is a block diagram showing an example of a functional configuration of a structuring device according to a first embodiment. The structuring device 100 receives, as input data, document data to be structurized 101, a processing module pool 102, a template data pool 103, and a template classifier 104. The structuring device 100 includes a classification unit 110 and a structuring processing unit 130.

[0015] The processing module pool 102 is a data area that stores a plurality of processing modules for structuring the document data to be structured 101. The processing module pool 102 has a plurality of processing modules, specifically, for example, a data load module 120, a line extraction module 121, a footnote extraction module 122, a figure / table extraction module 123, a caption extraction module 124, a formula extraction module 125, an auxiliary information extraction module 126, a paragraph extraction module 127, a chapter / section structure detection module 128, a column combination module 129, a page combination module 12A, and an output module 12B.

[0016] When there is no need to distinguish between data load module 120 to output module 12B, they will be referred to as processing modules 12#. Each of processing modules 12# is a software module that executes a unique process.

[0017] The template data pool 103 holds one or more template data that satisfy dependencies between a plurality of processing modules 12#. The template data includes a plurality of processing modules 12# in the processing module pool 102, and is configured according to the execution order of the plurality of processing modules 12#.

[0018] A classification unit 110 classifies the template data pool 103 using a template classifier 104, and outputs template data 111 suited to the layout of the document data 101 to be structurized.

[0019] The structuring processor 130 inputs the document data to be structurized 101, extracts a group of processing modules defined in the template data 111 from the processing module pool 102, and executes the extracted group of processing modules in the order defined in the template data 111. The structuring processor 130 then outputs structured data 131.

[0020] The structuring device 100 can handle any language, but in the first embodiment, an example in which English is handled will be described.

[0021] [Document data to be structured 101] Fig. 2 is an explanatory diagram showing an example of the document data to be structured 101 shown in Fig. 1. The document data to be structured 101 may be document data of any domain, but in the first embodiment, for convenience, an example of handling academic papers will be described. The document data to be structured 101 includes document data to be converted into structured document data. The data format of the document data is not important, but in the first embodiment, the document data to be structured 101 is a document in Portable Document Format (PDF) format for convenience.

[0022] 2, only the first page of the document data to be structurized 101 is shown, but there are pages 2 and onward. The upper left vertex of each page of the document data to be structurized 101 is set as the coordinate origin O, the width direction of the page is set as the X axis, and the height direction of the page is set as the Y axis.

[0023] The area above the first line L1, i.e., the area whose Y coordinate value is equal to or less than the Y coordinate value of the first line L1, is referred to as the header area 201. The area below the second line L2, i.e., the area whose Y coordinate value is equal to or more than the Y coordinate value of the first line L1, is referred to as the footer area 202. In the footer area 202, for example, a character string 220 indicating public information of the document data 101 to be structurized is written.

[0024] The area to the left of the first column line C1, i.e., the area whose X coordinate value is equal to or less than the coordinate value of the first column line C1, is referred to as a left margin area 203. The area to the right of the second column line C2, i.e., the area whose X coordinate value is equal to or greater than the coordinate value of the second column line C2, is referred to as a right margin area 204.

[0025] The area surrounded by the first row line L1, the second row line L2, the first column line C1, and the second column line C2 is referred to as the body region 205. The data in the body region 205 is referred to as the body. For convenience, it will be referred to as the body 205 below. In the body 205, a character string 251 describing "Test Paper" is a heading, and a character string 252 describing "Test Author" is the name of the author.

[0026] In the main text 205 , the bar graph 253 is a diagram, and the character string 254 below the bar graph 253 , “Fig. 1 is a bar graph,” is a caption for the bar graph 253 .

[0027] In the main text 205, a character string 255 indicating "y=Ax+B···(1)" is a mathematical formula. A character string 256 describing "* This work was conducted when the author was a master's student at the University" is a footnote.

[0028] In the body 205, a character string 257 beginning with "Abstract" other than the character strings 251, 252, and 254 to 256 is a body character string. The character string 257 is written in double columns.

[0029] [Processing module 12#] Returning to FIG. 1, each processing module 12# will now be described in detail.

[0030] [Data Load Module 120] The data load module 120 is, for example, a module that executes data loading of the document data to be structurized 101. The data load module 120 extracts information required for structuration (reading order, tokens, meta information, objects) from the document data to be structurized 101 for each page of the document data to be structurized 101.

[0031] FIG. 3 is a flowchart showing an example of a data loading process procedure performed by the data loading module 120. As shown in FIG.

[0032] (Step S301) The data load module 120 acquires the page height and width of the document data to be structurized 101. The page height is the length of the page in the Y-axis direction, and the page width is the length of the page in the X-axis direction.

[0033] (Step S302) The data load module 120 extracts the reading order of the tokens. A token is a character string indicating a processing unit, such as a word. If the document data 101 to be structurized has the reading order of the words embedded as metadata, the data load module 120 determines the reading order from the metadata.

[0034] The data load module 120 estimates the reading order if the reading order is not embedded in the document data to be structurized 101. The data load module 120 can estimate the reading order using a machine learning model such as LayoutReader, for example.

[0035] In the case of the document data 101 to be structurized in FIG. 2, the data load module 120 extracts the reading order of the tokens in the character strings 220, 251, 252, and 254-257.

[0036] (Step S303) The data load module 120 extracts tokens from the document data to be structured 101 according to the reading order determined or estimated in step S302. The data load module 120 stores the token strings in a page as page instances in the reading order for each page.

[0037] In the case of the document data 101 to be structurized in FIG. 2, the data load module 120 extracts tokens from within the character strings 220, 251, 252, and 254 to 257, and stores them as an instance of the first page.

[0038] (Step S304) The data load module 120 extracts meta information for each token. The meta information is associated with the token in the document data to be structurized 101 as part of the metadata, and includes, for example, the font size of the characters constituting the token, the font name of the characters, and the coordinate values ​​of the token or the characters constituting the token. If meta information is not associated as part of the metadata in the document data to be structurized 101, the structuring device 100 applies preset default meta information. The data load module 120 associates the meta information in the page with the token and stores it for each page as an instance of the page.

[0039] (Step S305) The data load module 120 extracts objects such as lines, figures, and images from the document data to be structured 101. The data load module 120 stores the objects in the page as object instances in the page instance for each page, in association with the page number. This completes the data load process by the data load module 120.

[0040] In the case of the document data 101 to be structurized in FIG. 2, the data load module 120 extracts the bar graph 253 and stores it as an instance of the first page.

[0041] [Line Extraction Module 121] Line extraction module 121 refers to the layout of tokens in document data to be structurized 101 (i.e., the coordinate values ​​of tokens), identifies lines in document data to be structurized 101, and extracts line elements of the identified lines from the page instance. A line element is a token string arranged in a line. For example, it is a token string of each line in character strings 220, 251, 252, 254 to 257. Line extraction module 121 is executed, for example, according to the algorithm shown in the flowchart of FIG.

[0042] 4 is a flowchart showing an example of a line extraction processing procedure by line extraction module 121. Note that line extraction module 121 uses a line information cache. Tokens of the same line are held in the line information cache by execution of line extraction module 121.

[0043] (Step S400) The line extraction module 121 sets a target token in the page instance. A target token is a token to be extracted. In the initial state, the target token is the first token in reading order. In the case of the document data 101 to be structured in FIG. 2, the first token is "Test" in the character string 251. When moving from step S403, the token ("Paper") next in reading order to the target token ("Test") registered in the line information cache at that time in step S403 is set as the new target token.

[0044] (Step S401) The line extraction module 121 determines whether there is a token immediately preceding the target token in reading order. If step S401 is true (step S401: True), the process proceeds to step S403. If step S401 is false (step S401: False), the target token is the first token in reading order, so the process proceeds to step S402.

[0045] (Step S402) Line extraction module 121 stores the token to be processed in the line information cache, and returns to step S400. Specifically, for example, line extraction module 121 registers the first token "Test" in character string 251 in the line information cache.

[0046] (Step S403) The line extraction module 121 calculates the absolute value of the difference between the average value of the Y coordinate values ​​(column coordinate values) of the tokens included in the line information cache and the Y coordinate value of the token to be processed. In step S402, one or more tokens are held in the line information cache, and the line extraction module 121 calculates the average value of the Y coordinate values ​​of one or more tokens.

[0047] Example 403-1: For example, the token to be processed is "Paper" in the character string 251. "Test" is stored as a token in the line information cache. The line extraction module 121 calculates the absolute value of the difference between the Y coordinate value of "Test" in the line information cache and the Y coordinate value of "Paper", which is the token to be processed.

[0048] Example 403-2: For example, the token to be processed is "Test" in character string 252. The line information cache stores "Test" and "Paper" as tokens. Line extraction module 121 calculates the absolute value of the difference between the average value of the Y coordinate value of "Test" and the Y coordinate value of "Paper" in the line information cache and the Y coordinate value of "Test" in character string 252, which is the token to be processed.

[0049] (Step S404) The line extraction module 121 judges whether the absolute value of the difference calculated in step S403 is equal to or less than a threshold value. The threshold value can be set by the user, but since the coordinate system in the first embodiment is standardized, a setting value assumed on the structuring device 100 side may be used as a default value of the threshold value.

[0050] If step S404 is true (step S404: True), the token included in the line information cache and the target token are considered to belong to the same line, so the process proceeds to step S402. For example, this is the case in example 403-1 above. In this case, in step S402, the line extraction module 121 registers the target token, "Paper", in the line information cache.

[0051] If step S404 is false (step S404: False), it is determined that the token to be processed is not included in the same line as the token included in the line information cache, and the process proceeds to step S405. For example, this corresponds to the case of the above-mentioned example 403-2.

[0052] (Step S405) The line extraction module 121 registers the token string in the line information cache as a line element, as a line instance in the page instance. For example, in the above example 403-2, since the line information cache stores "Test" and "Paper" as tokens, the line extraction module 121 registers the token strings "Test" and "Paper" as line instances in the page instance.

[0053] (Step S406) The line extraction module 121 initializes the line information cache with the current token to be processed. That is, the current token to be processed is stored in the line information cache as the first token of the next line. For example, in the case of the above-mentioned example 403-2, the line extraction module 121 deletes the token "Test" other than "Paper" in the character string 251, which is the current token to be processed, from the token string "Test" and "Paper".

[0054] In this way, by applying the line extraction process to the document data to be structured 101 after application of the data load module 120, it is possible to extract all line elements contained in the document data to be structured 101 from the page instance. This completes the line extraction process by the line extraction module 121.

[0055] [Footnote Extraction Module 122] The footnote extraction module 122 extracts footnotes from line elements in a line instance. A token string is stored in the line instance for each line. A footnote is a character string indicating a note added to the bottom of a page. The footnote extraction module 122 is executed on a page-by-page basis, for example, according to the algorithm shown in the flowchart of FIG. 5.

[0056] FIG. 5 is a flowchart showing an example of a procedure for the footnote extraction process performed by the footnote extraction module 122.

[0057] (Step S501) The footnote extraction module 122 attempts to estimate the coordinates of the footnote range in the processing target page of the structure target document data 101. The coordinate estimation can be realized, for example, by utilizing an object detection model such as X101 trained by the DocBank dataset.

[0058] (Step S502) As a result of the coordinate estimation attempt in step S501, the footnote extraction module 122 determines whether or not a coordinate area estimated to be a footnote (hereinafter, footnote estimated area) exists in the page of the document data to be structured 101. If step S502 is false (step S502: False), the footnote extraction process by the footnote extraction module 122 for the page ends, and if there is a next page, the footnote extraction module 122 executes the footnote extraction process with the next page as the page to be processed. If step S502 is true (step S502: True), the process proceeds to step S503.

[0059] (Step S503) The footnote extraction module 122 calculates an overlap rate between the footnote estimated region and the line element in the line instance. It is assumed that the footnote estimated region 261 is estimated in FIG.

[0060] Example 503-1: For example, the line element for which the overlap rate is to be calculated is the token string "Recently, several studies have succeeded" which is the bottom line of the left column in the character string 257. In this case, it is assumed that the overlap rate between the footnote inferred region 261 and the line element is calculated to be 40%.

[0061] Example 503-2: For example, the line element for which the overlap rate is to be calculated is the token string "* This work was conducted when the author," which is the string in the top line of the character string 256. In this case, the overlap rate between the footnote inferred region 261 and the line element is 100%. Similarly, the overlap rate of the token string "was a master's student at the University." in the bottom line of the character string 256 is also 100%.

[0062] (Step S504) The footnote extraction module 122 judges whether the overlap rate calculated in step S503 is equal to or greater than a threshold. The threshold can be set by the user, but since the overlap rate is calculated in the range of 0% to 100%, a set value assumed by the structuring device 100 may be used as a default value for the threshold. If step S504 is false (step S504: False), it is considered that there are no line elements to be extracted as footnotes, so the footnote extraction process by the footnote extraction module 122 for the page is terminated, and if there is a next page, the footnote extraction module 122 executes the footnote extraction process with the next page as the processing target page. If step S504 is true (step S504: True), the process proceeds to step S505.

[0063] In the above example 503-1, if the threshold is, for example, 60%, the token string "Recently, several studies have succeeded", which is a row element, is determined to not be a footnote, and step S504 is false (step S504: False).

[0064] On the other hand, in the case of example 503-2, the line elements, that is, the token strings in the upper and lower lines of the character string 256 are determined to be footnotes, and step S504 is true (step S504: True).

[0065] (Step S505) The footnote extraction module 122 extracts line elements to be footnotes from the line instance and deletes them from the line instance. In the case of the above example 503-2, the footnote extraction module 122 deletes each token string of the upper line and the lower line of the character string 256, which is the line element, from the line instance.

[0066] (Step S506) The footnote extraction module 122 associates the line element extracted in step S505 with the page number and the line number, and registers it in the instance of the footnote line as a line element of the footnote. After this, the footnote extraction process by the footnote extraction module 122 for the page is completed, and if there is a next page, the footnote extraction module 122 executes the footnote extraction process with the next page as the processing target page.

[0067] In the case of the above example 503-2, the footnote extraction module 122 associates each token string in the upper line and the lower line of the character string 256, which is a line element, with a page number and a line number and registers them as footnote line elements in an instance of the footnote line.

[0068] [Graph Extraction Module 123] The graphic / table extraction module 123 extracts line elements (hereinafter referred to as graphic / table line elements) contained in a graphic / table of the document data to be structurized 101 from line instances. The graphic / table extraction module 123 is executed on a page-by-page basis, for example, according to the algorithm shown in the flowchart of FIG.

[0069] FIG. 6 is a flowchart showing an example of a procedure for a diagram extraction process performed by the diagram extraction module 123. As shown in FIG.

[0070] (Step S601) The figure / table extraction module 123 attempts to estimate the coordinates of the figure / table range in the processing target page of the document data to be structured 101. The coordinate estimation can be realized by utilizing an object detection model such as X101 trained with the DocBank dataset, similar to the footnote extraction module 122 in Fig. 5. Furthermore, it can also be realized by an object detection model such as X101 trained with the Publaynet or TableBank dataset.

[0071] (Step S602) As a result of the coordinate estimation attempt in step S601, the graphic extraction module 123 determines whether or not a coordinate area estimated to be a graphic (hereinafter, graphic estimation area) exists. If step S602 is false (step S602: False), the graphic extraction process by the graphic extraction module 123 for the page ends, and if there is a next page, the graphic extraction module 123 executes the graphic extraction process on the next page as the processing target page. If step S602 is true (step S602: True), the process proceeds to step S603.

[0072] (Step S603) The diagram extraction module 123 determines whether any line elements of the row instance belong to the diagram estimated area. If step S603 is false (step S603: False), there are no line elements to be deleted from the row instance, so the diagram extraction process by the diagram extraction module 123 for the page ends, and if there is a next page, the diagram extraction module 123 executes the diagram extraction process on the next page as the processing target page. If step S603 is true (step S603: True), the process proceeds to 604.

[0073] (Step S604) The graphic extraction module 123 extracts and removes the corresponding line element from the line instance.

[0074] (Step S605) The diagram extraction module 123 registers the line element extracted in step S604 in the instance of the line in the diagram as a line element in the diagram. After this, the diagram extraction process by the diagram extraction module 123 for the page is completed, and if there is a next page, the diagram extraction module 123 executes the diagram extraction process on the next page as the processing target page.

[0075] In the example of Figure 2, since there are no tokens in the bar graph 253, nothing is registered in the instance of the line in the chart. However, if there is a token in the bar graph 253 (for example, the numbers on the scale are an explanation of the axis), it will be registered in the instance of the line in the chart as an element of the line in the chart.

[0076] [Caption Extraction Module 124] The caption extraction module 124 extracts line elements corresponding to the captions of the figures and tables. The caption extraction module 124 is executed on a page-by-page basis, for example, according to the algorithm shown in the flowchart of FIG.

[0077] FIG. 7 is a flowchart showing an example of a procedure for the caption extraction process performed by the caption extraction module 124.

[0078] (Step S701) The caption extraction module 124 attempts to estimate the coordinates of the range including the caption in the processing target page of the document data to be structured 101. The coordinate estimation can be realized by utilizing an object detection model such as X101 trained using the DocBank or Publaynet dataset, similar to the modules in Fig. 5 and Fig. 6.

[0079] (Step S702) The caption extraction module 124 determines whether or not a coordinate area estimated to be a caption (hereinafter, a caption estimated area) exists as a result of the coordinate estimation attempt in step S701. If step S702 is false (step S702: False), the caption extraction process by the caption extraction module 124 for the page ends, and if there is a next page, the caption extraction module 124 executes the caption extraction process on the next page as the processing target page. If step S702 is true (step S702: True), the process proceeds to step S703.

[0080] In the example of FIG. 2, it is assumed that two caption estimation areas 263 and 264 have been estimated.

[0081] (Step S703) The caption extraction module 124 attempts to estimate the coordinates of the figure / table range in the processing target page in the same manner as the figure / table extraction module 123. In the example of Fig. 2, it is assumed that a figure / table estimation area 262 has been estimated.

[0082] (Step S704) The caption extraction module 124 determines whether or not a graphic estimation area exists as a result of the coordinate estimation attempt in step S703. If step S704 is true (step S704: True), the process proceeds to step S705. If step S704 is false (step S704: False), the process proceeds to step S707.

[0083] (Step S705) The caption extraction module 124 regards the line elements in the caption estimation region as a caption, and calculates the centroid distance between the center of gravity of the graphic estimation region and the center of gravity of the caption estimation region. In the example of Fig. 2, the caption extraction module 124 calculates the centroid distance between the center of gravity of the graphic estimation region 262 and the center of gravity of the caption estimation region 263. In addition, the caption extraction module 124 calculates the centroid distance between the center of gravity of the graphic estimation region 262 and the center of gravity of the caption estimation region 264.

[0084] (Step S706) The caption extraction module 124 assigns, to each of the graphic row elements in the graphic row instance, a row element in the caption estimation region that has the smallest centroid distance from the graphic estimation region as a caption. In the example of FIG. 2, the caption estimation region that has the smallest centroid distance from the graphic estimation region 262 is the caption estimation region 263. Therefore, the caption extraction module 124 associates the token string "Fig. 1 is a bar graph.", which is the row element in the caption estimation region 263 that has the smallest centroid distance from the graphic estimation region 262, with each of the graphic row elements and stores it in the graphic row instance.

[0085] (Step S707) The caption extraction module 124 deletes the caption from the line instance. After that, the caption extraction process by the caption extraction module 124 for the page is completed, and if there is a next page, the caption extraction module 124 executes the caption extraction process on the next page as the processing target page.

[0086] In the above example, the caption extraction module 124 removes the caption line element "FIG. 1 is a bar graph." from the line instance.

[0087] [Formula Extraction Module 125] The mathematical expression extraction module 125 extracts row elements corresponding to mathematical expressions (hereinafter, mathematical expression row elements) from row elements in a row instance. The mathematical expression extraction module 125 is executed on a page-by-page basis according to the algorithm shown in the flowchart of FIG.

[0088] FIG. 8 is a flowchart showing an example of a mathematical formula extraction process performed by the mathematical formula extraction module 125.

[0089] (Step S801) The mathematical expression extraction module 125 attempts to estimate the coordinates of a mathematical expression range in a processing target page of the structure target document data 101. The coordinates of the mathematical expression range can be estimated by utilizing an object detection model such as X101 trained by the DocBank dataset, similar to the processing module 12# in Figs. 5 to 7.

[0090] (Step S802) The mathematical expression extraction module 125 judges whether or not there is a diagram area that is estimated to be a mathematical expression (hereinafter, mathematical expression estimated area). If step S802 is false (step S802: False), the mathematical expression extraction process by the mathematical expression extraction module 125 for the page is completed, and if there is a next page, the mathematical expression extraction module 125 executes the mathematical expression extraction process on the next page as the processing target page. If step S802 is true (step S802: True), the process proceeds to step S803.

[0091] (Step S803) The mathematical expression extraction module 125 calculates an overlap rate between the mathematical expression inference region and the row elements in the row instance. It is assumed that the mathematical expression inference region 265 in FIG.

[0092] Example 803-1: For example, the row element for which the overlap rate is to be calculated is the token string “y=Ax+B···(1)”, which is the character string 255. In this case, it is assumed that the overlap rate between the formula inference area 265 and the row element is calculated to be 100%.

[0093] (Step S804) The mathematical expression extraction module 125 judges whether the overlap rate calculated in step S803 is equal to or greater than a threshold value. The threshold value can be set by the user, but since the overlap rate is calculated in the range of 0% to 100%, a set value assumed on the structuring device 100 side may be used as the default value of the threshold value. If step S804 is false (step S802: False), the mathematical expression extraction process by the mathematical expression extraction module 125 for the page is completed, and if there is a next page, the mathematical expression extraction module 125 executes the mathematical expression extraction process on the next page as the processing target page. If step S804 is true (step S804: True), the process proceeds to step S805.

[0094] In the above example 803-1, if the threshold is, for example, 60%, the token string “y=Ax+B···(1)”, which is a row element, is determined to be a mathematical expression, and step S804 becomes true (step S504: True).

[0095] (Step S805) The mathematical expression extraction module 125 extracts the row elements determined to be mathematical expressions from the row instance as mathematical expression row elements and deletes them from the row instance. In the case of the above-mentioned example 803-1, the mathematical expression extraction module 125 deletes the token string "y=Ax+B···(1)" of the character string 255, which is the row element, from the row instance.

[0096] (Step S806) The mathematical expression extraction module 125 associates the line element extracted in step S805 with the page number and line number, and registers it as a mathematical expression line element in the instance of the mathematical expression corresponding line. After this, the mathematical expression extraction process by the mathematical expression extraction module 125 for the page is completed, and if there is a next page, the mathematical expression extraction module 125 executes the mathematical expression extraction process on the next page as the processing target page.

[0097] In the case of the above example 803-1, the formula extraction module 125 associates the token sequence “y=Ax+B···(1)” of the string 255, which is a line element, with the page number and line number and registers it as a formula line element in an instance of the formula corresponding line.

[0098] [Auxiliary information extraction module 126] The auxiliary information extraction module 126 extracts auxiliary information such as a header element, a footer element, etc. The auxiliary information extraction module 126 is executed on a line-by-line basis, for example, according to the algorithm shown in the flowchart of FIG.

[0099] FIG. 9 is a flowchart showing an example of a procedure for the auxiliary information extraction process performed by the auxiliary information extraction module 126.

[0100] (Step S901) The auxiliary information extraction module 126 determines whether the Y coordinate value of the top end of the row element is equal to or less than the first row threshold. In the example of Fig. 2, the auxiliary information extraction module 126 determines whether the Y coordinate value of the top end of the row element is equal to or less than the Y coordinate value of the first row line L1. If step S901 is true (step S901: True), the process proceeds to step S903. If step S901 is false (step S901: False), the process proceeds to step S902.

[0101] (Step S902) The auxiliary information extraction module 126 registers the line element as a header line element in the instance of the auxiliary information, and proceeds to step S909.

[0102] (Step S903) The auxiliary information extraction module 126 determines whether the Y coordinate value of the bottom end of the row element is equal to or greater than the second row threshold. In the example of Fig. 2, the auxiliary information extraction module 126 determines whether the Y coordinate value of the bottom end of the row element is equal to or less than the Y coordinate value of the second row line L2. If step S903 is true (step S903: True), the process proceeds to step S905. If step S903 is false (step S903: False), the process proceeds to step S904.

[0103] (Step S904) The auxiliary information extraction module 126 registers the line element as a footer line element in the auxiliary information instance, and proceeds to step S909. In the example of Fig. 2, the two-column token string "Annual Meeting 2023, pages 1234-1244" and "July 9-14, 2023." indicating the character string 220 is registered as a footer line element in the auxiliary information instance.

[0104] (Step S905) The auxiliary information extraction module 126 determines whether the X coordinate value of the left end of the row element is equal to or less than the first column threshold. In the example of Fig. 2, the auxiliary information extraction module 126 determines whether the X coordinate value of the left end of the row element is equal to or less than the X coordinate value of the first column line C1. If step S905 is true (step S905: True), the process proceeds to step S907. If step S905 is false (step S905: False), the process proceeds to step S906.

[0105] (Step S906) The auxiliary information extraction module 126 registers the line element in question as a line element within the left margin in an instance of the auxiliary information.

[0106] (Step S907) The auxiliary information extraction module 126 determines whether the X coordinate value of the right end of the row element is equal to or greater than the second column threshold. In the example of Fig. 2, the auxiliary information extraction module 126 determines whether the X coordinate value of the right end of the row element is equal to or greater than the X coordinate value of the second column line C2. If step S907 is true (step S907: True), the auxiliary information extraction process by the auxiliary information extraction module 126 for the row is completed, and if there is a next row, the auxiliary information extraction module 126 executes the auxiliary information extraction process with the next row as the processing target row. If step S907 is false (step S907: False), the process proceeds to step S908.

[0107] (Step S908) The auxiliary information extraction module 126 registers the line element in question as a line element within the right margin in an instance of the auxiliary information.

[0108] (Step S909) The auxiliary information extraction module 126 deletes the corresponding line element from the line instance. In the example of Fig. 2, the two-column token string "Annual Meeting 2023, pages 1234-1244" and "July 9-14, 2023." representing the character string 220 are deleted from the line instance.

[0109] Thereafter, the auxiliary information extraction process by the auxiliary information extraction module 126 for that page is completed, and if there is a next page, the auxiliary information extraction module 126 executes the auxiliary information extraction process on the next page as the processing target page.

[0110] The thresholds in steps S901, S903, S905, and S907 can be set by the user, but since the coordinate system in the first embodiment is standardized, the setting values ​​assumed on the structuring device 100 side may be used as default values ​​for the thresholds.

[0111] [Paragraph Extraction Module 127] The paragraph extraction module 127 extracts paragraph elements from line instances. The paragraph extraction module 127 is executed on a line-by-line basis, for example, according to the algorithm shown in the flowchart of FIG.

[0112] Fig. 10 is a flowchart showing an example of a paragraph extraction processing procedure by the paragraph extraction module 127. In the example of Fig. 2, at this stage, token strings indicating character strings 251, 252, and 257 among character strings 220, 251, 252, and 254 to 257 remain in the line instance. Note that the paragraph extraction module 127 uses a paragraph information cache. Initially, paragraph information is empty. Tokens of the same paragraph are held in the paragraph information cache by the execution of the paragraph extraction module 127.

[0113] (Step S1001) The paragraph extraction module 127 determines whether or not there is a line element immediately before the line element to be processed in the line instance. The line element to be processed is initially the first line element in the reading order of the line instance. In the case of the document data 101 to be structured in FIG. 2, the character string 251 "Test Paper" is the first line element. If the line element to be processed is the first line element, step S1001 is false (step S1001: False).

[0114] If step S1001 is true (step S1001: True), the process proceeds to step S1003. If step S1001 is false (step S1001: False), the process proceeds to step S1002.

[0115] (Step S1003) The paragraph extraction module 127 stores the line element to be processed in the paragraph information cache. When the paragraph extraction process by the paragraph extraction module 127 for the line is completed and there is a next line, the paragraph extraction module 127 executes the paragraph extraction process with the next line as the processing target line.

[0116] (Step S1002) The paragraph extraction module 127 calculates the absolute value of the difference between the X coordinate value of the right end of the line element to be processed and the X coordinate value of the right end of the immediately preceding line element (hereinafter referred to as the right end absolute value), and calculates the absolute value of the difference between the X coordinate value of the left end of the line element to be processed and the X coordinate value of the left end of the immediately preceding line element (hereinafter referred to as the left end absolute value).

[0117] (Step S1004) The paragraph extraction module 127 judges whether or not both the right end absolute value and the left end absolute value calculated in step S1002 are equal to or less than a threshold value. The threshold value can be set by the user, but since the coordinate system in the first embodiment is standardized, a setting value assumed on the structuring device 100 side may be used as a default value of the threshold value. If step S1004 is true (step S1004: True), the immediately preceding line element and the line element to be processed are considered to belong to the same paragraph, and the process proceeds to step S1003. If step S1004 is false (step S1004: False), the process proceeds to step S1005.

[0118] (Step S1005) The paragraph extraction module 127 judges whether only the left edge absolute value is equal to or smaller than the threshold value. The threshold value is the same value as in step S1004. If step S1005 is false (step S1005: False), the process proceeds to step S1009. If step S1005 is true (step S1005: True), the paragraph may end in the line element being processed, so the process proceeds to step S1006.

[0119] (Step S1006) The paragraph extraction module 127 determines whether the line element to be processed matches a regular expression for detecting an end-of-sentence expression. For example, the regular expression may be ".*?[.!?:;]"?\s*[0-9]*$".

[0120] The regular expression ".*?" is a part that indicates any character "." repeated 0 or more times, and "?" means a non-greedy match, and is used to match as few characters as possible. In other words, it matches characters up to the first occurrence of the following part (., !, ?, :, ;).

[0121] The regular expression "[.!?:;]" means to match any one of the characters included in the square brackets [ ]. Specifically, it matches any one of "., !, ?, :, ;".

[0122] The regular expression "?" indicates that the double quotation mark "" can occur 0 or 1 time. The "?" indicates that the previous element can occur 0 or 1 time.

[0123] The regular expression "\s*" means zero or more occurrences of a whitespace character (space, tab, newline, etc.). "\s" is an escape sequence that represents a whitespace character.

[0124] The regular expression "[0-9]*" means that it matches any number of digits, from 0 to 9, zero or more times.

[0125] The regular expression "$" indicates that it should match the end of the string, meaning that the regular expression will match anything following the end of the subject text string.

[0126] If step S1006 is false (step S1006: False), the paragraph is considered to end at the immediately preceding line element, and the process proceeds to step S1009. If step S1006 is true (step S1006: True), the paragraph ends at the current line element, and the process proceeds to step S1007.

[0127] (Step S1007) Since the line element registered in the paragraph information cache and the line element to be processed constitute the same paragraph, and the line element to be processed is considered to be the last line in the paragraph element, the paragraph extraction module 127 registers the line element registered in the paragraph information cache and the line element to be processed as an instance of a paragraph. Then, the process proceeds to step S1008.

[0128] (Step S1008) The paragraph extraction module 127 initializes a new paragraph information cache with an empty cache. When the paragraph extraction process by the paragraph extraction module 127 for the line is completed and there is a next line, the paragraph extraction module 127 executes the paragraph extraction process with the next line as the processing target line.

[0129] (Step S1009) The paragraph extraction module 127 registers the line elements in the paragraph information cache in the paragraph instance, and proceeds to step S1010.

[0130] (Step S1010) The paragraph extraction module 127 initializes the paragraph information cache with the line element to be processed. After that, the paragraph extraction process by the paragraph extraction module 127 for the line is completed, and if there is a next line, the paragraph extraction module 127 executes the paragraph extraction process with the next line as the line to be processed.

[0131] [Chapter Structure Detection Module 128] The chapter / section structure detection module 128 detects the chapter name for each paragraph. The chapter / section structure detection module 128 is executed for each paragraph according to the algorithm shown in the flowchart of FIG.

[0132] FIG. 11 is a flowchart showing an example of a chapter / section structure detection process performed by the chapter / section structure detection module 128.

[0133] (Step S1101) The chapter structure detection module 128 determines whether the paragraph element to be processed in the paragraph instance matches a regular expression for detecting a heading expression. For example, the regular expression can be "^([IVXLCDM\.]+|([A-Z0-9][0-9\.]*))\s([^\.]*)$".

[0134] The caret '^' in the regular expression marks the beginning of the string, meaning that the regular expression starts matching at the beginning of the string.

[0135] The regular expression "^([IVXLCDM\.]+|([A-Z0-9][0-9\.]*))" indicates that two regular expression patterns are separated by a "|" (pipe) and that either pattern will be matched. Specifically, it consists of the following two subpatterns:

[0136] The subpattern "([IVXLCDM\.]+" matches one or more repeats of the following characters: Roman numerals or a period ".".

[0137] The subpattern "([A-Z0-9][0-9\.]*))" matches a string that starts with an uppercase letter or digit and then matches zero or more repeated digits or periods. The subpattern "([A-Z0-9][0-9\.]*))" matches any sequence of letters or digits and periods.

[0138] The regular expression "\s" matches any whitespace character (space, tab, newline, etc.).

[0139] The regular expression "([^\.]*)" matches zero or more occurrences of any character other than a period ".".

[0140] The regular expression "$" matches the end of a text string.

[0141] If step S1101 is false (step S1102: False), the process proceeds to step S1104. If step S1101 is true (step S1101: True), the paragraph element to be processed is likely to be a heading, so the process proceeds to step S1102.

[0142] (Step S1102) The chapter / section structure detection module 128 determines whether the font size of the token in the paragraph element being processed is equal to or larger than the font size of the most frequent value of the token in the main text 205. For example, the chapter / section structure detection module 128 identifies the font size of each token in the main text 205 from the page instance, and calculates the most frequent value by assuming it to be the font size of the main text 205.

[0143] If step S1102 is true (step S1102: True), the process proceeds to step S1103. If step S1102 is false (step S1102: False), the paragraph element is not regarded as a heading, so that the chapter / section structure detection process by the chapter / section structure detection module 128 for the paragraph is terminated, and if there is a next paragraph, the chapter / section structure detection module 128 executes the chapter / section structure detection process with the next paragraph as the paragraph to be processed.

[0144] (Step S1103) The chapter and section structure detection module 128 determines whether the font type of the paragraph element to be processed is different from the font type of the most frequent value of the token in the main text 205. For example, the chapter and section structure detection module 128 identifies the font type, which is meta information of the token in the main text 205, from the page instance, and calculates the most frequent value. If step S1103 is true (step S1103: True), the process proceeds to step S1105. If step S1103 is false (step S1103: False), the paragraph element to be processed is not regarded as a heading, so that the chapter and section structure detection process by the chapter and section structure detection module 128 for the paragraph is terminated, and if there is a next paragraph, the chapter and section structure detection module 128 executes the chapter and section structure detection process with the next paragraph as the paragraph to be processed.

[0145] (Step S1104) The chapter and section structure detection module 128 determines whether the paragraph element to be processed matches the heading string specified by the user. Step S1104 is a process for dealing with heading expressions that cannot be covered by the regular expression in step S1101. If step S1104 is true (step S1104: True), the process proceeds to step S1105. If step S1104 is false (step S1103: False), the paragraph element to be processed is not considered to be a heading, so the chapter and section structure detection process by the chapter and section structure detection module 128 for the paragraph is terminated, and if there is a next paragraph, the chapter and section structure detection module 128 executes the chapter and section structure detection process with the next paragraph as the paragraph to be processed.

[0146] (Step S1105) The chapter structure detection module 128 records the paragraph element to be processed as a heading element together with the page number and the line number. After that, the chapter structure detection module 128 ends the chapter structure detection process for the paragraph, and if there is a next paragraph, the chapter structure detection module 128 executes the chapter structure detection process for the next paragraph as the paragraph to be processed.

[0147] [Column combination module 129] The column combining module 129 combines multiple columns when the paragraph elements across the multiple columns are the same paragraph element. The column combining module 129 is executed on a paragraph-by-paragraph basis, for example, according to the algorithm shown in the flowchart of FIG.

[0148] 12 is a flowchart showing an example of a column merging process procedure by the column merging module 129. The column merging module 129 uses a column merging cache. Initially, the column merging cache is empty, and tokens of the same paragraph are stored in the column merging cache by execution of the column merging module 129. Also, initially, the paragraph element at the top of the reading order in the paragraph instance is set as the paragraph element to be processed.

[0149] (Step S1201) The column binding module 129 determines whether a paragraph element exists in the column binding cache. Initially, the column binding cache is empty, so step S1201 is false (step S1201: False). If step S1201 is true (step S1201: True), the process proceeds to step S1203. If step S1201 is false (step S1201: False), the process proceeds to step S1202.

[0150] (Step S1202) The column combining module 129 initializes a column combining cache with the paragraph element to be processed. Then, when the column combining process by the column combining module 129 for the paragraph element to be processed is completed and there is a next paragraph, the column combining module 129 executes the column combining process with the paragraph element of the next paragraph as the paragraph element to be processed.

[0151] (Step S1203) The column merge module 129 judges whether the absolute value of the difference between the X coordinate value of the left end of the paragraph element stored in the column merge cache and the X coordinate value of the left end of the paragraph element to be processed (left end absolute value) is equal to or greater than a threshold. Note that the threshold can be set by the user, but since the coordinate system in the first embodiment is standardized, a setting value assumed on the structuring device 100 side may be used as the default value of the threshold. If step S1203 is true (step S1203: True), there is a possibility that one paragraph straddles two consecutive columns, so the process proceeds to step S1205.

[0152] The two consecutive columns are two consecutive columns on the same page, or the last column on the page and the first column on the next page. For example, if there are two columns (left column and right column) on a page, and the paragraph element to be processed is located at the end of the paragraph elements stored in the column merge cache and the paragraph element to be processed is located at the beginning of the right column, they will be merged into one paragraph element spanning the left and right columns depending on the determination results of steps S1205 to S1207. If step S1203 is false (step S1203: False), the process proceeds to step S1204.

[0153] (Step S1204) The column binding module 129 registers the paragraph element stored in the column binding cache as a body element in the page instance, and proceeds to step S1202.

[0154] (Step S1205) The column binding module 129 determines whether the paragraph elements stored in the column binding cache match a regular expression for detecting the end of a sentence. For example, the regular expression may be ".*?[。..!!??]"?[0-9]*$".

[0155] The regular expression "[。..!!??]" means to match any one of the characters included in the square brackets [ ]. Specifically, it matches any one of the Japanese punctuation marks "." and ",", the English punctuation marks ".", "!", and "?", and the corresponding full-width and half-width periods ".", "!", and "?".

[0156] The regular expression "[0-9]*" means that there can be zero or more occurrences of the digits 0 through 9. In other words, it matches any number of digits.

[0157] If step S1205 is true (step S1205: True), the process proceeds to step S1204, where the paragraph element stored in the column binding cache is registered as a body element in the page instance. If step S1205 is false (step S1205: False), the process proceeds to step S1206.

[0158] ".*?[。..、,,!!??::]$" (Step S1206) The column combining module 129 determines whether the paragraph element to be processed matches a regular expression for detecting the end of a sentence. For example, the regular expression may be ".*?[..,,,!!??::]$".

[0159] The regular expression "[。..、,,!!??::]" means to match any one of the characters included in the square brackets [ ]. Specifically, it matches any one of the Japanese punctuation marks "." and ",", the English punctuation marks ".", "!", "?", and ":", and their full-width and half-width variations ".", "!", "?", and ":".

[0160] If step S1206 is false (step S1206: False), the process proceeds to step S1204. If step S1206 is true (step S1205: True), the process proceeds to step S1207.

[0161] (Step S1207) The column binding module 129 judges whether the font size and font type of the paragraph element stored in the column binding cache match the font size and font type of the paragraph element to be processed. If step S1207 is false (step S1207: False), the process proceeds to step S1204. If step S1207 is true (step S1207: True), the process proceeds to step S1208.

[0162] (Step S1208) The column combining module 129 combines the paragraph element that does not match the regular expression stored in the column combining cache with the paragraph element to be processed that matches the regular expression, and proceeds to step S1209. As a result, the two consecutive columns are combined.

[0163] (Step S1209) The column combination module 129 registers the paragraph elements combined in step S1208 as body elements in the page instance, and proceeds to step S1210.

[0164] (Step S1210) The column combining module 129 initializes the column combining cache to be empty. After that, the column combining process by the column combining module 129 for the paragraph is completed, and if there is a next paragraph, the column combining module 129 executes the column combining process for the next paragraph as the paragraph to be processed.

[0165] [Page binding module 12A] The page merging module 12A merges multiple pages when paragraph elements across multiple pages are the same. The page merging module 12A is executed on a page-by-page basis, for example, according to the algorithm shown in the flowchart of FIG.

[0166] 13 is a flowchart showing an example of a page merging process procedure by the page merging module 12A. The page merging module 12A uses a page merging cache. Initially, the page merging cache is empty, and the page merging cache holds tokens of the same page by executing the page merging module 12A. Also, initially, the first page is the page to be processed.

[0167] (Step S1301) The page merging module 12A judges whether or not the paragraph element of the page to be processed exists in the paragraph instance. If step S1301 is false (step S1301: False), the process proceeds to step S1311. If step S1301 is true (step S1301: True), the process proceeds to step S1302.

[0168] (Step S1302) The page combination module 12A judges whether the page combination cache is empty. Initially, the page combination cache is empty, so step S1302 is true (step S1301: True). If step S1302 is true (step S1302: True), the process proceeds to step S1308. If step S1302 is false (step S1302: False), the paragraph element at the end of the page previous to the processing target page is stored, so the process proceeds to step S1303.

[0169] (Step S1303) The page binding module 12A determines whether the paragraph elements in the page binding cache match a regular expression that detects the end of a sentence. For example, the regular expression may be ".*?[。..!!??]"?[0-9]*$".

[0170] The regular expression "[。..!!??]" indicates that it matches any one of the characters contained within the square brackets [ ]. Specifically, it matches one of the Japanese punctuation marks "." and ",", the English punctuation marks ".", "!", and "?", and the corresponding full-width and half-width periods ".", "!", and "?". "[0-9]*" indicates that the digits 0 to 9 appear 0 or more times with "*". In other words, it matches any number of digits.

[0171] If step S1303 is true (step S1303: True), the process proceeds to step S1307. If step S1303 is false (step S1303: False), the process proceeds to step S1304.

[0172] (Step S1304) The page merging module 12A extracts the first paragraph element of the page to be processed, and the process proceeds to step S1305.

[0173] (Step S1305) The page merge module 12A judges whether the extracted first paragraph element matches the regular expression for detecting the end of a sentence. If step S1305 is false (step S1305: False), the process proceeds to step S1307. If step S1305 is true (step S1305: True), the process proceeds to step S1306.

[0174] (Step S1306) The page merging module 12A merges the paragraph element not matching the regular expression stored in the page merging cache with the first paragraph element matching the regular expression extracted in step S1304, and proceeds to step S1308. As a result, the previous page of the processing target page and the processing target page are merged by combining the last paragraph element of the page previous to the processing target page with the first paragraph element of the processing target page.

[0175] (Step S1307) The page combination module 12A registers the paragraph element stored in the page combination cache as a body element in the page instance, and proceeds to step S1308.

[0176] (Step S1308) The page merging module 12A registers the paragraph elements other than the last one in the processing target page as body elements in the page instance, and proceeds to step S1309.

[0177] (Step S1309) The page merging module 12A initializes the page merging cache with the last paragraph element of the processing target page. After this, the page merging process by the page merging module 12A for the processing target page is completed, and if there is a next page, the page merging module 12A executes the page merging process with the next page as the processing target page.

[0178] (Step S1311) The page merging module 12A judges whether the page merging cache is empty. If step S1311 is true (step S1311: True), the page merging process by the page merging module 12A for the page merging page is completed, and if there is a next page, the page merging module 12A executes the page merging process with the next page as the page to be processed. If step S1311 is false (step S1311: False), the process proceeds to step S1312.

[0179] (Step S1312) The page combination module 12A registers the paragraph element stored in the page combination cache as a body element in the page instance, and proceeds to step S1313.

[0180] (Step S1313) The page merging module 12A initializes the page merging cache to be empty. After that, the page merging process by the page merging module 12A for the processing target page is completed, and if there is a next page, the page merging module 12A executes the page merging process with the next page as the processing target page.

[0181] [Output module 12B] The output module 12B outputs the structured data 131. The output module 12B is executed for each paragraph element, for example, according to the algorithm shown in the flowchart of FIG.

[0182] FIG. 14 is a flowchart showing an example of an output process procedure by the output module 12B.

[0183] (Step S1401) The output module 12B stores, as output data, objects such as figures, footnotes, and metadata extracted by the data load module 120 through object extraction (step S305), and proceeds to step S1402.

[0184] (Step S1402) Output module 12B creates a chapter / section structure cache and proceeds to step S1403. The chapter / section structure cache is a dictionary-type cache and has a heading element area and a content element area inside. Output module 12B stores a heading element of a certain chapter / section structure in the heading element area, and stores a paragraph element belonging to that chapter / section structure in the content element area.

[0185] (Step S1403) The output module 12B executes a chapter / section structure creation process, and proceeds to step S1404. The chapter / section structure creation process will be described later with reference to FIG.

[0186] (Step S1404) The output module 12B associates the output data stored in step S1401 with the paragraph element in the content element region, and outputs it as structured data 131 together with the heading element in the heading element region.

[0187] [Chapter structure creation process (step S1403)] 15 is a flowchart showing a detailed example of a processing procedure of the chapter / section structure creation process (step S1403) by the output module 12B. Note that paragraph elements that have not been extracted in the reading order of the paragraph instance are treated as processing target paragraph elements.

[0188] (Step S1501) Output module 12B determines whether the paragraph element to be processed is a heading element based on the chapter / section structure detection result (step S1105) by chapter / section structure detection module 128. If step S1501 is true (step S1501: True), the process proceeds to step S1503. If step S1501 is false (step S1501: False), the process proceeds to step S1502.

[0189] (Step S1502) The output module 12B stores the paragraph element to be processed in the content element area of ​​the chapter / section structure cache, and proceeds to step S1404.

[0190] (Step S1503) The output module 12B stores the chapter / section structure cache as output data (heading elements in the heading element area and content elements in the content element area), and proceeds to step S1504.

[0191] (Step S1504) The output module 12B initializes the chapter / section structure cache to be empty, and proceeds to step S1505.

[0192] (Step S1505) The output module 12B stores the paragraph element to be processed in the heading element area of ​​the chapter / section structure cache that has been initialized to be empty, and proceeds to step S1404.

[0193] [Structured Data 131] 16 is an explanatory diagram showing an example of structured data 131. Structured data 131 is data obtained by structuring document data 101 to be structured. Structured data 131 is data obtained by structuring document data 101 to be structured for each section. A section is made up of a section name written in "xxx" and section information written in [yyy]. For example, "Test Paper", which is a heading in document data 101 to be structured, is stored in structured data 131 as section information with section name "doc_name".

[0194] Further, within the section information, more detailed section names and section information are stored. For example, the output data from the chapter structure cache (heading elements within the heading element region and content elements within the content element region) are stored as section information with the section name "content", and further, the heading element "Abstract" within the heading element region is stored as section information with the section name "title", and the content element "Thanks to the success of goal-oriented negotiation dialogue systems, studies of Negotiation … in the proposed data set." within the content element region is stored as section information with the section name "content".

[0195] The author's name "Test Author" in the document data to be structurized 101 is stored in "author" in the structured data 131. The footnote "* This work was conducted when the author was a master's student at the University." in the document data to be structurized 101 is stored in "foot notes" in the structured data 131 as one element of a character string with a list structure.

[0196] The character string 220 in the document data to be structurized 101, namely, "Annual Meeting 2023, pages 1234-1244" and "July 9-14, 2023.", is stored in "footers" as footer information in the structured data 131.

[0197] The element “Thanks to the success of goal-oriented Negotiation dialogue systems, . . . ” associated with “Abstract” in the document data to be structurized 101 is stored as one element of the list structure of “content” in the structured data 131 using dictionary-type data.

[0198] Similarly, the element “Negotiation is an essential task involved in our daily life...” associated with “1 Introduction” in the document data to be structurized 101 is stored as dictionary-type data in the structured data 131 as one element of the list structure of “content.”

[0199] The dictionary type data has title and content as keys, and stores section headings and associated text, respectively. The data format of the structured data 131 can be any data format as long as it can realize the above content. In this embodiment, it is assumed for convenience that the data format is JavaScript Object Notation (JSON).

[0200] Moreover, bar graph 253, which is a chart in document data to be structurized 101, is stored as a file path to bar graph 253 stored in an instance of an object in "figure" in structured data 131. Structuring device 100 can access bar graph 253 by the file path.

[0201] The character string 254 indicating a caption in the document data to be structurized 101, "Fig. 1 is a bar graph," is stored in "caption" in the structured data 131. Although not shown, the same applies to mathematical expressions and auxiliary information.

[0202] Furthermore, there are multiple granularities of structuring in structuring device 100, such as token (word) units, line units, paragraph units, and section units. These can be extracted by applying data load module 120, line extraction module 121, paragraph extraction module 127, and chapter structure detection module 128, respectively. The granularity of structuring can be adjusted by providing different types of output module 12B for each granularity.

[0203] [Dependencies between processing modules 12#] Dependencies are set for each processing module 12#. In other words, processing modules 12# cannot be applied in an order that does not satisfy the dependencies. The dependencies are defined as dependency data.

[0204] 17 is an explanatory diagram showing an example of dependency data in which dependencies are defined. Dependency data 1700 has a key 1701 and a value 1702. The key 1701 specifies the processing module 12# on which the dependency is to be performed. The value 1702 stores, in list form, the name of the dependency source processing module 12# that should be executed before the processing module 12# defined by the key 1701.

[0205] The dependency relationships between the processing modules 12# are defined by the template data 111 with reference to the dependency data 1700. The template data 111 is a module sequence that includes one or more arbitrary processing modules 12# included in the processing module pool 102 and indicates the application order of the processing modules 12#.

[0206] [Template data 111] 18 is an explanatory diagram showing an example of template data 111. Template data 1800 is an example of template data 111, and is defined in the following order: data load module 120, line extraction module 121, paragraph extraction module 127, chapter structure detection module 128, page combination module 12A, and output module 12B.

[0207] In the template data 1800, the data load module 120 is applied to the structuring target document data 101. Then, in order, the line extraction module 121, paragraph extraction module 127, chapter structure detection module 128, page combination module 12A, and output module 12B are applied.

[0208] Processing module 12# may be embedded in template data 1800, but any data format may be specified as long as it is capable of storing order data. For convenience, in the first embodiment, a list format is used. For example, a pointer to processing module 12# is embedded in template data 1800 in list format. In this case, structuring processing unit 130, which will be described later, acquires processing module 12# from processing module pool 102 by specifying the pointer.

[0209] [Settings data] 19 is an explanatory diagram showing an example of setting data. The setting data 1900 specifies the threshold, overlap rate, and character string by user operation. In the setting data 1900, the threshold for row extraction in step S404 is specified in "y_offset".

[0210] In "caption_overlap_threshold", a threshold value for the overlap rate regarding the footnote extraction in step S504 is specified. In "equation_overlap_threshold", a threshold value for the overlap rate regarding the formula extraction in step S804 is specified. In "header_offset", a threshold value for the header extraction in step S901 is specified. In "footer_offset", a threshold value for the footnote-corresponding line element extraction in step S903 is specified.

[0211] In "left_side_offset", the threshold value for the left edge of the page in step S905 is specified. In "right_side_offset", the threshold value for the right edge of the page in step S907 is specified. In "x_offset", the threshold value for paragraph extraction in step S1004 is specified. In "headline_names", the list of heading character strings specified by the user in step S1104 is specified. In "column_offset", the threshold value for column merging in step S1203 is specified.

[0212] [Classification Unit 110 and Template Classifier 104] Returning to FIG. 1, the classification unit 110 uses the template classifier 104 to execute an acquisition process for acquiring the document data to be structurized 101 as input data, and an extraction process for extracting template data 111 that is most appropriate for structuring the document data to be structurized 101 from the template data pool 103.

[0213] A rule-based method or a machine learning method can be used for the template classifier 104, but the rule-based method will be described in Example 1. An example in which the machine learning method is used for the template classifier 104 will be described in Example 2.

[0214] Rule-based template classifier 104 defines template data for each layout feature of document data to be structured 101. Rule-based template classifier 104 also defines processing module 12# for each layout feature of document data to be structured 101. Examples of layout features include the presence or absence of footnotes, headers, figures, tables, and formulas, and the number of columns in one page (e.g., double columns).

[0215] The layout features of the document data 101 to be structured described above are identified by the classification unit 110 in response to an input from the user.

[0216] Specifically, for example, when layout features of input document data to be structured 101 are input by a user operation, classification unit 110 refers to rule-based template classifier 104, identifies template data 111 from template classifier 104 that corresponds to the identified layout features input by the user operation, and extracts it from template data pool 103.

[0217] Fig. 20 is a flow chart showing an example of an extraction process procedure for template data 111. In Fig. 20, as an example, data load module 120, line extraction module 121, paragraph extraction module 127, chapter structure detection module 128, page merge module 12A and output module 12B are set as required processing modules 12# that are required for template data 111, and the other processing modules 12# are set as selectable processing modules 12# that can be appropriately selected by the user. Which processing modules 12# are set as required processing modules 12# and selectable processing modules 12# can be appropriately changed as long as the dependencies in dependency data 1700 are observed.

[0218] For example, data load module 120, line extraction module 121, paragraph extraction module 127, chapter structure detection module 128, and output module 12B may be designated as required processing modules 12#, and the other processing modules 12# may be designated as selectable processing modules 12#.

[0219] (Step S2001) The classification unit 110 adds a data load module 120 and a line extraction module 121 to the empty template data in accordance with the dependencies defined in the dependency data 1700 .

[0220] (Step S2002) The classification unit 110 judges whether or not the document data to be structurized 101 has a footnote as a layout feature. Specifically, for example, when a selection input of a footnote is accepted by a user operation, step S2002 becomes true (step S2002: True). On the other hand, when a selection input of a footnote is not accepted by a user operation, step S2002 becomes false (step S2002: False).

[0221] If step S2002 is false (step S2002: False), the process proceeds to step S2004. If step S2002 is true (step S2002: True), the process proceeds to step S2003.

[0222] (Step S2003) The classification unit 110 adds the footnote extraction module 122 to the template data to which the data load module 120 and the line extraction module 121 have been added, in accordance with the dependency relationship defined in the dependency relationship data 1700, and proceeds to step S2004.

[0223] (Step S2004) The classification unit 110 judges whether or not the document data to be structurized 101 has a header as a layout feature. Specifically, for example, if a header selection input is accepted by a user operation, step S2004 becomes true (step S2004: True). If a header selection input is not accepted by a user operation, step S2004 becomes false (step S2004: False).

[0224] If step S2004 is false (step S2004: False), the process proceeds to step S2006. If step S2004 is true (step S2004: True), the process proceeds to step S2005.

[0225] (Step S2005) Classification unit 110 adds auxiliary information extraction module 126 to the template data to which at least data load module 120 and line extraction module 121 have been added, in accordance with the dependency relationship defined in dependency relationship data 1700, and proceeds to step S2006.

[0226] (Step S2006) The classification unit 110 judges whether or not a diagram or a table is present as a layout feature in the document data to be structurized 101. Specifically, for example, if a selection input of a diagram or a table is accepted by a user operation, step S2006 becomes true (step S2006: True). If a selection input of a diagram or a table is not accepted by a user operation, step S2006 becomes false (step S2006: False).

[0227] If step S2006 is false (step S2006: False), the process proceeds to step S2008. If step S2006 is true (step S2006: True), the process proceeds to step S2007.

[0228] (Step S2007) The classification unit 110 adds a diagram extraction module 123 and a caption extraction module 124 to the template data to which at least the data load module 120 and the line extraction module 121 have been added, in accordance with the dependencies defined in the dependency data 1700, and proceeds to step S2008.

[0229] (Step S2008) The classification unit 110 judges whether or not there is a mathematical formula as a layout feature in the document data to be structurized 101. Specifically, for example, if a mathematical formula is selected and input by a user operation, step S2008 becomes true (step S2008: True). If a mathematical formula is not selected and input by a user operation, step S2008 becomes false (step S2008: False).

[0230] If step S2008 is false (step S2008: False), the process proceeds to step S2010. If step S2008 is true (step S2008: True), the process proceeds to step S2009.

[0231] (Step S2009) Classification unit 110 adds mathematical expression extraction module 125 to the template data to which at least data load module 120 and line extraction module 121 have been added, in accordance with the dependencies defined in dependency data 1700, and proceeds to step S2010.

[0232] (Step S2010) The classification unit 110 adds a paragraph extraction module 127 and a chapter structure detection module 128 to the template data to which at least the data load module 120 and the line extraction module 121 have been added, in accordance with the dependencies defined in the dependency data 1700, and proceeds to step S2011.

[0233] (Step S2011) The classification unit 110 judges whether the number of columns in the main text is double or more as a layout feature of the document data to be structurized 101. Specifically, for example, when a selection input indicating that the number of columns in the main text is double or more is accepted by a user operation, step S2011 becomes true (step S2008: True). When a selection input indicating that the number of columns in the main text is double or more is not accepted by a user operation, step S2011 becomes false (step S2011: False).

[0234] If step S2011 is false (step S2011: False), the process proceeds to step S2013. If step S2011 is true (step S2011: True), the process proceeds to step S2012.

[0235] (Step S2012) The classification unit 110 adds a column combination module 129 to the template data to which at least the data load module 120, the line extraction module 121, the paragraph extraction module 127, and the chapter structure detection module 128 have been added, in accordance with the dependencies defined in the dependency data 1700, and proceeds to step S2013.

[0236] (Step S2013) The classification unit 110 adds a page combination module 12A and an output module 12B to the template data to which at least a data load module 120, a line extraction module 121, a paragraph extraction module 127 and a chapter structure detection module 128 have been added, in accordance with the dependencies defined in the dependency data 1700, and proceeds to step S2014.

[0237] (Step S2014) Classification unit 110 generates and outputs template data 111 by connecting the template data to which at least data load module 120, line extraction module 121, paragraph extraction module 127, chapter structure detection module 128, page combination module 12A and output module 12B have been added in accordance with the dependency relationships of these processing modules 12#. This completes the extraction process of template data 111 by classification unit 110.

[0238] [Structuring Processor 130] Returning to Fig. 1, structuring processor 130 inputs document data to be structured 101, applies template data 111 extracted by classification unit 110 to the input document data to be structured 101, and executes structuring processing in accordance with the processing order in template data 111. Then, structuring processor 130 outputs structured data 131 as the execution result of the structuring processing. Furthermore, structuring processor 130 may store structured data 131 and document data to be structured 101 in association with each other.

[0239] [Settings execution screen] 21 is an explanatory diagram showing Example 1 of the setting execution screen of the structuring device 100 according to the embodiment 1. The setting execution screen 2100 is displayed on the structuring device 100 or another computer that can communicate with the structuring device 100 via a network.

[0240] The setting execution screen 2100 has a target file selection area 2101 , a selection button 2102 , a process button 2103 , a setting button 2104 , a chat field 2105 , a document file display area 2106 , a structured data display area 2107 , and a download button 2108 .

[0241] The target file selection area 2101 is a user interface that allows the user to select, for example, by a pull-down menu, the target file, which is the document data to be structured 101. The file name of the selected document data to be structured 101 ("XXX.pdf" is used as an example in FIG. 21) is displayed in the target file selection area 2101.

[0242] The selection button 2102 is a user interface for uploading the document data 101 to be structured selected in the target file selection area 2101, that is, for reading the document data 101 to be structured from the storage destination of the document data 101 to be structured.

[0243] The process button 2103 is a user interface for instructing the structuring device 100 to start executing processes (the extraction process by the classification unit 110 and the structuring process by the structuring processing unit 210).

[0244] 19, that is, is a user interface for reading from a storage destination of the setting data 1900. The read setting data 1900 is set in the corresponding processing module 12#.

[0245] Chat box 2105 is a display area in which a conversation between structuring device 100 and a user can be displayed in a chat format. When processing button 2103 is pressed to start the extraction process of template data 111 shown in Fig. 20, inquiries from structuring device 100 are displayed as speech bubbles 2151, 2153, and 2155 with their tips pointing left, and answers from the user are displayed as speech bubbles 2152 and 2154 with their tips pointing right.

[0246] The chat field 2105 also has an accept button 2156 and a reject button 2157. The accept button 2156 is a user interface for allowing the user to accept an inquiry from the structuring device 100 by a user operation and for displaying “Yes” in the chat field 2105.

[0247] The reject button 2157 is a user interface for allowing the user to reject an inquiry from the structuring device 100 by user operation and for displaying “No” in the chat box 2105 .

[0248] In the chat field 2105, speech bubble 2151 is displayed in the chat field 2105 as a result of execution of step S2002, speech bubble 2153 is displayed in the chat field 2105 as a result of execution of step S2004, and speech bubble 2155 is displayed in the chat field 2105 as a result of execution of step S2006.

[0249] Furthermore, when the user presses the accept button 2156 after the speech bubble 2151 is displayed, the speech bubble 2152 is displayed, and the structuring device 100 executes step S2003. When the user presses the reject button 2157 after the speech bubble 2153 is displayed, the speech bubble 2154 is displayed, and the structuring device 100 executes step S2005.

[0250] In the document file display area 2106, the document data 101 to be structurized that has been uploaded by pressing the select button 2102 is displayed.

[0251] In the structured data display area 2107, structured data 131 that is the execution result of the structuring processing unit 130 is displayed.

[0252] The download button 2108 is a user interface for downloading the structured data 131, that is, for acquiring the structured data 131 from the storage location of the structured data 131.

[0253] Fig. 22 is an explanatory diagram showing Example 2 of the setting execution screen of the structuring device 100 according to the first embodiment. In the setting execution screen 2200, the differences from the setting execution screen 2100 shown in Fig. 21 will be mainly described, so that the same components as those in the setting execution screen 2100 are denoted by the same reference numerals and the description thereof will be omitted.

[0254] The setting execution screen 2200 has an analysis button 2201 , a processing module pool display area 2202 , a template display area 2203 , and a processing button 2204 .

[0255] The analysis button 2201 is a user interface for starting template analysis in the chat field 2105 by a user operation.

[0256] The processing module pool display area 2202 is an area that displays each of the processing modules 12# in the processing module pool 102 as an icon.

[0257] Template display area 2203 is an area in which each of processing modules 12# constituting template data 111 is displayed as an icon.

[0258] When pressed, processing button 2204 serves as a user interface for instructing structuring device 100 to start structuring processing in structuring processing unit 130 in accordance with template data 111 displayed in template display area 2203 .

[0259] Furthermore, the user can move icons displayed in a shaded area between the processing module pool display area 2202 and the template display area 2203 by dragging and dropping.

[0260] <Checking for dependency relationships> Next, confirmation of the existence of dependency relationships between processing modules 12# will be described. It is considered that some users do not understand the nature of the dependency relationships between processing modules 12#. Therefore, structuring device 100 is configured to be able to execute a process of confirming the existence of dependency relationships between processing modules 12# after the above-mentioned classification unit 110 executes the process of extracting template data 111 and before structuring processing unit 130 executes the structuring process.

[0261] 23 is a flowchart illustrating an example of a procedure for checking the dependency relationships between the processing modules 12# by the structuring device 100 according to the first embodiment. Initially, an empty confirmed module list is prepared.

[0262] (Step S2301) Structuring device 100 determines whether or not there is an unselected processing module 12# in template data 111. If step S2301 is false (step S2301: False), the process proceeds to step S2308. If step S2301 is true (step S2301: True), the process proceeds to step S2302.

[0263] (Step S2302) Structuring device 100 selects an unselected processing module 12# from template data 111, and proceeds to step S2303. For example, if no processing module 12# is selected in template data 1800 shown in FIG. 18, structuring device 100 selects, for example, from the top data load module 120.

[0264] (Step S2303) The structuring device 100 judges whether the selection processing module 12# in step S2302 exists in the key 1701 of the dependency data 1700. If the selection processing module 12# exists in the key 1701, it determines that step S2301 is true (step S2303: True) and proceeds to step S2305. If the selection processing module 12# does not exist in the key 1701, it determines that step S2301 is false (step S2303: False) and proceeds to step S2304.

[0265] (Step S2304) If selected processing module 12# in template data 111 does not exist in key 1701, structuring device 100 cannot confirm that there is a dependency between processing modules 12#, and cannot execute structuring processing by structuring processing unit 130 based on template data 111. Therefore, structuring device 100 outputs error information indicating that to the user in a manner that is visible to the user. This ends the process of confirming the dependency between processing modules 12#.

[0266] (Step S2305) The structuring device 100 identifies the processing module 12# (hereinafter, dependency destination processing module 12#) of the value 1702 corresponding to the key 1701 in step S2303 from the dependency data 1700, and proceeds to step S2306.

[0267] (Step S2306) The structuring device 100 judges whether or not the dependency destination processing module 12# exists in the confirmed processing module list. If step S2306 is true (step S2306: True), the process proceeds to step S2301. If step S2306 is false (step S2306: False), the process proceeds to step S2307.

[0268] (Step S2307) The structuring device 100 registers the name of the dependency destination processing module 12# in the confirmed processing module list, and returns to step S2301.

[0269] (Step S2308) Structuring device 100 outputs the confirmed processing module list to the user in a visible manner, thereby completing the process of confirming the dependency relationships between processing modules 12#.

[0270] <Example of Hardware Configuration of Structuring Device 100> FIG. 24 is a block diagram showing an example of a hardware configuration of the structuring device 100 according to the first embodiment. The structuring device 100 includes a processor 2401, a storage device 2402, an input device 2403, an output device 2404, and a communication interface (communication IF) 2405. The processor 2401, the storage device 2402, the input device 2403, the output device 2404, and the communication IF 2405 are connected by a bus 2406. The processor 2401 controls the structuring device 100. The storage device 2402 is a working area for the processor 2401. The storage device 2402 is a non-transient or temporary recording medium that stores various programs and data. Examples of the storage device 2402 include a ROM (Read Only Memory), a RAM (Random Access Memory), a HDD (Hard Disk Drive), and a flash memory. The input device 2403 inputs data. The input device 2403 may be, for example, a keyboard, a mouse, a touch panel, a numeric keypad, a scanner, a microphone, or a sensor. The output device 2404 outputs data. The output device 2404 may be, for example, a display, a printer, or a speaker. The communication IF 2405 connects to a network and transmits and receives data.

[0271] The document data to be structurized 101, processing module pool 102, template data pool 103, template classifier 104, template data 111, and structured data 131 shown in FIG. 1 are stored, for example, in storage device 2402 of structuring device 100 shown in FIG. 24, or in storage device 2402 of another computer that can communicate with structuring device 100 via a network.

[0272] In addition, the processing module pool 102, classification unit 110, structuring processing unit 130 shown in FIG. 1, and the function of executing the confirmation process shown in FIG. 23 are specifically realized, for example, by having processor 2401 execute a program stored in storage device 2402 shown in FIG. 24.

[0273] In this way, according to the first embodiment, the document data to be structured 101 can be structured with high accuracy in accordance with the layout. EXAMPLES

[0274] Next, a description will be given of Example 2. In Example 1, the rule-based template classifier 104 is used, but in Example 2, a case where a machine learning template classifier 104 is used will be described. Note that in Example 2, differences from Example 1 will be mainly described, so that the same components as those in Example 1 are given the same reference numerals and descriptions thereof will be omitted.

[0275] <Example of functional configuration of structuring device> 25 is a block diagram showing a functional configuration example 1 of a structuring device according to Example 2. The structuring device 2500 has a template matching unit 2510 and a learning unit 2520. The template matching unit 2510 receives as input the document data to be structurized 101, the processing module pool 102, the template data pool 103, and supervised answer data 2501, and outputs label data 2502.

[0276] Learning unit 2520 receives template data 111 and template classifier 104 , and outputs trained template classifier 2504 .

[0277] Specifically, template matching unit 2510 and learning unit 2520 are realized by, for example, having processor 2401 execute a program stored in storage device 2402 shown in Fig. 24. Supervised answer data 2501 and trained template classifier 2504 are stored, for example, in storage device 2402 of structuring device 100 shown in Fig. 24, or in storage device 2402 of another computer capable of communicating with structuring device 100 via a network.

[0278] The supervised data 2501 input to the template matching unit 2510 is structured data. Specifically, for example, the supervised data 2501 is structured data that is output by structuring processing as a result of a user creating template data and inputting the document data 101 to be structured into the created template data.

[0279] [Template Matching Unit 2510] Specifically, for example, template matching unit 2510 executes a structuring process similar to that of structuring processing unit 130 shown in the first embodiment, for each piece of template data in template data pool 103 (hereinafter, referred to as comparison target template data), and generates structured data for each piece of comparison target template data.

[0280] The template matching unit 2510 executes matching between each of the structured data generated for each comparison target template data and the supervised answer data 2501, and determines the degree of match. Specifically, for example, the template matching unit 2510 calculates, as a score, an edit difference between the supervised answer data 2501 and the structured data obtained by structuring the structuring target document data 101 with each comparison target template data stored in the template data pool 103.

[0281] The score is calculated, for example, by an F value based on the degree of match on a character-by-character basis, the degree of match on a line-by-line basis, and the degree of match on a paragraph-by-paragraph basis. After calculating the scores, the template matching unit 2510 outputs the comparison template data with the best score as label data 2502.

[0282] [Learning Section 2520] The learning unit 2520 receives as input the label data 2502, the document data to be structurized 101, and the template classifier 104 based on machine learning, and outputs the trained template classifier 2504. Specifically, for example, the learning unit 2520 trains the template classifier 104 by machine learning based on the difference between the output result (template data) obtained as a result of inputting the document data to be structurized 101 into the template classifier 104, and the label data 2502, and outputs the trained template classifier 2504.

[0283] During learning, since there is no selection input of the processing module 12# by user operation, the learning unit 2520 selects the processing module 12# in a brute-force manner as long as the dependency is satisfied, thereby extracting the output result (template data) to be compared with the label data 2502.

[0284] Since the machine learning template classifier 104 is a machine learning model, any architecture can be used, for example, a LayoutLM model based on Transformer.

[0285] The template classifier 104 extracts, for example, layout-related features and character string-related features from the document data to be structurized 101, and performs optimization as a multi-class classification problem of selecting an optimal template from the template data stored in the template data pool 103. Therefore, a cross-entropy loss function is used for training the template classifier 104.

[0286] Fig. 26 is a block diagram showing a second example of a functional configuration of a structuring device 2500 according to the second embodiment. The structuring device 2500 includes a classification unit 2610 and a structuring processing unit 130 in addition to the configuration shown in Fig. 25. Specifically, the classification unit 2610 is realized by causing the processor 2401 to execute a program stored in the storage device 2402 shown in Fig. 24.

[0287] [Classification section 2610] The classification unit 2610 receives the document data to be structurized 101, the processing module pool 102, the template data pool 103, and the trained template classifier 2504, and outputs the template data 111.

[0288] Specifically, for example, the classification unit 2610 inputs the document data to be structurized 101 to the trained template classifier 2504 to select template data 111. That is, the trained template classifier 2504 selects optimal template data 111 from among template data that takes into account all possible combinations taking into account dependency relationships from the processing module pool 102, by solving a multi-class classification problem. The classification unit 2610 extracts the template data 111 selected by the trained template classifier 2504 from the template data pool 103.

[0289] [Settings execution screen] 27 is an explanatory diagram showing Example 1 of the setting execution screen according to Example 2. The setting execution screen 2700 has a target file selection area 2101, a selection button 2102, a setting button 2104, a chat field 2105, a document file display area 2106, a structured data display area 2107, a download button 2108, a used template display area 2701, and a process button 2702.

[0290] The process button 2702 is a user interface that can be operated by the user to start structuring processing of the structuring target document data 101. The used template display area 2701 is a user interface that displays the template data 111 that is output when the classification unit 2610 is completed after the process button 2702 is pressed by a user operation.

[0291] 28 is an explanatory diagram showing Example 2 of the setting execution screen according to Example 2. The setting execution screen 2800 has a target file selection area 2101, a selection button 2102, a setting button 2104, a document file display area 2106, a structured data display area 2107, a download button 2108, a process button 2204, an analysis button 2801, a processing module pool display area 2202, and a template display area 2203.

[0292] The analysis button 2801 is a user interface for instructing the structuring device 2500 to start the process of extracting the template data 111 in the classification unit 2610, by a user operation.

[0293] According to the second embodiment, it is possible to automatically extract the optimum template data 111 suited to the characteristics of the document data 101 to be structured, and to structure the document data 101 to be structured with high accuracy.

[0294] The present invention is not limited to the above-described embodiments, and includes various modified examples and equivalent configurations within the spirit of the appended claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those having all of the configurations described. Also, a part of the configuration of one embodiment may be replaced with a configuration of another embodiment. Also, a configuration of another embodiment may be added to a configuration of one embodiment. Also, a part of the configuration of each embodiment may be added, deleted, or replaced with another configuration.

[0295] Furthermore, each of the aforementioned configurations, functions, processing units, processing means, etc. may be realized in hardware, for example by designing some or all of them as an integrated circuit, or may be realized in software by a processor interpreting and executing a program that realizes each function.

[0296] Information such as programs, tables, files, etc. that realize each function can be stored in a storage device such as a memory, a hard disk, or an SSD (Solid State Drive), or in a recording medium such as an IC (Integrated Circuit) card, an SD card, or a DVD (Digital Versatile Disc).

[0297] In addition, the control lines and information lines shown are those considered necessary for the explanation, and do not necessarily show all the control lines and information lines necessary for implementation. In reality, it can be considered that almost all components are connected to each other. [Explanation of symbols]

[0298] 12# Processing module 12A Page Binding Module 12B Output Module 100 Structured Device 101 Document data to be structured 102 Processing Module Pool 103 Template Data Pool 104 Template Classifier 110 Classification Department 111 Template Data 120 Data Load Module 121 Line Extraction Module 122 Footnote Extraction Module 124 Caption Extraction Module 125 Formula Extraction Module 126 Auxiliary Information Extraction Module 127 Paragraph Extraction Module 128 Chapter Structure Detection Module 129 Column Combining Module 130 Structured Processing Unit 131 Structured Data 210 Structured Processing Unit 1700 Dependency Data 1800 template data 1900 Setting data 2100 Setting execution screen 2200 Setting execution screen 2401 Processor 2402 Storage Device 2500 Structured Device 2501 Correct Data 2502 Label Data 2504 Trained template classifier 2510 Template Matching Unit 2520 Learning Department 2610 Classification Department 2700 Setting execution screen 2800 Setting execution screen

Claims

1. A structuring device having a processor for executing a program and a storage device for storing the program, A processing module pool is accessible, the processing module pool storing a plurality of processing modules capable of executing processing based on layout-related features in document data, and a template data pool is accessible, the template data pool storing template data in which two or more processing modules combined in accordance with dependencies between the plurality of processing modules are defined, The processor, An acquisition process for acquiring document data to be structured; an extraction process for extracting specific template data from the template data pool based on the result of selection input of features related to the layout of the document data to be structurized acquired by the acquisition process; a structuring process for outputting first structured data by structuring the document data to be structured with the layout-related features by executing two or more specific processing modules constituting the specific template data extracted by the extraction process in accordance with a dependency relationship between the two or more specific processing modules; A structuring device comprising:

2. 2. A structuring device according to claim 1, the plurality of processing modules include a first processing module that is essential for all template data, and a second processing module that is selectable by the selection input; A structuring device comprising:

3. 3. A structuring device according to claim 2, the first processing module includes a line extraction module that extracts line elements that constitute lines from the document data, a paragraph extraction module that extracts paragraph elements that constitute paragraphs based on the line elements, and a detection module that detects whether the paragraph elements correspond to headings; In the structuring process, the processor generates the first structured data by distinguishing between paragraph elements that correspond to the heading and paragraph elements that do not correspond to the heading based on a detection result of the detection module. A structuring device comprising:

4. 4. A structuring device according to claim 3, the first processing module includes a page combining module that combines a first paragraph element of a first page in the document data with a last paragraph element of a second page immediately preceding the first page based on the paragraph element; In the structuring process, the processor generates the first structured data by distinguishing between paragraph elements corresponding to the heading and paragraph elements not corresponding to the heading based on the detection result of the detection module and the page combination result of the page combination module. A structuring device comprising:

5. 4. A structuring device according to claim 3, the first processing module includes a page merging module that, when two consecutive columns exist in the document data and the two consecutive paragraph elements satisfy a predetermined condition, merges the two consecutive paragraph elements; In the structuring process, the processor generates the first structured data by distinguishing between paragraph elements corresponding to the heading and paragraph elements not corresponding to the heading based on the detection result of the detection module and the page combination result of the page combination module. A structuring device comprising:

6. 2. A structuring device according to claim 1, a classifier that associates layout-related features in the document data with the processing modules; In the extraction process, the processor selects the specific processing module corresponding to the selection input result using the classifier, thereby extracting the specific template data. A structuring device comprising:

7. 2. A structuring device according to claim 1, a classifier is accessible that is trained to receive the document data and extract the template data from the template data pool based on layout-related features within the document data; In the extraction process, the processor extracts the specific template data by using the classifier to select the specific processing module corresponding to the result of the selection input. A structuring device comprising:

8. 8. A structuring device according to claim 7, The processor, a determination process for determining whether or not second structured data corresponding to the document data to be structured matches third structured data for each of the template data in the template data pool, the third structured data being obtained by executing the structuring process on the document data to be structured for each of the template data, and outputting the template data from which the second structured data is generated as label data based on a result of the determination; executing a learning process for learning the classifier based on template data obtained by inputting the document data to be structured into the classifier and label data output by the determination process; In the extraction process, the processor selects the specific processing module corresponding to the result of the selection input using a classifier trained by the learning process, thereby extracting the specific template data. A structuring device comprising:

9. A structuring method executed by a structuring device having a processor that executes a program and a storage device that stores the program, comprising: A processing module pool is accessible, the processing module pool storing a plurality of processing modules capable of executing processing based on layout-related features in document data, and a template data pool is accessible, the template data pool storing template data in which two or more processing modules combined in accordance with dependencies between the plurality of processing modules are defined, The processor, An acquisition process for acquiring document data to be structured; an extraction process for extracting specific template data from the template data pool based on the result of selection input of features related to the layout of the document data to be structurized acquired by the acquisition process; a structuring process for outputting first structured data by structuring the document data to be structured with the layout-related features by executing two or more specific processing modules constituting the specific template data extracted by the extraction process in accordance with a dependency relationship between the two or more specific processing modules; A structuring method comprising:

10. A processor having access to a processing module pool storing a plurality of processing modules capable of executing processing based on features related to layout in document data, and a template data pool storing template data in which two or more processing modules combined according to dependencies between the plurality of processing modules are defined, An acquisition process for acquiring document data to be structured; an extraction process for extracting specific template data from the template data pool based on the result of selection input of features related to the layout of the document data to be structurized acquired by the acquisition process; a structuring process for outputting first structured data by structuring the document data to be structured with the layout-related features by executing two or more specific processing modules constituting the specific template data extracted by the extraction process in accordance with a dependency relationship between the two or more specific processing modules; A structured program characterized by executing the above.

Images