Document processing device, document processing method, and recording medium
The document processing device effectively addresses the challenge of processing non-standard documents by dividing images into structural elements and extracting relevant data, enabling efficient generation of structured data from diverse document types.
Patent Information
- Application Number
- PCT/JP2023/017838
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-12
- Publication Date
- 2024-11-21
AI Technical Summary
Existing document processing techniques struggle to generate structured data from non-standard documents with variable formats and layouts, such as emails and social media posts, as they lack methods to effectively divide and extract elements from these types of documents.
A document processing device and method that divides document images into regions based on structural elements, using techniques like image conversion and optical character recognition to extract character and non-character elements, and generates structured data by identifying and processing sentence, list, signature, and image elements.
Enables the generation of structured data from various documents, including non-standard ones, by accurately identifying and extracting elements, thereby improving data processing efficiency and usability.
Smart Images

Figure JP2023017838_21112024_PF_FP_ABST
Abstract
Description
Document processing device, document processing method, and recording medium
[0001] The present disclosure relates to techniques available for processing documents.
[0002] 2. Description of the Related Art A technique has been proposed for acquiring structured data of elements such as sentences and figures contained in a document based on a document image corresponding to the imaged document.
[0003] Specifically, for example, Patent Document 1 discloses a technology for generating a structured document based on a document image corresponding to an article in a newspaper, magazine, etc., by assigning a reading order to multiple regions within the article.
[0004] International Publication No. WO2021 / 084702
[0005] However, Patent Document 1 does not specifically mention a method for generating structured data corresponding to elements contained in unstructured documents that do not have a fixed format or layout, such as emails and posts on SNS (Social Networking Service). Therefore, the technology disclosed in Patent Document 1 has a problem in that it may not be possible to generate structured data corresponding to elements contained in unstructured documents.
[0006] An object of the present disclosure is to provide a document processing device capable of generating structured data according to elements contained in various documents such as unstructured documents.
[0007] In one aspect of the present disclosure, a document processing device includes a division means for dividing an image of a document into areas for each element according to a criterion for dividing the image of the document into areas for each element that represent the structure of the document, and an acquisition means for acquiring character strings contained in the areas for each element.
[0008] In another aspect of the present disclosure, a document processing method divides an image of a document into areas for each element representing the structure of the document according to a criterion for dividing the image of the document into areas for each element, and obtains character strings contained in the areas for each element.
[0009] In yet another aspect of the present disclosure, a recording medium records a program that causes a computer to execute a process of dividing an image of a document into areas for each element according to criteria for dividing the image of the document into elements that represent the structure of the document, and obtaining character strings contained in the areas for each element.
[0010] According to the present disclosure, it is possible to generate structured data according to elements contained in various documents such as unstructured documents.
[0011] 1. A block diagram showing an example of the hardware configuration of an information processing apparatus according to the first embodiment. 2. A block diagram showing an example of the functional configuration of the information processing apparatus according to the first embodiment. 3. A diagram showing an example of a document file processed by the information processing apparatus according to the first embodiment. 4. A diagram showing an example of a case where line numbers are assigned to the document file of FIG. 3. 5. A diagram for explaining an example of a document area detected by the information processing apparatus according to the first embodiment. 6. A diagram showing an example of structured data based on sentence information extracted by the information processing apparatus according to the first embodiment. 7. A diagram showing an example of structured data based on list information extracted by the information processing apparatus according to the first embodiment. 8. A diagram showing an example of structured data based on signature information extracted by the information processing apparatus according to the first embodiment. 9. A diagram showing an example of structured data based on image information extracted by the information processing apparatus according to the first embodiment. 10. A flowchart showing an example of processing performed in the information processing apparatus according to the first embodiment. 11. A diagram showing an example of document data acquired by processing according to Modification 1 of the first embodiment. 12. A diagram showing an example of document data acquired by processing according to Modification 2 of the first embodiment. 13. A diagram showing an example of document image data that is the target of processing according to Modification 3 of the first embodiment. 14. A diagram showing an example of a processing result corresponding to the document image data of FIG. 13. 15. A diagram showing a schematic configuration of a scanner device according to Modification 4 of the first embodiment. 16. A block diagram showing an example of the functional configuration of a document processing apparatus according to the second embodiment. 10 is a flowchart illustrating an example of processing performed in a document processing apparatus according to a second embodiment.
[0012] Hereinafter, preferred embodiments of the present disclosure will be described with reference to the drawings.
[0013] <First embodiment> [Hardware configuration] Fig. 1 is a block diagram showing an example of the hardware configuration of an information processing device according to the first embodiment. As shown in Fig. 1, the information processing device 100 has an interface (IF) 111, a processor 112, a memory 113, a recording medium 114, and a database (DB) 115. Note that the information processing device 100 may be realized as a scanner device having a mounting unit and an imaging unit, for example, as in Variation 4 described below.
[0014] The IF 111 inputs and outputs data to and from an external device. For example, document data corresponding to an unimaged document file and document image data corresponding to an imaged document file are input to the information processing device 100 through the IF 111.
[0015] Document data may include, for example, emails, papers, and conversation logs from a messaging application. Document data may be data containing only text, or may be data containing both text and images. Document image data may include, for example, at least one of a Portable Document Format (PDF) file, a Joint Photographic Experts Group (JPEG) file, a Portable Network Graphics (PNG) file, a Tag Image File Format (TIFF) file, and a screenshot. Document image data may be data obtained by converting document data containing only text into an image, or may be data obtained by converting document data containing both text and images into an image. Document image data is not limited to the above examples.
[0016] The processor 112 is a computer such as a CPU (Central Processing Unit), and executes a program prepared in advance to control the entire information processing device 100. Specifically, the processor 112 performs, for example, processing related to extraction of elements included in document image data, and processing related to generation of structured data according to the type of element extracted from the document image data.
[0017] The memory 113 is configured by a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The memory 113 is also used as a working memory while the processor 112 is executing various processes.
[0018] The recording medium 114 is a non-volatile, non-transitory recording medium such as a disk-shaped recording medium or a semiconductor memory, and is configured to be detachable from the information processing device 100. The recording medium 114 records various programs to be executed by the processor 112. When the information processing device 100 executes various processes, the programs recorded on the recording medium 114 are loaded into the memory 113 and executed by the processor 112.
[0019] The DB 115 stores, for example, data input via the IF 111 and processing results obtained by processing by the processor 112 .
[0020] [Functional Configuration] Fig. 2 is a block diagram showing an example of the functional configuration of the information processing device according to the first embodiment. The information processing device 100 has a function as a document processing device. As shown in Fig. 2, the information processing device 100 also has a data acquisition unit 11, a document image generation unit 12, an element extraction unit 13, and an information extraction unit 14.
[0021] The data acquisition unit 11 acquires document data DCA and document image data DGA as input data. The data acquisition unit 11 outputs the document data DCA to the document image generation unit 12. The data acquisition unit 11 outputs the document image data DGA to the element extraction unit 13.
[0022] The document image generating unit 12 functions as an identifier generating unit. The document image generating unit 12 acquires document data DXA by adding information that can identify lines (for example, a line number, a numeric string, a code string, a barcode, a two-dimensional barcode, etc.) to the document data DCA output from the data acquiring unit 11.
[0023] The document image generating unit 12 may assign, as the identifiable information, a code representing each character of a string of characters included in a line. The target to which one piece of identifiable information is assigned may be one line, multiple lines, or a part of a line. The document image generating unit 12 may also generate information that can identify characters included in a line (or generate information that can identify the characters by converting the characters), and assign the created information to the line. These examples will be described in Modifications 1 to 3 below. Note that, for convenience of explanation, the following description will be given assuming that the identifiable information is a line number.
[0024] The document image generation unit 12 acquires the document data DXA by, for example, assigning line numbers to positions corresponding to line break positions in the document data DCA. The document image generation unit 12 also generates document image data DCB by performing image conversion processing on the document data DXA. The document image generation unit 12 also outputs the document image data DCB generated as described above and the document data DXA that is the source of the document image data DCB to the element extraction unit 13.
[0025] The element extraction unit 13 functions as a division unit. Based on the document data DXA and document image data DCB output from the document image generation unit 12, the element extraction unit 13 extracts multiple document elements DE, which are different types of elements, contained in the document image data DCB. The element extraction unit 13 outputs, to the information extraction unit 14, character element data DCD, which indicates the extraction results of multiple character elements from the multiple document elements DE, and non-character element data DCE, which indicates the extraction results of non-character elements from the multiple document elements DE. Based on the document image data DGA output from the data acquisition unit 11, the element extraction unit 13 extracts multiple document elements DE contained in the document image data DGA. The element extraction unit 13 outputs, to the information extraction unit 14, character element data DGD, which indicates the extraction results of multiple character elements from the multiple document elements DE, and non-character element data DGE, which indicates the extraction results of non-character elements from the multiple document elements DE. The element extraction unit 13 includes a detection unit 13A, a character element extraction unit 13B, and a non-character element extraction unit 13C.
[0026] The multiple document elements DE may include elements such as sentences, lists, signatures, document titles, chapter titles, section titles, list titles, headers, footers, tables, images, charts, and chart titles. Furthermore, list elements among the multiple document elements DE may include elements such as bulleted lists, ruled tables, indentations, and columns. Furthermore, chart elements among the multiple document elements DE may include elements such as graphs and diagrams. Furthermore, signature elements among the multiple document elements DE may include elements such as the name of the creator of the document. Furthermore, image elements among the multiple document elements DE may include elements such as images containing text. Furthermore, character elements may include elements expressed in text. Specifically, character elements may include elements such as sentences, lists, signatures, document titles, chapter titles, section titles, list titles, headers, footers, page numbers, dates, comments, and chart titles. Furthermore, non-character elements may include elements expressed in a format other than text. Specifically, non-character elements may include elements such as images and charts.
[0027] The detection unit 13A processes the document image data DCB using the detection model ODM to detect multiple document elements DE included in the document image data DCB and at least one document area DCR corresponding to an area in the document image data DCB where the multiple document elements DE exist. The detection unit 13A also assigns a label indicating one of the multiple document elements DE to each document area DCR detected from the document image data DCB.
[0028] The detection unit 13A processes the document image data DGA using the detection model ODM to detect multiple document elements DE included in the document image data DGA and at least one document area DGR corresponding to an area in the document image data DGA where the multiple document elements DE exist. The detection unit 13A also assigns a label indicating one of the multiple document elements DE to each of the document areas DGR detected from the document image data DGA.
[0029] According to this embodiment, for example, models such as Faster R-CNN (Convolutional Neural Network), YOLO (You Look Only Once), and SSD (Single Shot MultiBox Detector) can be used as the detection model ODM. Furthermore, according to this embodiment, data in which a label indicating one of a plurality of document elements DE is assigned to a document area included in document image data can be used as training data for the detection model ODM. Furthermore, according to this embodiment, the detection model ODM can be constructed by performing machine learning using the aforementioned training data. Furthermore, in this embodiment, it is desirable that the document area be set and detected as a rectangular area.
[0030] Based on the label assigned to the document area DCR, character element extraction unit 13B identifies a character area DCC, which is an area corresponding to a character element among the multiple document elements DE. Character element extraction unit 13B also extracts from document data DCA character elements corresponding to the line numbers of the character areas DCC in document image data DCB. Through this processing, character element extraction unit 13B can extract multiple character elements from document data DCA as document elements contained in each of the multiple document areas DCR corresponding to the multiple character areas DCC. Character element extraction unit 13B also outputs data including the character elements extracted from document data DCA to information extraction unit 14 as character element data DCD.
[0031] Note that character element extraction unit 13B may extract character elements by performing a process different from the process described above. Specifically, character element extraction unit 13B may extract character elements from document image data DCB by, for example, trimming a character area DCC from document image data DCB and performing OCR (Optical Character Recognition) processing on the character area DCC.
[0032] Based on the label assigned to the document area DGR, the character element extraction unit 13B identifies a character area DGC, which is an area corresponding to a character element among the multiple document elements DE. The character element extraction unit 13B also performs OCR processing on the character area DGC to extract character elements contained in the character area DGC from the document image data DGA. Through this processing, the character element extraction unit 13B can extract multiple character elements from the document image data DGA as document elements contained in each of the multiple document areas DGR corresponding to the multiple character areas DGC. Alternatively, the character element extraction unit 13B may extract characters contained in a character string represented by identifiable information from the identifiable information. The character element extraction unit 13B also outputs data containing the character elements extracted from the document image data DGA to the information extraction unit 14 as character element data DGD.
[0033] In this embodiment, character element extraction unit 13B may extract, as a plurality of character elements, elements represented as groups of characters arranged according to mutually different rules. Therefore, hereinafter, a case will be mainly described as an example in which data including sentence elements, list elements, and signature elements is extracted as character element data, and the character element data is output to information extraction unit 14.
[0034] The non-character element extraction unit 13C identifies non-character areas DCF, which are areas corresponding to non-character elements among the multiple document elements DE, based on the labels assigned to the document areas DCR. The non-character element extraction unit 13C also extracts non-character elements from the document image data DCB by trimming the non-character areas DCF. Through this processing, the non-character element extraction unit 13C can extract multiple non-character elements from the document image data DCB as document elements contained in each of the multiple document areas DCR corresponding to the multiple non-character areas DCF. The non-character element extraction unit 13C also outputs data including the non-character elements extracted from the document image data DCB to the information extraction unit 14 as non-character element data DCE.
[0035] The non-character element extraction unit 13C identifies non-character regions DGH, which are regions corresponding to non-character elements among the multiple document elements DE, based on the labels assigned to the document regions DGR. The non-character element extraction unit 13C also extracts non-character elements from the document image data DGA by trimming the non-character regions DGH. Through this processing, the non-character element extraction unit 13C can extract multiple non-character elements from the document image data DGA as document elements contained in each of the multiple document regions DGR corresponding to the multiple non-character regions DGH. The non-character element extraction unit 13C also outputs data including the non-character elements extracted from the document image data DGA to the information extraction unit 14 as non-character element data DGE.
[0036] In the following description, a case will be mainly explained as an example in which data including image elements is extracted as non-character element data and the non-character element data is output to the information extraction unit 14 .
[0037] The information extraction unit 14 functions as an acquisition unit. The information extraction unit 14 extracts information for each type of element included in the character element data DCD and non-character element data DCE output from the element extraction unit 13. The information extraction unit 14 generates structured data DCX based on the information extracted from the character element data DCD and non-character element data DCE, and outputs the generated structured data DCX to the outside. The information extraction unit 14 extracts information for each type of element included in the character element data DGD and non-character element data DGE output from the element extraction unit 13. The information extraction unit 14 generates structured data DGX based on the information extracted from the character element data DGD and non-character element data DGE, and outputs the generated structured data DGX to the outside. The information extraction unit 14 includes a sentence information extraction unit 14A, a list information extraction unit 14B, a signature information extraction unit 14C, and an image information extraction unit 14D.
[0038] The sentence information extraction unit 14A extracts sentence information from sentence elements included in the character element data DCD and generates structured data DCT based on the extracted sentence information. The sentence information extraction unit 14A also extracts sentence information from sentence elements included in the character element data DGD and generates structured data DGT based on the extracted sentence information.
[0039] The list information extraction unit 14B extracts list information from the list elements included in the character element data DCD, and generates structured data DCL based on the extracted list information. The list information extraction unit 14B also extracts list information from the list elements included in the character element data DGD, and generates structured data DGL based on the extracted list information.
[0040] The signature information extraction unit 14C extracts signature information from the signature elements included in the character element data DCD and generates structured data DCS based on the extracted signature information. The list information extraction unit 14B extracts signature information from the signature elements included in the character element data DGD and generates structured data DGS based on the extracted signature information.
[0041] The image information extraction unit 14D extracts image information from image elements included in the non-character element data DCE and generates structured data DCG based on the extracted image information. The image information extraction unit 14D also extracts image information from image elements included in the non-character element data DGE and generates structured data DGG based on the extracted image information.
[0042] According to the above-described processing, the information extraction unit 14 can generate, as the structured data DCX, data including at least one of the structured data DCT, DCL, DCS, and DCG. Furthermore, according to the above-described processing, the information extraction unit 14 can generate, as the structured data DGX, data including at least one of the structured data DGT, DGL, DGS, and DGG. Furthermore, the structured data DCX and DGX may be output to an output destination according to the data format. Specifically, the structured data DCX and DGX may be output to, for example, a relational database or a graph database.
[0043] [Specific Example] Next, a specific example according to the present embodiment will be described. Note that the following description will be given taking as an example a case where a document file DFA corresponding to unimaged document data, as shown in Fig. 3, is output from the data acquisition unit 11 to the document image generation unit 12. Fig. 3 is a diagram showing an example of a document file processed by the information processing apparatus according to the first embodiment.
[0044] The document image generation unit 12 generates, for example, a document file DYA as shown in FIG. 4 by assigning line numbers to positions corresponding to line break positions in the document file DFA. FIG. 4 is a diagram showing an example of the case where line numbers are assigned to the document file of FIG. 3. As shown in FIG. 4, line numbers "1" to "29" are assigned to the left edge of the document file DYA along the vertical direction of the document file DYA. The document image generation unit 12 also generates a document image file DFB by performing an image conversion process on the document file DYA. That is, the left edge of the document image file DFB is assigned the same line numbers as the document file DYA. The document image generation unit 12 also outputs the document image file DFB and the document file DYA that was the source of the document image file DFB to the element extraction unit 13.
[0045] According to the above-described process, the document image generation unit 12 can link an identifier that identifies a character string included in a document file DFA to the document file DFA. Also, according to the above-described process, the document image generation unit 12 can link a line number in the document file DFA as an identifier that identifies a character string included in the document file DFA.
[0046] The detection unit 13A processes the document image file DFB using the detection model ODM to detect multiple document elements DE included in the document image file DFB and document areas DFR corresponding to areas in the document image file DFB where the multiple document elements DE exist. Through this processing, the detection unit 13A can detect, for example, document areas DFR1, DFR2, DFR3, and DFR4 as shown in FIG. 5 . FIG. 5 is a diagram illustrating an example of a document area detected by the information processing apparatus according to the first embodiment. Furthermore, the detection unit 13A assigns a label indicating one of the multiple document elements DE to the document areas DFR1, DFR2, DFR3, and DFR4 detected from the document image file DFB.
[0047] The document region DFR1 is detected as a region corresponding to a sentence element among the plurality of document elements DE. The document region DFR1 is also given a label of "sentence" indicating that it corresponds to a sentence element among the plurality of document elements DE.
[0048] The document region DFR2 is detected as a region corresponding to a list element among the plurality of document elements DE. The document region DFR2 is also given a label "list" indicating that it corresponds to a list element among the plurality of document elements DE.
[0049] The document region DFR3 is detected as a region corresponding to the signature element among the plurality of document elements DE. The document region DFR3 is also given a label of "signature" indicating that it corresponds to the signature element among the plurality of document elements DE.
[0050] The document region DFR4 is detected as a region corresponding to an image element among the plurality of document elements DE. The document region DFR4 is also given the label "image" indicating that it corresponds to an image element among the plurality of document elements DE.
[0051] Based on the labels assigned to the document areas DFR1, DFR2, DFR3, and DFR4, the character element extraction unit 13B identifies character areas DFC, which are areas corresponding to character elements among the multiple document elements DE. Specifically, the character element extraction unit 13B identifies the document areas DFR1, DFR2, and DFR3 in the document image file DFB illustrated in FIG. 5 as character areas DFC. Furthermore, for each character area DFC identified from the multiple document areas DFR, the character element extraction unit 13B extracts from the document file DYA character elements corresponding to the line numbers of each character area DFC in the document image file DFB. Through this process, the character element extraction unit 13B can extract multiple sentences existing in the portion of the document area DFR1 from line numbers "1" to "13" (see FIG. 5) as sentence elements from the document file DYA. Furthermore, according to the above-described process, character element extraction unit 13B can extract, as list elements, from document file DYA, the bulleted items present in the portion from line numbers "15" to "22" (see FIG. 5) of document area DFR2, which has been detected as character area DFC. According to the above-described process, character element extraction unit 13B can extract, as signature elements, from document file DYA, the signatures present in the portion from line numbers "25" to "29" (see FIG. 5) of document area DFR3, which has been detected as character area DFC. According to this process, character element extraction unit 13B can extract character elements present in document areas DFR1, DFR2, and DFR3 with suitable accuracy. Furthermore, character element extraction unit 13B outputs data including sentence elements, list elements, and signature elements extracted from document file DYA to information extraction unit 14 as character element data DFD.
[0052] The non-character element extraction unit 13C identifies an image area DFF, which is an area corresponding to an image element among the multiple document elements DE, based on the labels assigned to the document areas DFR1, DFR2, DFR3, and DFR4. Specifically, the non-character element extraction unit 13C identifies the document area DFR4 in the document image file DFB illustrated in FIG. 5 as the image area DFF. The non-character element extraction unit 13C also extracts image elements from the document image file DFB by trimming the document area DFR4 identified as the image area DFF. The image elements extracted from the document image file DFB include, for example, an image simulating the character string "IT japan expo" displayed on a monitor screen, as shown in FIG. 5. In other words, the non-character element extraction unit 13C can extract, as image elements from the document image file DFB, an image resembling a monitor and the character string "IT japan expo" contained in the image. The non-character element extraction unit 13C also outputs data containing the image elements extracted from the document image file DFB to the information extraction unit 14 as non-character element data DFE.
[0053] According to the above-described process, the element extraction unit 13 can divide the document image file DFB into document areas DFR1, DFR2, DFR3, and DFR4 corresponding to the areas of each element, based on a plurality of document elements DE corresponding to criteria for dividing the document image file DFB into elements representing the document structure. According to the above-described process, the element extraction unit 13 can extract list elements corresponding to bulleted lists included in the document image file DFB as elements representing the document structure. According to the above-described process, the element extraction unit 13 can extract signature elements representing the creator of the document as elements representing the document structure. According to the above-described process, the element extraction unit 13 can extract image elements corresponding to the image included in document area DFR4 in the document image file DFB and the character string "IT japan expo" included in the image as elements representing the document structure.
[0054] The sentence information extraction unit 14A performs information extraction processing, such as open information extraction, on sentence elements included in the character element data DFD to extract, as sentence information, combinations of subject, predicate, and object in each of multiple sentences included in the sentence elements. Specifically, the sentence information extraction unit 14A performs information extraction processing, such as open information extraction, on the sentence, "This exhibition is an IT exhibition consisting of six specialized exhibitions." This allows the combination of "this exhibition," "is," and "IT exhibition consisting of six specialized exhibitions" to be extracted as sentence information. Furthermore, the sentence information extraction unit 14A generates, as structured data based on the extracted sentence information, structured data in a table format, such as that shown in FIG. 6 . FIG. 6 is a diagram illustrating an example of structured data based on sentence information extracted by the information processing device according to the first embodiment. For simplicity of illustration, the structured data DFT in FIG. 6 omits the descriptions of subject, predicate, and object combinations corresponding to some sentences of the sentence elements included in the character element data DFD.
[0055] The list information extraction unit 14B performs information extraction processing, such as text processing, on the list elements included in the character element data DFD to extract combinations of items and values included in the list elements as list information. Specifically, the list information extraction unit 14B performs text processing, such as normalization processing, which includes removing line breaks and line start characters included in the list elements. Through such processing, the list information extraction unit 14B can extract list information including, for example, a combination of "session date," which corresponds to the item, and "May 16 (Mon) - May 22 (Sun), 2022," which corresponds to the value. Furthermore, the list information extraction unit 14B generates structured data DFL in a tabular format, such as that shown in FIG. 7, as structured data based on the list information extracted as described above. FIG. 7 is a diagram illustrating an example of structured data based on the list information extracted by the information processing device according to the first embodiment.
[0056] The signature information extraction unit 14C extracts named entities included in the signature elements as signature information by performing information extraction processing, such as named entity extraction processing, on the signature elements included in the character element data DFD. Through such processing, the signature information extraction unit 14C can extract signature information including, for example, "IT Japan Expo Spring Management Office." Furthermore, the signature information extraction unit 14C generates structured data DFS in a table format, such as that shown in FIG. 8, as structured data based on the extracted signature information. FIG. 8 is a diagram showing an example of structured data based on signature information extracted by the information processing device according to the first embodiment. The structured data DFS includes "values" corresponding to each named entity extracted as signature information and "items" set according to the attributes of each named entity. The structured data DFS includes the following items: "organization," "telephone number," and "email address." The structured data DFS also includes "IT Japan Expo Spring Management Office," which is a value corresponding to "organization," "TEL: 045-XXXX-XXXX," which is a value corresponding to "telephone number," and "it_spring@it_expo2022spring.example.com," which is a value corresponding to "email." In other words, the combination of items and values included in the structured data DFS corresponds to identification information that identifies the creator of the document file DFA.
[0057] The image information extraction unit 14D, for example, performs OCR processing on image elements included in the non-character element data DFE to extract character strings included in the image elements as image information. Furthermore, the image information extraction unit 14D, for example, inputs the image elements included in the non-character element data DFE into a caption generation model to extract captions describing the image elements as image information. That is, the image information extraction unit 14D can extract character strings included in the image elements and captions describing the image elements by performing information extraction processing, such as OCR processing and caption generation processing, on the image elements included in the non-character element data DFE. According to this embodiment, for example, a Transformer model and a model including a convolutional neural network can be used as the caption generation model. Furthermore, according to this embodiment, for example, a captioned image dataset, such as Microsoft® Common Object in Context, can be used as training data for the caption generation model. According to the above-described processing, the image information extraction unit 14D can extract image information including, for example, "IT japan expo," which corresponds to a character string included in an image element, and "monitor," which corresponds to a caption explaining the image element. Furthermore, the image information extraction unit 14D generates, as structured data based on the image information extracted as described above, structured data DFG in a table format, for example, as shown in FIG. 9 . FIG. 9 is a diagram showing an example of structured data based on image information extracted by the information processing device according to the first embodiment.
[0058] According to the above-described process, the information extraction unit 14 can acquire items or values included in the itemized list elements of the character element data DFD for each line in the document area DFR2. According to the above-described process, the information extraction unit 14 can acquire identification information identifying the creator of the document file DFA from a character string included in the document area DFR3. According to the above-described process, the information extraction unit 14 can acquire a character string included in an image in the document file DFA and a caption describing the image from the document area DFR4. According to the above-described process, the information extraction unit 14 can acquire a character string associated with an identifier (line number) in the document file DFA as a character string included in the document areas DFR1, DFR2, and DFR3. According to the above-described process, the information extraction unit 14 can generate data including the structured data DFT, DFL, DFS, and DFG as the structured data DCX. Furthermore, the information extraction unit 14 outputs the structured data DCX including the structured data DFT, DFL, DFS, and DFG to the outside.
[0059] On the other hand, the user can check the contents of the structured data by accessing the database in which the structured data DCX is stored using, for example, a BI (Business Intelligence) tool or the like.
[0060] [Processing Flow] Next, a description will be given of the flow of processing performed in the information processing apparatus according to the first embodiment. Fig. 10 is a flowchart showing an example of processing performed in the information processing apparatus according to the first embodiment.
[0061] First, the information processing apparatus 100 acquires the document data DCA and the document image data DGA as input data (step S11).
[0062] Next, the information processing device 100 acquires document data DXA by assigning line numbers to the document data DCA acquired in step S11, and generates document image data DCB by performing image conversion processing on the acquired document data DXA (step S12).
[0063] Next, based on the document image data DCB acquired in step S12, the information processing device 100 detects a plurality of document elements DE contained in the document image data DCB and at least one document area DCR corresponding to the area in the document image data DCB where the plurality of document elements DE exist (step S13). Also, based on the document image data DGA acquired in step S11, the information processing device 100 detects a plurality of document elements DE contained in the document image data DGA and at least one document area DGR corresponding to the area in the document image data DGA where the plurality of document elements DE exist (step S13). Note that it is desirable that the information processing device 100 perform processing using the detection model ODM in step S13.
[0064] Next, the information processing device 100 extracts document elements included in the document area DCR detected in step S13 based on the document data DXA and document image data DCB acquired in step S12 (step S14). The information processing device 100 also extracts document elements included in the document area DGR detected in step S13 based on the document image data DGA acquired in step S11 (step S14). The process of step S14 allows the information processing device 100 to extract character elements and non-character elements as document elements. The process of step S14 also allows the information processing device 100 to extract, for example, sentence elements, list elements, and signature elements as character elements. The process of step S14 also allows the information processing device 100 to extract, for example, image elements as non-character elements.
[0065] Next, the information processing device 100 extracts information for each type of document element extracted in step S14 and generates structured data based on the extracted information (step S15). In step S15, the information processing device 100 performs, for example, an information extraction process corresponding to each of sentence elements, list elements, and signature elements to extract information from each element, and generates structured data based on the extracted information. In step S15, the information processing device 100 also performs, for example, an information extraction process corresponding to an image element to extract information from the image element, and generates structured data based on the extracted information. Furthermore, by performing the process in step S15, the information processing device 100 can generate structured data DFT, DFL, DFS, and DFG as exemplified in FIGS. 6 to 9.
[0066] As described above, according to this embodiment, structured data can be generated by extracting multiple character elements included in document image data and performing information extraction processing corresponding to each of the extracted character elements. Also, as described above, according to this embodiment, structured data can be generated by extracting non-character elements included in document image data and performing information extraction processing corresponding to the extracted non-character elements. Therefore, according to this embodiment, structured data can be generated according to elements included in various documents, such as unstructured documents.
[0067] [Modifications] Modifications of the present embodiment will be described below. For simplicity, specific descriptions of parts to which the above-described processes can be applied will be omitted as appropriate.
[0068] (Variation 1) According to this embodiment, the document image generation unit 12 may perform a process of adding coordinate information to the document data output from the data acquisition unit 11 instead of line numbers. By such a process, the document image generation unit 12 can acquire document data DPA, for example, as shown in FIG. 11 , with coordinate information added to the left edge. Furthermore, according to the above-described process, the document image generation unit 12 can add coordinate information corresponding to a line break position in the document data as an identifier for identifying a character string included in the document data. FIG. 11 is a diagram showing an example of document data acquired by a process according to Variation 1 of the first embodiment. The coordinate information, such as "0, 0.99," added to the left edge of the document data DPA may be added as a coordinate value that indicates, for example, a vertical position within the document data and changes depending on the font size, etc.
[0069] (Variation 2) According to this embodiment, the document image generation unit 12 may perform processing to add code information equivalent to a one-dimensional code or a two-dimensional code instead of line numbers to the document data output from the data acquisition unit 11. By performing such processing, the document image generation unit 12 can acquire document data DQA having a two-dimensional code added to the left edge, as shown in Fig. 12, for example. Fig. 12 is a diagram showing an example of document data acquired by processing according to Variation 2 of the first embodiment.
[0070] According to this modification, when the document image generating unit 12 assigns code information to the document data, the document image generating unit 12 may embed line numbers in the document data into the code information. By such processing, the document image generating unit 12 can embed line numbers such as "1" to "29" in the document file DYA in FIG. 4 into the code information.
[0071] According to this modification, when assigning a two-dimensional code to document data, the document image generation unit 12 may embed a character string for each line in the document data into the two-dimensional code. According to this process, the document image generation unit 12 can embed, for example, the character string "Universe Education Co., Ltd., Minami-sama" included in the document data DCA of FIG. 3 into the two-dimensional code and assign the two-dimensional code to a position corresponding to the first line of the document data DCA. Furthermore, according to the above process, the document image generation unit 12 can create a code representing the character string included in the document data as an identifier for identifying the character string included in the document data and associate the created code with the document data. Furthermore, according to the above process, the character element extraction unit 13B can acquire the character string restored by reading the two-dimensional code as a character string for each line in the document data. In other words, according to the above process, the character element extraction unit 13B can acquire the character string included in the document data without using the document data.
[0072] (Variation 3) According to this embodiment, for example, document image data DRB as shown in Fig. 13 may be input to the element extraction unit 13. Also, according to this embodiment, the detection unit 13A performs processing on the document image data DRB using the detection model ODM, thereby obtaining processing results such as those shown in Fig. 14. Fig. 13 is a diagram showing an example of document image data that is the target of processing according to Variation 3 of the first embodiment. Fig. 14 is a diagram showing an example of processing results corresponding to the document image data of Fig. 13.
[0073] According to the processing result of FIG. 14 , the document area in the document image data DRB is indicated by a dashed line. According to the processing result of FIG. 14 , the area of the multiple document elements DE where a header element exists is labeled "Header." According to the processing result of FIG. 14 , the area of the multiple document elements DE where a document title element exists is labeled "Document Title." According to the processing result of FIG. 14 , the area of the multiple document elements DE where a signature element exists is labeled "Signature." According to the processing result of FIG. 14 , the area of the multiple document elements DE where a chapter title element exists is labeled "Chapter Title." The chapter title element may include a character string indicating the chapter number. According to the processing result of FIG. 14 , the area of the multiple document elements DE where a section title element exists is labeled "Section Title." The section title element may include a character string indicating the section number. According to the processing result of FIG. 14 , the area of the multiple document elements DE where a sentence element exists is labeled "Sentence." 14, a label "list" is assigned to an area of the plurality of document elements DE where a list element (a table using ruled lines) exists. According to the processing result of FIG. 14, a label "list title" is assigned to an area of the plurality of document elements DE where a list title element exists. According to the processing result of FIG. 14, a label "chart" is assigned to an area of the plurality of document elements DE where a chart element (line graph) exists. According to the processing result of FIG. 14, a label "chart title" is assigned to an area of the plurality of document elements DE where a chart title element exists. According to the processing result of FIG. 14, a label "footer" is assigned to an area of the plurality of document elements DE where a footer element exists.
[0074] Furthermore, according to this embodiment, for example, a document element corresponding to a label assigned by the detection unit 13A is extracted by the element extraction unit 13, and information extraction processing corresponding to the document element is performed in the information extraction unit 14, thereby generating structured data based on the information extracted as a processing result of the information extraction processing.
[0075] (Modification 4) This embodiment can also be applied to a device including a control unit having the same functions as the information processing device 100. Specifically, this embodiment can also be applied to a scanner device 200 as shown in Fig. 15, for example. The scanner device 200 has a mounting unit 71, an imaging unit 72, and a control unit 73. Fig. 15 is a diagram showing the schematic configuration of a scanner device according to Modification 4 of the first embodiment.
[0076] The placement unit 71 has a configuration on which an object DX to be scanned, such as a paper document, can be placed.
[0077] The imaging unit 72 has, for example, a camera, etc. The imaging unit 72 is provided in a position facing the surface of the object DX placed on the placement unit 71. The imaging unit 72 has a configuration that allows it to scan the object DX placed on the placement unit 71.
[0078] The control unit 73 is configured as a control means having the same functions as the information processing device 100. The control unit 73 also controls the imaging unit 72 to acquire an image DXG of the surface of the object DX by scanning the surface of the object DX placed on the placement unit 71. The control unit 73 also performs processing to extract character strings included in the image DXG.
[0079] According to the configuration of this modified example as described above, it is possible to generate structured data according to elements contained in various documents such as unstructured documents.
[0080] Second Embodiment FIG. 16 is a block diagram showing an example of the functional configuration of a document processing apparatus according to a second embodiment.
[0081] The document processing device 500 according to this embodiment has the same hardware configuration as the information processing device 100. The document processing device 500 also has a dividing unit 511 and an acquiring unit 512.
[0082] The dividing unit 511 can be realized, for example, by using the function of the element extracting unit 13 of the first embodiment. The acquiring unit 512 can be realized, for example, by using the function of the information extracting unit 14 of the first embodiment.
[0083] FIG. 17 is a flowchart for explaining an example of processing performed in the document processing apparatus according to the second embodiment.
[0084] The dividing unit 511 divides the image of the document into regions for each element that represents the structure of the document, in accordance with a criterion for dividing the image of the document for each element (step S51).
[0085] The acquiring unit 512 acquires the character string included in the area for each element (step S52).
[0086] According to this embodiment, structured data can be generated according to elements contained in various documents such as unstructured documents.
[0087] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.
[0088] (Supplementary Note 1) A document processing device comprising: a division means for dividing an image of a document into areas for each element according to a criterion for dividing the image of the document into elements that represent the structure of the document; and an acquisition means for acquiring character strings included in the areas for each element.
[0089] (Supplementary Note 2) The document processing device of Supplementary Note 1, wherein the structure includes list elements corresponding to itemized lists, and the acquisition means acquires items or values included in the itemized lists for each line in an area corresponding to the list element among the areas for each element.
[0090] (Supplementary Note 3) The document processing device of Supplementary Note 1 or 2, wherein the structure includes a signature element representing a creator who created the document, and the acquisition means acquires identification information that identifies the creator from a character string included in an area corresponding to the signature element.
[0091] (Supplementary Note 4) The document processing device according to any one of Supplementary Notes 1 to 3, wherein the structure includes image elements corresponding to images in the document and character strings included in the images in the document, and the acquisition means acquires character strings included in the images in the document and captions that describe the images in the document.
[0092] (Supplementary Note 5) A document processing device according to any one of Supplementary Notes 1 to 4, further comprising an identifier creation means for linking information capable of identifying a character string contained in the document to the document, wherein the acquisition means uses the information to acquire a character string contained in an area for each element.
[0093] (Supplementary Note 6) The document processing device according to Supplementary Note 5, wherein the acquiring means acquires a character string linked to the information in the document as a character string included in an area for each element.
[0094] (Supplementary Note 7) The document processing device according to Supplementary Note 6, wherein the identifier creating means associates the information with a line number in the document.
[0095] (Supplementary Note 8) The document processing device according to Supplementary Note 6, wherein the identifier generating means assigns, as the information, a line number corresponding to a line break position in the document.
[0096] (Supplementary Note 9) The document processing device according to Supplementary Note 6, wherein the identifier generating means assigns coordinate information corresponding to a line break position in the document as the information.
[0097] (Supplementary Note 10) The document processing device according to Supplementary Note 5, wherein the identifier creation means creates a code representing a character string included in the document as the information, and links the created code to the document.
[0098] (Supplementary Note 11) A document processing method comprising: dividing an image of a document into regions for each element according to a criterion for dividing the image of the document into elements that represent the structure of the document; and acquiring character strings included in the regions for each element.
[0099] (Supplementary Note 12) A recording medium having recorded thereon a program that causes a computer to execute a process of dividing an image of a document into areas for each element according to a criterion for dividing the image of the document into elements that represent the structure of the document, and acquiring character strings included in the areas for each element.
[0100] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments and examples. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure.
[0101] 11 Data acquisition unit 12 Document image generation unit 13 Element extraction unit 14 Information extraction unit 100 Information processing device
Claims
1. A document processing device comprising: a division means for dividing an image of a document into areas for each element according to a criterion for dividing the image of the document into elements that represent the structure of the document; and an acquisition means for acquiring character strings contained in the areas for each element.
2. A document processing device as described in claim 1, wherein the structure includes list elements corresponding to bullet points, and the acquisition means acquires items or values included in the bullet points for each line in an area corresponding to the list element among the areas for each element.
3. A document processing device as described in claim 1 or 2, wherein the structure includes a signature element representing the creator who created the document, and the acquisition means acquires identification information identifying the creator from a character string included in an area corresponding to the signature element.
4. A document processing device as claimed in any one of claims 1 to 3, wherein the structure includes image elements corresponding to images in the document and character strings contained in the images in the document, and the acquisition means acquires character strings contained in the images in the document and captions describing the images in the document.
5. A document processing device as claimed in any one of claims 1 to 4, further comprising an identifier creation means for linking information capable of identifying a character string contained in the document to the document, wherein the acquisition means uses the information to acquire a character string contained in an area for each element.
6. The document processing device according to claim 5, wherein said acquiring means acquires a character string linked to said information in said document as a character string included in an area for each of said elements.
7. The document processing device according to claim 6, wherein the identifier generating means associates the information with a line number in the document.
8. The document processing device according to claim 6, wherein said identifier generating means assigns, as said information, a line number corresponding to a line break position in said document.
9. The document processing device according to claim 6, wherein said identifier generating means assigns, as said information, coordinate information corresponding to a line break position in said document.
10. The document processing device according to claim 5, wherein said identifier creation means creates, as said information, a code representing a character string contained in said document, and links said created code to said document.
11. A document processing method comprising: dividing an image of a document into areas for each element according to a criterion for dividing the image of the document into elements that represent the structure of the document; and acquiring character strings contained in the areas for each element.
12. A recording medium having recorded thereon a program that causes a computer to execute a process of dividing an image of a document into areas for each element according to criteria for dividing the image of the document into elements that represent the structure of the document, and acquiring character strings contained in the areas for each element.
Citation Information
Patent Citations
Document processor and document processing method
JP2003288334A
Document processor, document processing method, and document processing program
JP2008305088A
Image processing method, image processing apparatus, image forming apparatus, program and recording medium
JP2010288229A
File name creation apparatus and file name creation program
JP2013074609A
Document management system, image forming apparatus, document reproduction device, document management method, and computer program
JP2015226298A
Cited By
PDF text extraction method based on image segmentation and OCR
CN120496081A
