Image processing device, image processing system, output device, image processing method, and image processing program
The image processing system addresses inefficiencies in character recognition by generating training data with randomly arranged item information within constraints, enhancing accuracy and reducing labor through automated data generation and labeling.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- SHARP KK
- Filing Date
- 2024-11-27
- Publication Date
- 2026-06-08
Smart Images
Figure 2026092950000001_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to a technique for performing image processing (OCR) such as character recognition on images.
Background Art
[0002] Conventionally, techniques have been proposed for appropriately extracting character strings from images of documents (papers) such as forms by OCR processing. For example, a technique for generating data in which form images with various layouts are generated and data linking the result of OCR processing of the generated form images and the text data used when generating the form images is used as learning data is known (see, for example, Patent Document 1).
Prior Art Documents
Patent Documents
[0003]
Patent Document 1
Summary of the Invention
Problems to be Solved by the Invention
[0004] However, in the conventional technology, since information on a large number of arrangement patterns corresponding to various layouts is required, there is a problem that the efficiency of generating learning data is poor.
[0005] An object of the present disclosure is to efficiently generate learning data corresponding to character recognition processing for recognizing characters from document images, and to provide an image processing apparatus, an image processing system, an output apparatus, an image processing method, and an image processing program capable of improving character recognition accuracy.
Means for Solving the Problems
[0006] <00An image processing device according to one aspect of the present disclosure generates training data used for machine learning of character recognition processing that recognizes characters from a document image. The image processing device comprises an acquisition processing unit, a document image generation processing unit, and a training data generation processing unit. The acquisition processing unit acquires constraints for arranging item information of a document. The document image generation processing unit generates a document image in which the item information is randomly arranged based on the constraints. The training data generation processing unit generates data as training data in which the document image, labels representing items, text information used when generating the document image, and position information of the text information are associated with each other.
[0007] An image processing system according to one aspect of this disclosure comprises an image processing device and a learning device that generates a trained model by performing machine learning using the training data generated by the image processing device.
[0008] An output device according to another aspect of the present disclosure uses the trained model generated by the learning device to perform character recognition processing on an input image and outputs the character recognition result.
[0009] An image processing method relating to another aspect of the present disclosure generates training data used for machine learning of character recognition processing that recognizes characters from a document image. The image processing method is a method in which one or more processors perform the following steps: obtain constraints for arranging item information of a document; generate a document image in which the item information is randomly arranged based on the constraints; and generate data as training data in which the document image, labels representing items, text information used when generating the document image, and positional information of the text information are associated with each other.
[0010] An image processing program according to another aspect of this disclosure is a program that generates training data used for machine learning of character recognition processing that recognizes characters from document images. The image processing program obtains constraints for arranging item information in a document, Based on the aforementioned constraints, one or more processors are made to perform the following actions: generate a document image in which the item information is randomly arranged; and generate data as training data in which the document image, labels representing the items, text information used when generating the document image, and positional information of the text information are associated with each other. [Effects of the Invention]
[0011] According to this disclosure, it is possible to provide an image processing device, an image processing system, an output device, an image processing method, and an image processing program that can efficiently generate training data corresponding to character recognition processing that recognizes characters from document images, and improve the accuracy of character recognition. [Brief explanation of the drawing]
[0012] [Figure 1] Figure 1 is a functional block diagram showing the configuration of an image processing system according to an embodiment of this disclosure. [Figure 2] Figure 2 shows an example of a document (quotation) according to the embodiment of this disclosure. [Figure 3] Figure 3 shows an example of the format of a form according to the embodiment of this disclosure. [Figure 4] Figure 4 shows an example of the format of a form according to the embodiment of this disclosure. [Figure 5] Figure 5 is a flowchart showing an example of the procedure for generating training data performed in the image processing apparatus according to the embodiment of this disclosure. [Figure 6] Figure 6 is a flowchart showing an example of the procedure for setting item information performed in the image processing apparatus according to the embodiment of this disclosure. [Figure 7A] Figure 7A shows an example of a template corresponding to the specification table according to the embodiment of this disclosure. [Figure 7B] Figure 7B shows an example of a template corresponding to the specification table according to the embodiment of this disclosure. [Figure 7C]FIG. 7C is a diagram showing an example of a template corresponding to a specification according to an embodiment of the present disclosure. [Figure 8] FIG. 8 is a diagram showing an example of input data for items according to an embodiment of the present disclosure. [Figure 9] FIG. 9 is a diagram showing a specific example of a method for obtaining rectangular position information of text according to an embodiment of the present disclosure. [Figure 10] FIG. 10 is a diagram showing a specific example of data input to each item of a template of a specification according to an embodiment of the present disclosure. [Figure 11] FIG. 11 is a flowchart showing an example of a procedure for layout arrangement processing executed in an image processing apparatus according to an embodiment of the present disclosure. [Figure 12] FIG. 12 is a diagram showing an example of a form of a document according to an embodiment of the present disclosure. [Figure 13] FIG. 13 is a diagram showing an example of a form of a document according to an embodiment of the present disclosure. [Figure 14] FIG. 14 is a diagram showing a specific example of a method for calculating position information on a rectangular document according to an embodiment of the present disclosure. [Figure 15] FIG. 15 is a diagram showing a specific example of keys and values included in a rectangle according to an embodiment of the present disclosure.
Mode for Carrying Out the Invention
[0013] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. Note that the following embodiments are an example of embodying the present disclosure and do not have the character of limiting the technical scope of the present disclosure.
[0014] [First Embodiment] Figure 1 is a block diagram showing the configuration of an image processing system 10 according to the first embodiment of this disclosure. The image processing system 10 comprises an image processing device 1 and an output device 2. The image processing device 1 performs a process to generate training data (training data) used for machine learning of character recognition processing to recognize characters from a form image. The image processing device 1 also performs machine learning using the training data to generate a trained model for performing character recognition processing on an input image. The output device 2 uses the trained model to perform OCR processing (character recognition processing) on the input image to be recognized and outputs the OCR result (character recognition result). Note that the form is an example of a document in this disclosure. Furthermore, the document in this disclosure is not limited to a form; it may be other documents or papers (reports, notifications, etc.).
[0015] As shown in Figure 1, the image processing device 1 includes a control unit 11, a storage unit 12, an operation display unit 13, a communication unit 14, and the like. The image processing device 1 may consist of one or more cloud servers, or one or more physical servers.
[0016] The communication unit 14 is a communication interface for connecting the image processing device 1 to the network N1 by wire or wireless connection and for performing data communication with external devices (e.g., output device 2) via the network N1 in accordance with a predetermined communication protocol. The network N1 consists of, for example, the internet, a LAN, etc.
[0017] The operation display unit 13 is a user interface comprising a display unit such as a liquid crystal display or an organic EL display that displays various information, and an operation unit such as a mouse, keyboard, or touch panel that accepts input. For example, the operation display unit 13 accepts instructions for generating training data and displays training data.
[0018] The storage unit 12 is a non-volatile storage unit such as an HDD (Hard Disk Drive), SSD (Solid State Drive), or flash memory that stores various types of information. The storage unit 12 stores a control program, such as a learning data generation program (an example of an image processing program in this disclosure), which causes the control unit 11 to execute the learning data generation process (see Figure 5) described later. For example, the learning data generation program is non-temporarily recorded on a computer-readable recording medium such as a CD or DVD, read by a reader device (not shown) such as a CD drive or DVD drive provided by the image processing device 1, and stored in the storage unit 12. The learning data generation program may also be distributed from a cloud server and stored in the storage unit 12.
[0019] Furthermore, the storage unit 12 may store image data (such as scanned data) of documents acquired from external devices. The image data may be used to generate training data.
[0020] The control unit 11 of the image processing device 1 includes control devices such as a CPU, ROM, and RAM. The CPU is a processor that performs various arithmetic operations. The ROM stores control programs such as a BIOS and OS in advance to allow the CPU to perform various operations. The RAM stores various information and is used as a temporary storage memory (work area) for the various operations performed by the CPU. The control unit 11 controls the image processing device 1 by executing various control programs stored in advance in the ROM or storage unit 12 using the CPU.
[0021] Here, we will describe the types of documents that the image processing system 10 of this disclosure performs image processing (character recognition processing) on. Figure 2 shows an example of a document (quotation). Figure 3 shows a general layout of the document. As shown in Figures 2 and 3, the document contains multiple items, and each item is arranged according to a predetermined rule. The items included in the document vary depending on the type of document. For example, types of documents include "quotation," "purchase order," "invoice," and "receipt."
[0022] The "Quotation" section includes items such as quotation number, quotation issuance date, quoted amount (excluding / including tax), delivery date, quotation validity period, transaction method, registration number (number of qualified invoice issuer), issuing company information (company name, address, telephone number, contact person's name, etc.), receiving company information (company name, address, telephone number, contact person's name, etc.), and itemized list (product name / model number, quantity, unit price, individual amounts, etc.).
[0023] The "Order Form" section includes items such as the order form number, order form issuing department, order amount (excluding / including tax), transaction method, registration number (number of qualified invoice issuer), issuing company information (company name, address, telephone number, contact person's name, etc.), receiving company information (company name, address, telephone number, contact person's name, etc.), and itemized list (quotation number, item name / model number, quantity, unit price, individual amount, etc.).
[0024] The "Invoice" section includes items such as invoice number, invoice date, invoice amount (excluding / including tax), tax amount, payment due date, bank transfer information (bank name, branch name, account number, account holder name, etc.), registration number (number of qualified invoice issuers), issuing company information (company name, address, telephone number, contact person's name, etc.), receiving company information (company name, address, telephone number, contact person's name, etc.), and itemized statement (order number, transaction date, item name / model number, quantity, unit price, individual amounts, etc.).
[0025] The "receipt" section includes items such as receipt number, receipt issue date, receipt amount (excluding / including tax), tax amount, recipient's name, issuer information (company name, address, phone number, contact person's name, etc.), and item details.
[0026] Figure 4 shows the names of items commonly found in general forms. While the names may differ depending on the type of form or the related items, a group of similar related items is referred to as an "item (or group of items)." For example, "Issuer items (group of issuer items)" include the issuing company name, issuing address, registration number, issuing telephone number, issuing fax number, issuing email address, and issuing contact person. "Recipient items (group of recipient items)" include the recipient company name, recipient address, and recipient contact person. "Due date items (group of due date items)" include payment due date and delivery date. "Total amount items" include total amount including tax, total amount excluding tax, tax amount, and discounts. "Item items (group of item items)" include number, item name, quantity, unit, unit price, discount, tax amount, and individual amounts. "Bank transfer items (group of bank transfer items)" include bank name, branch name, and account number. Furthermore, in accordance with the Electronic Bookkeeping Law, in order to store and manage a large amount of data from various documents, it is necessary to extract various items for each type of document, and then aggregate, store, and manage them.
[0027] In order to sort through a large volume of such forms and extract and manage the information contained therein, a method (AI-OCR) has recently been adopted that uses AI to extract text regions from images of forms, processes these text regions using OCR to convert them into text, and then obtains the item information contained therein from the positional information and text of the text regions.
[0028] One method for recognizing items is to identify them based on the text content and rectangular position information of each item, determining that an item is specific if it meets certain conditions. For example, since the issue date of each form is often located in the upper right corner of the form, the system can identify and extract date-like formats ("year / month / day") within that area. However, the issue date is not always located in the upper right corner; it may also be in the upper left corner. Furthermore, there are many date formats different from the issue date (transaction date, deadline, etc.), making it difficult to determine the correct issue date. In addition, since forms contain multiple items, the specific conditions for each item can be complexly intertwined, conflicting conditions may arise, and recognition can become difficult.
[0029] Therefore, one possible approach is to use AI to learn and recognize multiple items listed on the forms (such as issue number, issue date, issuer information, and total amount). However, training an AI to correctly recognize multiple items requires a vast amount of data, which is time-consuming and laborious. For example, it would be necessary to collect a huge number of forms in various formats, and collecting forms becomes particularly difficult if there are restrictions on confidentiality, such as disclosing amounts or item names. Furthermore, even if many forms are collected, it is extremely laborious to assign labels to each individual item listed on the forms to indicate what each item represents.
[0030] Therefore, it is conceivable to prepare known items in advance. If each item can be arranged like an actual form, a vast amount of data can be easily created. Then, by training the AI with this vast amount of data, it becomes possible to improve recognition accuracy. However, it is necessary to create many known items, and since these items are not necessarily in the same area, each item must be placed in various positions.
[0031] The image processing system 10 according to this embodiment has a configuration that can solve each of the above-mentioned problems. Specifically, since multiple related items grouped into item groups are not placed separately from each other, the image processing system 10 inputs text information based on a template for each item group and places it for each item group. The image processing system 10 also prepares multiple templates for each item, randomly selects one from among them, and inputs the text information related to that item into the appropriate place in the selected template. Furthermore, the image processing system 10 randomly selects text formatting (font, size, italics, underline, bold, text color, background color, etc.) within the template, while imposing constraints that ensure consistency within the template so that the size does not differ drastically from character to character and does not look unnatural.
[0032] Furthermore, the image processing system 10 ensures consistency between different items by, for example, inputting the total value of items listed in the details sheet so that it matches the total amount on the document. Also, the image processing system 10 determines and inputs the expiration date so that it does not become a date that predates the issue date.
[0033] Furthermore, the image processing system 10 arranges randomly selected text information on a form based on layout constraints, while maintaining consistency between different items, using a randomly selected template as described above.
[0034] The constraints on the above layout include, for example, "the document type, recipient information, greeting, and subject line will never be at the bottom," "the bank transfer information, transaction method, expiration date, and delivery location will never be at the top," and "the greeting will not appear next to the itemized statement items."
[0035] The image processing system 10, while satisfying the constraints described above, can create a layout (form image) that closely resembles an existing form by randomly arranging each item. The specific configuration of the image processing device 1 for realizing the above configuration is described below.
[0036] Specifically, as shown in Figure 1, the control unit 11 includes various processing units such as an acquisition processing unit 111, a report image generation processing unit 112, a training data generation processing unit 113, and a training processing unit 114. The control unit 11 functions as these various processing units by executing various processes according to the training data generation program. Some or all of the processing units included in the control unit 11 may be composed of electronic circuits. The training data generation program may be a program that causes multiple processors to function as these various processing units.
[0037] The acquisition processing unit 111 acquires the constraints for arranging the item information of the form. The form image generation processing unit 112 generates a form image in which the item information is randomly arranged based on the constraints. The training data generation processing unit 113 generates training data by associating the form image, labels representing the items, the text information used when generating the form image, and the position information of the text information with each other.
[0038] For example, the form image generation processing unit 112 sets the information (item information) for each item included in the form. Specifically, the form image generation processing unit 112 randomly acquires (selects) items.
[0039] Furthermore, the form image generation processing unit 112 selects a template for each item. Specific examples of templates will be described later.
[0040] Furthermore, the document image generation processing unit 112 determines and inputs data for each acquired item based on the selected template. For example, the document image generation processing unit 112 determines (selects) a greeting message for the document type, determines (selects) a deadline for the issue date, and determines (selects) a total amount for the details sheet. For example, if the document image generation processing unit 112 selects an issue date, it will determine the deadline to a date after the issue date. Note that the aforementioned deadline may be something that does not affect the issue date (for example, "one month later"). The document image generation processing unit 112 also determines the total amount to be the sum of the individual amounts (subtotals) listed in the details sheet.
[0041] Furthermore, the form image generation processing unit 112 determines the format (font type, font size, font color, whether italics are used, whether bold text is used, whether underlined, etc.) for the input data.
[0042] Furthermore, the form image generation processing unit 112 converts the template into an image, performs character recognition, and obtains rectangular position information indicating the start and end points of each text. The form image generation processing unit 112 also obtains text information, rectangular position information of the text, and the label of the item for each item.
[0043] Furthermore, the form image generation processing unit 112 performs layout placement processing for the configured items. Specifically, the acquisition processing unit 111 acquires the layout constraints (the constraints mentioned above) for placing each item on the form. The form image generation processing unit 112 then determines the placement position of each item within the range of the aforementioned constraints.
[0044] Furthermore, the form image generation processing unit 112 calculates rectangular position information on the form based on the determined placement position and the rectangular position information of the text corresponding to each item. The specific calculation method will be described later. Once the form image generation processing unit 112 calculates the rectangular position information on the form for each item, it generates a form image based on said rectangular position information. In this way, the form image generation processing unit 112 generates a form image for each type of form. The form image generation processing unit 112 is an example of the document image generation processing unit of this disclosure.
[0045] The training data generation processing unit 113 generates training data by associating the report image, labels representing items, text information used when generating the report image, and positional information of the text information with each other.
[0046] In this way, the control unit 11 generates a predetermined number of forms. The control unit 11 uses the data from each generated form as training data for machine learning.
[0047] The learning processing unit 114 performs machine learning using the training data and generates a trained model. Specifically, the learning processing unit 114 uses machine learning on training data for each form, which includes text information for each item, text rectangle position information, and item labels, to generate the trained model.
[0048] Machine learning includes algorithms such as supervised learning (using supervised data), unsupervised learning (using unsupervised data), and reinforcement learning. Furthermore, a technique called "deep learning," which learns to extract features themselves, is used to implement these methods. In this embodiment, the learning processing unit 114 has a learning model based on the various algorithms described above. The learning processing unit 114 can perform machine learning using supervised and unsupervised data as input data to generate a trained model that performs character recognition processing. In other words, the image processing device 1 functions as a learning device that generates a trained model.
[0049] The trained model can be applied to various output devices 2 (such as character recognition devices). For example, as shown in Figure 1, when an input image to be recognized is input to the output device 2 (such as a user terminal), the output device 2 uses the trained model to perform OCR processing on the input image and outputs the OCR result.
[0050] Specifically, output device 2 extracts character rectangles from the input image, performs OCR processing on the extracted character rectangles using the trained model, and outputs the OCR result (character recognition result). Each process in output device 2 can be performed using well-known techniques.
[0051] The trained model may be downloaded to the output device 2 for use, or it may be stored on a server (cloud server) and accessed from the user terminal via the internet or the like for use. For example, when an arbitrary input image is input to the user terminal, the trained model outputs the optimal character recognition result.
[0052] [Training data generation process] Figure 5 is a flowchart showing an example of the procedure for generating training data performed in the image processing device 1.
[0053] This disclosure can be understood as a method for generating training data (image processing method of this disclosure) that performs one or more steps included in the training data generation process. Furthermore, one or more steps included in the training data generation process described herein may be omitted as appropriate. In addition, the execution order of each step in the training data generation process may differ to the extent that similar effects are produced. Furthermore, although this description uses the case in which the control unit 11 of the image processing device 1 executes each step in the training data generation process as an example, in other embodiments, one or more processors may distribute and execute each step in the training data generation process.
[0054] <Step S11> In step S11, the control unit 11 obtains the number of training data to be generated. That is, the control unit 11 obtains the required number of data for the form that is the target of the training data. For example, if the user specifies the number to be generated, the control unit 11 obtains that number.
[0055] <Step S12> In step S12, the control unit 11 sets the information for each item included in the form. A specific example of the item information setting process will be described later (see Figure 6).
[0056] <Step S13> In step S13, the control unit 11 arranges the layout for the items set in step S12. A specific example of the layout arrangement process will be described later (see Figure 7).
[0057] <Step S14> In step S14, the control unit 11 determines whether the number of data points in the generated forms has reached the specified number of generated data points. If the control unit 11 determines that the number of data points in the generated forms has reached the specified number of generated data points (S14: Yes), it terminates the learning data generation process. On the other hand, if the control unit 11 determines that the number of data points in the generated forms has not reached the specified number of generated data points (S14: No), it returns to step S12 and executes the item information setting process and the layout placement process. The control unit 11 repeatedly executes the item information setting process and the layout placement process until the number of data points in the generated forms reaches the specified number of generated data points.
[0058] Furthermore, when the number of data points in the generated forms reaches the specified number, the control unit 11 performs machine learning using the data from each generated form as training data to generate a trained model.
[0059] [Setting item information process] Figure 6 is a flowchart showing an example of the procedure for setting item information (step S12 in Figure 5) performed in the image processing device 1.
[0060] <Step S21> In step S21, the control unit 11 selects an item and determines the data for the selected item. For example, the control unit 11 selects the issue date as an item and determines the expiration date for the issue date. Alternatively, the control unit 11 selects the details sheet as an item and determines the total amount for the details sheet.
[0061] <Step S22> In step S22, the control unit 11 randomly selects a predetermined template from pre-prepared templates for an item. Figures 7A to 7C show specific examples of templates corresponding to detail tables. The control unit 11 selects one of the templates corresponding to the detail tables in Figures 7A to 7C.
[0062] <Step S23> In step S23, the control unit 11 inputs data for each item based on the selected template. Specifically, as shown in Figure 8, the control unit 11 generates input data for each item from a database or random data. In addition, for some items (for example, "unit"), the control unit 11 may randomly select whether the data is "present" or "absent" ("present / absent random").
[0063] Furthermore, there are items where random data determination would result in inconsistencies between items (for example, "Individual Amount"). In this case, the control unit 11 may determine the data through calculation. Also, for items where only one of them is needed (for example, "Total Amount Excluding Tax" and "Total Amount Including Tax"), the control unit 11 may randomly select one of them ("Random Selection"). The term "Random Characters" in Figure 8 means that one or more characters are randomly selected from numbers, English letters (lowercase / uppercase), kanji, katakana, symbols, etc., to form a string, but the premise is that the selected characters match the information of the item. For example, if the item is "Quantity," the control unit 11 will not select anything else, even if randomly, because it only contains "numbers" and "Chinese numerals."
[0064] In this way, the control unit 11 pre-registers multiple templates (see Figures 7A to 7C) for each item, and inputs random information (see Figure 8) corresponding to each item for a template randomly selected from among these multiple templates. The control unit 11 also randomly inputs the item value information for multiple related items in a way that ensures consistency between items.
[0065] <Step S24> In step S24, the control unit 11 determines the formatting (font type, font size, font color, whether italics are used, whether bold text is used, whether underlined, etc.) for the data entered into the template. In this case, the control unit 11 randomly selects and determines the formatting while setting constraints to ensure consistency within the template, so that the size does not differ drastically from character to character and does not look unnatural.
[0066] <Step S25> In step S25, the control unit 11 acquires rectangular position information of the text. Specifically, the control unit 11 converts the template into an image, performs string recognition, and acquires rectangular position information indicating the start and end points of each text. For example, in the item numbers of the detail table group shown in Figure 9, the control unit 11 acquires the horizontal (x) and vertical (y) coordinates [(50,60), (70,80)] representing the start and end points of the "1" recognized by string recognition.
[0067] <Step S26> In step S26, the control unit 11 determines whether or not there is another item that depends on the set item. If the control unit 11 determines that there is another item (S26: Yes), it moves the process to step S27. On the other hand, if the control unit 11 determines that there is no other item (S26: No), it moves the process to step S28.
[0068] <Step S27> In step S27, the control unit 11 selects a dependent item. Then, the control unit 11 moves the process to step S22, randomly selects a template that matches the selected item (S22), and inputs data into the template based on the data content of the previously entered item (S23).
[0069] For example, the control unit 11 randomly inputs data into each item (product name, quantity, unit, unit price, discount, amount) of the selected template (see Figure 7A) in the item group of details shown in Figure 10, and inputs the tax-exclusive amount, total discount amount, tax amount (10%), and tax-inclusive amount of the item group of details into the tax-exclusive total amount, discount amount, tax amount, and grand total of the total amount item group which depend on the tax-exclusive total amount, discount amount, tax amount, and grand total. In this example, the total amount item group corresponds to other items that depend on the item group of details.
[0070] Subsequently, the control unit 11 determines the format for the data entered into the template (S24) and obtains the rectangular position information of the text (S25).
[0071] <Step S28> In step S28, the control unit 11 determines whether or not there are items to be entered in the template. If the control unit 11 determines that there are items to be entered in the template (S28: Yes), it returns to step S21 and executes the process described above. On the other hand, if the control unit 11 determines that there are no items to be entered in the template (S28: No), it terminates the item information setting process and moves the process to step S13 (see Figure 5). The control unit 11 executes the above process until there are no more items to be entered in the template, and obtains text information, text rectangle position information, and item labels for each item.
[0072] [Layout placement process] Figure 11 is a flowchart showing an example of the layout placement process (step S13 in Figure 5) performed in the image processing device 1.
[0073] <Step S31> In step S31, the control unit 11 obtains layout constraints for arranging each item on the form. These constraints include, for example, "the document type, recipient item, greeting, and subject will never be at the bottom," "the bank transfer item, transaction method, expiration date, and delivery location will never be at the top," and "the greeting will not appear next to the item details."
[0074] If the aforementioned constraints are removed and each item is simply placed randomly, as shown in Figure 12, all items may end up clustered in the center of the form, resulting in an inappropriate overall layout. Alternatively, as shown in Figure 13, the overall layout may be appropriate, but the placement of each item may be impractical. When training data with layouts that deviate from actual forms is generated and used for training, the accuracy of character recognition for actual forms will decrease.
[0075] Therefore, the control unit 11 generates training data by setting constraints regarding the layout of the form.
[0076] <Step S32> In step S32, the control unit 11 randomly determines the placement position of each item within the constraints of the layout.
[0077] <Step S33> In step S33, the control unit 11 calculates and updates the rectangular position information on the form based on the determined placement position and the text rectangular position information for each item. For example, with respect to item number "1" of the detail table item group shown in Figure 9, if the top left corner, which is the starting point of the determined placement position of the detail table item group, is at (500, 1300) on the form (see Figure 14), the control unit 11 can calculate the coordinates of the start and end points of the text rectangular position information for item number "1" as (550, 1360) and (570, 1380), respectively, by adding the coordinates of the start and end points on the form. In this way, the control unit 11 updates the rectangular position of the text for each item to its rectangular position on the form.
[0078] <Step S34> In step S34, the control unit 11 determines whether or not there are items to be placed. If the control unit 11 determines that there are items to be placed (S34: Yes), it proceeds to step S32. On the other hand, if the control unit 11 determines that there are no items to be placed (S34: No), it terminates the layout placement process and proceeds to step S14 (see Figure 5). The control unit 11 continues to perform the above process until there are no more items to be placed. As a result, a report image is automatically generated, and the text information, text rectangle position information, and item labels for each item are acquired.
[0079] In this way, the control unit 11 generates a dataset in which data that should have the same item value information across multiple items is aligned, and generates a report image using this dataset. Specifically, the control unit 11 generates a dataset so that information that should have the same information across item information (such as the sum of amounts in the details report and the total amount) is aligned, and automatically lays out the report using the dataset.
[0080] As described above, the control unit 11 repeatedly performs the item information setting process (see Figure 6) and the layout placement process (see Figure 11) until the number of data points in the generated forms reaches the specified number of generated forms, thereby generating training data. The control unit 11 performs machine learning using the data from each generated form as training data and generates a trained model.
[0081] The generated trained model is fed into output device 2 (character recognition device). When output device 2 acquires an input image to be recognized, it performs a process to extract a character rectangle from the input image. Output device 2 then performs OCR processing on the extracted character rectangle using the trained model and outputs the OCR result (character recognition result).
[0082] [Second Embodiment] A second embodiment of the image processing system relating to this disclosure will now be described. Note that the same configuration as in the first embodiment will not be described.
[0083] Basically, each item has a key that represents the content of that item and a value that represents the value of that item.
[0084] For example, in the "Issuance Date: 2024 / 3 / 1" entry on the quotation, "Issuance Date:" is the key representing the content of the "Issuance Date" item, and "2024 / 3 / 1" is the value representing the value of that item.
[0085] A single rectangle may contain both the key and the value, or the key and value may be located far apart on the same row, or even vertically. Also, "2024 / 3 / 1" alone doesn't tell us what the date is, but the key reveals it's the "issue date."
[0086] Furthermore, in the case of a detailed table like the one shown in Figure 15, the "Item Name," "Quantity," "Unit Price," and "Subtotal" in the top row of the detailed table are keys for the values below them (XXX, 5, 100, 500), respectively, and each value is a value.
[0087] The image processing device 1 may also be trained to learn the key-value relationship of an item in order to accurately recognize what that item is by inputting text and its rectangular position information into the AI.
[0088] Image processing device 1 separates the content of an item from the value of that item and generates a rectangle, location information, and text. In the example above, image processing device 1 associates the label "Issue Date k", the text "Issue Date:", and its rectangular location information as the key for "Issue Date", and associates the label "Issue Date v", the text "2024 / 3 / 1", and its rectangular location information as the value for "Issue Date", thereby learning the relationship between the key and value of "Issue Date".
[0089] In this way, by separating the key and value of each item and associating them with labels, text, and rectangular position information, the relationships between the text content and position information are learned, thereby improving recognition accuracy.
[0090] Since some rectangles in the input for item recognition processing do not have separate keys and values, it is acceptable to randomly train the model without separating the keys and values during training.
[0091] [Third Embodiment] A third embodiment of the image processing system relating to this disclosure will now be described. Note that the same configuration as that of the first and second embodiments will not be described.
[0092] For training purposes, the text in the generated form image and its content must be identical. However, in actual inference, there may be discrepancies between the text in the form image and the recognized text. For example, if the issue date in the form image is "2024 / 3 / 1," but the recognized text is "2O24 / 3 / I" (where 0 is misrecognized as the letter O (uppercase O) and 1 as the letter I (uppercase I)), then if the model was trained using the correct character format ("number / number / number"), it may not be able to recognize characters other than numbers as the issue date.
[0093] Therefore, in order to improve the resistance of misrecognition of the input text during inference, the image processing device 1 may intentionally replace some characters with characters different from the correct characters during training.
[0094] [Features of this disclosure] As described above, the image processing device 1 according to this embodiment is an image processing system that generates training data used for machine learning of character recognition processing that recognizes characters from a form image (document image). Specifically, the image processing system 10 acquires constraints for arranging the item information of the form, and generates a form image in which the item information is randomly arranged based on the constraints. The image processing system 10 then generates training data by associating the form image, labels representing the items, the text information used when generating the form image, and the positional information of the text information with each other.
[0095] For example, the image processing system 10 automatically lays out multiple fields (details, issuer information, recipient company information, expiration date, etc.) randomly, taking into account pre-set layout constraints for each type of document (quotation, invoice, delivery note, etc.), and further generates random information for each field. The image processing system 10 also uses a field information template, which is a collection of related fields, and automatically inputs random field values from each field information template to ensure consistency of field values for specified fields. The image processing system 10 also generates a dataset to ensure that information that should be the same across fields (such as the sum of amounts in the details and the total amount) matches, and uses this dataset for automatic layout. Furthermore, the image processing system 10 prepares multiple templates for each field and inputs random information corresponding to each field into a randomly selected template. Finally, the image processing system 10 generates training data by linking the text information used when generating the document image, the position information of the text information, and the document image.
[0096] According to the above configuration, it is possible to efficiently generate a large amount of training data for machine learning that closely resembles actual forms. Furthermore, it is possible to improve the accuracy of machine learning, character recognition, and AI processing in electronic ledger management.
[0097] The document images in this disclosure are not limited to forms, but may also be various general document images. For example, the document images in this disclosure may be various reports that use templates, such as work reports, business trip reports, sales reports, survey reports, work reports, business reports, and financial statements. By applying the features of this disclosure to these document images (template documents), it becomes possible to perform high-precision character recognition processing, manage them in appropriate folders, and centrally manage information, thereby improving the efficiency of office work. When applying the features of this disclosure to the aforementioned document images, the image processing system 10 generates document images based on pre-set layout constraints for each document attribute (form, report, notification, etc.) and document type (for forms: quotation, invoice, delivery note, etc.; for reports: work report, business trip report, sales report, survey report, work report, business report, financial statement, etc.).
[0098] In the image processing system 10, the image processing device 1 and the output device 2 may be configured as a single integrated device. Alternatively, each processing unit of the image processing device 1 (acquisition processing unit 111, report image generation processing unit 112, training data generation processing unit 113, and training processing unit 114) may be distributed across multiple devices. For example, the training processing unit 114 may be included in a different device (training device) from the image processing device 1. In this case, the image processing system 10 may consist of the image processing device 1 and a training device that generates a trained model by performing machine learning using the training data generated by the image processing device 1.
[0099] Furthermore, the control unit 11 of the image processing device 1 controls the entire image processing device 1. The control unit 11 realizes various functions by reading and executing various programs stored in the storage unit 12 (for example, storage or ROM). The control unit 11 may be realized by one or more control devices / arithmetic units (CPU (Central Processing Unit), SoC (System on a Chip)). Also, the control unit 11 may be composed of one or more control circuits (electronic circuits).
[0100] [Disclosure Note] The following is an overview of the disclosures extracted from the above-described embodiments. Note that each configuration and processing function described in the following notes can be selected and combined as desired.
[0101] <Note 1> An image processing device that generates training data used for machine learning in character recognition processing, which recognizes characters from document images, A processing unit that obtains constraints for arranging document item information, A document image generation processing unit generates a document image in which the item information is randomly arranged based on the aforementioned constraints, A training data generation processing unit generates training data by associating the document image, labels representing items, text information used when generating the document image, and positional information of the text information with each other. An image processing device equipped with the following features.
[0102] <Note 2> The document image generation processing unit randomly acquires the item information. The image processing device described in Appendix 1.
[0103] <Note 3> The document image generation processing unit generates the document image for each type of document. The image processing apparatus described in Appendix 1 or 2.
[0104] <Note 4> The document image generation processing unit randomly inputs the item value information of multiple related items in a manner that ensures consistency between items. An image processing device as described in any of the appendices 1 to 3.
[0105] <Note 5> The document image generation processing unit generates a dataset in which data that should have the same item value information among the multiple items is aligned, and generates the document image using the dataset. An image processing device as described in any of the appendices 1 to 4.
[0106] <Note 6> The document image generation processing unit pre-registers multiple templates for each item, and inputs random information corresponding to each item for a template randomly selected from among the multiple templates. An image processing device as described in any of the appendices 1 to 5.
[0107] <Note 7> The document image generation processing unit separates the content of the item from the value of the item and generates a rectangle, position information, and text. An image processing device as described in any of the appendices 1 to 6.
[0108] <Note 8> An image processing device as described in any of Appendix 1 to 7, A learning device that generates a trained model by performing machine learning using the training data generated by the image processing device, An image processing system equipped with the following features.
[0109] <Note 9> An output device that performs character recognition processing on an input image using the trained model generated by the learning device described in Appendix 8, and outputs the character recognition result.
[0110] <Note 10> An image processing method for generating training data used in machine learning for character recognition processing that recognizes characters from document images, To obtain the constraints for arranging the item information in the document, Based on the aforementioned constraints, generate the document image in which the item information is randomly arranged, The process involves generating data as training data by associating the document image, the label representing the item, the text information used to generate the document image, and the positional information of the text information with each other. An image processing method performed by one or more processors.
[0111] <Note 11> An image processing program that generates training data used for machine learning in character recognition processing, which recognizes characters from document images, To obtain the constraints for arranging the item information in the document, Based on the aforementioned constraints, generate the document image in which the item information is randomly arranged, The process involves generating data as training data by associating the document image, the label representing the item, the text information used to generate the document image, and the positional information of the text information with each other. An image processing program for causing one or more processors to execute, or a non-temporary computer-readable recording medium on which the image processing program is recorded. [Explanation of Symbols]
[0112] 1: Image processing device 2: Output device 10: Image Processing System 11: Control Unit 12: Storage section 13: Operation display section 14: Communications Department 111: Acquisition Processing Unit 112: Form Image Generation Processing Unit 113: Training data generation processing unit 114: Learning Processing Unit
Claims
1. An image processing device that generates training data used for machine learning in character recognition processing, which recognizes characters from document images, A processing unit that obtains constraints for arranging document item information, A document image generation processing unit generates a document image in which the item information is randomly arranged based on the aforementioned constraints, A training data generation processing unit generates training data by associating the document image, labels representing items, text information used when generating the document image, and positional information of the text information with each other. An image processing device equipped with the following features.
2. The document image generation processing unit randomly acquires the item information. The image processing apparatus according to claim 1.
3. The document image generation processing unit generates the document image for each type of document. The image processing apparatus according to claim 1.
4. The document image generation processing unit randomly inputs the item value information of multiple related items in a manner that ensures consistency between items. The image processing apparatus according to claim 1.
5. The document image generation processing unit generates a dataset in which data that should have the same item value information among the multiple items is aligned, and generates the document image using the dataset. The image processing apparatus according to claim 1.
6. The document image generation processing unit pre-registers multiple templates for each item, and inputs random information corresponding to each item for a template randomly selected from among the multiple templates. The image processing apparatus according to claim 1.
7. The document image generation processing unit separates the content of the item from the value of the item and generates a rectangle, position information, and text. The image processing apparatus according to claim 1.
8. An image processing apparatus according to any one of claims 1 to 7, A learning device that generates a trained model by performing machine learning using the training data generated by the image processing device, An image processing system equipped with the following features.
9. An output device that performs character recognition processing on an input image using the trained model generated by the learning device described in claim 8, and outputs the character recognition result.
10. An image processing method for generating training data used in machine learning for character recognition processing that recognizes characters from document images, To obtain the constraints for arranging the item information in the document, Based on the aforementioned constraints, generate the document image in which the item information is randomly arranged, The process involves generating data as training data by associating the document image, the label representing the item, the text information used to generate the document image, and the positional information of the text information with each other. An image processing method performed by one or more processors.
11. An image processing program that generates training data used for machine learning in character recognition processing, which recognizes characters from document images, To obtain the constraints for arranging the item information in the document, Based on the aforementioned constraints, generate the document image in which the item information is randomly arranged, The process involves generating data as training data by associating the document image, the label representing the item, the text information used to generate the document image, and the positional information of the text information with each other. An image processing program that is executed by one or more processors.