Information processing device, image processing device, generation method, and program

Abstract

To generate a highly robust trained model while reducing the time and effort required to collect training images.SOLUTION: An information processing system 100, functioning as an information processing device, comprises: acquisition means for acquiring a training document image 114 and table structure information contained in the training document image 114; generation means for generating a masked image in which an object region in the training document image 114 is masked; and a learning unit 123 which generates a trained model which outputs table structure information included in a document image by training using the masked image and data representing the position of the table structure based on the acquired information as a training dataset.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] TECHNICAL FIELD This disclosure relates to processing document images. [Background technology]

[0002] There are documents such as invoices and purchase orders that contain itemized lists. Itemized lists contain information such as product names, quantities, unit prices, and amounts, which provide a breakdown of the invoice or order amount. One method for appropriately extracting the information contained in the itemized list from a document image of such a document is to identify the table structure within the document.

[0003] Patent Document 1 describes a method for acquiring information about a table structure based on position information of character blocks included in an image of the table structure. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 5-334490 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the method of Patent Document 1 may not be able to properly acquire information about the table structure. For example, the method of Patent Document 1 does not take into account cell merging, so there is a risk that the position of a cell may be erroneously identified from a table structure in which cells are merged. Also, extraction of character blocks may fail from a table structure without ruled lines, and in such cases, there is a risk that the method of Patent Document 1 may erroneously identify the position of a cell.

[0006] Therefore, in order to appropriately acquire table structure information, it is conceivable to collect document images for training, perform machine learning using the collected document images, and generate a trained model that outputs table structure information contained in the document images. However, since there are many notation variations that can be contained in document images, in order to generate a highly robust trained model that appropriately outputs table structure information even when various document images are input, it is necessary to perform machine learning using various training document images. In this case, the burden of collecting document images for training increases. [Means for solving the problem]

[0007] The information processing device of the present disclosure is characterized by having an acquisition means for acquiring a training document image and information on a table structure contained in the training document image; a generation means for generating a mask image in which an object area in the training document image is masked; and a learning means for generating a trained model that outputs information on the table structure contained in the document image by training using the mask image and data representing the position of the table structure based on the information as a training dataset. [Effects of the Invention]

[0008] According to the present disclosure, a highly robust trained model can be generated while reducing the effort required to collect images for training. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of an information processing system. [Figure 2] FIG. 2 is a diagram showing an example of the hardware configuration of each device in the information processing system. [Figure 3] FIG. 2 is a diagram showing a sequence of the information processing system. [Figure 4] 10 is a flowchart for generating a table structure specific model. [Figure 5] 10A and 10B are diagrams illustrating an example of a learning document image and table structure coordinate information. [Figure 6]FIG. 10 is a diagram showing an example of a mask image and a correct label image included in a training dataset. [Figure 7] 10 is a flowchart showing the flow of processing for identifying a table structure in a document image. [Figure 8] FIG. 10 is a diagram showing an example of an inference result of a table structure identification model. [Figure 9] FIG. 10 is a diagram illustrating an example of a method for extracting an item value string. [Figure 10] FIG. 10 is a diagram illustrating an example of a confirmation screen. [Figure 11] 10 is a flowchart for generating a table structure specific model. [Figure 12] FIG. 10 is a diagram for explaining a method for correcting a mask area. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the technology of the present disclosure will be described with reference to the drawings. Note that the components described in the following embodiments are merely examples and are not intended to limit the scope of the technology of the present disclosure.

[0011] <Embodiment 1> [System Configuration] Fig. 1 is a diagram showing an example of the configuration of an information processing system. As shown in Fig. 1, the information processing system 100 includes, for example, an image forming device 110, a learning device 120, and an information processing server 130, which are connected to one another via a network 104. In the information processing system 100, the image forming device 110, the learning device 120, and the information processing server 130 may be configured to be connected to the network 104 in multiple rather than single connections. For example, the information processing server 130 may be configured to be composed of a first server device having high-speed computing resources and a second server device having large-capacity storage, which are connected to one another via the network 104.

[0012] The image forming apparatus 110 is an information processing apparatus realized by an MFP (Multi-Function Peripheral) having multiple functions such as printing, scanning, faxing, etc. The image forming apparatus 110 includes at least an image acquisition unit 118.

[0013] The image forming apparatus 110 has a scanner device 206 (see FIG. 2). The scanner device 206 optically reads an original document 111, which is a document with character strings printed on a print medium such as paper, and the image acquisition unit 118 performs predetermined image processing on the resulting data to generate a document image 113. Also, for example, the image acquisition unit 118 receives FAX data 112 transmitted from a FAX transmitter (not shown), and generates the document image 113 by performing predetermined FAX image processing. The image acquisition unit 118 transmits the generated document image 113 to the information processing server 130.

[0014] The information processing device having the image acquisition unit 118 may be an information processing device such as a PC (Personal Computer) in addition to the image forming device 110 having the above-mentioned scanning and faxing functions. For example, the PC may generate a document image 113 such as a PDF or JPEG using a document creation application running on the PC, and transmit the document image 113 to the information processing server 130.

[0015] The learning device 120 is an information processing device having functional units of a learning data generation unit 122 and a learning unit 123. A plurality of learning document images 114 provided by an engineer and table structure coordinate information 115 corresponding to each of the plurality of learning document images 114 are input to the learning device 120. The learning data generation unit 122 generates a learning dataset based on the plurality of learning document images 114 and the table structure coordinate information 115. Details will be described later. The learning unit 123 uses the learning dataset to generate a table structure identification model 116, which is a trained model. Details of the table structure identification model 116 will be described later.

[0016] The information processing server 130 is an image processing device having the functional units of an information processing unit 131 and a data management unit 132. The information processing unit 131 recognizes multiple character strings included in the received document image 113, identifies the position of each cell of the schedule included in the document image 113 using the table structure identification model 116, and extracts item value character strings 117 from the schedule. The extracted item value character strings 117 are character strings for each item, such as product name, quantity, unit price, and amount. The character string extraction process will be described in detail later. The data management unit 132 stores the data of the item value character strings 117 extracted by the information processing unit 131.

[0017] The network 104 is realized by a LAN, a WAN, or the like, and is a communication unit that connects the image forming device 110, the learning device 120, and the information processing server 130 to one another and transmits and receives data between the devices.

[0018] Note that the functional units of the image acquisition unit 118, the learning data generation unit 122, the learning unit 123, the information processing unit 131, and the data management unit 132 do not have to be included in each device as shown in Fig. 1. For example, the learning data generation unit 122 and the learning unit 123 may be included in an information processing server. Alternatively, all of the functional units may be included in one device included in the information processing system 100.

[0019] [Hardware configuration of image forming device] FIG. 2 is a diagram showing an example of the hardware configuration of the image forming device 110, the learning device 120, and the information processing server 130 included in the information processing system 100 of FIG.

[0020] 2(a) is a diagram showing the hardware configuration of the image forming apparatus 110. The image forming apparatus 110 has a CPU 201, a ROM 202, a RAM 204, a printer device 205, a scanner device 206, and a document transport device 207. It also has a storage 208, an input device 209, a display device 210, and an external interface 211, which are connected to one another via a data bus 203.

[0021] The CPU 201 is a control unit for controlling the overall operation of the image forming apparatus 110. The CPU 201 starts up the system of the image forming apparatus 110 by executing a startup program stored in the ROM 202, and realizes functions of the image forming apparatus 110, such as printing, scanning, and faxing, by executing a control program stored in the storage 208.

[0022] ROM 202 is realized by non-volatile memory and is a storage unit for storing a startup program that starts up image forming apparatus 110. Data bus 203 is a communication unit for transmitting and receiving data between devices that make up image forming apparatus 110. RAM 204 is realized by volatile memory and is a storage unit used as a work memory when CPU 201 executes a control program.

[0023] The printer device 205 is an image output device that prints and outputs a document image on a print medium such as paper. The scanner device 206 is an image input device that is used to optically read a print medium (document) such as paper on which characters, diagrams, etc. are printed to generate a document image. The document transport device 207 is realized by an ADF (Auto Document Feeder) or the like, detects documents placed on a platen, and transports the detected documents one by one to the scanner device 206. The storage 208 is realized by an HDD (Hard Disk Drive) or the like, and is a storage unit for storing the control program and document images described above.

[0024] The input device 209 is realized by a touch panel, hard keys, or the like, and accepts operation input from the user to the image forming apparatus 110. The display device 210 is realized by a liquid crystal display, or the like, and is a display unit for displaying and outputting a setting screen of the image forming apparatus 110 to the user. The external interface 211 connects the image forming apparatus 110 and the network 104, and is an interface unit for receiving fax data from a fax transmitter (not shown) and transmitting document images to the information processing server 130.

[0025] [Hardware configuration of the learning device] 2(b) is a diagram showing the hardware configuration of the learning device 120. As shown in FIG. 2(b), the learning device 120 has a CPU 231, a ROM 232, a RAM 234, a storage 235, an input device 236, a display device 237, an external interface 238, and a GPU 239, which are connected to each other via a data bus 233.

[0026] The CPU 231 is a control unit for controlling the overall operation of the learning device 120. The CPU 231 starts up the system of the learning device 120 by executing a boot program stored in the ROM 232. Then, the CPU 231 generates a table structure identification model by executing a learning program stored in the storage 208. The ROM 232 is realized by non-volatile memory and is a storage unit for storing the boot program that starts up the learning device 120. The data bus 233 is a communication unit for transmitting and receiving data between devices that make up the learning device 120. The RAM 234 is realized by volatile memory and is a storage unit used as work memory when the CPU 231 executes the learning program. The storage 235 is realized by an HDD (Hard Disk Drive) or the like and is a storage unit for storing the learning program and table structure identification model described above.

[0027] The input device 236 is realized by a mouse, keyboard, etc., and accepts operational input from engineers to the learning device 120. The display device 237 is realized by a liquid crystal display, etc., and is a display unit that displays and outputs the setting screen of the learning device 120 to the engineer. The CPU 231 operates as a display control unit that controls the screen displayed on the display device 237. The external interface 238 connects the learning device 120 to the network 104, and is an interface unit that receives learning document images from a PC, etc. (not shown), and transmits the table structure identification model 116 to the information processing server 130.

[0028] The GPU 239 is a computing unit configured with an image processing processor. The GPU 239 executes computations for generating the table structure identification model 116 based on the training dataset, for example, in accordance with control commands given from the CPU 231.

[0029] 1 are realized by the CPU 231 executing a predetermined program, but this is not limiting. Other hardware, such as a GPU 239 for accelerating calculations or a field programmable gate array (FPGA) (not shown), may also be used. Each functional unit may be realized by a combination of software and hardware, such as a dedicated IC, or some or all of the functions may be realized solely by hardware.

[0030] [Hardware configuration of information processing server] 2(c) is a diagram showing the hardware configuration of the information processing server 130. As shown in FIG. 2(c), the information processing server 130 has a CPU 261, a ROM 262, a RAM 264, a storage 265, an input device 266, a display device 267, and an external interface 268, which are connected to each other via a data bus 263.

[0031] The CPU 261 is a control unit for controlling the overall operation of the information processing server 130. The CPU 261 starts up the system of the information processing server 130 by executing a boot program stored in the ROM 262, and executes an information processing program stored in the storage 265 to perform information processing such as character recognition and information extraction.

[0032] The ROM 262 is realized by a non-volatile memory and is a storage unit for storing a boot program that starts up the information processing server 130. The data bus 263 is a communication unit for transmitting and receiving data between devices that make up the information processing server 130. The RAM 264 is realized by a volatile memory and is a storage unit used as a work memory when the CPU 261 executes the information processing program.

[0033] The storage 265 is realized by an HDD (Hard Disk Drive) or the like, and is a memory unit for storing the aforementioned information processing program, document image 113, table structure specification model 116, data of item value string 117, and the like.

[0034] The input device 266 is realized by a mouse, a keyboard, or the like, and receives operation inputs to the information processing server 130 from a user who uses the information processing server 130 or an engineer who controls the information processing server 130 .

[0035] The display device 267 is realized by a liquid crystal display or the like, and is a display unit for displaying and outputting the setting screen of the information processing server 130 to a user who uses the information processing server 130 or an engineer who controls the information processing server 130. The CPU 261 operates as a display control unit that controls the screen displayed on the display device 267.

[0036] The external interface 268 is an interface unit that connects the information processing server 130 and the network 104 , and receives the table structure identification model 116 from the learning device 120 and the document image 113 from the image forming device 110 .

[0037] 1 are realized by the CPU 261 executing a predetermined program, but this is not limiting. Other hardware such as a GPU (Graphics Processing Unit) or an FPGA (Field Programmable Gate Array) may also be used to speed up calculations. Each functional unit may be realized by a combination of software and hardware such as a dedicated IC, or some or all of the functions may be realized solely by hardware.

[0038] [Sequence of generating trained model] 3 is a diagram showing a usage sequence of the information processing system 100 in FIG. 1. The symbol "S" in the explanation of each process indicates a step in the sequence, and this also applies to subsequent flowcharts. For convenience of explanation, the operations of a user or engineer are also explained using steps.

[0039] The steps in Figure 3(a) represent the flow when an engineer develops a table structure specifying model, and are a diagram explaining the flow of generating a table structure specifying model by the learning device 120. Details of S302 to S304 in Figure 3(a) will be described later using Figure 4.

[0040] In S301, an engineer of the information processing system 100 inputs, to the learning device 120, a plurality of learning document images and table structure coordinate information corresponding to each of the plurality of learning document images.

[0041] In S302, the learning device 120 generates a learning data set based on the data input in S301.

[0042] In S303, the learning device 120 uses the learning dataset generated in S302 to train a model that outputs information about the table structure in the document image, and generates a table structure identification model that is a trained model.

[0043] In S304, the learning device 120 transmits the table structure identification model generated in S303 to the information processing server 130.

[0044] In S305, the information processing server 130 stores the table structure identification model transmitted in S304 in the storage 265.

[0045] [Field value string extraction process sequence] 3(b) is a diagram illustrating the flow of processing in which the information processing server 130 extracts character strings in a table included in the document image 113 input by the user. Details of S313 in FIG. 3(b) will be described with reference to FIG. 7.

[0046] In S311, a user of the information processing system 100 places a paper document (original) on the image forming device 110 and instructs the image forming device 110 to scan the document in order to obtain data on character strings contained in a table in the document.

[0047] In S312, the scanner device 206 of the image forming apparatus 110 reads the placed paper document, and the image acquisition unit 118 generates a document image that is an image of the scanned document. Then, the image acquisition unit 118 transmits the generated document image to the information processing server 130.

[0048] In S313, the information processing server 130 acquires the document image transmitted in S312 from the image forming apparatus 110. Then, the information processing server 130 uses the table structure identification model to identify the table structure included in the acquired document image, and extracts character strings from the area of the identified table structure.

[0049] In S314, the information processing server 130 outputs the character string extracted in S313 to the user.

[0050] [Table structure specific model generation process] Next, the details of the process for generating a table structure identification model, which is a trained model, will be described. Hereinafter, unless otherwise specified, the term table structure identification model refers to a trained table structure identification model. The table structure identification model is a trained model that outputs information for identifying the area of each cell that makes up a table (table structure) contained in an input image as an inference result (output result). Once the area of each cell is properly identified, the character string within each cell can be properly extracted.

[0051] As a training dataset used to train a pre-training table structure identification model, training images and correct label images indicating the positions of table cells included in the training images are created. A table structure identification model is generated by training the pre-training table structure identification model using the training dataset. The table structure identification model can identify the pixels belonging to the cells that make up the table included in the image based on the features of the input image and identify the table structure.

[0052] When an image is input, the table structure identification model acquires the image's features by performing convolution processing or the like on the image within the model, and identifies each pixel based on the acquired features as to whether it corresponds to a table cell. At this time, if the input image contains text, information such as the shape and design of the text is also acquired as features. For the same reason, in the learning phase, if the learning image contains text, the information about the text is also taken into account when learning.

[0053] For example, an image may include a statement that shows the breakdown of the total price. The text included in the statement may have a wide variety of variations in its appearance. Examples of variations include the design of the text, such as the font, size, and color of the text, and the character types for each language, such as Japanese, English, and Arabic. Furthermore, because the item names on the statement include various product names, the variations in appearance also include variations in the wording.

[0054] In this way, to generate a highly robust table structure identification model that can appropriately identify table structures even when images containing character strings with various notation variations are input, a large number of variations of images must be prepared in advance as training images. However, preparing a large number of variations of training images is a burden. For example, documents such as forms containing detailed tables are internal company information, so it is difficult to collect a large number of variations of form images as training images.

[0055] Furthermore, even during inference, the table structure identification model may not be able to make correct inferences from images of forms that have untrained variations because the image features are unknown. For example, for a table structure identification model trained only with Japanese images, the shapes of character strings in other languages are unknown, and the features obtained from the image are unknown. Therefore, for example, if an image of only Arabic is input to a table structure identification model trained only with Japanese images, the table structure identification model may have difficulty identifying the correct table structure. Therefore, in this embodiment, a method for performing learning and inference while suppressing dependence on character string information is described.

[0056] FIG. 4 is a flowchart illustrating the details of S302 to S304 in FIG. 3(a) that are executed by the learning device 120. The series of processes shown in the flowchart in FIG. 4 are performed by at least one of the CPU 231 and GPU 239 of the learning device 120 by loading program code stored in the ROM 232 or storage 235 into the RAM 234 and executing it. In addition, some or all of the functions of the steps in FIG. 4 may be realized by hardware such as an ASIC or electronic circuit. In the following explanation, as an example, the CPU 231 executes the processing of each step.

[0057] In S401, the CPU 231 acquires a plurality of sets of learning document images and table structure coordinate information of the learning document images input by the engineer in S301 of FIG.

[0058] FIG. 5(a) is a diagram showing an example of a learning document image acquired in S401. Learning document image 500 in FIG. 5(a) shows a document image of an invoice. Learning document image 500 acquired in S401 includes a table structure. The table structure is made up of multiple rectangles (cells). The table structure is, for example, a statement showing the breakdown of the invoice amount as shown in FIG. 5(a), and the following explanation will be given assuming that the table structure included in the document image is a statement. In FIG. 5(a), the area indicated by the dashed line in learning document image 500 indicates the area of statement 501 in learning document image 500. Learning document image 500 is, for example, a full-color (three-channel RGB) image, with a width of 2480 pixels and a height of 3508 pixels, equivalent to A4 size at 300 dpi.

[0059] FIG. 5(b) is a diagram showing an example of table structure coordinate information acquired in S401. Table structure coordinate information is, for example, text data describing the coordinates of the position of a spreadsheet included in the input learning document image and the coordinates of the positions of each cell that makes up the spreadsheet. Coordinates 511 of table structure coordinate information 510 in FIG. 5(b) describe the coordinates of the position and size of the area of spreadsheet 501 in learning document image 500. Coordinates x and y in coordinates 511 indicate the coordinates of the upper left corner of the area of spreadsheet 501. The coordinates are values when the upper left corner of learning document image 500 is the origin. Coordinates 511 also describe the width w and height h of the area of spreadsheet 501.

[0060] Cell coordinates 512 describe the coordinates of the position of each cell that constitutes detailed table 501 contained in the learning document image and the size of each cell. Coordinates x and y in coordinates 512 indicate the coordinates of the upper left position of the corresponding cell. The coordinates are values when the upper left of learning document image 500 is the origin. Coordinates 512 also describe the width w and height h of the corresponding cell. Although some details are omitted in Figure 5(b), the coordinates of the position and size of each of the multiple cells that constitute detailed table 501 are described.

[0061] Although there are no limitations on the method for setting the coordinates of the cells that make up the schedule included in the learning document image, it is desirable to set them so that the areas of multiple cells do not overlap each other and so that they do not overlap the ruled lines of the schedule.

[0062] The next steps S402 to S407 are loop processing, which is executed repeatedly for the number of learning document images acquired in S401. That is, in S403, CPU 231 selects a learning document image to be processed from the multiple learning document images acquired in S401. Then, the processing of S404 to S406 is performed on the learning document image to be processed. When the processing for the learning document image to be processed is completed, in S407 it is determined whether processing has been performed on all of the multiple learning document images. If it is determined that processing has not been performed on all learning document images, the process returns to S403, and a learning document image to be processed is selected from the unprocessed learning document images. In the loop processing, the following processing is performed.

[0063] In S404, the CPU 231 executes character detection processing on the learning document image to detect character regions included in the learning document image. A character region is a circumscribed rectangular region of a text string (character string) consisting of adjacent characters. If there is a blank space or a ruled line between characters, each character is detected as a separate character region.

[0064] 6(a) is a diagram for explaining the result of character detection in S404, and the solid-line rectangular area indicates the detected character area. When the learning document image 500 is selected as the processing target, the area including each character string in the learning document image 500 is detected as a character area.

[0065] In S405, the CPU 231 performs mask processing on the character area detected in S404 to generate a mask image.

[0066] FIG. 6(b) is a diagram illustrating the results of the masking process in S405. The masking process for the character region is a process of turning all pixels in the character region detected in S404 black, i.e., a process of setting all RGB values to 0. As a result, a mask image 610 as shown in FIG. 6(b) is generated. Note that the masking process in S405 has been described as turning all pixels in the character region black, but this is only one example of masking process, and the color used to mask the character region may be a color other than the background color. Furthermore, the masking color may be different for each character region. For example, the masking process may be performed using a color randomly selected from colors different from the background color of the cell region.

[0067] In the masking process of this embodiment, the masked area is not limited to the text area. For example, a document image may contain an image of a company logo, and such a logo may also be masked. In other words, a mask image may be generated such that all objects, such as text and images, contained in the document image are masked.

[0068] In S406, the CPU 231 generates a correct label image using table structure coordinate information corresponding to the learning document image to be processed. The correct label image is an image in which correct labels are assigned to pixels corresponding to cells that make up the schedule included in the learning document image to be processed.

[0069] Fig. 6(c) is a diagram showing an example of a correct label image generated in S406. As shown in Fig. 6(c), the correct label image 620 is a binary image in which, for example, white pixels are pixels with correct labels that correspond to cells that make up the schedule, and black pixels are pixels other than those with correct labels. The size of the correct label image is the same as the image size of the learning document image to be processed.

[0070] The correct label image is generated by determining the pixel values of pixels belonging to cells as values indicating white, based on the position and size information of each cell described in the table structure coordinate information 510 corresponding to the learning document image to be processed. In the case of an 8-bit image, the pixel values of pixels belonging to cells are determined to be 255. The pixel values of pixels not belonging to cells are determined to be values indicating black. In other words, the pixel values of pixels not belonging to cells are determined to be 0.

[0071] By performing S404 to S406, a training dataset consisting of two pieces of data, a mask image 610 and a supervised label image 620, is generated from the training document image to be processed. The training dataset is a dataset that uses the mask image 610 as input data and the supervised label image 620 as training data.

[0072] By repeating the loop processing of S402 to S407 until there are no more unprocessed learning document images, multiple learning data sets generated from the multiple learning document images acquired in S401 are acquired. If the CPU 231 determines in S407 that all learning document images have been processed, the process proceeds to S408.

[0073] In S408, the CPU 231 generates a table structure identification model by machine learning using the generated training dataset. The table structure identification model of this embodiment is trained so that when a mask image generated by masking a document image is input, a binary image in which the areas of each cell of a table included in the document image are white pixels is output as an inference result image.

[0074] Note that a known machine learning method may be used to train the table structure identification model. For example, a fully convolutional network (FCN) that detects objects in an image at the pixel level in machine learning may be used. Furthermore, a general object recognition method such as YOLO (YouOnlyLookOnce) that estimates the area of an object in an image may also be used.

[0075] For example, in S408, first, a pre-training table structure identification model is prepared that outputs an inference result image as an inference result when an image containing a table structure is input. A mask image from the training dataset is input to the pre-training table structure identification model, and the model outputs an inference result image. The pre-training table structure identification model is trained so that the value indicating the discrepancy between the output inference result image and the correct label image from the training dataset approaches 0, thereby generating a trained table structure identification model.

[0076] In S409, the CPU 231 transmits the table structure identification model generated in S408 to the information processing server 130. Then, the table structure identification model is saved in the storage 265 in the information processing server 130.

[0077] [Table structure identification process] Fig. 7 is a flowchart illustrating the details of S313 in Fig. 3(b) executed by the information processing server 130. The series of processes shown in the flowchart in Fig. 7 are performed by the CPU 261 of the information processing server 130 by loading program code stored in the ROM 262 or the storage 265 into the RAM 264 and executing it. Also, some or all of the functions of the steps in Fig. 7 may be realized by hardware such as an ASIC or an electronic circuit. In the following explanation, as an example, it is assumed that the CPU 261 executes the processing of each step.

[0078] In S701, the CPU 261 acquires a table structure identification model stored in the storage 265. The table structure identification model is a trained model generated as a result of the processing of the flowchart in FIG.

[0079] In S702, the CPU 261 acquires the document image transmitted from the image forming apparatus 110 in S312 of Fig. 3. The document image acquired in S702 becomes the image to be subjected to the table structure identification process.

[0080] 8(a) is a diagram showing an example of a document image acquired in S702. The document image 800 acquired in S702 will be described as a document image that includes a detailed table 801 as a table structure, similar to the learning document image.

[0081] In S703, the CPU 261 executes character detection processing on the document image acquired in S702 to detect character regions included in the document image. The method of character detection processing may be the same as the character detection processing described in S404.

[0082] In S704, the CPU 261 performs mask processing on the character region detected in S703 to generate a mask image corresponding to the document image. The method for generating the mask image may be the same as the method for generating the mask image described in S405.

[0083] A mask image 810 in FIG. 8(b) shows an example of a mask image obtained as a result of mask processing performed on the document image 800 in FIG. 8(a) acquired in S702.

[0084] In S705, the CPU 261 inputs the mask image generated in S704 to the table structure specifying model acquired in S701, and as a result, acquires an inference result image output by the table structure specifying model.

[0085] FIG. 8(c) is a diagram showing an example of an inference result image output by the table structure identification model when the mask image 810 in FIG. 8(b) is input to the table structure identification model. As shown in FIG. 8(c), the inference result image 820 has the same configuration as the correct label image 620 in FIG. 6(c). That is, in the inference result image 820, the pixel values in the cells constituting the schedule 801 indicate that the cells are cells. The value indicating that the cells is, for example, a pixel value of 255. An area 821 in the inference result image 820 is the area of the schedule 801, and the white areas in the area 821 are areas of pixels with values indicating that the cells are cells. Furthermore, in the inference result image 820, the pixel values of pixels other than the cells indicate that the cells are not cells. The value indicating that the cells are not cells is, for example, a pixel value of 0, and the black areas in the inference result image 820 are areas with values indicating that the cells are not cells.

[0086] In S706, the CPU 261 performs character recognition processing on the character areas detected in S703 to recognize character strings contained in each detected character area. The character recognition processing may use an existing character recognition technology that identifies characters based on pixel information contained in a character area.

[0087] In S707, the CPU 261 identifies the structure of the schedule 801 included in the document image acquired in S702, based on the inference result image 820. In this embodiment, the CPU 261 identifies the area of the cells in each row of the schedule 801 included in the document image.

[0088] 9 is a diagram for explaining a method for specifying the area of cells in each row of a detailed table 801 included in a document image from an inference result image 820. A method for specifying the area of cells in each row when an inference result image 820 is obtained from a document image 800 will be explained.

[0089] First, pixels having values indicating that they are cells in the inference result image 820 obtained in S705, i.e., closed regions of continuous white pixels in the inference result image 820, are extracted. The extracted closed regions are extracted as the regions of cells that make up the detailed table 801 included in the inference result image 820. For example, as shown in FIG. 9(b), cells 921, 922, 923, and 924 are extracted from the detailed table region 821 in the inference result image 820.

[0090] Next, from the extracted cells, cell 921 located in the upper left corner of document image 800 is selected. Then, for example, if the Y coordinate value of the upper left corner of selected cell 921 is y and the height value is h, cells whose upper left y coordinate values are within the range from y±H to y+h±H are identified from the extracted cells, where H is a predetermined constant. This identifies cells 922, 923, and 924 corresponding to selected cell 921. These identified cells 922, 923, and 924 are identified as cells belonging to the same row as selected cell 921. Furthermore, by repeatedly performing the same process except for the selected cell and cells identified as being in the same row as the selected cell, cells in each row of schedule 801 can be extracted.

[0091] In S708, the CPU 261 extracts an item value character string from the character strings recognized in S706. Specifically, for each row identified in S707, the CPU 261 obtains the character string contained in the cell.

[0092] For example, as shown in FIG. 9(a), "item name" is obtained as character string 901 contained in cell 921 in the first row of statement 801. Similarly, "unit price" is obtained as character string 902 contained in cell 922, "quantity" is obtained as character string 903 contained in cell 923, and "amount" is obtained as character string 904 contained in cell 924. By performing the same process for each row, it is possible to extract character strings in the cells for each row. Of these, for example, the character string obtained from the cell in the first row is obtained as a character string representing the name of the item (item name). Furthermore, character strings obtained from the second row onwards are obtained as character strings corresponding to the items (item value character strings). From statement 801, the item names of item name, unit price, quantity, and amount, and the character strings corresponding to the items are obtained as item value character strings.

[0093] It is also possible to identify columns in a schedule based on the position and size of each identified cell. By identifying columns, for example, it is possible to extract a list of product names from a column that stores product names. In this way, by identifying the table structure, it is possible to appropriately extract information written in a schedule included in a document.

[0094] In S709, the CPU 261 performs processing for presenting the item value string acquired in S708 to the user. Specifically, the CPU 261 performs control so that a confirmation screen for presenting the item value string to the user is displayed on any display device of the information processing system 100.

[0095] 10 is a diagram showing an example of a confirmation screen displayed as a result of S709. The confirmation screen 1000 is displayed, for example, on the display device 210 of the image forming apparatus 110 or the display device 267 of the information processing server 130. The confirmation screen 1000 includes a preview image display area 1001, a result display area 1002, and an end button 1013.

[0096] The result display area 1002 is an area where character strings 1005 to 1008 obtained from cells constituting the schedule included in the document image 800 are displayed as item value character strings corresponding to the respective items. Edit buttons 1009 to 1012 corresponding to the respective items are provided, and the user can correct the character strings 1005 to 1008 obtained as item value character strings by pressing the edit buttons 1009 to 1012. The preview image display area 1001 displays a preview of the document image 800 currently being processed, and the rows 1004 containing the character strings 1005 to 1008 displayed in the result display area 1002 are highlighted. The schedule area 1003 is also highlighted. The flowchart in FIG. 10 ends when the end button 1013 is pressed.

[0097] As described above, in this embodiment, a mask image obtained by masking a character region of a document image is used as training data input to the model. The mask image displays character strings more abstractly than the document image. Therefore, this embodiment reduces the likelihood of the model learning depending on the spelling variations of the character string. Therefore, this embodiment also reduces the amount of training data, thereby reducing the effort required to collect training data that covers a wide variety of spellings. Furthermore, this embodiment can generate a trained model that performs inference without depending on the spelling variations of the character string, thereby improving the robustness of the trained model.

[0098] It is also possible to identify the table structure by simply regarding the character regions in a table as cell regions without using a trained model. However, with this method, if no ruled lines are drawn, character strings contained in two adjacent cells may be combined and erroneously detected as a single character region, which may result in an inability to properly identify the table structure. On the other hand, even in this embodiment, adjacent character strings may be erroneously detected as a single character region and masked. However, in this embodiment, the training dataset also includes data from such cases. Therefore, the table structure identification model of this embodiment, trained using such a training dataset, can output information appropriate for identifying the table structure even when a mask image in which erroneously detected character regions are masked is input.

[0099] It is also possible to allow a model to learn and infer without relying on the character strings by deleting the character strings in a document image. However, when the character strings are deleted, the information required for the model to acquire the features in the image is reduced, making the model unsuitable as input data for learning. According to this embodiment, masking the character strings allows the character strings to be treated abstractly, thereby preventing the information required for acquiring the features in the image from being reduced.

[0100] <Embodiment 2> In the first embodiment, a method was described in which character regions in a schedule are masked and character strings are treated abstractly, thereby enabling the model to learn and make inferences while reducing dependency on variations in the notation of character strings. In this embodiment, a method is described in which the variation of a training data set is increased while minimizing the effort required to collect data. This embodiment will be described focusing on the differences from the first embodiment. Portions not specifically mentioned have the same configuration and processing as the first embodiment.

[0101] Fig. 11 is a flowchart of this embodiment that explains in detail steps S302 to S304 in Fig. 3(a) that are executed by the learning device 120. The series of processes shown in the flowchart in Fig. 11 are performed by at least one of the CPU 231 and GPU 239 of the learning device 120 expanding program code stored in the ROM 232 or storage 235 into the RAM 234 and executing it. In the following explanation, it is assumed, as an example, that the CPU 231 executes the processing of each step.

[0102] The flowchart in Fig. 11 is obtained by adding new steps to the flowchart in Fig. 4 of embodiment 1. Therefore, in the flowchart in Fig. 11, steps that are the same as steps in the flowchart in Fig. 4 are assigned the same numbers. Therefore, in the explanation of the flowchart in Fig. 11, steps that are different from the flowchart in Fig. 4 will be explained.

[0103] In the flowchart of FIG. 11, the process of S1101 is performed after S405 ends and before S406 is performed. In S1101, the CPU 231 corrects the mask image generated in S405 to generate a new mask image different from the mask image generated in S405. The correction is a process of expanding, dividing, adding, deleting, or reducing the mask area included in each cell. Character strings of various lengths can be entered within table cells included in the document image to be processed, as long as they fit within the cell. For this reason, a new mask image can be generated by performing a correction process, such as expanding the size, on the mask area of the mask image generated in S405. Note that the number of mask images newly generated in S1101 is not limited. Two or more new mask images may be generated.

[0104] The mask image generated in S1101 is also stored as a training data set paired with the correct labeled image of the training document image to be processed, similar to the mask image generated in S405, and is then used as data for training the table structure identification model in S408.

[0105] Thus, in this embodiment, since at least two mask images before and after correction are generated from one learning document image, at least two learning data sets can be generated from one learning document image. Therefore, it is possible to increase the number of learning data sets while suppressing the labor of collecting the number of learning document images.

[0106] FIG. 12 is a diagram for explaining a method of correcting a mask region executed in S1101. FIG. 12(a) is a diagram showing an image of the明细表501 in the mask image generated in S405. Taking the mask region included in the mask image of FIG. 12(a) as an example, a method of correcting the mask region will be described.

[0107] First, an expansion of the mask region, which is one of the mask region correction methods, will be described. The expansion of the mask region is a process of expanding the mask region so as to fit within the cell. By expanding the mask region, it is possible to pseudo-reproduce a document in which a long string is described within the cell.

[0108] FIG. 12(b) is a diagram for explaining the expansion of the mask region in which the left-justified string is masked. FIG. b-1 in FIG. 12(b) is a diagram showing the cell 1202 before correction. The upper left coordinates of the cell 1202 are (x1, y1), the width is w1, and the height is h1. Also, the mask region 1212 is one in which the left-justified string is masked. Assume that the upper left coordinates of the mask region 1212 before correction included in the cell 1202 are (x2, y2), the width is w2, and the height is h2. When expanding the mask region 1212 in FIG. 12(b), the mask region 1212 is expanded to the right within the range that fits within the cell 1202. FIG. b-2 in FIG. 12(b) is an example of the expanded mask region. The expanded mask region 1222 is a mask region in which the width of the mask region 1212 is corrected to w2 + d. d is set to a value within the range of 0 < d < (x1 + w1) - (x2 + w2) so that the mask region fits within the cell 1202. Also, the value of d may be randomly determined within the range.

[0109] FIG. 12(c) is a diagram for explaining the expansion of a mask region in which a right-justified string is masked. FIG. c-1 in FIG. 12(c) is a diagram showing cell 1203 before correction. The upper left coordinates of cell 1203 are (x1, y1), the width is w1, and the height is h1. Also, mask region 1213 is one in which a right-justified string is masked. Assume that the upper left coordinates of the mask region 1213 before correction included in cell 1203 are (x2, y2), the width is w2, and the height is h2. When expanding the mask region 1213, the region is expanded leftward within the range where the mask region 1213 fits into cell 1203. FIG. c-2 in FIG. 12(c) is an example of the expanded mask region. The expanded mask region 1223 has its upper left coordinates as (x2−d, y2) and the width is expanded to w2 + d. By setting d to a value within the range of 0 < d < x2−x1, the mask region 1223 is made to fit into cell 1203. Also, the value of d may be randomly determined as long as it is within the range.

[0110] FIG. 12(d) is a diagram for explaining the expansion of a mask region in which a centered string is masked. FIG. d-1 in FIG. 12(d) is a diagram showing cell 1204 before correction. The upper left coordinates of cell 1204 are (x1, y1), the width is w1, and the height is h1. Also, the mask region 1214 before expansion is one in which a centered string is masked. Assume that the upper left coordinates of the mask region 1214 before correction included in cell 1204 are (x2, y2), the width is w2, and the height is h2. When expanding the mask region 1214, the region is expanded in both the left and right directions within the range where the mask region 1214 fits into cell 1204. FIG. d-2 in FIG. 12(d) is an example of the expanded mask region. The expanded mask region 1224 has its upper left coordinates as (x2−d, y2) and the width is expanded to w2 + 2d. By setting d to a value within the ranges of 0 < d < x2−x1 and 0 < d < (x1 + w1)−(x2 + w2), the mask region 1224 is made to fit into cell 1204. Also, the value of d may be randomly determined as long as it is within the ranges.

[0111] Next, the division of the mask area, which is one of the methods for correcting the mask area, will be described. The division of the mask area means dividing a single mask area into multiple mask areas. According to the division of the mask area, for example, when there is a large blank space between the characters in a cell like "￥ 100", a state where multiple character areas are detected from within the cell can be pseudo-replicated.

[0112] FIG. 12(e) is a diagram for explaining the division of a mask area in which a horizontally centered character string is masked. FIG. e-1 in FIG. 12(e) is a diagram showing the cell 1204 before correction. The upper left coordinate of the cell 1204 is (x1, y1), the width is w1, and the height is h1. Let the upper left coordinate of the mask area 1214 before correction included in the cell 1204 be (x2, y2), the width be w2, and the height be h2. FIG. e-2 in FIG. 12(e) is an example of the mask area after division. The division is performed by setting the area with width d in the mask area 1214 before division as the background color. As a result, two mask areas 1234 and 1244 are replicated. d is set to be within the mask area in the range of 0 < d < w2. The value of d may be randomly determined within the range, and multiple non-masked areas may be set.

[0113] Next, the addition of the mask area, which is one of the methods for correcting the mask area, will be described. The addition of the mask area means setting a mask area in a cell that does not contain a character string. By adding the mask area, a cell that pseudo-contains a character string can be replicated.

[0114] FIG. 12(f) is a diagram for explaining the addition of a mask area. FIG. f-1 in FIG. 12(f) is a diagram showing cell 1205 before correction. The upper left coordinates of cell 1205 are (x1, y1), the width is w1, and the height is h1. Since there is no string in cell 1205, there is no mask area. FIG. f-2 in FIG. 12(f) is an example of the added mask area. Assume that the upper left coordinates of the newly added mask area 1225 are (x2, y2), the width is w2, and the height is h2. Each value is set so that the mask area 1225 fits within cell 1205. The value of x2 is within the range of x1 < x2 < x1 + w1. The value of y2 is within the range of y1 < y2 < y1 + h1. The value of w2 is within the range of 0 < w2 < w1 - (x2 - x1). The value of h2 is within the range of 0 < h2 < h1 - (y2 - y1). These values may be randomly determined as long as they are within the range. Also, a plurality of mask areas may be added within the cell area.

[0115] Also, in an actual specification, strings are often described in all items such as product name, quantity, unit price, etc. Therefore, it is preferable to add a mask area to a row with no description within the cell area. In this case, a mask area may also be added to other cells in the same row. Also, in an actual specification, when strings are described in all items of a certain row, it is often the case that strings are described in all items of all the rows above that row. Therefore, a mask area may be added to each cell in each column of one row immediately below the row with the string description or a plurality of consecutive rows. Thereby, a mask image for learning that more reproduces the characteristics of the actual data can be generated.

[0116] Next, a method of deleting a mask area, which is one of the mask area correction methods, will be described. Deleting a mask area means setting the color of the mask area to the same color as the background color of the cell area. Thereby, a cell that seemingly does not contain a string can be reproduced.

[0117] FIG. 12(g) is a diagram illustrating the deletion of a mask area. Diagram g-1 in FIG. 12(g) shows cell 1204 before correction. Diagram g-2 in FIG. 12(g) is an example of a deleted mask area. To delete a mask area, the background color of cell 1204 is identified based on the pixel values of the pixels in cell 1204 that are not included in mask area 1214. A cell without a mask area can be generated by using the identified background color value as the color of mask area 1214. In an actual statement, a row containing a cell with no text often lacks text for all items in the row, such as product name, quantity, and unit price. Therefore, the deletion of a mask area may involve deleting the mask areas of all cells in the same row. Furthermore, in an actual statement, if no text is written for all items in a row, all rows below that row often lack text. Therefore, the deletion of mask areas may involve deleting one or several consecutive rows immediately above the row with no text. This allows for the generation of a learning mask image that more closely resembles the characteristics of the actual data.

[0118] Next, we will explain how to expand the mask area vertically, which is one method of correcting the mask area. Vertical expansion is a process that expands the mask area vertically so that it fits within the cell. This makes it possible to simulate, for example, a character string with a large font size being written in the cell area.

[0119] FIG. 12(h) is a diagram for explaining the vertical expansion of the mask region. FIG. h-1 in FIG. 12(h) is a diagram showing the cell 1204 before correction. The upper left coordinates of the cell 1204 are (x1, y1), the width is w1, and the height is h1. Also, the mask region 1214 is one in which the centered string is masked. Let the upper left coordinates of the mask region 1214 before correction included in the cell 1204 be (x2, y2), the width be w2, and the height be h2. FIG. h-2 in FIG. 12(h) is an example of the expanded mask region. The expanded mask region 1254 is an image obtained by expanding the mask region 1214 by d in both the upward and downward directions. That is, the mask region 1254 has the upper left coordinates as (x2, y2 - d) and is expanded to a height of h2 + 2d. By setting d to a value within the range of 0 < d < y2 - y1 and 0 < d < (y1 + h1) - (y2 + h2), the mask region 1254 can be made to fit within the cell 1204. Also, the value of d may be randomly determined as long as it is within the range. Furthermore, the value of d may use different values in the upward and downward directions, or only one of the upward and downward regions may be expanded.

[0120] Next, a method for reducing the mask region, which is one of the methods for correcting the mask region, will be described. Reduction is a process of narrowing the region in the width direction or the vertical direction. Thereby, for example, it is possible to pseudo-reproduce the case where a string with a small font size is described in the cell region.

[0121] FIG. 12(i) is a diagram for explaining the reduction of the mask region. In FIG. i-1 in FIG. 12(i), it is a diagram showing the cell 1206 before correction. The upper left coordinates of the cell 1206 are (x1, y1), the width is w1, and the height is h1. Assume that the upper left coordinates of the mask region 1216 before correction included in the cell 1206 are (x2, y2), the width is w2, and the height is h2. FIG. i-2 in FIG. 12(i) is an example of the mask region after reduction. The mask region 1226 after reduction is corrected to a width of w2 - dw and a height of h2 - dh. By setting dw within the range of 0 < dw < w2 and dh as a value within the range of 0 < dh < h2, the mask region 1226 is prevented from disappearing. Also, the values of dw and dh may be randomly determined as long as they are within the range. Although an example of reducing the regions in the downward and rightward directions with respect to the mask region 1216 before correction is shown, the regions in the upward or leftward direction may be reduced, and the combination of the directions of the regions to be reduced is not limited.

[0122] The mask region to be corrected in S1101 may be the mask regions included in all the cells in the specification 501, or may be some of the cells in the specification 501. Also, in S1101, each of the above-described correction methods may be executed multiple times, or a combination of multiple correction methods may be executed. Further, correction processing may be applied according to the description seen in the actual specification. For example, since the character sizes in each column of the specification are often the same between cells, for example, the same correction processing may be performed for all the cells in the same column.

[0123] As described above, according to this embodiment, by performing correction processing on the mask region to generate a new mask image, it is possible to increase the number of learning data sets while reducing the labor of collecting learning document images.

[0124] <Other Embodiments> The present disclosure can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0125] The disclosure of the above-described embodiment includes the following configurations.

[0126] (Configuration 1) an acquisition means for acquiring a learning document image and information about a table structure included in the learning document image; a generation means for detecting an object region from the learning document image and generating a mask image in which the object region in the learning document image is masked; a learning means for generating a trained model that outputs information about the table structure included in a document image by performing training using the mask image and data representing the position of the table structure based on the information as a training dataset; and An information processing device comprising:

[0127] (Configuration 2) The table structure is made up of a plurality of cells, The data representing the positions of the table structure is data representing the positions of the plurality of cells. 2. The information processing device according to configuration 1,

[0128] (Configuration 3) The data representing the position of the table structure is a binary image in which a first value is held in pixels in an area corresponding to a cell in the learning document image, and a second value is held in other pixels. 3. The information processing device according to configuration 2.

[0129] (Configuration 4) The object is a character 4. The information processing device according to any one of configurations 1 to 3.

[0130] (Configuration 5) The learning means The mask image is used as input data, and the data representing the position of the table structure is used as a correct label to perform learning. 5. The information processing device according to any one of configurations 1 to 4.

[0131] (Configuration 6) The generating means masks the area of the object with a color different from the background color of the learning document image. 6. The information processing device according to any one of configurations 1 to 5.

[0132] (Configuration 7) further comprising a second generating means for performing a predetermined process on the mask image to generate a new mask image; The learning means The new mask image and the data representing the position of the table structure are further used as a training data set to perform training. 7. The information processing device according to any one of configurations 1 to 6.

[0133] (Configuration 8) The predetermined process includes at least one of a process of expanding a masked region, a process of reducing a masked region, a process of dividing a masked region, a process of deleting a masked region, and a process of adding a new masked region. 8. The information processing device according to configuration 7.

[0134] (Configuration 9) A first acquisition means for acquiring the trained model generated by the information processing device according to any one of configurations 1 to 8; a second acquiring means for acquiring a document image including a table structure; a detection means for detecting an object area from the document image; a generating means for generating a mask image in which the region of the object in the document image is masked; an identification means for identifying the position of a table structure included in the document image based on an output result obtained by inputting the mask image into the trained model; 1. An image processing device comprising:

[0135] (Configuration 10) character recognition means for recognizing character strings included in the document image; an extraction means for extracting character strings included in the specified table structure from the recognized character strings; 10. The image processing device according to configuration 9, further comprising:

[0136] (Configuration 11) The apparatus further includes a display control means for displaying a screen on which the extracted character string is presented to a user. 11. The image processing device according to configuration 10, further comprising:

[0137] (Configuration 12) The trained model outputs as the output result a binary image in which a first value is held in pixels in an area corresponding to cells constituting a table structure in the document image and a second value is held in other pixels. 12. The image processing device according to any one of configurations 9 to 11,

[0138] (Configuration 13) an acquiring step of acquiring a learning document image and information about a table structure included in the learning document image; a generating step of detecting an object region from the learning document image and generating a mask image in which the object region in the learning document image is masked; a learning step of generating a trained model that outputs information about the table structure included in a document image by performing training using the mask image and data representing the position of the table structure based on the information as a training dataset; A method for generating a trained model, comprising:

[0139] (Configuration 14) A program for causing a computer to execute each means of the information processing device according to any one of configurations 1 to 8. [Explanation of symbols]

[0140] 120 Learning Device 122 Learning data generation unit 123 Learning Department

Claims

1. an acquisition means for acquiring a learning document image and information about a table structure included in the learning document image; a generation means for generating a mask image in which an object region in the learning document image is masked; a learning means for generating a trained model that outputs information about the table structure included in a document image by performing training using the mask image and data representing the position of the table structure based on the information as a training data set; An information processing device comprising:

2. The table structure is made up of a plurality of cells, The data representing the positions of the table structure is data representing the positions of the plurality of cells.

2. The information processing apparatus according to claim 1, wherein:

3. The data representing the position of the table structure is a binary image in which a first value is held in pixels in an area corresponding to a cell in the learning document image, and a second value is held in other pixels.

3. The information processing apparatus according to claim 2, wherein:

4. The object is a character 2. The information processing apparatus according to claim 1, wherein:

5. The learning means The mask image is used as input data, and the data representing the position of the table structure is used as a correct label to perform learning.

2. The information processing apparatus according to claim 1, wherein:

6. The generating means masks the area of the object with a color different from the background color of the learning document image.

2. The information processing apparatus according to claim 1, wherein:

7. a second generating means for performing a predetermined process on the mask image to generate a new mask image; The learning means The new mask image and the data representing the position of the table structure are further used as a training data set to perform training.

2. The information processing apparatus according to claim 1, wherein:

8. The predetermined processing includes at least one of a processing for expanding a masked region, a processing for reducing a masked region, a processing for dividing a masked region, a processing for deleting a masked region, and a processing for adding a new masked region.

8. The information processing apparatus according to claim 7,

9. A first acquisition means for acquiring the trained model generated by the information processing device according to any one of claims 1 to 8; a second acquiring means for acquiring a document image including a table structure; a detection means for detecting an object area from the document image; a generating means for generating a mask image in which the region of the object in the document image is masked; an identification means for identifying the position of a table structure included in the document image based on an output result obtained by inputting the mask image into the trained model; 1. An image processing device comprising:

10. character recognition means for recognizing character strings included in the document image; an extraction means for extracting character strings included in the specified table structure from the recognized character strings; 10. The image processing device according to claim 9, further comprising:

11. The apparatus further includes a display control means for displaying a screen on which the extracted character string is presented to a user.

11. The image processing device according to claim 10, further comprising:

12. The trained model outputs, as the output result, a binary image in which a first value is held in pixels in an area corresponding to cells constituting a table structure in the document image and a second value is held in other pixels.

10. The image processing device according to claim 9,

13. an acquiring step of acquiring a learning document image and information about a table structure included in the learning document image; a generating step of generating a mask image in which an object region in the learning document image is masked; a learning step of generating a trained model that outputs information about the table structure included in a document image by performing training using the mask image and data representing the position of the table structure based on the information as a training dataset; A method for generating a trained model, comprising:

14. A program for causing a computer to execute each means of the information processing apparatus according to any one of claims 1 to 8.

Images