Information processing device, extraction processing device, image processing system, control method for information processing device, and program

Abstract

To provide an information processing apparatus configured so that learning data with various noises can be generated, and robust learning against noises can be performed, as a result, the influence of noises caused by scanning or the like can be sufficiently learned, and the deterioration of extraction accuracy of handwritten characters caused by the influence of noises can be suppressed, an extraction processing apparatus, an image processing system, a control method of the information processing apparatus, and a program.SOLUTION: A learning apparatus synthesizes a handwriting image (a foreground image obtained by randomly cut out in S501 to S505) with a noise image (a noise image randomly cut out in S507 to S508) to generate a synthesized image (S509), generates a correct label indicative of handwriting pixels in the handwriting image (S506), and associates the synthesized image with the correct label, as learning data for learning of a model which estimates handwriting pixels from an image (S510).SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an extraction processing device, an image processing system, a control method for an information processing device, and a program. [Background technology]

[0002] In recent years, the spread of computers has brought about changes in the working environment, and the digitization of business documents is progressing. This digitization is now extending to documents containing handwritten characters, and technology for extracting and digitizing handwritten characters is being studied.

[0003] Patent Document 1 discloses a technique for extracting handwritten areas during insurance document inspection and varying the number of manual checks depending on whether the character recognition of the contents is possible. This technique makes it possible to extract handwritten characters with information value written in handwritten areas of printed documents. For tasks such as extracting specific objects (handwritten characters in Patent Document 1) from images, a technique has been proposed for processing images to enhance learning data so that the shape of the object to be extracted and the type of background can be learned under various conditions (Patent Document 2).

[0004] In Patent Document 2, efficient learning is achieved by extracting the object to be recognized from an image, performing image transformations such as rotation and scaling, and pasting the resulting image onto another background image to create learning data. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2010-122791 [Patent Document 2] US-B-010546217 [Non-patent literature]

[0006] [Non-Patent Document 1] Long, et al. “Fully Convolutional Networks for Semantic Segmentation”, The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 3431-3440 Summary of the Invention [Problem to be solved by the invention]

[0007] When documents are digitized using an image processing device such as an MFP (Multifunction Peripheral), noise occurs in the image with each scan (dirt on the document table, streaks, dust, etc.). This noise is unique to devices with scanning functions and will be referred to as "scan noise" hereafter. Since this type of scan noise occurs in a random position with each scan, when creating training data, it is preferable to generate images so that the noise occurs in as many different positions as possible.

[0008] However, when training data is created using a method of enhancing images using scanned images as in Patent Document 2, the position at which scan noise appears is fixed, making it difficult to perform training that is robust against the effects of noise. For this reason, when training is performed using training data created as in Patent Document 2, it is not possible to fully learn about the effects of noise caused by scanning and the like, and there is a possibility that the effects of noise may reduce the accuracy of handwritten character extraction.

[0009] The present invention has been made to solve the above-mentioned problems, and an object of the present invention is to provide a mechanism that can generate training data for various noises and enable robust learning against noise. [Means for solving the problem]

[0010] The present invention includes: a first acquisition means for acquiring a handwritten image; Randomly cut out from an image generated by scanning a blank documentThe image processing system is characterized by comprising: a second acquisition means for acquiring a noise image; a synthesis means for generating a synthetic image by synthesizing the handwritten image and the noise image; a correct label generation means for generating correct labels indicating handwritten pixels in the handwritten image; and a data generation means for generating training data for generating a trained model that estimates handwritten pixels from an image based on the synthetic image and the correct labels. [Effects of the Invention]

[0011] According to the present invention, it is possible to create training data for various noises, enabling robust training against noise. As a result, it is possible to sufficiently learn the influence of noise caused by scanning and the like, and to suppress a decrease in the accuracy of handwritten character extraction due to the influence of noise. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of an image processing system according to an embodiment of the present invention. [Figure 2] FIG. 1 is a hardware configuration diagram of an image processing device, a learning device, and an image processing server. [Figure 3] FIG. 2 is a block diagram showing an outline of the logical configuration of a learning device. [Figure 4] 4 is a flowchart illustrating a learning process in the first embodiment. [Figure 5] 4 is a flowchart illustrating a learning data generation process in the first embodiment. [Figure 6] FIG. 3 is a diagram showing an example of a foreground original image in the first embodiment. [Figure 7] FIG. 3 is a diagram showing an example of a noise source image in the first embodiment. [Figure 8A] 5A to 5C are diagrams illustrating the compositing of a foreground image and a noise image in the first embodiment. [Figure 8B] 5A to 5C are diagrams illustrating the compositing of a foreground image and a noise image in the first embodiment. [Figure 9] 6 is a flowchart illustrating handwriting extraction request processing and handwriting extraction processing in the first embodiment. [Figure 10] 4A to 4C are diagrams illustrating an example of a processing target image and a result of estimating handwritten pixels from the processing target image in the first embodiment. [Figure 11] 3A to 3C are diagrams illustrating an example of a handwritten extraction image and a background extraction image according to the first embodiment. [Figure 12] 10A and 10B are diagrams illustrating the compositing of a background original image and a learning image in the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Below, we will explain a method for extracting handwritten characters from handwritten forms using a neural network that performs handwriting extraction and is trained using training data generated including noise, and for character recognition of the written content and saving it in a database, etc.

[0014] [First embodiment] FIG. 1 is a diagram showing an example of the configuration of an image processing system according to an embodiment of the present invention. The image processing system 100 of this embodiment includes an image processing device 101, a learning device 102, and an image processing server 103. The image processing device 101, the learning device 102, and the image processing server 103 are connected via a network 104.

[0015] The image processing device 101 is a digital multifunction peripheral or the like, and has a scanning function and a printing function. The image processing device 101 uses a scanning function to scan a blank document with only handwritten text and generate image data (hereinafter, this image will be referred to as a "foreground original image"). The image processing device 101 scans multiple documents in this way and generates multiple foreground original images. Furthermore, the image processing device 101 generates image data containing noise generated by scanning, for example, a blank document (hereinafter, this image is referred to as a "noise-source image"). Then, the image processing device 101 transmits the foreground original image and the noise original image generated as described above to the learning device 102 via the network 104 .

[0016] The learning device 102 generates training data for training a neural network that performs handwriting extraction from the foreground original image and noise original image generated by the image processing device 101. The learning device 102 then uses the generated training data to train the neural network and generate training results (neural network parameters, etc.). The learning device 102 transmits the training results to the image processing server 103 via the network 104.

[0017] Furthermore, when extracting handwriting, the image processing device 101 scans a document containing handwriting to obtain scanned image data to be processed (hereinafter, this scanned image data will be referred to as a "processing target image"). Then, the image processing device 101 transmits the processing target image to the image processing server 103 obtained as described above via the network 104.

[0018] The image processing server 103 extracts handwriting from the processing target image generated by the image processing device 101. Using the learning results generated by the learning device 102, the image processing device 101 performs inference using a neural network to extract handwritten pixels from the processing target image, and generates an image (image of handwritten portion, image of printed portion) in which the handwritten portion and the printed portion (background portion) are separated. The image processing server 103 may be configured to transmit the images (image of the handwritten portion and image of the typed portion) generated as described above to a handwriting OCR server (not shown) and a typed OCR server (not shown), respectively, and receive the OCR processing results. Note that OCR is an abbreviation for Optical Character Recognition.

[0019] The hardware configuration of the image processing device 101, the learning device 102, and the image processing server 103 will be described below with reference to FIG. FIG. 2A is a block diagram showing an example of the hardware configuration of the image processing device 101. As shown in FIG. The image processing device 101 includes a CPU 201, a ROM 202, a RAM 204, a printer device 205, a scanner device 206, a document transport device 207, a storage 208, an input device 209, a display device 210, and an external interface 211. These devices are connected via a data bus 203 so as to be able to communicate with each other.

[0020] The CPU 201 is a controller for overall control of the image processing device 101. The CPU 201 starts an operating system (OS) stored in a storage 208 using a boot program stored in a ROM 202. A controller program stored in the storage 208 is executed on this OS. The controller program is a program for controlling the image processing device 101. The CPU 201 overall controls each device connected via a data bus 203. The RAM 204 operates as a temporary storage area such as the main memory or work area of ​​the CPU 201.

[0021] The printer device 205 prints image data onto paper (recording material). This can be an electrophotographic printing method using a photosensitive drum or photosensitive belt, or an inkjet method that ejects ink from a minute nozzle array to print an image directly onto paper, but any method is acceptable.

[0022] The scanner device 206 scans an original such as paper using an optical reader such as a CCD (Charge Coupled Device), obtains electrical signal data, converts it, and generates scanned image data. The original transport device 207, such as an ADF (Auto Document Feeder), transports originals placed on a platen on the original transport device 207 to the scanner device 206 one sheet at a time.

[0023] The storage 208 is a readable / writable non-volatile memory such as a hard disk drive (HDD) or a solid state drive (SSD), in which various data such as the controller program described above is recorded. The input device 209 is an input device configured with a touch panel, hard keys, etc. The input device 209 accepts an operation instruction from the user and transmits instruction information including an instruction position to the CPU 201.

[0024] The display device 210 is a display device such as an LCD or a CRT, and displays the display data generated by the CPU 201. The CPU 201 determines which operation has been performed based on the instruction information received from the input device 209 and the display data being displayed on the display device 210. In response to this determination result, the CPU 201 controls the image processing device 101 and generates new display data to display on the display device 210.

[0025] The external interface 211 transmits and receives various data, including image data, to and from external devices via a network such as a LAN, a telephone line, or near-field wireless communication such as infrared. The external interface 211 receives PDL data from external devices such as the learning device 102 or a PC (not shown). The CPU 201 interprets the PDL data received by the external interface 211 and generates an image. The generated image is printed by the printer device 205 or stored in the storage 208. The external interface 211 also receives image data from external devices such as the image processing server 103. The CPU 201 prints the received image data using the printer device 205, stores it in the storage 208, or transmits it to other external devices via the external interface 211. The external interface 211 also transmits image data generated by the scanner device 206 and image data stored in the storage 208 to external devices such as the learning device 102 and the image processing server 103.

[0026] FIG. 2B is a block diagram showing an example of the hardware configuration of the learning device 102. The learning device 102 includes a CPU 231, a ROM 232, a RAM 234, a storage 235, an input device 236, a display device 237, an external interface 238, and a GPU 239. These components can send and receive data to and from each other via a data bus 233.

[0027] The CPU 231 is a controller for controlling the entire learning device 102. The CPU 231 starts the OS stored in the storage 235 using a boot program stored in the ROM 232, which is a non-volatile memory. The OS runs a learning data generation program and a learning program stored in the storage 235. The CPU 231 generates learning data by executing the learning data generation program. The CPU 231 also runs the learning program to train a neural network that extracts handwriting. The CPU 231 controls each unit via buses such as the data bus 233.

[0028] The RAM 234 operates as a temporary storage area such as a main memory or work area for the CPU 231 . The storage 235 is a non-volatile memory such as an HDD or SSD that can be read and written, and stores various data such as the learning data generation program and learning program described above.

[0029] The input device 236 is an input device configured with a mouse, keyboard, etc. The display device 237 is the same as the display device 210 described with reference to FIG. The external interface 238 is similar to the external interface 211 described with reference to FIG. 2(a). The GPU 239 is an image processing processor that cooperates with the CPU 231 to generate image data and perform neural network training.

[0030] FIG. 2C is a block diagram showing an example of the hardware configuration of the image processing server 103. As shown in FIG. The image processing server 103 includes a CPU 261, a ROM 262, a RAM 264, a storage 265, an input device 266, a display device 267, and an external interface 268. These components can transmit and receive data to and from each other via a data bus 263.

[0031] The CPU 261 is a controller for controlling the entire image processing server 103. The CPU 261 starts up the OS stored in the storage 265 using a boot program stored in the ROM 262, which is a non-volatile memory. On this OS, the image processing server program stored in the storage 265 is executed. By executing this image processing server program, the CPU 261 extracts and erases handwritten pixels from the image to be processed. The CPU 261 controls each unit via buses such as the data bus 263.

[0032] RAM 264 serves as a temporary storage area for the CPU 261's main memory and work area. It works. The storage 265 is a non-volatile memory such as an HDD or SSD that can be read and written, and stores various data such as the image processing program described above.

[0033] The input device 266 is similar to the input device 236 described with reference to Fig. 2(b). The display device 267 is similar to the display device 210 described with reference to Fig. 2(a). The external interface 268 is similar to the external interface 211 described with reference to FIG. 2(a).

[0034] The learning device 102 and the image processing server 103 are information processing devices, and may be realized by a single computer or multiple computers. For example, the learning device 102 and the image processing server 103 may be configured to be realized using cloud computing technology. The learning device 102 and the image processing server 103 may be configured to be realized by a cloud service or the like.

[0035] FIG. 3 is a block diagram showing an outline of the logical configuration of the learning device 102. The training data generation unit 301 generates training data for training a neural network. The training data generation unit 301 is realized by the CPU 231 loading a training data generation program into the RAM 234 and executing it.

[0036] The learning unit 302 learns the neural network using the learning data generated by the learning data generation unit 301. The learning unit 302 is realized by the CPU 231 loading a learning program into the RAM 234 and executing it. The CPU 231 may be configured to execute part of the calculation processing executed by the learning data unit 301 and the learning unit 302 in cooperation with the GPU 239 .

[0037] Next, the learning process performed by the learning device 102 will be described. 4 is a flowchart illustrating the learning process in the first embodiment. This process is implemented by the learning unit 302 of the learning device 102, etc. That is, the CPU 231 loads a learning program, etc. into the RAM 234 and executes it. This learning process is started when the user performs a predetermined operation via the input device 209 of the image processing device 101. In this embodiment, the mini-batch method is used for learning the neural network, but this is not limitative. The learning process may be started not only by the above-mentioned means, but also by an input received from a PC (not shown) or the like via the network 104 as a trigger.

[0038] First, in S401, the CPU 231 initializes the neural network. That is, the CPU 231 constructs a neural network and randomly determines and initializes the values ​​of each parameter included in the neural network. The neural network to be constructed may have a variety of structures, but may take the form of, for example, FCN (Fully Convolutional Networks) as shown in Non-Patent Document 1.

[0039] Next, in S402, the CPU 231 acquires training data. The CPU 231 executes a training data generation process, which will be described later, to acquire a predetermined number of pieces of training data (mini-batch size, for example, "10"). Note that here, the CPU 231 may be configured to acquire training data that has been generated in advance by the training data generation unit 301 and stored in the storage 235 or the like.

[0040] Next, in S403 to S404, the CPU 231 performs learning of the neural network. First, in S403, the CPU 231 calculates the error of the neural network. First, the CPU 231 inputs an input image included in each learning data into the neural network to obtain an output. The output is an image of the same image size as the input image, and as a prediction result, pixels determined to be handwritten have pixel values ​​indicating handwriting, and pixels determined not to be handwritten have pixel values ​​indicating non-handwriting. The CPU 231 then evaluates the difference between the output and the correct labeled image to determine the error. Cross entropy can be used as an index for this evaluation.

[0041] Next, in S404, the CPU 231 adjusts the parameters of the neural network. That is, the CPU 231 changes the parameter values ​​of the neural network by the backpropagation method based on the error calculated in S403.

[0042] Next, in S405, CPU 231 determines whether or not to end the learning. More specifically, CPU 231 determines whether or not the processes of S402 to S404 above have been performed a predetermined number of times (for example, "60,000 times"). The predetermined number of times can be determined by a user's input operation at the start of this learning process. If the predetermined number of times has not yet been performed, CPU 231 determines not to end the process (determines NO in S405), transitions to S402, and continues learning of the neural network.

[0043] On the other hand, if the process has already been performed the predetermined number of times, the CPU 231 determines that the process is to be ended (YES in S405), and the CPU 231 shifts the process to S406. In S406, the CPU 231 transmits the neural network parameters adjusted in S404 to the image processing server 103 as the learning results, and ends the processing of this flowchart.

[0044] Next, the learning data generation process performed by the learning device 102 will be described. 5 is a flowchart illustrating the learning data generation process in the first embodiment. This process is implemented by the learning data generation unit 301 of the learning device 102. That is, the CPU 231 loads learning data generation into the RAM 234 and executes it.

[0045] First, in S501, the CPU 231 selects and reads out a foreground original image stored in the storage 235. This foreground original image will be described with reference to FIG. FIG. 6 is a diagram showing an example of a foreground original image. As shown in Fig. 6, the foreground original image is an image on which characters and the like are written only by hand. The foreground original image is generated, for example, by scanning a document on which handwritten characters 601 to 603 are written on a blank sheet of paper 600 using the image processing device 101, as shown in Fig. 6. The learning device 102 receives a plurality of foreground original images generated as described above from the image processing device 101 and stores them in the storage 235. In S501 above, one of the foreground original images is randomly selected.

[0046] Next, in S502, the CPU 231 rotates and processes the foreground original image read out in S501. The rotation angle is determined by randomly selecting from a predetermined range (for example, between -10 degrees and +10 degrees). Next, in S503, CPU 231 generates image data by cutting out a portion (for example, a size of "vertical × horizontal = 512 × 512") of the foreground original image rotated in S502 (hereinafter, this image data will be referred to as "foreground image"). The position of the cutout is determined randomly.

[0047] Next, in S504, CPU 231 scales and processes the foreground image generated in S503. The scale is determined by randomly selecting from a predetermined range (for example, between 50% and 150%). Furthermore, a portion of the scaled foreground image (for example, a size of 256 x 256) is cropped from the center or the like to update the foreground image.

[0048] Next, in S505, CPU 231 processes the foreground image by changing the luminance of each pixel. Specifically, CPU 231 converts the foreground image into grayscale and changes the luminance of the foreground image using gamma correction. The gamma value is randomly selected from a predetermined range (for example, between 0.1 and 10.0).

[0049] Next, in S506, the CPU 231 generates a correct label image for the foreground image (correct label generation process). First, the CPU 231 performs a binarization process on the foreground image. The CPU 231 sets pixel values ​​that are lower than a predetermined threshold to values ​​indicating handwriting (for example, the same applies to values ​​after "255") and other pixel values ​​to values ​​indicating non-handwriting (for example, the same applies to values ​​after "0"), generating image data as a correct label image for the foreground image. The binarization method is not limited to a method using a threshold, and any known binarization method can be applied.

[0050] Next, in S507, the CPU 231 selects and reads out the noise source image stored in the storage 235. This noise source image will be described with reference to FIG. FIG. 7 is a diagram showing an example of a noise source image. The noise source image is an image containing noise that appears in an image when a document or the like is scanned, and is generated, for example, by scanning a blank sheet of paper with the image processing device 101. The learning device 102 is assumed to have received the noise source image generated as described above from the image processing device 101 and stored it in the storage 235 in advance.

[0051] FIG. 7 shows an example in which noise 704 and noise 705 occur in a scanned image 702 obtained from a blank sheet of paper 701 via the scanning function of the image processing device 101 . Noise 704 represents, for example, scan noise that appears randomly during scanning, and noise 705 represents, for example, streak-like noise caused by dirt on the image processing device 101. In addition to these, noises include noise caused by the reading device (CCD, line sensor), compression noise caused by image compression performed during image processing, and the like.

[0052] Next, in S508, the CPU 231 generates image data by cutting out a portion of the noise original image acquired in S507 (for example, a size of "vertical × horizontal = 512 × 512") (hereinafter, this image data will be referred to as the "noise image"). The cutting position is determined randomly. Note that the pixel values ​​of the cut-out noise image may be evaluated, and a noise image that does not contain a certain amount of noise may not be adopted, but may be cut out from a different position. For example, if the luminance value of a pixel value is below a threshold value (for example, "200"), it is determined that noise has occurred in the pixel, and a determination is made for each pixel in the cut-out image. If the luminance value is below a predetermined percentage (for example, "10%" of the number of pixels), the cut-out noise image may not be adopted, and another noise image may be cut out again. Note that the noise determination process is not limited to this form, and other forms of evaluating the presence or absence of noise may also be used.

[0053] Next, in S509, CPU 231 generates an image by combining the foreground image acquired in S501 to S505 above with the noise image acquired in S507 to S508 above (hereinafter, this image will be referred to as the "composite image"). For example, CPU 231 combines the images by comparing the same coordinates of the foreground image and the noise image and creating a new image using pixel values ​​of the image with lower brightness. Note that the combining method is not limited to the above, and for example, pixel values ​​may be determined by blending pixels at the same coordinates in each image at a fixed ratio.

[0054] Next, in S510, the CPU 231 associates the composite image generated by composition in S509 with the correct labeled image generated in S506, and stores the result in a predetermined area of ​​the storage 235 as learning data.

[0055] Next, in S511, the CPU 231 determines whether or not to end the learning data generation process. If the predetermined number of pieces of learning data have not yet been generated, the CPU 231 determines that the process is not yet ended (NO in S511) and transitions the process to S501. On the other hand, if the predetermined number of pieces of learning data have already been generated, the CPU 231 determines to end the process (determines YES in S511) and ends the process of this flowchart.

[0056] The training data generated as described above will be described below with reference to FIGS. 8A and 8B. 8A and 8B are diagrams showing an example of a composite image obtained by combining a foreground image and a noise image in the first embodiment. Hereinafter, FIGS. 8A and 8B will be collectively referred to as FIG.

[0057] FIG. 8(a) shows a composite image of learning images in the first embodiment. Foreground image 801 is an example of a foreground image generated by S501 to S505 in Fig. 5. This is an image in which handwritten character 601 in original foreground image 600 shown in Fig. 6 is partially cut out and rotated. The noise image 802 is an example of a noise image generated by steps S507 and S508 in Fig. 5. This is an image generated by partially cutting out the noise 704 in the noise original image 702 illustrated in Fig. 7.

[0058] A composite image 803 is an example of a composite image generated in S508 of Fig. 5. This image is generated by comparing pixel values ​​at the same coordinates of the foreground image 801 and the noise image 802, for example. The correct labeled image 804 is an example of a correct labeled image generated in S506 of Fig. 5. The pixels are configured to represent the handwritten character portion of the image 801, and the image is treated as training data to be paired with the composite image 803.

[0059] 8(b) to 8(d) show the composite image at the pixel level. 8(b), 8(c), and 8(d) are partial images cut out at the same coordinates (width: x1 to x2, height: y1 to y2) from a foreground image 801, a noise image 802, and a composite image 803, respectively. The rectangles represent the pixels that make up the images.

[0060] As described above, each pixel of the composite image 807 is determined based on the pixel values ​​at the same coordinates in the foreground image 805 and noise image 806 to be composited. For pixel 810 in the composite image 807, the luminance values ​​of pixel 808 in the foreground image 805 and pixel 809 in the noise image 806, which have the same coordinates, are compared, and the luminance value of the pixel with the lower luminance value is adopted. For pixel 810, the luminance value of pixel 808 in the foreground image 805 is determined to be lower, and is adopted as the luminance value of pixel 810. On the other hand, for pixel 813, pixel 811 in the foreground image 805 and pixel 812 in the noise image 806 are compared, and pixel 812 in the noise image 806, which is determined to have the lower luminance value, is adopted as pixel 813 in the composite image 807.

[0061] As described above, in this embodiment, a handwritten character image (foreground image) and a noise image are superimposed to generate a composite image, and the composite result is combined with label information of the handwritten character (correct label image) to generate training data. This makes it possible to create training data for various noises, enabling robust learning against noise.

[0062] Next, a case will be described in which image data is read from a document using the scanning function of the image processing device 101, and handwritten characters are extracted by the image processing server 103. First, the image processing device 101 scans an original document containing printed and handwritten characters to obtain an image to be processed. Then, the image to be processed is sent to the image processing server 103 to request extraction of handwritten characters. The following description will be given with reference to FIG. 9(a).

[0063] 9(a) is a flowchart showing an example of handwriting extraction request processing by the image processing device 101. This processing is realized by the CPU 201 of the image processing device 101 reading out a controller program recorded in the storage 208, expanding it into the RAM 204, and executing it. This processing is started when the user performs a predetermined operation via the input device 209 of the image processing device 101.

[0064] First, in S901, the CPU 201 controls the scanner device 206 and the document transport device 207 to scan a document and generate an image to be processed. The image to be processed is generated as full-color (RGB three-channel) image data. The document to be scanned here is, for example, a form such as an order slip as shown in FIG. 10(a), with values ​​written by hand to the right of each item. Next, in S902, the CPU 201 transmits the handwriting extraction processing target generated in S901 to the image processing server 103 via the external interface 211.

[0065] Next, the handwriting extraction process performed by the image processing server 103 will be described. The image processing server 103 receives the image to be processed from the image processing device 101, and obtains image data by extracting handwritten characters contained in the scanned image data. Hereinafter, this will be described with reference to FIG. 9(b).

[0066] 9(b) is a flowchart showing an example of handwriting extraction processing by the image processing server 103. This processing is realized by the CPU 261 reading out the image processing server program stored in the storage 265, expanding it in the RAM 264, and executing it. This processing is started, for example, when the user turns on the power of the image processing server 103.

[0067] First, in S911, the CPU 261 loads a neural network for extracting handwriting. First, the CPU 261 constructs the same neural network as in S401 of Fig. 4. Then, in S406 of Fig. 4, the learning results (neural network parameters) transmitted from the learning device 102 are reflected in the constructed neural network. This generates a trained model for estimating handwritten pixels from an image. It is assumed that the image processing server 103 has previously received the learning results transmitted from the learning device 102 in S406 of FIG.

[0068] Next, in S912, the CPU 261 determines whether or not the processing target image to be processed has been received and acquired from the image processing device 101. If the processing target image has been acquired via the external interface 268 (YES in S912), the CPU 261 proceeds to S913. On the other hand, if not (NO in S912), the CPU 261 proceeds to S917.

[0069] In S913, the CPU 261 extracts handwritten pixels from the processing target image received from the image processing device 101. First, the CPU 261 grayscales the processing target image. Then, this grayscaled processing target image is input to the neural network constructed in S911 above to estimate handwritten pixels. That is, the handwritten pixels are estimated using the trained model generated in S911 above. As a result, image data is obtained that is the same size as the processing target image and in which a value indicating handwritten (for example, "1") is recorded for pixels determined to be handwritten, and a value indicating not handwritten (for example, "0") is recorded for pixels determined not to be handwritten. The following description will be made with reference to FIG. 10. The image in Figure 10(a) is the "image to be processed," and the image resulting from estimating handwritten pixels is shown in Figure 10(b). In Figure 10, to make the results easier to see, pixels with a value of "0" are shown in black, and pixels with a value of "1" are shown in white. Hereafter, this image data will be referred to as the "estimation result."

[0070] Next, in S914, the CPU 261 uses the estimation result created in S913 as a mask to extract the portion corresponding to the handwriting from the image to be processed, and creates an image of only the handwriting. Specifically, an image of the same size as the image to be processed is created by substituting the pixel values ​​of the image to be processed for coordinates (pixels) where the estimation result shows handwriting, and "255" for coordinates (pixels) where the estimation result shows no handwriting. Hereinafter, this image will be referred to as the "handwriting extracted image." An example of the handwriting extracted image created here is shown in Figure 11(a).

[0071] Next, in S915, the CPU 261 uses the estimation result created in S913 as a mask to extract a portion corresponding to the background from the image to be processed, and creates an image of only the background. Specifically, an image of the same size as the image to be processed is created in which the pixel values ​​of the image to be processed are assigned to coordinates (pixels) where the estimation result is not handwritten, and "255" is assigned to coordinates (pixels) where the estimation result is handwritten. Hereinafter, this image will be referred to as the "background extracted image." An example of the background extracted image created here is shown in Figure 11(b).

[0072] Next, in S916, the CPU 261 outputs the handwritten extraction image and the background extraction image. Specifically, the CPU 261 transmits the handwritten extraction image obtained in S914 and the background extraction image obtained in S915 to the image processing device 101 via the external interface 268.

[0073] Next, in S917, the CPU 261 determines whether or not to end the process. If the user performs a predetermined operation such as turning off the power of the image processing server 103, the CPU 261 determines to end the process (determines YES in S917) and ends the process of this flowchart. Otherwise, the CPU 261 determines not to end the process (determines NO in S917) and transitions the process to S912.

[0074] In S916 above, the handwriting extraction result is sent to the image processing device 101, but other processing may be performed. For example, the handwriting extraction image and background extraction image may be sent to a handwriting OCR server (not shown) and a type OCR server (not shown) via the external interface 268, respectively, to be used as input for each OCR, or may be sent to a DB server (not shown) and stored therein. In this way, the output destination of the handwriting extraction result is not limited.

[0075] As described above, image data containing only handwritten characters and image data containing only noise are used to generate training data for training a neural network that extracts handwriting by combining the handwritten characters and noise so that they overlap.The neural network is then trained using the generated training data, and the training results can be used to extract handwritten character regions from scanned images.In this case, by combining a noise image with the training data for training, it is possible to obtain accurate extraction results regardless of the influence of noise generated during scanning.

[0076] In this embodiment, the training data generation unit 301 and the training unit 302 are implemented in the training device 102, but they may be implemented in separate devices. In this case, the device implementing the training data generation unit 301 transmits the training data generated by the training data generation unit 301 to the device implementing the training unit 302. Then, the training unit 302 trains the neural network based on the received training data. Furthermore, the image processing device 101 and the image processing server 103 have been described as separate devices, but the present invention is not limited to this configuration and multiple functions may be integrated into the same device.

[0077] In this embodiment, the training data generation process is called to generate training data in the training process. However, a large amount of training data may be generated in advance by the training data generation process, and mini-batch size samples may be taken from the training data as needed during the training process.

[0078] Alternatively, a foreground original image may be generated by scanning a document with only a seal (a document with only a seal impression on a blank sheet of paper), or by scanning the back side of a thin sheet of paper on which an electronic document has been printed. This allows for the synthesis and generation of training data for training a neural network that detects pixels corresponding to a seal or show-through.

[0079] The training data may also include the foreground image and the corresponding correct labeled image obtained in S505 or S506 of Fig. 5. The training data may also include the noise image obtained in S508 of Fig. 5 and the corresponding correct labeled image, which has the same image size as the noise image and in which all pixels have values ​​indicating that the image is not handwritten.

[0080] In this embodiment, the input image is generated as a grayscale image, but it may be generated as a full-color image or other format. In this embodiment, image processing such as rotation is not performed on the noise image, but a method may be used to generate an image that includes image processing (S502, S504, S505, etc. in FIG. 5) in the same way as the foreground image.

[0081] Second Embodiment In the first embodiment, a noise image is synthesized with a handwritten foreground image. However, it is also possible to synthesize the foreground image with an image containing background information such as type and lines (hereinafter referred to as the "original background image"), and then synthesize the noise image with the synthesized image. This embodiment will be described below with reference to FIG. 12.

[0082] FIG. 12(a) is a diagram illustrating an example of a background original image in the second embodiment. A background original image 1201 shows the format of a form. In the second embodiment, as shown in Fig. 12(a), an image including type 1202 representing input items of a form, such as "furigana" and ruled lines 1203, is used as background information for synthesis.

[0083] FIG. 12(b) shows a synthesized image of the learning images in the second embodiment, and shows a synthesis example in which a background image is added to the example in the first embodiment using a foreground image and a noise image. In the example of FIG. 12(b), first, foreground image 1211 and background image 1212 are composited to generate composite image 1213. The background image 1212 is obtained by performing the same processes as in S501 to S505 of FIG. 5 in the first embodiment on an original background image (e.g., original background image 1201) and partially cropping it. Then, a noise image 1214 is further composited with the generated composite image 1213 to generate final training data image 1215. The method of generating the foreground image and noise image is the same as in the first embodiment.

[0084] In the training data generation process of each of the above embodiments, when a noise image is combined with a foreground image or a composite image obtained by combining a foreground image and a background image, a certain percentage of images may be provided without noise images. This makes it possible to include training data with little influence of noise.

[0085] In the above embodiments, a model using a neural network is exemplified as a model for estimating handwritten pixels from an image, but the present invention is not limited to this.

[0086] In the above embodiments, a configuration has been described in which a composite image is generated using a handwritten character image and an image containing only noise, and the composite image is then combined with label information for the handwritten characters to perform learning. This makes it possible to create learning data for various noises, enabling robust learning against noise. As a result, it is possible to fully learn the effects of noise caused by scanning and other factors, and to suppress a decrease in the accuracy of handwritten character extraction due to the effects of noise.

[0087] It goes without saying that the configurations and contents of the various data described above are not limited to those described above, and that the data may be configured in various configurations and contents depending on the application and purpose. Although one embodiment has been described above, the present invention can be embodied as, for example, a system, an apparatus, a method, a program, a storage medium, etc. Specifically, the present invention may be applied to a system made up of multiple devices, or may be applied to an apparatus made up of a single device. Furthermore, the present invention also includes any combination of the above embodiments.

[0088] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. Furthermore, the present invention may be applied to a system made up of multiple devices, or to an apparatus made up of a single device. The present invention is not limited to the above-described embodiments, and various modifications (including organic combinations of the embodiments) are possible based on the spirit of the present invention, and these modifications are not excluded from the scope of the present invention. In other words, all configurations that combine the above-described embodiments and their modifications are included in the present invention. [Explanation of symbols]

[0089] 101 Image processing device 102 Learning Device 103 Image Processing Server

Claims

1. a first acquisition means for acquiring a handwritten image; a second acquisition means for acquiring a noise image randomly extracted from an image generated by scanning a blank document; a synthesis means for generating a synthesis image by synthesizing the handwritten image and the noise image; a correct label generating means for generating correct labels indicating handwritten pixels in the handwritten image; a data generation means for generating training data for generating a trained model that estimates handwritten pixels from an image based on the synthesized image and the correct answer label; An information processing device comprising:

2. 2. The information processing apparatus according to claim 1, wherein the first acquisition means acquires, as the handwritten image, an image randomly cut out from an image generated by reading a handwritten document.

3. 3. The information processing apparatus according to claim 1, wherein pixel values ​​of the composite image generated by the combining means are determined by comparing pixel values ​​of corresponding pixels of the handwritten image and the noise image.

4. 4. The information processing apparatus according to claim 1, wherein the correct label generating means generates the correct label by binarizing the handwritten image.

5. 5. The information processing device according to claim 1, wherein the synthesis means generates the synthesized image by synthesizing the handwritten image, an image including background information, and the noise image.

6. The information processing device according to any one of claims 1 to 5, further comprising: a model generation means for generating the trained model that estimates handwritten pixels from an image using the training data generated by the data generation means.

7. 6. An extraction processing device comprising: an estimation means for estimating handwritten pixels from an image to be processed using the trained model generated using the training data generated by the data generation means of the information processing device according to any one of claims 1 to 5.

8. 8. The extraction processing device according to claim 7, further comprising a creating means for creating an image in which a portion corresponding to handwriting is extracted from the image to be processed and an image in which a portion corresponding to a background is extracted from the image to be processed, based on the result of estimation by the estimating means.

9. 9. An image processing system comprising: an information processing device according to claim 1; and an extraction processing device according to claim 7.

10. a first acquisition step of acquiring a handwritten image; a second acquisition step of acquiring a noise image randomly extracted from an image generated by scanning a blank document; a combining step of combining the handwritten image and the noise image to generate a combined image; a correct label generation step of generating correct labels indicating handwritten pixels in the handwritten image; a data generation step of generating training data for generating a trained model that estimates handwritten pixels from an image based on the synthesized image and the correct answer label; 1. A method for controlling an information processing device, comprising:

11. A program for causing a computer to function as each of the means according to any one of claims 1 to 6.

Images