Image processing device, image processing method and storage medium

Abstract

The present disclosure relates to an image processing device, an image processing method and a storage medium. A first decoder obtains an area including a target recognition object based on a feature map obtained by performing convolution processing on a processing target image. Next, the first decoder obtains a partial feature map corresponding to a portion of the obtained area including the target recognition object from the feature map. Then, a second decoder obtains pixels corresponding to the target recognition object based on the partial feature map. This reduces the amount of calculation required by a decoder unit included in a neural network.

Description

Technical Field

[0001] Aspects of the present disclosure generally relate to an image processing system, an image processing method, and a storage medium, each for obtaining pixels of an object using a neural network. Background Art

[0002] Recently, an operation is performed in which a paper document such as a business form having characters written therein by a user is scanned and a handwritten optical character recognition (OCR) process is performed on an image obtained by such scanning to computerize (convert into digital data) the handwritten characters.

[0003] It is desirable that a partial image representing handwritten characters is separated from a scanned image obtained by scanning a business form having the handwritten characters and input to a handwriting OCR engine for handwriting OCR. The reason for this is that since the handwriting OCR engine is optimized for recognizing handwritten characters, if a character image printed with a specific character font such as a type font or a graphic such as an icon is printed therein is included in the image input to the handwriting OCR engine, the recognition accuracy of the handwriting OCR engine may be reduced.

[0004] Therefore, it is conceivable that the handwriting OCR engine processes an image obtained by extracting only the pixels of the handwritten characters. Fig.22A An example of an image of a business form with a handwritten full name on it is shown, and Fig. 22B Shown by Fig.22A The image shown in FIG. 1 is an image obtained by extracting pixels of handwritten characters (that is, an image obtained by converting pixels of ruled lines and typeface characters into white pixels). Fig.22A In the image shown in , the presence of the ruled lines makes it possible to see that the handwriting is "ALF" and "REDFORD". However, if the handwriting OCR engine has Fig. 22B In the example shown in FIG. 1 , the recognition process is performed on the image obtained by extracting only the pixels of the handwritten characters, and the delimiting position of such a handwritten character string is unclear, and the handwritten character string may be recognized as "ALFRED" and "FORD". It is desirable that the partial images obtained by dividing the respective areas of the fill-in column are input to the handwriting OCR engine respectively.

[0005] On the other hand, recently, in the field of image recognition, neural networks have attracted attention. For example, a neural network is known that is used to classify pixels included in an image into predetermined categories (classes) and is configured with an encoder and a decoder. The neural network causes the encoder to perform convolution calculations on an input image and output a feature map to distinguish the categories of each pixel of the image. Then, the neural network causes the decoder to perform deconvolution calculations on the feature map output from the encoder and output an image (i.e., a probability map) composed of probability values, each of which indicates the category to which each pixel of the input image belongs.

[0006] It is proposed to apply a neural network for performing category classification to the following tasks: separating a scanned image into an image of handwritten characters and an image other than the image of handwritten characters, and extracting pixels of handwritten characters from the input image. "Recognizing Challenging Handwritten Annotations with Fully Convolutional Networks", 16th International Conference on Frontiers in Handwriting Recognition 2018 discussed performing category classification to distinguish whether each pixel included in the input image is a pixel constituting a handwritten character. In addition, in the following description, the process for classifying and extracting pixels constituting handwritten characters is referred to as "handwriting extraction", pixels extracted by handwriting extraction are referred to as "handwriting extraction pixels", and images obtained by handwriting extraction are referred to as "handwriting extraction images".

[0007] In addition, "Handwritten Text Line Segmentation Using Fully Convolutional Network", 2017 14th IAPR International Conference on Document Analysis and Recognition discusses applying a neural network for performing category classification to the following task: classifying an area (fill-in field area) in which characters are written included in a business form image, thereby creating a neural network for estimating and outputting an area including handwritten fill-in on an input image. In addition, in the following description, the process for estimating an area including handwritten fill-in is referred to as "handwritten area estimation". In addition, the area obtained by handwritten area estimation is referred to as a "handwritten area".

[0008] In addition, on the other hand, it is well known that neural networks require a lot of calculations. In the case where the above two processing operations for handwriting area estimation and handwriting extraction (i.e., multiple tasks) are configured with multiple neural networks suitable for each processing operation, two neural networks are required, so the increase in calculation load is more obvious.

[0009] In addition, Japanese Patent Application Laid-Open No. 2019-177032 discusses a neural network that is capable of integrating parts of the processing operations of an encoder of a neural network that are common between multiple tasks, thereby reducing the amount of calculation.

[0010] According to the technology discussed in Japanese Patent Application Laid-Open No. 2019-177032, the amount of calculation can be reduced by sharing the calculation in the encoder unit between multiple neural networks. However, even with the technology discussed in Japanese Patent Application Laid-Open No. 2019-177032, the amount of calculation in the decoder unit is still large. Summary of the invention

[0011] According to one aspect of the present disclosure, an image processing device includes: at least one memory storing instructions; and at least one processor executing the instructions to perform the following operations: using a neural network to obtain a region including a target recognition object included in a processing target image, and using the neural network to obtain pixels corresponding to the target recognition object included in the processing target image, wherein the neural network includes: an encoder unit configured to obtain a feature map by performing convolution processing on the processing target image; a first decoder unit configured to obtain the region including the target recognition object based on the feature map; a partial feature map obtaining unit configured to obtain a partial feature map corresponding to a portion of the region including the target recognition object obtained by the first decoder unit from the feature map; and a second decoder unit configured to obtain pixels corresponding to the target recognition object based on the partial feature map.

[0012] Further features of the present disclosure will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a diagram showing the configuration of an image processing system.

[0014] Figure 2A is a diagram showing the configuration of an image processing apparatus. Figure 2B is a diagram showing the configuration of a learning device. Figure 2C is a diagram showing the configuration of an image processing server. Figure 2D is a diagram showing a configuration of an optical character recognition (OCR) server.

[0015] Figure 3Ais a diagram showing a learning sequence of an image processing system.

[0016] Figure 3B is a diagram showing a usage sequence of the image processing system.

[0017] Figure 4A , Figure 4B , Figure 4C and Figure 4D An example of a business form is shown.

[0018] Figure 5A is a diagram showing a learning document scanning screen. Figure 5B : is a diagram showing a handwriting extraction correct answer data creation screen. Figure 5C : is a diagram showing a handwriting area estimation correct answer data creation screen. Figure 5D is a diagram showing a business form processing screen.

[0019] Fig. 6A : is a flowchart showing the flow of document sample image generation processing. Figure 6B : is a flowchart showing the flow of document sample image receiving processing. Figure 6C : is a flowchart showing the flow of the correct answer data generation process.

[0020] Fig. 7A : is a flowchart showing the flow of the learning data generation process. Figure 7B : is a flowchart showing the flow of the learning process.

[0021] Fig. 8A is a diagram showing a configuration example of learning data for handwriting extraction. Figure 8B is a diagram showing a configuration example of learning data used for handwritten area estimation.

[0022] Fig.9A Detailed description of the process of converting a business form into text request. Fig. 9B The flowchart shows the flow of the business form to text conversion process.

[0023] Fig. 10A and Fig. 10B This is a diagram showing an overview of data generation processing in business form to text conversion processing.

[0024] Fig.11 is a diagram showing the configuration of a neural network.

[0025] Fig.12 is a diagram showing the relationship between a handwritten area and a feature map.

[0026] Fig.13 : is a flowchart showing the flow of the handwriting area feature acquisition process.

[0027] Fig.14A : is a flowchart showing the flow of business form to text conversion processing in the second exemplary embodiment. Fig. 14B is a flowchart showing the flow of handwriting area selection processing in the second exemplary embodiment.

[0028] Fig.15 is a diagram showing a configuration of a neural network in the third exemplary embodiment.

[0029] Fig.16A is a flowchart showing the flow of learning data generation processing in the third exemplary embodiment. Fig. 16B is a flowchart showing the flow of a learning process in the third exemplary embodiment.

[0030] Fig.17 : is a flowchart showing the flow of business form to text conversion processing in the third exemplary embodiment.

[0031] Fig.18A , Fig.18B , Fig.18C and Fig.18D is a diagram showing an example of a business form in the fourth exemplary embodiment.

[0032] Fig.19A is a diagram showing a configuration example of learning data for seal extraction. Fig.19B is a diagram showing a configuration example of learning data for seal area estimation.

[0033] Fig. 20A : is a flowchart showing the flow of seal verification request processing. Fig. 20B : is a flowchart showing the flow of the seal verification process.

[0034] Fig.21A is a diagram showing an example of a processing target image. Fig.21B is a diagram showing an example of estimation results performed by a neural network for handwriting area estimation and handwriting extraction.

[0035] Fig.22A and Fig. 22B It is a figure which shows the example of filling in a business form. DETAILED DESCRIPTION

[0036] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. In addition, the configuration for implementing the present disclosure is not limited to the configuration described in the exemplary embodiments. Some parts of the configuration described in the exemplary embodiments may be omitted or replaced by their equivalents within the scope of allowing similar advantageous effects to be obtained.

[0037] Furthermore, in the following description, performing optical character recognition (OCR) on a handwriting extraction image is referred to as “handwriting OCR.” Handwriting OCR can be used to convert handwritten characters into text data (convert into digital data).

[0038] The first exemplary embodiment relates to an operation example of performing handwriting area estimation and handwriting extraction using a neural network and reducing the amount of calculation of the neural network.

[0039] <Image Processing System>

[0040] Figure 1 1 is a diagram showing a configuration of an image processing system 100. The image processing system 100 includes an image processing apparatus 101, a learning apparatus 102, an image processing server 103, and an OCR server 104. The image processing apparatus 101, the learning apparatus 102, the image processing server 103, and the OCR server 104 are interconnected via a network 105.

[0041] The image processing apparatus 101 is a so-called digital multifunction peripheral (MFP), and includes a printing function and a scanning function (ie, a function serving as an image acquisition unit 111 ).

[0042] In the case of performing learning on a business form, the image processing device 101 scans a plurality of sheets of a document such as a business form used as a sample to generate a plurality of pieces of image data (hereinafter, these image data are referred to as "document sample images"). The document used as a sample includes a document with handwritten characters therein. Then, the image processing device 101 transmits the document sample images to the learning device 102 via the network 105, so that the learning device 102 performs learning processing on the business form.

[0043] Furthermore, in the case of performing processing of a business form image, the image processing apparatus 101 scans a document including a target recognition object (handwritten characters) to obtain image data (hereinafter, the image data is referred to as a "processing target image"). Then, the image processing apparatus 101 transmits the obtained processing target image to the image processing server 103 via the network 105, so that the image processing server 103 performs recognition processing of the business form.

[0044] The learning device 102 functions as an image accumulation unit 115 that accumulates document sample images generated by the image processing device 101. In addition, the learning device 102 also functions as a learning data generation unit 112 that generates learning data from the images accumulated as above. The learning data is data for performing learning of a neural network that performs handwriting area estimation and handwriting extraction. The learning device 102 also functions as a learning unit 113 that performs learning of the neural network using the generated learning data. The learning processing of the learning unit 113 is used to generate a learning result (e.g., a parameter of the neural network). The learning device 102 sends the learning result (learning model) to the image processing server 103 via the network 105. The following reference is made to Fig.11 The neural network in the first exemplary embodiment is described.

[0045] The image processing server 103 functions as an image conversion unit 114 that converts a processing target image. The image conversion unit 114 generates an image as a handwriting OCR target from the processing target image. Therefore, the image conversion unit 114 performs handwriting area estimation on the processing target image generated by the image processing device 101. The image processing server 103 uses the learning result generated by the learning device 102 to perform inference through a neural network to estimate (identify) an area in which characters are handwritten and included in the processing target image. This results in obtaining a handwriting area. Here, the handwriting area is information indicating a partial area included in the processing target image, and is expressed as information including, for example, a specific pixel position (coordinates) on the processing target image and a width and height from the specific pixel position. In addition, depending on the number of items written in the business form, the handwriting area may include a plurality of handwriting areas. In addition, the image conversion unit 114 performs handwriting extraction in the handwriting area obtained by the handwriting area estimation. At this time, the image processing device 101 uses the learning result generated by the learning device 102 to perform inference through a neural network to extract (identify) pixels (pixel positions) of handwritten characters included in the handwriting area. Then, the image processing device 101 obtains a handwriting extraction image. Here, the handwriting area is an area divided based on, for example, each fill-in box in the processing target image. On the other hand, the handwriting extraction image is an image obtained by identifying the pixels of the handwritten characters in the handwriting area. Therefore, even if a part of the lead type font characters is included in the handwriting area, since only the pixels of the handwritten characters are identified in the handwriting extraction image, the pixels of the lead type font characters are not included in the handwriting extraction image. Therefore, performing processing based on the results of handwriting area estimation and handwriting extraction makes it possible to process only the handwritten characters included in the processing target image while distinguishing the handwritten characters for each individual fill-in box. Then, the image conversion unit 114 sends the handwriting extraction image to the OCR server 104. Therefore, the image conversion unit 114 instructs the OCR server 104 to set the handwriting extraction image, which is obtained by extracting the handwritten characters included in the estimated handwriting area, as the handwriting OCR target area. In addition, the image conversion unit 114 refers to the handwriting area and the handwriting extraction image, and thus generates an image obtained by removing the pixels of the handwritten characters from a specific pixel position (coordinate) on the processing target image (hereinafter referred to as a "type font image"). Then, the image conversion unit 114 generates information about the following area (hereinafter referred to as a "type font area"), which is an area on the type font image and includes type font characters that are the type font OCR target. The generation of the type font area is described below. Then, the image conversion unit 114 sends the type font image and the type font area to the OCR server 104. Therefore, the image conversion unit 114 instructs the OCR server 104 to perform type font OCR with respect to the type font area on the type font image.Image conversion unit 114 receives the result of handwriting OCR and the result of typeface OCR from OCR server 104. Then, image conversion unit 114 integrates the received results into text data and sends the text data to image processing apparatus 101. Hereinafter, the text data is referred to as "business form text data".

[0046] The OCR server 104 includes functions as a handwriting OCR unit 116 and a typeface OCR unit 117, wherein the handwriting OCR unit 116 is suitable for performing character recognition of handwritten characters and the typeface OCR unit 117 is suitable for performing character recognition of typeface characters. Upon receiving the handwriting extraction image, the OCR server 104 causes the handwriting OCR unit 116 to perform handwriting OCR processing on the handwriting extraction image to obtain text data (OCR result). The handwriting OCR unit 116 sends the obtained text data to the image processing server 103. In addition, when the OCR server 104 receives the typeface image and the typeface area, it causes the typeface OCR unit 117 to perform typeface OCR processing on the typeface area included in the typeface image to obtain text data. The typeface OCR unit 117 sends the obtained text data to the image processing server 103.

[0047] <Configuration of Neural Network>

[0048] Describe the configuration of the neural network in this system. Fig.11 1 is a diagram showing the configuration of the neural network 1100. The neural network 1100 in the first exemplary embodiment performs various types of processing operations with respect to the input of an image. Therefore, the neural network 1100 performs handwriting area estimation and handwriting extraction on the input image. Therefore, the neural network 1100 in the first exemplary embodiment has a configuration obtained by combining a plurality of neural networks that process respective different tasks. Fig.11 In the example shown in , the neural network 1100 has a configuration obtained from a combination of a handwriting area estimation neural network and a handwriting extraction neural network. The handwriting area estimation neural network and the handwriting extraction neural network share an encoder, the details of which are described below. In addition, in the first exemplary embodiment, it is assumed that the image to be input to the neural network 1100 is a grayscale (single-channel) image, but it may also be another type of image, such as a color (three-channel) image.

[0049] like Fig.11As shown in , the neural network 1100 includes an encoder unit 1101, a pixel extraction decoder unit 1112 participating in handwriting extraction, and a region estimation decoder unit 1122 participating in handwriting region estimation. The encoder unit 1101 and the pixel extraction decoder unit 1112 constitute a handwriting region estimation neural network. In addition, the encoder unit 1101 and the region estimation decoder unit 1122 constitute a handwriting region estimation neural network. Such two neural networks share the encoder unit 1101, which is a layer that performs similar calculations in the two neural networks. Then, switching is performed between the pixel extraction decoder unit 1112 and the region estimation decoder unit 1122 depending on the task. Upon receiving an input image, the neural network 1100 performs calculations in the encoder unit 1101. Then, the neural network 1100 inputs the result of the calculation (feature map) to the pixel extraction decoder unit 1112 and the region estimation decoder unit 1122, and then outputs the handwriting extraction result via the calculation in the pixel extraction decoder unit 1112, and outputs the handwriting region estimation result via the calculation in the region estimation decoder unit 1122.

[0050] The encoder unit 1101 includes convolution layers EC1, EC2, EC3 and EC4 and pooling layers EP1, EP2 and EP3. The convolution layers EC1, EC2, EC3 and EC4 perform convolution calculations on each input with filters having respective predetermined kernel sizes to extract features, thereby outputting feature maps of a predetermined number of channels. Here, the feature map is a set of feature information about handwritten fill-in. The outputs of the convolution layers EC1, EC2, EC3 and EC4 and the pooling layers EP1, EP2 and EP3 are feature maps. In addition, the convolution layer EC1 receives an input image, and the convolution layers EC2, EC3 and EC4 receive feature maps output from the corresponding immediately preceding layers (pooling layers (described below)). The pooling layers EP1, EP2 and EP3 output the maximum values ​​of the respective predetermined kernel size regions to the feature maps output from the respective immediately preceding convolution layers EC1, EC2 and EC3, thereby extracting more important feature information from the input feature maps. In this way, the encoder unit 1101 converts the input image into feature information required for handwriting area estimation and handwriting extraction. In addition, in the first exemplary embodiment, for example, the pooling layers EP1, EP2, and EP3 reduce the resolution of each input feature map to half and output the feature map at the reduced resolution.

[0051] The pixel extraction decoder unit 1112 includes deconvolution layers DDp1, DDp2, DDp3, and DDp4 and upsampling layers DUp1, DUp2, and DUp3. The deconvolution layers DDp1, DDp2, DDp3, and DDp4 perform deconvolution calculations on the respective input feature maps with filters having respective predetermined kernel sizes. Then, the deconvolution layers DDp1, DDp2, DDp3, and DDp4 convert feature information included in the feature maps and gradually estimate the category of each pixel of the input image (whether the pixel of interest is a handwritten pixel). The upsampling layers DUp1, DUp2, and DUp3 increase the resolution of each feature map output from each immediately preceding deconvolution layer DDp1, DDp2, and DDp3, thereby restoring the resolution reduced by the pooling layers EP1, EP2, and EP3. In addition, in the first exemplary embodiment, for example, the upsampling layers DUp1, DUp2, and DUp3 double the resolution of each input feature map and output each feature map at the doubled resolution. In addition, the deconvolution layer DDp4 outputs images of various categories for which the category probability value of each pixel of the input image has been estimated (the number of images to be output is the number of channels = the number of categories required for classification). The pixel extraction decoder unit 1112 normalizes the output of the deconvolution layer DDp4 to a probability distribution, and outputs a graph representing the probability of the corresponding category of each pixel. Then, the pixel extraction decoder unit 1112 divides the pixels of each category with the highest probability, and performs imaging of the pixels, thereby outputting a handwriting extraction image. Here, as a method of normalizing to a probability distribution, for example, a softmax function is used. Handwriting extraction is performed by a subnetwork including an encoder unit 1101 and a pixel extraction decoder unit 1112.

[0052] In the above-mentioned calculation of the pixel extraction decoder unit 1112, since the resolution of the feature map is reduced via the pooling layers EP1, EP2, and EP3, the feature map to be calculated by the pixel extraction decoder unit 1112 loses local features. Therefore, the feature map that has been extracted by the encoder unit 1101 during the processing stage is passed to the pixel extraction decoder unit 1112 to increase the estimation accuracy of the class probability value. Therefore, the output of the convolution layer EC3 is connected to the output of the upsampling layer DUp1, the output of the convolution layer EC2 is connected to the output of the upsampling layer DUp2, and the output of the convolution layer EC1 is connected to the output of the upsampling layer DUp3. In the following, this processing method is referred to as "jump connection".

[0053] The region estimation decoder unit 1122 includes deconvolution layers DDr1, DDr2, and DDr3 and upsampling layers DUr1, DUr2, and DUr3. The deconvolution layers DDr1, DDr2, and DDr3 constituting the region estimation decoder unit 1122 perform respective operations similar to respective operations of the deconvolution layers DDp1, DDp2, and DDp3 constituting the pixel extraction decoder unit 1112. The upsampling layers DUr1, DUr2, and DUr3 also perform respective operations similar to respective operations of the upsampling layers DUp1, DUp2, and DUp3 constituting the pixel extraction decoder unit 1112. In addition, the upsampling layer DUr3 outputs an image of each category in which the category probability value of each pixel of the input image has been estimated (the number of images to be output is the number of channels = the number of categories required for classification). The region estimation decoder unit 1122 normalizes the output of the upsampling layer DUr3 into a probability distribution, and outputs a map representing the probability of each category for each pixel. Then, the area estimation decoder unit 1122 divides the pixels of each category with the highest probability and estimates the handwritten area. Here, as a method of normalizing to a probability distribution, for example, a softmax function is used. The handwritten area estimation is performed by a subnetwork including the encoder unit 1101 and the area estimation decoder unit 1122.

[0054] Then, a skip connection is introduced to increase the estimation accuracy of the class probability value even in the calculation performed by the region estimation decoder unit 1122. Therefore, the output of the convolution layer EC3 is connected to the output of the upsampling layer DUr1, and the output of the convolution layer EC2 is connected to the output of the upsampling layer DUr2.

[0055] In addition, although in the first exemplary embodiment, an example has been described in which one convolution layer is arranged between pooling layers in the encoder unit 1101, a plurality of convolution layers may be arranged therebetween. In addition, an example has been described in which one deconvolution layer is arranged between upsampling layers in each of the region estimation decoder unit 1122 and the pixel extraction decoder unit 1112, a plurality of deconvolution layers may be arranged therebetween. In addition, in the case where a convolution layer and a subsequent pooling layer are regarded as constituent units, although in the first exemplary embodiment, three constituent units are provided in the region estimation decoder unit 1122, more than three or less than three constituent units may be provided therein. In addition, in the case where an upsampling layer and a subsequent deconvolution layer are regarded as constituent units, in the first exemplary embodiment, two constituent units are provided in the region estimation decoder unit 1122, and three constituent units are provided in the pixel extraction decoder unit 1112. However, the first exemplary embodiment is not limited thereto, and constituent units more than or less than this number may be provided therein. Furthermore, although in the first exemplary embodiment, an example has been described in which each pooling layer outputs a maximum value, other configurations may be employed in which, for example, each pooling layer outputs an average value of feature map pixels corresponding to a kernel.

[0056] The above-mentioned region estimation decoder unit 1122 has a configuration similar to that of the pixel extraction decoder unit 1112 except for the deconvolution layer DDp4, but may be configured to have a configuration having a smaller amount of calculation than that of the pixel extraction decoder unit 1112. For example, with respect to the deconvolution layers DDr1, DDr2, and DDr3 of the region estimation decoder unit 1122, the number of output channels of each layer is set to half the number of output channels of the deconvolution layers DDp1, DDp2, and DDp3 of the pixel extraction decoder unit 1112, wherein the deconvolution layers DDp1, DDp2, and DDp3 perform operations similar to those of the deconvolution layers DDr1, DDr2, and DDr3. In addition, similarly, with respect to the upsampling layers DUr1, DUr2, and DUr3 of the region estimation decoder unit 1122, the number of output channels of each layer is set to half the number of output channels of the upsampling layers DUp1, DUp2, and DUp3 of the pixel extraction decoder unit 1112. The reason for this is that the handwriting extraction task and the handwriting area estimation task differ in difficulty level, wherein the handwriting extraction task extracts pixels of the handwritten part one by one, and the handwriting area estimation task estimates the area of ​​a set of pixels of the handwritten part. Therefore, the area estimation decoder unit 1122 that processes the handwriting area estimation task having a lower difficulty level than the handwriting extraction task can be configured as a small-scale neural network having a smaller amount of calculation than the pixel extraction decoder 1112. Simplifying the configuration of the area estimation decoder unit 1122 enables the amount of calculation to be reduced to a greater extent than performing handwriting area estimation in a more complex configuration equivalent to the configuration of the pixel extraction decoder unit 1112. In addition, in the first exemplary embodiment, an example is described in which the scale of the neural network is reduced by reducing the number of convolutional layers or reducing the number of output channels of each convolutional layer. Not only can the number of convolutional layers or the number of output channels of each convolutional layer be reduced, but also, for example, the number of convolutions of each convolutional layer can be reduced.

[0057] In the above-mentioned neural network 1100, as mentioned above, the pixel extraction decoder unit 1112 and the region estimation decoder unit 1122 share the calculations performed by the encoder unit 1101. Therefore, the feature map output from the encoder unit 1101 is set as the input to the pixel extraction decoder unit 1112 and the region estimation decoder unit 1122. Here, the handwritten pixels are pixels present in the handwritten area. Therefore, a feature map obtained by narrowing the input image to a portion corresponding to the handwritten area estimated by the region estimation decoder unit 1122 is obtained, and the feature map is input to the pixel extraction decoder unit 1112. This makes it possible to reduce the amount of calculation required to perform handwriting extraction in a non-handwritten area where no handwritten pixels exist, thereby reducing the amount of calculation. The following reference is made to Fig.9A and Fig. 9B as well as Fig.13 Describe this.

[0058] <Learning Sequence>

[0059] Describe the learning sequence in this system. Figure 3A is a diagram showing a learning sequence of the image processing system 100 .

[0060] When the user issues an instruction to the image processing apparatus 101 to read a document in step S301 , then in step S302 , the image acquisition unit 111 reads the document to generate a document sample image.

[0061] In step S303, the image acquisition unit 111 sends the document sample image generated in the above-mentioned manner to the learning data generation unit 112. In addition, the image acquisition unit 111 may attach identification (ID) information to the document sample image. The ID information is, for example, information for identifying the image processing device 101 used as the image acquisition unit 111. In addition, the ID information may be user identification information for identifying a user operating the image processing device 101 or group identification information for identifying a group to which the user belongs.

[0062] In step S304 , the learning data generating unit 112 accumulates the document sample image in the image accumulating unit 115 after receiving the transmitted document sample image.

[0063] In step S305, when the user issues an instruction to the learning device 102 to attach the correct answer data related to the document sample image, the learning data generation unit 112 acquires the correct answer data. Then, in step S306, the learning data generation unit 112 accumulates the correct answer data in the image accumulation unit 115 while associating the correct answer data with the document sample image. The correct answer data is data used for learning of the neural network. The method of attaching the correct answer data is described below. Then, in step S307, the learning data generation unit 112 generates learning data based on the multiple data accumulated in the above-mentioned manner. At this time, the learning data generation unit 112 can generate learning data using only the document sample image based on specific ID information. After that, in step S308, the learning data generation unit 112 sends the learning data to the learning unit 113. In the case of generating learning data using only the image based on the specific ID information, the learning data generation unit 112 also sends the ID information together with the learning data. In step S309, the learning unit 113 performs learning processing based on the received learning data and updates the learning model. The learning unit 113 may retain a learning model for each piece of ID information and perform learning using only the learning data corresponding to the ID information.Associating the ID information and the learning model with each other in the above-mentioned manner enables establishment of a learning model dedicated to a specific use environment.

[0064] <Use Sequence>

[0065] Describe the sequence of use in this system. Figure 3B is a diagram showing a usage sequence of the image processing system 100 .

[0066] When the user issues an instruction to read a document (business form) in step S351, the image acquisition unit 111 reads the document to generate a processing target image in step S352. Figure 4A Business Form 400 or Figure 4B . These business forms include fields for names (first name field 401 and last name field 402 and first name field 451), address fields 403 and 452, and telephone number fields 404 and 453, in which the names, addresses, and telephone numbers have been input in handwriting, respectively. In addition, titles 411, 412, 413, 414, 461, 462, and 463 of item names corresponding to the respective fields each contain an image of lead-type font characters previously printed on a document. In addition, since the arrangement of these fields (the layout of the business form) is determined by the business form creation source, it is possible to generate an image that is different for each business form read (i.e., an image of a so-called atypical business form).

[0067] In step S353, the image acquisition unit 111 transmits the processing target image read in the above-mentioned manner to the image conversion unit 114. In addition, at this time, the image acquisition unit 111 may attach ID information to the data to be transmitted.

[0068] In step S354, upon receiving the transmitted data, the image conversion unit 114 receives an instruction to convert the processing target image into text data. At this time, the image conversion unit 114 stores information about the image acquisition unit 111 as a reply destination of the data. In step S355, upon receiving an instruction to modify a handwritten character, the image conversion unit 114 requests an updated learning model from the learning unit 113. In step S356, in response to the request from the image conversion unit 114, the learning unit 113 sends the updated learning model to the image conversion unit 114. In the case where the request made by the image conversion unit 114 includes designated ID information, the learning unit 113 sends the learning model corresponding to the ID information. In step S357, the image conversion unit 114 performs handwriting area estimation and handwriting extraction on the processing target image based on the acquired learning model. Then, in step S358, the image conversion unit 114 sends the handwriting extraction image generated for each handwriting area to the handwriting OCR unit 116. In step S359, the handwriting OCR unit 116 performs handwriting OCR processing on the handwriting extraction image to generate text data (handwriting). In step S360, the handwriting OCR unit 116 sends the generated text data (handwriting) to the image conversion unit 114. Then, in step S361, the image conversion unit 114 generates a type font image and a type font area from the processing target image. Then, in step S362, the image conversion unit 114 sends the type font image and the type font area to the type font OCR unit 117. In step S363, the type font OCR unit 117 performs type font OCR processing on the type font image to generate text data (type font). Then, in step S364, the type font OCR unit 117 sends the generated text data (type font) to the image conversion unit 114. Then, in step S365, the image conversion unit 114 generates business form text data based on at least the text data (handwriting) and the text data (type font). In step S366, the image conversion unit 114 sends the business form text data to the image acquisition unit 111. In step S367, upon receiving the business form text data, the image acquisition unit 111 presents a use screen of the business form text data to the user. Thereafter, the image acquisition unit 111 outputs the business form text data according to the purpose of the business form text data. For example, the image acquisition unit 111 sends the business form text data to a business system (not shown) provided externally separately from the image processing system 100.

[0069] <Device Configuration>

[0070] In order to realize the above-mentioned system, each device has the following configuration. Figure 2A is a diagram showing the configuration of the image processing apparatus 101 . Figure 2Bis a diagram showing the configuration of the learning device 102 . Figure 2C is a diagram showing the configuration of the image processing server 103 . Figure 2D is a diagram showing the configuration of the OCR server 104 .

[0071] like Figure 2A As shown in FIG. 1 , the image processing apparatus 101 includes a central processing unit (CPU) 201, a read-only memory (ROM) 202, a random access memory (RAM) 204, a printer device 205, a scanner device 206, a document transport device 207, a storage section 208, an input device 209, a display device 210, and an external interface 211. The devices and units of the image processing apparatus 101 are interconnected via a data bus 203 in a manner capable of communicating with each other.

[0072] The CPU 201 is a controller configured to comprehensively control the image processing apparatus 101. The CPU 201 activates an operating system (OS) through a boot program stored in the ROM 202. A controller program stored in the storage section 208 is executed on the OS. The controller program is a program for controlling the image processing apparatus 101. The CPU 201 comprehensively controls devices and units interconnected by a data bus 203. The RAM 204 is used as a temporary storage area of ​​the CPU 201, such as a main memory or a work area.

[0073] The printer device 205 is a device for printing image data on a sheet (recording material or paper). The printer device 205 may be of any printing type, such as an electrophotographic printing type using a photosensitive drum or a photosensitive belt or an inkjet printing type that ejects ink from a micro nozzle array to print an image directly on a sheet. The scanner device 206 scans a document such as a paper document using an optical scanning device such as a charge coupled device (CCD) sensor to obtain electrical signal data, and then generates image data by converting the electrical signal data. In addition, a document conveying device 207 such as an automatic document feeder (ADF) conveys documents placed on a document positioning plate of the document conveying device 207 to the scanner device 206 one by one.

[0074] The storage unit 208 is a readable and writable non-volatile memory, such as a hard disk drive (HDD) or a solid state drive (SSD), on which various data are recorded, including, for example, the controller program mentioned above. The input device 209 is a device configured with, for example, a touch panel or a hardware key. The input device 209 receives instructions issued by an operation performed by a user. Then, the input device 209 transmits instruction information including a designated position to the CPU 201. The display device 210 is, for example, a liquid crystal display (LCD) or a cathode ray tube (CRT). The display device 210 displays image data generated by the CPU 201. The CPU 201 determines what operation the user has performed based on the instruction information received from the input device 209 and the display data displayed on the display device 210. Then, in response to the result of the determination, the CPU 201 controls the image processing apparatus 101 and generates new display data so that the display device 210 displays the new display data.

[0075] The external interface 211 transmits and receives various data including image data with respect to an external device via a network such as a local area network (LAN), a telephone line, or short-range wireless communication using infrared rays. The external interface 211 receives page description language (PDL) data from an external device such as a learning device 102 or a personal computer (PC) (not shown). The CPU 201 interprets the PDL data received by the external interface 211 to generate an image. The generated image is printed by the printer device 205 or stored in the storage unit 208. In addition, the external interface 211 receives image data from an external device such as an image processing server 103. The received image data is printed by the printer device 205, stored in the storage unit 208, or sent to another external device via the external interface 211.

[0076] Figure 2B The learning apparatus 102 shown in FIG. 1 includes a CPU 231, a ROM 232, a RAM 234, a storage section 235, an input device 236, a display device 237, an external interface 238, and a graphics processing unit (GPU) 239. The devices and units of the learning apparatus 102 can transmit and receive data with respect to each other via a data bus 233.

[0077] The CPU 231 is a controller configured to control the entire learning device 102. The CPU 231 activates the OS through a boot program stored in the ROM 232, which is a non-volatile memory. The CPU 231 executes a learning data generation program and a learning program stored in the storage unit 235 on the OS. The CPU 231 generates learning data by executing the learning data generation program. In addition, the CPU 231 learns a neural network that performs handwriting extraction by executing the learning program. The CPU 231 controls the devices and units of the learning device 102 via a bus such as a data bus 233.

[0078] The RAM 234 is used as a temporary storage area such as a main memory or a work area of ​​the CPU 231. The storage section 235 is a readable and writable nonvolatile memory, and has recorded thereon the above-mentioned learning data generation program and learning program.

[0079] The input device 236 is a device configured with, for example, a mouse and a keyboard. The display device 237 is similar to the reference Figure 2A A display device 210 is described.

[0080] External interface 238 is similar to reference Figure 2A Describe the external interface 211.

[0081] The GPU 239 is an image processing processor, and performs generation of image data and learning of a neural network in cooperation with the CPU 231 .

[0082] Figure 2C The image processing server 103 shown in FIG. 1 includes a CPU 261, a ROM 262, a RAM 264, a storage section 265, an input device 266, a display device 267, and an external interface 268. The devices and units of the image processing server 103 can transmit and receive data with respect to each other via the data bus 263.

[0083] The CPU 261 is a controller configured to control the entire image processing server 103. The CPU 261 activates the OS through a boot program stored in the ROM 262, which is a nonvolatile memory. The CPU 261 executes the image processing server program stored in the storage section 265 on the OS. The CPU 261 performs handwriting area estimation and handwriting extraction on the processing target image by executing the image processing server program. The CPU 261 controls the devices and units of the image processing server 103 via a bus such as a data bus 263.

[0084] The RAM 264 is used as a temporary storage area such as a main memory or a work area of ​​the CPU 261. The storage section 265 is a readable and writable nonvolatile memory, and has the above-mentioned image processing server program recorded thereon.

[0085] Input device 266 is similar to reference Figure 2B The input device 236 described above. The display device 267 is similar to the reference Figure 2A A display device 210 is described.

[0086] External interface 268 is similar to reference Figure 2A Describe the external interface 211.

[0087] Figure 2D The OCR server 104 shown in FIG. 1 includes a CPU 291, a ROM 292, a RAM 294, a storage section 295, an input device 296, a display device 297, and an external interface 298. The devices and units of the OCR server 104 can transmit and receive data with respect to each other via a data bus 293.

[0088] The CPU 291 is a controller configured to control the entire OCR server 104. The CPU 291 activates the OS through a boot program stored in the ROM 292, which is a nonvolatile memory. The CPU 291 executes an OCR server program stored in the storage section 295 on the OS. The CPU 291 recognizes the handwritten characters of the handwriting extraction image and the type font characters of the type font image by executing the OCR server program, and converts the handwritten characters of the handwriting extraction image and the type font characters of the type font image into text data. The CPU 291 controls the devices and units of the OCR server 104 via a bus such as a data bus 293.

[0089] The RAM 294 is used as a temporary storage area such as a main memory or a work area of ​​the CPU 291. The storage section 295 is a readable and writable nonvolatile memory, and has the above-mentioned OCR server program recorded thereon.

[0090] Input device 296 is similar to reference Figure 2B The input device 236 described above. The display device 297 is similar to the reference Figure 2A A display device 210 is described.

[0091] External interface 298 is similar to reference Figure 2A Describe the external interface 211.

[0092] <Operation Screen>

[0093] Issued via the following operation screen Figure 3AThe user instruction in step S301 in the learning sequence shown in . Figure 5A 5 is a diagram showing a learning document scanning screen. The learning document scanning screen 500 is an example of a screen displayed on the display device 210 of the image processing apparatus 101. Figure 5A As shown in FIG. 5 , the learning document scanning screen 500 includes a preview area 501 , a scan button 502 , and a send start button 503 .

[0094] The scan button 502 is a button for starting to read a document set on the scanner device 206. After scanning is completed, a document sample image is generated and then displayed in the preview area 501. If another document is set on the scanner device 206 and the scan button 502 is pressed again, a plurality of document sample images are collectively stored.

[0095] When the document is read, the transmission start button 503 becomes pressable to issue an instruction. In response to pressing the transmission start button 503 to issue an instruction, the document sample image is transmitted to the learning apparatus 102.

[0096] In addition, the following operation screen is issued Figure 3A The user instruction in step S305 in the learning sequence shown in . Figure 5B : is a diagram showing a handwriting extraction correct answer data creation screen. Figure 5C 2 is a diagram showing a handwriting area estimation correct answer data creation screen. The user performs operations based on the display contents of the handwriting extraction correct answer data creation screen and the handwriting area estimation correct answer data creation screen, thereby creating correct answer data.

[0097] The handwriting extraction correct answer data creation screen 520 is an example of a screen displayed on the display device 237 of the learning device 102. In addition, the handwriting extraction correct answer data creation screen 520 may be configured to be displayed on a display device of a different device (such as a PC) connected to the learning device 102 via a network, and to be used to issue instructions to the learning device 102 via the different device. Figure 5B As shown in , the handwriting extraction correct answer data creation screen 520 includes an image display area 521 , an image selection button 522 , an enlargement button 523 , a reduction button 524 , an extraction button 525 , an estimation button 526 , and a storage button 527 .

[0098] The image selection button 522 is a button for selecting a document sample image received from the image processing device 101 and accumulated in the image accumulation unit 115. In response to pressing the image selection button 522 to issue an instruction, a selection screen (not shown) is displayed to enable selection of a document sample image. In response to selecting a document sample image, the selected document sample image is displayed in the image display area 521. The user performs an operation on the document sample image displayed in the image display area 521 to create correct answer data.

[0099] The zoom-in button 523 and the zoom-out button 524 are buttons for respectively zooming in and out the display in the image display area 521. In response to pressing the zoom-in button 523 or the zoom-out button 524 to issue an instruction, the document sample image displayed in the image display area 521 is zoomed in or out to facilitate the creation of correct answer data.

[0100] Extract button 525 and estimate button 526 are buttons for selecting which correct answer data of handwriting extraction and handwriting area estimation is to be created. In response to selecting one of the buttons, the selected button is highlighted. In response to selecting extract button 525, a state for creating correct answer data of handwriting extraction is set. When extract button 525 has been selected, the user creates correct answer data of handwriting extraction by performing the following operations. Figure 5B As shown in , the user operates the mouse cursor via the input device 236 to trace and select handwritten characters in the document sample image displayed in the image display area 521. The learning data generating unit 112 records the pixel position selected by the user operation in the document sample image upon receiving the operation. Therefore, the correct answer data for handwriting extraction is the position of the pixel corresponding to the handwritten portion in the document sample image. On the other hand, in response to selecting the estimate button 526, a state for creating correct answer data for handwriting area estimation is set. When the estimate button 526 has been selected, the user creates correct answer data for handwriting area estimation by performing the following operations. As Figure 5C As indicated by the dotted box shown in , the user operates the mouse cursor via the input device 236. Then, the user selects the inside of the area surrounded by the grid lines and filled with handwritten characters included in the document sample image displayed in the image display area 521 (here, the inside of the fill-in column and not including the grid lines). Therefore, this is an operation of selecting an area for each fill-in column of the business form. When receiving this operation, the learning data generating unit 112 records the area selected by the above-mentioned operation. Therefore, the correct answer data for the handwritten area estimation is the area inside the fill-in column on the document sample image (the area filled with handwritten characters) (hereinafter, the area filled with handwritten characters will be referred to as "handwritten area").

[0101] The storage button 527 is a button for storing the created correct answer data. The correct answer data extracted by handwriting is accumulated in the image accumulation unit 115 as the following image. The image has the same size (width and height) as the document sample image. The value of each pixel of the handwritten character position selected by the user is a value indicating the handwritten part (for example, "255", and this also applies to the following description). In addition, the value of each pixel at a position other than the handwritten character position is a value that does not indicate the handwritten part (for example, "0", and this also applies to the following description). Hereinafter, the image as the correct answer data extracted by handwriting is referred to as a "handwriting extraction correct answer image". Figure 4C An example of a handwriting extraction correct answer image is shown. In addition, the correct answer data of the handwriting area estimation is accumulated in the image accumulation unit 115 as the following image. The image has the same size (width and height) as the document sample image. The value of each pixel corresponding to the handwriting area selected by the user is a value indicating the handwriting area (for example, "255", and this also applies to the following description). In addition, the value of each pixel corresponding to the area other than the handwriting area is a value that does not indicate the handwriting area (for example, "0", and this also applies to the following description). Hereinafter, the image of the correct answer data as the handwriting area estimation is referred to as the "handwriting area estimation correct answer image". Figure 4D An example of a handwritten area estimation correct answer image is shown.

[0102] In addition, the following operation screen is issued Figure 3B The user instruction in step S351 in the learning sequence shown in . Figure 5D is a diagram showing a business form processing screen. Figure 5D As shown in FIG. 5 , the business form processing screen 540 includes a preview area 541 , a scan button 542 , and a send start button 543 .

[0103] The scan button 542 is a button for starting reading of a document set on the scanner device 206. When scanning is completed, a processing target image is generated and then displayed in the preview area 541.

[0104] When the document is read, the transmission start button 543 becomes pressable to issue an instruction. In response to the transmission start button 543 being pressed to issue an instruction, the processing target image is transmitted to the image processing server 103.

[0105] <Document Sample Image Generation Processing>

[0106] Next, the document sample image generation process performed by the image processing apparatus 101 is described. Fig. 6A2 is a flowchart showing the flow of document sample image generation processing. This processing is realized by the CPU 201 reading out the controller program stored in the storage unit 208, loading the controller program onto the RAM 204, and executing the controller program. This processing is started by the user operating the input device 209 of the image processing apparatus 101.

[0107] In step S601, the CPU 201 determines whether an instruction for scanning a document has been issued. In the case where the user has performed a predetermined operation to scan a document (pressed the scan button 502 to issue an instruction) via the input device 209, the CPU 201 sets the determination result to "yes" ("yes" in step S601), and then advances the process to step S602. If not, the CPU 201 sets the determination result to "no" ("no" in step S601), and then advances the process to step S604.

[0108] In step S602, the CPU 201 controls the scanner device 206 and the document transport device 207 to scan a document, thereby generating a document sample image. The document sample image is generated as grayscale image data.

[0109] In step S603 , the CPU 201 transmits the document sample image generated in step S602 to the learning apparatus 102 via the external interface 211 .

[0110] In step S604, the CPU 201 determines whether to end the processing. In the case where the user has performed a predetermined operation to end the document sample image generation processing, the CPU 201 sets the determination result to "yes" ("yes" in step S604), and then ends the processing. If not, the CPU 201 sets the determination result to "no" ("no" in step S604), and then returns the processing to step S601.

[0111] Through the above-described processing, the image processing apparatus 101 generates a document sample image and transmits the document sample image to the learning apparatus 102. Depending on an operation performed by a user or the number of documents placed on the document transport device 207, a plurality of document sample images are acquired.

[0112] <Document Sample Image Receiving Process>

[0113] Next, the document sample image receiving process performed by the learning device 102 is described. Figure 6B 2 is a flowchart showing the flow of the document sample image receiving process. This process is realized by the CPU 231 reading out the learning data generation program stored in the storage unit 235, loading the learning data generation program onto the RAM 234, and executing the learning data generation program. This process is started by the user turning on the power of the learning device 102.

[0114] In step S621, the CPU 231 determines whether the document sample image has been received. If the image data has been received via the external interface 238, the CPU 231 sets the determination result to "yes" ("yes" in step S621), and then advances the process to step S622. If not, the CPU 231 sets the determination result to "no" ("no" in step S621), and then advances the process to step S623.

[0115] In step S622 , the CPU 231 stores the received document sample image in a predetermined area of ​​the storage section 235 .

[0116] In step S623, the CPU 231 determines whether to end the processing. In the case where the user has performed a predetermined operation to end the document sample image receiving processing, such as powering off the learning device 102, the CPU 231 sets the determination result to "yes" ("yes" in step S623), and then ends the processing. If not, the CPU 231 sets the determination result to "no" ("no" in step S623), and then returns the processing to step S621.

[0117] <Correct Answer Data Generation Process>

[0118] Next, the correct answer data generation process performed by the learning device 102 is described. Figure 6C : is a flowchart showing the flow of the correct answer data generation process.

[0119] This process is implemented by the learning data generating unit 112 of the learning apparatus 102. This process is started by the user performing a predetermined operation via the input device 236 of the learning apparatus 102.

[0120] In step S641, the CPU 231 determines whether an instruction to select a document sample image has been issued. In the case where the user has performed a predetermined operation via the input device 236 to select a document sample image (pressing the image selection button 522 to issue an instruction), the CPU 231 sets the determination result to "yes" ("yes" in step S641), and then advances the process to step S642. If not, the CPU 231 sets the determination result to "no" ("no" in step S641), and then advances the process to step S643.

[0121] In step S642 , the CPU 231 reads out the document sample image selected by the user in step S641 from the storage section 235 , and outputs the read document sample image to the user (displays it in the image display area 521 ).

[0122] In step S643, CPU 231 determines whether the user has issued an instruction to input the correct answer data. If the user has performed the operation of tracking the handwritten characters in the document sample image or tracking the grid frame with the handwritten characters filled therein as mentioned above via input device 236, then CPU 231 sets the determination result to "yes" ("yes" in step S643), and then proceeds to step S644. If not, then CPU 231 sets the determination result to "no" ("no" in step S643), and then proceeds to step S647.

[0123] In step S644, the CPU 231 determines whether the correct answer data input by the user is the correct answer data of handwriting extraction. If the user has performed an operation of issuing an instruction to create the correct answer data of handwriting extraction (selecting the extraction button 525), the CPU 231 sets the determination result to "yes" ("yes" in step S644), and then advances the processing to step S645. If not, that is, in the case where the correct answer data input by the user is the correct answer data of handwriting area estimation (selecting the estimation button 526), ​​the CPU 231 sets the determination result to "no" ("no" in step S644), and then advances the processing to step S646.

[0124] In step S645, the CPU 231 temporarily stores the handwriting extracted correct answer data input by the user in the RAM 234. As mentioned above, the handwriting extracted correct answer data is position information about each pixel corresponding to the handwritten portion in the document sample image.

[0125] In step S646, the CPU 231 temporarily stores the correct answer data of handwriting area estimation input by the user in the RAM 234. As mentioned above, the correct answer data of handwriting area estimation is area information corresponding to the handwriting area in the document sample image.

[0126] In step S647, the CPU 231 determines whether an instruction to store the correct answer data has been issued. In the case where the user has performed a predetermined operation via the input device 236 to store the correct answer data (pressing the storage button 527 to issue an instruction), the CPU 231 sets the determination result to "yes" ("yes" in step S647), and then advances the process to step S648. If not, the CPU 231 sets the determination result to "no" ("no" in step S647), and then advances the process to step S650.

[0127] In step S648, CPU 231 generates and stores a handwriting extraction correct answer image as the correct answer data for handwriting extraction. CPU 231 generates a handwriting extraction correct answer image by performing the following operations. CPU 231 generates an image having the same size as the document sample image read out in step S642 as a handwriting extraction correct answer image. CPU 231 sets all pixels of the generated image to values ​​that do not indicate a handwritten portion. Then, CPU 231 refers to the position information temporarily stored in RAM 234 in step S645, and changes the value of each pixel in the corresponding position on the handwriting extraction correct answer image to a value indicating a handwritten portion. CPU 231 stores the handwriting extraction correct answer image generated in this way in a predetermined area of ​​storage unit 235, and associates the handwriting extraction correct answer image with the document sample image read out in step S642.

[0128] In step S649, CPU 231 generates and stores a handwriting area estimation correct answer image as the correct answer data for handwriting area estimation. CPU 231 generates a handwriting area estimation correct answer image by performing the following operations. CPU 231 generates an image having the same size as the document sample image read out in step S642 as a handwriting area estimation correct answer image. CPU 231 sets all pixels of the generated image to values ​​that do not indicate a handwriting area. Then, CPU 231 refers to the area information temporarily stored in RAM 234 in step S646, and changes the value of each pixel included in the corresponding area on the handwriting area estimation correct answer image to a value indicating a handwriting area. CPU 231 stores the handwriting area estimation correct answer image generated in this way in a predetermined area of ​​storage unit 235, and associates the handwriting area estimation correct answer image with the document sample image read out in step S642 and the handwriting extraction correct answer image generated in step S648.

[0129] In step S650, the CPU 231 determines whether to end the process. In the case where the user has performed a predetermined operation to end the correct answer data generation process, the CPU 231 sets the determination result to "yes" ("yes" in step S650), and then ends the process. If not, the CPU 231 sets the determination result to "no" ("no" in step S650), and then returns the process to step S641.

[0130] <Learning Data Generation Process>

[0131] Next, the learning data generation process performed by the learning device 102 is described. Fig. 7A2 is a flowchart showing the flow of a learning data generation process. This process is implemented by the learning data generation unit 112 of the learning apparatus 102. This process is started by a user performing a predetermined operation via the input device 236 of the learning apparatus 102.

[0132] First, in step S701, the CPU 231 selects and reads out a document sample image stored in the storage unit 235. Figure 6B In the process in step S622 shown in FIG. 2 , a plurality of document sample images are recorded in the storage section 235 , and thus the CPU 231 randomly selects one document sample image from among the plurality of document sample images.

[0133] In step S702, the CPU 231 reads out the handwriting extraction correct answer image stored in the storage section 235. Since the handwriting extraction correct answer image associated with the document sample image read out in step S701 is stored in the storage section 235 by the processing in step S648, the CPU 231 reads out the stored handwriting extraction correct answer image.

[0134] In step S703, the CPU 231 reads out the handwriting area estimated correct answer image stored in the storage section 235. Since the handwriting area estimated correct answer image associated with the document sample image read out in step S701 is stored in the storage section 235 by the processing in step S649, the CPU 231 reads out the stored handwriting area estimated correct answer image.

[0135] In step S704, the CPU 231 cuts out a portion (eg, a size of 256 pixels high by 256 pixels wide) of the document sample image read out in step S701 to generate an input image used in learning data. The CPU 231 determines the cutting position in a random manner.

[0136] In step S705, CPU 231 cuts off a portion of the handwriting extraction correct answer image read out in step S702 to generate a correct answer label image (training data or correct answer image data) used in the learning data for handwriting extraction. Hereinafter, the correct answer label image is referred to as a "handwriting extraction correct answer label image". It is assumed that the cutting position and size are similar to those of the input image cut from the document sample image in step S704.

[0137] In step S706, the CPU 231 crops a portion of the handwriting area estimation correct answer image read out in step S703 to generate a correct answer label image used in the learning data for handwriting area estimation (hereinafter, the correct answer label image is referred to as a "handwriting area estimation correct answer label image"). It is assumed that the cropping position and size are similar to those of the input image cropped from the document sample image in step S704.

[0138] In step S707, the CPU 231 associates the input image generated in step S704 and the handwriting extraction correct answer label image generated in step S705 with each other, and stores the associated images as learning data for handwriting extraction in a predetermined area of ​​the storage section 235. In the first exemplary embodiment, the CPU 231 stores data such as Fig. 8A The learning data shown in .

[0139] In step S708, the CPU 231 associates the input image generated in step S704 and the handwriting area estimation correct answer label image generated in step S706 with each other, and stores the associated image as learning data for handwriting area estimation in a predetermined area of ​​the storage section 235. In the first exemplary embodiment, the CPU 231 stores data such as Figure 8B In addition, the handwriting area estimation correct answer label image is associated with the input image generated in step S704, resulting in the handwriting extraction correct answer label image generated in step S705 being also associated.

[0140] In step S709, the CPU 231 determines whether to end the learning data generation process. If a predetermined amount of learning data has been generated so far (the predetermined amount is determined, for example, by the user specifying via the input device 236 of the learning device 102 at the beginning of this flowchart), the CPU 231 sets the determination result to "yes" ("yes" in step S709), and then ends the process. If not, the CPU 231 sets the determination result to "no" ("no" in step S709), and then returns the process to step S701.

[0141] Through the above processing, learning data for the neural network 1100 is generated. In order to increase the versatility of the neural network, the learning data can be modified. For example, the input image is enlarged or reduced at a magnification determined by randomly selecting from a predetermined range (e.g., between 50% and 150%). The handwriting area estimated correct answer label image and the handwriting extracted correct answer label image are also enlarged or reduced in a similar manner. Alternatively, the input image is rotated at a rotation angle determined by randomly selecting from a predetermined range (e.g., between -10 degrees and 10 degrees). The handwriting area estimated correct answer label image and the handwriting extracted correct answer label image are also rotated in a similar manner. If the change in enlargement or rotation is considered, when the input image, the handwriting area estimated correct answer label image, and the handwriting extracted correct answer label image are cropped in steps S704, S705, and S706, the CPU 231 crops with a slightly larger size (e.g., a size of 512 pixels high by 512 pixels wide). Then, after performing the change of enlargement or rotation, the CPU 231 performs cropping from the central portion, in this way reaching the final size of each of the input image, the handwriting area estimation correct answer label image, and the handwriting extraction correct answer label image (e.g., 256 pixels high by 256 pixels wide). Alternatively, the brightness of each pixel of the input image can be changed and modified. For example, the CPU 231 uses gamma correction to change the brightness of each pixel of the input image. The CPU 231 randomly selects and determines the gamma value from a predetermined range (e.g., between 0.1 and 10.0).

[0142] <Learning Process>

[0143] Next, the learning process performed by the learning device 102 is described. Figure 7B 2 is a flowchart showing the flow of the learning process. The process is implemented by the learning unit 113 of the learning device 102. The process is started by the user performing a predetermined operation via the input device 236 of the learning device 102. In addition, in the first exemplary embodiment, it is assumed that a mini-batch method is used for the learning of the neural network 1100.

[0144] First, in step S731, the CPU 231 initializes the neural network 1100. Therefore, the CPU 231 constructs the neural network 1100 by randomly determining the values ​​of the parameters included in the neural network 1100 and initializes the neural network 1100.

[0145] In step S732, the CPU 231 acquires the learning data. The CPU 231 executes Fig. 7A The learning data generation process shown in the flowchart of is performed to obtain a predetermined amount (minimum batch size, for example, "10") of learning data.

[0146] In step S733, the CPU 231 obtains the output of the encoder unit 1101 of the neural network 1100. Therefore, the CPU 231 inputs the input image included in each of the learning data for handwriting area estimation and the learning data for handwriting extraction to the neural network 1100, and obtains the feature map output from the encoder unit 1101.

[0147] In step S734, the CPU 231 calculates the error in the handwriting area estimation result of the neural network 1100. Therefore, the CPU 231 inputs the feature map acquired in step S733 to the area estimation decoder unit 1122, and obtains the output of the area estimation decoder unit 1122. The output is an image having the same image size as the image size of the input image, and is configured so that: pixels determined as included in the handwriting area as an estimation result have values ​​indicating the handwriting area, and pixels not so determined have values ​​not indicating the handwriting area. Then, the CPU 231 evaluates the difference between the output and the handwriting area estimation correct answer label image included in the learning data, thereby obtaining the error. Cross entropy can be used as an indicator for this evaluation.

[0148] In step S735, the CPU 231 calculates the error in the handwriting extraction result of the neural network 1100. Therefore, the CPU 231 inputs the feature map obtained in step S733 to the pixel extraction decoder unit 1112, and obtains the output of the pixel extraction decoder unit 1112. The output is an image whose image size is the same as the image size of the input image, and is configured so that: pixels determined as handwritten parts as estimation results have values ​​indicating handwritten parts, and pixels not so determined have values ​​not indicating handwritten parts. Then, the CPU 231 evaluates the difference between the output and the handwriting extraction correct answer label image included in the learning data, thereby obtaining the error. As with handwriting area estimation, cross entropy can be used as an indicator for evaluation.

[0149] In step S736, the CPU 231 adjusts the parameters of the neural network 1100. Therefore, the CPU 231 changes the values ​​of the parameters of the neural network 1100 by the back propagation method based on the errors calculated in steps S734 and S735.

[0150] In step S737, CPU 231 determines whether to end the learning process. CPU 231 performs this determination in the following manner. CPU 231 determines whether the processing in steps S732 to S736 has been performed a predetermined number of times (e.g., 60,000 times). The predetermined number of times can be determined, for example, by the user performing an operation input at the beginning of this flowchart. If the processing has been performed a predetermined number of times, CPU 231 sets the determination result to "yes" ("yes" in step S737), and then advances the processing to step S738. If not, CPU 231 sets the determination result to "no" ("no" in step S737), and then returns the processing to step S732, thereby continuing the learning of the neural network 1100.

[0151] In step S738 , the CPU 231 transmits the parameters of the neural network 1100 adjusted in step S736 to the image processing server 103 as a learning result.

[0152] <Business form conversion to text request processing>

[0153] Next, a description is given of a business form conversion to text request process performed by the image processing device 101. The image processing device 101 scans a business form in which handwritten characters are filled to generate a processing target image. Then, the image processing device 101 sends the processing target image to the image processing server 103 and requests the image processing server 103 to perform the business form conversion to text process. Fig.9A 2 is a flowchart showing the flow of a business form conversion to text request process. This process is realized by the CPU 201 of the image processing apparatus 101 reading out a controller program stored in the storage unit 208, loading the controller program onto the RAM 204, and executing the controller program. This process is started by a user performing a predetermined operation via the input device 209 of the image processing apparatus 101.

[0154] First, in step S901, the CPU 201 causes the scanner device 206 and the document transport device 207 to scan a document to generate a processing target image. The processing target image is generated as grayscale image data.

[0155] In step S902 , the CPU 201 transmits the processing target image generated in step S901 to the image processing server 103 via the external interface 211 .

[0156] In step S903, the CPU 201 determines whether the processing result has been received from the image processing server 103. If the processing result has been received from the image processing server 103 via the external interface 211, the CPU 201 sets the determination result to "Yes" ("Yes" in step S903), and then advances the processing to step S904. If not, the CPU 201 sets the determination result to "No" ("No" in step S903), and then repeats the processing in step S903.

[0157] In step S904, the CPU 201 outputs the processing result received from the image processing server 103, that is, the business form text data generated by recognizing the handwritten characters or typeface characters included in the processing target image generated in step S901. For example, the CPU 201 can send the business form text data to the sending destination set by the user operating the input device 209 via the external interface 211.

[0158] <Business Forms to Text Processing>

[0159] Next, the business form-to-text conversion process performed by the image processing server 103 is described. Fig. 9B The flowchart shows the flow of the business form to text conversion process. Fig. 10A and Fig. 10B 1 is a diagram showing an outline of data generation processing in business form to text conversion processing. The image processing server 103 serving as the image conversion unit 114 receives the processing target image from the image processing device 101, and obtains text data by performing OCR on the lead font characters and handwritten characters included in the scanned image data. The image processing server 103 causes the lead font OCR unit 117 to perform OCR on the lead font characters. The image processing server 103 causes the handwriting OCR unit 116 to perform OCR on the handwritten characters. The business form to text conversion processing is realized by the CPU 261 reading out the image processing server program stored in the storage unit 265, loading the image processing server program onto the RAM 264, and executing the image processing server program. This processing is started by the user powering on the image processing server 103.

[0160] First, in step S951, the CPU 261 loads Fig.11 The neural network 1100 shown in FIG. 1 performs handwriting area estimation and handwriting extraction. The CPU 261 constructs the neural network 1100 with Figure 7B The CPU 261 then reflects the learning result (parameters of the neural network 1100) transmitted from the learning device 102 in step S738 in the constructed neural network 1100.

[0161] In step S952, the CPU 261 determines whether a processing target image has been received from the image processing device 101. If the processing target image has been received via the external interface 268, the CPU 261 sets the determination result to "yes" ("yes" in step S952), and then advances the process to step S953. If not, the CPU 261 sets the determination result to "no" ("no" in step S952), and then advances the process to step S965. Here, it is assumed that the processing target image has been received. Fig. 10A Business Form 400 ( Figure 4A The business form 400 shown in FIG. 4 is used as a processing target image.

[0162] After executing step S952, then in steps S953 to S958, the CPU 261 inputs the processing target image received from the image processing device 101 to the neural network 1100, and performs handwriting area estimation and handwriting extraction. The neural network 1100 has a configuration in which, as described above with reference to Fig.11 As described above, the pixel extraction decoder unit 1112 and the area estimation decoder unit 1122 share the calculation result obtained by the encoder unit 1101, that is, the extracted feature map. The handwritten pixels exist within the handwritten area. Taking advantage of this, the CPU 261 narrows the feature map output from the pixel extraction decoder unit 1112 to an area corresponding to the handwritten area estimated by the area estimation decoder unit 1122. Then, the CPU 261 inputs the feature map included in the narrowed area to the pixel extraction decoder unit 1112. This makes it possible to reduce the amount of calculation required to perform handwriting extraction in a non-handwritten area, which is an area in which no handwritten pixels exist.

[0163] In step S953 , the CPU 261 inputs the processing target image received from the image processing apparatus 101 to the neural network 1100 constructed in step S951 , and obtains the feature map output from the encoder unit 1101 .

[0164] In step S954, the CPU 261 estimates a handwritten area from the processing target image received from the image processing device 101. Therefore, the CPU 261 inputs the feature map acquired in step S953 to the area estimation decoder unit 1122 to estimate the handwritten area. The CPU 261 obtains the following image data as the output of the neural network 1100. Therefore, the CPU 261 obtains image data having an image size that is the same as the image size of the processing target image, and the image data is configured such that: a value indicating a handwritten area is recorded on a pixel determined as being included in the handwritten area as a result of the estimation, and a value not indicating a handwritten area is recorded on a pixel determined as not being included in the handwritten area. In addition, as mentioned above, in Figure 3A In step S305 shown in , the user has created the correct answer data of the handwriting area estimation in the position corresponding to the fill-in column associated with each fill-in item of the business form. Therefore, the area estimation decoder unit 1122 of the neural network 1100 has learned this in advance, and therefore outputs pixels with values ​​indicating the handwriting area for each fill-in column. In addition, since the output of the neural network 1100 is the estimation result of each pixel, and the estimated area is not necessarily an accurate rectangle, it will be difficult to process, so the CPU 261 sets the bounding rectangle including the estimated area. The setting of the bounding rectangle can be achieved by applying known techniques thereto. Each bounding rectangle can be expressed as area coordinate information consisting of the upper left endpoint, width and height on the processing target image. The CPU 261 sets the rectangle information group obtained in this way as the handwriting area. Fig. 10A A dotted-line frame 1002 shown in illustrative example represents an example of a handwritten area estimated with respect to the processing target image (business form 400).

[0165] In step S955, the CPU 261 performs processing to obtain a feature map of a portion corresponding to each handwritten area in the feature map output from the convolutional layers EC1, EC2, EC3, and EC4 obtained in step S953 based on all the handwritten areas estimated in step S954. In addition, in the following description, the feature map (partial feature map) of the area corresponding to the handwritten area on the feature map output from each convolutional layer is referred to as a "handwritten area feature map".

[0166] Here, the relationship between the estimation result of the handwritten area of ​​the image input to the neural network 1100 and the feature maps output from the convolutional layers EC1, EC2, EC3 and EC4 is described. Fig.121 is a diagram for explaining the relationship between the estimation result of the handwritten area of ​​the image input to the neural network 1100 and the feature map output from the convolution layers EC1, EC2, EC3 and EC4. When receiving the input image, the neural network 1100 extracts features at each convolution layer of the encoder unit 1101 to output a feature map, and causes the region estimation decoder unit 1122 to process the feature map to estimate the category to which each pixel of the input image belongs. Fig.12 , the features of the dotted line portion in the feature map extracted by each layer of the encoder unit 1101 are processed by the corresponding layer of the region estimation decoder unit 1122, and the dotted line region corresponding to the dotted line portion of the feature map in the output image is output as the handwritten region. In addition, referring to Fig.12 , the feature maps output from the convolutional layers EC1, EC2, and EC3 are reduced in resolution at respective predetermined magnifications through the pooling layers EP1, EP2, and EP3. On the other hand, the output image from the region estimation decoder unit 1122 is made to have the same resolution as that of the feature map output from the convolutional layer EC1 through the upsampling layers DUr1, DUr2, and DUr3.

[0167] The handwritten area feature acquisition process is described considering the above relationship between the estimation result of the handwritten area and the feature map output from the convolutional layers EC1, EC2, EC3 and EC4. Fig.13 : is a flowchart showing the flow of the handwriting area feature acquisition process.

[0168] In step S1301 , the CPU 261 selects one of the handwriting areas estimated in step S954 .

[0169] In step S1302, the CPU 261 obtains a feature map (handwritten area feature map) of a portion corresponding to the handwritten area selected in step S1301 with respect to the feature map output from the convolution layer EC4. Specifically, the CPU 261 calculates a handwritten area feature map to be output from the convolution layer EC4 (a feature map to be input to the pixel extraction decoder unit 1112) by multiplying the area coordinate information about the handwritten area selected in step S1301 by the size ratio between the output of the convolution layer EC1 and the output of the convolution layer EC4. Here, the handwritten area selected in step S1301 has been obtained as a circumscribed rectangle including the area obtained by handwritten area estimation. Therefore, the handwritten area is an area obtained by separating the non-handwritten area surrounding the handwritten area. On the other hand, in the case of convolution processing in the deconvolution layer DDp1, the CPU 261 may also refer to the non-handwritten pixels surrounding the handwritten area. In this case, the CPU 261 expands the handwritten area obtained by estimation to the surrounding to a lesser extent. For example, the CPU 261 expands the handwritten area to the surroundings with a width obtained by subtracting "1" from the kernel size (width and height) to be applied to the convolution process. The CPU 261 can calculate a handwritten area feature map based on the area obtained in this way. In addition, in the case of the convolution process, the CPU 261 can also refer to non-handwritten pixels that further surround the handwritten area. Therefore, the CPU 261 can apply the following area as a handwritten area, which is obtained by expanding the handwritten area with a width larger than the width obtained by subtracting "1" from the kernel size.

[0170] After deriving the handwritten area feature map to be output from the convolution layer EC4, the CPU 261 advances the processing to step S1303.

[0171] In step S1303, the CPU 261 also derives the handwriting area feature map to be output from the convolutional layers EC1, EC2, and EC3 to perform skip connection. Fig.12 As shown in , as the output of each convolution layer, the feature map obtained by halving the output resolution of the previous layer is output through the pooling layers EP1, EP2 and EP3 located between the convolution layers. To this end, CPU 261 calculates the handwriting area feature map to be output from the convolution layer EC3 by doubling the coordinate information (vertical position, horizontal position, width and height) related to the handwriting area feature map output from the convolution layer EC4 calculated in step S1302. In addition, CPU261 calculates the handwriting area feature map to be output from the convolution layer EC2 by quadrupling the coordinate information. In addition, CPU261 calculates the handwriting area feature map to be output from the convolution layer EC1 by octupling the coordinate information.

[0172] In step S1304, the CPU 261 determines whether all handwriting areas have been processed. If the processing operations in steps S1302 and S1303 have been performed on all handwriting areas estimated in step S954, the CPU 261 sets the determination result to "yes" ("yes" in step S1304), and then ends the handwriting area feature acquisition processing. Then, the CPU 261 returns the processing to Fig. 9B The business form shown in is converted into text and the process proceeds to step S956.

[0173] If not, the CPU 261 sets the determination result to "No" ("No" in step S1304), and then returns the process to step S1301, where the CPU 261 selects and processes the next handwriting area.

[0174] In step S956, the CPU 261 inputs all the handwriting area feature maps output from the convolution layers EC1, EC2, EC3, and EC4 of the encoder unit 1101 obtained in step S955 to the pixel extraction decoder unit 1112. Then, the CPU 261 causes the pixel extraction decoder unit 1112 to estimate the handwritten pixels within the range of all handwriting areas on the feature maps output from the respective convolution layers EC1, EC2, EC3, and EC4. As an output of the neural network 1100, the following image data is obtained. Therefore, the obtained image data is image data having an image size that is the same as the image size of the handwriting area, and is configured such that: a pixel determined as a handwriting portion as a result of the estimation has a value indicating that the pixel is a handwriting portion, and a pixel not determined as a handwriting portion has a value indicating that the pixel is not a handwriting portion. Then, the CPU 261 extracts pixels located at the same position as the position of a pixel having a value indicating that the pixel is a handwriting portion included in the obtained image data from the processing target image, thereby generating a handwriting extraction image. This results in obtaining Fig. 10A The handwriting extraction image 1001 shown in FIG. Fig. 10A As shown in , handwriting extraction image 1001 is an image including only handwriting pixels in the handwriting area. The handwriting extraction images are output, and the number of handwriting extraction images corresponds to the number of handwriting area feature maps input to the pixel extraction decoder unit 1112.

[0175] In step S957, the CPU 261 sends all the handwriting extraction images generated in step S956 to the handwriting OCR unit 116 via the external interface 268. Then, the CPU 261 causes the handwriting OCR unit 116 to perform handwriting OCR on all the handwriting extraction images. Handwriting OCR can be achieved by applying known techniques thereto. Performing handwriting OCR on the handwriting extraction images prevents or reduces printed contents such as lead-type font characters or marks in a handwriting fill-in column from becoming a target of handwriting OCR, such as in Figure 4A In addition, estimating adjacent items such as the name entry field 401 and the last name entry field 402 as separate handwriting areas to perform handwriting OCR thereon prevents or reduces characters or items from being connected to each other.

[0176] In step S958, CPU 261 determines whether all handwriting OCR results have been received from handwriting OCR unit 116. The handwriting OCR result is text data obtained by handwriting OCR unit 116 identifying handwritten characters included in the handwriting area. If all handwriting OCR results have been received from handwriting OCR unit 116 via external interface 268, CPU 261 sets the determination result to "yes" ("yes" in step S958), and then advances the processing to step S959. If not, CPU 261 sets the determination result to "no" ("no" in step S958), and then repeats the processing in step S958. Through the processing performed so far, CPU 261 has obtained the handwriting area (coordinate information) and the text data obtained by identifying the handwriting characters included in the handwriting area. CPU 261 stores the obtained handwriting area and text data in RAM 264 as handwriting information table 1003.

[0177] In step S959, the CPU 261 generates a typeface image by removing the handwriting portion from the processing target image based on the coordinate information related to the handwriting area generated in step S954 and all the handwriting extraction images generated in step S956. The CPU 261 changes the pixels included in the processing target image and located at the following positions to white (red, green, blue (RGB) = (255, 255, 255)), which are the same as the positions of the pixels having values ​​indicating that the pixels are handwriting portions in all the handwriting extraction images generated in step S956. This results in obtaining Fig. 10A The typeface image 1004 shown in FIG.

[0178] In step S960, CPU 261 extracts a lead font area from the lead font image generated in step S959. CPU 261 extracts a partial area on the lead font image including lead font characters as a lead font area. Here, the partial area is a collection (object) of printed content, and is, for example, a character line composed of a plurality of characters, a sentence composed of a plurality of character lines, or an object such as a picture, a photo, a table, or a graph. As a method for extracting a partial area, for example, the following method can be used. The method binarizes the lead font image into black and white to generate a binary image. In the binary image, the method extracts a portion in which black pixels are connected to each other (connected black pixels), and creates a rectangle of circumscribed connected black pixels. The method evaluates the shape and size of the rectangle to obtain a rectangular group of characters or partial characters. For the rectangular group, the method evaluates the distance between the rectangles and integrates rectangles whose distances therebetween are less than or equal to a predetermined threshold, thereby obtaining a rectangular group of characters. In the case where rectangles of characters of similar sizes exist side by side in a close manner, the method integrates these rectangles to obtain a rectangular group of character lines. In the case where rectangles of character lines with similar shorter sides exist side by side at regular intervals, the method integrates the rectangles to obtain a rectangular group of sentences. In addition, the method also obtains rectangles including objects other than characters, lines and sentences, such as pictures, photos, tables or graphs. The method eliminates isolated rectangles for characters or parts of characters from the rectangles extracted in the above-mentioned manner. The method sets the remaining rectangles as partial regions. Fig. 10A This processing step can extract multiple background partial areas from the background sample image.

[0179] In step S961, the CPU 261 sends the typeface image generated in step S959 and the typeface area acquired in step S960 to the typeface OCR unit 117 via the external interface 268, and causes the typeface OCR unit 117 to perform typeface OCR. Typeface OCR can be implemented by applying known techniques thereto.

[0180] In step S962, the CPU 261 determines whether a lead font OCR result has been received from the lead font OCR unit 117. The lead font OCR result is text data obtained by the lead font OCR unit 117 recognizing the lead font characters included in the lead font area. If the lead font OCR result has been received from the lead font OCR unit 117 via the external interface 268, the CPU 261 sets the determination result to "yes" ("yes" in step S962), and then advances the processing to step S963. If not, the CPU 261 sets the determination result to "no" ("no" in step S962), and then repeats the processing in step S962. Through the processing performed so far, the CPU 261 has obtained the lead font area (coordinate information) and the text data obtained by recognizing the lead font characters included in the lead font area. The CPU 261 stores the obtained lead font area and text data in the RAM 264 as the lead font information table 1006.

[0181] In step S963, CPU 261 integrates the handwriting OCR results and the typeface OCR results received from handwriting OCR unit 116 and typeface OCR unit 117, respectively. CPU 261 evaluates the original positional relationship between the handwriting area and the typeface area and the semantic relationship between the text data, which are the handwriting OCR results and the typeface OCR results, thereby estimating the association between the handwriting OCR results and the typeface OCR results. CPU 261 performs this estimation based on handwriting information table 1003 and typeface information table 1006. Then, CPU 261 identifies the handwriting area included in the name entry column 401 as the handwriting area closest to the typeface area included in the name title 411. CPU 261 can consider not only the distance, but also the positional relationship between the handwriting area and the typeface area, such as the vertical and horizontal relationship. In addition, if the text data of the handwriting OCR result as the handwriting area is a string including the surname in the full name, then CPU 261 can identify the text data as the surname in the full name. Therefore, CPU 261 evaluates the typeface OCR result and the handwriting OCR result as the relationship between the items and the pairs of values ​​including the surname in the full name. Similarly, for the name in the full name, considering the distance between the typeface area and the handwriting area and the handwriting OCR result as the string including the name in the full name, CPU 261 evaluates the typeface OCR result and the handwriting OCR result as the relationship between the items and the pairs of values ​​including the names in the full name. CPU 261 can integrate surname and name into the pair including the items and values ​​about the full name. In addition, similarly, about the address, considering the distance between the typeface area and the handwriting area and the handwriting OCR result as the string including the place name, CPU 261 evaluates the relationship between the typeface OCR result and the handwriting OCR result. Regarding the telephone number, considering the distance between the typeface area and the handwriting area and the fact that the handwriting OCR result consists of digits, the CPU 261 evaluates the relationship between the typeface OCR result and the handwriting OCR result. In this way, the CPU 261 integrates the handwriting OCR result and the typeface OCR result into one or more pairs, each pair including an item and a value, thereby generating the business form text data 1007.

[0182] In step S964 , the CPU 261 sends the generated business form data to the image acquisition unit 111 .

[0183] In step S965, the CPU 261 determines whether to end the processing. In the case where the user has performed a predetermined operation to, for example, power off the image processing server 103, the CPU 261 sets the determination result to "yes" ("yes" in step S965), and then ends the processing. If not, the CPU 261 sets the determination result to "no" ("no" in step S965), and then returns the processing to step S952.

[0184] As described above, according to the first exemplary embodiment, the image processing system 100 uses a neural network having a configuration in which an encoder unit (a layer that performs similar calculations) is shared between a handwriting area estimation neural network and a handwriting extraction neural network. Then, the image processing system 100 estimates a handwriting area from a scanned image of a business form including handwritten characters, thereby generating a handwriting extraction image including only a handwriting portion corresponding to the estimated handwriting area. At this time, since handwritten pixels exist in the handwriting area, the image processing system 100 inputs a feature map obtained in the following manner into a decoder unit for handwriting extraction, the feature map being obtained by narrowing down to only the handwriting area estimated by the decoder unit for handwriting area estimation. This makes it possible to reduce the amount of calculation required to perform handwriting extraction in a non-handwriting area where handwritten pixels do not exist, and thus reduce the amount of calculation used for generation.

[0185] Furthermore, although in the first exemplary embodiment, in the business form to text conversion process, the image processing system 100 performs a process of inputting a feature map obtained by narrowing down to only a handwritten area estimated by the decoder unit for handwritten area estimation to the decoder unit for handwritten extraction, the image processing system 100 may perform a similar process even in the learning process. In that case, even in the learning process, the image processing system 100 calculates an error in the neural network for handwritten area estimation in step S734. Next, the image processing system 100 performs Fig.13 Next, in step S735, the image processing system 100 inputs the feature map obtained by narrowing down to only a part corresponding to the handwriting area estimated by the decoder unit for handwriting area estimation to the decoder unit for handwriting extraction.

[0186] In a second exemplary embodiment, a method is described in which, when a feature map obtained by narrowing down to a handwriting area acquired only by handwriting area estimation is input to a decoder unit for handwriting extraction, a feature map obtained by further narrowing down the area is input. In the first exemplary embodiment, for example, erroneous extraction of a handwriting area or the possibility of a small area where handwriting extraction is not necessary is not considered, which may occur due to, for example, a small pen stain caused by an unintentional contact of a pen with a table. In a second exemplary embodiment, in response to this situation, a method is described in which the area of ​​the feature map to be input to the decoder unit for handwriting extraction is further narrowed. In addition, the configuration of the image processing system 100 according to the second exemplary embodiment is similar to that in the first exemplary embodiment except for the characteristic parts. Therefore, similar constituent elements are assigned the same respective figure marks, and their detailed descriptions are omitted herein.

[0187] <Business Forms to Text Processing>

[0188] The business form to text conversion process performed by the image processing server 103 in the second exemplary embodiment is described. Fig.14A 2 is a flowchart showing the flow of the business form to text conversion process in the second exemplary embodiment. This process is realized by the CPU 261 reading out the image processing server program stored in the storage unit 265, loading the image processing server program into the RAM 264, and executing the image processing server program. This process is started by the user turning on the power of the image processing server 103.

[0189] The processing operations in steps S951 to S954 are the same as Fig. 9B The processing operations in the steps having the corresponding same step numbers in the flowcharts of FIG. 1 are similar.

[0190] In step S1401, the CPU 261 performs handwriting area selection processing on the handwriting area generated in step S954, and updates the number of areas each generated as a handwriting area. Fig. 14B Describe the handwriting area selection process. Fig. 14B is a flowchart showing the flow of handwriting area selection processing in the second exemplary embodiment.

[0191] In step S1451 , the CPU 261 selects one of the handwriting areas estimated in step S954 .

[0192] In step S1452, CPU 261 determines whether the area of ​​the handwriting area selected in step S1451 corresponds to a small area. CPU 261 counts the number of pixels corresponding to the handwriting area, and if the counted number is less than or equal to a predetermined threshold value, CPU 261 sets the determination result to "yes" ("yes" in step S1452), and then advances the process to step S1453. If not, CPU 261 sets the determination result to "no" ("no" in step S1452), and then returns the process to step S1451.

[0193] In step S1453, the CPU 261 records a value indicating that the area is not a handwriting area on the pixel group in the handwriting area generated in step S954 corresponding to the handwriting area determined as "No" in step S1452. This makes it possible to eliminate small areas where handwriting extraction is unnecessary, the area of ​​the small area being less than or equal to a predetermined threshold.

[0194] In step S1454, CPU 261 determines whether all handwriting areas have been processed. If the processing in step S1452 has been performed on all handwriting areas estimated in step S954, CPU 261 sets the determination result to "yes" ("yes" in step S1454), and then ends the handwriting area selection processing. If not, CPU 261 sets the determination result to "no" ("no" in step S1454), and then returns the processing to step S1451, in which CPU 261 selects and processes the next handwriting area.

[0195] As described above, according to the second exemplary embodiment, for the estimated handwriting area, the image processing system 100 can eliminate small areas where handwriting extraction is not necessary. This makes it possible to eliminate areas where handwriting extraction is not necessary, thereby further reducing the amount of calculation required for the neural network 1100 to perform handwriting extraction. In addition, eliminating erroneously extracted areas of handwriting area estimation makes it possible to increase the handwriting OCR recognition rate.

[0196] In the third exemplary embodiment, in addition to the method for converting a business form into a text process described in the first and second exemplary embodiments, a method for determining whether there is handwritten filling in the processing target image is described. In the first and second exemplary embodiments, the feature map obtained by narrowing down to only the handwritten area of ​​the processing target image is input to the decoder unit for handwriting extraction, thereby reducing the amount of calculation in the entire neural network. However, in the case where there is no handwritten filling in the processing target image, useless calculations may be spent on handwritten area estimation. For example, not only can application forms related to service use be distributed at the window of the service provider, but in recent years, their forms can be released to the public via the Internet. The applicant can download the form in advance, print the form, write in the form, and then rush to the window, which can shorten the time of the process at the window. Since the form is an electronic file, the user can also electronically input the filling content into the form and print the form in which the filling content is written. Therefore, in this case, the application form consists of only lead font characters. The service provider processes and converts the business form consisting of handwritten characters and lead font characters or the business form consisting of only lead font characters into text data.

[0197] In the third exemplary embodiment, a method is described as follows: in the above case, before performing handwriting area estimation and handwriting extraction, it is determined whether there is handwriting fill-in in the processing target image, and if there is handwriting fill-in, handwriting area estimation and handwriting extraction are performed. In addition, in the following description, determining whether there is handwriting fill-in in the processing target image is referred to as "handwriting existence or non-existence determination". In addition, determining the existence of handwriting fill-in through handwriting existence or non-existence determination is referred to as "handwriting existence". In addition, determining the absence of handwriting fill-in through handwriting existence or non-existence determination is referred to as "handwriting non-existence". In addition, the configuration of the image processing system 100 according to the third exemplary embodiment is similar to that in the first and second exemplary embodiments except for the characteristic parts. Therefore, similar constituent elements are assigned corresponding identical figure marks, and their detailed descriptions are omitted here.

[0198] <Configuration of Neural Network>

[0199] The configuration of the neural network in the third exemplary embodiment is described. Fig.15 1 is a diagram showing the configuration of the neural network of the image processing system 100 in the third exemplary embodiment. The neural network 1500 in the third exemplary embodiment performs various types of processing operations on the input image. For example, the neural network 1500 performs handwriting presence or absence determination, handwriting area estimation, and handwriting extraction on the input image. Therefore, the neural network 1500 in the third exemplary embodiment has a configuration consisting of a combination of a plurality of neural networks that process respective different tasks. For example, Fig.15 The neural network 1500 shown in has a configuration consisting of a combination of a handwriting presence or absence determination neural network, a handwriting area estimation neural network, and a handwriting extraction neural network. In addition, in the third exemplary embodiment, it is assumed that the input image to the neural network 1500 is a grayscale (single-channel) image, but it may be another type of image, such as a color (three-channel) image.

[0200] like Fig.15 As shown in , the neural network 1500 includes an encoder unit 1101, a pixel extraction decoder unit 1112, a region estimation decoder unit 1122 and a fully connected layer L.

[0201] As in the conventional configuration, the neural network 1500 includes a handwriting extraction neural network composed of an encoder unit 1101 and a pixel extraction decoder unit 1112. In addition, the neural network 1500 also includes a handwriting area estimation neural network composed of the encoder unit 1101 and the area estimation decoder unit 1122. In addition, the neural network 1500 also includes a handwriting presence or absence determination neural network composed of the encoder unit 1101 and the fully connected layer L. In the third exemplary embodiment, the three neural networks share the encoder unit 1101, which is a layer that performs similar calculations in the respective neural networks. Then, switching is performed between the pixel extraction decoder unit 1112, the area estimation decoder unit 1122, and the fully connected layer L depending on the task. After receiving the input image, the neural network 1500 performs calculations in the encoder unit 1101. Then, the neural network 1500 inputs the calculation results to the fully connected layer L, the area estimation decoder unit 1122, and the pixel extraction decoder unit 1112, and then outputs the handwriting presence or absence determination result via the calculation in the fully connected layer L. Furthermore, the neural network 1500 outputs a handwriting area estimation result via calculation in the area estimation decoder unit 1122. Furthermore, the neural network 1500 outputs a handwriting extraction result via calculation in the pixel extraction decoder unit 1112. Moreover, in the third exemplary embodiment, in the case where the fully connected layer L has outputted "handwriting does not exist", the neural network 1500 does not perform calculations in the area estimation decoder unit 1122 and the pixel extraction decoder unit 1112. Details thereof are described below.

[0202] The input to the fully connected layer L is generated by converting the feature map obtained as the final output of the encoder unit 1101 via the pooling layer EP3. This conversion can be performed by using, for example, the following method. When the feature map output from the pooling layer EP3 consists of multiple channels, the method calculates the average value of all pixel values ​​in each channel. Then, the method integrates the average values ​​calculated for the number of channels and converts it into a vector, and sets the vector as the input to the fully connected layer L. In addition, another conversion method can be adopted: for example, each channel is converted from a 2×2 dimensional matrix to a 1×2 dimensional vector, and the vector connected with the vector calculated for the number of channels is set as the input to the fully connected layer L. Then, the fully connected layer L converts the input and passes the converted input through an activation function, thereby outputting a feature variable, which is a value obtained by converting the feature into a numerical value. The neural network 1500 performs a conversion that normalizes the feature variable as the output from the fully connected layer L to a probability distribution, thereby converting the feature variable into the probability of handwriting existence and the probability of handwriting non-existence, and performing handwriting existence or non-existence determination based on the probability. Here, as a method of normalizing to a probability distribution, a softmax function is used, for example. Furthermore, although the output of the pooling layer EP3 is set as the input to the fully connected layer L in the third exemplary embodiment, each of the outputs of the pooling layers EP1 and EP2 may be set as the input to the fully connected layer L.

[0203] <Learning Data Generation Process>

[0204] Next, the learning data generation process performed by the learning device 102 is described. Fig.16A 2 is a flowchart showing the flow of a learning data generation process. This process is implemented by the learning data generation unit 112 of the learning apparatus 102. This process is started by a user performing a predetermined operation via the input device 236 of the learning apparatus 102.

[0205] The processing operations in steps S701 to S708 are the same as Fig. 7A The processing operations in the steps having the corresponding same step numbers in the flowcharts of FIG. 1 are similar.

[0206] In step S1601, CPU 231 determines whether the handwriting extraction correct answer label image generated in step S705 includes pixels having values ​​indicating a handwritten portion. CPU 231 scans the handwriting extraction correct answer label image generated in step S705, and determines whether there is a value indicating a handwritten portion therein. If there is a value indicating a handwritten portion, CPU 231 sets the determination result to "yes" ("yes" in step S1601), and then advances the process to step S1602. If not, CPU 231 sets the determination result to "no" ("no" in step S1601), and then advances the process to step S1603. In addition, the following configuration may be adopted here: a threshold is set in advance regarding the number of pixels having values ​​indicating a handwritten portion, and if the number of pixels whose values ​​indicate a handwritten portion exceeds the threshold, CPU 231 sets the determination result to "yes".

[0207] In step S1602, for the input image generated in step S704 corresponding to the handwriting extraction correct answer label image determined as "yes" in step S1601, the CPU 231 generates a value indicating the presence of handwriting as a correct answer label used in the learning data for handwriting presence or absence determination. In addition, in the following description, the correct answer label is referred to as a "handwriting presence or absence determination correct answer label." Then, the CPU 231 associates the input image and the handwriting presence or absence determination correct answer label with each other, and stores the associated image and label as learning data for handwriting presence or absence determination in a predetermined area of ​​the storage unit 235.

[0208] In step S1603, for the input image generated in step S704 corresponding to the handwriting extraction correct answer label image determined as "No" in step S1601, the CPU 231 generates a value indicating that handwriting does not exist as a handwriting presence or absence determination correct answer label. Then, the CPU 231 associates the input image and the handwriting presence or absence determination correct answer label with each other, and stores the associated image and label in a predetermined area of ​​the storage section 235 as learning data for handwriting presence or absence determination.

[0209] <Learning Process>

[0210] Next, the learning process performed by the learning device 102 is described. Fig. 16B 2 is a flowchart showing the flow of the learning process. This process is implemented by the learning unit 113 of the learning device 102. This process is started by the user performing a predetermined operation via the input device 236 of the learning device 102. In addition, in the third exemplary embodiment, it is assumed that the mini-batch method is used for the learning of the neural network 1500.

[0211] The processing operations in steps S731 to S733 are the same as Figure 7B The processing operations in the steps having the corresponding same step numbers in the flowcharts of FIG. 1 are similar.

[0212] In step S1631, the CPU 231 calculates the error in the handwriting presence or absence determination result of the neural network 1500. Therefore, the CPU 231 inputs the feature map acquired in step S733 into the fully connected layer L, and obtains the output of the fully connected layer L. The output is a probability distribution normalized with respect to the probability of handwriting presence of the input image and the probability of handwriting absence of the input image. Then, the CPU 231 evaluates the difference between the output and the handwriting presence or absence determination correct answer label included in the learning data, thereby obtaining the error. As with handwriting extraction, cross entropy can be used as an indicator for evaluation.

[0213] The processing operations in steps S734 to S735 are the same as those in steps S734 to S735. Figure 7B The processing operations in the steps having the corresponding same step numbers in the flowcharts of FIG. 1 are similar.

[0214] In step S1632, the CPU 231 adjusts the parameters of the neural network 1500. Therefore, the CPU 231 changes the values ​​of the parameters of the neural network 1500 by the back propagation method based on the errors calculated in steps S1631, S734, and S735.

[0215] <Business Forms to Text Processing>

[0216] Next, the business form-to-text conversion process performed by the image processing server 103 is described. Fig.17 2 is a flowchart showing the flow of the business form to text conversion process. The business form to text conversion process is realized by the CPU 261 reading the image processing server program stored in the storage unit 265, loading the image processing server program onto the RAM 264, and executing the image processing server program. This process is started when the user powers on the image processing server 103.

[0217] The processing operations in steps S951 to S953 are the same as Fig. 9B The processing operations in the steps having the corresponding same step numbers in the flowcharts of FIG. 1 are similar.

[0218] In step S1701, the CPU 261 performs handwriting presence or absence determination on the processing target image received from the image processing apparatus 101. The CPU 261 inputs the feature map acquired in step S953 to the fully connected layer L, thus causing the neural network 1500 to determine the presence or absence of handwriting. As an output of the neural network 1500, in the case where the presence of handwriting has been acquired as a determination result, a value indicating the presence of handwriting is output, and in the case where the absence of handwriting has been acquired as a determination result, a value indicating the absence of handwriting is output.

[0219] In step S1702, the CPU 261 determines whether handwriting exists in the processing target image. If a value indicating the presence of handwriting has been obtained in the handwriting presence or absence determination performed in step S1701, the CPU 261 sets the determination result to "yes" ("yes" in step S1702), and then advances the process to step S954. If not, the CPU 261 sets the determination result to "no" ("no" in step S1702), and then advances the process to step S959.

[0220] The processing operations in steps S954 to S958 are the same as Fig. 9B The processing operations in the steps having the corresponding same step numbers in the flowcharts of FIG. 1 are similar.

[0221] In step S959, the CPU 261 generates a typeface image. In the case where the determination result in step S1702 is "Yes", the CPU 261 generates a typeface image by Fig. 9B The handwritten portion is removed from the processing target image in the processing operation in the steps with the same step number in the flowchart of the corresponding step number to generate a lead type font image. In the case where the determination result in step S1702 is "No", the CPU 261 directly sets the processing target image to a lead type font image.

[0222] The processing operations in steps S960 to S965 are the same as those in steps S961 to S963. Fig. 9B The processing operations in the steps having the corresponding same step numbers in the flowcharts of FIG. 1 are similar.

[0223] As described above, according to the third exemplary embodiment, before performing handwriting area estimation and handwriting extraction, the image processing system 100 determines whether there is handwriting filling in the processing target image, and if there is handwriting filling, performs handwriting area estimation and handwriting extraction. This makes it possible to reduce the amount of calculation required in the case where there is no handwriting filling in the processing target image.

[0224] In the fourth exemplary embodiment, a case where the neural network and the configuration in the present disclosure are applied to a task different from the tasks in the first, second, and third exemplary embodiments is described. In addition, the configuration in the present disclosure is a configuration in which a feature map obtained by narrowing down to a region including only a target recognition object estimated by a region estimation decoder unit is input to a decoder unit that extracts pixels of the target recognition object. In the fourth exemplary embodiment, a case in which a seal image (an image of an imprinted seal) included in a document is taken as a target recognition object is described. Therefore, an example of a configuration is described in which a neural network is used for a process of extracting pixels constituting a seal image and a process of estimating a region including the seal image. For example, with respect to an account transfer request form, a seal included in a business form can be verified relative to a seal registered in, for example, a database. According to the fourth exemplary embodiment, the image processing system 100 is capable of extracting a region including a seal and its pixels, thereby automating the extraction of a seal as a verification target. In addition, the image processing system 100 is capable of separating a printed portion (a printed portion for guiding a seal imprint position, such as "LS") used as a seal background from the seal from each other, thereby enabling verification to be performed with higher accuracy.

[0225] In addition, in the following description, estimating the area including the seal is referred to as "seal area estimation", and extracting the seal for each pixel is referred to as "seal extraction". Moreover, the image processing system 100 in the fourth exemplary embodiment is a system in which the extraction target and the area estimation target in the first, second and third exemplary embodiments are changed from handwritten characters to seals. Therefore, the configuration and processing operation of the image processing system 100 in the fourth exemplary embodiment are similar to those in the first, second and third exemplary embodiments, and the "handwritten part", "handwritten characters" and "filling in of handwritten characters" described in the first, second and third exemplary embodiments are replaced with "seals". In the following description, the details of the learning data used in the learning process of the neural network are described, which is the difference between the fourth exemplary embodiment and the first, second and third exemplary embodiments. In addition, the details of the seal verification request process and the seal verification process are described. In addition, similar constituent elements between the fourth exemplary embodiment and the first, second and third exemplary embodiments are assigned corresponding identical reference numerals, and their detailed description is omitted here.

[0226] <Operation Screen>

[0227] As in the first exemplary embodiment, the user performs operations based on the seal extraction correct answer data creation screen and the seal area estimation correct answer data creation screen, thereby creating correct answer data. In addition, the seal extraction correct answer data creation screen and the seal area estimation correct answer data creation screen are screens in which Figure 5A-5DIn the screen shown in , the target specified by the user is changed to a seal, and the operation performed by the user on it is similar to the above-mentioned operation. In addition, the document sample image in the fourth exemplary embodiment is, for example, Fig.18A Business Form 1800 or Fig.18B Business form 1850 shown in .

[0228] The correct answer data for seal extraction is accumulated in the image accumulation unit 115 as the following image. The image has the same size (width and height) as the document sample image. The value of each pixel of the seal position selected by the user is a value indicating a seal (for example, "255", and this also applies to the following description). The value of each of the other pixels is a value not indicating a seal (for example, "0", and this also applies to the following description). Hereinafter, this image used as the correct answer data for seal extraction is referred to as a "seal extraction correct answer image". Fig.18C An example of a seal extraction correct answer image is shown.

[0229] In addition, the correct answer data of the seal area estimation is accumulated in the image accumulation unit 115 as the following image. The image has the same size (width and height) as the document sample image. The value of each pixel corresponding to the seal area selected by the user is a value indicating the seal area (for example, "255", and this also applies to the following description). The value of each of the other pixels is a value that does not indicate the seal area (for example, "0", and this also applies to the following description). Hereinafter, this image used as the correct answer data of the seal area estimation is referred to as a "seal area estimation correct answer image". Fig.18D An example of a seal area estimation correct answer image is shown.

[0230] <Learning Data Generation Process>

[0231] A learning data generation process similar to that in the first exemplary embodiment is implemented by the learning data generation unit 112 of the learning apparatus 102. This process is started by a user performing a predetermined operation via the input device 236 of the learning apparatus 102.

[0232] As in the first exemplary embodiment, the CPU 231 generates a correct answer label image used in the learning data for seal extraction (hereinafter, the correct answer label image is referred to as a "seal extraction correct answer label image"), and stores the seal extraction correct answer label image as learning data for seal extraction in a predetermined area of ​​the storage section 235. In the fourth exemplary embodiment, the storage unit 235 stores the correct answer label image such as the seal extraction correct answer label image. Fig.19A The learning data shown in .

[0233] Furthermore, as in the first exemplary embodiment, the CPU 231 generates a correct answer label image used in the learning data for seal area estimation (hereinafter, the correct answer label image is referred to as a "seal area estimation correct answer label image"), and stores the seal area estimation correct answer label image as the learning data for seal area estimation in a predetermined area of ​​the storage section 235. In the fourth exemplary embodiment, the storage unit 235 stores the seal area estimation correct answer label image. Fig.19B The learning data shown in .

[0234] <Seal verification request processing>

[0235] Next, the seal verification request process performed by the image processing device 101 is described. The image processing device 101 scans a business form on which a seal is imprinted to generate a processing target image. Then, the image processing device 101 sends the processing target image to the image processing server 103 and requests the image processing server 103 to perform seal verification. Fig. 20A 2 is a flowchart showing the flow of seal verification request processing. This processing is realized by the CPU 201 of the image processing apparatus 101 reading out a controller program stored in the storage unit 208, loading the controller program onto the RAM 204, and executing the controller program. This processing is started by the user performing a predetermined operation via the input device 209 of the image processing apparatus 101.

[0236] The processing operations in steps S901 to S903 are the same as Fig.9A The processing operations in the steps having the corresponding same step numbers in the flowcharts of FIG. 1 are similar.

[0237] In step S2001, the CPU 201 outputs the processing result received from the image processing server 103, that is, the result of verifying the seal included in the processing target image generated in step S901. For example, the CPU 201 transmits the verification result to the transmission destination set by the user operating the input device 209 via the external interface 211.

[0238] <Seal verification processing>

[0239] Next, the seal authentication processing executed by the image processing server 103 is described. Fig. 20Bis a flowchart showing the flow of seal verification processing. In addition, the image processing system 100 in the fourth exemplary embodiment includes a seal verification server (not shown) instead of the OCR server 104 described in the first exemplary embodiment. The seal verification server includes a seal database that registers and accumulates a seal image and registrant information about the seal (e.g., name, address, and account number) while associating the two with each other. In addition, the seal verification server includes a seal verification unit that verifies the sent seal image relative to the seal image previously registered in the seal database, and outputs a result indicating whether the seal image is currently registered. The image processing server 103 serving as the image conversion unit 114 receives the processing target image from the image processing device 101, and verifies the seal included in the scanned image data using the seal database included in the seal verification server, thereby obtaining a verification result. The seal verification server causes the seal verification unit to verify the seal using the seal database. The seal verification process is realized by the CPU 261 reading out the image processing server program stored in the storage unit 265, loading the image processing server program onto the RAM 264, and executing the image processing server program. This process is started when the user powers on the image processing server 103.

[0240] The processing operation in steps S951 to S956 is similar to the following processing operation, in which Fig. 9B In the steps with the same corresponding step numbers in the flowchart, "handwriting" is replaced by "stamp".

[0241] In step S2051, the CPU 261 sends the seal extraction image generated in step S956 to the seal verification unit via the external interface 268. Then, the CPU 261 causes the seal verification unit to perform seal verification. Seal verification can be achieved by applying known techniques thereto. Performing seal verification on the seal extraction image makes it possible to prevent or reduce the printed portion for guiding the seal imprinting position, such as "LS" in the seal background, from becoming the target of seal verification.

[0242] In step S2052, CPU 261 determines whether a seal verification result has been received from the seal verification unit. The seal verification result is a value indicating whether the seal extraction image is currently registered in the seal database. For example, in the case where the seal extraction image is currently registered, the seal verification result represents "1" as output, and in the case where the seal extraction image is not currently registered, the seal verification result represents "0" as output. If the seal verification result has been received from the seal verification unit via external interface 268, CPU 261 sets the determination result to "yes" ("yes" in step S2052), and then advances the processing to step S2053. If not, CPU 261 sets the determination result to "no" ("no" in step S2052), and then repeats the processing in step S2052.

[0243] In step S2053 , the CPU 261 transmits the seal authentication result received in step S2052 to the image acquisition unit 111 .

[0244] In step S2054, the CPU 261 determines whether to end the processing. In the case where the user performs a predetermined operation such as powering off the image processing server 103, the CPU 261 sets the determination result to "yes" ("yes" in step S2054), and then ends the processing. If not, the CPU 261 sets the determination result to "no" ("no" in step S2054), and then returns the processing to step S952.

[0245] As described above, according to the fourth exemplary embodiment, the neural network and configuration of the present disclosure are not limited to handwriting area estimation and handwriting extraction, but can also be applied to seal area estimation and seal extraction. Here, the configuration of the present disclosure is a configuration for inputting a feature map obtained by narrowing down to a target recognition area estimated only by a recognition target area estimation decoder unit to a decoder unit for extracting target recognition pixels. As another example of application, when a business form consisting only of lead font characters is assumed, the neural network and configuration of the present disclosure can also be applied to estimation of a lead font area and to lead font pixel extraction for extracting lead font pixels contained in a lead font area, which lead font area is an item that includes only lead font characters.

[0246] The present disclosure may also be implemented by the following process: supplying a program for implementing one or more functions of the above exemplary embodiments to a system or device via a network or storage medium, and causing one or more processors included in the system or device to read out and execute the program. In addition, the present disclosure may also be implemented by a circuit (e.g., an application specific integrated circuit (ASIC)) that implements such one or more functions.

[0247] The present disclosure can also be applied to a system composed of multiple devices and a system composed of one device. For example, in the above exemplary embodiment, the learning data generating unit 112 and the learning unit 113 are configured to be implemented in the learning device 102, but can be configured to be implemented in separate devices. In this case, the device for implementing the learning data generating unit 112 sends the learning data generated by the learning data generating unit 112 to the device for implementing the learning unit 113. Then, the learning unit 113 performs learning of the neural network based on the received learning data. Although the image processing device 101 and the image processing server 103 have been described as separate devices, the image processing device 101 can be configured to include the functions of the image processing server 103. Although the image processing server 103 and the OCR server 104 have been described as separate devices, the image processing server 103 can be configured to include the functions of the OCR server 104.

[0248] The present disclosure is not limited to the above exemplary embodiments, but can be modified in various ways (including organic combinations of some or all of the above exemplary embodiments) based on the gist of the present disclosure, and such modifications are not intended to be excluded from the scope of the present disclosure. Therefore, all configurations obtained by combining some or all of the above exemplary embodiments and their variations are included in the present disclosure.

[0249] Although in the above exemplary embodiment, a method of determining extraction of a type font area based on the connectivity of pixels as in step S959 has been described, a type font area may be estimated using a neural network as in handwriting area estimation. For example, a user selects a type font area in the same manner as creating a handwriting area estimation correct answer image, creates correct answer data based on the type font area, constructs a new neural network that performs type font OCR area estimation, and causes the neural network to perform learning based on the correct answer data.

[0250] In the above exemplary embodiment, learning data is generated by the learning data generation process at the time of learning processing. However, a configuration may be adopted in which a large amount of learning data is generated in advance by the learning data generation process, and at the time of learning processing, learning data corresponding to the minimum batch size is sampled as needed.

[0251] In the above-described exemplary embodiments, the input image is generated as a grayscale image, but may be generated as another type of image, such as a full-color image.

[0252] Furthermore, finally, an example of an estimation result obtained by performing learning by a neural network for handwriting extraction and a neural network for handwriting area estimation according to the method described in the present disclosure is shown. Fig.21AAn example of processing a target image is shown. Fig.21B An example of the estimation results obtained by the neural network for handwriting area estimation and the neural network for handwriting extraction is shown. Fig.21A The target image is processed as shown in Fig.21B The estimation results are shown in . Fig.21B , handwritten characters (character parts) extracted by the neural network for handwriting extraction and handwritten areas (white shaded parts) estimated by the neural network for handwriting area estimation are shown.

[0253] Other embodiments

[0254] The (one or more) embodiments of the present disclosure may also be implemented by a computer or method of a system or device, the computer of the system or device reads and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be more completely referred to as a "non-transitory computer-readable storage medium") to perform one or more functions of (one or more) of the above embodiments and / or includes one or more circuits (e.g., application-specific integrated circuits (ASICs)) for performing one or more functions of (one or more) of the above embodiments, and the method is performed by a computer of the system or device, for example, by reading and executing computer executable instructions from a storage medium to perform one or more functions of (one or more) of the above embodiments and / or controlling one or more circuits to perform one or more functions of (one or more) of the above embodiments. The computer may include one or more processors (e.g., a central processing unit (CPU), a microprocessing unit (MPU)), and may include a network of separate computers or separate processors to read and execute computer executable instructions. Computer executable instructions may be provided to a computer, for example, from a network or a storage medium. The storage medium may include, for example, a hard disk, a random access memory (RAM), a read-only memory (ROM), a storage portion of a distributed computing system, an optical disk (such as a compact disk (CD), a digital versatile disk (DVD), or a Blu-ray disk (BD)). TM ), one or more of a flash memory device, a memory card, etc.

[0255] The embodiments of the present invention may also be implemented by providing software (program) for performing the functions of the above-described embodiments to a system or device via a network or various storage media, and a computer or a central processing unit (CPU) or a microprocessing unit (MPU) of the system or device reads and executes the program.

[0256] While the present disclosure has been described with reference to exemplary embodiments, it is to be understood that the present disclosure is not limited to the disclosed exemplary embodiments.The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. An image processing device, comprising: A first obtaining component, used for obtaining a region including a target recognition object included in a processing target image using a neural network; as well as a second obtaining component, for obtaining pixels corresponding to the target recognition object included in the processing target image using a neural network, Wherein, the neural network comprises: an encoder unit configured to obtain a feature map by performing a convolution process on the processing target image; A first decoder unit is configured to obtain a region including the target recognition object based on the feature map; a partial feature map obtaining unit configured to obtain, from the feature map, a partial feature map corresponding to a portion of the area including the target recognition object obtained by the first decoder unit; and The second decoder unit is configured to obtain pixels corresponding to the target recognition object based on the partial feature map.

2. The image processing apparatus according to claim 1, wherein: The partial feature map obtaining unit obtains a partial feature map corresponding to a portion of an area excluding an area having a predetermined number of pixels or less from among the area including the target recognition object obtained by the first decoder unit.

3. The image processing apparatus according to claim 1, wherein: The target recognition object includes handwritten characters.

4. The image processing apparatus according to claim 1, wherein: The target identification object includes an embossed seal.

5. A computer-readable storage medium storing instructions, which, when executed by a computer, cause the computer to perform the following operations: Using a neural network to obtain a region including a target recognition object included in a processing target image; as well as obtaining pixels corresponding to the target recognition object included in the processing target image using a neural network, in, The neural network comprises: an encoder unit configured to obtain a feature map by performing a convolution process on the processing target image; A first decoder unit is configured to obtain a region including the target recognition object based on the feature map; a partial feature map obtaining unit configured to obtain, from the feature map, a partial feature map corresponding to a portion of the area including the target recognition object obtained by the first decoder unit; and The second decoder unit is configured to obtain pixels corresponding to the target recognition object based on the partial feature map.

6. An image processing method, comprising: Using a neural network to obtain a region including a target recognition object included in a processing target image; as well as obtaining pixels corresponding to the target recognition object included in the processing target image using a neural network, Wherein, the neural network comprises: an encoder unit configured to obtain a feature map by performing a convolution process on the processing target image; A first decoder unit is configured to obtain a region including the target recognition object based on the feature map; a partial feature map obtaining unit configured to obtain, from the feature map, a partial feature map corresponding to a portion of the area including the target recognition object obtained by the first decoder unit; and The second decoder unit is configured to obtain pixels corresponding to the target recognition object based on the partial feature map.

Images