Character recognition processing system, character recognition processing method, and program

Abstract

To provide a character recognition processing system, a character recognition processing method, and a program with which an appropriate result of character recognition processing can be obtained corresponding to each of a plurality of different types of character recognition object images.SOLUTION: In a character recognition processing system comprising user terminals used respectively in manufacturing plants and a character recognition server, the character recognition server 200 comprises: a type determination unit 222 that determines whether a character recognition object image based on a picked-up image is of a first type in which a recognition object character string is a single row or of a second type in which a plurality of recognition object character strings are arranged at predetermined positions according to a predetermined format; and a character recognition processing unit 223 that executes first character recognition pre-processing corresponding to the first type on the character recognition object image determined to be of the first type, executes second character recognition pre-processing corresponding to the second type on the character recognition object image determined to be of the second type, and recognizes characters from the character recognition object image on which the first character recognition pre-processing or the second character recognition pre-processing is executed.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to a character recognition processing system, a character recognition processing method, and a program.

Background Art

[0002] There is known a technique for structuring character information extracted from a document image obtained by scanning a paper form with a scanner (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In character recognition processing, as character recognition target images, there are cases corresponding to each of a type that is a single-line string such as the expiration date of a food product, etc., and a type in which strings are arranged for each of a plurality of predetermined items according to a prescribed format such as a form. When performing character recognition processing corresponding to a plurality of different types of character recognition target images, it is required to obtain appropriate character recognition processing results for each type.

[0005] The present invention has been made in view of such circumstances, and an object thereof is to obtain appropriate character recognition processing results corresponding to each of a plurality of different types of character recognition target images.

Means for Solving the Problems

[0006] One aspect of the present invention for solving the above-described problems is When capturing a single line of text, the type of text to be recognized is determined based on the character recognition target image corresponding to the frame area of ​​the captured image captured by the user so that the single line of text to be recognized fits within the frame area, or based on the character recognition target image corresponding to the image area of ​​the captured image captured by the user so that multiple texts to be recognized fit within the image area.A type determination unit that determines whether the string to be recognized is of type 1, which consists of a single line, or type 2, which consists of multiple strings to be recognized arranged in predetermined positions according to a predetermined format, The type of character recognition target is Character recognition determined to be of the first type subject For images, perform the first character recognition preprocessing corresponding to the first type mentioned above. The type of character recognition target is The character recognition processing system comprises a character recognition processing unit that performs a second character recognition preprocessing corresponding to the second type for character recognition target images determined to be of the second type, and recognizes characters from character recognition target images that have undergone the first character recognition preprocessing or the second character recognition preprocessing.

[0007] One aspect of the present invention is a character recognition processing method in a character recognition processing system, When capturing a single line of text, the type of text to be recognized is determined based on the character recognition target image corresponding to the frame area of ​​the captured image captured by the user so that the single line of text to be recognized fits within the frame area, or based on the character recognition target image corresponding to the image area of ​​the captured image captured by the user so that multiple texts to be recognized fit within the image area. A type determination step that determines whether the string to be recognized is of type 1, which consists of a single line, or type 2, which consists of multiple strings to be recognized arranged in predetermined positions according to a predetermined format, The type of character recognition target is Character recognition determined to be of the first type subject For images, perform the first character recognition preprocessing corresponding to the first type mentioned above. The type of character recognition target is The character recognition processing method comprises the steps of: performing a second character recognition preprocessing corresponding to the second type for character recognition target images determined to be of the second type; and recognizing characters from character recognition target images that have undergone the first character recognition preprocessing or the second character recognition preprocessing.

[0008] One aspect of the present invention relates to a computer in a character recognition processing system, When capturing a single line of text, the type of text to be recognized is determined based on the character recognition target image corresponding to the frame area of ​​the captured image captured by the user so that the single line of text to be recognized fits within the frame area, or based on the character recognition target image corresponding to the image area of ​​the captured image captured by the user so that multiple texts to be recognized fit within the image area. A type determination unit that determines whether the string to be recognized is of type 1, which consists of a single line, or type 2, which consists of multiple strings to be recognized arranged in predetermined positions according to a predetermined format. The type of character recognition target is Character recognition determined to be of the first type subject For images, perform the first character recognition preprocessing corresponding to the first type mentioned above. The type of character recognition target isThis program performs a second character recognition preprocessing corresponding to the second character recognition type on character recognition target images that have been determined to be of the second type, and functions as a character recognition processing unit that recognizes characters from character recognition target images that have undergone the first character recognition preprocessing or the second character recognition preprocessing. [Effects of the Invention]

[0009] As described above, the present invention provides the effect of obtaining appropriate character recognition processing results for each of several different types of character recognition target images. [Brief explanation of the drawing]

[0010] [Figure 1] This figure shows an example of the configuration of the character recognition processing system in this embodiment. [Figure 2] This figure shows an example of the application screen of the character recognition support application displayed on the user terminal of this embodiment. [Figure 3] This figure shows an example of the configuration of the character recognition server in this embodiment. [Figure 4] This figure shows an example of format data in this embodiment. [Figure 5] This figure shows an example of a string rule table in this embodiment. [Figure 6] This figure shows an example of a character conversion table in this embodiment. [Figure 7] This figure shows an example of the processing procedure that the character recognition server in this embodiment performs in response to character recognition. [Figure 8] This flowchart shows an example of the processing procedure that the character recognition server in this embodiment executes in response to binarization. [Figure 9] This flowchart shows an example of the processing procedure that the character recognition server in this embodiment performs in response to tilt correction. [Figure 10] This flowchart shows an example of the processing procedure that the character recognition server in this embodiment executes in response to character correction. [Figure 11] This is a diagram showing an image for character recognition of a single-line character string type in this embodiment. [Figure 12] This is a diagram showing an image for character recognition of a formatted document type in this embodiment. [Figure 13] This is a diagram for explaining an example of inclination correction in this embodiment. [Figure 14] This is a diagram for explaining an example of generation of a combined character string in this embodiment.

Mode for Carrying Out the Invention

[0011] [Configuration Example of Character Recognition Processing System] FIG. 1 shows a configuration example of the character recognition processing system of this embodiment. The character recognition processing system of this embodiment performs character recognition processing by OCR (Optical Character Reader) on a predetermined target in the manufacturing factory FC as support for product manufacturing management in the manufacturing factory FC, and enables a user such as an administrator in the manufacturing factory FC to confirm the character recognition result.

[0012] The character recognition processing system of this embodiment includes a user terminal 100 used in each manufacturing factory FC and a character recognition server २००.

[0013] The user terminal 100 is used to image a character recognition target in the manufacturing factory FC. Further, the user terminal 100 can display the character recognition processing result. The user terminal 100 may be configured to operate a character recognition support application corresponding to the character recognition processing system of this embodiment. The character recognition support application can transmit an image for character recognition obtained according to the user's imaging operation to the character recognition server 200 and display the character recognition processing result executed on the image for character recognition received by the character recognition server 200.

[0014] The user terminal 100 may be, for example, a mobile device such as a smartphone or tablet owned by the user. In this case, the user operates the user terminal 100 to capture images of the object to be recognized. Alternatively, the user terminal may consist of a camera that captures images of the object to be recognized on a product being transported, for example, by a conveyor belt, and a computer device connected to the camera.

[0015] In this embodiment, the character recognition target may be, for example, the expiration date printed on the product packaging. Alternatively, the character recognition target may be information related to manufacturing management, such as temperature, displayed on manufacturing equipment or manufacturing management equipment. Such expiration dates and information such as temperature displayed on equipment are recognized as single-line strings.

[0016] Furthermore, the objects of character recognition may be labels attached to products, etc. Also, the character recognition information may be forms related to orders, purchase orders, etc. For such labels and forms, the characters included in each item, which are placed in a predetermined position according to a prescribed format (style), within the scope of the label or form, will be the targets of character recognition.

[0017] The character recognition support system of this embodiment is capable of performing character recognition processing for both single-line strings (single-line strings: an example of the first type) and documents with a defined format, such as labels and forms (formatted documents: an example of the second type), as types of characters to be recognized.

[0018] [About the character recognition support application] Figure 2 shows an example of the application screen of a character recognition support application displayed on the user terminal 100. Figure 2(A) is an example of the application screen when a single-line string is the target of character recognition, and Figure 2(B) is an example of the application screen when a formatted document is the target of character recognition. In the application screen shown in the figure, the captured image area AR1 is the area where the captured image taken by the camera is displayed. Within the captured image area AR1, a frame area AR2 is placed to serve as a guide when capturing single-line text.

[0019] Frame area AR2 is located within the captured image area AR1. Frame area AR2 represents the portion of the captured image that is captured as the image to be used for character recognition, corresponding to the case where a single line of text is the target of character recognition.

[0020] When the user chooses a single-line string as the target for character recognition, they perform the image capture operation so that the string to be recognized fits within the frame area AR2, as shown in Figure 2(A). When the user chooses a formatted document as the target for character recognition, they perform the image capture operation so that the formatted document fits within the captured image area AR1. In response to the image capture operation, the user terminal 100 transmits to the character recognition server 200, as character recognition target images, the image portion corresponding to the captured image area AR1 and the image portion extracted from the captured image corresponding to the frame area AR2. The user terminal 100 may also transmit the image portion corresponding to the captured image area AR1 and coordinate information corresponding to the frame area AR2 within that image portion.

[0021] The character recognition server 200 determines, based on the received image to be recognized, whether the subject of the recognition is a single-line string or a formatted document, and performs character recognition processing according to the determined subject. The character recognition server 200 then transmits the string obtained as a result of the character recognition processing to the user terminal 100. When the user terminal 100 receives a string as a result of character recognition processing, it displays the received string in the recognition result area AR3 on the application screen. Alternatively, the user terminal 100 may store the received string as a result of character recognition processing in its own memory. Or, the user terminal 100 may transmit the character recognition processing result to a predetermined device, server, etc. (not shown) in the manufacturing plant FC. The equipment and servers in the manufacturing plant FC may store the transmitted character recognition results. The character recognition results stored in the user terminal 100 or the equipment and servers within the manufacturing plant FC may be used for various manufacturing management purposes at the manufacturing plant FC. When the character recognition server 200 selects a formatted document as the target for character recognition, it transmits information about the strings of characters for each predetermined item recognized in the captured formatted document to the user terminal 100 as the character recognition processing result. In this case as well, the user terminal 100 may store the received character recognition processing results in its own storage unit or in a device, server, etc., within the manufacturing plant FC. In this case as well, the user terminal 100 may display the character recognition processing results in a predetermined manner. For example, the user terminal 100 may display a list of recognized strings for each item name that is the target of character recognition in the formatted document.

[0022] [Example of a character recognition server configuration] Figure 3 shows an example of the functional configuration of the character recognition server 200. The character recognition server 200 in this figure comprises a communication unit 201, a control unit 202, and a storage unit 203. The communications unit 201 communicates with the user terminal 100 via the network.

[0023] The control unit 202 performs various controls on the character recognition server 200. The control unit 202 includes an image acquisition unit 221, a type determination unit 222, a character recognition processing unit 223, and a processing result transmission unit 224. The image acquisition unit 221 acquires the image to be recognized for character recognition that has been sent from the user terminal 100.

[0024] The type determination unit 222 determines whether the type of the image to be recognized for character recognition, acquired by the image acquisition unit 221, is a single-line string or a formatted document.

[0025] The character recognition processing unit 223 performs character recognition processing on the acquired image to be recognized. The character recognition processing unit 223 performs processing according to the type of image to be recognized determined by the type determination unit 222. In other words, if the type determination unit 222 determines that the image to be recognized is a single-line string, the character recognition processing unit 223 performs character recognition processing corresponding to a single-line string, and if the type determination unit 222 determines that the image to be recognized is a formatted document, the character recognition processing unit 223 performs character recognition processing corresponding to a formatted document.

[0026] The processing result transmission unit 224 transmits the character recognition processing result, which indicates the character string recognized by the character recognition processing unit 223, to the user terminal 100 that sent the image to be recognized.

[0027] The memory unit 203 stores various types of information that the character recognition server 200 supports. The memory unit 203 includes a format database memory unit 231, a string rule table memory unit 232, and a character conversion table memory unit 233. The format database storage unit 231 stores the format database. The format database is a database of formats for each formatted document registered from the manufacturing plant FC as a target for character recognition.

[0028] The format database stores format data for each format. Figure 4 shows an example of format data corresponding to one format. The format data in this figure includes areas for format ID, format name, image data, and item data.

[0029] The format ID field stores a format ID that uniquely identifies the corresponding format.

[0030] The format name field stores the name of the format assigned to the corresponding format (format name).

[0031] The image data area stores image data of formatted documents in the corresponding format. The image data of the formatted documents may be, for example, an image file obtained by converting the formatted document file into an image file.

[0032] The item data area stores the corresponding data (item data) for each item designated as a target for character recognition in the corresponding format.

[0033] The item data corresponding to a single item includes the item ID, item name, and coordinate area. The item ID field stores the item ID that uniquely identifies the corresponding item. The item name area stores the name (item name) assigned to the corresponding item. The coordinate area stores the coordinates where the corresponding items are located in the formatted document. The coordinates may be defined, for example, by setting a bounding rectangle around the string placed in the corresponding item in the formatted document and indicating the position of that bounding rectangle.

[0034] Furthermore, the format data corresponding to one format may, for example, store the factory ID that indicates the manufacturing plant FC that registered the corresponding format.

[0035] Let's return to the explanation in Figure 3. The string rule table storage unit 232 stores the string rule table. The string rule table stores the string rule corresponding to each single-line string that each manufacturing plant FC targets for character recognition, and the string rule for each item in the formatted document that each manufacturing plant FC has registered as a target for character recognition.

[0036] Figure 5 shows an example of a string rule table. Figure 5(A) is a string rule table when single-line strings are the target of character recognition. A string rule table corresponding to a single-line string has a structure in which one or more string rules are defined for a single-line string. A sequence number is assigned to each string rule corresponding to a single-line string. Figure 5(B) shows the string rule table corresponding to the formatted document. The string rule table for a formatted document stores a format ID corresponding to each formatted document, and for each format ID, it stores one or more item IDs included in the formatted document and one or more string rules corresponding to each item ID. The one or more string rules corresponding to the item indicated by a single item ID are assigned a sequence number in ascending order from 1. String rules specify conditions for characters or ranges of characters, for example, for each digit of a character string. Note that for single-line strings or item strings that do not have string rules, the corresponding string rules do not need to be stored in the string rule table.

[0037] Let's return to the explanation in Figure 3. The character conversion table storage unit 233 stores the character conversion table. The character conversion table is used to correct the recognition of characters recognized by the character recognition process, as will be described later.

[0038] Figure 6 shows an example of a character conversion table. In this character conversion table, one or more conversion candidate characters are associated with each character to be converted. Furthermore, a priority (priority order) is associated with each conversion candidate character associated with a single character to be converted.

[0039] [Example of processing procedure] Refer to the flowcharts in Figures 7 to 10 to explain an example of the processing procedure performed by the character recognition server 200. In addition, Figures 11 to 14 will be referred to as appropriate in the explanation of Figures 7 to 10 to explain the processing.

[0040] The flowchart in Figure 7 shows an example of the processing procedure that the character recognition server 200 performs in response to character recognition. Step S100: The user terminal 100, on which the character recognition support application is installed, transmits the image of the character to be recognized to the character recognition server 200 when it has captured an image of the character to be recognized. The image acquisition unit 221 in the character recognition server 200 acquires the image of the character to be recognized transmitted from the user terminal 100.

[0041] Step S102: The type determination unit 222 determines whether the type of character recognition target corresponding to the character recognition target image acquired in step S100 is a single-line string or a formatted document.

[0042] Referring to Figures 11 and 12, an example of determining the type of character to be recognized in step S102 will be explained. Figure 11 shows the image P1 targeted for character recognition as a single-line string. Figure 12 shows the image P2 targeted for character recognition as a formatted document. Image P1 corresponds to the image portion extracted corresponding to frame area AR2. Image P2 corresponds to the image portion corresponding to captured image area AR1. In the following explanation, unless otherwise specified, the image P1 and P2 targeted for character recognition will be referred to simply as "image P." In step S102, the type determination unit 222 measures the height h1 of the character recognition target image P1 acquired in step S100, and measures the largest value among the heights corresponding to the character strings present in the character recognition target image P1 as height h2. The type determination unit 222 compares h2 / h1 with a predetermined threshold K. If h2 / h1 is greater than the threshold K, the type determination unit 222 determines that the character recognition target is of the type of single-line character string, and if h2 / h1 is less than or equal to the threshold K, it determines that the character recognition target is of the type of formatted document.

[0043] Furthermore, the type determination unit 222 may perform the type determination of the character recognition target in step S102 as follows. The type determination unit 222 searches for the feature quantity that is closest to the feature quantity of the character recognition target image P2 obtained in step S100, from among the image data feature quantities stored in the format database corresponding to each format, similar to step S112 above. If no image data features with an approximate value greater than a predetermined threshold are found, it means that there is no image data with the same content and format as the character recognition target image P2. In this case, the type determination unit 222 may determine that the type of character recognition target is a single-line string. On the other hand, if image data features with a matching format are found for the character recognition target image P, then the image data features determined to match are corresponding to a formatted document. In this case, the type determination unit 222 determines that the type of character recognition target is a formatted document. Furthermore, if the type determination in step S102 is performed in this manner, and the type of document to be recognized is determined to be a formatted document, then the format identification is considered to have been performed at the stage when the determination result in step S102 is obtained. In this case, the processing in step S112 in Figure 7 may be omitted.

[0044] Return to Figure 7 for the explanation. Step S104: The type determination unit 222 determines whether the type of character recognition target determined in step S104 is a single-line string or a formatted document.

[0045] Step S106: If step S104 determines that the type of character to be recognized is a single-line string, the character recognition processing unit 223 executes the preprocessing steps S106 to S110 (an example of the first character recognition preprocessing). As preprocessing corresponding to the type of single-line string, the character recognition processing unit 223 first performs binarization of the character recognition target image P in step S106.

[0046] Referring to the flowchart in Figure 8, an example of the binarization process procedure as step S106 will be explained. Step S200: The character recognition processing unit 223 resizes the character recognition target image P1 corresponding to the type of single-line string. As part of the resizing, the character recognition processing unit 223 changes the height (vertical) and width (horizontal) of the character recognition target image P1 to predetermined numbers of pixels. Due to this resizing process, depending on the size of the characters to be recognized, white areas may appear in the character parts during the adaptive binarization described later. The resizing in step S200 can be adjusted so that white areas do not occur for the size of the characters to be recognized.

[0047] Step S202: The character recognition target image P1 acquired in step S100 is a color image. Therefore, the character recognition processing unit 223 performs grayscale conversion, converting the character recognition target image resized in step S200 into a grayscale image.

[0048] Step S204: The character recognition processing unit 223 performs smoothing of the character recognition target image, which has been converted to grayscale in step S202. Smoothing removes noise from the character recognition target image.

[0049] Step S206: Depending on the relationship between the background color and the color of the characters in the character recognition target image P1 acquired in step S100, the character recognition processing unit 223 performs a black-and-white inversion process if the background is black and the characters are white in the character recognition target image smoothed in step S204. As a result, the character recognition target image becomes one with black characters on a white background, making it suitable for the next adaptive binarization.

[0050] Step S208: The character recognition processing unit 223 performs adaptive binarization on the character recognition target image that has been inverted in black and white in step S206. In adaptive binarization, since the binarization threshold is determined for each pixel while binarization is performed, an appropriate binarization result can be obtained in response to situations where the brightness changes in parts of the character recognition target image.

[0051] Step S210: The character recognition processing unit 223 performs a resize on the character recognition target image, which has been adaptively binarized in step S208, to return it to its original size, for example, before it was resized in step S200.

[0052] Return to Figure 7 for the explanation. Step S108: The character recognition processing unit 223 performs tilt correction on the character recognition target image that was binarized in step S106. The character string in the character recognition target image P acquired in step S100 may be tilted due to the angle relationship between the subject and the camera at the time of imaging. For character recognition processing, the characters to be recognized must be horizontal. Therefore, as tilt correction, the character recognition processing unit 223 corrects the character string in the character recognition target image P so that it becomes horizontal.

[0053] Referring to the flowchart in Figure 9, an example of the tilt correction process procedure as step S108 will be explained. Step S300: The character recognition processing unit 223 performs a process to convert the characters in the character recognition target image, which has been binarized in step S106, to white. At this time, the background is converted to black. As a result, a character recognition target image with a black background and white characters is obtained.

[0054] Step S302: The character recognition processing unit 223 performs a fill operation on the character string in the character recognition target image. The fill operation may be, for example, a dilation operation in morphological transformation. For example, in such a dilation operation, an N x N pixel area centered on each pixel of the character recognition target image is extracted, and if at least one white pixel is included, the pixel value is output as white, and if no white pixels are included, it is output as black, thereby dilating the white pixel area. As the fill process is performed on the entire image to be recognized, the area in the image P10 obtained in step S300 in which a character string (recognition target character string) exists is converted into a filled area AR21 consisting of a single white block, as shown in Figure 13(B), for example, as shown in Figure 13(A).

[0055] Step S304: The character recognition processing unit 223 calculates the bounding rectangle SQ (Figure 13(B)) corresponding to the filled area AR21 obtained in step S302.

[0056] Step S306: The character recognition processing unit 223 performs a rotation process corresponding to tilt correction on the circumscribing rectangle SQ calculated in step S304. The character recognition processing unit 223 calculates the tilt angle of the calculated circumscribing rectangle SQ with respect to the horizontal, and rotates the circumscribing rectangle SQ so that the calculated tilt angle becomes zero. Furthermore, if multiple bounding rectangles SQ are calculated in accordance with the multiple filled regions AR21 obtained in step S302, the character recognition processing unit 223 may perform tilt correction using the bounding rectangle SQ closest to the center position in the character recognition target image P10. Alternatively, the character recognition processing unit 223 may perform tilt correction using the bounding rectangle SQ with a height greater than or equal to a certain level among the multiple bounding rectangles SQ calculated.

[0057] Return to Figure 7 for the explanation. Step S110: The character recognition processing unit 223 extracts (extracts) the image portion to be recognized as a character. That is, the character recognition processing unit 223 applies the original string to the bounding rectangle SQ that has been tilt-corrected in step S108. The character recognition processing unit 223 may extract the image portion of the bounding rectangle SQ to which the string has been applied from the image P10 to be recognized as the image portion to be recognized as a character. This provides the image portion to be recognized as a character, which includes the horizontal string. Furthermore, by extracting the image portion to be recognized as a character in this way, noise that is recognized as a character other than the string corresponding to the item can be removed.

[0058] Step S112: If it is determined in step S104 that the type of the image P1 to be recognized is a formatted document, the character recognition processing unit 223 performs preprocessing in steps S112 to S116 (an example of second character recognition preprocessing). As preprocessing corresponding to the type of formatted document, the character recognition processing unit 223 first performs format identification in step S112. Format identification is the process of identifying the format of the formatted document to which the image P1 to be recognized corresponds.

[0059] Refer again to Figure 12 to explain the format identification by step S112. Step S100 yields a character recognition target image P2, which captures the label shown in Figure 12. In this case, steps S102 and S104 determine that the character recognition target image P2 is a formatted document. In this case, the character recognition processing unit 223 calculates the feature quantities of the character recognition target image P2 as format identification in step S112. The type determination unit 222 also calculates the feature quantities of the image data stored in the format database corresponding to each format. Alternatively, the image data feature quantities for each format may be stored in the format database beforehand, and the type determination unit 222 may use the stored image data feature quantities. The type determination unit 222 searches for the feature quantities of the image data of the format that are closest to the feature quantities of the character recognition target image P2. The format corresponding to the retrieved feature quantities is identified as the format of the formatted document to which the character recognition target image P1 corresponds.

[0060] As described above, when the character recognition processing unit 223 searches for image data features in a format that matches the features of the image P2 to be recognized, it may extract the portion of the acquired image P2 that does not contain the string corresponding to the item. The character recognition processing unit 223 may then search for image data features in a format that contains the same features as the extracted portion of the image.

[0061] For example, in the character recognition target image P2, each item contains a corresponding string area AR12 (AR12-1 to AR12-7). The strings shown in the string area AR12 corresponding to each item are not fixed and are often changed. Therefore, even with formatted documents of the same format, the characters of the items may differ between the content of the formatted document in the character recognition target image P2 and the content of the image data of the formatted document stored in the format data. In other words, when comparing the feature quantities of the character recognition target image P2 with the feature quantities of the image data stored in the format data, a difference of a certain magnitude may occur, potentially reducing the accuracy of the search. Therefore, as described above, the accuracy of the search can be improved by using the feature quantities of the image portion of the target image P2 that does not contain the string corresponding to the item in the search.

[0062] Specifically, in the example shown in Figure 12, the character recognition processing unit 223 may extract the image portion AR13 from the image P2 to be recognized and use it for searching. The image portion AR13 is effective for searching because it does not contain a string corresponding to the item, has fixed content, and its image is distinctive. The character recognition processing unit 223 may, for example, estimate the region in the image to be recognized where a string corresponding to an item is presumed to be located, and may decide that the region containing the image, etc., which has as few estimated regions as possible, will be used as the image portion for searching. This process of determining the image portion may be performed, for example, using a trained model that has learned the correspondence between images of various formatted documents and the image portion to be used for searching.

[0063] Step S114: The character recognition processing unit 223 extracts the image portion corresponding to the formatted document from the character recognition target image acquired in step S100.

[0064] Step S116: The character recognition processing unit 223 extracts the image portion corresponding to the item from the image portion of the formatted document extracted in step S114 as the image portion to be recognized as character. Specifically, if the image portion of the formatted document extracted in step S114 is as shown in Figure 12, the character recognition processing unit 223 may extract the string area AR12 (AR12-1 to AR12-7) as the image portion to be recognized as character. In this case, the character recognition processing unit 223 may use the coordinate information for each item in the format data of the format identified in step S112 to identify the position of each item in the image to be recognized, and extract the image portion corresponding to the identified position as the image portion to be recognized. By extracting the image portion to be recognized in this way, noise that is recognized as characters other than the string corresponding to the item can be removed. Furthermore, items containing fixed strings that remain unchanged, such as the URL in the string area AR12-7 in Figure 12, may be excluded from the image portion targeted for character recognition. To achieve this, for example, the data for the item corresponding to the string area AR12-7 may be excluded from the item data in the format data.

[0065] Furthermore, when extracting the image portion corresponding to the formatted document in step S114, the character recognition processing unit 223 may perform tilt correction. In this case, as tilt correction, the character recognition processing unit 223 may perform a projection transformation so that the feature portion in the image portion corresponding to the formatted document that matches the format data is in the same position as the feature portion in the format data that matches the feature.

[0066] Step S118: The character recognition processing unit 223 performs character recognition processing on the character recognition target image portion extracted in step S110 or step S116. In other words, the character recognition processing unit 223 recognizes the character string in the character recognition target image portion.

[0067] Step S120: The result of the character recognition process in step S118 may contain errors. Therefore, the character recognition processing unit 223 performs character correction (character correction) on the character string recognized in step S118.

[0068] Referring to the flowchart in Figure 10, an example of the character correction processing procedure as step S122 will be explained. As can be understood from the following explanation, the character correction process is a process that determines the final character recognition result for each part of the image targeted for character recognition.

[0069] Step S400: The character recognition processing unit 223 selects one character recognition target image portion to be corrected from among the character recognition target image portions that have undergone character recognition processing in step S118.

[0070] Step S402: The character recognition processing unit 223 determines whether there are multiple recognized string units as a result of the character recognition processing in step S118 for the character recognition target image portion to be corrected. A string unit is a unit consisting of one or more characters. If one character is separated from other characters by a certain distance in the horizontal direction, one unit string is formed by that one character, and if multiple characters are adjacent to each other in the horizontal direction and are separated by a certain distance, one unit string is formed by multiple characters.

[0071] Step S404: If it is determined in step S402 that there are multiple unit strings, the character recognition processing unit 223 generates a combined string by concatenating the multiple unit strings according to the arrangement of the multiple unit strings present in the character recognition target image portion.

[0072] Refer to Figure 14 for an example of setting a combined string. This figure shows an example of the results of the character recognition process included in the character recognition target image portion P20 selected in step S400. In the character recognition target image portion P20 of 1, there is one unit string corresponding to a single line string, or one unit string corresponding to one item in the format. However, if there is noise in the character recognition target image portion that could not be removed by the previous processing, for example, multiple unit strings may be obtained as a result of the character recognition processing in step S118. In the figure, the character recognition processing results show that there are four recognition unit strings: UN-1 in the upper left, UN-2 in the upper right, UN-3 in the lower left, and UN-4 in the lower right. In each of the recognition unit strings UN(UN-1, UN-2, UN-3, UN-4), each character forming the string is recognized.

[0073] In cases where multiple recognition unit strings UN exist in a single character recognition target image portion P20, the character recognition processing unit 223 concatenates the unit strings for each combination of recognition unit strings UN that could be the correct unit string to set a concatenated unit string. Specifically, in the example shown in Figure 14, the character recognition processing unit 223 sets each of the individual recognition unit strings UN-1, UN-2, UN-3, and UN-4 as a concatenated unit string. Furthermore, the character recognition processing unit 223 sets a concatenated unit string using multiple recognition unit strings UN that exist horizontally without overlapping within the character recognition target image portion P20. Specifically, the character recognition processing unit 223 may set a concatenated unit string using recognition unit strings UN-1 and UN-2, and may set a concatenated unit string using recognition unit strings UN-3 and UN-2. Note that recognition unit strings UN-1 and UN-4 overlap horizontally, as shown by the dashed line, and are therefore not set as a concatenated unit string.

[0074] The character recognition processing unit 223 generates a single string (combined string) by sequentially concatenating the strings from left to right of the recognition unit string UN for each concatenation unit string set as described above. Specifically, if the string of recognition unit string UN-1 is "12345" and the string of recognition unit string UN-2 is "AB-CD", the concatenation unit string will be "12345AB-CD". For concatenation unit strings with a single recognition unit string UN, the string in the corresponding recognition unit string UN may be used directly as the combined string. Furthermore, as a result of the character recognition processing in step S402, a recognition unit string UN-5 may be obtained that has a portion that extends beyond the outer frame of the character recognition target image portion P20. In such cases, the character recognition processing unit 223 may exclude the recognition unit string UN-5 from the concatenated unit string in step S404.

[0075] Step S406: The character recognition processing unit 223 determines whether each combined string generated in step S404 matches the corresponding string rule stored in the string rule table storage unit 232 (rule matching determination).

[0076] Step S408: The character recognition processing unit 223 determines whether or not there is a combined string that has been determined to match the corresponding string rule based on the rule matching determination in step S406.

[0077] Step S410: If one of the concatenated strings generated in step S404 corresponds to the correct character recognition result, but step S408 determines that there is no concatenated string that matches the corresponding string rule, then the concatenated string that corresponds to the correct character recognition result contains a misrecognized character. Therefore, the character recognition processing unit 223 performs character correction and rule matching for each combined string. When correcting a combined string, the character recognition processing unit 223 utilizes the character conversion table stored in the character conversion table. In the combined string to be corrected, the character recognition processing unit 223 uses the character conversion table to convert characters that have been identified as not matching the string rules as a result of the rule matching check. For example, if the character recognized as the letter "O" in the combined string is to be converted, the character recognition processing unit 223 refers to the character conversion table (Figure 6) and first converts the character to the number "0", which has the highest priority among the conversion candidate characters associated with the letter "O" as the character to be converted. The character recognition processing unit 223 performs rule matching again for each corrected combined string.

[0078] Step S412: If it is determined in step S402 that there is only one unit string, the character recognition processing unit 223 performs a rule matching check on that single unit string.

[0079] Step S414: The character recognition processing unit 223 determines whether or not a result of the rule matching determination in step S412 was obtained in which the string matches the corresponding string rule.

[0080] Step S416: If it is determined in step S414 that the string does not match the corresponding string rule, the character recognition processing unit 223 corrects the character identified as not matching the string rule in the target unit string by converting it using the character conversion table, in accordance with step S410. The character recognition processing unit 223 then performs a rule matching check again on the corrected unit string.

[0081] Step S418: The character recognition processing unit 223 determines the final character recognition result corresponding to the selected character recognition target image portion. Specifically, if the character recognition processing unit 223 determines in step S408 that there is a concatenated string that matches the string rule, it confirms the concatenated string that matches the string rule as the final recognized string. Furthermore, if the character recognition processing unit 223 determines in step S414 that the target unit string matches the string rule, it confirms the target unit string as the final recognized string. Furthermore, if the character recognition processing unit 223 has gone through the processing in step S410, it confirms the corrected concatenated string that was determined to match the string rule in step S410 as the finally recognized string. If, as a result of the processing in step S410, it is determined that there is no corrected concatenated string that matches the string rule, it may be determined that there is a character recognition error. Furthermore, if the character recognition processing unit 223 determines in step S414 that the string matches the string rule, it confirms the target unit string as the finally recognized string. Furthermore, if step S416 has been completed, the character recognition processing unit 223 confirms the corrected unit string as the final recognized string if the corrected unit string in step S416 matches the string rule. On the other hand, if step S416 has been completed, and the corrected unit string in step S416 does not match the string rule, it may be determined that there is a character recognition error.

[0082] Step S420: The character recognition processing unit 223 determines whether the character correction (confirmation of character recognition results) process for all character recognition target image portions has been completed at this stage. If it is determined that processing for all parts of the image to be recognized has not been completed, the process returns to step S400, and processing for the next part of the image to be recognized is executed. If the type of image to be recognized is a single-line string, there is only one image portion to be recognized, and therefore, in the first step S420, it is determined that the processing targeting all parts of the image to be recognized is complete. If the type of image to be recognized is a formatted document, in step S420, it is determined that the processing targeting all parts of the image to be recognized is complete once the processing for each corresponding image portion to be recognized for each item in the corresponding format has been completed.

[0083] Return to Figure 7 for the explanation. Step S122: The processing result transmission unit 224 transmits character recognition result information, which indicates the final character recognition result obtained in step S418, to the user terminal 100 that transmitted the image to be recognized. The user terminal 100 outputs the received character recognition result information by displaying it or by other means. Note that the process in step S122 is a procedure for transmitting character recognition result information as soon as the character recognition result is determined, so that the character recognition result is displayed immediately, as in the example in Figure 2. For example, the processing result transmission unit 224 may store the final character recognition result from step S418 and, upon request from the user terminal 100, transmit the stored character recognition result information to the user terminal 100.

[0084] At the manufacturing plant FC, users such as administrators can use the character recognition result information transmitted from the character recognition server 200 to manage manufacturing processes. For example, in a manufacturing plant (FC), checking the character recognition results for printed information on specified items such as the product's expiration date on the product packaging can efficiently detect printing errors in the product. Furthermore, the user of the manufacturing plant FC may register correct string information such as expiration dates with the character recognition server 200, and the character recognition server 200 may compare the character recognition result with the registered string. If there is a discrepancy, it may notify the user terminal 100 that a printing error has occurred.

[0085] Furthermore, for example, in a manufacturing plant (FC), if the model numbers attached to products are not coded using barcodes or similar methods, users can efficiently perform inventory management by utilizing the results of character recognition processing for the model numbers.

[0086] Furthermore, the user terminal 100 can capture images of instrument readings from equipment in the manufacturing plant FC, and the character recognition results of these readings can be stored (remembered) for use in equipment maintenance.

[0087] Furthermore, by having the user capture images of the documents using the user terminal 100 and storing the character recognition results for each document, the user can manage documents using a computer, for example, by classifying them according to the order number.

[0088] The character recognition server 200 may acquire image data obtained by scanning printed single-line text strings or forms, as well as document files including PDF format, and use them as targets for character recognition. It may also acquire image data or document files output as electronic data from any other system and use them as targets for character recognition.

[0089] Furthermore, the applications of the character recognition processing system of this embodiment are not particularly limited and may be used in environments other than manufacturing plant fuel cells.

[0090] Alternatively, the functions of the user terminal 100, character recognition server 200, etc., may be implemented by recording the program for implementing these functions on a computer-readable recording medium, loading the program recorded on this recording medium into a computer system, and executing it. Here, "loading the program recorded on the recording medium into a computer system and executing it" includes installing the program into the computer system. Here, "computer system" includes hardware such as the OS and peripheral devices. Furthermore, "computer system" may include multiple computer devices connected via a network including communication lines such as the Internet, WAN, LAN, and dedicated lines. Furthermore, "computer-readable recording medium" refers to portable media such as flexible disks, magneto-optical disks, ROMs, CD-ROMs, and storage devices such as hard disks built into the computer system. Thus, the recording medium storing the program may also be a non-transient recording medium such as a CD-ROM. Furthermore, the recording medium also includes internal or external recording media accessible from the distribution server for distributing the program. The program code stored on the distribution server's recording medium may be different from the program code in a format executable by the terminal device. In other words, the format in which the program is stored on the distribution server is irrelevant, as long as it can be downloaded from the distribution server and installed in a form that can be executed on the terminal device. Furthermore, the program may be divided into multiple parts, each downloaded at a different time and then combined on the terminal device, and each divided program may be distributed by a different distribution server. In addition, "computer-readable recording medium" includes volatile memory (RAM) within computer systems that act as servers or clients when a program is transmitted over a network, which retains the program for a certain period of time. Moreover, the above program may be intended to implement only a part of the functions described above.Furthermore, the above-mentioned functions may be implemented in combination with programs already recorded in the computer system, such as so-called differential files (differential programs). [Explanation of symbols]

[0091] 100 User terminal, 200 Character recognition server, 201 Communication unit, 202 Control unit, 203 Storage unit, 221 Image acquisition unit, 222 Type determination unit, 223 Character recognition processing unit, 224 Processing result transmission unit, 231 Format database storage unit, 232 String rule table storage unit, 233 Character conversion table storage unit

Claims

1. A type determination unit that determines whether the type of character recognition target is a first type in which the character recognition target is a single line, or a second type in which multiple character recognition target is arranged in predetermined positions according to a predetermined format, based on a character recognition target image corresponding to the frame area of ​​an image captured by the user so that the character recognition target is a single line in the frame area of ​​an image captured by the user so that multiple character recognition target is in the image capture area, or a character recognition target image corresponding to the image capture area of ​​an image captured by the user so that multiple character recognition target is in the image capture area. For character recognition target images in which the type of character recognition target is determined to be the first type, a first character recognition preprocessing corresponding to the first type is performed; for character recognition target images in which the type of character recognition target is determined to be the second type, a second character recognition preprocessing corresponding to the second type is performed; and a character recognition processing unit recognizes characters from character recognition target images in which the first character recognition preprocessing or the second character recognition preprocessing has been performed. A character recognition processing system equipped with the following features.

2. The character recognition processing unit is configured to perform tilt correction in the first character recognition preprocessing to make one line of the character recognition target string horizontal within the character recognition target image, and the tilt correction is performed by forming a bounding rectangle around the one line of the character recognition target string and performing tilt correction on the formed bounding rectangle. The character recognition processing system according to claim 1.

3. The character recognition processing unit forms the bounding rectangle by performing a filtering process on the character recognition target image, which is color-coded with a first color for the character to be recognized and a second color for the background, such that pixels of the first color that are in a predetermined positional relationship with pixels of the second color are converted to the second color. The character recognition processing system according to claim 2.

4. The character recognition processing unit, if it performs character recognition on the character recognition target image portion corresponding to one character string, and if one or more unit strings exist as a result, generates a combined string obtained by combining multiple unit strings that are positioned horizontally without overlap, and selects the combined string that satisfies the conditions of the corresponding string from among the generated combined strings as the character recognition result. A character recognition processing system according to any one of claims 1 to 3.

5. A character recognition processing method in a character recognition processing system, A type determination step in which the type of character recognition target is determined is either a first type in which the character recognition target is a single line, or a second type in which multiple character recognition target is arranged in predetermined positions according to a predetermined format, based on a character recognition target image corresponding to the frame area of ​​an image captured by the user so that the character recognition target is a single line in the frame area of ​​the image captured by the user so that the character recognition target is a single line in the frame area of ​​the image captured by the user so that multiple character recognition target is arranged in predetermined positions according to a predetermined format, For character recognition target images in which the type of character recognition target is determined to be the first type, a first character recognition preprocessing corresponding to the first type is performed; for character recognition target images in which the type of character recognition target is determined to be the second type, a second character recognition preprocessing corresponding to the second type is performed; and a character recognition processing step of recognizing characters from character recognition target images in which the first character recognition preprocessing or the second character recognition preprocessing has been performed. A character recognition processing method comprising the following:

6. Computers in character recognition processing systems A type determination unit determines whether the type of character recognition target is a first type in which the character recognition target is a single line, or a second type in which multiple character recognition target is arranged in predetermined positions according to a predetermined format, based on a character recognition target image corresponding to the frame area of ​​an image captured by the user so that the character recognition target is a single line in the frame area of ​​the image captured by the user so that multiple character recognition target is arranged in predetermined positions according to a predetermined format. For character recognition target images whose type is determined to be the first type, a first character recognition preprocessing corresponding to the first type is performed. For character recognition target images whose type is determined to be the second type, a second character recognition preprocessing corresponding to the second type is performed. A character recognition processing unit recognizes characters from character recognition target images that have undergone the first or second character recognition preprocessing. A program designed to function as such.

Images