Character recognition method and character recognition device

Abstract

To provide a character recognition method capable of facilitating the correct character recognition of a character recognition object.SOLUTION: The character recognition method for recognizing a character written in a reading object including a frame line includes a step for acquiring an image captured by a camera, a step for cutting out a recognition object area for recognizing the character from the image, a step for removing a noise including the frame line in the recognition object area, a step for performing recognition processing for recognizing a plurality of characters in the recognition object area, and a step for determining the failure degree of the recognition result of a character string including the plurality of characters by the recognition processing on the basis of the recognition result for each character in the recognition object area.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001] The present disclosure relates to a character recognition method and a character recognition device for recognizing characters. [Background technology]

[0002] Conventionally, a document identification device that recognizes characters written on a document including ruled lines is known. This document identification device includes an image acquisition means for acquiring a filled-out document image on which characters are written on the document, a character recognition means for executing a character recognition process for recognizing characters written in a reading area surrounded by ruled lines in the filled-out document image, a character string determination means for determining whether a specific character is included at the beginning or end of a character string recognized from the reading area by the character recognition means, a ruled line detection means for detecting a ruled line included in the reading area in the filled-out document image when the character string determination means determines that a specific character is included at the beginning or end of the character string, a ruled line correction means for correcting the ruled line detected by the ruled line detection means to a single solid line, and a control means for causing the character recognition means to execute the character recognition process again on the filled-out document image in which the ruled lines have been corrected by the ruled line correction means. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2017-33438 A Summary of the Invention [Problem to be solved by the invention]

[0004] The present disclosure provides a character recognition method and a character recognition device that can facilitate correct character recognition of a character recognition target. [Means for solving the problem]

[0005] One aspect of the present disclosure is a character recognition method for recognizing characters written on a reading object including a frame line, the character recognition method including the steps of: acquiring an image captured by a camera; extracting a recognition target area from the image in which characters are to be recognized; removing noise including the frame line within the recognition target area; performing a recognition process for recognizing multiple characters within the recognition target area; and determining a degree of failure of a recognition result of the recognition process for a character string including the multiple characters based on the recognition result for each character in the recognition target area.

[0006] One aspect of the present disclosure is a character recognition device that includes a processor and recognizes characters written on a reading object that includes a frame line, wherein the processor acquires an image captured by a camera, extracts a recognition target area from the image in which characters are to be recognized, removes noise including the frame line within the recognition target area, performs a recognition process to recognize multiple characters within the recognition target area, and determines a degree of failure of the recognition result of the recognition process for a character string including the multiple characters based on the recognition result for each character in the recognition target area. Effect of the Invention

[0007] According to the present disclosure, it is possible to facilitate correct character recognition of a character recognition target. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating a use case of a character recognition device according to an embodiment of the present disclosure. [Diagram 2] A block diagram showing an example of the internal configuration of a character recognition device. [Diagram 3] FIG. 1 is a diagram showing an example of a captured image of a reading target; [Figure 4] FIG. 1 is a diagram for explaining a first example of character recognition processing and failure determination processing; [Diagram 5] FIG. 11 is a diagram for explaining a second example of the character recognition process and the failure determination process. [Figure 6] FIG. 13 is a diagram showing a display example of a re-imaging guide screen; [Figure 7]A diagram showing an example of the character recognition result screen [Figure 8] Flowchart showing a first operation example of the character recognition device [Figure 9] Flowchart showing a second operation example of the character recognition device [Figure 10] Flowchart showing a third operation example of the character recognition device [Figure 11] Figure for supplementary explanation of pretreatment DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the embodiments will be described in detail with reference to the drawings as appropriate. However, more detailed explanation than necessary may be omitted. For example, detailed explanation of already well-known matters or duplicate explanation of substantially the same configuration may be omitted. This is to avoid the following explanation becoming unnecessarily redundant and to facilitate understanding by those skilled in the art. Note that the attached drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

[0010] (How the embodiments of the present disclosure were achieved)

[0011] In the document identification device of Patent Document 1, the presence of ruled lines in the document may allow the detection of specific characters that are similar in shape to the ruled lines (for example, "1 (number)," "I (uppercase English letter)," "l (lowercase English letter)," "+ (plus)," "# (sharp)," or "- (minus or hyphen, etc.)"). However, the document identification device of Patent Document 1 does not reacquire the completed document image, so the image to be recognized as the character is not changed. Therefore, even if the character recognition process is performed again by the character recognition means on the completed document image with the corrected ruled lines, misrecognition of characters may occur similarly.

[0012] In addition, the document identification device of Patent Document 1 does not consider whether a character string including a plurality of characters is broken (unreliable) as a recognition result by character recognition processing. As a result, the document identification device of Patent Document 1 judges that as long as the character string can be recognized, even if the character string contains a misrecognized character, there is no problem with the recognition result. Therefore, the character string judged by the document identification device to be free of problems as a recognition result includes a mixture of characters that contain misrecognized characters and those that do not. Here, if a misrecognized character is included, the user may change the shooting conditions and perform the character recognition processing again. However, since the document identification device of Patent Document 1 does not have an index indicating whether the recognition result is broken, the user must judge whether the recognition result is broken without any assistance. Therefore, the configuration of the document identification device of Patent Document 1 requires the user to perform a cumbersome confirmation task, and there is also a risk of erroneous judgment due to user error. Therefore, when a character string is recognized, it is desirable to further obtain an indicator showing whether the recognition result is faulty, and to make suggestions to the user based on this indicator as to whether or not the character recognition process should be performed again.

[0013] In the following embodiment, a character recognition method and a character recognition device that can easily perform correct character recognition of a character recognition target will be described.

[0014] (Embodiment) FIG. 1 is a diagram illustrating a use case of a character recognition device according to an embodiment of the present disclosure.

[0015] The character recognition device 100 reads characters written on a read object TG. Characters here broadly include letters, numbers, symbols, etc. The read object TG is, for example, a slip for a package to be transported. The read object TG may be something other than a slip, such as a medical questionnaire, application form, order form, or other read object. The character recognition device 100 may be a PC (Personal Computer), a tablet terminal, a smartphone, or the like.

[0016] The character recognition device 100 includes, for example, a camera 13 (see FIG. 2), and performs image processing on a captured image of a read target TG. The character recognition device 100 performs OCR (Optical Character Recognition / Reader) processing (hereinafter also referred to as "character recognition processing") on the captured image, and recognizes characters written on the read target TG.

[0017] FIG. 2 is a block diagram showing an example of the internal configuration of the character recognition device 100. As shown in FIG.

[0018] The character recognition device 100 includes a processor 11, a memory 12, a camera 13, an input device 14, and a display device 15. The input device 14 and the display device 15 may be integrated into one device, such as a touch panel.

[0019] The processor 11 may be configured using, for example, a Central Processing Unit (CPU) or a Digital Signal Processor (DSP). The processor 11 may be configured using various integrated circuits (for example, a Large Scale Integration (LSI) or a Field Programmable Gate Array (FPGA)). The processor 11 realizes various functions by executing programs stored in the memory 12. The processor 11 comprehensively controls each part of the character recognition device 100 and performs various processes.

[0020] The processor 11 includes an image processing unit 11A as a functional unit. The image processing unit 11A executes image processing on the captured image output from the camera 13. The image processing includes preprocessing, character recognition processing, and failure determination processing, which will be described later.

[0021] The memory 12 includes a primary storage device (e.g., Random Access Memory (hereinafter referred to as "RAM") or Read Only Memory (hereinafter referred to as "ROM")). The memory 12 may include a secondary storage device (e.g., a Hard Disk Drive (hereinafter referred to as "HDD") or a Solid State Drive (hereinafter referred to as "SSD")) or a tertiary storage device (e.g., an optical disk or an SD card). The memory 12 may also be an external storage medium, and may be detachable from the character recognition device 100. The memory 12 stores various data, information, programs, etc.

[0022] The memory 12 stores, for example, an image captured by the camera 13. The memory 12 stores, for example, information (format information) on a predetermined format of the read target TG. The format includes, for example, the position, size, and range of the recognition target area R1 on the read target TG. The recognition target area R1 is an area in which one or more characters can be written for which character recognition processing is performed by the processor 11. The recognition target area R1 may be determined, for example, according to the area of ​​each item (e.g., postal code, telephone number, address, name, delivery date) on a slip. In addition, the memory 12 stores, for example, various thresholds (thresholds th0, th1, th2...) used to derive the degree of failure of the recognition result of a character string obtained by the character recognition processing. The memory 12 stores the recognition result by the character recognition processing. This recognition result may include the recognition result for each character written in the recognition target area R1 (i.e., the recognition result for one character) and the recognition result for each recognition target area (i.e., the recognition result for all the characters written in the recognition target area). The recognition result of one character is, for example, a recognized character. The recognition result of all the characters written in the recognition target area is, for example, a recognized character string. The memory 12 may also store various types of trained models.

[0023] The camera 13 is configured to have at least a lens (not shown) and an image sensor (not shown). The image sensor is, for example, a solid-state imaging element such as a Charged-Coupled Device (CCD) or a Complementary Metal-Oxide-Semiconductor (CMOS), and converts an optical image formed on an imaging surface into an electric signal. The camera 13 captures an image of the read target TG based on, for example, a user operation via the input device 14, and outputs the captured image of the read target TG to the image processing unit 11A of the processor 11. The camera 13 may also capture an image automatically according to a predetermined condition. The camera 13 may also capture an image of the read target TG continuously based on one user operation to obtain a plurality of captured images. In FIG. 2, the camera 13 may be provided outside the character recognition device 100. In this case, the character recognition device 100 and the camera 13 may be connected by wire via a USB or the like, or may be connected wirelessly via a wireless LAN or the like. The camera 13 may also be a camera that can be installed away from the character recognition device 100, such as a network camera.

[0024] The input device 14 may include various buttons, keys, a keyboard, a mouse, a touch panel, a microphone, or other input devices. The input device 14 accepts input of various data or information. The input device 14 is operated by a user of the character recognition device 100, for example.

[0025] The display device 15 is configured using a display such as an LCD (Liquid Crystal Display) or an organic EL (Electroluminescence) display. The display device 15 displays various data or information, etc. The display device 15 may display an image of the reading target TG output from the processor 11, a re-imaging guide screen G11 (described later), a character recognition result screen G12 (described later), etc.

[0026] Next, the processing contents of the image processing section 11A of the processor 11 will be illustrated.

[0027] The image processing unit 11A acquires a captured image captured by the camera 13. The image processing unit 11A performs various image processing on the captured image. For example, the image processing unit 11A cuts out one or more recognition target regions R1 from the captured image of the reading target TG based on, for example, format information stored in the memory 12. The recognition target region R1 may coincide with the character writing region described above.

[0028] The image processing unit 11A performs a predetermined pre-processing on the cut-out recognition target region R1. This pre-processing includes a process of deleting noise including a frame line WK (ruled line) in the recognition target region R1. The frame line WK here broadly includes a frame line WK1 for assisting in writing characters, a frame line WK2 for dividing a plurality of regions (for example, an address entry field, a telephone number (also simply written as telephone) entry field) in the reading target TG, and other frame lines. In addition to the frame line WK, the noise deleted in the pre-processing includes, for example, dots and lines due to stains on the paper surface. Here, since various methods for deleting noise from an image are known, detailed description will be omitted. In addition, the pre-processing may include a predetermined correction process in the captured image and a process of removing the background. The image processing unit 11A may execute the pre-processing using, for example, a pre-processing trained model stored in the memory 12.

[0029] The image processing unit 11A performs character recognition processing to recognize one or more characters in the recognition target area R1. The image processing unit 11A may perform the character recognition processing by, for example, using a trained model for character recognition stored in the memory 12. The image processing unit 11A may perform the character recognition processing by, for example, pattern matching. The character recognition processing provides a recognition result for each character in the recognition target area R1. That is, an individual recognition result (also called an individual recognition result) for each character is obtained. The image processing unit 11A detects each character existence area in which each character is detected by the individual recognition result for each character. The character existence area may have, for example, a rectangular shape surrounding the recognized character, or may have other shapes. The character existence area is detected for each character, so that the number of character existence areas is the same as the number of characters. Also, when multiple characters exist in the recognition target area R1, a character string is recognized by recognizing the multiple characters as a result of the character recognition processing.

[0030] The image processing unit 11A judges the degree of failure of the overall recognition result (also called the overall recognition result) of the characters in the recognition target area R1, that is, the degree of failure of the recognition result of the character string, based on the recognition result of each character in the recognition target area R1. The degree of failure is an index indicating how low the reliability of the recognition result of the character string is. In other words, the higher the degree of failure of the recognition result of the character string, the lower the reliability of the recognition result of the character string, and the lower the degree of failure of the recognition result of the character string, the higher the reliability of the recognition result of the character string. The degree of failure is derived for each recognition target area R1. The degree of failure may be expressed, for example, by any value between 0 and 1, or may be expressed by other values ​​or value ranges. For example, if the degree of failure is equal to or greater than a threshold th0 (for example, 0.8), it may be judged that there is a failure, and if the degree of failure is less than the threshold th0, it may be judged that there is no failure. Details of the method of judging the degree of failure will be described later.

[0031] For example, when it is determined that the recognition result is failed, the image processing unit 11A may generate a re-imaging guide screen G11 according to the degree of failure of the character string recognition result, and output the re-imaging guide screen G11 to the display device 15. The re-imaging guide screen G11 is a screen for guiding the user of the character recognition device 100 to re-imaging the reading target TG.

[0032] Depending on the degree of failure of the character string recognition result, for example when it is determined that the recognition result is failed, image processing unit 11A may generate a character recognition result screen G12 and output it to display device 15. Character recognition result screen G12 is a screen that includes at least information on the recognition result by the character recognition process, and may include a message that notifies the user of character recognition device 100 that the character string recognition result is failed.

[0033] Character recognition device 100 may include a communication device. The communication device communicates various data or information according to a wired or wireless communication method. The communication device may be communicatively connected to a network NT. The communication method used by the communication device may include a local area network (LAN), a wide area network (WAN), a mobile phone network, power line communication, short-range wireless communication (e.g., Bluetooth (registered trademark), NFC (registered trademark)), and other communication methods.

[0034] For example, an external server may perform at least a part of the processing by image processing unit 11A, and necessary data may be communicated between the external server and the image processing unit 11A via a communication device. Also, character recognition device 100 may not include camera 13, and a communication device may receive captured images from a device including camera 13 and send capture instructions to camera 13.

[0035] FIG. 3 is a diagram showing an example of a captured image G1 obtained by capturing an image of a reading target TG.

[0036] In the example of Fig. 3, the read object TG is a slip handwritten by a customer using a transportation service using a predetermined writing instrument. Note that the method of writing characters is not limited to handwriting, and may be printing, stamping, etc.

[0037] The reading target TG includes a recognition target area R1. Note that the recognition target area R1 shown in FIG. 3 is an example and is not limited to this. The position, size, range, etc. of the recognition target area R1 are determined depending on the type of the reading target TG. The image processing unit 11A can recognize the position, size, range, etc. of each recognition target area by acquiring format information related to the reading target TG stored in the memory 12. Note that the image processing unit 11A may recognize the type of the reading target TG, for example, by reading a mark printed on the reading target TG, or may have the user specify the type via the input device 14.

[0038] For example, in the reading target TG, within the recognition target area R1, information regarding the delivery destination delivered by the transportation service, such as zip code, telephone number, delivery address, recipient name, delivery date, etc., can be handwritten. Also, in the reading target TG, within the recognition target area R1, information regarding the requester (i.e., customer) using the transportation service, such as zip code, telephone number, requester address, requester name, etc., can be handwritten.

[0039] FIG. 4 is a diagram for explaining a first example of character recognition processing and breakdown determination processing.

[0040] The image processing unit 11A acquires the captured image G1 of the reading target TG before the character recognition processing, cuts out the recognition target area R1 from the captured image G1, and performs preprocessing on the recognition target area R1. The recognition target area R1 cut out in FIG. 4 is, for example, the recognition target area R10 in FIG. 3.

[0041] The image processing unit 11A recognizes the characters within the preprocessed recognition target area R1 in the character recognition processing. In the example of FIG. 4, each character of the address "AA Prefecture BB City CC Town 1-2-3" is recognized one by one, and a character existence area R2 is detected for each character. Also, the image processing unit 11A may change the display mode of the frame indicating the outer edge of the character existence area R2 according to the recognition level (e.g., recognition score) of the character recognition processing. Note that since the character existence area R2 is an area where a character is recognized to exist, for example, when the sizes of a plurality of characters are different in handwriting, the sizes of the character existence areas R2 detected for each character may also be different. Also, in FIG. 4, a part of the frame line WK existing below the recognition target area R1 is detected as four characters (e.g., "one", "-"). Also, since "-" exists in "1-2-3" described in the address, the degree of breakdown of the recognition result of the character string increases, or in other words, the reliability decreases.

[0042] In the breakdown determination process, the image processing unit 11A counts the number of specific characters obtained as the recognition result by the character recognition process, and determines the degree of breakdown of the recognition result of the character string based on the number of specific characters. The specific characters here are characters similar to the frame line WK, and may be, for example, "一", "-", "1", "I", etc. Since the frame line WK may remain in the recognition target area R1 even after the preprocessing is performed, the frame line WK and the specific characters may be mixed. In this case, if both are recognized as specific characters, they cannot be distinguished. That is, the recognized specific characters may include frame noise that is actually not a character but the frame line WK.

[0043] As an example, the image processing unit 11A determines that the greater the number of specific characters, the greater the degree of breakdown, and the smaller the number of specific characters, the smaller the degree of breakdown. For example, when the number of specific characters is equal to or greater than the threshold th1 (for example, 3), the image processing unit 11A determines that the degree of breakdown is equal to or greater than the threshold th0, and the recognition result of the character string is broken. This is because when the number of specific characters is large, there is a high possibility that the frame line WK is included. Also, when the number of specific characters is less than the threshold th1, the image processing unit 11A determines that the degree of breakdown is less than the threshold th0, and the recognition result of the character string is not broken. This is because when the number of specific characters is small, there is a low possibility that the frame line WK is included. In the example of FIG. 4, since 4 specific characters SC are recognized, it is determined that the recognition result of the character string is broken.

[0044] FIG. 5 is a diagram for explaining a second example of the character recognition process and the breakdown determination process. Note that the processes up to the character recognition process are the same as those in FIG. 4, so the description is omitted.

[0045] The image processing unit 11A recognizes the characters in the preprocessed recognition target area R1 by the character recognition process, and detects the character existence area R2 within the recognition target area R1 (see (11)).

[0046] In the failure determination process, the image processing unit 11A determines the degree of failure of the character string recognition result based on the position or positional relationship of each detected character existence region R2. For example, the image processing unit 11A may detect the center position CR of each character existence region R2 (see (12)) and determine the degree of failure of the character string recognition result based on the center position CR of each character existence region R2. In this case, the image processing unit 11A arranges each square SQ having a side of length l1 centered on each center position CR in the captured image G1 (see (13)), and arranges a straight line L1 so that the number of lines passing through each square SQ is maximized (see (14)). In this case, the image processing unit 11A calculates the number of squares SQ that the straight line L1 does not pass through. The length l1 of one side of the square SQ represents the size of the square SQ and indicates the allowable amount of positional variation of the character, that is, the character existence region R2.

[0047] As an example, the image processing unit 11A may determine that the greater the number of squares SQ through which the straight line L1 does not pass, the greater the degree of failure, and that the smaller the number of squares SQ through which the straight line L1 does not pass, the smaller the degree of failure. For example, when the number of squares SQ through which the straight line L1 does not pass is equal to or greater than a threshold th2, the image processing unit 11A determines that the degree of failure is equal to or greater than a threshold th0, and that the recognized character string is broken. When the number of squares SQ through which the straight line L1 does not pass is less than the threshold th2, the image processing unit 11A determines that the degree of failure is less than a threshold th0, and that the recognition result of the character string is not broken. Compared with the characters corresponding to the other squares SQ (square SQ1), the characters corresponding to the squares SQ through which the straight line L1 does not pass are shifted in a direction perpendicular to the arrangement direction of the characters (i.e., the vertical direction of the characters). Therefore, when the number of squares SQ through which the straight line L1 does not pass is large, it can be said that the possibility of the frame line WK being included is high, and when the number of squares SQ through which the straight line L1 does not pass is small, it can be said that the possibility of the frame line WK being included is low. The character arrangement direction corresponds to the direction of the characters to be written in the recognition target area R1, for example. For example, the image processing unit 11A determines the direction along the straight line L1 as the direction of the characters to be written in the recognition target area R1.

[0048] Note that image processor 11A may not evaluate all characters where line L1 does not pass through square SQ in the same way, but may change the degree of influence of the misaligned character on the evaluation of the degree of failure according to the amount of misalignment. For example, when the distance from line L1 to center position CR or square SQ is far, image processor 11A can determine that the character is definitely misaligned, and increases the influence on the evaluation of the degree of failure. Conversely, when the distance from line L1 to center position CR or square SQ is short, image processor 11A reduces the influence on the evaluation of the degree of failure because there is a risk that the character has been erroneously determined to be misaligned due to factors such as side l1 of square SQ being too short.

[0049] Next, an example of a screen display will be described.

[0050] 6 is a diagram showing a display example of the re-imaging guide screen G11. Information provided by the re-imaging guide screen G11 is an example of a recognition failure alert for re-imaging.

[0051] The re-imaging guidance screen G11 includes guidance information I1 for re-imaging the read target TG. The guidance information includes, for example, message information such as "Recognition failed. Do you want to re-imaging?". The re-imaging guidance screen G11 also includes an OK button for instructing re-imaging and a NO button for not instructing re-imaging. The OK button and the NO button function as the input device 14 for accepting user operations. The OK button and the NO button may be physical buttons or functional buttons. The user checks the re-imaging guidance screen G11 and selects the OK button via the input device 14, so that the character recognition device 100 can re-image the read target TG even if the recognition result of the character string is broken.

[0052] 7 is a diagram showing a display example of the character recognition result screen G12. The information provided by the character recognition result screen G12 is an example of a recognition failure alert for prompting the user to confirm the information.

[0053] The character recognition result screen G12 includes an image g31 obtained by cutting out the recognition target region R1 in the captured image G1 or a preprocessed image g31 of the cut out image, and a display field F1 for displaying the recognition result of the character recognition process for recognizing the characters in the recognition target region R1. The character recognition result screen G12 also includes an OK button and a NO button similar to those in Fig. 6. The user can check the re-imaging guide screen G11 and select the OK button via the input device 14, so that the character recognition device 100 can re-image the read target TG even if the recognition result of the character string is not correct.

[0054] Moreover, the information displayed in the display field F1 of the recognition result can be changed by selecting the display field F1 via the input device 14. For example, the image processing unit 11A may correct or delete a specific character displayed in the display field F1 via the input device 14. That is, even if the frame line WK is recognized as a specific character SC, the character string displayed in the display field F1 can be corrected to a correct character string by the user deleting the specific character SC via the input device 14. In this case, the correct character can be obtained as the character in the recognition target area R1 without re-imaging the read target TG.

[0055] Next, an example of the operation of the character recognition device 100 will be described. FIG. 8 is a flowchart showing a first operation example of the character recognition device 100.

[0056] First, the camera 13 captures an image of a read target TG (e.g., a slip). The image processing unit 11A acquires a captured image G1 in which the captured read target TG appears (S11). The image processing unit 11A refers to the format information and detects and acquires (cuts out) a recognition target area R1 from the captured image G1 (S12). The image processing unit 11A pre-processes the recognition target area R1. The image processing unit 11A recognizes characters written in the recognition target area R1 (S13).

[0057] The image processing unit 11A judges the degree of failure of the character string recognition result (i.e., the overall recognition result) (S14). In the first operation example, the degree of failure is represented by a binary value. When it is judged that the character string recognition result by the character recognition process is failed, the image processing unit 11A displays at least one of a re-imaging guide screen G11 and a character recognition result screen G12 via the display device 15 (S15). The character recognition result screen G12 may be displayed including information indicating an error related to the failure of the character string recognition result.

[0058] On the other hand, if the character string recognition result is not determined to be corrupted, the image processing unit 11A stores this recognition result in the memory 12 (S16). The recognition result here is a character recognition result that is not corrupted (has high reliability) and may include information on the recognized character string.

[0059] According to such a first operation example, when the character string recognition result is corrupted, character recognition device 100 displays the screen of step S15, so that the user can understand that it is necessary to re-image the read target TG, correct or delete the character recognition result, etc. Therefore, the user can easily perform correct character recognition of the character recognition target by performing a user operation via input device 14 to re-image the read target TG, correct or delete the character recognition result, etc. Furthermore, character recognition device 100 can use the character recognition result in subsequent processing by storing a character recognition result that is not corrupted.

[0060] Fig. 9 is a flowchart showing a second operation example of character recognition device 100. In Fig. 9, character recognition device 100 repeats the process until the recognition result is not corrupted, that is, until the recognition result is successful. Note that in Fig. 9, the same process as in Fig. 8 is given the same step number, and the description thereof is omitted or simplified.

[0061] First, the character recognition device 100 executes the process of steps S11 to S14 similar to those in FIG. 8. If it is determined in step S14 that the recognition result of the character string is not correct, the image processing unit 11A instructs the camera 13 to re-image the reading object TG (S21). In response to this instruction, the camera 13 re-images the reading object TG. The image processing unit 11A acquires a captured image G1 of the re-imaged reading object TG (S11). That is, after the process of step S21, the process proceeds to step S11, and the processes of steps S11 to S14 are repeated again. This repetition is performed until it is no longer determined in step S14 that the recognition result of the character string is not correct.

[0062] If it is not determined in step S14 that the character string recognition result is corrupted, image processing section 11A stores the recognition result in memory 12 (S16).

[0063] According to the second operation example, character recognition device 100 can obtain a captured image from which the frame lines can be suitably removed in preprocessing, for example, by changing the amount of light received by camera 13 depending on the timing of image capture. By obtaining a captured image from which the frame lines can be suitably removed in such preprocessing, it is possible to prevent the frame lines from remaining after preprocessing, resulting in the image being erroneously recognized as a character. Therefore, character recognition device 100 can easily perform correct character recognition.

[0064] Fig. 10 is a flowchart showing a third operation example of the character recognition device 100. In Fig. 10, the character recognition device 100 captures a plurality of (N) captured images G1 of the reading target TG, performs character recognition processing and failure determination processing based on each captured image G1, and obtains a suitable character recognition result for the captured image G1. Note that in Fig. 10, the same steps as those in Fig. 8 or 9 are denoted by the same step numbers, and their explanations are omitted or simplified.

[0065] First, the character recognition device 100 executes the processes of steps S11 to S13 similar to those in FIG. 8. Next, the image processing unit 11A derives (e.g., calculates) the degree of failure of the recognition result of the character string (S31). The image processing unit 11A temporarily stores information on the recognition result including the degree of failure derived in step S31 in the memory 12 (S32). The information on the recognition result includes information on the recognized character string. The image processing unit 11A judges whether or not the derivation of the degree of failure has been completed N times (S33). If the derivation of the degree of failure has not been completed N times (No in step S33), the character recognition device 100 repeats the processes of steps S11 to S13 and S31 to S33. That is, the character recognition device 100 repeats the imaging process, character recognition process, and failure judgment process (here, the failure degree derivation process) of the read target TG N times.

[0066] When the derivation of the degree of corruption has been completed N times (Yes in step S33), the image processing unit 11A determines the recognition result with the smallest degree of corruption as a result of the N times of corruption determination processing, and stores this recognition result in the memory 12 (S34). The recognition result here is the least corrupted (most reliable) character recognition result among the multiple (N) recognition results, and may include information on the recognized character string. Therefore, the processor 11 can use the most reliable character string in subsequent processing. Note that the image processing unit 11A may determine a recognition result with a small degree of corruption (for example, below a predetermined threshold value) even if the degree of corruption is not the smallest.

[0067] 10, after the imaging process, character recognition process, and failure determination process for one captured image G1 are completed, the imaging process, character recognition process, and failure determination process for the next captured image G1 are performed, and this series of processes is repeated N times, but this is not limited to the above. The character recognition device 100 may first obtain multiple captured images G1 by continuous shooting, and perform the character recognition process and failure determination process for each captured image G1 in parallel.

[0068] According to the third operation example, the character recognition device 100 can obtain a captured image from which the frame lines can be officially removed by preprocessing, since the amount of light received by the camera 13 varies depending on the timing of capturing the image. By obtaining a captured image from which the frame lines can be officially removed by such preprocessing, it is possible to suppress erroneous recognition of the image as a character due to the frame lines remaining even after preprocessing. In addition, an optimal character recognition result can be obtained based on a plurality of degrees of failure derived based on each of a plurality of captured images G1. Therefore, even if the character recognition device 100 tries not to re-capture the image as much as possible, it is possible to obtain a character string as a more reliable character recognition result, which can be used for subsequent processing.

[0069] Next, the preprocessing by the character recognition device 100 will be described in more detail. FIG. 11 is a diagram for providing a supplementary explanation of the pre-processing.

[0070] The image processor 11A may extract multiple recognition target regions R1 (e.g., recognition target regions R10 and R11) from the captured image G1 of the reading target TG (see (21)), and connect these recognition target regions R1 to generate one recognition target region R1 (e.g., recognition target region R15). In this case, each character included in the recognition target regions R10 and R11 becomes a character recognition target.

[0071] Furthermore, the recognition target area R1 of the reading target TG is often not actually black and white, but is colored in a color such as gray. Furthermore, the color of characters printed on the reading target TG is often not pure black. For this reason, the image processing unit 11A may execute pre-processing such as processing to correct image quality (e.g., blur correction) and correction to remove the background. This allows information on the recognition target area R1 (e.g., recognition target area R15) with the image quality corrected and the background removed to be obtained (see (22)).

[0072] Then, the image processing unit 11A executes character recognition processing on the cut-out image of the preprocessed recognition target area R1 (see (23)).

[0073] According to the character recognition device 100 in such a first embodiment, since the degree of breakdown of the recognition result of the character string is determined, the degree of breakdown is obtained as an index indicating whether there is a problem with the recognition result of the character string, and based on this index, it is possible to more easily and correctly perform character recognition on the character recognition target. Further, the character recognition device 100 can determine the presence or absence of breakdown according to the magnitude of the degree of breakdown. When there is a breakdown, the reading target TG can be re-imaged by the camera 13, or it can be notified to the user by displaying on the screen (for example, the re-imaging guidance screen G11 and the character recognition result screen G12) that re-imaging or the like is necessary.

[0074] Further, even when some frame line noises remain without being completely removed in the preprocessing, the character recognition device 100 can notify that the remaining frame line noises are affecting the recognition result of the character string. For example, when the number of recognized specific characters SC is large, the character recognition device 100 can calculate the degree of breakdown according to the number of recognized specific characters SC, thereby estimating that frame line noises are included. Also, when the recognized multiple characters, that is, the character existence regions R2, are not arranged linearly along the straight line L1, the frame line WK can be recognized as a specific character SC, that is, it can be estimated that frame line noises are included. Further, when specific characters SC such as hyphens or the Chinese numeral 'one' are arranged continuously, and the number of these continuously arranged specific characters SC is large (equal to or more than a predetermined number), it can be estimated that the specific characters SC include frame line noises. Also, by re-taking an image with the camera 13 and performing character recognition processing, the character recognition device 100 can easily obtain a correct character recognition result without reprocessing the captured image G1 of the reading target TG for character recognition.

[0075] In this embodiment, it is exemplified that the characters described on the reading target TG are in horizontal writing (the horizontal direction in FIG. 3), but it is not limited to this, and vertical writing may also be used. In this case, the direction in which the straight line L1 extends may be along the character arrangement direction in the case of vertical writing instead of the character arrangement direction in the case of horizontal writing.

[0076] In this embodiment, the derivation of the failure degree is exemplified by derivation of the failure degree taking into account the position of the character existence region R2 and derivation of the failure degree taking into account the recognition result of the character recognition. However, the present invention is not limited to this, and character recognition device 100 may derive the failure degree of the recognition result by combining these two derivation methods.

[0077] (Other variations) In this embodiment, the image processing unit 11A judges the degree of failure by comparing the number of specific characters with the threshold value, but the degree of failure may be judged by other criteria based on the number of specific characters. For example, the image processing unit 11A may judge the degree of failure based on the ratio of the number of recognized characters to the number of specific characters. In general, the longer the character string, the higher the possibility of including specific characters. For example, when "Room 5, 3-4-2, chome" is written as "2-3-4-5" or when a long syllable is included in the name of an apartment building, the number of "-" characters will be large even if the character string is correctly recognized. Therefore, if the number of specific characters and the threshold value are directly evaluated, the possibility of erroneously judging that the character string is broken in such a case increases. On the other hand, even in such a case, if the characters are correctly recognized, it is rare that the specific characters occupy the majority of the recognized characters. Therefore, the image processing unit 11A can judge the degree of failure more accurately when the character string is long by judging the degree of failure based on the ratio of characters, etc.

[0078] In addition, in this embodiment, when counting the number of specific characters for determining the degree of failure, the image processing unit 11A may take into consideration whether the characters before and after the specific character are specific characters. For example, if the characters before and after the specific character are characters other than the specific character, the image processing unit 11A may not use the specific character for determining the degree of failure. For example, the image processing unit 11A may not count the specific character as a specific character even if it corresponds to the specific character. When an address is written as "2-3-4-5", the "-" between the numbers is correctly recognized with a high probability. On the other hand, when specific characters are consecutive or appear in a place not between other characters, such as "2--3-4-5", "-2-3-4-5", and "2-3-4-5-", there is a high possibility that a misrecognition has occurred because it does not match the description of a general address. Therefore, by counting only when a specific character is consecutive or when the specific character is not sandwiched between other characters, the image processing unit 11A can prevent a correctly recognized specific character from being treated as an erroneously recognized specific character, which may affect the determination of the degree of failure. Note that, although "-" is described as an example in the above example, the image processing unit 11A may also treat other specific characters as consecutive specific characters when they are consecutive. For example, when "+-" is detected, the image processing unit 11A may count the number of specific characters as two, regarding it as "two consecutive specific characters."

[0079] In addition, in this embodiment, when counting the number of specific characters to determine the degree of failure, the image processing unit 11A may set a difference in the amount of the specific character to be added for each character. For example, in writing a general address, the possibility of using "#" and "+" is low, but the possibility of using "1" is high. Therefore, if "#" and "1" are treated equally, the correct recognition result will be easily determined to be a failure. For example, since both "#+-l" and "11-1" have four specific characters, if the condition for failure is set to three or more specific character types, both character strings will be determined to be a failure. In other words, although "11-1" is more likely to be a correct recognition result, it will be treated the same as "#+-1", which is clearly a failure. Therefore, for example, the image processing unit 11A can reduce the amount of addition when "1" is found compared to the amount of addition when "#" and "+" are found, thereby reducing the possibility of "11-1" being determined to be a failure while maintaining the standard for determining that "#+-1" is a failure.

[0080] In this embodiment, an example of a procedure for determining the degree of failure of the character string recognition result based on the position or positional relationship of each detected character existence region R2 has been described. However, the above-mentioned procedure is merely one example for realizing the idea of ​​determining the degree of failure based on the number of characters whose positions in a direction perpendicular to the direction in which the characters should be written are shifted from other characters among the recognized characters. This idea can be realized by various procedures, for example, as exemplified below.

[0081] The above-described procedure includes a process of arranging the straight line L1 based on the center position CR of each character, but the image processing unit 11A may arrange the straight line L1 by other processes. For example, if the direction of the characters to be written in the recognition target area R1 (e.g., horizontal or vertical writing) is associated in advance, the image processing unit 11A may set the straight line L1 as a straight line that is parallel to this direction and passes through the center of the recognition target area R1. Here, information indicating the direction of the characters to be written in the recognition target area R1 may be held in the memory 12 for each recognition target area R1, for example.

[0082] In the above-described procedure, the image processor 11A arranges a square SQ centered on the center position CR of each character, but the shape of this region may be other shapes such as a rectangle, a circle, or a triangle. Also, the image processor 11A does not need to set the center position CR of each character as the center of the region. However, in order to perform failure judgment accurately, it is desirable to align the positional relationship between the center position CR of each character and each region. This is because there is a risk that variations in the positional relationship between the center position CR of each character and each region will be erroneously judged as a deviation in the position of the character.

[0083] In the above-mentioned procedure, the image processor 11A detects the center position CR of each character. However, in the determination of the degree of failure, it is sufficient to evaluate the position of each character in a direction perpendicular to the direction in which the multiple characters should be written, so only the center position in this perpendicular direction may be detected. However, when the arrangement of the straight line L1 is set based on the center position of each character or an area including the center position, if the position of each character in the direction in which the multiple characters should be arranged is unknown, the inclination of the straight line L1 cannot be specified. Therefore, when adopting the procedure of arranging the straight line L1, the image processor 11A may also obtain information on this position. This position does not necessarily have to be the center position of each character in the direction in which the multiple characters should be arranged, and may be, for example, the edge of the character existence area.

[0084] Also, image processor 11A may determine whether there is a character that is misaligned with other characters without arranging straight line L1. For example, image processor 11A can determine whether there is a misaligned character by detecting the center position of each character in a direction perpendicular to the direction in which the multiple characters are to be written and checking whether there is an outlier included in the distribution of the detected center positions.

[0085] (Overview of the embodiment) As a result, the present disclosure includes at least the following items. Note that in parentheses, examples of components corresponding to the above-described embodiment are given, but the present disclosure is not limited to these.

[0086] (Item 1) A character recognition method for recognizing characters written on a read target (read target TG) including a frame line (frame line WK), comprising: A step of acquiring an image (captured image G1) captured by a camera (camera 13); A step of extracting a recognition target region (recognition target region R1) for recognizing characters from the image; removing noise including a frame line from within the recognition target region; performing a recognition process to recognize a plurality of characters within the recognition target area; determining a degree of failure of a recognition result of the character string including the plurality of characters obtained by the recognition process based on a recognition result for each character in the recognition target area; A character recognition method comprising the steps of:

[0087] As a result, the character recognition method determines the degree of failure in the recognition result of a character string, and obtains the degree of failure as an index of whether or not there is a problem with the recognition result of the character string, making it easier to correctly recognize the character recognition target based on this index.

[0088] (Item 2) The step of determining the degree of bankruptcy includes: determining the degree of corruption based on the number of predetermined characters (specific characters SC) obtained as a recognition result by the recognition process; Item 1. A character recognition method according to item 1.

[0089] For example, if there are many predetermined characters similar to frame lines, not only are there characters that are recognized as predetermined characters as intended, but there is also a possibility that the frame lines, etc., are recognized as predetermined characters. Depending on how many such predetermined characters exist, it is possible to determine how inaccurately the character string is read as a character recognition result.

[0090] (Item 3) The step of determining the degree of bankruptcy includes: determining the degree of corruption based on a position of each character recognized by the recognition process; 3. The character recognition method according to item 1 or 2.

[0091] This allows the character recognition method to determine, in consideration of the detection position of each recognized character, how accurately the character string has been read as a result of character recognition.

[0092] (Item 4) The step of determining the degree of bankruptcy includes: determining the degree of failure based on the number of characters that are shifted from other characters in a direction perpendicular to a direction in which the plurality of characters are to be written, among the plurality of characters recognized by the recognition process; Item 3. The character recognition method according to item 3.

[0093] For example, if there are a large number of characters that are recognized as being out of alignment with other characters, it is highly likely that the recognized characters include a border or the like that is far from the normal position of the character string. The character recognition method can estimate the number of such characters and determine the degree of failure of the character string recognition result.

[0094] (Item 5) a direction in which the plurality of characters should be written is defined in association with the recognition target area; Item 4. A character recognition method according to item 4.

[0095] This allows the character recognition method to easily determine the direction in which multiple characters should be written without having to perform any calculations or the like.

[0096] (Item 6) The step of determining the degree of bankruptcy includes: determining that a direction along which a straight line (straight line L1) that passes through each of predetermined ranges (squares SQ) including each center position (center position CR) of each character recognized by the recognition process is the maximum is a direction along which the plurality of characters should be written; Item 4. A character recognition method according to item 4.

[0097] This allows the character recognition method to derive the direction in which multiple characters should be written, based on the representative center position of each recognized character, even if the direction is not specified in advance.

[0098] (Item 7) The step of determining the degree of bankruptcy includes: determining that a character through which the straight line does not pass is a character whose position in a direction perpendicular to a direction in which the plurality of characters are to be written is shifted from other characters; Item 6. The character recognition method according to item 6.

[0099] This allows the character recognition method to determine characters that are shifted in a direction perpendicular to the character arrangement direction by using a predetermined range that includes the center position of the character.

[0100] (Item 8) The step of determining the degree of bankruptcy includes: If the degree of failure is equal to or greater than a predetermined threshold (threshold th0), determining that the recognition result of the character string is failed. 8. The character recognition method according to any one of items 1 to 7.

[0101] This makes it possible to check whether or not the character recognition method results in a failure in the recognition of the character string.

[0102] (Item 9) if it is determined that the recognition result of the character string is corrupted, repeating the steps of acquiring the image, cutting out the recognition target area, removing the noise, performing the recognition processing, and determining the degree of corruption until the recognition result of the character string is no longer determined to be corrupted. Item 8. The character recognition method according to item 8.

[0103] This enables the character recognition method to obtain good character recognition results (character string recognition results).

[0104] (Item 10) The step of acquiring the images includes acquiring a plurality of images captured by the camera; The step of acquiring the recognition target region includes a step of cutting out the recognition target region for each of the images, The step of removing noise includes removing the noise for each of the images; the step of performing the recognition processing includes a step of performing the recognition processing for each of the images, The step of determining the degree of corruption includes a step of determining the degree of corruption for each of the images, The character recognition method further comprises: determining a recognition result of the character string having the smallest degree of corruption; 8. The character recognition method according to any one of items 1 to 7.

[0105] This allows the character recognition method to select an image and character string recognition result with the least degree of corruption from among the multiple images obtained. Therefore, the character recognition method can obtain a good character recognition result without re-imaging, and can obtain a character recognition result in which the character recognition target is correctly recognized in a short time.

[0106] (Item 11) If it is determined that the character string recognition result is unsuccessful, the method further includes a step of re-imaging an image of a subject including the recognition target area by the camera. Item 8. The character recognition method according to item 8.

[0107] As a result, even if the character string recognition result is not satisfactory, the character recognition method can obtain a new recognition result by re-imaging. In this case, for example, the amount of light varies depending on the timing of imaging, so the captured image may differ. Therefore, even if the character string recognition result is unsuccessful, the character recognition method can obtain an image in which it is easy to remove the frame line from the recognition target area by pre-processing by re-imaging, making it easier to correctly recognize characters.

[0108] (Item 12) If it is determined that the character string recognition result is corrupted, displaying information (guidance information I1) regarding the corrupted character string on a display device. Item 8. The character recognition method according to item 8.

[0109] This allows the character recognition method to provide the user with information about the failure of the character string recognition process, so that the user can take appropriate action, such as correcting some characters using an input device or instructing the user to re-take an image.

[0110] (Item 13) A character recognition device (character recognition device 100) that includes a processor (processor 11) and recognizes characters written on a reading target including a frame line, The processor, Acquire an image captured by a camera, Extracting a recognition target area for recognizing characters from the image; removing noise including the frame line within the recognition target region; performing a recognition process to recognize a plurality of characters within the recognition target area; determining a degree of failure of a recognition result of the character string including the plurality of characters obtained by the recognition process based on a recognition result for each character in the recognition target area; Character recognition device.

[0111] This allows the character recognition device to obtain the same effect as item 1.

[0112] Although various embodiments have been described above with reference to the drawings, it goes without saying that the present disclosure is not limited to such examples. It is clear that a person skilled in the art can come up with various modified or amended examples within the scope of the claims, and it is understood that these also naturally belong to the technical scope of the present disclosure. In addition, the components in the above embodiments may be arbitrarily combined within the scope of the invention.

[0113] In addition, the above embodiment may also be applicable to a program that realizes the functions of a character recognition method, which is supplied to a computer (e.g., character recognition device 100) via a network or various storage media, and which is read and executed by the processor of this computer, as well as to a storage medium on which this program is stored. [Industrial Applicability]

[0114] The present disclosure is useful for a character recognition method and a character recognition device that can easily perform correct character recognition of a character recognition target. [Explanation of symbols]

[0115] 100 character recognition device 11 Processors 11A Image processing unit 12. Memory 13. Camera 14 Input Devices 15 Display Devices TG Read target R1 Recognition target area R2 Character existence area WK Border

Claims

1. A character recognition method for recognizing characters written on a reading target including a frame line, comprising: acquiring an image captured by a camera; A step of extracting a recognition target area for recognizing characters from the image; removing noise including a frame line from within the recognition target region; performing a recognition process to recognize a plurality of characters within the recognition target area; determining a degree of failure of a recognition result of the character string including the plurality of characters obtained by the recognition process based on a recognition result for each character in the recognition target area; A character recognition method comprising the steps of:

2. The step of determining the degree of bankruptcy includes: determining the degree of corruption based on the number of predetermined characters obtained as a recognition result by the recognition process; 2. The character recognition method according to claim 1.

3. The step of determining the degree of bankruptcy includes: determining the degree of corruption based on a position of each character recognized by the recognition process; 2. The character recognition method according to claim 1.

4. The step of determining the degree of bankruptcy includes: determining the degree of failure based on the number of characters that are shifted from other characters in a direction perpendicular to a direction in which the plurality of characters are to be written, among the plurality of characters recognized by the recognition process; 4. The character recognition method according to claim 3.

5. a direction in which the plurality of characters should be written is defined in association with the recognition target area; 5. The character recognition method according to claim 4.

6. The step of determining the degree of bankruptcy includes: determining a direction along which a straight line passing through each of predetermined ranges including each center position of each character recognized by the recognition process is the largest as a direction along which the plurality of characters should be written; 5. The character recognition method according to claim 4.

7. The step of determining the degree of bankruptcy includes: determining that a character through which the straight line does not pass is a character whose position in a direction perpendicular to a direction in which the plurality of characters are to be written is shifted from other characters; 7. The character recognition method according to claim 6.

8. The step of determining the degree of bankruptcy includes: determining that the recognition result of the character string is corrupted when the degree of corruption is equal to or greater than a predetermined threshold value; 3. The character recognition method according to claim 1 or 2.

9. if it is determined that the recognition result of the character string is corrupted, repeating the steps of acquiring the image, cutting out the recognition target area, removing the noise, performing the recognition processing, and determining the degree of corruption until the recognition result of the character string is no longer determined to be corrupted.

9. The character recognition method according to claim 8.

10. The step of acquiring the images includes acquiring a plurality of images captured by the camera; The step of extracting a recognition target region includes a step of extracting the recognition target region for each of the images, The step of removing noise includes removing the noise for each of the images; the step of performing the recognition processing includes a step of performing the recognition processing for each of the images, The step of determining the degree of corruption includes a step of determining the degree of corruption for each of the images, The character recognition method further comprises: determining a recognition result of the character string having the smallest degree of corruption; 2. The character recognition method according to claim 1.

11. If it is determined that the character string recognition result is unsuccessful, the method further includes a step of re-imaging an image of a subject including the recognition target area by the camera.

9. The character recognition method according to claim 8.

12. If it is determined that the character string recognition result is corrupted, displaying information regarding the corrupted character string on a display device.

9. The character recognition method according to claim 8.

13. A character recognition device that includes a processor and recognizes characters written on a reading target including a frame line, The processor, Acquire an image captured by a camera, Extracting a recognition target area for recognizing characters from the image; removing noise including the frame line within the recognition target region; performing a recognition process to recognize a plurality of characters within the recognition target area; determining a degree of failure of a recognition result of the character string including the plurality of characters obtained by the recognition process based on a recognition result for each character in the recognition target area; Character recognition device.

Images