Image judgment method, image judgment device, and character recognition method

Abstract

To provide an image determination method capable of effectively detecting blur in captured images in images of a reading target in which the recognition target is described.SOLUTION: The image determination method is an image determination method performed by a terminal apparatus including a camera capable of imaging an object with characters in a prescribed format. The method includes the steps of: imaging the object by the camera; detecting a point of a blur determination that is outside the character recognition area and at least a part of the predetermined format from the captured image of the captured object; and determining whether the captured image is blurred or not based on the detected blur determination point.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an image assessment method, an image assessment device, and a character recognition method. [Background technology]

[0002] Patent document 1 discloses a mobile phone that includes a shooting target setting means that can be set to either a normal mode in which an image is the shooting target or an OCR function mode in which characters or barcodes are the shooting target, a display means that displays the captured image, a focus state determination means that determines the focus state of the characters or barcodes captured in the OCR function mode, a recognition means that recognizes the captured characters or barcodes, and a recognition result storage means that stores the recognition results obtained by the recognition means. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2006-180476 Summary of the Invention [Problem to be solved by the invention]

[0004] The present disclosure has been devised in consideration of the above-described conventional circumstances, and aims to provide an image assessment method, an image assessment device, and a character recognition method that more effectively detect blurring in an image captured when capturing an image of a reading target on which a recognition target is written. [Means for solving the problem]

[0005] The present disclosure relates to an image determination method performed by a terminal device equipped with a camera capable of capturing an image of an object on which characters are written in a predetermined format, the method including: capturing an image of the object with the camera; detecting at least one straight line that is outside a character recognition area and is at least a part of the predetermined format from the captured image of the object as a blur determination point; and determining whether the captured image is a blurred image based on the detected blur determination point. In the determination, the thickness of the line detected as the blur determination portion is measured, and line thickness information corresponding to the detected line is obtained from line thickness information included in the predetermined format. If it is determined that the measured thickness is equal to or greater than a predetermined multiple of the obtained thickness, the captured image is determined to be a blurred image. The present invention provides an image evaluation method.

[0006] The present disclosure also provides a method for detecting blurred images, the method comprising: detecting, from a captured image of the object having characters written in a predetermined format, at least one straight line that is outside a character recognition area and is at least a part of the predetermined format, as a blur determination point; and determining, based on the detected blur determination point, whether the captured image is a blurred image. The determination unit measures the thickness of the line detected as the blur determination portion, acquires line thickness information corresponding to the detected line from line thickness information included in the predetermined format, and determines that the measured thickness is a predetermined multiple of the acquired thickness or more, and determines that the captured image is a blurred image. The present invention provides an image assessment device.

[0007] The present disclosure also provides a character recognition method performed by a terminal device equipped with a camera capable of capturing an image of an object on which characters are written in a predetermined format, the method comprising: capturing an image of the object using the camera; detecting at least one straight line that is outside a character recognition area and is at least a part of the predetermined format from the captured image of the object as a blur determination point; determining whether the captured image is a blurred image based on the detected blur determination point; if it is determined that the captured image is a blurred image, performing character recognition of the characters written in the character recognition area using a first character recognition model; if it is determined that the captured image is not a blurred image, performing recognition of the characters written in the character recognition area using a second character recognition model; and outputting the recognized character information. In the determination, the thickness of the line detected as the blur determination portion is measured, and line thickness information corresponding to the detected line is obtained from line thickness information included in the predetermined format. If it is determined that the measured thickness is equal to or greater than a predetermined multiple of the obtained thickness, the captured image is determined to be a blurred image. A character recognition method is provided. [Effects of the Invention]

[0008] According to the present disclosure, when imaging a reading target on which a recognition target is written, blur in the captured image can be detected more effectively. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating a use case of the character recognition system according to the first embodiment. [Figure 2] FIG. 1 is a block diagram showing an example of the internal configuration of a terminal device according to a first embodiment. [Figure 3] FIG. 10 is a diagram showing an example of a captured image of a first reading target; [Figure 4] FIG. 10 is a diagram showing an example of a character recognition area in a first reading object; [Figure 5] FIG. 10 is a diagram showing an example of a captured image of a second reading target; [Figure 6] A comparison of an example of a blurred image and a normal image. [Figure 7] FIG. 10 is a diagram showing an example of a character recognition result screen. [Figure 8] FIG. 10 is a diagram showing an example of a re-imaging selection screen; [Figure 9] 10 is a flowchart showing a first example of an operation procedure of a terminal device according to the first embodiment. [Figure 10] Flowchart showing a second example of an operation procedure of the terminal device in the first embodiment [Figure 11] FIG. 10 is a diagram illustrating an example of a first image processing procedure of a terminal device. [Figure 12] FIG. 10 is a diagram illustrating an example of a second image processing procedure of the terminal device. DETAILED DESCRIPTION OF THE INVENTION

[0010] (Background to this disclosure) Conventional mobile phones display a captured image of the target character or barcode on a display screen, or display the focused (hereinafter referred to as "blurred") image on the display screen. This allows the mobile phone to recognize the target character using a captured image captured in good condition (i.e., not blurred). Recently, there has been a demand for recognizing characters written on slips and other documents using smartphones, tablet devices, and the like. However, characters written on slips and other documents can be handwritten using any writing implement, printed using any printing machine, or stamped using a stamp, and therefore the size, thickness, ink, and other characteristics of the characters are indefinite. This makes it difficult for mobile phones to determine the blurred state of the target character, resulting in failure to recognize the written character. Therefore, users must take the slip home and manually input the characters written on the slip into another device while visually checking them, which is time-consuming.

[0011] Furthermore, the process of detecting characters from a captured image generally places a heavy load on the mobile phone, so when detecting characters that are handwritten, printed, or stamped, such as on a slip, and then detecting blurring of the detected characters, the processing is very heavy and places a heavy load on the mobile phone.

[0012] Hereinafter, with reference to the drawings as appropriate, detailed descriptions of embodiments that specifically disclose the configurations and operations of the image assessment method, image assessment device, and character recognition method according to the present disclosure will be described in detail. However, more detailed descriptions than necessary may be omitted. For example, detailed descriptions of already well-known matters or redundant descriptions of substantially identical configurations may be omitted. This is to avoid unnecessary redundancy in the following description and to facilitate understanding by those skilled in the art. Note that the accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure and are not intended to limit the subject matter recited in the claims.

[0013] (Embodiment 1) A use case of the character recognition system according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram illustrating a use case of the image assessment system according to the first embodiment.

[0014] Note that the terminal device P1 shown in Fig. 1 is an example and is not limited to this. For example, the terminal device P1 may be any terminal device that is equipped with a camera 13 (see Fig. 2) and is capable of transmitting and receiving data via a network NW (see Fig. 2), and may be, for example, a tablet terminal, a notebook PC (Personal Computer), etc. Furthermore, an example will be described in which the read object TG in the first embodiment is a slip for a package to be transported, but is not limited to this. For example, the read object TG may be anything that has a predetermined format and a predetermined area in which characters are written, such as a medical questionnaire, an application form, an order form, etc.

[0015] The terminal device P1 is equipped with a camera 13 (see FIG. 2), and performs image processing on a captured image of a read object TG, detects blur from the captured image (specifically, the read object TG shown in the captured image), and determines whether the captured image is a blurred image (hereinafter referred to as "blur determination"). Based on the blur determination result, the terminal device P1 performs OCR (Optical Character Recognition / Reader) processing (hereinafter referred to as "character recognition processing") on the captured image to recognize characters written on the read object TG. The terminal device P1 transmits the character recognition result to a server S1 (see FIG. 2), which is an example of an external terminal, via a network NW (see FIG. 2). The terminal device P1 may transmit a captured image that is determined not to be blurred based on the blur determination result (hereinafter referred to as "normal image") to the server S1.

[0016] The read object TG has a predetermined format and a character writing area in which characters are written. The read object TG has handwritten, printed, or stamped characters that are different from the characters included in the predetermined format of the read object TG. For example, if the read object TG is a slip for a package to be transported, the read object TG has handwritten, printed, or stamped information about the destination (delivery destination), information about the sender requesting delivery of the package, etc. written on it.

[0017] Note that characters written by hand, printed, or stamped on the target TG may smudge or have unclear outlines depending on the ink used, the amount of ink, the material of the target TG, or the environment at the time of writing (for example, low humidity, high humidity, etc.). Also, characters written by hand, printed, or stamped on the target TG may have different character sizes, line thicknesses, etc.

[0018] The internal configuration of the terminal device P1 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing an example of the internal configuration of the terminal device P1 in embodiment 1. Note that although the number of terminal devices shown in Fig. 2 is one, two or more terminal devices may be connected to the server S1 via the network NW so as to be able to send and receive data therebetween.

[0019] The terminal device P1 is wirelessly connected to the server S1 via a network NW. The wireless communication here refers to, for example, short-range wireless communication such as Bluetooth (registered trademark) or NFC (registered trademark), or communication via a wireless LAN (Local Area Network) such as Wi-Fi (registered trademark). The terminal device P1 performs a blur determination process on a captured image of the read target TG. Based on the blur determination result, the terminal device P1 performs a character recognition process on the characters written on the read target TG, and transmits the character recognition result (i.e., the recognized character information) recognized by the character recognition process to the server S1 for storage. The terminal device P1 includes a communication unit 10, a control unit 11, a memory 12, a camera 13, an input unit 14, and a display unit 15. The input unit 14 and the display unit 15 may be integrated into one unit, such as a touch panel.

[0020] The communication unit 10 is connected to the network NW so as to be able to communicate wirelessly, and transmits and receives data to and from the server S1 via the network NW. The communication unit 10 transmits a normal image (i.e., a captured image of the read object TG) output from the control unit 11 to the server S1, and transmits to the server S1 a character recognition result of the read object TG output from the control unit 11 and the captured image of the read object TG in association with each other. Note that the captured image of the read object TG transmitted in association with the character recognition result may be a captured image determined to be blurred (hereinafter referred to as a "blurred image"), or may be a normal image.

[0021] The control unit 11 is configured using, for example, a CPU (Central Processing Unit), a DSP (Digital Signal Processor), or an FPGA (Field Programmable Gate Array), and controls the operation of each unit of the terminal device P1. The control unit 11 works in cooperation with the memory 12 to comprehensively perform various processes and controls. Specifically, the control unit 11 references the programs and data stored in the memory 12 and executes the programs to realize the functions of each unit. Note that each unit referred to here is the image processing unit 11A.

[0022] The image processing unit 11A, which is an example of a detection unit and a determination unit, performs image processing on the captured image output from the camera 13 and performs blur determination processing on the read object TG that appears in the captured image. The image processing unit 11A in the first embodiment detects one or more straight lines or recognition marks (for example, the recognition mark RG1 shown in FIG. 3 or the recognition mark RG2 shown in FIG. 4) from the read object TG that appears in the captured image, and determines whether the captured image is a blurred image (i.e., whether the read object TG that appears in the captured image is blurred) based on each of the detected one or more straight lines or the detected recognition marks. Note that the number of straight lines detected here is not limited to one, and may be two or more. By increasing the number of detected straight lines, the image processing unit 11A can more accurately determine the blur of the captured image that appears in the read object TG.

[0023] First, an example of executing the blur determination process based on one or more straight lines will be described. The straight lines here include, for example, frame lines that form a predetermined format of the reading object TG, ruled lines that are included in the predetermined format and that assist in writing characters, etc.

[0024] The image processing unit 11A performs a Hough transform on the captured image of the read target TG and detects any one or more straight lines from among the multiple straight lines detected by the Hough transform. While the first embodiment exemplifies detecting straight lines by performing a Hough transform, the method for detecting straight lines is not limited to this, and other known techniques may be used. Furthermore, the straight lines detected here may be of different line types (e.g., solid lines, dashed lines, dot-dash lines, etc.), may have different line thicknesses, or may have the same thickness. The image processing unit 11A cuts out the areas surrounding each of the detected straight lines, extracts the edges of each straight line, and calculates the edge strength for each extracted straight line.

[0025] The edge strength here refers to the degree of change in density at the boundary between the edge of the line and its surroundings. The stronger the edge strength, the greater the change in density at the boundary, indicating a clearer distinction between the line and its surroundings. The weaker the edge strength, the smaller the change in density at the boundary, indicating a less clear distinction between the line and its surroundings.

[0026] The image processing unit 11A calculates edge strength statistics based on the calculated edge strengths for each straight line. Note that the statistics referred to here may be the maximum or minimum value of the calculated edge strengths for each straight line, or the average value of the calculated edge strengths for each straight line. The image processing unit 11A determines whether the captured image output from the camera 13 is a blurred image (i.e., whether the captured image is blurred) based on the calculated edge strength statistics.

[0027] Here, the thickness of a line when blurring occurs is greater than the thickness of a line when blurring does not occur. Therefore, instead of edge intensity, image processing unit 11A may calculate the thickness of the detected line and determine whether or not the image is blurred by comparing the calculated thickness with thicknesses of lines pre-stored in memory 12. For example, image processing unit 11A may determine whether or not the calculated thickness of the line is equal to or less than a predetermined multiple (e.g., 1 time or more and 1.2 times, 1.5 times, 2 times, etc.) of the thickness of the line pre-stored in memory 12, and if it is determined that the calculated thickness is equal to or less than the predetermined multiple, determine that the image is not blurred.

[0028] Furthermore, when blurring occurs, the contrast between an arbitrary point (position) on the line and a position near the line (i.e., an arbitrary nearby point (position)) becomes small. The image processing unit 11A calculates the luminance of the detected line and the luminance of a nearby position of the line, instead of edge intensity. The image processing unit 11A may determine whether or not the image is blurred based on the difference (luminance difference) between the calculated maximum luminance of an arbitrary point on the line and the minimum luminance of a nearby position corresponding to the line, or the difference (luminance difference) between the calculated minimum luminance of an arbitrary point on the line and the maximum luminance of a nearby position corresponding to the line. For example, the image processing unit 11A may determine that the image is blurred if it is determined that the calculated luminance difference is not equal to or greater than a predetermined value. Note that the predetermined value may be set to an arbitrary value based on the color of the line and the background color of the read target TG.

[0029] Next, an example of executing the blur determination process based on the recognition mark will be described. The recognition mark here is, for example, a predetermined format of the reading object TG, such as a logo, a mark, or an arbitrary figure.

[0030] The image processor 11A detects a recognition mark from a captured image of the reading target TG. The image processor 11A cuts out a surrounding area including the detected recognition mark, extracts the edge of the recognition mark, calculates the edge strength of the extracted recognition mark, and uses the calculated edge strength as a statistical quantity of the edge strength. The edge strength here indicates the degree of change in shading at the boundary between the edge of the recognition mark and a position adjacent to the recognition mark.

[0031] The image processing unit 11A determines whether or not the captured image output from the camera 13 is a blurred image based on the calculated edge strength statistics.

[0032] The image processing unit 11A may determine whether or not the image is blurred by calculating the size (area) of the detected recognition mark instead of the edge intensity and comparing the calculated size (area) of the recognition mark with the size (area) of the recognition mark previously stored in the memory 12. For example, the image processing unit 11A may determine whether or not the calculated size (area) of the recognition mark is equal to or smaller than a predetermined multiple (e.g., 1 time or more and 1.2 times, 1.5 times, 2 times, etc.) of the size (area) of the recognition mark previously stored in the memory 12, and may determine that the image is not blurred if it is determined that the calculated size (area) is equal to or smaller than the predetermined multiple.

[0033] Furthermore, instead of edge intensity, the image processing unit 11A calculates the luminance of the detected recognition mark and the luminance of a position adjacent to the straight line. The image processing unit 11A may determine whether or not the image is blurred based on the difference (luminance difference) between the calculated maximum luminance of the straight line and the minimum luminance of a position adjacent to the recognition mark, or the difference (luminance difference) between the calculated minimum luminance of the straight line and the maximum luminance of a position adjacent to the recognition mark. For example, the image processing unit 11A may determine that the image is blurred if it determines that the calculated luminance difference is less than a predetermined value. Note that the predetermined value may be set to any value based on the recognition mark color and background color of the reading target TG.

[0034] When the image processing unit 11A determines that the captured image is a blurred image, it generates a screen (re-image selection screen SC5, see FIG. 8) that prompts the user of the terminal device P1 to re-image the read target TG, and outputs it to the display unit 15, or it generates a character recognition result screen SC4 (see FIG. 7) that includes a message MSG4 (see FIG. 7) that notifies that the image is a blurred image, and outputs it to the display unit 15. Here, when the image processing unit 11A executes character recognition processing using a blurred image, it executes character recognition processing using a blurred image recognition model (an example of a first character recognition model) stored in the memory 12.

[0035] On the other hand, if the image processing unit 11A determines that the captured image is not a blurred image (i.e., a normal image), it performs character recognition processing using a recognition model for normal images (an example of a second character recognition model) stored in the memory 12, generates a character recognition result screen, and outputs it to the display unit 15.

[0036] The image processing unit 11A may evaluate the degree of blur (index) of the captured image based on the calculated edge strength statistics or brightness difference. In such a case, the image processing unit 11A acquires the character recognition result of the character recognition region by performing a weighting process based on the evaluated degree of blur on each character recognition result obtained by performing character recognition processing using a blurred image recognition model and a normal image recognition model. The image processing unit 11A may also acquire the character recognition result of the character recognition region by performing character recognition processing using a recognition model corresponding to the evaluated degree of blur.

[0037] Based on the control command output from the input unit 14, the image processing unit 11A associates the character recognition result with the captured image on which the character recognition process was performed, stores the result in the memory 12, and outputs the result to the communication unit 10 for transmission to the server S1.

[0038] Alternatively, the image processing unit 11A may reduce the size of the captured image output from the camera 13 and perform a Hough transform on the reduced captured image to detect straight lines. In this case, the image processing unit 11A extracts each character recognition area (i.e., an area containing characters to be subjected to the character recognition process) from the entire captured image of the read target TG, and performs the character recognition process on each of the extracted character recognition areas. In this way, the image processing unit 11A can reduce the processing load required for the blur determination process by performing the blur determination process using the reduced captured image, and can reduce the processing load required for character detection by performing the character detection process. This allows the image processing unit 11A to speed up both the blur determination process and the character detection process. Furthermore, the image processing unit 11A extracts each extracted character recognition area from the captured image and performs the character recognition process for each extracted character recognition area, thereby reducing the processing load required for the character recognition process and increasing the speed.

[0039] The memory 12 includes, for example, a random access memory (RAM) serving as a work memory used when the control unit 11 executes each process, and a read-only memory (ROM) for storing programs and data that define the operation of the control unit 11. The RAM temporarily stores data or information generated or acquired by the control unit 11. The ROM stores programs that define the operation of the control unit 11. The memory 12 stores a normal image recognition model for recognizing characters written in a captured image determined to be a normal image, and a blurred image recognition model for recognizing characters written in a captured image determined to be a blurred image. Note that if the character recognition process is performed only when the image processing unit 11A determines the image to be normal, the memory 12 does not need to store the blurred image recognition model. The memory 12 may also store format information for each read target TG, the number of lines for each line thickness, identification mark information, character recognition area information (e.g., coordinates, etc.), various thresholds and predetermined values ​​used in the blur determination process, and multiple recognition models for recognizing characters corresponding to the degree of blur in the captured image.

[0040] The camera 13 is configured to have at least a lens (not shown) and an image sensor (not shown). The image sensor is a solid-state imaging element such as a CCD (Charged-Coupled Device) or a CMOS (Complementary Metal-Oxide-Semiconductor), and converts an optical image formed on an imaging surface into an electrical signal. The camera 13 captures an image of the reading target TG based on a user operation via the input unit 14, and outputs the captured image of the reading target TG to the image processing unit 11A of the control unit 11.

[0041] The input unit 14 is a user interface configured using, for example, a touch panel, a keyboard, a mouse, etc. The input unit 14 converts the received distributor TC operation into an electrical signal (control command) and outputs it to the control unit 11. The input unit 14 may be a touch panel configured integrally with the display unit 15.

[0042] The display unit 15 is configured using a display such as an LCD (Liquid Crystal Display) or an organic EL (Electroluminescence) display, etc. The display unit 15 displays the captured image of the reading target TG output from the control unit 11, a character recognition result screen SC4 (see FIG. 7), a re-imaging selection screen SC5 (see FIG. 8), etc.

[0043] The network NW connects the terminal device P1 and the server S1 so as to enable data transmission and reception between them.

[0044] The server S1 is connected to the terminal device P1 via a network NW so as to be able to communicate wirelessly with the terminal device P1. The server S1 associates the captured image of the read target TG or the character recognition result transmitted from the terminal device P1 with the captured image of the read target TG corresponding to the character recognition result, and stores and manages the associated images for each terminal device P1.

[0045] When only the captured image of the read target TG is transmitted from the terminal device P1, the server S1 may be able to perform character recognition processing in the same manner as the image processing unit 11A included in the terminal device P1.

[0046] A straight line or a recognition mark RG1 detected from the first reading target TG1 will be described with reference to Fig. 3. Fig. 3 is a diagram showing an example of a captured image IMG1 of the first reading target TG1. The first reading target TG1 shown in Fig. 3 is a slip as an example, but it goes without saying that the first reading target TG1 is not limited to this. Similarly, it goes without saying that the straight line or the recognition mark RG1 described with reference to Fig. 3 is merely an example and is not limited to this.

[0047] The first reading target TG1 is a slip handwritten by a customer using a specified writing instrument who uses a transportation service. The method of writing characters is not limited to handwriting, and may be printing, affixing a stamp, or the like. The recognition mark RG1 is a mark added for recognizing or identifying the reading target, and may be, for example, a logo, a mark, or any graphic. The image processing unit 11A may detect the recognition mark RG1 from the captured image IMG1 and compare the detected recognition mark RG1 with each of the multiple recognition marks stored in the memory 12 based on the detected recognition mark RG1. When the image processing unit 11A determines that the read object is the first read object TG1 based on the collation result, it acquires information about the first read object TG1 stored in the memory 12 (for example, format information (position information of one or more detected straight lines and line thickness information corresponding to these straight lines, etc.), position information of each character recognition area, various thresholds used for blurred image determination, information such as predetermined values, etc.), and distinguishes the first read object TG1 from other read objects related to the first read object TG1 (for example, a second read object TG2 which is a copy sheet of characters written on the first read object TG1, etc.). Note that the distinction between the first read object TG1 and other read objects related to the first read object TG1 may be performed based on the thickness, line type, number, etc. of all straight lines in the format of the first read object TG1, or may be performed by pattern matching the format of the first read object TG1.

[0048] For example, the first reading target TG1 shown in Fig. 3 has handwritten information about the recipient to be delivered by the transportation service, such as a postal code "XXX-YYYY," a telephone number "0123-456-789," a recipient address "C XYZ, A City, A Prefecture," a recipient's name "○○○ △△," and a delivery date "1st (Mon) 5th (Sun)." Also, the first reading target TG1 has handwritten information about the sender (i.e., customer) who will use the transportation service, such as a postal code "XXX-YXYY," a telephone number "1111-222-333," a sender's address "Room F ZZZ, D City, A Prefecture," and a sender's name "XXX □□."

[0049] The first read target TG1 is captured by the camera 13 included in the terminal device P1. The image processing unit 11A performs a Hough transform on the captured image IMG1 of the captured first read target TG1 to detect one or more straight lines LN11, LN12, LN13, LN14, LN21, and LN22. The image processing unit 11A may detect one or more straight lines LN11, LN12, LN13, LN14, LN21, and LN22 after reducing the size of the captured image IMG1 of the first read target TG1. Each of the four straight lines LN11 to LN14 is thicker than each of the two straight lines LN21 to LN22. The image processing unit 11A determines whether the captured image IMG1 is a blurred image based on these detected straight lines.

[0050] Alternatively, the image processing unit 11A may detect a recognition mark RG1 from the captured first reading target TG1 and determine whether the captured image IMG1 is a blurred image based on the detected recognition mark RG1. In such a case, the image processing unit 11A omits the above-described Hough transform process, the process of detecting one or more straight lines, etc.

[0051] An example of the character recognition area AR1 of the first reading target TG1 will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of the character recognition area AR1 of the first reading target TG1. Note that each of the character recognition areas AR1 of the first reading target TG1 shown in Fig. 4 is an example and is not limited to this.

[0052] The first reading target TG1 has at least one character recognition area AR1, AR2, which is an entry location (area) where a customer writes (writes) characters. The image processing unit 11A detects characters (handwritten characters, printed characters, stamped characters, etc.) written by the customer from the captured image IMG1, and determines these detected areas as character recognition areas AR1. The image processing unit 11A extracts each of the determined or specified multiple character recognition areas AR1 and performs character recognition processing.

[0053] The image processing unit 11A may also perform pattern matching based on the detected format of the first read target TG1, matching one or more straight lines included in the format, or matching the recognition mark RG1, to identify the first read target TG1 stored in the memory 12, and acquire position (coordinate) information for at least one character recognition area of ​​the first read target TG1 stored in the memory 12 by referring to the format information of the first read target TG1 (i.e., position information for each of all character recognition areas). The image processing unit 11A may also compare the acquired position (coordinate) information for each character recognition area of ​​the first read target TG1 with the position of the first read target TG1 shown in the captured image IMG1 to identify the position (coordinate) of each of the character recognition areas AR1 and AR2. The image processing unit 11A extracts each of the determined or identified character recognition areas AR1 and AR2, and performs character recognition processing on the characters written in each of the extracted character recognition areas AR1 and AR2. Here, the image processing unit 11A may perform a detection process for the character recognition area AR2 in which no characters are written among all the character recognition areas AR1 and AR2, and then perform a character recognition process only on the character recognition area AR1 in which characters are written.

[0054] An example of a character recognition area AR2 of the second reading target TG2 will be described with reference to Fig. 5. Fig. 5 is a diagram showing an example of a captured image IMG2 obtained by capturing an image of the second reading target TG2.

[0055] 5 is a copy paper on which characters written on the first read object TG1 are copied. The second read object TG2 has a format in which, for example, the multiple ruled lines DLN1 and DLN2 on the first read object TG1, which are intended to assist customers in writing characters, are omitted. Furthermore, the recognition mark RG2 on the second read object TG2 is different from the recognition mark RG1 on the first read object TG1.

[0056] The image processing unit 11A performs a Hough transform on the captured image IMG2 of the second read target TG2 to detect all of the straight lines LN31 and LN41. The image processing unit 11A classifies all of the detected straight lines by line thickness or line type. For example, the image processing unit 11A classifies the straight lines as having ten straight lines LN31 with a first line thickness and two straight lines LN41 with a second line thickness that is thinner than the first line thickness. The image processing unit 11A compares the classification result (i.e., the format information showing ten straight lines of the first thickness and two straight lines of the second thickness) with the format information of all read targets stored in the memory 12, and identifies the read targets.

[0057] In addition, the image processing unit 11A may detect a recognition mark RG2 from the captured image IMG2 in which the second read object TG2 is captured, and compare the detected recognition mark RG2 with the recognition marks possessed by each of all read objects stored in the memory 12 to identify the read object, or may identify the read object by pattern matching based on the format of the second read object TG2.

[0058] In this way, by identifying the first read target TG1 and the second read target TG2, the image processing unit 11A can associate the captured image IMG1 of the first read target TG1 with the captured image IMG2 of the second read target TG2, and the character recognition results of these captured images IMG1 and IMG2. This allows the image processing unit 11A to support management of the captured images of the read targets and the character recognition results that are sent to and stored in the server S1 from one or more terminal devices.

[0059] The above-described association between the captured images of the plurality of read targets and the character recognition results is not essential and may be omitted. In such a case, the terminal device P1 omits the process of specifying the read targets and the process of identifying the read targets.

[0060] The blur determination process will be described in more detail with reference to Fig. 6. Fig. 6 is a diagram comparing an example of a blurred image and an example of a normal image. In the example shown in Fig. 6, the detection result of one straight line will be described in detail for ease of understanding.

[0061] The captured image SC31 is a captured image that has been determined to be a blurred image by the image processing unit 11A of the terminal device P1, while the captured image SC32 is a captured image that has been determined to be a normal image by the image processing unit 11A.

[0062] The image processing unit 11A detects a straight line LN311 from the captured image SC31 and cuts out an area AR311 including the periphery of the straight line LN311. The image processing unit 11A extracts the edges of the straight line LN311 from the cut-out area AR311 and calculates the edge strength. The image processing unit 11A calculates edge strength statistics based on the calculated edge strengths of one or more straight lines. The image processing unit 11A refers to the memory 12 to obtain a threshold value for determining a blurred image corresponding to the reading target shown in the captured image SC31, and determines whether the calculated edge strength statistics are less than the threshold value. Here, the image processing unit 11A determines that the edge strength statistics are less than the threshold value and determines that the captured image SC31 is a blurred image.

[0063] Similarly, the image processing unit 11A detects a straight line LN312 from the captured image SC32 and cuts out an area AR312 including the periphery of the straight line LN312. The image processing unit 11A extracts the edges of the straight line LN312 from the cut-out area AR312 and calculates the edge intensities. The image processing unit 11A calculates edge intensity statistics based on the calculated edge intensities of one or more straight lines. The image processing unit 11A determines whether the calculated edge intensity statistics are less than a threshold value. Here, the image processing unit 11A determines that the edge intensity statistics are not less than the threshold value and determines that the captured image SC32 is a normal image.

[0064] Next, an example will be described in which it is determined whether an image is blurred or not based on the thickness of an extracted straight line.

[0065] The image processing unit 11A measures the line thickness W1 of the straight line LN311 based on the extracted edge of the straight line LN311. The image processing unit 11A refers to the memory 12 to acquire information about the line thickness of the line to be read that appears in the captured image SC31, and determines whether the measured line thickness W1 is a predetermined multiple (for example, 1 time or more, such as 1.2 times, 1.5 times, 2 times, etc.) of the acquired line thickness or more. Here, the image processing unit 11A determines that the measured line thickness W1 is a predetermined multiple of the acquired line thickness or more, and determines that the captured image SC31 is a blurred image.

[0066] Similarly, the image processing unit 11A measures the line thickness W2 of the straight line LN312 based on the extracted edge of the straight line LN312. The image processing unit 11A determines whether the measured line thickness W2 is equal to or greater than a predetermined multiple of the acquired line thickness. Here, the image processing unit 11A determines that the measured line thickness W2 is not equal to or greater than the predetermined multiple of the acquired line thickness, and determines that the captured image SC32 is a normal image.

[0067] Next, an example will be described in which it is determined whether an image is blurred or not based on the difference in brightness between the brightness of an arbitrary point (position) on the extracted straight line and the brightness of a position near the straight line.

[0068] The image processing unit 11A calculates the luminance of an arbitrary point (position) on the extracted straight line LN311 and the luminance of an arbitrary point (position) in an area AR311 near the extracted straight line LN311. The image processing unit 11A calculates the luminance difference (difference) between the calculated luminance on the straight line LN311 and the luminance of the area AR311 near the straight line LN311. The image processing unit 11A refers to the memory 12 to acquire a threshold (predetermined value) for the luminance difference corresponding to this reading target, and determines whether the calculated luminance difference is equal to or greater than the predetermined value (i.e., whether the detected straight line is blurred). Here, the image processing unit 11A determines that the calculated luminance difference is not equal to or greater than the predetermined value, and determines that the captured image SC31 is a blurred image.

[0069] The image processing unit 11A calculates the luminance of an arbitrary point (position) on the extracted straight line LN312 and the luminance of an area AR312 near the extracted straight line LN312. The image processing unit 11A calculates the luminance difference (difference) between the calculated luminance on the straight line LN312 and the luminance of the area AR312 near the straight line LN312. The image processing unit 11A determines whether the calculated luminance difference is equal to or greater than a predetermined value (i.e., whether the detected straight line is blurred). Here, the image processing unit 11A determines that the calculated luminance difference is equal to or greater than the predetermined value, and determines that the captured image SC32 is a normal image.

[0070] Note that the image processing unit 11A may calculate the luminance of at least one arbitrary position (for example, positions PT31, PT32 shown in FIG. 6) in the regions AR311, AR312 as the luminance of the positions near the straight lines LN311, LN312, or may calculate the average luminance of the entire region AR311. Alternatively, the image processing unit 11A may calculate the luminance of the entire regions AR311, AR312, and select either the maximum or minimum luminance value from the calculated luminance of the entire regions AR311, AR312 based on the luminance on the straight line LN311. Specifically, if the color of the straight lines LN311, LN312 is black (low brightness) and the color near the straight lines LN311, LN312 is white (high brightness), etc., the image processing unit 11A may select the maximum brightness value among the brightness values ​​of the calculated entire areas AR311, AR312, and if the color of the straight line LN311 is white (high brightness) and the color near the straight lines LN311, LN312 is black (low brightness), etc., the image processing unit 11A may select the minimum brightness value among the brightness values ​​of the calculated entire areas AR311, AR312.

[0071] Next, the character recognition result screen SC4 will be described with reference to Fig. 7. Fig. 7 is a diagram showing an example of the character recognition result screen SC4.

[0072] The image processing unit 11A performs character recognition processing on the cut-out image IMG41 of the cut-out character recognition area, and recognizes the characters "0801234" written in the character recognition area. As a result of the character recognition processing, the image processing unit 11A generates a character recognition result screen SC4 based on the recognized character information "0801234", and outputs it to the display unit 15.

[0073] The character recognition result screen SC4 is generated to include a cut-out image IMG41 on which the character recognition process has been executed, a character recognition result DP41 including character information OCR41 "0801234" which is the character recognition result recognized by the character recognition process, and buttons BT41 and BT42. Note that the character recognition result screen SC4 shown in Fig. 7 shows an example in which it is generated to include one cut-out image IMG41 (i.e., one character recognition area) and the character recognition result DP41 corresponding to this cut-out image IMG41, but it goes without saying that if there are multiple character recognition areas, it may be generated to include multiple cut-out images and character recognition results corresponding to each of the multiple cut-out images.

[0074] 7 shows an example of a screen displayed when character recognition processing is performed using a captured image determined to be a blurred image. When character recognition processing is performed using a captured image determined to be a blurred image, the image processing unit 11A generates a message MSG4 indicating that the captured image is a blurred image, such as "The captured image is blurred. Please check it carefully," and generates a character recognition result screen SC4 including the generated message MSG4. This allows the terminal device P1 to notify the user that the captured image is a blurred image, and to suggest a decrease in the character recognition accuracy of the character recognition result or to prompt the user to check the character recognition result more carefully.

[0075] Furthermore, when the image processing unit 11A performs character recognition processing using a captured image determined to be a blurred image, it highlights the character recognition result DP41 or the character information OCR41 by surrounding it with a frame line of a predetermined color (for example, red, blue, green, etc.). This allows the terminal device P1 to highlight the character recognition result (character information) that should be checked more carefully, and therefore makes the character recognition result (character information) that should be checked more carefully visible to the user.

[0076] The button BT41 is selected (pressed) by a user operation when it is determined that the characters displayed in the cropped image IMG41 match the character information OCR41. Specifically, the input unit 14 generates a control command indicating that the characters displayed in the cropped image IMG41 match the character information OCR41, based on the user's selection (pressing) operation of the button BT41, and outputs the control command to the image processing unit 11A in the control unit 11. Based on the control command output from the input unit 14, the image processing unit 11A associates the captured image of the reading target TG with the character information OCR41, and transmits the associated image to the server S1 via the communication unit 10.

[0077] The button BT42 is selected (pressed) by a user operation when it is determined that the characters displayed in the cropped image IMG41 do not match the character information OCR41. Specifically, the input unit 14 generates a control command indicating that the characters displayed in the cropped image IMG41 do not match the character information OCR41, based on the user's operation of selecting (pressing) the button BT41, and outputs the control command to the image processing unit 11A in the control unit 11. Based on the control command output from the input unit 14, the image processing unit 11A may generate a screen (not shown) on which the current character recognition result (character information OCR41) can be corrected by a user operation (manual input), output the screen to the display unit 15, or may perform imaging of the read target TG and character recognition processing again.

[0078] The re-imaging selection screen SC5 will be described with reference to Fig. 8. Fig. 8 is a diagram showing an example of the re-imaging selection screen SC5. Note that the re-imaging selection screen SC5 shown in Fig. 8 is just an example and is not limiting.

[0079] The re-imaging selection screen SC5 is generated by the image processing unit 11A and output to and displayed on the display unit 15 when the captured image is determined to be a blurred image. The display process of the re-imaging selection screen SC5 is executed by the terminal device P1 in the process of step St21 shown in FIG. 9, for example. Note that generation of the re-imaging selection screen SC5 is not essential, and may be executed only if a setting has been made in advance to perform re-imaging when the captured image is determined to be a blurred image. Furthermore, this setting may be made by the user.

[0080] The re-imaging selection screen SC5 is generated to include a message MSG5 such as "The captured image is blurred. Do you want to capture it again?" that prompts the user to re-image the read target TG, and a plurality of buttons BT51 and BT52.

[0081] Button BT51 is selected (pressed) by a user operation when re-imaging the read target TG. Specifically, based on the user's operation of selecting (pressing) button BT51, input unit 14 generates a control command requesting re-imaging of the read target TG and outputs it to control unit 11. Based on the control command output from input unit 14, control unit 11 drives camera 13 so that the read target TG can be re-imaged.

[0082] Button BT52 is selected (pressed) by a user operation when re-imaging of the read target TG is not to be performed. Specifically, based on the user's selection (pressing) operation of button BT51, input unit 14 generates a control command to omit re-imaging of the read target TG and outputs the control command to image processing unit 11A in control unit 11. Based on the control command output from input unit 14, image processing unit 11A executes extraction processing of at least one character recognition region from the captured image determined to be a blurred image, and character recognition processing using a blurred image recognition model. Note that image processing unit 11A may terminate the character recognition processing based on the control command output from input unit 14.

[0083] Next, a first operation procedure executed by the terminal device P1 will be described with reference to Fig. 9. Fig. 9 is a flowchart showing an example of the first operation procedure of the terminal device P1 in the first embodiment.

[0084] The input unit 14 converts the received distributor TC operation into an electrical signal (control command) and outputs it to the control unit 11. The control unit 11 outputs the control command output from the input unit 14 to the camera 13 to capture an image of the slip that is the object to be read TG. The camera 13 captures an image of the object to be read TG and outputs the captured image of the object to be read TG to the image processing unit 11A in the control unit 11 (St11).

[0085] The image processing unit 11A reduces the image size of the captured image output from the camera 13 (St12). For example, if the image size is 3000×2000, the image processing unit 11A reduces it to a smaller image size such as 600×400. Note that the processing of step St12 is not essential and may be omitted.

[0086] The image processing unit 11A detects each of a plurality of straight lines from the captured image or the reduced captured image (St13). The image processing unit 11A cuts out the periphery of each of the detected straight lines and extracts the edges of each of the straight lines (St14).

[0087] The image processing unit 11A calculates edge strength from each edge of the extracted straight lines, and calculates statistics of the edge strength of the captured image based on these edge strengths (St15).

[0088] In the processing of step St15, the image processing unit 11A may calculate the thickness of one or more detected straight lines instead of the edge strength, or may calculate the maximum brightness value on the detected straight lines and the minimum brightness value at positions near the straight lines.

[0089] Here, the image processing unit 11A identifies the read object TG by comparing the arrangement pattern of each of the multiple lines in the captured image or the number of lines classified by line thickness with the format for each read object TG stored in the memory 12 (i.e., the arrangement pattern of each of the multiple lines, the number of lines classified by line type or thickness, etc.). The image processing unit 11A may also detect a recognition mark (e.g., a logo, mark, or any graphic of a transportation company) from the captured image of the read object TG and identify the read object TG by comparing the detected recognition mark with the recognition mark for each read object TG stored in the memory 12. Note that the processing of step St16 is not essential and may be omitted, or may be performed at another timing, such as after step St12.

[0090] The image processing unit 11A determines whether the captured image is a blurred image based on whether the calculated statistical amount of edge strength is less than a threshold value (St17).

[0091] If the image processing unit 11A determines in the process of step St17 that the edge intensity statistics are less than the threshold and that the captured image is a blurred image (St18, YES), it extracts at least one character recognition area for executing character recognition processing from the reading target TG shown in the captured image (St20).The image processing unit 11A uses the blurred image recognition model stored in the memory 12 to execute character recognition processing on each of the extracted character recognition areas, thereby recognizing the written characters (St23).

[0092] Here, the image processing unit 11A is set in advance or by user settings, etc., to perform character recognition processing only when the image is normal, and if, in the processing of step St17, the edge strength statistics are less than the threshold and the captured image is determined to be a blurred image (St18, YES), it determines whether or not a setting has been made in advance to re-image the read target TG (St19).

[0093] If the image processing unit 11A determines in the processing of step St19 that a setting for re-imaging the read target TG has been made in advance (St19, YES), the image processing unit 11A proceeds to the processing of step St21, and may generate a re-imaging selection screen SC5 (see FIG. 8) for prompting (requesting) the user to re-imaging the read target TG, and output and display the screen on the display unit 15 (St21). This allows the image processing unit 11A to improve the character recognition accuracy of the character recognition processing. On the other hand, if the image processing unit 11A determines in the processing of step St19 that a setting for re-imaging the read target TG has not been made in advance (St19, NO), the image processing unit 11A extracts at least one character recognition area for executing character recognition processing from the read target TG shown in the captured image (St20).

[0094] On the other hand, if the image processing unit 11A determines in the process of step St17 that the edge intensity statistics are not less than the threshold and that the captured image is not a blurred image (St18, NO), it extracts at least one character recognition area for which character recognition processing is to be performed from the read object TG shown in the captured image (St20). Here, if the image processing unit 11A has performed the process of step St12, it performs the process of step St20 using the original captured image (i.e., before reduction). The image processing unit 11A uses a normal image recognition model stored in the memory 12 to perform character recognition processing on each of the extracted character recognition areas and recognizes the written characters (St22). Note that the normal image recognition model referred to here is a model used when the image is determined to be normal.

[0095] The image processing unit 11A acquires the character recognition result (i.e., character information for each character recognition area) by character recognition processing using the normal image recognition model or the blurred image recognition model. The image processing unit 11A generates a character recognition result screen SC4 (see FIG. 7) including the acquired character recognition result and a clipped image of the extracted character recognition area, and outputs it to the display unit 15 for display (St24). Note that the character recognition result screen SC4 may be generated to include only the character recognition result.

[0096] In the above-described first operation procedure, the image processing unit 11A has been described as performing character recognition processing using either a blurred image recognition model or a normal image recognition model, but character recognition processing may also be performed using a blurred image recognition model and a normal image recognition model.

[0097] In such a case, in the process of step St17, the image processing unit 11A evaluates the degree of blur (index) of the captured image based on the calculated edge intensity statistics. Furthermore, the image processing unit 11A performs character recognition processing using a blurred-image recognition model and a normal-image recognition model on the character recognition region extracted in the process of step St20. The image processing unit 11A weights the character recognition results using the blurred-image recognition model and the normal-image recognition model based on the degree of blur evaluated in the process of step St17, and acquires the character recognition results for the character recognition region. For example, if the image processing unit 11A evaluates the degree of blur to be 30%, it weights the character recognition result using the blurred-image recognition model by 30% and the character recognition result using the normal-image recognition model by 70%, and acquires the character recognition results for the character recognition region.

[0098] Furthermore, the image processing unit 11A may execute character recognition processing for at least one extracted character recognition area using a recognition model corresponding to the evaluated degree of blur. For example, if the image processing unit 11A evaluates the degree of blur to be 30%, it executes character recognition processing for the character recognition area using a recognition model corresponding to this degree of blur, and obtains a character recognition result for the character recognition area.

[0099] This allows the image processing unit 11A to more adaptively select a recognition model to be used for character recognition processing based on the degree of blur in the captured image, and to improve the accuracy of character recognition by performing character recognition processing that is more suitable for the degree of blur.

[0100] Next, a second operation procedure executed by the terminal device P1 will be described with reference to Fig. 10. Fig. 10 is a flowchart showing an example of the second operation procedure of the terminal device P1 in embodiment 1. The second operation procedure shown in Fig. 10 is an operation procedure for repeatedly capturing an image of the read object TG until it is determined that the captured image of the read object TG is not a blurred image (that is, until a normal image is acquired).

[0101] The input unit 14 converts the received distributor TC operation into an electrical signal (control command) and outputs it to the control unit 11. The control unit 11 outputs the control command output from the input unit 14 to the camera 13 to capture an image of the slip that is the reading target TG. The camera 13 captures an image of the reading target TG and outputs the captured image of the reading target TG to the image processing unit 11A in the control unit 11 (St31).

[0102] The image processing unit 11A detects each of a plurality of straight lines from the captured image or the reduced captured image (St32). The image processing unit 11A cuts out the periphery of each of the detected straight lines and extracts the edges of each of the straight lines (St33).

[0103] The image processing unit 11A calculates edge strength from each edge of the extracted multiple straight lines, and calculates statistics of the edge strength of the captured image based on these edge strengths (St34). Note that in the processing of step St34, the image processing unit 11A may calculate the thickness of one or more detected straight lines instead of the edge strength, or may calculate the maximum value of luminance on the detected straight lines and the minimum value of luminance at positions near the straight lines.

[0104] Here, the image processing unit 11A identifies the read object TG by comparing the arrangement pattern of each of the multiple lines in the captured image or the number of lines classified by line thickness with the format for each read object TG stored in the memory 12 (i.e., the arrangement pattern of each of the multiple lines, the number of lines classified by line type or thickness, etc.). The image processing unit 11A may also detect a recognition mark (e.g., a logo, mark, or any graphic of a transportation company) from the captured image of the read object TG and identify the read object TG by comparing the detected recognition mark with the recognition mark for each read object TG stored in the memory 12. The processing of step St16 is not essential and may be omitted, or may be performed at another timing, such as after step St32.

[0105] The image processing unit 11A determines whether the captured image is a blurred image based on whether the calculated statistical amount of edge strength is less than a threshold value (St36).

[0106] If the image processing unit 11A determines in the process of step St36 that the statistical amount of edge intensity is less than the threshold and that the captured image is a blurred image (St37, YES), the process proceeds to the process of step St31.

[0107] On the other hand, if the image processing unit 11A determines in the processing of step St36 that the edge intensity statistics are not less than the threshold and that the captured image is not a blurred image (St37, NO), it extracts at least one character recognition area for executing character recognition processing from the reading target TG shown in the captured image (St38).The image processing unit 11A executes character recognition processing on each of the extracted character recognition areas using a normal image recognition model stored in the memory 12, and recognizes the written characters (St39).Note that if the terminal device P1 does not execute the first operation procedure and executes only the second operation procedure, the memory 12 does not need to store a blurred image recognition model.

[0108] The image processing unit 11A acquires the character recognition result (i.e., character information for each character recognition area) by the character recognition process using the normal image recognition model. The image processing unit 11A generates a character recognition result screen SC4 (see FIG. 7) including the acquired character recognition result and a clipped image of the extracted character recognition area, and outputs it to the display unit 15 for display (St40). Note that the character recognition result screen SC4 may be generated to include only the character recognition result.

[0109] As described above, the terminal device P1 in the first embodiment can acquire a normal image of the reading target TG and perform character recognition processing using the acquired normal image. This allows the terminal device P1 to suppress a decrease in character recognition accuracy and suppress an increase in the number of times and amount of corrections made by the user to the character recognition results.

[0110] Furthermore, when the user causes the server S1 to execute the processes executed in steps St38 to St39 (that is, when the user requests the server S1 to transmit only the captured image of the read target TG), the terminal device P1 transmits only the captured image determined to be a normal image to the server S1. In such a case, the process of step St40 may be omitted or may be performed by the server S1 and the terminal device P1. For example, in step St40, the server S1 generates a character recognition result screen SC4 and transmits the generated character recognition result screen SC4 to the terminal device P1. The terminal device P1 outputs (displays) the character recognition result screen SC4 transmitted from the server S1 on the display unit 15.

[0111] Next, the first captured image selection procedure executed by the terminal device P1 will be described with reference to Fig. 11. Fig. 11 is a diagram illustrating the first captured image selection procedure of the terminal device P1. Note that the first captured image selection procedure here is processing executed in the second operation procedure shown in Fig. 10.

[0112] In the first captured image selection procedure, the camera 13 in the terminal device P1 captures images of the read target TG multiple times at approximately the same time (St41). The multiple captures mentioned here may be achieved by extracting frames before and after a single captured image captured by a user operation, or may be achieved by capturing a video of the read target TG and extracting each of the multiple captured images included in the captured video. The camera 13 outputs each of the captured multiple captured images to the image processing unit 11A of the control unit 11. The process of step St41 is the process executed in step St31.

[0113] The image processing unit 11A executes blur determination processing in parallel for each of the plurality of captured images at approximately the same time, based on each of the plurality of captured images output from the camera 13 (St42). Note that the blur determination processing (processing of step St42) referred to here is the processing executed in steps St32 to St37 of the second operation procedure shown in Fig. 10. Specifically, in the example shown in Fig. 11, the image processing unit 11A executes blur determination processing in parallel for each of the four captured images, based on each of the four captured images output from the camera 13.

[0114] The image processing unit 11A selects one captured image with the least blur (i.e., the largest edge intensity statistic) as the normal image from among at least one captured image determined to be a normal image as a result of the processing in step St42 (St43). The image processing unit 11A performs character recognition using the selected captured image (St44). Here, the processing in step St43 is processing that is executed at the timing of transition from the processing in step St37 to the processing in step St38. The processing in step St44 is processing that is executed in steps St39 to St40.

[0115] 11, the camera 13 captures images of the read target TG multiple times at time t11 (St41) and outputs each of the four captured images to the image processing unit 11A of the control unit 11. The image processing unit 11A acquires each of the four captured images output from the camera 13 and performs blur determination processing on each of the four captured images approximately simultaneously at time t12 (St42). That is, in the first captured image selection procedure, the terminal device P1 captures images of the read target TG using the camera 13 and performs multiple processes (each of "Process 1," "Process 2," "Process 3," and "Process 4" shown in FIG. 11) of determining blur of the captured images approximately simultaneously or in parallel.

[0116] As described above, the terminal device P1 captures images of the read target TG multiple times and selects the captured image with the least blur (i.e., the largest edge intensity statistic) from among the captured images, thereby selecting the captured image on which the character recognition process is to be performed. As a result, when repeatedly capturing images of the read target TG until it is determined that the captured image of the read target TG is not a blurred image (i.e., until a normal image is acquired), as shown in the second example of the operation procedure, the terminal device P1 can perform the blur determination process for each of the multiple captured images in parallel, thereby reducing the number of times the user re-imaging, and making the blur determination process more efficient. Furthermore, because the terminal device P1 can perform the blur determination process for each of the multiple captured images in parallel, the time required to generate and display (present) the character recognition result screen SC4 using a normal image can be further reduced.

[0117] Next, the second captured image selection procedure executed by the terminal device P1 will be described with reference to Fig. 12. Fig. 12 is a diagram illustrating the second captured image selection procedure of the terminal device P1. Note that the second captured image selection procedure here is processing executed in the second operation procedure shown in Fig. 10.

[0118] In the second captured image selection procedure, the camera 13 in the terminal device P1 captures a moving image of the read object TG and sequentially outputs each of the multiple captured images (i.e., frames) included in the captured moving image to the image processing unit 11A of the control unit 11. The image processing unit 11A sequentially executes a blur determination process on each of the captured images output from the camera 13. If the image processing unit 11A determines that the captured image is a normal image as a result of the blur determination process, it generates and outputs a control command to the camera 13 to end capturing of the read object TG, and executes character recognition processing on the captured image determined to be a normal image.

[0119] The processes of steps St51A to St51D are the same as the processes executed in step St31 shown in Fig. 10. The blur determination process (the processes of steps St52A to St52D) referred to here is the process executed in steps St32 to St37 of the second operation procedure shown in Fig. 10.

[0120] 12, camera 13 performs a first imaging at time t21 (St51A), a second imaging at time t22 (St51B), a third imaging at time t23 (St51C), and a fourth imaging at time t24 (St51D). After performing the fourth imaging, camera 13 ends the imaging process based on a control command output from image processing unit 11A.

[0121] In the example shown in FIG. 12, the image processing unit 11A acquires a first captured image captured by the camera 13 at time t21 and performs blur determination processing on the first captured image at time t22 (St52A). If the image processing unit 11A determines that the first captured image is a blurred image, it acquires a second captured image captured by the camera 13 at time t22 and performs blur determination processing on the second captured image at time t23 (St52B). If the image processing unit 11A determines that the second captured image is a blurred image, it acquires a third captured image captured by the camera 13 at time t23 and performs blur determination processing on the third captured image at time t24 (St52C). If the image processing unit 11A determines that the third captured image is a normal image, it generates and outputs a control command to the camera 13 to end image capture, and performs character recognition processing on the third captured image determined to be a normal image (St53).

[0122] As described above, the terminal device P1 sequentially performs an imaging process of the read target TG using the camera 13 and a blur determination process of the captured image using the image processing unit 11A, and if the captured image is determined to be a normal image as a result of the blur determination process, it performs a character recognition process, thereby enabling more efficient acquisition of the normal image. Furthermore, the terminal device P1 continues to perform the imaging process and the blur determination process until a normal image is acquired, eliminating the need for the user to perform a re-imaging operation.

[0123] As described above, the terminal device P1 in embodiment 1 is equipped with a camera 13 capable of capturing an image of a read object TG (an example of an object) on which characters are written in a predetermined format, and captures an image of the read object TG using the camera 13. From the captured image of the read object TG, it detects blur determination points (e.g., a straight line, one or more straight lines, a recognition mark, all straight lines forming the predetermined format, etc.) that are outside the character recognition area (e.g., each of the character recognition areas AR1 and AR2 shown in Figure 4) and are at least part of the predetermined format, and determines whether the captured image is a blurred image based on the detected blur determination points.

[0124] As a result, the terminal device P1 in embodiment 1 can more efficiently determine whether the captured image itself is a blurred image, even when the characters written on the read object TG are characters written by hand, printed, or stamped, and the characters are blurred or the outlines are unclear due to the ink used, the amount of ink, the material of the read object TG, or the environment at the time of writing (for example, low humidity, high humidity, etc.). Furthermore, as a result, the terminal device P1 can determine whether the read object TG itself shown in the captured image is blurred, rather than whether the characters themselves written on the read object TG are blurred.

[0125] Furthermore, as described above, the terminal device P1 in the first embodiment calculates the edge strength of the detected blur determination portion (for example, one or more straight lines, recognition marks RG1, RG2, etc.), and determines whether or not the captured image is a blurred image based on the calculated edge strength of the blur determination portion. As a result, the terminal device P1 in the first embodiment can determine whether or not the read object TG itself shown in the captured image is blurred, based on the edge strength of the blur determination portion detected from the read object TG shown in the captured image.

[0126] Furthermore, as described above, the terminal device P1 in the first embodiment further calculates edge strength statistics based on the calculated edge strength, and determines that the captured image is a blurred image if it determines that the calculated statistics are less than the first threshold. This allows the terminal device P1 in the first embodiment to determine whether the read object TG itself shown in the captured image is blurred or not, based on the edge strength statistics of the blur determination portion detected from the read object TG shown in the captured image.

[0127] Furthermore, as described above, the terminal device P1 in the first embodiment calculates a first luminance on the blur determination location and a second luminance near the blur determination location from the detected blur determination location, and determines that the captured image is a blurred image if it is determined that the luminance difference between the calculated first luminance and the second luminance is less than a second threshold value (an example of a predetermined value). As a result, the terminal device P1 in the first embodiment can determine whether the read object TG itself shown in the captured image is blurred, based on the luminance difference of the blur determination location detected from the read object TG shown in the captured image.

[0128] Furthermore, as described above, the terminal device P1 in the first embodiment reduces the image size of the acquired captured image and detects a blur determination portion, which is at least a part of a predetermined format outside the character recognition area, from the reduced captured image. This allows the terminal device P1 in the first embodiment to reduce the processing load required for the detection process of the blur determination portion and the blur image determination process, thereby making these processes more efficient and speeding up.

[0129] Furthermore, as described above, the terminal device P1 in the first embodiment captures a plurality of captured images, detects a blur determination portion from each of the captured images, and determines the captured image in which the blur determination portion with the smallest blur is detected as a non-blurred image (i.e., a normal image). This allows the terminal device P1 in the first embodiment to more efficiently acquire a blurred image with less blur or a normal image.

[0130] Furthermore, as described above, when the terminal device P1 in the first embodiment determines that the captured image is a blurred image, it re-images the read object TG, re-detects the blur determination portion from the re-imaged captured image, and re-determines whether the re-imaged captured image is a blurred image based on the re-detected blur determination portion. As a result, the terminal device P1 in the first embodiment can repeatedly image the read object TG and perform the blur determination process on the captured image until it acquires a normal image.

[0131] Furthermore, as described above, the terminal device P1 in the first embodiment detects a blur determination portion from a captured image, and executes in parallel a first process of determining whether the captured image is a blurred image or not based on the detected blur determination portion, and a second process of re-imaging the read object TG using the camera 13, and if it is determined that the captured image is not a blurred image, the second process being executed in parallel is stopped. As a result, the terminal device P1 in the first embodiment can further shorten the time until a normal image is acquired by executing in parallel the first process of imaging the read object TG and the second process of determining blur of the captured image.

[0132] Furthermore, as described above, the blur determination portion detected in the first embodiment stores the thickness information of all the lines included in the predetermined format, detects at least one line as the blur determination portion, measures the thickness of the line, obtains the thickness information of the line corresponding to the detected line, and determines that the captured image is a blurred image if it is determined that the measured thickness is equal to or greater than a predetermined multiple of the obtained thickness. As a result, the terminal device P1 in the first embodiment can determine whether the reading object TG itself shown in the captured image is blurred, based on the thickness information of the at least one detected line and the thickness information of this line stored in the memory 12.

[0133] As described above, the blur determination portion detected in the first embodiment is a recognition mark (an example of a predetermined mark) included in a predetermined format. As a result, the terminal device P1 in the first embodiment can determine whether or not the read object TG itself shown in the captured image is blurred, based on at least two straight lines forming the recognition mark included in the predetermined format of the read object TG, rather than on the written characters.

[0134] As described above, the terminal device P1 in embodiment 1 comprises a camera 13 capable of capturing an image of a reading object TG on which characters are written in a predetermined format, an image processing unit 11A (an example of a detection unit) that detects a blur determination point that is outside the character recognition area and is at least a part of the predetermined format from the captured image captured by the camera 13, and an image processing unit 11A (an example of a determination unit) that determines whether the captured image is a blurred image or not based on the detected blur determination point.

[0135] As a result, the terminal device P1 in the first embodiment calculates the edge strength of the detected blur determination portion (for example, each of two or more straight lines, recognition marks RG1, RG2, etc.), and determines whether or not the captured image is a blurred image based on the calculated edge strength of the blur determination portion. As a result, the terminal device P1 in the first embodiment can determine whether or not the read object TG itself shown in the captured image is blurred, based on the edge strength of the blur determination portion detected from the read object TG shown in the captured image.

[0136] As described above, the terminal device P1 in embodiment 1 captures an image of a read object TG on which characters are written in a predetermined format using the camera 13, detects a blur determination point from the captured image of the read object TG that is outside the character recognition area and is at least part of the predetermined format, determines whether the captured image is a blurred image or not based on the detected blur determination point, and if it determines that the captured image is a blurred image, performs character recognition of the characters written in the character recognition area using a recognition model for blurred images (an example of a first character recognition model), and if it determines that the captured image is not a blurred image, performs recognition of the characters written in the character recognition area using a recognition model for normal images (an example of a second character recognition model), and outputs the recognized character information.

[0137] As a result, the terminal device P1 in embodiment 1 can more efficiently determine whether the captured image itself is a blurred image, and by selectively using a character recognition model for blurred images and a character recognition model for normal images based on the determination result, it is possible to perform character recognition processing more adaptively and further improve character recognition accuracy.

[0138] Although various embodiments have been described above with reference to the accompanying drawings, the present disclosure is not limited to such examples. It is clear that those skilled in the art can conceive of various modifications, alterations, substitutions, additions, deletions, and equivalents within the scope of the claims, and it is understood that these also fall within the technical scope of the present disclosure. Furthermore, the components of the various embodiments described above may be combined in any manner without departing from the spirit of the invention. [Industrial Applicability]

[0139] INDUSTRIAL APPLICABILITY The present disclosure is useful as providing an image assessment method, an image assessment device, and a character recognition method that more effectively detect blurring in an image captured when capturing an image of a reading target on which a recognition target is written. [Explanation of symbols]

[0140] 10. Communications Department 11 Control section 11A Image processing unit 12 Memory 13 Camera 14 Input section 15 Display section NW Network P1 terminal equipment S1 Server SC4 character recognition result screen SC5 Re-imaging selection screen TG reading target TG1 First reading target TG2 Second reading target

Claims

1. An image determination method performed by a terminal device equipped with a camera capable of capturing an image of an object having characters written in a predetermined format, taking an image of the object with the camera; Detecting at least one straight line that is outside a character recognition area and is at least a part of the predetermined format from the captured image of the object as a blur determination portion; determining whether the captured image is a blurred image based on the detected blur determination portion; In the determination, measuring the thickness of the straight line detected as the blur determination portion; acquiring line thickness information corresponding to the detected line from line thickness information included in the predetermined format; If it is determined that the measured thickness is equal to or greater than a predetermined multiple of the acquired thickness, it is determined that the captured image is a blurred image. Image judgment method.

2. Calculating edge strength of the detected blur determination portion; determining whether the captured image is the blurred image based on the calculated edge intensity of the blur determination portion; The image determination method according to claim 1 .

3. further calculating statistics of the edge strength based on the calculated edge strength; If it is determined that the calculated statistical amount is less than a first threshold, it is determined that the captured image is the blurred image. The image determination method according to claim 2 .

4. calculating a first luminance on the detected blur determination location and a second luminance in the vicinity of the blur determination location from the detected blur determination location; When it is determined that the calculated luminance difference between the first luminance and the second luminance is less than a second threshold, it is determined that the captured image is the blurred image. The image determination method according to claim 1 .

5. reducing an image size of the acquired captured image, and detecting the blur determination portion, which is at least a part of the predetermined format outside a character recognition area, from the reduced captured image; The image determination method according to claim 1 .

6. Taking a plurality of the captured images, Detecting the blur determination portion from each of the plurality of captured images; determining that the captured image in which the blur determination portion with the smallest blur is detected among the plurality of blur determination portions is not the blurred image; The image determination method according to claim 1 .

7. If it is determined that the captured image is a blurred image, re-capturing the object is performed; Redetecting the blur determination portion from the re-captured captured image; re-determining whether the re-captured captured image is the blurred image based on the re-detected blur determination portion; The image determination method according to claim 1 .

8. a first process of detecting the blur determination portion from the captured image, and determining whether the captured image is the blurred image based on the detected blur determination portion, and a second process of re-imaging the object using the camera, are executed in parallel; If it is determined that the captured image is not the blurred image, the second process being executed in parallel is stopped. The image determination method according to claim 1 .

9. the blur determination portion is a predetermined mark included in the predetermined format, The image determination method according to claim 1 .

10. a camera capable of capturing an image of an object having characters written in a predetermined format; a detection unit that detects, from the captured image of the object captured by the camera, at least one straight line that is outside a character recognition area and that is at least a part of the predetermined format as a blur determination portion; a determination unit that determines whether the captured image is a blurred image based on the detected blur determination portion, The determination unit measuring the thickness of the straight line detected as the blur determination portion; acquiring line thickness information corresponding to the detected line from line thickness information included in the predetermined format; If it is determined that the measured thickness is equal to or greater than a predetermined multiple of the acquired thickness, it is determined that the captured image is a blurred image. Image assessment device.

11. A character recognition method performed by a terminal device equipped with a camera capable of capturing an image of an object on which characters are written in a predetermined format, taking an image of the object with the camera; Detecting at least one straight line that is outside a character recognition area and is at least a part of the predetermined format from the captured image of the object as a blur determination portion; determining whether the captured image is a blurred image based on the detected blur determination portion; If it is determined that the captured image is the blurred image, a first character recognition model is used to perform character recognition of characters written in the character recognition area; If it is determined that the captured image is not the blurred image, a second character recognition model is used to recognize the characters written in the character recognition area; outputting the recognized character information; In the determination, measuring the thickness of the straight line detected as the blur determination portion; acquiring line thickness information corresponding to the detected line from line thickness information included in the predetermined format; If it is determined that the measured thickness is equal to or greater than a predetermined multiple of the acquired thickness, it is determined that the captured image is a blurred image. Character recognition method.

Images