Image processing device, image processing system, output device, image processing method, and image processing program

Abstract

The present invention provides an image processing device, an image processing system, an output device, an image processing method, and an image processing program that can improve the accuracy of character recognition for characters that are close to each other. [Solution] The acquisition processing unit 111 acquires image data containing characters. The rectangle candidate detection processing unit 113 detects character rectangle candidates from the image data and calculates the confidence level of the detected rectangle candidates. The single character extraction processing unit 114 corrects the confidence level of the rectangle candidates to be corrected among the first and second rectangle candidates, using a method for calculating the degree of overlap, which is selected according to the relationship between the type of the first character corresponding to the first rectangle candidate and the type of the second character corresponding to the second rectangle candidate, when at least a portion of the first and second rectangle candidates detected by the rectangle candidate detection processing unit 113 overlap with each other. The single character extraction processing unit 114 extracts single character rectangles based on the corrected confidence level.

Description

Technical Field

[0001] This disclosure relates to a technique for performing image processing such as character recognition on an input image.

Background Art

[0002] Conventionally, techniques for recognizing (OCR processing) handwritten characters in documents, forms, etc. are known. For example, a technique for extracting a handwritten area filled in by hand and a printed area where typeface is printed from a form, and recognizing the character string in the printed area when the degree of overlap with the handwritten area satisfies a predetermined condition is known (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the conventional technology, for example, when two characters are close to each other and the character area rectangles overlap each other, one of the characters may be removed, or an unnecessary character area rectangle other than the two character area rectangles may be extracted, resulting in a problem of reduced character recognition accuracy.

[0005] An object of this disclosure is to provide an image processing apparatus, an image processing system, an output device, an image processing method, and an image processing program capable of improving the character recognition accuracy of characters that are close to each other.

Means for Solving the Problems

[0006] An image processing apparatus according to one aspect of the present disclosure comprises an acquisition processing unit, a detection processing unit, a correction processing unit, and an extraction processing unit. The acquisition processing unit acquires image data including a target for detection. The detection processing unit detects rectangle candidates for the target for detection from the image data and calculates the confidence level of the detected rectangle candidates. When at least a portion of the first rectangle candidates and second rectangle candidates detected by the detection processing unit overlap, the correction processing unit corrects the confidence level of the rectangle candidate to be corrected among the first rectangle candidates and the second rectangle candidates, using a method for calculating the degree of overlap, which is selected according to the relationship between the type of first detection target corresponding to the first rectangle candidate and the type of second detection target corresponding to the second rectangle candidate. The extraction processing unit extracts the rectangle of the target for detection based on the confidence level corrected by the correction processing unit.

[0007] An image processing system according to another aspect of this disclosure comprises the image processing device and a learning device that generates a trained model by performing machine learning using the training data generated by the image processing device.

[0008] An output device according to another aspect of the present disclosure uses the trained model generated by the learning device to perform character recognition processing on an input image and outputs the character recognition result.

[0009] An output device according to another aspect of this disclosure includes a control unit that presents to the user the character recognition results obtained by character recognition processing in the image processing device.

[0010] An image processing method according to another aspect of the present disclosure is performed by one or more processors, which includes acquiring image data including a target to be detected; detecting rectangle candidates for the target to be detected from the image data and calculating the confidence level of the detected rectangle candidates; correcting the confidence level of the rectangle candidates to be corrected among the first and second rectangle candidates, using a method for calculating the degree of overlap, which is selected according to the relationship between the type of first target to be detected corresponding to the first rectangle candidate and the type of second target to be detected corresponding to the second rectangle candidate, if at least a portion of the detected first rectangle candidates and second rectangle candidates overlap with each other; and extracting the rectangle of the target to be detected based on the corrected confidence level.

[0011] An image processing program according to another aspect of this disclosure is a program that causes one or more processors to perform the following actions: acquire image data including a target to be detected; detect rectangle candidates for the target to be detected from the image data and calculate the confidence level of the detected rectangle candidates; correct the confidence level of the rectangle candidates to be corrected among the first and second rectangle candidates, using a method for calculating the degree of overlap, which is selected according to the relationship between the type of first target to be detected corresponding to the first rectangle candidate and the type of second target to be detected corresponding to the second rectangle candidate, when at least a portion of the detected first and second rectangle candidates overlap with each other; and extract the rectangle of the target to be detected based on the corrected confidence level. [Effects of the Invention]

[0012] According to this disclosure, it is possible to provide an image processing device, an image processing system, an output device, an image processing method, and an image processing program that can improve the accuracy of character recognition between characters that are close to each other. [Brief explanation of the drawing]

[0013] [Figure 1] Figure 1 is a functional block diagram showing the configuration of an image processing system according to an embodiment of this disclosure. [Figure 2] Figure 2 shows an example of an input image (receipt) according to the embodiment of this disclosure. [Figure 3] FIG. 3 is a diagram showing an example of a single-character rectangle detected in an input image according to an embodiment of the present disclosure. [Figure 4] FIG. 4 is a diagram showing an example of a single-character rectangle extracted in a conventional character recognition process. [Figure 5] FIG. 5 is a diagram showing an example of a single-character rectangle extracted in a conventional character recognition process. [Figure 6] FIG. 6 is a diagram showing an example of a rectangle candidate detected in a character recognition process according to an embodiment of the present disclosure. [Figure 7] FIG. 7 is a diagram showing an example of a rectangle candidate detected in a character recognition process according to an embodiment of the present disclosure. [Figure 8] FIG. 8 is a graph showing the relationship between the degree of overlap (index) and the correction coefficient used in the character recognition process according to an embodiment of the present disclosure. [Figure 9] FIG. 9 is a diagram showing an example of a method for calculating the first index and the second index according to an embodiment of the present disclosure. [Figure 10] FIG. 10 is a diagram showing a specific example of a correction process using the first index (IoU) according to an embodiment of the present disclosure. [Figure 11] FIG. 11 is a diagram showing a specific example of a correction process using the first index (IoU) according to an embodiment of the present disclosure. [Figure 12] FIG. 12 is a diagram showing a specific example of a correction process using the second index (IoS) according to an embodiment of the present disclosure. [Figure 13] FIG. 13 is a diagram showing a specific example of a correction process using the second index (IoS) according to an embodiment of the present disclosure. [Figure 14] FIG. 14 is a flowchart showing an example of a procedure of a character recognition process executed in an image processing apparatus according to an embodiment of the present disclosure. [Embodiments of the Invention]

[0014] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. Note that the following embodiments are an example of embodying the present disclosure and do not have the character of limiting the technical scope of the present disclosure.

[0015] FIG. 1 is a block diagram showing the configuration of an image processing system 10 according to an embodiment of the present disclosure. The image processing system 10 includes an image processing apparatus 1 and a learning apparatus 2. The image processing apparatus 1 is an information processing apparatus that recognizes a character string included in an input image (image data) and executes character recognition processing (OCR processing) on the recognized character string to output a character recognition result. The learning apparatus 2 is an information processing apparatus that performs machine learning using input data (learning data) input from the image processing apparatus 1 and generates a learned model for performing character recognition on an input image.

[0016] As shown in FIG. 1, the image processing apparatus 1 includes a control unit 11, a storage unit 12, an operation display unit 13, a communication unit 14, and the like. The image processing apparatus 1 may be one or more cloud servers or one or more physical servers.

[0017] The communication unit 14 is a communication interface for connecting the image processing apparatus 1 to the network N1 by wire or wirelessly and performing data communication according to a predetermined communication protocol with an external device (for example, the learning apparatus 2) via the network N1. The network N1 is composed of, for example, the Internet, a LAN, or the like.

[0018] The operation display unit 13 is a user interface including a display unit such as a liquid crystal display or an organic EL display for displaying various information, and an operation unit such as a mouse, a keyboard, or a touch panel for receiving operations.

[0019] The storage unit 12 is a non-volatile storage unit such as an HDD (Hard Disk Drive), SSD (Solid State Drive), or flash memory that stores various types of information. The storage unit 12 stores a control program, such as a character recognition program (an example of an image processing program in this disclosure), which causes the control unit 11 to execute the character recognition processing described later. For example, the character recognition program is non-temporarily recorded on a computer-readable recording medium such as a CD or DVD, read by a reader (not shown) such as a CD drive or DVD drive provided by the image processing device 1, and stored in the storage unit 12. The character recognition program may also be distributed from a cloud server and stored in the storage unit 12.

[0020] Furthermore, the memory unit 12 stores image data (such as scanned data) of documents and other items acquired from external devices.

[0021] Figure 2 shows a receipt as an example of a document (form). As shown in Figure 2, the receipt includes multiple items such as the date of issue, recipient, issuer's contact information, and amount. The receipt in Figure 2 also includes printed text T1 and handwritten text T2. For example, a user scans the receipt using a scanner, multifunction printer, etc., and uploads the image data (input image) to the image processing device 1. Alternatively, a user can take a picture of the receipt with a camera on an operating terminal (e.g., a smartphone) and upload the image data to the image processing device 1. When the control unit 11 acquires the image data of the receipt, it stores it in the storage unit 12. In another embodiment, the control unit 11 may acquire a document file of the receipt created on an external device and store the document file in the storage unit 12.

[0022] The control unit 11 includes control devices such as a CPU, ROM, and RAM. The CPU is a processor that performs various arithmetic operations. The ROM stores control programs such as a BIOS and OS in advance to allow the CPU to perform various operations. The RAM stores various information and is used as a temporary storage memory (work area) for the various operations performed by the CPU. The control unit 11 controls the image processing device 1 by executing various control programs stored in advance in the ROM or storage unit 12 using the CPU.

[0023] However, with conventional technology, a problem arises where character recognition accuracy decreases, for example, when two characters are close together and their character area rectangles overlap, one of the characters may be removed, or unnecessary character area rectangles other than the two character area rectangles may be extracted.

[0024] For example, Figure 3 shows a receipt image with a single character rectangle K1 for typed character T1 and a single character rectangle K2 for handwritten character T2. In the handwritten date T2 "2014 / 2 / 18" shown in Figure 4, if the distance between the character " / " and "1" is small, "1" may not be extracted as a single character rectangle and may be removed. For example, if we use NMS (Non-Maximum Suppression), a known object detection technique, and process it to keep the rectangle candidate with the highest score (confidence) among the detected characters " / " and "1", the rectangle candidate for " / " with the higher score will be extracted as a single character rectangle, and the rectangle candidate for "1" with the lower score will be removed. Alternatively, Soft-NMS can be used to extract the rectangle candidate with the lower score as a single character rectangle. However, while Soft-NMS makes it possible to extract the low-scoring "1" as a single character rectangle, a problem arises where the character "," is extracted not only as a single character rectangle for handwritten characters but also as a single character rectangle for typed characters, as shown in Figure 5.

[0025] Thus, with conventional technology, it is difficult to simultaneously prevent the removal of adjacent (or overlapping) single-character rectangles and prevent the extraction of unnecessary single-character rectangles, resulting in a problem of reduced character recognition accuracy. In contrast, the image processing device 1 according to this disclosure has a configuration that can improve the character recognition accuracy of characters that are close to each other, as shown below.

[0026] Specifically, as shown in Figure 1, the control unit 11 includes various processing units such as an acquisition processing unit 111, a string extraction processing unit 112, a rectangle candidate detection processing unit 113, a single character extraction processing unit 114, a recognition processing unit 115, and an output processing unit 116. The control unit 11 functions as one of these various processing units by executing various processes according to the character recognition program. Some or all of the processing units included in the control unit 11 may be composed of electronic circuits. The character recognition program may be a program that causes multiple processors to function as the various processing units.

[0027] The acquisition processing unit 111 acquires an input image containing the characters to be detected. Specifically, the acquisition processing unit 111 acquires an image (character image data) that is subject to character recognition. For example, the acquisition processing unit 111 acquires character image data from a document image, such as a receipt shown in Figure 2, which contains handwritten characters, printed characters, etc.

[0028] The string extraction processing unit 112 extracts string rectangles corresponding to strings composed of multiple characters from the input image. Specifically, the string extraction processing unit 112 performs string recognition processing by performing document part recognition processing on the input image acquired by the acquisition processing unit 111 to recognize strings composed of multiple characters. The string extraction processing unit 112 also sets string rectangles corresponding to the recognized strings. That is, the string extraction processing unit 112 recognizes blocks of multiple characters as string rectangles. For example, in the input image shown in Figure 2, the string extraction processing unit 112 extracts string rectangles corresponding to the handwritten strings ("2014 / 2 / 18", "￥121,000").

[0029] The rectangle candidate detection processing unit 113 detects multiple rectangle candidates corresponding to each of the multiple characters from the input image. In this embodiment, the rectangle candidates represent rectangles that are candidates for single-character rectangles. The rectangle candidate detection processing unit 113 also calculates a score (confidence level, likelihood) for the detected rectangle candidates. Specifically, the rectangle candidate detection processing unit 113 performs single-character recognition processing to recognize characters one by one in the input image acquired by the acquisition processing unit 111. The rectangle candidate detection processing unit 113 also detects rectangle candidates corresponding to the recognized characters. That is, the rectangle candidate detection processing unit 113 recognizes the smallest unit character as a rectangle candidate.

[0030] For example, the rectangle candidate detection processing unit 113 detects multiple rectangle candidates corresponding to each character of the printed character T1 and multiple rectangle candidates corresponding to each character of the handwritten character T2. Figure 6 shows an example in which nine rectangle candidates K0 were detected in the string rectangle of the handwritten character T2 issue date "2014 / 2 / 18". Figure 7 shows an example in which eight rectangle candidates K0 were detected in the string rectangle of the handwritten character T2 amount "121,000". Note that in the case of "," shown in Figure 7, both the handwritten character rectangle candidate and the printed character rectangle candidate are detected. Furthermore, as shown in Figures 6 and 7, the rectangle candidate detection processing unit 113 calculates a score for each detected rectangle candidate.

[0031] The single-character extraction processing unit 114 extracts multiple single-character rectangles corresponding to each character from the rectangle candidate detection processing unit 113. Specifically, the single-character extraction processing unit 114 extracts each rectangle candidate as a single-character rectangle when the distance between multiple rectangle candidates is large and they do not overlap. For example, in the example shown in Figure 6, the single-character extraction processing unit 114 extracts "2", "0", "1", "4", " / ", "2", and "8" as single-character rectangles. Also, for example, in the example shown in Figure 7, the single-character extraction processing unit 114 extracts "1", "2", "1", "0", "0", and "0" as single-character rectangles.

[0032] In response to this, the single-character extraction processing unit 114 performs a correction process to correct the score (confidence level) of each rectangle candidate when multiple rectangle candidates are close together and overlap. For example, in the example shown in Figure 6, the single-character extraction processing unit 114 performs a correction process because the rectangle candidate for the character " / " and the rectangle candidate for the character "1" overlap. Also, in the example shown in Figure 7, the single-character extraction processing unit 114 performs a correction process because the handwritten rectangle candidate and the printed rectangle candidate detected for the character "," overlap. The single-character extraction processing unit 114 is an example of the correction processing unit and extraction processing unit of this disclosure.

[0033] A specific example of the correction process is described below. When at least a portion of the first and second rectangle candidates detected by the rectangle candidate detection process 113 overlap, the single-character extraction process 114 performs a correction process to correct the score (confidence level) of the rectangle candidate to be corrected among the first and second rectangle candidates, using a method for calculating the degree of overlap between the first and second characters, which is selected according to the relationship between the type of the first character corresponding to the first rectangle candidate and the type of the second character corresponding to the second rectangle candidate. The single-character extraction process 114 then extracts a single-character rectangle based on the corrected score. The rectangle candidate with the lower score (confidence level) among the first and second rectangle candidates is the one to be corrected. In other words, the single-character extraction process 114 corrects the score of the rectangle candidate with the lower score among the first and second rectangle candidates.

[0034] For example, the single-character extraction processing unit 114 uses Soft-NMS and corrects the score of the rectangle candidate to be corrected by using a correction coefficient based on different indicators (degree of overlap) depending on the type (attribute) of the rectangle candidate. Figure 8 shows a graph showing the relationship between indicators (first indicator: IoU, second indicator: IoS) that represent the degree of overlap between multiple rectangle candidates and the correction coefficient. Figure 9 shows a specific example of the calculation method for the first and second indicators. As shown in Figure 8, a graph having the characteristic that the correction coefficient becomes smaller as the degree of overlap (ratio) increases is an example of the correction data of this disclosure. The calculation method for the first indicator is an example of the first calculation method of this disclosure, and the calculation method for the second indicator is an example of the second calculation method of this disclosure. In the graph shown in Figure 8, the correction coefficient is set to 1.0 in the interval where the degree of overlap is low, from indicator 0 to Nt (threshold), and the correction process is omitted in the interval where the degree of overlap is less than the threshold.

[0035] In Figure 9, for example, if the area of ​​the first candidate rectangle is Sa, the area of ​​the second candidate rectangle is Sb, and the area of ​​the overlapping portion between the first and second candidate rectangles is Sc, then the first index (IoU; Intersection over Union) is expressed as Sc / (Sa+Sb-Sc), and the second index (IoS; Intersection over smaller) is expressed as Sc / Sb. Here, the candidate rectangle to be corrected is the second candidate rectangle.

[0036] Taking the issue date "2014 / 2 / 18" of the handwritten character T2 as an example, since both the first rectangular candidate " / " and the second rectangular candidate "1" are handwritten characters (same type), the single-character extraction processing unit 114 calculates a correction coefficient using the first index (IoU), as shown in Figure 10. Specifically, the single-character extraction processing unit 114 calculates the first index (IoU = Sc / (Sa + Sb - Sc)) based on the area Sa of the first rectangular candidate " / ", the area Sb of the second rectangular candidate "1", and the overlapping area Sc of the two (see Figure 9). Here, we assume that "a1" is calculated as the first index (IoU). Then, the single-character extraction processing unit 114 calculates a correction coefficient corresponding to IoU = a1 using the graph. Here, we assume that "b1" is calculated as the correction coefficient. Here, if the score of the second rectangle candidate "1" to be corrected is "90", the single-character extraction processing unit 114 corrects the score by multiplying it by a correction coefficient, as shown in Figure 11. As a result, the corrected score of the second rectangle candidate "1" becomes "90 × b1". If the corrected score is above the threshold, the single-character extraction processing unit 114 extracts the second rectangle candidate "1" to be corrected as a single-character rectangle. This makes it possible to extract the character "1" (see Figure 4), which was removed in the conventional technology, as a single-character rectangle.

[0037] Thus, when the characters of the first rectangle candidate and the second rectangle candidate are of the same type, the single-character extraction processing unit 114 sets the correction coefficient to a large value by using the first index (IoU), which has a small impact from the degree of overlap. This makes it difficult to lower the corrected score, making it difficult for the rectangle candidate to be corrected to be deleted, and enabling it to be appropriately extracted as a single-character rectangle. Note that "same type" includes cases where the characters of multiple rectangle candidates are all handwritten characters, and cases where the characters of multiple rectangle candidates are all printed characters.

[0038] In contrast, for the handwritten character T2 amount "121,000", the handwritten character of the first rectangular candidate corresponding to the character "," and the typeset character of the second rectangular candidate are of different types. Therefore, as shown in Figure 12, the single-character extraction processing unit 114 calculates a correction coefficient using the second index (IoS). Specifically, the single-character extraction processing unit 114 calculates the second index (IoS = Sc / Sb) based on the area Sb and overlapping area Sc of the second rectangular candidate of the typeset character (see Figure 9). Here, we assume that "a2" is calculated as the second index (IoS). Then, the single-character extraction processing unit 114 calculates a correction coefficient corresponding to IoS = a2 using the graph. Here, we assume that "b2" is calculated as the correction coefficient. If the score of the character "," of the second rectangular candidate to be corrected is "70", then, as shown in Figure 13, the single-character extraction processing unit 114 corrects the score by multiplying it by the correction coefficient. As a result, the corrected score for the second rectangle candidate "," becomes "70 × b2". The single-character extraction processing unit 114 deletes the second rectangle candidate to be corrected if the corrected score is below the threshold. This makes it possible to delete rectangle candidates of typefaces (see Figure 5) that were unnecessarily extracted in the conventional technology.

[0039] Thus, when the characters of the first rectangle candidate and the second rectangle candidate are of different types, the correction coefficient is set to a small value by using the second index (IoS), which has a greater influence on the degree of overlap. This makes it easier to lower the corrected score, making it easier to eliminate the rectangle candidates to be corrected and preventing the extraction of unnecessary single-character rectangles. "Different types" include combinations of handwritten and printed characters in multiple rectangle candidates.

[0040] As described above, the single-character extraction processing unit 114 selects a calculation method depending on whether the type of the first character and the type of the second character are the same. Specifically, the single-character extraction processing unit 114 selects a first calculation method using a first index (IoU) (see Figures 10 and 11) when both the first and second characters are handwritten or typeset, and selects a second calculation method using a second index (IoS) (see Figures 12 and 13) when one of the first and second characters is handwritten and the other is typeset. Furthermore, the first calculation method is a method for calculating the ratio of the area of ​​the overlapping portion to the total area of ​​the first rectangle candidate and the second rectangle candidate (degree of overlap), and the second calculation method is a method for calculating the ratio of the area of ​​the overlapping portion to the area of ​​the rectangle candidate to be corrected.

[0041] In other words, the single-character extraction processing unit 114 calculates a correction coefficient corresponding to the ratio calculated by the first calculation method when both the first and second characters are handwritten or typeset, and multiplies the score of the rectangle candidate to be corrected by this correction coefficient to correct the score. When one of the first and second characters is handwritten and the other is typeset, the single-character extraction processing unit 114 calculates a correction coefficient corresponding to the ratio calculated by the second calculation method, and multiplies the score of the rectangle candidate to be corrected by this correction coefficient to correct the score. Furthermore, the single-character extraction processing unit 114 calculates the correction coefficient by referring to correction data (the graph shown in Figure 8) which has the characteristic that the correction coefficient decreases as the ratio increases.

[0042] Furthermore, the single-character extraction processing unit 114 extracts a candidate rectangle to be corrected as a single-character rectangle if the corrected score is equal to or greater than a threshold, and does not extract a candidate rectangle to be corrected as a single-character rectangle if the corrected score is less than the threshold.

[0043] As a result, as shown in Figure 9, the second index (IoS) of the second calculation method is less affected by the area of ​​the first rectangle candidate and more affected by the overlapping area of ​​the rectangle candidates to be corrected. Therefore, the greater the degree of overlap, the lower the score becomes, making it easier to exclude rectangle candidates. In contrast, the first index (IoU) of the first calculation method is more affected by the area of ​​the first rectangle candidate, so the effect of the degree of overlap is smaller, making it easier to get a higher score, and thus making it harder to exclude rectangle candidates.

[0044] The recognition processing unit 115 performs character recognition processing (OCR processing) of the string. Specifically, the recognition processing unit 115 performs OCR processing based on the string rectangle extracted by the string extraction processing unit 112 and the multiple single-character rectangles extracted by the single-character extraction processing unit 114.

[0045] For example, the recognition processing unit 115 performs preprocessing to improve the accuracy of OCR (processing such as background removal, inversion, line removal, seal removal, and italic correction), and then performs the existing OCR processing.

[0046] The output processing unit 116 outputs the OCR result (character recognition result). For example, the output processing unit 116 outputs the OCR result to the requesting party that sent the character recognition request for the input image.

[0047] Furthermore, the output processing unit 116 outputs training data to the learning device 2 (see Figure 1). The control unit 11 generates single-character rectangles containing the corrected scores by the single-character extraction processing unit 114 as training data to be used for machine learning. The output processing unit 116 outputs the training data (teacher data) containing the single-character rectangles that have undergone correction processing by the single-character extraction processing unit 114 to the learning device 2.

[0048] The learning device 2 uses the training data generated by the image processing device 1 to perform machine learning and generate a trained model.

[0049] Machine learning includes algorithms such as supervised learning (using supervised data), unsupervised learning (using unsupervised data), and reinforcement learning. Furthermore, a technique called "deep learning," which learns to extract features themselves, is used to implement these methods. In this embodiment, the learning device 2 has a learning model based on the various algorithms described above. The learning device 2 can perform machine learning using supervised and unsupervised data as input data and generate a trained model that performs character recognition processing.

[0050] The pre-trained model can be applied to the image processing device 1. For example, as shown in Figure 1, when an input image to be recognized is input to the image processing device 1, the image processing device 1 uses the pre-trained model to perform OCR processing on the input image and outputs the OCR result. The image processing device 1 is an example of an output device in this disclosure.

[0051] Furthermore, the trained model may be downloaded to the image processing device 1 for use, or it may be stored on a server (cloud server) and accessed from the user terminal via the internet or the like for use. For example, when an arbitrary input image is input to the user terminal, the trained model outputs the optimal character recognition result. That is, the user terminal may use the trained model generated by the learning device 2 to perform OCR processing on the input image and output the OCR result. The user terminal may also include a control unit that presents the user with the OCR result of performing OCR processing on a string using the corrected single-character rectangle corrected by the image processing device 1. The user terminal is an example of an output device in this disclosure.

[0052] [Character recognition processing] Figure 14 is a flowchart showing an example of the character recognition process performed in the image processing device 1.

[0053] This disclosure can be understood as a character recognition method (image processing method of this disclosure) that performs one or more steps included in the character recognition process. Furthermore, one or more steps included in the character recognition process described herein may be omitted as appropriate. In addition, the execution order of each step in the character recognition process may differ to the extent that similar effects are produced. Furthermore, although this description uses the case in which the control unit 11 of the image processing device 1 performs each step in the character recognition process as an example, in other embodiments, one or more processors may distribute and execute each step in the character recognition process. In addition, when the control unit 11 acquires character image data from an external device, it is possible to execute the character recognition process in parallel for each piece of character image data.

[0054] <Step S1> In step S1, the control unit 11 determines whether or not it has acquired character image data. Specifically, the control unit 11 acquires character image data of a document (for example, a receipt in Figure 2) from an external device. If the control unit 11 has acquired character image data (S1: Yes), it proceeds to step S2. The control unit 11 waits until it has acquired character image data (S1: No).

[0055] <Step S2> In step S2, the control unit 11 detects multiple single-character rectangle candidates corresponding to each of the multiple characters from the character image data. Specifically, the control unit 11 detects individual characters in the input image and designates the rectangle of each detected character as a rectangle candidate. For example, as shown in Figure 6, the control unit 11 detects nine rectangle candidates K0 in the string rectangle of the handwritten character T2 issue date "2014 / 2 / 18". Also, as shown in Figure 7, for example, the control unit 11 detects eight rectangle candidates K0 in the string rectangle of the handwritten character T2 amount "121,000".

[0056] <Step S3> In step S3, the control unit 11 determines whether the plurality of rectangle candidates extracted in step S2 include any duplicate rectangle candidates. In another embodiment, the control unit 11 may determine whether there are any rectangle candidates among the plurality of rectangle candidates whose distance from each other is less than a predetermined distance.

[0057] The control unit 11 proceeds to step S4 if there are duplicate rectangle candidates among the multiple rectangle candidates (S3: Yes). On the other hand, the control unit 11 proceeds to step S7 if there are no duplicate rectangle candidates among the multiple rectangle candidates (S3: No).

[0058] <Step S4> In step S4, the control unit 11 determines whether the types of the overlapping rectangle candidates are different. Specifically, the control unit 11 determines whether the character types of the overlapping rectangle candidates are a combination of handwritten characters and printed characters. If the character types of the overlapping rectangle candidates are a combination of handwritten characters and printed characters (S4:Yes), the control unit 11 proceeds to step S51. On the other hand, if the character types of the overlapping rectangle candidates are a combination of handwritten characters or a combination of printed characters (S4:No), the control unit 11 proceeds to step S52.

[0059] <Step S51> In step S51, the control unit 11 calculates a correction coefficient to correct the score of the rectangle candidate to be corrected using the second index (IoS) (see Figure 9). Specifically, the control unit 11 calculates the correction coefficient corresponding to the second index (IoS) using the graph shown in Figure 8. For example, in the example shown in Figure 7, the character "," in the second rectangle candidate is a combination of a handwritten rectangle candidate and a printed rectangle candidate, so the control unit 11 calculates the second index (IoS) (IoS=a2) as shown in Figure 12, and calculates the correction coefficient "b2" corresponding to the second index "a2". After step S51, the control unit 11 moves the process to step S6.

[0060] <Step S52> In step S52, the control unit 11 calculates a correction coefficient to correct the score of the rectangle candidate to be corrected using the first index (IoU) (see Figure 9). Specifically, the control unit 11 calculates the correction coefficient corresponding to the first index (IoU) using the graph shown in Figure 8. For example, in the example shown in Figure 6, the characters " / " and "1" of the second rectangle candidate are a combination of handwritten character rectangle candidates, so the control unit 11 calculates the first index (IoU) (IoU=a1) as shown in Figure 10, and calculates the correction coefficient "b1" corresponding to the first index "a1". After step S52, the control unit 11 moves the process to step S6.

[0061] <Step S6> In step S6, the control unit 11 corrects the score of the rectangle candidate to be corrected. Specifically, if the types of overlapping rectangle candidates are different (S4: Yes), the control unit 11 corrects the score of the second rectangle candidate to be corrected using the correction coefficient calculated by the second index (IoS) (step S51). For example, as shown in Figures 12 and 13, if the score of the character "," in the second rectangle candidate to be corrected is "70", the control unit 11 multiplies the score by the correction coefficient to correct the score to "70 × b2".

[0062] Furthermore, if the types of overlapping rectangle candidates are the same (S4: No), the control unit 11 corrects the score of the second rectangle candidate to be corrected using the correction coefficient (step S52) calculated by the first index (IoU). For example, as shown in Figures 10 and 11, if the score of the character "1" of the second rectangle candidate to be corrected is "90", the control unit 11 multiplies the score by the correction coefficient to correct the score to "90 × b1".

[0063] <Step S7> In step S7, the control unit 11 extracts single-character rectangles. Specifically, the control unit 11 extracts a rectangle candidate as a single-character rectangle if its score is above a threshold. If multiple rectangle candidates overlap (S3: Yes), the control unit 11 extracts a rectangle candidate as a single-character rectangle if its score after correction using the correction coefficient (steps S51, S52) is above a threshold. Although not shown in Figure 14, the control unit 11 also extracts character rectangles based on the input image.

[0064] <Step S8> In step S8, the control unit 11 performs OCR processing. Specifically, the control unit 11 performs existing preprocessing such as background removal, inversion, line removal, stamp removal, and italic correction. The control unit 11 also performs existing OCR processing on the string rectangle and single-character rectangle after the preprocessing. After performing the OCR processing, the control unit 11 outputs the OCR results.

[0065] As described above, the control unit 11 executes the character recognition process. Furthermore, the control unit 11 repeatedly executes the character recognition process each time it acquires character image data (input image) to be recognized.

[0066] As described above, the image processing apparatus 1 according to this embodiment performs the following: acquires image data including the character to be detected; detects candidate rectangles of the character from the image data and calculates the confidence level of the detected candidate rectangles; corrects the confidence level of the candidate rectangle to be corrected among the first and second candidates if at least a portion of the detected first candidate rectangle and second candidate rectangles overlap with each other, using a method for calculating the degree of overlap, which is selected according to the relationship between the type of the first character corresponding to the first candidate rectangle and the type of the second character corresponding to the second candidate rectangle; and extracts the character rectangle based on the corrected confidence level.

[0067] According to the above configuration, for example, if the first and second characters are of the same type (e.g., both handwritten characters, or both typeset characters), using the first calculation method that utilizes the first index (IoU), which has a small impact from the degree of overlap, makes it difficult to reduce the corrected confidence score, thus making it difficult to eliminate candidate rectangles to be corrected. This makes it possible to appropriately extract single-character rectangles. On the other hand, if the first and second characters are of different types (e.g., a combination of handwritten characters and typeset characters), using the second calculation method that utilizes the second index (IoS), which has a large impact from the degree of overlap, makes it easier to reduce the corrected confidence score, thus making it easier to eliminate candidate rectangles to be corrected. This makes it possible to prevent the extraction of unnecessary single-character rectangles.

[0068] Therefore, it is possible to prevent the removal of adjacent character rectangles and to prevent the extraction of unnecessary character rectangles, thereby improving the character recognition accuracy of characters that are close to each other.

[0069] Furthermore, the control unit 11 of the image processing device 1 may display on the operation display unit 13 the calculation method used for the correction process from among the first calculation method and the second calculation method so as to be identifiable. In addition, the control unit 11 may display rectangular candidates for each character on the operation display unit 13, and may also display in an identifiable manner the types of the first character and the second character whose rectangular candidates overlap with each other. For example, the control unit 11 may display the frame images of the first rectangular candidate and the second rectangular candidate with different colors.

[0070] In the embodiments described above, characters are used as examples of the detection target, but the detection target of this disclosure is not limited to characters, but may also be objects, people, etc. Furthermore, the detection target of this disclosure may be a combination of characters and non-characters.

[0071] Furthermore, in the image processing system 10, the image processing device 1 and the learning device 2 may be configured as a single integrated device. Also, each processing unit of the image processing device 1 (acquisition processing unit 111, string extraction processing unit 112, rectangle candidate detection processing unit 113, single character extraction processing unit 114, recognition processing unit 115, output processing unit 116) may be distributed across multiple devices.

[0072] In the image processing system 10 according to this disclosure, the control unit 11 of the image processing device 1 controls the entire image processing device 1. The control unit 11 realizes various functions by reading and executing various programs stored in the storage unit 12 (for example, storage or ROM). The control unit 11 may be realized by one or more control devices / arithmetic units (CPU (Central Processing Unit), SoC (System on a Chip)). The control unit 11 may also be composed of one or more control circuits (electronic circuits).

[0073] [Disclosure Note] The following is an overview of the disclosures extracted from the above-described embodiments. Note that each configuration and processing function described in the following notes can be selected and combined as desired.

[0074] <Note 1> An acquisition processing unit that acquires image data including the target to be detected, A detection processing unit that detects candidate rectangles to be detected from the image data and calculates the confidence level of the detected candidate rectangles, When at least a portion of the first rectangle candidate and the second rectangle candidate detected by the detection processing unit overlap with each other, a correction processing unit corrects the reliability of the rectangle candidate to be corrected among the first rectangle candidate and the second rectangle candidate, using a method for calculating the degree of overlap selected according to the relationship between the type of first detection target corresponding to the first rectangle candidate and the type of second detection target corresponding to the second rectangle candidate. An extraction processing unit extracts the rectangle to be detected based on the confidence level corrected by the correction processing unit, An image processing device equipped with the following features.

[0075] <Note 2> The object to be detected is a character, The correction processing unit selects the calculation method according to whether the type of the first character and the type of the second character are the same. The image processing device described in Appendix 1.

[0076] <Note 3> The aforementioned categories include handwritten characters and printed characters. The correction processing unit selects a first calculation method when both the first character and the second character are either handwritten characters or typed characters, and selects a second calculation method when one of the first character and the second character is a handwritten character and the other is a typed character. The image processing apparatus described in Appendix 2.

[0077] <Note 4> The first calculation method is a method for calculating the ratio of the area of ​​the overlapping portion to the total area of ​​the first candidate rectangle and the second candidate rectangle, The second calculation method is a method for calculating the ratio of the area of ​​the overlapping portion to the area of ​​the candidate rectangle to be corrected. The image processing apparatus described in Appendix 3.

[0078] <Note 5> The correction processing unit, When both the first character and the second character are handwritten characters or typeset characters, a correction coefficient corresponding to the ratio calculated by the first calculation method is calculated, and the confidence level of the rectangle candidate to be corrected is corrected by multiplying the said correction coefficient by the said confidence level. When one of the first and second characters is a handwritten character and the other is a printed character, a correction coefficient corresponding to the ratio calculated by the second calculation method is calculated, and the confidence level of the rectangle candidate to be corrected is corrected by multiplying the said correction coefficient by the said confidence level. The image processing apparatus described in Appendix 4.

[0079] <Note 6> The correction processing unit calculates the correction coefficient by referring to correction data having the characteristic that the correction coefficient decreases as the ratio increases. The image processing apparatus described in Appendix 5.

[0080] <Note 7> The extraction processing unit extracts the rectangle candidate to be corrected as the rectangle to be detected if the confidence level corrected by the correction processing unit is equal to or greater than a threshold, and does not extract the rectangle candidate to be corrected as the rectangle to be detected if the confidence level corrected by the correction processing unit is less than the threshold. An image processing device as described in any of the appendices 1 to 6.

[0081] <Note 8> The rectangle to be detected, including the confidence level corrected by the correction processing unit, is generated as training data to be used in machine learning. An image processing device as described in any of the appendices 1 to 7.

[0082] <Note 9> An image processing device as described in any of Appendix 1 to 8, A learning device that generates a trained model by performing machine learning using the training data generated by the image processing device, An image processing system equipped with the following features.

[0083] <Note 10> An output device that performs character recognition processing on an input image using the trained model generated by the learning device described in Appendix 9, and outputs the character recognition result.

[0084] <Note 11> An output device comprising a control unit that presents to the user the character recognition results obtained when character recognition processing is performed in an image processing device described in any of Appendix 1 to 8.

[0085] <Note 12> Acquiring image data that includes the target to be detected, The process involves detecting candidate rectangles to be detected from the aforementioned image data, and calculating the confidence level of the detected candidate rectangles. When at least a portion of the detected first rectangular candidate and second rectangular candidate overlap with each other, the reliability of the rectangle candidate to be corrected among the first and second rectangular candidates is corrected using a method for calculating the degree of overlap, which is selected according to the relationship between the type of first detection target corresponding to the first rectangular candidate and the type of second detection target corresponding to the second rectangular candidate. Based on the corrected confidence level, the rectangle to be detected is extracted, An image processing method performed by one or more processors.

[0086] <Note 13> Acquiring image data that includes the target to be detected, The process involves detecting candidate rectangles to be detected from the aforementioned image data, and calculating the confidence level of the detected candidate rectangles. When at least a portion of the detected first rectangular candidate and second rectangular candidate overlap with each other, the reliability of the rectangle candidate to be corrected among the first and second rectangular candidates is corrected using a method for calculating the degree of overlap, which is selected according to the relationship between the type of first detection target corresponding to the first rectangular candidate and the type of second detection target corresponding to the second rectangular candidate. Based on the corrected confidence level, the rectangle to be detected is extracted, An image processing program for causing one or more processors to execute, or a non-temporary computer-readable recording medium on which the detection program is recorded. [Explanation of Symbols]

[0087] 1: Image processing device 2: Learning device 10: Image Processing System 11: Control Unit 111: Acquisition Processing Unit 112: String extraction processing unit 113: Rectangle Candidate Detection Processing Unit 114: Single character extraction processing unit 115: Recognition Processing Unit 116: Output Processing Unit

Claims

1. An acquisition processing unit that acquires image data including the target to be detected, A detection processing unit that detects candidate rectangles to be detected from the image data and calculates the confidence level of the detected candidate rectangles, When at least a portion of the first rectangle candidate and the second rectangle candidate detected by the detection processing unit overlap with each other, a correction processing unit corrects the reliability of the rectangle candidate to be corrected among the first rectangle candidate and the second rectangle candidate, using a method for calculating the degree of overlap selected according to the relationship between the type of first detection target corresponding to the first rectangle candidate and the type of second detection target corresponding to the second rectangle candidate. An extraction processing unit extracts the rectangle to be detected based on the confidence level corrected by the correction processing unit, An image processing device equipped with the following features.

2. The object to be detected is a character, The correction processing unit selects the calculation method according to whether the type of the first character and the type of the second character are the same. The image processing apparatus according to claim 1.

3. The aforementioned categories include handwritten characters and printed characters. The correction processing unit selects a first calculation method when both the first character and the second character are either handwritten characters or typed characters, and selects a second calculation method when one of the first character and the second character is a handwritten character and the other is a typed character. The image processing apparatus according to claim 2.

4. The first calculation method is a method for calculating the ratio of the area of ​​the overlapping portion to the total area of ​​the first candidate rectangle and the second candidate rectangle, The second calculation method is a method for calculating the ratio of the area of ​​the overlapping portion to the area of ​​the candidate rectangle to be corrected. The image processing apparatus according to claim 3.

5. The correction processing unit, When both the first character and the second character are handwritten characters or printed characters, a correction coefficient corresponding to the ratio calculated by the first calculation method is calculated, and the confidence level of the rectangle candidate to be corrected is corrected by multiplying the said correction coefficient by the said confidence level. When one of the first and second characters is a handwritten character and the other is a printed character, a correction coefficient corresponding to the ratio calculated by the second calculation method is calculated, and the confidence level of the rectangle candidate to be corrected is corrected by multiplying the said correction coefficient by the said confidence level. The image processing apparatus according to claim 4.

6. The correction processing unit calculates the correction coefficient by referring to correction data having the characteristic that the correction coefficient decreases as the ratio increases. The image processing apparatus according to claim 5.

7. The extraction processing unit extracts the rectangle candidate to be corrected as the rectangle to be detected if the confidence level corrected by the correction processing unit is equal to or greater than a threshold, and does not extract the rectangle candidate to be corrected as the rectangle to be detected if the confidence level corrected by the correction processing unit is less than the threshold. The image processing apparatus according to claim 1.

8. The rectangle to be detected, including the confidence level corrected by the correction processing unit, is generated as training data to be used in machine learning. The image processing apparatus according to any one of claims 1 to 7.

9. The image processing apparatus according to claim 8, A learning device that generates a trained model by performing machine learning using the training data generated by the image processing device, An image processing system equipped with the following features.

10. An output device that performs character recognition processing on an input image using the trained model generated by the learning device described in claim 9, and outputs the character recognition result.

11. An output device comprising a control unit that presents to the user the character recognition result obtained by performing character recognition processing in an image processing apparatus according to any one of claims 1 to 7.

12. Acquiring image data that includes the target to be detected, The process involves detecting candidate rectangles to be detected from the aforementioned image data, and calculating the confidence level of the detected candidate rectangles. When at least a portion of the detected first rectangle candidate and second rectangle candidate overlap with each other, the reliability of the rectangle candidate to be corrected among the first and second rectangle candidates is corrected using a method for calculating the degree of overlap, which is selected according to the relationship between the type of first detection target corresponding to the first rectangle candidate and the type of second detection target corresponding to the second rectangle candidate. Based on the corrected confidence level, the rectangle to be detected is extracted, An image processing method performed by one or more processors.

13. Acquiring image data that includes the target to be detected, The process involves detecting candidate rectangles to be detected from the aforementioned image data, and calculating the confidence level of the detected candidate rectangles. When at least a portion of the detected first rectangle candidate and second rectangle candidate overlap with each other, the reliability of the rectangle candidate to be corrected among the first and second rectangle candidates is corrected using a method for calculating the degree of overlap, which is selected according to the relationship between the type of first detection target corresponding to the first rectangle candidate and the type of second detection target corresponding to the second rectangle candidate. Based on the corrected confidence level, the rectangle to be detected is extracted, An image processing program that is executed by one or more processors.

Images