Information processing system, information processing method, and program

Abstract

To provide an information processing system and an information processing apparatus configured to allow a character recognition result to be efficiently checked with high accuracy.SOLUTION: An information processing system includes an image acquisition unit 12, a character recognition unit 13, a cropped image generation unit 15, an image classification unit 16, and a display control unit 17. The image acquisition unit 12 acquires a target object image of a target object with characters shown thereon. The character recognition unit 13 recognizes the characters based on the acquired target object image. The cropped image generation unit 15 generates cropped images cropped from the target object image, for each of the recognized characters. The image classification unit 16 classifies the generated cropped images, based on a result recognized by the character recognition unit 13, for each group of images having the same recognition result. The display control unit 17 displays, for each recognition result, the cropped images with the same recognition result included in the group of the images having the same recognition result.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an information processing system, an information processing method, and a program that can be applied to checking character recognition results, etc. [Background technology]

[0002] Patent Document 1 discloses a character recognition correction system that enables an operator to accurately and efficiently make corrections to the character recognition results obtained by an OCR (Optical Character Recognition / Reader) on image data such as handwritten forms.

[0003] In the character recognition correction system described in Patent Document 1, the character recognition results and image data (original image) are displayed on the operator's terminal so that they can be observed in comparison. If the operator finds an unsatisfactory part in the character recognition results, he or she can use the search function provided in the character recognition correction system to search for the correct information to replace that part. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2004-220181 Summary of the Invention [Problem to be solved by the invention]

[0005] When character recognition is performed using OCR as described in Patent Document 1, there is a demand for a technique that enables the character recognition results to be checked efficiently and with high accuracy.

[0006] In view of the above circumstances, an object of the present invention is to provide an information processing system, an information processing method, and a program that enable character recognition results to be checked efficiently and with high accuracy. [Means for solving the problem]

[0007] In order to achieve the above object, an information processing system according to an embodiment of the present invention includes an image acquisition unit, a character recognition unit, a cut-out image generation unit, an image classification unit, and a display control unit. The image acquisition unit acquires an object image of an object on which characters are displayed. The character recognition unit recognizes the characters based on the acquired object image. The image generation unit generates a cut-out image cut out from the object image for each of the recognized characters. The image classification unit classifies the generated cut-out images into groups of identical recognition result images that result in the same recognition result, based on the recognition result by the character recognition unit. The display control unit displays, for each recognition result, cut-out images that are included in the group of identical recognition result images and result in the same recognition result.

[0008] The display control unit may display the clipped images that result in the same recognition result so that they can be compared with each other.

[0009] The display control unit may display the cut-out images that result in the same recognition result in an aligned manner, or may display the cut-out images by switching them in order at predetermined time intervals.

[0010] The display control unit may arrange and display the cut-out images that result in the same recognition result in an order based on the feature amounts of the cut-out images, or may switch and display the cut-out images in an order based on the feature amounts of the cut-out images.

[0011] The display control unit may arrange and display the cut-out images that result in the same recognition result in order of similarity, or may switch and display the cut-out images in order of similarity.

[0012] The display control unit may arrange and display the cut-out images that result in the same recognition result in an order based on the reliability of the recognition result, or may switch and display the cut-out images in an order based on the reliability of the recognition result.

[0013] the information processing system further includes an input receiving unit that receives an input of a selection for the displayed extracted image, and a recognition result image generating unit that generates a recognition result image including a recognition result by the character recognition unit, When a selection input for the cut-out image is accepted, the display control unit may highlight and display at least one of the area of ​​the object image corresponding to the selected cut-out image and the area of ​​the recognition result image corresponding to the selected cut-out image.

[0014] The input receiving unit may receive input of corrections to characters included in the area displayed in a highlighted manner in the recognition result image. In this case, the information processing system may further include a storage unit that stores the input corrected characters.

[0015] The display control unit may sequentially switch and display the cut-out images that result in the same recognition result at a predetermined time interval. In this case, the input receiving unit may receive an input to stop switching the display while a predetermined cut-out image is being displayed as an input to select the displayed predetermined cut-out image.

[0016] The information processing system may further include a recognition result image generation unit that generates a recognition result image including a recognition result by the character recognition unit. In this case, the display control unit may display the object image and the recognition result image so that they can be compared.

[0017] The display control unit may display a GUI (Graphical User Interface) for correcting the recognition result obtained by the character recognition unit.

[0018] The characters may be numbers from 0 to 9.

[0019] An information processing method according to one aspect of the present invention is an information processing method executed by a computer system, and includes acquiring an object image of an object on which text is displayed. The characters are recognized based on the acquired object image. A cut-out image is generated from the object image for each of the recognized characters. The generated cut-out images are classified into groups of identical recognition result images that have the same recognition result based on the character recognition result. The cut-out images that are included in the group of identical recognition result images and result in the same recognition result are displayed for each recognition result.

[0020] A program according to an aspect of the present invention causes a computer system to execute the information processing method.

[0021] An information processing system according to another aspect of the present invention includes an image acquisition unit, a character recognition unit, a storage unit, an image generation unit, an image classification unit, and a display control unit. The image acquisition unit acquires an object image of an object on which characters are displayed. The character recognition unit recognizes the characters based on the acquired object image. The storage unit stores a character string of interest that is a target of attention. The image cutout unit generates cutout images cut out for each corresponding character string corresponding to the stored target character string of interest from the object image based on the recognition result by the character recognition unit. The image classification classifies the generated cut-out images into groups of identical character string images that have the same target character string. The display control unit displays the cut-out images included in the group of identical character string images for each of the same target character strings.

[0022] An information processing method according to another aspect of the present invention is an information processing method executed by a computer system, and includes acquiring an object image of an object on which text is displayed. The characters are recognized based on the acquired object image. A character string of interest that is the subject of attention is stored. Based on the character recognition result, an extracted image is generated, which is extracted from the object image for each corresponding character string that corresponds to the stored target character string of interest. The generated cut-out images are classified into groups of identical character string images that have the same target character string. The cut-out images included in the group of identical character string images are displayed for each of the same target character strings.

[0023] A program according to another aspect of the present invention causes a computer system to execute the information processing method. [Effects of the Invention]

[0024] As described above, according to the present invention, it is possible to check the character recognition results efficiently and with high accuracy. [Brief explanation of the drawings]

[0025] [Figure 1] 1 is a schematic diagram illustrating an example of the configuration of a character recognition system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a block diagram illustrating an example of a functional configuration of a server device. [Figure 3] 10 is a flowchart illustrating an example of a character recognition operation performed by the server device. [Figure 4] FIG. 10 is a schematic diagram showing an example of a photographed image of a time card. [Figure 5] FIG. 10 is a schematic diagram illustrating an example of recognition result information. [Figure 6] FIG. 10 is a schematic diagram illustrating an example of a recognition result image. [Figure 7] FIG. 2 is a schematic diagram illustrating an example of character information. [Figure 8] FIG. 10 is a schematic diagram for explaining classification of extracted images. [Figure 9] FIG. 10 is a schematic diagram showing an example of highlighting an area corresponding to a selected clipped image. [Figure 10] FIG. 10 is a schematic diagram illustrating an example of correction of a recognition result. [Figure 11] FIG. 2 is a block diagram showing a more detailed configuration example of the character recognition system. [Figure 12] 10 is a flowchart showing an example of an operation from receiving a photographed image from an image input unit to displaying a decomposed print image by a print image display unit. [Figure 13] 10 is a flowchart showing an example of operations from receiving an input of a print image selection from an operator to correcting the read result. [Figure 14] 1 is a schematic diagram showing an example of a character that can be recognized using the present technology; [Figure 15] FIG. 2 is a block diagram showing an example of the hardware configuration of a computer 60 that can be used as an operator terminal and a server device. DETAILED DESCRIPTION OF THE INVENTION

[0026] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0027] [Character Recognition System] Fig. 1 is a schematic diagram showing an example of the configuration of a character recognition system according to one embodiment of the present invention. The character recognition system 1 shown in Fig. 1 is one embodiment of an information processing system according to the present invention.

[0028] The character recognition system 1 includes an operator terminal 2 and a server device 3.

[0029] The operator terminal 2 is a terminal used by the operator 4, and has a photographing function, an image display function, and an input reception function. The photographing function can be realized by installing a photographing device such as a digital camera. The display function can be realized by installing a display device using, for example, an LCD or EL. The input reception function can be realized by installing an operation device such as a keyboard, a pointing device, or a touch panel.

[0030] 1, a smartphone is used as the operator terminal 2. A digital camera (not shown) of the smartphone realizes a photographing function, and a touch panel 5 of the smartphone realizes an image display function and an input reception function.

[0031] 1, the operator terminal 2 and the server device 3 are communicably connected to each other via a network 6. The network 6 is constructed, for example, by the Internet or a wide area communication network. Alternatively, any WAN (Wide Area Network) or LAN (Local Area Network) may be used, and the protocol for constructing the network 6 is not limited.

[0032] The operator terminal 2 and the server device 3 each have hardware necessary for a computer, such as a processor such as a CPU, GPU, or DSP, memory such as a ROM or RAM, and a storage device such as an HDD (see FIG. 15). The processor loads the program according to the present invention stored in the storage unit or memory into the RAM and executes it, thereby executing the information processing method (character recognition method / character recognition result presentation method) according to the present invention.

[0033] As the operator terminal 2 having the photographing function, the image display function, and the input reception function, it is also possible to use other computers such as a tablet terminal or a notebook PC.

[0034] It is also possible to connect a digital camera and a notebook PC so that they can communicate with each other via wire or wirelessly, and use both devices as a set as the operator terminal 2. In other words, it is also possible to use a set of multiple devices that each have a photographing function, an image display function, and an input reception function.

[0035] The server device 3 may be any computer such as various PCs or tablet terminals.

[0036] In this embodiment, the server device 3 provides a character recognition application service (hereinafter referred to as a character recognition application) that includes presenting the character recognition results to the operator 4.

[0037] The character recognition application is provided as, for example, a web application, and the operator 4 can use the character recognition application on a web page displayed by a web browser using the operator terminal 2.

[0038] Of course, the present invention is not limited to this, and the character recognition application may be provided as a native application. In this case, the character recognition application program is downloaded to the operator terminal 2 of the operator 4.

[0039] It is also possible to install the functions of the server device 3 described below in the operator terminal 2, and to configure the character recognition system 1 using the operator terminal 2 alone. For example, it is also possible to operate a character recognition application downloaded to the operator terminal 2 as a stand-alone application. In addition, various other modes for using the character recognition application may be adopted.

[0040] In this embodiment, in the character recognition application, the server device 3 executes various processes such as character recognition, presentation of the character recognition results, acceptance of input of corrections to the character recognition results from the operator 4, correction of the character recognition results, etc. The server device 3 also executes generation and output of various GUIs (Graphical User Interfaces), such as a GUI for checking the character recognition results and a GUI for correcting the character recognition results.

[0041] 1, the server device 3 is illustrated as a computer consisting of a single housing. However, the configuration is not limited to this, and the character recognition system 1 may be realized by multiple server devices with various functions working together.

[0042] For example, the operator 4 starts a character recognition application using the operator terminal 2. Then, the operator 4 transmits a captured image of an object on which characters to be recognized are displayed to the server device 3 via the network 6.

[0043] 1, in this embodiment, an object on which characters to be recognized are displayed will be described using, as an example, a time card 7 on which arrival and departure times, etc. are printed (punched). Note that "an object on which characters are displayed" refers to a state in which characters are written on the object by any method, such as printing, imprinting, or handwriting, and the characters are displayed.

[0044] For example, the operator 4 inputs an instruction to execute a character recognition function (character reading function) in the character recognition application. For example, the operator 4 selects a button or the like displayed on the touch panel 5 of the operator terminal 2 for executing character recognition.

[0045] The operator terminal 2 is switched to a photographing mode, and the operator 4 photographs the time card 7 using the operator terminal 2. The photographed image of the time card 7 is sent to the server device 3, and the server device 3 performs character recognition on the characters such as the punched time.

[0046] The result of the character recognition performed by the server device 3 is transmitted to the operator terminal 2. For example, a GUI including the character recognition result is generated, transmitted to the operator terminal 2, and displayed on the touch panel 5. The operator 4 can check the character recognition result and input corrections, etc.

[0047] It is also possible to use a scanned image obtained by an image scanner, etc., as the target image for character recognition. For example, it is possible to generate a scanned image of the time card 7 by an image scanner, send it from the operator terminal 2 to the server device 3, and perform character recognition of the time stamped, etc.

[0048] The photographed image of the time card 7 photographed by the operator terminal 2 and the scanned image of the time card 7 generated by the image scanner are embodiments of the object image according to the present invention.

[0049] In this embodiment, a time card 7 is taken as an example of an object on which characters according to the present invention are displayed. However, the scope of application of the present invention is not limited to the time card 7, and the present invention can be applied to any object on which characters to be recognized are displayed.

[0050] Furthermore, "characters" according to the present invention include various characters such as numbers, alphabets, hiragana, katakana, kanji, and characters used in various countries around the world. Also included in the "characters" according to the present invention are various symbols such as mathematical symbols, chemical symbols, and map symbols. Of course, it is also possible to implement the present invention by limiting the "characters" to be recognized to numbers from 0 to 9, alphabets, etc.

[0051] In this embodiment, various information for configuring the character recognition system 1 is stored in the storage unit of the server device 3 (see FIG. 15). For example, captured images of objects on which characters are displayed, character recognition results, format information for generating various GUIs, etc. are stored. The storage unit of the server device 3 also stores recognition result information (see FIG. 5) and character information (see FIG. 7), which will be described later. Any other information related to character recognition may be stored.

[0052] 1, the server device 3 is configured on the network 6. However, the configuration is not limited to this, and the server device 3 may be configured near the operator terminal 2, and the server device 3 and the operator terminal 2 may be connected to each other so as to be able to communicate with each other via wire or wirelessly.

[0053] Fig. 2 is a block diagram showing an example of the functional configuration of a server device 3 according to one embodiment of the present invention. In this embodiment, as shown in Fig. 2, a processor of the server device 3 executes a program according to the present technology to configure the following functional blocks: an image acquisition unit 12, a character recognition unit 13, a recognition result image generation unit 14, a cut-out image generation unit 15, an image classification unit 16, a display control unit 17, and an input acceptance unit 18. Of course, dedicated hardware such as an IC (integrated circuit) may be used to realize each functional block.

[0054] The program is installed in the server device 3 via, for example, various recording media. Alternatively, the program may be installed via the Internet or the like. There are no limitations on the type of recording medium on which the program is recorded, and any computer-readable recording medium may be used. For example, any computer-readable non-transitory storage medium may be used.

[0055] 3 is a flowchart showing an example of the character recognition operation by the server device 3. First, the image acquisition unit 12 acquires a photographed image of an object on which characters are displayed (step 101). In this embodiment, a photographed image of a time card 7 on which the arrival and departure times of work and the like are printed is acquired. The acquired photographed image of the time card 7 is stored in the memory unit of the server device 3.

[0056] 4 is a schematic diagram showing an example of a captured image 20 of a time card 7. As shown in FIG. 4, the time card 7 has a printing area 21 for times such as clock-in and clock-out. In this embodiment, the following printing columns are configured from the left side of the printing area 21. The printing columns can also be called punch columns.

[0057] "Date"...Numbers 1 to 15 are pre-printed. Numbers 16 to 31 are pre-printed on the back. "Regular arrival time": The regular arrival time will be printed. "Leave on time": The time of leaving within working hours will be printed. "Left within working hours": The return time from leaving work during working hours will be printed. "Leave work on time": The normal time for leaving work is clocked in.

[0058] In the example shown in FIG. 4, the normal arrival time and normal departure time are printed for each of the "Regular Arrival Time" and "Regular Leave Time" columns for the 1st, 3rd, 5th to 7th, 10th, and 12th to 15th days.

[0059] The character recognition unit 13 performs character recognition using OCR based on the acquired photographed image 20, and recognizes the characters displayed on the time card 7 (step 102). In this embodiment, in the printing area 21 shown in FIG. 4, the pre-printed characters for "date," the characters 1 to 15, the characters for "start on time," the characters for "within time," the characters for "leave" below "within time," the character for "start" below "within time," and the character for "leave on time" are recognized. The character recognition unit 13 also recognizes the time to be printed in each printing column of the printing area 21.

[0060] In character recognition by OCR, for example, a character area in which characters are displayed is detected from the captured image 20, and the characters included in the detected character area are recognized one by one. There are no other specific limitations on the algorithm for performing character recognition by OCR, and any algorithm may be used.

[0061] Deep learning can be used for character recognition. LSTM (Long Short Term Memory), a type of deep learning like Tesseract, or the YOLO (You Only Look Once) model for object detection can be used. For example, by using AI (artificial intelligence) that performs deep learning, character recognition can be performed with high accuracy. OCR using AI technology is also called AI-OCR.

[0062] 5 is a schematic diagram showing an example of recognition result information. In this embodiment, the character recognition unit 13 performs character recognition to generate recognition result information 23 shown in FIG.

[0063] As shown in FIG. 5, the recognition result information 23 includes the following information: "Date": Information indicating the date. In this embodiment, any one of 1 to 15 is stored. For example, in the case of the normal work start time of "8:55" on "Day 1" in FIG. 4, "1" is stored in "Date."

[0064] "Print column": The column in which the recognized time is printed. In this embodiment, one of "Leave on time," "Leave on time," "Leave on time," or "Leave on time" is stored. For example, index numbers 1 to 7 may be assigned in order to these print columns, and the index numbers may be stored. For example, if the normal arrival time on "Day 1" is "8:55," then "Leave on time" or the index number "1" for "Leave on time" is stored.

[0065] "Recognition result (time)": The recognized time. If the normal start time for "Day 1" is "8:55", the recognition result of "8:55" will be stored.

[0066] "Position (area)": The position on the captured image 20 at the recognized time is stored. For example, the X and Y coordinates of the pixel in the upper left corner of the recognized time and the number of pixels for the width and height of the character area are stored. For example, in the case of a normal work start time of "8:55" on "Day 1," (X, Y) = (102, 175), width = 16, and height = 12 are stored. This information means that the characters "8:55" were recognized from an image area of ​​16 pixels wide and 12 pixels high, starting from the pixel whose X and Y coordinates on the captured image 20 are (102, 175).

[0067] "Element 1" to "Element N" store information about each character that makes up the "recognition result (time)." Specifically, the "recognition result (1 character)" and "position (area)" of the leftmost character that makes up the "recognition result (time)" are stored in "Element 1." The "recognition result (1 character)" and "position (area)" of the next character to the right are stored in "Element 2."

[0068] For example, if the normal start time for "Day 1" is "8:55", the following information is stored: "Element 1" “Recognition result (1 character)”…”8” "Position (area)" ... "(X, Y) = (102, 175), Width = 5, Height = 12" "Element 2" “Recognition result (1 character)”… “:” "Position (area)" ... "(X, Y) = (107, 175), Width = 1, Height = 12" "Element 3" “Recognition result (1 character)”…”5” "Position (area)" ... "(X, Y) = (108, 175), Width = 5, Height = 12" "Element 4" “Recognition result (1 character)”…”5” "Position (area)" ... "(X, Y) = (113, 175), width = 5, height = 12"

[0069] The specific data format of the recognition result information 23 is not limited, and any data format may be used.

[0070] The recognition result image generating unit 14 generates a recognition result image including the recognition result by the character recognition unit 13 (step 103). The recognition result image includes any image including the recognition result by the character recognition unit 13. Of course, the recognition result image also includes text data of the recognition result. The recognition result image also includes any image that allows the operator 4 to understand what kind of recognition result has been obtained.

[0071] 6 is a schematic diagram showing an example of a recognition result image. Note that the following description will be given taking as an example the print area 21 of the time card 7, which is the target of character recognition.

[0072] 6, the recognition result image 25 is constructed by inputting the recognition result by the character recognition unit 13 into a time card image 26 that resembles the print area 21 of the time card 7. As shown in FIG. 6, the time card image 26 has almost the same configuration as the photographed image 20, and is configured with print columns for "Date" (1 to 15) and "Clock in on time," "Clock out within time," "Clock in within time," and "Clock out on time." Format information for the time card image 26 may be stored in advance in the storage unit of the server device 3.

[0073] 5 from the storage unit of the server device 3. Then, based on the read recognition result information 23, the recognition result image generating unit 14 inputs the "recognition result (time)" into each "print column" of the time card image 26. As a result, the recognition result image 25 is generated.

[0074] 2 controls the display of various images on the touch panel 5 of the operator terminal 2 as a function of the character recognition application. For example, the display control unit 17 controls the display of text, icons, etc. for using the character recognition application, the display of character recognition results, etc. In this embodiment, the display control unit 17 also controls the display of various GUIs, such as a GUI for checking the character recognition results and a GUI for correcting the character recognition results.

[0075] 6, the recognition result image 25 and the captured image 20 are shown side by side. With the configuration shown in FIG. 6, the display control unit 17 may display the recognition result image 25 and the captured image 20 so that they can be compared as a GUI for checking the character recognition result (hereinafter referred to as image comparison display).

[0076] The operator 4 can check the character recognition result by checking the recognition result image 25 and the captured image 20, which are displayed on the touch panel 5 of the operator terminal 2 so as to be comparable to each other. Of course, other display methods may be employed to display the recognition result image 25 and the captured image 20 so as to be comparable to each other, such as enlarging and displaying the same portions side by side.

[0077] The cut-out image generating unit 15 generates cut-out images cut out for each recognized character from the photographed image 20 (step 104). In this embodiment, the cut-out images are generated by cutting out the numbers 0 to 9 from the time printed in the print area 21 for each character.

[0078] 5 from the storage unit of the server device 3. Then, based on the information of "element 1" to "element N" in the recognition result information 23, the cut-out image generation unit 15 cuts out an image area defined by the "position (area)" of an element whose "recognition result (one character)" is a number from 0 to 9, and generates the cut-out image.

[0079] The cut-out image generating unit 15 also generates character information 28 shown in Fig. 7. The character information 28 is information about each cut-out image, and is managed by breaking down the "recognition result (time)" of the recognition result information 23 shown in Fig. 5 into numbers and assigning an ID to each number.

[0080] As shown in FIG. 7, the text information 28 includes the following information: "ID": ID for identifying each number that makes up the "recognition result (time)". In this embodiment, the numbers "1" and "2" are assigned as "IDs" to each number that makes up the "recognition result (time)" in order from the left.

[0081] "Recognition result (1 character)": Recognized number. The same information as "Recognition result (1 character)" in "Element 1" to "Element N" of recognition result information 23 is stored. "Date": Information indicating the date. The same information as "Date" in the recognition result information 23 is stored. "Print column": The print column in which the recognized number was printed. The same information as the "print column" of the recognition result information 23 is stored. "Position (area)": The "position (area)" of the "recognition result (time)" including the recognized number. The same information as the "position (area)" of the "recognition result (time)" in the recognition result information 23 is stored. This "position (area)" is referenced when performing highlighting, which will be explained later. Instead of or in addition to this information, the "position (area)" of the "recognition result (one character)" may be stored. In other words, the "position (area)" of the "recognition result (one character)" within "element 1" to "element N" in the recognition result information 23 may be stored.

[0082] "Cut-out image (link)": A link to a cut-out image stored in the storage unit of the server device 3 is stored.

[0083] For example, taking the normal work start time "8:55" for "1st day" in FIG. 5 as an example, the text information 28 is as follows:

[0084] "ID"..."1" “Recognition result (1 character)”…”8” "Date"..."1" "Print column"..."Regular departure" or "Regular departure" index number "1" "Position (area)" ... "(X, Y) = (102, 175), Width = 16, Height = 12" "Cut-out image (link)"

[0085] "ID"..."2" “Recognition result (1 character)”…”5” "Date"..."1" "Print column"..."Regular departure" or "Regular departure" index number "1" "Position (area)" ... "(X, Y) = (102, 175), Width = 16, Height = 12" "Cut-out image (link)"

[0086] "ID"..."3" “Recognition result (1 character)”…”5” "Date"..."1" "Print column"..."Regular departure" or "Regular departure" index number "1" "Position (area)" ... "(X, Y) = (102, 175), Width = 16, Height = 12" "Cut-out image (link)"

[0087] The image classification unit 16 classifies the generated cut-out images into groups of identically recognized images based on the recognition results of the character recognition unit 13 (step 105).

[0088] Fig. 8 is a schematic diagram for explaining classification of cut-out images. The image classification unit 16 reads out the character information 28 shown in Fig. 7 from the storage unit of the server device 3. Then, based on the character information 28, the cut-out images 30 (30(0) to 30(9)) are classified into groups of identical recognition result images that result in the same recognition result.

[0089] For example, the image classification unit 16 performs a search for the "recognition result (character)" of the character information 28, extracts character information 28 whose "recognition result (character)" is the same number, reads out the cut-out image 30 from the "cut-out image (link)" of the character information 28, and acquires it as a group of images with the same recognition result.

[0090] 8, the second cut-out image 30(8) from the right of the group of identical recognition result images for which the recognition result is "8" displays the number 3. This means that the character recognition result by the character recognition unit 13 was incorrect, and the number 3 was recognized as "8."

[0091] The result of the printed number 3 being erroneously recognized as "8" is stored in the "recognition result (time)" and "recognition result (one character)" of the recognition result information 23 shown in Fig. 5. Also in the character information 28 shown in Fig. 7, "8" is stored in the "recognition result (one character)" and a link to the cut-out image 30 in which the number 3 is displayed is stored in the "cut-out image (link)".

[0092] The display control unit 17 displays the cut-out images 30 that are included in the group of identical recognition result images and result in the same recognition result for each recognition result (step 106). In this embodiment, the display control unit 17 displays the cut-out images 30(0) to 30(9) shown in Fig. 8 for each recognition result "0" to "9".

[0093] The method of displaying the cutout image 30 for each recognition result includes any display method that allows cutout images that result in the same recognition result to be compared with each other. For example, cutout images 30 that result in the same recognition result are displayed together for each recognition result.

[0094] For example, the display control unit 17 can align and display cropped images 30 that result in the same recognition result in a configuration similar to that shown in Fig. 8. For example, as shown in Fig. 8, a group of identical recognition result images that result in the recognition result "8" are aligned in a row from side to side and displayed on the touch panel 5 of the operator terminal 2. This allows the operator 4 to easily find the cropped image 30 (8) that displays the number 3. As a result, it becomes possible to easily grasp that the character recognition result is incorrect, i.e., that there is a reading error.

[0095] Furthermore, the display control unit 17 can also display cropped images 30 that result in the same recognition result by switching them in order at a predetermined time interval. For example, a group of identical recognition result images that result in the recognition result "8" shown in Fig. 8 are displayed in the same position on the touch panel 5 of the operator terminal 2 by switching them in order from the left or right side. The time interval may be set arbitrarily, for example, to 0.2 seconds, 0.5 seconds, or the like.

[0096] By displaying the same numbers like in a flip book, it is possible to create an animation-like display in which the same numbers fluctuate. Human vision has a high ability to detect changes, so it is easy to detect when the same numbers change to different numbers. Since this can be detected even when the display is switched at high speed, it is possible to find reading errors in a short time. It is also possible to check large amounts of data in a short time.

[0097] That is, by checking the cutout images 30(8) displayed in sequence, the operator 4 can easily find that the cutout image 30(8) displaying the number 3 is mixed in. In other words, as a result, it becomes possible to easily understand that the character recognition result is incorrect. Note that the method of displaying cutout images 30 with the same recognition result by switching them in sequence at a predetermined time interval can also be called switching display or automatic flipping display.

[0098] In addition, when cropped images 30 that produce the same recognition results are aligned and displayed, or switched and displayed in order at predetermined time intervals, it is also possible to set the order in which they are aligned or switched and displayed.

[0099] In this embodiment, the display control unit 17 can arrange and display cut-out images 30 that result in the same recognition result in an order based on the feature amounts of the cut-out images 30. The display control unit 17 can also switch and display cut-out images 30 that result in the same recognition result in an order based on the feature amounts of the cut-out images 30.

[0100] For example, based on the feature amount, it is possible to arrange and display the cut-out images 30 that will result in the same recognition result in order of similarity. Also, based on the feature amount, it is possible to switch and display the cut-out images 30 that will result in the same recognition result in order of similarity.

[0101] The cut-out images 30 that result in the same recognition result are displayed in a line such that similar images are arranged in order, or similar images are displayed in order by switching between them. This allows the operator 4 to easily distinguish the differences between the cut-out images 30. This makes it possible to easily find cut-out images 30 that have an incorrect recognition result.

[0102] Any information can be used as the feature, such as the size of the image, the number of black dots in the image that represent the print area, the color spectrum, etc. Any other feature that can determine the similarity can be used, and for example, multidimensional feature can be detected.

[0103] 8, the size of the cut-out image 30 is detected as a feature amount, and the cut-out images 30 are arranged in descending order of size. Displaying the cut-out images in this order is also effective.

[0104] For example, the feature amount of each of the numbers 0 to 9 may be stored in advance as a reference feature amount in the storage unit of the server device 3. Then, the feature amount of the print area in the cut-out image 30 is compared with the reference feature amount of the number that is the recognition result of the cut-out image 30, and the cut-out images 30 are sorted in order of the degree of similarity with the reference feature amount, or are displayed by switching between them. Such processing is also possible.

[0105] The detection of the feature amount of the cut-out image 30 may be performed by the image classification unit 16 or the display control unit 17. Alternatively, a functional block for detecting the feature amount of the cut-out image 30 may be configured separately.

[0106] In this embodiment, the display control unit 17 can display the cut-out images 30 that result in the same recognition result in an order based on the reliability of the recognition result. The display control unit 17 can also display the cut-out images 30 that result in the same recognition result in an order based on the reliability of the recognition result.

[0107] Specific examples of the order based on the reliability of the recognition results include an order from highest to lowest reliability of the recognition results, or an order from lowest to highest reliability of the recognition results.

[0108] The reliability of the recognition result of the cut-out image 30 can be detected based on, for example, the above-mentioned feature amount. For example, if the size of the cut-out image 30 is small or if the number of black dots is small, it can be determined that there is a high possibility of erroneous recognition, i.e., the reliability is low.

[0109] Alternatively, the feature amount of each of the numbers 0 to 9 is stored in advance as a reference feature amount in the storage unit of the server device 3. The feature amount of the print area in the cut-out image 30 can be compared with the reference feature amount of the number that is the recognition result of the cut-out image 30, and the similarity to the reference feature amount can be used as a parameter representing the reliability of the recognition result. The higher the similarity to the reference feature amount, the higher the reliability of the recognition result can be determined to be.

[0110] Alternatively, when character recognition is performed using OCR, a recognition likelihood (an evaluation index of the accuracy of recognition) may be output along with the recognition result. The recognition likelihood may also be used as the reliability of the extracted image 30.

[0111] In this way, the cut-out images 30 that result in the same recognition result are displayed in order of increasing reliability (or decreasing reliability), or are alternately displayed in order of increasing reliability (or decreasing reliability). This allows the operator 4 to focus on the cut-out images 30 that have low reliability and are likely to be misrecognized. This makes it possible to easily find the cut-out images 30 that have an incorrect recognition result.

[0112] 2 receives various inputs from the operator 4 using the operator terminal 2. The input receiving unit 18 can determine what instructions or information have been input based on a signal (operation signal) in response to an operation of the touch panel 5 of the operator terminal 2 by the operator 4.

[0113] In this embodiment, the input receiving unit 18 determines whether or not a selection input for the cut-out image 30 displayed on the touch panel 5 of the operator terminal 2 has been received (step 107). For example, the operator 4 inputs a selection of the cut-out image 30 that may be misrecognized (read error).

[0114] For example, suppose that cut-out images 30(8) that result in the same recognition results are arranged and displayed in the same configuration as in Fig. 8. Suppose that the operator 4 inputs an operation to select the cut-out image 30(8) that displays the number 3. In this case, the input receiving unit 18 determines that the input to select the cut-out image 30 has been received.

[0115] Also, assume that cut-out images 30(8) that result in the same recognition result are displayed in sequence at predetermined time intervals. If the operator 4 inputs an input to stop the display switching while cut-out image 30(8) displaying the number 3 is displayed, it is determined that an input to select cut-out image 30(8) displaying the number 3 has been accepted. In this manner, in this embodiment, when display switching is performed, an input to stop the display switching while a predetermined cut-out image is displayed can be accepted as an input to select the displayed predetermined cut-out image.

[0116] For example, the operator 4 stops (or pauses or interrupts) the switching display when the cutout image 30(8) showing the number 3 is displayed while the cutout image 30(8) is being switched and displayed. By accepting this input as an input for selecting a cutout image, high operability can be achieved.

[0117] If the input of selection for the cut-out image 30 is not accepted (No in step 107), the process ends. For example, if the results of character recognition are all correct, the cut-out image 30 is not selected and the process ends.

[0118] If the input of the selection for the cut-out image 30 is accepted (Yes in step 107), the display control unit 17 highlights and displays the area corresponding to the selected cut-out image 30 (step 108).

[0119] 9 is a schematic diagram showing an example of highlighting the area corresponding to the selected cropped image 30. In FIG. 9, an example of highlighting is shown in the image comparison display shown in FIG.

[0120] For example, assume that the operator 4 selects the cutout image 30(8) displaying the number 3. The display control unit 17 reads out the text information 28 shown in Fig. 7 for the cutout image 30(8) displaying the number 3, and refers to the "date," "print column," "position (area)," etc.

[0121] As a result, in the recognition result image 25 shown in Fig. 9, it is possible to highlight the print column of "Date" = 10 and "Print column" = "Leave on time" as the area 32 corresponding to the selected cut-out image 30(8). The "8" in the "Recognition result (time)" = "21:28" in the print column is the recognition result of the selected cut-out image 30(8). In other words, the number 3 has been erroneously recognized as an "8."

[0122] In addition, in the captured image 20 shown in Figure 9, it is possible to highlight the area specified by "Position (area)" as the area 33 corresponding to the selected cropped image 30(8). The highlighted area 33 corresponds to the printing field for "Date" = 10 and "Printing field" = "Leave on time" in the captured image 20. The cropped image of the "3" part of "21:23" displayed in the highlighted area becomes cropped image 30(8) in which the number 3 is displayed.

[0123] In this way, by selecting the cropped image 30 that results in the same recognition result displayed for each recognition result, the operator 4 can quickly and easily determine which printed portion in the recognition result image 25 and the captured image 20 has been misrecognized.

[0124] Any display method may be adopted as a method for highlighting and displaying the region 32 and the region 33. For example, any method that allows the operator 4 to distinguish the region 32 and the region 33 is included in the highlighting according to the present invention.

[0125] The input accepting unit 18 determines whether or not an input for correction to the characters included in the highlighted area 32 in the recognition result image 25 has been accepted (step 109). The characters included in the highlighted area 32 correspond to the recognition result by the character recognition unit 13.

[0126] If the input of correction to the characters included in the highlighted area 32 is not accepted (No in step 109), the process ends.

[0127] If an input for correction to the characters included in the highlighted area 32 is accepted (Yes in step 109), the recognition result is corrected (step 110). For example, the display control unit 17 displays a GUI for correcting the recognition result by the character recognition unit 13. The input accepting unit 18 accepts the input of correction by the operator 4.

[0128] 10 is a schematic diagram showing an example of correction of the recognition result. For example, the operator 4 selects the area 32 corresponding to the cut-out image 30(8) of the recognition result image 25. A GUI is displayed that allows the operator to input numbers into the printing field corresponding to the area 32. The operator 4 corrects "21:28" in the printing field corresponding to the area 32 to "21:23" and inputs it. The correction is accepted, and the recognition result is corrected.

[0129] The correction content is saved in the storage unit of the server device 3 (step 111). In this embodiment, in the character information 28 shown in FIG. 7 for the cut-out image 30(8) displaying the number 3, the "recognition result (one character)" is changed from "8" to "3." Also, in the recognition result information 23 shown in FIG. 5 in which the area 33 of the captured image 20 shown in FIG. 10 is stored as the "position (area)," the "recognition result (time)" is changed from "21:28" to "21:23." Also, the "recognition result (one character)" of "element 4" is changed from "8" to "3." As a result, the correction content of the recognition result is saved.

[0130] The GUI for correcting the recognition result and the operation method for correcting the recognition result by the operator 4 are not limited and may be set arbitrarily. In this embodiment, the storage unit of the server device 3 functions as an embodiment of a storage unit that stores the input corrected characters.

[0131] A more detailed embodiment of the character recognition system 1 shown in Fig. 1 will now be described. Fig. 11 is a block diagram showing a more detailed configuration example of the character recognition system 1. The character recognition system 1 according to this embodiment has an image input unit 35, a timestamp reading unit 36, a print image decomposition unit 37, an image search unit 38, a print image display unit 39, a timestamp correction unit 40, a time card display unit 41, a highlighting unit 42, a timestamp identification unit 43, and a print image selection unit 44.

[0132] The character recognition system 1 also includes a print image information storage unit 45 and a recognition result information storage unit 46.

[0133] Image input unit 35 shown in Fig. 11 functions as an embodiment of the image acquisition unit according to the present invention, and is configured, for example, within server device 3 shown in Fig. 1. Also, embossment reading unit 36 ​​functions as an embodiment of the character recognition unit according to the present invention, and is configured, for example, within server device 3 shown in Fig. 1.

[0134] The print image decomposition unit 37 shown in Fig. 11 functions as an embodiment of the cut-out image generation unit according to the present invention, and is configured, for example, within the server device 3 shown in Fig. 1. The print image corresponds to an embodiment of the cut-out image according to the present invention.

[0135] The image search unit 38 shown in FIG. 11 functions as an embodiment of the image classification unit according to the present invention, and is configured, for example, within the server device 3 shown in FIG.

[0136] The print image information storage unit 45 and the recognition result information storage unit 46 shown in FIG.

[0137] The print image display unit 39, stump correction unit 40, time card display unit 41, highlighting unit 42, and stump identification unit 43 shown in Fig. 11 function as an embodiment of the display control unit according to the present invention, and are configured, for example, within the server device 3 shown in Fig. 1. Of these, the stump correction unit 40 also functions as an embodiment of the input acceptance unit according to the present invention. Furthermore, the time card display unit 41 also functions as an embodiment of the recognition result image generation unit according to the present invention.

[0138] A print image selection unit 44 shown in FIG. 11 functions as an embodiment of the input reception unit according to the present invention, and is configured, for example, within the server device 3 shown in FIG.

[0139] The image input unit 35 inputs the photographed image 20 of the time card 7 into the system and outputs it to the stump reading unit 36. The stump reading unit 36 ​​performs character recognition using OCR to read the printed time, date, printing fields, etc. from the photographed image 20. Then, the recognition result information 23 shown in FIG. 23 is stored in the recognition result information storage unit 46.

[0140] When the recognition result information 23 is stored, the time card display unit 41 is activated and a recognition result image 25, as shown in Fig. 6, is generated. The recognition result image 25 and the captured image 20 are also displayed on the touch panel 5 of the operator terminal 2 so that they can be compared.

[0141] Furthermore, once character recognition is performed by the time stamp reader 36 and the recognition result information 23 is stored, the print image decomposer 37 is activated. The print image decomposer 37 reads the photographed image 20 of the time card 7 and the recognition result information 23 from the recognition result information storage unit 46.

[0142] Then, for all punches in the recognition result information 23, the numbers used are extracted character by character and assigned an ID. Then, images of numbers are extracted character by character from the photographed image 20 of the time card 7, associated with the ID, and stored in the print image information storage unit 45. For example, the character information 28 shown in FIG. 7 can be used as the print image information.

[0143] The print image decomposition unit 37 extracts all numbers from all stamps, associates the print image information with the print image, and stores the associations in the print image information storage unit 45, then ends the process and activates the image search unit 38.

[0144] The image search unit 38 sequentially searches and extracts numbers from 0 to 9 from the print image information storage unit 45. Specifically, by specifying a number, print image information and print images of the same number are extracted from the print image information storage unit. The extracted print images for each number are output to the print image display unit 39, and an order is assigned according to the feature amount of the print image.

[0145] The print image display unit 39 displays the print images to which the numbers have been assigned, arranged in the assigned order for each number, on the touch panel 5 of the operator terminal 2. Alternatively, the print image display unit 39 may switch and display the print images to which the numbers have been assigned in the assigned order at the same display position at predetermined time intervals.

[0146] The print image selection unit 44 determines whether or not the operator 4 has input a print image selection, and if a print image selection has been input, outputs print image information regarding the print image selected by the operator 4 to the embossing identification unit 43.

[0147] The embossing identification unit 43 refers to the received print image information, identifies the embossing date and print field contained in the print image selected by the operator 4, and outputs them to the highlighting unit 42. The highlighting unit 42 uses the identified date and print field information to highlight (highlight) the corresponding area of ​​the recognition result image 25 and the captured image 20 displayed on the touch panel 5 of the operator terminal 2.

[0148] The embossing correction unit 40 determines whether or not the operator 4 has input an input to correct the embossed information, and if an input to correct the recognition result has been input, corrects the embossed information.

[0149] FIG. 12 is a flowchart showing an example of the operation from when the photographed image 20 is received from the image input unit 35 to when the decomposed print image is displayed by the print image display unit 39.

[0150] When the photographed image 20 is input by the image input unit 35 (step 201), the input photographed image 20 is stored in the recognition result information storage unit 46 (step 202), and then the stamp printed on the photographed image 20 is read (step 203).

[0151] The read stamp is broken down into individual numbers (step 204), and the position information on the photographed image 20 and the print image are stored as print image information in the print image information storage unit 45 (step 205).

[0152] Once all the print image information has been saved, the process begins to display the numbers from 0 to 9 for each recognition result. In step 206, the number is set to 0, and it is confirmed that the number does not exceed 9 (step 207). If the number does not exceed 9, print image information (print image) that matches the number to be processed is searched for and extracted from print image information storage unit 45 (step 208).

[0153] An order is assigned to the print images based on the extracted feature amounts of the print images (step 209), and the print images are displayed in an aligned order in the assigned order, or are displayed by switching between them (step 210).

[0154] To process the next number, the number is incremented by one (step 211) and the process returns to step 207. If the number to be processed exceeds 9 in step 207, processing of all numbers is complete and the process ends.

[0155] FIG. 13 is a flowchart showing an example of the operation from receiving input of the print image selection from the operator 4 to correcting the reading result when a misrecognition (reading error) is found in the print image displayed by the print image display unit 39.

[0156] When a print image that has been erroneously classified and displayed is found among the print images displayed by the print image display unit 39, the operator 4 selects the print image.

[0157] When the print image selection unit 44 receives the selection input from the operator 4 (step 301), the date and print field corresponding to the print image are identified from the print image information of the selected print image (step 302).

[0158] Based on the identified date and printing column information, the corresponding punch (printing column) is highlighted in the recognition result image 25 displayed by the time card display unit 41 (step 303). Next, the area displaying the corresponding punch on the original photographed image 20 displayed on the time card display unit 41 is highlighted (step 304).

[0159] The embossing correction unit 40 waits for correction input from the operator 4, and when it receives the instruction input (step 305), it checks whether the instruction is for correction (step 306), and if it is an instruction for correction, it makes the requested correction (step 307), and stores the correction result in the recognition result information storage unit 46 (step 308), thereby completing the process.

[0160] As described above, in the character recognition system 1 according to this embodiment, the clipped images 30 clipped for each recognized character are classified into identical recognition result image groups that result in the same recognition result. Then, the display control unit 17 displays the clipped images 30 that result in the same recognition result and are included in the identical recognition result image group for each recognition result. This allows the operator 4 to check the character recognition results efficiently and with high accuracy.

[0161] In attendance management using time cards7, the task of aggregating the printed times at the end of the month and calculating payroll is a time-consuming and burdensome task for the person in charge. To solve this problem, attendance management systems have been proposed that electronically record arrival and departure times and automatically calculate payroll.

[0162] However, in workplaces with a small number of employees or where part-time employees are frequently replaced, time cards are still widely used because they are inexpensive and do not require registration, which means a lot of work is required to tally the data at the end of the month.

[0163] In workplaces where time cards 7 are used, a system has been proposed that uses OCR to read the time printed on the time card 7 and reduce the burden on the person in charge of tasks such as tallying up the time at the end of the month.

[0164] However, when OCR is used to recognize characters, reading errors often occur due to lighting conditions, shadows, or dirt on the time card when photographing or scanning the time card. If the time is read incorrectly, the work hours will not be calculated correctly, which can lead to incorrect payroll calculations, which can be a major problem. For this reason, even if time cards are read using OCR, manual checking is still required.

[0165] When checking the results of character recognition using OCR, it is not easy for humans to find errors in a string of numbers. For example, the "3" and "8" look similar in shape, as in "17:13" and "17:18," and are easily overlooked. Also, in a company where the regular working time is "17:15," leaving work at "17:13" is considered early, but leaving work at "17:18" is considered regular, which has a very different meaning in terms of attendance management.

[0166] The person performing the check (operator 4) compares the OCR reading results displayed on the screen with the photographed image 20 used for the reading, moving their eyes from side to side. Alternatively, they compare the OCR reading results displayed on the screen with the actual time card 7 in hand, moving their eyes up and down.

[0167] This type of confirmation method requires a lot of eye movement, which is a significant burden. In addition, when comparing, the time printed on the time card 7 must be memorized and then compared with the time read, which places a high cognitive load on the worker, making it easy to make mistakes when working for long periods of time.

[0168] For example, when checking many time cards 7, it is difficult for the operator 4 to accurately find misrecognition (reading error) using only the image comparison display as exemplified in FIG.

[0169] In the character recognition system 1 according to the present technology, cut-out images 30 are generated by cutting out numbers character by character from the original captured image 20, and are displayed together for each recognition result. The operator 4 can check whether there are any numbers other than 0 in the cut-out image 30(0) whose recognition result is "0", or whether there are any numbers other than 1 in the cut-out image 30(1) whose recognition result is "1".

[0170] Therefore, the operator 4 can compare and check the cut-out images 30 in the same number unit, with minimal eye movement and very little information to remember, so the cognitive load is also sufficiently low. As a result, it becomes possible to quickly and accurately discover misrecognition, and to efficiently and accurately check the character recognition results.

[0171] As a result, the burden on the operator 4 who performs tasks such as tallying up data at the end of the month can be significantly reduced, and payroll calculations and the like can also be made easier.

[0172] Furthermore, with this character recognition system 1, it is possible to quickly and efficiently find the position of the print that will cause a reading error and easily correct it, as shown in Figures 9 and 10. This makes it possible to achieve extremely high operability.

[0173] <Other embodiments> The present invention is not limited to the above-described embodiment, and various other embodiments can be realized.

[0174] (Regarding the "characters" to be recognized) Figure 14 is a schematic diagram showing an example of characters that can be recognized using this technology. In the example shown in Figure 14A, in each printing column for "Leave on time," "Leave on time," "Leave on time," and "Leave on time," not only the time but also the day of the week is printed in kanji to the left of the time. In the example shown in Figure 14A, the following characters are printed: "Date" = "1" "Regular departure" = "Sunday 8:21" "Date" = "2" "Regular departure" = "Mon 8:21" "Date" = "3" "Regular departure" = "Tuesday 8:20" "Date" = "3" "Leave time" = "Tuesday 9:55"

[0175] In the example shown in Figure 14B, in each of the print columns for "Leave on time," "Leave on time," "Leave on time," and "Leave on time," not only the time but also the day of the week is printed in English (first two letters) to the left of the time. As shown in Figure 14B, the English notation of the day of the week (first two letters) is printed horizontally and continuously from bottom to top. The English notation of the day of the week (first two letters) is also printed in small letters.

[0176] In the example shown in Figure 14C, in each of the print columns for "Leave on time," "Leave on time," "Leave on time," and "Leave on time," not only the time but also the date is printed in two-digit numbers to the left of the time. As shown in Figure 14C, the two-digit numbers for the date are printed horizontally and consecutively from bottom to top. The two-digit numbers for the date are also printed in small letters.

[0177] In this way, it is possible to print not only the time but also the day of the week and date in each of the printing columns for "Leave on time," "Leave on time," "Leave on time," and "Leave on time," and naturally, these days of the week and dates can be recognized by the character recognition system 1 related to the present technology.

[0178] In this case, for example, this technology can be applied to the English notation of the day of the week (first two letters) shown in Fig. 14B as one character, and also to the two-digit number of the date shown in Fig. 14C as one character.

[0179] In other words, "characters" to which this technology can be applied include sets of multiple characters. For example, characters indicating units such as "kilogram" and "gram" composed of small characters can also be recognized as a single character set.

[0180] For example, the cutout images 30 including "SU (landscape / top-bottom continuous printing)," "MO (landscape / top-bottom continuous printing)," and "TU (landscape / top-bottom continuous printing)" as shown in Fig. 14B are generated as cutout images 30. Then, the cutout images 30 are classified according to the recognition results of "SU (landscape / top-bottom continuous printing)," "MO (landscape / top-bottom continuous printing)," and "TU (landscape / top-bottom continuous printing)."

[0181] For example, suppose that a group of identical recognition result images that are the recognition result of "SU (landscape / vertical continuous printing)" includes a cutout image 30 of "TU (landscape / vertical continuous printing)", and the images are displayed in an aligned manner or switched, etc. The operator 4 can easily find the cutout image 30 of "TU (landscape / vertical continuous printing)" that has been erroneously recognized.

[0182] 14C, a cut-out image 30 including "01 (landscape / vertical continuous printing)," "02 (landscape / vertical continuous printing)," and "03 (landscape / vertical continuous printing)" is generated. Then, the cut-out image 30 is classified according to the recognition results of "01 (landscape / vertical continuous printing)," "02 (landscape / vertical continuous printing)," and "03 (landscape / vertical continuous printing)."

[0183] For example, suppose that the group of identical recognition result images that result in the recognition of "02 (landscape / vertical continuous printing)" includes a cutout image 30 of "03 (landscape / vertical continuous printing)", and the images are displayed in an aligned manner or switched, etc. The operator 4 can easily find the cutout image 30 of "03 (landscape / vertical continuous printing)" that has been misrecognized.

[0184] (Applying this technology to character strings) This technology can also be applied to a character string in which multiple "characters" are lined up, which are recognized by the character recognition unit 13 shown in Fig. 2. For example, this technology can be applied to the following character string:

[0185] Chemical formulas of H2O, CO2, etc. Currency units such as USD, JPY, VND, etc. Units such as km, cm, mm, mg, db, etc. Ticker symbols used to identify stocks on stock markets such as the US Any other string

[0186] As described with reference to Fig. 14, this technology can be applied to a character string consisting of multiple characters as one "character." On the other hand, this technology can also be applied to a character string in which multiple "characters" to be recognized are arranged, and an example will be described below.

[0187] For example, a character string to be recognized is stored in a storage unit of the server device 3 shown in Fig. 3. This character string corresponds to a character string of interest that is a target of attention according to the present technology, and will be referred to as a "character string of interest" hereinafter. This "character string of interest" can also be said to be a character string pattern that is a target of attention.

[0188] The cut-out image generation unit 15 determines whether or not a stored "target character string of interest" exists in the object image based on the recognition result for each "character" by the character recognition unit 13. For example, it is possible to determine whether or not a "target character string of interest" exists based on the recognition result for the recognized "character" and the position information of the "character" on the object image.

[0189] For example, assume that "JPY" is stored as a "character string of interest." If the "characters" recognized as "J," "P," and "Y" are arranged in this order as the recognition results by the character recognition unit 13, it is possible to determine that a "character string of interest" (="JPY") exists.

[0190] The strings recognized as "J," "P," and "Y" arranged in this order correspond to the "target string of interest" (= "JPY") displayed in the target image, and will be referred to as the "corresponding string" below.

[0191] The cut-out image generation unit 15 generates cut-out images cut out from the object image for each "corresponding character string" corresponding to the "target character string of interest." In this example, the cut-out images are recognized as "J," "P," and "Y," and the character strings arranged in this order are cut out as a group to form a single image. For example, by cutting out a rectangular area with a diagonal line connecting the top left pixel of the first character and the bottom right pixel of the last character, it is possible to generate cut-out images for each "corresponding character string." Of course, this method is not limited to this.

[0192] The determination by the cut-out image generation unit 15 of the existence or non-existence of a "target character string" and the generation of a cut-out image of a "corresponding character string" corresponding to the "target character string" can be summarized as follows: the cut-out image generation unit 15 generates a cut-out image cut out for each "corresponding character string" corresponding to the "target character string" stored from the object image, based on the recognition result by the character recognition unit 13.

[0193] The image classification unit 16 classifies the generated cut-out images into groups of identical character string images that have the same target character string. That is, cut-out images of "corresponding character strings" are classified for each "target character string." For example, cut-out images of character strings that are recognized as "J," "P," and "Y" and arranged in that order are classified into a group of identical character string images of the "target character string" (="JPY").

[0194] The display control unit 17 displays the cut-out images included in the same character string image group for each of the same target character strings. For example, the "corresponding character strings" classified as the "target character string" (="JPY") are displayed in an aligned manner or switched between displays.

[0195] For example, suppose that a character string such as "SPY" is present on the target image and is erroneously recognized as the "corresponding character string" of the "target character string" (= "JPY"). Operator 4 can easily find the cut-out image of "SPY" from among the aligned cut-out images of "JPY" or from among the cut-out images of "JPY" that are displayed by switching at predetermined time intervals.

[0196] The number of characters in the string stored as the "target string of interest" is not limited, and the present technology can be applied to any number of strings.

[0197] It is also possible to combine the method of extracting recognized characters one by one and displaying each of the same recognition results with the method of extracting each "corresponding character string" corresponding to the stored "target character string" and displaying each of the same target character strings.

[0198] For example, both the display of cutout images for each character and the display of cutout images for each "corresponding character string" may be executed as shown in Fig. 8. Alternatively, it is also possible to set the system so that cutout images for each character are generated for characters that are not stored as "character strings of interest," and cutout images for each "corresponding character string" are generated for characters that are stored as "character strings of interest."

[0199] 14B, the character recognition unit 13 may be trained to recognize, for example, the horizontal "SU" as a single character. Alternatively, the horizontal "SU" may be stored as a "character string of interest," and the horizontal "S" and "U" may be recognized individually, and the horizontal "SU" may be used as a "corresponding character string" corresponding to the "character string of interest" to generate a set of cut-out images.

[0200] Furthermore, when the character recognition unit 13 is trained to recognize the horizontal "SU" as a single character, it is also possible to have it recognized as the "day" of the week.

[0201] 9, both the region 32 corresponding to the selected cropped image 30(8) in the recognition result image 25 and the region 33 corresponding to the selected cropped image 30(8) in the captured image 20 are highlighted. This is not limiting, and only at least one of the region 32 and the region 33 may be highlighted. For example, there may be a case where only the recognition result image 25 in which the region 32 is highlighted is displayed, or a case where only the captured image 20 in which the region 33 is highlighted is displayed.

[0202] The character recognition system 1 shown in FIG. 1 can also be called a time card reading system or a time card confirmation system.

[0203] FIG. 15 is a block diagram showing an example of the hardware configuration of a computer 60 that can be used as the operator terminal 2 and the server device 3.

[0204] The computer 60 includes a CPU 61, a ROM 62, a RAM 63, an input / output interface 65, and a bus 64 that interconnects these components. The input / output interface 65 is connected to a display unit 66, an input unit 67, a storage unit 68, a communication unit 69, a drive unit 70, and the like. The display unit 66 is a display device using, for example, a liquid crystal display, an electroluminescent display, etc. The input unit 67 is, for example, a keyboard, a pointing device, a touch panel, or other operating device. When the input unit 67 includes a touch panel, the touch panel can be integrated with the display unit 66. The storage unit 68 is a non-volatile storage device such as a HDD, flash memory, or other solid-state memory. The drive unit 70 is a device capable of driving a removable storage medium 71 such as an optical storage medium or magnetic recording tape. The communication unit 69 is a modem, router, or other communication device that can be connected to a LAN, WAN, or the like and that communicates with other devices. The communication unit 69 may communicate either wired or wirelessly. The communication unit 69 is often used separately from the computer 60. Information processing by computer 60 having the above hardware configuration is realized by software stored in storage unit 68, ROM 62, etc., working in cooperation with the hardware resources of computer 60. Specifically, the information processing method according to the present invention is realized by loading a program constituting the software stored in ROM 62, etc., into RAM 63 and executing it. The program is installed in the computer 60 via, for example, the recording medium 71. Alternatively, the program may be installed in the computer 60 via a global network or the like. Any other computer-readable non-transitory storage medium may be used.

[0205] The information processing method (character recognition method / method for presenting character recognition results) and program according to the present invention may be executed by cooperation of multiple computers connected to each other so as to be able to communicate via a network or the like, thereby constructing an information processing system or information processing device according to the present invention. That is, the information processing method and program according to the present invention can be executed not only in a computer system configured by a single computer, but also in a computer system in which a plurality of computers operate in conjunction with each other. In this disclosure, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all the components are in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0206] The execution of the information processing method and program according to the present invention by a computer system includes both cases where the acquisition of an object image, character recognition, generation of a cut-out image, classification of the cut-out image, display control of the cut-out image, generation of a recognition result image, acceptance of input for selection and correction, correction processing, etc. are performed by a single computer, and cases where each process is performed by a different computer. Furthermore, the execution of each process by a specific computer includes having another computer execute part or all of the process and obtaining the results. That is, the information processing method and program according to the present invention can also be applied to a cloud computing configuration in which a single function is shared and processed jointly by a plurality of devices via a network.

[0207] The character recognition system, operator terminal, server terminal, photographing device, cropped image, recognition result image, each configuration of each GUI, acquisition of object image, character recognition, generation of cropped image, classification of cropped image, display control of cropped image, generation of recognition result image, acceptance of input for selection and correction, correction process, and other processing flows described with reference to each drawing are merely one embodiment and can be arbitrarily modified without departing from the spirit of the present invention. In other words, any other configuration, algorithm, etc. for implementing the present invention may be adopted.

[0208] In this disclosure, to facilitate understanding of the explanation, words such as "approximately," "almost," and "roughly" may be used as appropriate. However, there is no clear difference between using and not using words such as "approximately," "almost," and "roughly." In other words, in the present disclosure, concepts that define shape, size, positional relationship, state, etc., such as "center," "middle," "uniform," and "equal" are concepts that include "substantially center," "substantially central," "substantially uniform," "substantially equal," etc. For example, this also includes a state where the value is within a predetermined range (for example, a range of ±10%) based on the criteria of "perfectly centered," "perfectly central," "perfectly uniform," "perfectly equal," etc. Therefore, even if the words "roughly," "almost," "approximately," etc. are not added, the concept expressed by adding "roughly," "almost," "approximately," etc. may be included. Conversely, a state expressed by adding "roughly," "almost," "approximately," etc. does not necessarily exclude a complete state.

[0209] In this disclosure, expressions using "more than," such as "greater than A" and "smaller than A," are expressions that comprehensively include both concepts that include equivalent to A and concepts that do not include equivalent to A. For example, "greater than A" is not limited to cases that do not include equivalent to A, but also includes "A or greater." Furthermore, "smaller than A" is not limited to "less than A," but also includes "A or less." When implementing the present technology, specific settings and the like may be appropriately adopted from the concepts included in "greater than A" and "smaller than A" so as to achieve the effects described above.

[0210] It is also possible to combine at least two of the features of the present invention described above. That is, the various features described in each embodiment may be combined in any way without distinguishing between the embodiments. Furthermore, the various effects described above are merely examples and are not limiting, and other effects may also be achieved. [Explanation of symbols]

[0211] 1. Character recognition system 2...Operator terminal 3. Server equipment 4...Operator 6. Network 7...Time card 20...Photo 21...Print area 25...Recognition result image 26...Time card image 30...Cutout image 32...Area corresponding to the selected cut-out image in the recognition result image 33...Area corresponding to the selected cut-out image in the captured image 60...Computer

Claims

1. an image acquisition unit that acquires an object image of an object on which characters are displayed; a character recognition unit that recognizes the characters based on the acquired object image; a cutout image generation unit that generates a cutout image cut out from the object image for each of the recognized characters; an image classification unit that classifies the generated cut-out images into groups of identically recognized images based on the recognition result by the character recognition unit; a display control unit that displays, for each recognition result, cut-out images that are included in the group of identical recognition result images and result in the same recognition result; An information processing system comprising:

2. 2. The information processing system according to claim 1, The display control unit displays the cut-out images that result in the same recognition result so that they can be compared with each other. Information processing system.

3. 2. The information processing system according to claim 1, The display control unit displays the cutout images that result in the same recognition result in an aligned manner, or switches the images in order at predetermined time intervals. Information processing system.

4. 4. The information processing system according to claim 3, The display control unit displays the cutout images that result in the same recognition result by arranging them in an order based on the feature amount of the cutout images, or by switching between the orders based on the feature amount of the cutout images. Information processing system.

5. 4. The information processing system according to claim 3, The display control unit displays the cutout images that result in the same recognition result by arranging them in order of similarity, or by switching between the orders of similarity. Information processing system.

6. 4. The information processing system according to claim 3, The display control unit arranges and displays the cut-out images that result in the same recognition result in an order based on the reliability of the recognition result, or switches and displays the cut-out images in an order based on the reliability of the recognition result. Information processing system.

7. The information processing system according to claim 1, further comprising: an input receiving unit that receives an input of a selection for the displayed cut-out image; and a recognition result image generating unit that generates a recognition result image including a recognition result by the character recognition unit, When an input of a selection for the cut-out image is accepted, the display control unit highlights and displays at least one of a region of the object image corresponding to the selected cut-out image and a region of the recognition result image corresponding to the selected cut-out image. Information processing system.

8. 8. The information processing system according to claim 7, the input receiving unit receives an input of a correction to a character included in the area displayed in a highlighted manner in the recognition result image, The information processing system further includes a storage unit that stores the input corrected characters. Information processing system.

9. 8. The information processing system according to claim 7, the display control unit switches and displays the cut-out images that result in the same recognition result in order at a predetermined time interval; The input receiving unit receives an input for stopping the switching display while a predetermined cut-out image is being displayed as an input for selecting the displayed predetermined cut-out image. Information processing system.

10. The information processing system according to claim 1, further comprising: a recognition result image generation unit that generates a recognition result image including a recognition result by the character recognition unit; The display control unit displays the object image and the recognition result image so as to be comparable. Information processing device.

11. 2. The information processing device according to claim 1, The display control unit displays a GUI (Graphical User Interface) for correcting the recognition result obtained by the character recognition unit. Information processing device.

12. 2. The information processing system according to claim 1, The characters are numbers from 0 to 9 Information processing system.

13. Acquire an object image of the object on which the text is displayed; Recognizing the character based on the acquired object image; generating a cut-out image cut out for each of the recognized characters from the object image; The generated cut-out images are classified into groups of identical recognition result images that have the same recognition result based on the character recognition result; The cut-out images that are included in the group of identical recognition result images and result in the same recognition result are displayed for each recognition result. An information processing method implemented by a computer system.

14. Acquire an object image of the object on which the text is displayed; Recognizing the character based on the acquired object image; generating a cut-out image cut out for each of the recognized characters from the object image; The generated cut-out images are classified into groups of identical recognition result images that have the same recognition result based on the character recognition result; The cut-out images that are included in the group of identical recognition result images and result in the same recognition result are displayed for each recognition result. A program that causes a computer system to do something.

15. an image acquisition unit that acquires an object image of an object on which characters are displayed; a character recognition unit that recognizes the characters based on the acquired object image; a storage unit that stores a target character string to be focused; a cut-out image generating unit that generates a cut-out image cut out from the object image for each corresponding character string that corresponds to the stored target character string of interest based on a recognition result by the character recognition unit; an image classification unit that classifies the generated cut-out images into groups of identical character string images that have the same target character string; a display control unit that displays the cut-out images included in the same character string image group for each of the same target character strings; An information processing system comprising:

16. Acquire an object image of the object on which the text is displayed; Recognizing the character based on the acquired object image; memorizing a target character string to be focused on; generating a cut-out image cut out for each corresponding character string corresponding to the stored target character string from the object image based on the character recognition result; The generated cut-out images are classified into groups of identical character string images each having the same target character string; The cut-out images included in the same character string image group are displayed for each of the same target character strings. An information processing method implemented by a computer system.

17. Acquire an object image of the object on which the text is displayed; Recognizing the character based on the acquired object image; memorizing a target character string to be focused on; generating a cut-out image cut out for each corresponding character string corresponding to the stored target character string from the object image based on the character recognition result; The generated cut-out images are classified into groups of identical character string images each having the same target character string; The cut-out images included in the same character string image group are displayed for each of the same target character strings. A program that causes a computer system to do something.

Images