Image forming apparatus and character recognition method

Abstract

To provide an image forming apparatus that is easily used by a user who is not skilled in the operation of an information processing device and mainly writes a document by hand, and can print the document in an easy-to-read typeface, and a character recognition method.SOLUTION: A multifunction printer 100 as an image forming apparatus comprises: an instruction receiving unit 52 that receives an instruction from a user including designation of the type of a handwritten document to be read; an image reading unit 13 that reads an image of the document; a character recognition unit 35 that extracts one or more character areas from the read image, and performs character recognition by using a dictionary for character recognition; a display unit 51 that displays a result of the character recognition for checking; and a printing unit 14 that performs printing of the document by using a font corresponding to the characters for which the character recognition is performed. The character recognition unit 35 performs character recognition for each of the character areas by using a dictionary for character recognition according to the type of the document.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an image forming apparatus having a function for reading an image of a handwritten document and a character recognition function for performing character recognition on the read image, and a method for character recognition of a handwritten document. [Background technology]

[0002] Image forming devices installed in public places such as convenience stores are used by a wide range of users, including those who have difficulty writing legibly due to insufficient muscle strength due to aging or other reasons. In recent years, information processing devices such as personal computers have been proposed that are easy to operate regardless of age or disability. For example, as an information processing device that inputs handwritten character data and performs character recognition, one has been proposed that inputs voice data for directly or indirectly identifying the handwritten characters around the time the handwritten data is input. The output priority of multiple character recognition candidates obtained by character recognition of the handwritten character data is determined with reference to the voice data. This aims to improve the character recognition rate (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 8-221408 Summary of the Invention [Problem to be solved by the invention]

[0004] Users who are skilled in operating information processing devices can create documents using information processing devices instead of handwritten documents, but some elderly users are not proficient in operating information processing devices. The present disclosure has been made in consideration of the above circumstances, and provides a method for printing in a typeface that is easy to use and read, even for users who are not familiar with operating information processing devices and who mainly write handwritten documents. [Means for solving the problem]

[0005] The present disclosure provides an image forming device comprising an instruction receiving unit that receives instructions from a user including a specification of the type of handwritten document to be read, an image reading unit that reads an image of the document, a character recognition unit that extracts one or more character areas from the read image and performs character recognition using a character recognition dictionary, a display unit that displays the results of the character recognition for confirmation, and a printing unit that prints the document using a font corresponding to the recognized characters, wherein the character recognition unit performs the character recognition for each character area using a character recognition dictionary corresponding to the type of the document.

[0006] Also, from a different perspective, the present disclosure provides a character recognition method for a document manuscript, comprising the steps of: a control unit of an image forming apparatus receiving instructions from a user including specification of the type of handwritten document manuscript to be read; a step of reading the document manuscript using an image reading unit; a character recognition step of extracting one or more character areas from the read image and performing character recognition using a character recognition dictionary; a step of displaying the results of the character recognition for confirmation; and a step of printing the document manuscript using a font corresponding to the recognized characters using a printing unit, wherein the character recognition step performs the character recognition for each character area using a character recognition dictionary corresponding to the type of the document manuscript. [Effects of the Invention]

[0007] The image forming device according to the present disclosure comprises a character recognition unit, a display unit that displays the results of the character recognition for confirmation, and a printing unit that prints the document manuscript using a font corresponding to the recognized characters. The character recognition unit performs the character recognition using a character recognition dictionary that corresponds to the type of document manuscript, making it easy to use even for users who are not familiar with operating information processing devices and who mainly write handwritten documents, and it is possible to print in a typeface that is easy to read. The character recognition method according to the present disclosure also achieves the same effects. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a block diagram showing a configuration of a multifunction peripheral shown as an example of an image forming apparatus in the present disclosure. [Figure 2] 2 is a perspective view of the appearance of a portion corresponding to a document reading unit in the multifunction peripheral shown in FIG. 1. FIG. [Figure 3] 1. FIG. 4 is an explanatory diagram showing an example of a first operation screen displayed when a handwritten character recognition+print job is executed in the multifunction peripheral shown in FIG. [Figure 4] 1. FIG. 4 is an explanatory diagram showing an example of a second operation screen displayed when a handwritten character recognition+print job is executed in the multifunction peripheral shown in FIG. [Figure 5] 1. FIG. 4 is an explanatory diagram showing an example of a third operation screen displayed when a handwritten character recognition+print job is executed in the multifunction peripheral shown in FIG. [Figure 6] 1. FIG. 10 is an explanatory diagram showing an example of a fourth operation screen displayed when a handwritten character recognition+print job is executed in the multifunction peripheral shown in FIG. [Figure 7] 1. FIG. 11 is an explanatory diagram showing an example of a fifth operation screen displayed when a handwritten character recognition+print job is executed in the multifunction peripheral shown in FIG. [Figure 8] 1. FIG. 11 is an explanatory diagram showing an example of a sixth operation screen displayed when a handwritten character recognition+print job is executed in the multifunction peripheral shown in FIG. [Figure 9] 1. FIG. 11 is an explanatory diagram showing an example of a seventh operation screen displayed when a handwritten character recognition+print job is executed in the multifunction peripheral shown in FIG. [Figure 10] 10 is a first flowchart showing an example of handwritten character recognition+print job processing executed by a control unit in the multifunction peripheral shown in FIG. [Figure 11] 10 is a second flowchart showing an example of the handwritten character recognition+print job processing executed by the control unit in the multifunction peripheral shown in FIG. [Figure 12]1. FIG. 4 is an explanatory diagram showing an example of a first operation screen displayed when printing a format for handwritten character recognition is executed in the multifunction peripheral shown in FIG. [Figure 13] 1. FIG. 4 is an explanatory diagram showing an example of a second operation screen displayed when printing a format for handwritten character recognition is executed in the multifunction peripheral shown in FIG. [Figure 14] 4 is a flowchart showing an example of processing related to printing a handwritten character recognition format executed by a control unit in the multifunction peripheral shown in FIG. 1; [Figure 15] 2 is an explanatory diagram showing an example of an operation screen that displays candidates for character recognition in the multifunction peripheral shown in FIG. 1. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0009] The present disclosure will be described in further detail below with reference to the accompanying drawings. Note that the following description is illustrative in all respects and should not be construed as limiting the present disclosure. (Embodiment 1) First, an image forming apparatus according to the present disclosure will be described. Fig. 1 is a block diagram showing the configuration of a multifunction peripheral shown as an example of an image forming apparatus according to the present disclosure. Fig. 2 is an external perspective view of a portion of the multifunction peripheral shown in Fig. 1 that corresponds to a document reading unit.

[0010] As shown in FIG. 1, the multifunction peripheral 100 includes a control unit 12, an image reading unit 13, a printing unit 14, an operation unit 15, and a storage unit 18. The control unit 12 is mainly configured with a processor and a memory as hardware resources, and is a circuit configured with hardware resources such as an input / output interface circuit, a timer circuit, and a communication circuit in addition to the processor. The processor executes a control program stored in the memory to realize processes related to operation recognition and display control. It also realizes jobs related to document scanning and image printing, i.e., processes related to a series of image processing. In other words, the software resources and hardware resources work together to realize the function of the control unit 12 that controls the multifunction peripheral 100.

[0011] The control unit 12 according to the present disclosure includes an operation control unit 21, a file control unit 22, a job control unit 24, and an image processing unit 25. It also optionally includes a format providing unit 23, a voice recognition unit 27, and a voice synthesis unit 28. The image processing unit 25 includes a reading control unit 33, a print control unit 34, and a character recognition unit 35, and optionally includes a layout determination unit 36. The operation unit 15 according to the present disclosure includes a display unit 51 and an instruction receiving unit 52. Moreover, the storage unit 18 according to the present disclosure includes a dictionary storage unit 81 and, optionally, a format storage unit 82 .

[0012] The operation control unit 21 provided in the control unit 12 is connected to the operation unit 15, recognizes user operations received by the instruction receiving unit 52 of the operation unit 15, and generates display data to be displayed on the display unit 51 of the operation unit 15. As shown in FIG. 2, the operation unit 15 is provided on the front side of the multifunction peripheral 100. The operation unit 15 has a display device (display unit 51) that displays the operation screen generated by the operation control unit 21. It also has an operation detection device (instruction receiving unit 52) ​​that receives instructions from the user on the operation screen. For example, a liquid crystal display device or an OLED (Organic Light Emitting Display) can be used as the display device, and a touch panel can be used as the operation detection device. The operation control unit 21 communicates with the above-mentioned operation detection device and recognizes the user's operation detected by the operation detection device, i.e., the instruction from the user. It also transmits display data to the above-mentioned display device to display it as the operation screen. The file control unit 22 included in the control unit 12 manages the data stored in the storage unit 18 and controls reading and writing.

[0013] The image processing unit 25 included in the control unit 12 generates image data of the document based on the signal of the scanned image output from the image reading unit 13. In the case of a scan job, the generated image data of the document is stored in the memory unit 18 or transmitted to an external device via a communication circuit (not shown). The reading control unit 33 controls the processes related to the reading of the document. In the case of a copy job, the image data of the document is sent to the printing unit 14 and printed. In the case of a print job, image data for printing is generated based on job data received from an external device via the communication circuit, and printed by the printing unit 14. The printing control unit 34 controls the processes related to the printing. The character recognition unit 35 recognizes characters included in the image data of the document. Preferably, the layout determination unit 36 ​​determines the layout pattern of character areas included in the image data of the document. The character recognition unit 35 and the layout determination unit 36 ​​will be described later.

[0014] The job control unit 24 provided in the control unit 12 performs the corresponding processing when an instruction from a user recognized by the operation control unit 21 or an instruction from an external device serving as a print host via the communication circuit relates to job setting or execution. For example, if the instruction relates to job setting, the target setting is updated. If the instruction involves updating the operation screen, the operation control unit 21 generates display data for the updated operation screen and sends it to the display device of the operation unit 15. This updates the operation screen. When an instruction to execute a job is received, the job control unit 24 controls the image reading unit 13, printing unit 14, and image processing unit 25 according to the type of job received (e.g., scan job, copy job, print job), controls the flow of image data, and executes the instructed job. More specifically, the job control unit 24 recognizes the states detected by sensors (not shown) arranged in the image reading unit 13 and the printing unit 14. Then, the job control unit 24 controls the operations of motors, actuators, devices (not shown) arranged in the image reading unit 13 and the printing unit 14. The job control unit 24 also controls the image-related processing of the image processing unit 25.

[0015] The storage unit 18 includes either or both of a nonvolatile memory such as a hard disk drive (HDD) or a solid state drive (SDD) and a volatile memory such as a dynamic random access memory (DRAM) that stores image data. Although the storage unit 18 is shown as a block separate from the control unit 12 in FIG. 1, it may alternatively be included in the control unit 12. That is, the storage unit 18 may alternatively be mounted on the same circuit board as the control unit 12. The dictionary storage unit 81 of the memory unit 18 stores dictionaries according to the type of document, which are used by the character recognition unit 35 when performing character recognition. The optional format storage unit 82 stores print data in a predetermined format. Furthermore, if the image processing unit 25 includes the layout determination unit 36, the format storage unit 82 may store a database in which various layout patterns are registered. The layout determination unit 36 ​​may refer to the database in which these layout patterns are stored when determining a layout pattern according to the type of document.

[0016] The image reading unit 13 reads an image of an original document under the control of the job control unit 24 and converts it into an RGB (red, green, and blue) color image signal (RGB analog signal). That is, it executes image reading processing for copy jobs and scan jobs. The original document to be read is set by the user on the document feeder 31 or document table 32. The document feeder 31 transports the original document one sheet at a time to a predetermined reading position. The document table 32 is made of a flat, transparent glass member. The image reading unit 13 includes an image sensor (not shown in FIGS. 1 and 2). It also includes a scanning mechanism (not shown in FIGS. 1 and 2) for scanning the original document placed on the document table 32. The image processing unit 25 generates image data of the document based on the signal of the image read by the image sensor. It also generates image data for printing based on print data received from an external device via the communication circuit. The generated image data is sent to the printing unit 14 and printed, or transmitted to the external device via the communication circuit.

[0017] The printing unit 14 feeds print sheets one by one from one of the paper feed trays 41A, 41B, and 41C that accommodate print sheets, prints an image on the print sheet, and outputs it to an output tray. As shown in Figures 2 and 3, the multifunction device 100 outputs printed print sheets to one of the three output trays 42A, 42B, and 42C.

[0018] <Handwriting recognition + print job> Next, we will explain the processing flow for handwritten character recognition and print jobs. 3 to 9 are explanatory diagrams showing examples of operation screens displayed on the operation unit 15 when a handwriting recognition+print job is executed. Also, Fig. 10 and Fig. 11 are flowcharts showing examples of processing of a handwriting recognition+print job executed by the control unit 12. The following description will be made with reference to the flowchart and the operation screen. The control unit 12 displays the operation screen 130 (see FIG. 3) on the operation unit 15, which accepts the selection of a job to be executed by the multifunction peripheral 100, and waits for an instruction from the user (step S11 shown in FIG. 10).

[0019] When the [Handwriting recognition + printing] key 131 is touched on the operation screen 130, the control unit 12 responds to the touch by switching the operation screen to an operation screen 140 (see FIG. 3) for selecting the type of document (step S13 in FIG. 10). The control unit 12 displays a message 141 saying "Please select the type of document" along with multiple document type selection keys 142 on the operation screen 140. Preferably, the control unit 12 also causes the voice synthesis unit 28 to present the content of the message 141 to the user by voice. For other operation screens described below, the content of the displayed messages is also preferably presented to the user by voice. The document types displayed on the operation screen 140 are postcards and New Year's cards. Furthermore, a [Resume, Letter Paper] key 143 is displayed in order to accept the selection of a document type not displayed on the operation screen 140 (see FIG. 4). When the [Resume, Letter Paper] key 143 is touched, the control unit 12 switches the operation screen to an operation screen 150 (see FIG. 5) for selecting another document type in response to the touch.

[0020] On the operation screen 150, the control unit 12 displays a message 151 saying "Please select the type of document," along with a plurality of document type selection keys 152. The document types displayed on the operation screen 150 are letter paper and resume. Furthermore, a [Postcard, New Year's card] key 153 is displayed in case the user returns to the operation screen 140 to accept the selection of the document type (see FIG. 5). When either the document type selection key 142 or 152 in FIG. 4 or FIG. 5 is touched (step S15 in FIG. 10), in response to the operation, the control unit 12 switches the operation screen to the operation screen 160 (see FIG. 6) that instructs the reading of the document (step S17 in FIG. 10). On the operation screen 160, the control unit 12 displays a message 161 saying "Place the original and press 'Read Original'," along with an original set illustration 162 showing the user how to place the original. Furthermore, it displays a [Read Original] key 163 that instructs the user to read the original (see FIG. 6).

[0021] When a handwritten document is set according to the document set illustration 162 and the [Document Read] key 163 is touched (step S19 in FIG. 10), the control unit 12 responds to this operation by causing the image reading unit 13 to read the document (step S21 in FIG. 10). Preferably, the control unit 12 then references a database stored in the format storage unit 82 that contains layout patterns commonly used for each document type, depending on the selected document type, and estimates the layout of the character area in the read handwritten document (step S23 in FIG. 10). The application of such layout patterns is particularly effective for documents that contain a mixture of vertical and horizontal writing, characters of different sizes, and columns. Control unit 12 refers to the character recognition dictionary stored in dictionary storage unit 81 according to the selected document type and performs character recognition for each character area (step S25 in FIG. 10). Compared to applying the same character recognition dictionary regardless of document type, using a character recognition dictionary according to the document type increases the priority of words and phrases frequently used in the selected document type, enabling highly accurate character recognition. Then, the operation screen is switched to operation screen 170 (see FIG. 7) (step S27 in FIG. 10).

[0022] The control unit 12 displays an image of the scanned handwritten manuscript as a scanned image preview 172 on the operation screen 170, along with a message 171 saying, "Please read the following sentence aloud." Preferably, the control unit 12 requests the user to read each character area aloud, in units of one or more character areas extracted from the handwritten manuscript according to the layout pattern of the manuscript determined by the layout determination unit 36. The control unit 12 indicates the character area to be read aloud with a rectangular framed reading focus 173, indicating which part the user should read aloud. Furthermore, the control unit 12 displays a microphone icon 174, and changes its color, shade, or size to indicate that the reading performed by the user has been recognized by voice.

[0023] The control unit 12 may request reading of all character areas, or may request reading of only character areas whose reliability (character recognition accuracy) in the character recognition process is below a predetermined threshold. The reliability may be determined, for example, based on the degree of similarity between handwritten characters on the document and a character recognition dictionary. By requesting reading of only character areas whose reliability is below a threshold, the time required for reading can be reduced compared to when reading of all character areas. Here, character regions may be determined not simply based on layout, but also based on whether the results of character recognition for each character region result in a phrase with reasonable meaning. If a phrase with reasonable meaning is not obtained, it is preferable to shift the boundaries between adjacent character regions and perform character recognition to determine character regions in units that make more sense. The character recognition dictionary may be one that performs character recognition by referring not only to the shape of each character but also to the shape of a phrase consisting of multiple characters.

[0024] Preferably, the control unit 12 moves the reading focus 173 and asks the user to read each character area. More specifically, when the target character area indicated by the reading focus 173 is read aloud by the user (step S31 shown in FIG. 11), the control unit 12 causes the voice recognition unit 27 to recognize the voice. Then, the control unit 12 performs a transcription process corresponding to the reading voice (step S33). The result of the transcription from the reading voice is then compared with candidates obtained by performing character recognition processing by the character recognition unit 35. Preferably, the control unit 12 replaces the target character area indicated by the reading focus 173 with characters determined as a result of character recognition and voice recognition. When replacing, the control unit 12 replaces the character area indicated by the reading focus 173 with font characters.

[0025] In this way, each character region is replaced with a recognition result based on character recognition by the character recognition unit 35 and voice recognition by the voice recognition unit 27. The character recognition unit 35 may obtain multiple candidates as a result of character recognition. Preferably, the control unit 12 does not determine the character recognition result or narrow down the candidates based solely on the degree of similarity between the characters or phrases to be recognized and the character recognition dictionary, but determines the character recognition result or narrows down the candidates by taking into account the recognition result of the voice read by the voice recognition unit 27 (step S35).

[0026] When the character recognition process for one character area is completed, the control unit 12 determines whether there are any remaining character areas to be read aloud (step S37). If there are any remaining character areas to be read aloud (No in step S37), the control unit 12 moves the read-aloud focus shown in Fig. 7 to the next target (step S39). Then, the process returns to step S31 and waits for the target character area to be read aloud. On the other hand, if there are no remaining character regions to be read out (Yes in step S37), the control unit 12 switches the operation screen to operation screen 180 (see FIG. 8) (step S41 in FIG. 11). In this way, by preferably using voice recognition in combination, it becomes easy to use even for users who are not good at operating information processing devices, and it becomes possible to obtain character recognition results with higher accuracy.

[0027] Alternatively, instead of or in addition to the mode in which control unit 12 requests reading aloud while moving reading focus 173, the reading-aloud process may be performed as follows: When the user reads aloud the phrase he or she wishes to correct, speech recognition unit 27 recognizes the reading, determines the corresponding character area, and performs character recognition of the character area taking the reading aloud into consideration. In this case, control unit 12 may move reading focus 173 to the character area determined to be the target of character recognition, and then display the results of the character recognition.

[0028] Control unit 12 displays a message 181 saying "Please check the layout" on operation screen 180, along with a preview image of the confirmed character recognition result as character recognition preview 182. Furthermore, control unit 12 displays a "Change layout" key 183 and a "Print" key 184. When the [Print] key 184 is touched on the operation screen 180 (Yes in step S43), the control unit 12 replaces the handwritten characters in the handwritten document with the recognized font, executes printing (step S51), and ends the job. On the other hand, when the [Change Layout] key 183 is touched on the operation screen 180 (No in step S43 and Yes in step S45), in response to the operation, the control unit 12 switches the operation screen to an operation screen 190 (see FIG. 9) for selecting another layout (step S53 in FIG. 11).

[0029] The control unit 12 displays a message 191 saying "Please select a layout" along with a layout selection key 192 on the operation screen 190. Furthermore, a [To another layout] key 193 is displayed for accepting the selection of a layout not on the operation screen 190. When any of the layout selection keys 192 is touched, the control unit 12 responds to the operation by switching the operation screen to a screen similar to the operation screen 180 shown in FIG. 8 (via "Yes" in step S55 of FIG. 11 and then to step S41 of the same). However, the character recognition preview 182 in this case corresponds to the selected layout. The character recognition results for each character region are not changed and only the layout is changed. The above is the procedure by which the multifunction peripheral 100 in this embodiment reads a handwritten document, performs character recognition, and prints the document using a font corresponding to the recognized characters.

[0030] (Embodiment 2) In the first embodiment, document manuscripts with relatively free layouts, such as postcards or letter paper, are assumed. In contrast, there are also cases where characters are handwritten on documents with a predetermined format, such as various application forms for government services. For documents with a predetermined format, the image forming device can store or reference the format, and the handwritten text on a printed version of the format can be read as the handwritten document manuscript. In this way, instead of the layout determination unit 36 ​​estimating the layout of the manuscript as in the first embodiment, the accurate layout of the manuscript can be determined based on the pre-stored or pre-referenced format. Furthermore, since the content to be entered in each field of the predetermined format is known, highly accurate character recognition is possible by referring to a dictionary not only corresponding to the manuscript type but also corresponding to the content of each field.

[0031] In this embodiment, the format is assumed to be stored in advance in the format storage unit 82 of the storage unit 18. However, the present invention is not limited to this, and the format may be stored, for example, in an external server with which the multifunction device 100 can communicate, and the format may be referenced or downloaded as needed. 12 and 13 are explanatory diagrams showing examples of operation screens displayed when printing a handwriting recognition format in the present disclosure. Also, Fig. 14 is a flowchart showing an example of processing related to printing a handwriting recognition format executed by the control unit. The following description will be given with reference to the flowchart and the operation screen. As in the first embodiment, the control unit 12 causes the operation unit 15 to display the operation screen 130 (see FIG. 3) that accepts the selection of a job to be executed by the multifunction device 100, and the control unit 12 waits for an instruction from the user (step S61 shown in FIG. 14).

[0032] When the "Print format for handwriting recognition" key 132 is touched on the operation screen 130 shown in FIG. 3, the control unit 12 responds to the touch by performing the following series of processes as the format provider 23. First, the operation screen is switched to the operation screen 220 (see FIG. 12) for selecting the format to be printed (step S63 in FIG. 14). The control unit 12 then displays a message 221 saying, "Please select the format of the document to be printed," along with multiple format selection keys 222 on the operation screen 220. The operation screen 220 displays postcard and New Year's card format types. Furthermore, a [Resume, Letter Paper, Other] key 223 is displayed in order to accept the selection of a document type not displayed on the operation screen 140 (see FIG. 12). When the [Resume, Letter Paper, Other] key 223 is touched, the control unit 12 switches to an operation screen (not shown) in response to the touch to select another format. The other formats may include application documents for the aforementioned government services, etc. Other formats may include application document formats for private banks, insurance companies, postal services, courier services, etc. When any of the format selection keys 222 is touched (step S65 in FIG. 14), in response to the touch, the control unit 12 switches the operation screen to the operation screen 230 (see FIG. 13) (step S67 in FIG. 14).

[0033] The control unit 12 displays a message 231 saying "Please check the format of the document to be printed" on the operation screen 230, along with a preview image of the selected format as a format preview 232. Furthermore, a "Select another format" key 233 and a "Print" key 234 are displayed. When the [Print] key 234 is touched on the operation screen 230 (Yes in step S69), the control unit 12 executes printing in the selected format (step S71) and ends the job. On the other hand, if the [Select another format] key 233 is touched on the operation screen 230 (No in step S69 and Yes in step S73), in response to that operation, the control unit 12 switches the operation screen to an operation screen (not shown) for selecting another format (going to step S63 in FIG. 14 via step S73). The above is the procedure by which the multifunction device 100 prints the handwriting recognition form in this embodiment. The user's handwriting on the printed form in this way is read in the same way as in the first embodiment. It is assumed that the handwritten character recognition form has an image attached to it indicating that it is a printed form. Here, a two-dimensional code is used as an example. The image forming device that reads the two-dimensional code refers to the corresponding form and a dictionary corresponding to the document type of the form and each column of the form to perform character recognition. In addition, if auxiliary lines such as frames indicating where characters should be written by hand or notes are printed on the form, the control unit 12 may remove the auxiliary lines and print only the characters that have been recognized.

[0034] (Embodiment 3) In the first embodiment, the character recognition unit 35 determines or narrows down candidates based on the degree of similarity between the handwritten character to be recognized and the character recognition dictionary, preferably by further referring to the recognition result of the read-out voice. In this embodiment, rather than determining or narrowing down the candidates for recognition, the character recognition unit 35 presents the user with a wide range of words and phrases having the same or similar meaning as candidates.

[0035] 15 is an explanatory diagram showing an example of an operation screen in this embodiment on which the control unit 12 performs character recognition and displays candidates, which can be said to correspond to FIG. The control unit 12 displays an image of the scanned handwritten document as a scanned image preview 252 together with a message 251 on an operation screen 250 shown in Fig. 15, and also displays a reading focus 253. Furthermore, it displays a microphone icon 254. These correspond to the operation screen 170, message 171, scanned image preview 172, reading focus 173, and microphone icon 174 shown in Fig. 7, respectively.

[0036] 7, message 251 says, "Please select the content appropriate for the boxed area from the right column." The control unit 12 displays character recognition candidates 255, which are not shown in FIG. 7, to the right of the scanned image preview 252. Character recognition candidates 255 correspond to the "right column" in message 251. Character recognition candidates 255 include three candidates (candidates 1 to 3) assigned unique numbers 1 to 3. Candidate 1 is a verbatim recognition of handwritten characters in the character area indicated by readout focus 253, and is the result of character recognition in the first embodiment. In contrast, candidate 2 is a word or phrase with the same pronunciation and meaning as candidate 1, but with some characters replaced with kanji, i.e., similar characters. It can also be said to be a word or phrase with a similar meaning to candidate 1. Candidate 3 is a word or phrase with a similar meaning to candidate 1. Not limited to this example, control unit 12 displays characters or phrases with the same or similar meaning as candidate 1 as candidates.

[0037] In response to a user's operation of touching any of the candidates 1 to 3, the control unit 12 determines the touched candidate as the result of character recognition and replaces the handwritten characters in the reading focus 253 with the font characters of the selected candidate. A candidate may be selected not only by a touch operation, but also by the user's voice reading out one of the numbers 1 to 3 specific to the candidate. Furthermore, if none of the candidates 1 to 3 is the intended one, the user may read out the intended content, and the control unit 12 may add a new candidate 4 to the character recognition candidates 255 in response to the voice reading out the content. Alternatively, the new candidate may be displayed instead of the candidates 1 to 3. In this way, by displaying at least one of the characters that are identical or similar to the result of character recognition, or the words that are identical or similar to the result, as candidates for character recognition on the display unit, it becomes possible to present a variety of candidates to the user and accept their selection.

[0038] In the present disclosure above, a dictionary is a database in which at least one of characters, words, phrases, and sentences is registered, and is matched with an object to be recognized.

[0039] It should be understood that the present disclosure also includes any combination of the above-described aspects. In addition to the above-described embodiments, various modifications of the present disclosure are possible. These modifications should not be interpreted as not falling within the scope of the present disclosure. The invention according to the present disclosure should include all modifications that have the meaning equivalent to the scope of the claims and that fall within the scope of the present disclosure. [Explanation of symbols]

[0040] 12: Control unit, 13: Image reading unit, 14: Printing unit, 15: Operation unit, 18: Memory unit, 21: Operation control unit, 22: File control unit, 23: Format providing unit, 24: Job control unit, 25: Image processing unit, 27: Voice recognition unit, 28: Voice synthesis unit, 31: Document feeder, 32: Document table, 33: Reading control unit, 34: Printing control unit, 35: Character recognition unit, 36: Layout determination unit, 41A, 41B, 41C: Paper feed tray, 42A, 42B, 42C: Output tray, 51: Display unit, 52: Instruction receiving unit, 81: Dictionary storage unit, 82: Format storage unit 100: Multifunction device, 130,140,150,160,170,180,190,220,230,250: Operation screen, 131: [Handwriting recognition + print] key, 132: [Print format for handwriting recognition] key, 141,151,161,171,181,191,221,231,251: Message, 142,152: Document type selection key, 143: [Resume, letter paper] key, 153: [Postcard, New Year's card] key, 162: Document set illustration, 163: [Document reading] key, 172,252: Scanned image preview, 173,253: Reading focus, 174,254: Microphone icon, 182: Character recognition preview, 183: [Change layout] key, 184: [Print] key, 192: Layout selection key, 193: [To other layouts] key, 222: Format selection key, 223: [To resume, letter paper, etc.] key, 232: Format preview, 233: [Select another format] key, 234: [Print] key, 255: Character recognition candidate

Claims

1. an instruction receiving unit that receives instructions from a user, including a designation of the type of handwritten document to be read; an image reading unit that reads an image of the handwritten document; a layout determination unit that extracts one or more character regions from the read image and determines a layout pattern of words and phrases according to the type of the handwritten document; a character recognition unit that performs character recognition for each word in each character region using a character recognition dictionary that corresponds to the type of the handwritten document; a display unit that displays the result of the character recognition for confirmation; a control unit that prompts a user to read out the character recognition results displayed on the display unit for each character area according to the layout pattern; a voice recognition unit that recognizes the voice read aloud in response to the request; a printing unit that prints the handwritten document using a font corresponding to the recognized characters, The character recognition unit further uses the results of the recognition by the voice recognition unit to correct the character recognition of each word as necessary.

2. a format storage unit for storing in advance a plurality of layout formats for handwritten document manuscripts, each of which has an image indicating that the layout format is a printed format; a format providing unit that receives an instruction from a user to select one of the layout formats via the instruction receiving unit, and causes the printing unit to print the selected layout format and provide it to the user; 2. The image forming apparatus according to claim 1, wherein when a document on which characters are handwritten in a provided layout format is read by the image reading unit, the layout determining unit determines the layout of words and phrases based on the layout format.

3. 2. The image forming apparatus according to claim 1, wherein the character recognition unit calculates the reliability of the character recognition results for each word based on the degree of similarity with the character recognition dictionary, and asks the user to read out the word if the calculated reliability does not meet a predetermined standard.

4. the voice recognition unit recognizes a voice reading out a phrase that the user wants to correct from the character recognition results displayed on the display unit; 2. The image forming apparatus according to claim 1, wherein the character recognition unit corrects the character recognition of the corresponding character region by further using the result of the recognition by the voice recognition unit.

5. 2. The image forming apparatus according to claim 1, wherein the character recognition unit displays, in addition to the character recognition results obtained using the character recognition dictionary, at least one of characters that are synonymous or similar to the results, and synonymous or similar words and phrases as candidates for character recognition on the display unit.

6. A control unit of the image forming apparatus receiving instructions from a user including a designation of the type of handwritten document to be read; and, reading the handwritten document manuscript using an image reading unit; extracting one or more character regions from the read image and determining a layout pattern of words and phrases according to the type of the handwritten document; a character recognition step of performing character recognition for each word or phrase in each character region using a character recognition dictionary corresponding to the type of the handwritten document; displaying the character recognition results for confirmation; a step of asking a user to read out the displayed character recognition results for each character area according to the layout pattern; Recognizing the reading voice in response to the request using a voice recognition unit; further using the results of the speech recognition unit to correct the character recognition of each word as needed; and printing the handwritten document manuscript using a printing unit with a font corresponding to the recognized characters.

Images