Image forming apparatus and learning device

Abstract

To easily restore abnormality of a character string included in an image.SOLUTION: The image forming apparatus 20 forms an image on a recording medium P. The image forming apparatus 20 includes an acquisition unit 201, a storage unit 23, a control unit 21, and a display unit 25. The acquiring unit 201 acquires first data D1 including a first image G1 including at least a character string C. The storage unit 23 stores the learned prediction model M1. When the first data D1 is input, the trained prediction model M1 detects the abnormality df of the first image G1, predicts the first repair candidate rt1 of the abnormality df when the abnormality df is detected, and outputs the second data D2 in which the abnormality df is replaced with the first repair candidate rt1. The control unit 21 inputs the first data D1 to the trained prediction model M1, and the second data D2 is output from the trained prediction model M1. The display unit 25 displays the second image G2 including at least the first repair candidate rt1 based on the second data D2 output from the trained prediction model M1.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image forming apparatus and a learning apparatus. [Background technology]

[0002] The image forming device in Patent Document 1 performs character recognition processing on a scanned document image to obtain character data, and if a missing portion is detected in the character data, it extracts a candidate character or word to correct the missing portion from dictionary data based on pre-stored dictionary data in which multiple characters and multiple words are registered, and corrects the missing portion with the extracted character or word. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-106028 Summary of the Invention [Problem to be solved by the invention]

[0004] The image forming device of Patent Document 1 cannot correct characters and words that are not registered in the dictionary data. In other words, it is necessary to register the characters and words included in the document image in the dictionary data before reading the document image, which is time-consuming.

[0005] The present invention has been made in consideration of the above-mentioned problems, and its purpose is to provide an image forming apparatus and a learning apparatus that can easily repair abnormalities in character strings contained in an image. [Means for solving the problem]

[0006] An image forming apparatus according to the present invention forms an image on a recording medium. The image forming apparatus according to the present invention includes an acquisition unit, a storage unit, a control unit, and a display unit. The acquisition unit acquires first data including a first image that includes at least a character string. The storage unit stores a trained prediction model. When the first data is input, the trained prediction model detects an anomaly in the first image, and when the anomaly is detected, predicts a first repair candidate for the anomaly, and outputs second data in which the anomaly has been replaced with the first repair candidate. The control unit inputs the first data to the trained prediction model, and the trained prediction model outputs the second data. The display unit displays a second image including at least the first repair candidate based on the second data output from the trained prediction model. [Effects of the Invention]

[0007] According to the present invention, it is possible to easily correct an abnormality in a character string included in an image. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a functional block diagram illustrating an image forming apparatus according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram showing an example of a first image corresponding to first data. [Figure 3] FIG. 10 is a diagram showing an example of a second image corresponding to second data. [Figure 4] FIG. 10 shows a second image generated by the trained prediction model. [Figure 5] FIG. 2 is a diagram showing a first image acquired by the image forming apparatus. [Figure 6] FIG. 2 is a diagram showing a second image generated based on the first image. [Figure 7] 4 is a flowchart showing an image forming method in the image forming apparatus according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the drawings, the same or corresponding parts are designated by the same reference characters and description thereof will not be repeated.

[0010] The configuration of an image forming apparatus 20 according to this embodiment will be described with reference to Fig. 1. Fig. 1 is a functional block diagram showing the image forming apparatus 20 according to this embodiment.

[0011] The image forming device 20 forms an image on a recording medium P such as paper based on image data. The image forming device 20 includes a control unit 21, an image reading unit 22, a storage unit 23, an image forming unit 24, a display unit 25, and a reception unit 26.

[0012] The image forming unit 24 forms an image on the recording medium P. The image forming unit 24 forms an image on the recording medium P by, for example, a so-called "electrophotographic method."

[0013] The image reading unit 22 reads an image. The image reading unit 22 includes, for example, a CIS (Contact Image Sensor) unit, and the CIS unit reads the image. The image reading unit 22 generates first data D1 indicating the read image and outputs the first data D1 to the control unit 21.

[0014] The display unit 25 is, for example, a liquid crystal display device, and displays an operation input screen. The operation input screen is a screen for inputting various operations. The reception unit 26 receives operation inputs from the user. For example, the display unit 25 and the reception unit 26 are integrally formed as a touch panel. In this case, the reception unit 26 is a touch sensor, and detects a touch position on the display unit 25. The touch position is the position where the user touches the screen of the display unit 25. Therefore, the user can input operations by touching the operation input screen displayed on the display unit 25. Note that the reception unit 26 is not limited to a touch panel, and may be a physical key provided separately from the display unit 25.

[0015] The control unit 21 includes a processor such as a CPU (Central Processing Unit). The storage unit 23 includes a storage device such as a semiconductor memory or an HDD (Hard Disk Drive). The storage unit 23 stores various data and control programs. The various data includes image data. The control unit 21 controls the image reading unit 22, the image forming unit 24, the display unit 25, and the reception unit 26.

[0016] The control unit 21 also includes an acquisition unit 201 and a generation unit 202. Specifically, the processor of the control unit 21 functions as the acquisition unit 201 and the generation unit 202 by executing a control program in the storage unit 23.

[0017] The acquiring unit 201 receives and acquires the first data D1 output from the image reading unit 22. Note that the first data D1 may be generated by a terminal device such as a PC connected to an external network via a communication port, a wireless module, or the like (not shown), in addition to being generated by the image reading unit 22. In this case, the acquiring unit 201 communicates with the terminal device via the communication port, the wireless module, or the like (not shown), and receives the first data D1 from the terminal device.

[0018] In this embodiment, the storage unit 23 stores a trained prediction model M1. The trained prediction model M1 detects an abnormality df in the first data D1 as a result of machine learning. When the trained prediction model M1 detects an abnormality df, it predicts a first repair candidate rt1 for the abnormality df and outputs second data D2 in which the abnormality df has been replaced with the first repair candidate rt1.

[0019] Next, processing of the trained prediction model M1 will be described with reference to Figures 2 and 3. Figure 2 is a diagram showing an example of a first image G1 corresponding to first data D1. Figure 3 is a diagram showing an example of a second image G2 corresponding to second data D2.

[0020] 2, the first image G1 includes at least a character string C. The first image G1 includes a plurality of character strings C. As a result of machine learning, the trained prediction model M1 is able to identify the character string C on a word-by-word basis.

[0021] For example, among the character string C included in the first image G1, the character string C1 "forward" is divided into a portion p1 and a portion p2. In other words, the character string C1 is a character string including an abnormal df divided into multiple portions. As a result of machine learning, the trained prediction model M1 can detect the character string C1 included in the first image G1 as an abnormal df.

[0022] The control unit 21 inputs the first data D1 acquired by the acquisition unit 201 to the trained prediction model M1. When the trained prediction model M1 detects a character string C1 from the input first data D1, it predicts a character string C2 (FIG. 3) in which the portions p1 and p2 are integrated as a first repair candidate rt1. The trained prediction model M1 replaces the character string C1 with the predicted character string C2 in the first image G1. Similar to the replacement of the character string C1 with the character string C2, the trained prediction model M1 replaces the character string C divided into multiple parts with a character string in which the multiple parts are integrated to generate a second image G2 (FIG. 3).

[0023] In this way, by using the trained prediction model M1, the character string C1 divided into multiple parts can be easily repaired by replacing it with a single character string C2.

[0024] The trained prediction model M1 outputs second data D2 indicating the generated second image G2 to the control unit 21.

[0025] When the control unit 21 receives the second data D2 output from the trained prediction model M1, the control unit 21 controls the display unit 25 to display the second image G2. The control unit 21 also controls the image forming unit 24 to form the second image G2 on the recording medium P.

[0026] By making it possible to replace the abnormal df with the first repair candidate rt1 based on machine learning, the abnormal df of the character string C included in the first image G1 can be easily repaired. For example, before the image forming unit 24 forms an image on the recording medium P, the second image G2, which is the first image G1 with the repaired abnormal df, can be displayed on the display unit 25, allowing the user of the image forming device 20 to visually recognize the second image G2.

[0027] Next, the first repair candidate rt1 will be further described with reference to Fig. 4. Fig. 4 is a diagram showing a second image G21 generated by the trained prediction model M1. In the second image G21 shown in Fig. 4, the character string C containing the abnormal df has been replaced with the character string C21 "internship," which is the first repair candidate rt1.

[0028] In this embodiment, the trained prediction model M1 predicts a plurality of repair candidates rt for an abnormality df, and replaces the abnormality df with one of the plurality of repair candidates rt as a first repair candidate rt1.

[0029] Specifically, for a string C containing an abnormal df, the trained prediction model M1 predicts three repair candidates rt in addition to the string C21: the string C22 “apprenticeship,” the string C23 “externship,” and the string C24 “on-the-job training.” The trained prediction model M1 selects one of the strings C21, C22, C23, and C24 as a first repair candidate rt1 and replaces it with the string C containing the abnormal df.

[0030] The trained prediction model M1 outputs the second data D2 including the character strings C22, C23, and C24 other than the character string C21 selected as the first repair candidate rt1.

[0031] When the control unit 21 receives the second data D2 output from the trained prediction model M1, the control unit 21 controls the display unit 25 to display images corresponding to the character strings C22, C23, and C24, respectively, together with the second image G21.

[0032] By predicting a plurality of repair candidates rt and displaying them on the display unit 25, the user of the image forming apparatus 20 can be provided with options other than the first repair candidate rt1.

[0033] In this embodiment, after outputting the second data D2, it is possible to further replace the first repair candidate rt1. For example, the receiving unit 26 (FIG. 1) receives a selection operation to select one of the repair candidates rt other than the first repair candidate rt1. Specifically, the receiving unit 26 (FIG. 1) detects a touch position on one of the images corresponding to the character strings C22, C23, and C24.

[0034] When the receiving unit 26 (FIG. 1) receives the selection operation, the control unit 21 (FIG. 1) replaces the first repair candidate rt1 in the second data D2 with the repair candidate rt selected by the selection operation. As a result, the control unit 21 (FIG. 1) generates third data D3 representing the second image G21 in which the first repair candidate rt1 has been replaced with the other repair candidate rt.

[0035] In this way, by selecting the first restoration candidate rt1 from the multiple restoration candidates rt, it becomes easier to restore the second image G21 as expected by the user.

[0036] Based on the generated third data D3, the control unit 21 controls the display unit 25 to display images corresponding to the character strings C22, C23, and C24, respectively, along with the second image G21 in which the first repair candidate rt1 has been replaced with another repair candidate rt.

[0037] The repair candidates rt may be generated by an input operation to the reception unit 26, in addition to being predicted by the trained prediction model M1.

[0038] In this embodiment, the trained prediction model M1 may be updated by performing machine learning on the first data D1 and the third data D3 in association with each other.

[0039] Specifically, the generation unit 202 performs machine learning by associating the first data D1 with the third data D3. For example, the storage unit 23 stores a plurality of first data D1 previously input to the image forming apparatus 20 and a plurality of past second data D2 corresponding to the plurality of past first data D1. The generation unit 202 generates a first data set by adding new first data D1 to the plurality of past first data D1, and a second data set by adding third data D3 to the plurality of past second data D2. Each of the first data D1 in the first data set is an example of first data for learning. Each of the second data D2 and third data D3 in the second data set is an example of second data for learning.

[0040] The machine learning algorithm is not particularly limited as long as it is supervised learning, and may be, for example, a decision tree, a nearest neighbor method, a naive Bayes classifier, a support vector machine, or a neural network. Therefore, the trained prediction model M1 includes a decision tree, a nearest neighbor method, a naive Bayes classifier, a support vector machine, or a neural network. In the machine learning for generating the trained prediction model M1, backpropagation may be used.

[0041] In this embodiment, the machine learning algorithm that generates the trained prediction model M1 is a neural network. That is, the trained prediction model M1 includes a neural network. The neural network includes an input layer, one or more intermediate layers, and an output layer. Preferably, the neural network is a deep neural network (DNN), a recurrent neural network (RNN), or a convolutional neural network (CNN), and performs deep learning.

[0042] A deep neural network includes an input layer, multiple hidden layers, and an output layer. A convolutional neural network includes an input layer, multiple convolutional layers, multiple pooling layers, a fully connected layer, and an output layer. In a convolutional neural network, convolutional layers and pooling layers alternate between the input layer and the fully connected layer.

[0043] It is particularly preferable that the machine learning algorithm that generates the trained prediction model M1 is a convolutional neural network, that is, it is particularly preferable that the trained prediction model M1 includes a convolutional neural network.

[0044] The generation unit 202 performs machine learning by inputting the first data set and the second data set into a neural network to generate a trained prediction model M1. Specifically, the discrimination model generation unit 101 performs machine learning by associating each first data D1 in the first data set with the corresponding second data D2 or third data D3 in the second data set.

[0045] When first data D1 is input to the neural network, the generation unit 202 performs machine learning so that, if the input first data D1 includes an abnormality df, second data D2 or third data D3, which is first data D1 in which the abnormality df has been replaced with a repair candidate rt, is output, thereby generating a trained prediction model M1. The generation unit 202 updates the trained prediction model M1 stored in the storage unit 23 to the newly generated trained prediction model M1.

[0046] In this way, by using the third data D3 containing repair candidates rt based on the user's selection for machine learning to update the trained prediction model M1, the prediction accuracy of the repair candidates rt by the trained prediction model M1 can be further improved.

[0047] Next, an abnormal df other than the division of the character string C and the replacement of the abnormal df will be described with reference to Fig. 5 and Fig. 6. Fig. 5 is a diagram showing a first image G12 acquired by the image forming apparatus 20. Fig. 6 is a diagram showing a second image G22 generated based on the first image G12.

[0048] The first image G12 includes, for example, the character string C31 “HENLO,” the character string C32 “Im,” and the character string C33 “barfoo.” The character strings C31, C32, and C33 are each a character string that includes incorrect spelling (spelling errors).

[0049] As a result of machine learning, the trained prediction model M1 is able to detect the misspelled character strings C31, C32, and C33 as anomalous df.

[0050] The control unit 21 inputs first data D12 representing a first image G12 to the trained prediction model M1. When the trained prediction model M1 detects character strings C31, C32, and C33 from the input first data D12, it predicts corrected character strings C34 "HELLO," C35 "I'm," and C36 "Barfoo" (FIG. 3) as first repair candidates rt1. The trained prediction model M1 replaces the character strings C31, C32, and C33 in the first image G12 with the predicted character strings C34, C35, and C36, respectively, to generate a second image G22 (FIG. 3).

[0051] The trained prediction model M1 outputs to the control unit 21 second data D22 indicating the generated second image G22.

[0052] When the control unit 21 receives the second data D22 output from the trained prediction model M1, the control unit 21 controls the display unit 25 to display the second image G22. The control unit 21 also controls the image forming unit 24 to form the second image G22 on the recording medium P.

[0053] In addition to splitting and misspelling of character strings C, the trained prediction model M1 can detect, as abnormalities df, faded or worn character strings C, and missing character strings C that are located in predetermined positions such as page numbers.

[0054] When the control unit 21 displays the second image G2 on the display unit 25, the control unit 21 may display the repair candidate rt in the second image G2 in a color scheme, background color, or transparency different from that of the other character strings C.

[0055] Next, an image forming method in the image forming apparatus 20 according to this embodiment will be described with reference to Fig. 7. Fig. 7 is a flowchart showing the image forming method in the image forming apparatus 20 according to this embodiment.

[0056] When the receiving unit 26 receives an instruction to form an image on the recording medium P, the acquiring unit 201 acquires the first data D1 (step S11).

[0057] The control unit 21 inputs the first data D1 acquired by the acquisition unit 201 into the trained prediction model M1 (step S12).

[0058] The trained prediction model M1 determines whether the input first data D1 includes an abnormality df (step S13).

[0059] If the input first data D1 includes an abnormality df (Yes in step S13), the trained prediction model M1 predicts one or more repair candidates rt for the abnormality df (step S14).

[0060] The trained prediction model M1 selects one of the repair candidates rt as a first repair candidate rt1 (step S15), and replaces it with the abnormal df (step S16).

[0061] The trained prediction model M1 outputs a second image G2 including the first restoration candidate rt1 and second data D2 including restoration candidates rt other than the first restoration candidate rt1 (step S17).

[0062] On the other hand, if the input first data D1 does not contain an abnormality df (No in step S13), the learned prediction model M1 outputs the first data D1 as the second data D2 (step S17) without replacing the string C in the first data D1 with the repair candidate rt (step S18).

[0063] The control unit 21 controls the display unit 25 to display the second image G2 and the restoration candidates rt other than the first restoration candidate rt1 based on the second data D2 output from the trained prediction model M1 (step S19).

[0064] The receiving unit 26 waits for a selection operation to select one of the repair candidates rt other than the first repair candidate rt1 (step S20). When the receiving unit 26 receives the selection operation (Yes in step S20), the control unit 21 controls the display unit 25 to replace the first repair candidate rt1 in the second image G2 with the repair candidate rt selected by the selection operation and display it (step S21).

[0065] The control unit 21 controls the image forming unit 24 to form a second image G2 on the recording medium P (step S22).

[0066] On the other hand, if the accepting unit 26 does not accept a selection operation within the predetermined period of time (No in step S20), the control unit 21 controls the image forming unit 24 to form a second image G2 on the recording medium P (step S22).

[0067] In this embodiment, a learning device 200 that generates a trained discrimination model may generate the trained prediction model M1 separately from the image forming device 20. For example, the learning device 200 includes an acquisition unit 201 and a generation unit 202. The learning device 200 may be provided outside the image forming device 20. In this case, the learning device 200 transmits the generated trained prediction model M1 to the image forming device 20. The image forming device 20 receives the trained prediction model M1 transmitted from the learning device 200 and stores it in the storage unit 23.

[0068] The generation unit 202 performs machine learning by associating a first data set including a plurality of past first data D1 input to the image forming device 20 or the learning device 200 with a second data set including a plurality of past second data D2, thereby generating a trained prediction model M1. In particular, the generation unit 202 performs machine learning by associating a first training data set showing images including a training character string (character string C1) divided into a plurality of parts (parts p1 and p2) as shown in FIG. 2 with a second training data set showing images including a training character string (character string C2) in an integrated state, thereby generating a trained prediction model M1 that predicts the character string C2 as a repair candidate rt for the character string C1. Therefore, the trained prediction model M1 can repair, for example, a torn image.

[0069] The embodiments of the present invention have been described above with reference to the drawings. However, the present invention is not limited to the above embodiments and can be implemented in various forms without departing from the spirit of the present invention. Furthermore, the components disclosed in the above embodiments can be modified as appropriate. For example, some of the components shown in one embodiment may be added to the components of another embodiment, or some of the components shown in one embodiment may be deleted from the embodiment.

[0070] Furthermore, the drawings mainly show each component in a schematic manner to facilitate understanding of the invention, and the thickness, length, number, spacing, etc. of each component shown in the drawings may differ from the actual ones due to the convenience of creating the drawings. Furthermore, the configuration of each component shown in the above embodiment is merely an example and is not particularly limited, and it goes without saying that various modifications are possible within a range that does not substantially deviate from the effects of the present invention. [Industrial Applicability]

[0071] The present invention can be used in the field of image forming devices. [Explanation of symbols]

[0072] 20: Image forming device 21: Control unit 22: Image reading unit 23: Storage section 24: Image forming unit 25:Display section 26: Reception 101: Discrimination model generation unit 200: Learning device 201: Acquisition Department 202 :Generation part C, C1, C2, C21~C24, C31~C36: String D1, D12: First data D2, D22: Second data D3: Third data G1, G12: First image G2, G21, G22: Second image M1: Prediction model P: Recording medium df: Abnormal p1, p2 :partial rt :Repair candidate rt1: First repair candidate

Claims

1. An image forming apparatus for forming an image on a recording medium, an acquisition unit that acquires first data including a first image that includes at least a character string; a storage unit that stores a trained prediction model that, when the first data is input, detects an anomaly in the first image, predicts a first repair candidate for the anomaly when the anomaly is detected, and outputs second data in which the anomaly is replaced with the first repair candidate; a control unit that inputs the first data into the trained prediction model and outputs the second data from the trained prediction model; a display unit that displays a second image including at least the first restoration candidate based on the second data output from the trained prediction model; An image forming apparatus comprising:

2. the trained prediction model predicts a plurality of repair candidates for the anomaly, selects one of the plurality of repair candidates as the first repair candidate to replace the anomaly, and outputs the second data including repair candidates other than the first repair candidate; The image forming apparatus according to claim 1 , wherein the display unit further displays repair candidates other than the first repair candidate.

3. a receiving unit that receives a selection operation to select one of the repair candidates other than the first repair candidate; 3. The image forming apparatus according to claim 2, wherein when the receiving unit receives the selection operation, the control unit replaces the first repair candidate in the second data with the repair candidate selected by the selection operation to generate third data.

4. The image forming apparatus according to claim 3 , further comprising a generating unit that performs machine learning by associating the first data with the third data to generate or update the trained prediction model.

5. 5. The image forming apparatus according to claim 1, wherein, when the character string is included in the first image in a state where the character string is divided into a plurality of parts, the trained prediction model replaces the plurality of parts with the character string obtained by integrating the plurality of parts as the first repair candidate.

6. 6. The image forming apparatus of claim 5, further comprising a generation unit that performs machine learning by associating first training data indicating an image including a training character string divided into multiple parts with second training data indicating an image including the training character string in which the multiple parts included in the first training data have been integrated, to generate the trained prediction model.

7. an acquisition unit that acquires first training data representing an image including a training character string divided into a plurality of parts, and second training data representing an image including the training character string obtained by integrating the plurality of parts included in the first training data; a generation unit that performs machine learning by associating the first learning data with the second learning data to generate the trained prediction model; A learning device comprising:

Images