Information processing apparatus, information processing system, information processing method, and program

Abstract

To enable a user to correct an automatically corrected image.SOLUTION: An information processing apparatus includes an image generator that generates a first correction image by correcting document data based on first correction information output from a correction information estimation model to which the document data including an image read from a document to which a correction instruction is added is input, a display controller that displays the first correction image, and a receiver that receives an input of second correction information for the document data from a user. The correction information estimation model is a machine learning model that has learned a correspondence relationship between image data including a correction instruction and correction information indicating correction content corresponding to the correction instruction.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing system, an information processing method, and a program. [Background technology]

[0002] 2. Description of the Related Art Conventionally, a technique has been considered in which an image read from a document on which correction instructions have been handwritten is automatically corrected in accordance with the correction instructions.

[0003] For example, Patent Document 1 discloses an image forming device that reads an image from a manuscript that has been edited or revised, extracts an image of the edited portion from the read image, and edits the image read from the manuscript in accordance with the instructions in the extracted image of the edited portion. Summary of the Invention [Problem to be solved by the invention]

[0004] However, with conventional techniques, the user cannot give instructions for re-correcting an image that has been automatically corrected by the image forming apparatus.

[0005] The present invention has been made in view of the above points, and has as its object to enable a user to modify an automatically modified image. [Means for solving the problem]

[0006] In order to solve the above problem, an information processing device has an image generation unit that generates a first corrected image by correcting manuscript data based on first correction information output by a correction information estimation model that has received input of manuscript data including an image read from a manuscript to which correction instructions have been added, a display control unit that displays the first corrected image, and a reception unit that receives input of second correction information for the manuscript data from a user, and is characterized in that the correction information estimation model is a machine learning model that has learned the correspondence between image data including correction instructions and correction information that indicates the correction content according to the correction instructions. [Effects of the Invention]

[0007] The automatically corrected image may be made available for user correction. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating an example of a configuration of an information processing system 1 according to an embodiment of the present invention. [Figure 2] 1 is a diagram illustrating an example of a hardware configuration of an image forming apparatus 10 according to an embodiment of the present invention. [Figure 3] 2 is a diagram illustrating an example of a hardware configuration of a server device 20 according to an embodiment of the present invention. [Figure 4] 1 is a diagram illustrating an example of a functional configuration of an information processing system 1 according to an embodiment of the present invention. [Figure 5] 10 is a flowchart illustrating an example of a processing procedure for generating learning data for the correction information estimation model 23. [Figure 6] FIG. 10 is a diagram for explaining the designation of a correction symbol, a portion to be cancelled, and a portion to be corrected. [Figure 7] FIG. 10 is a diagram for explaining designation of a destination of a correction portion. [Figure 8] FIG. 10 is a diagram showing another example of a correction instruction. [Figure 9] 10 is a flowchart illustrating an example of a processing procedure for correcting manuscript data. [Figure 10] FIG. 10 is a sequence diagram illustrating an example of a processing procedure for correcting manuscript data. [Figure 11] FIG. 10 is a diagram showing a display example of a first correction image. [Figure 12] FIG. 10 is a diagram showing a display example of manuscript data. [Figure 13] FIG. 10 is a diagram showing a display example of a second correction image. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Fig. 1 is a diagram showing an example of the configuration of an information processing system 1 according to an embodiment of the present invention. As shown in Fig. 1, the information processing system 1 includes an image forming apparatus 10 and a server apparatus 20. The image forming apparatus 10 is connected to the server apparatus 20 via a network such as a LAN (Local Area Network) or the Internet.

[0010] The image forming apparatus 10 is an information processing apparatus that reads image data from a manuscript and executes various processes on the image data. In this embodiment, correction instructions are added (revised) to the manuscript. The correction instructions are information indicating changes to the characters themselves contained in the manuscript, changes to the position of the characters, etc. The image forming apparatus 10 makes corrections to data including an image read from the manuscript (i.e., image data read from the manuscript; hereinafter referred to as "manuscript data") according to the correction content indicated in the correction instructions, thereby generating an image corresponding to the image data after correction (hereinafter referred to as "revised image"). Note that the characters contained in the manuscript may be handwritten or typed. The correction instructions may be added to the manuscript by handwriting, or may be added by other methods.

[0011] The server device 20 receives manuscript data from the image forming device 10 and generates information indicating the content of the revisions implied by the revision instructions included in the manuscript data (hereinafter referred to as "revision information"). The server device 20 transmits the generated revision information to the image forming device 10. The revision information is information that mechanically indicates the content of the revisions. Therefore, the image forming device 10 can correct the manuscript data based on the revision information. The revision information is generated (estimated) using a trained machine learning model. The server device 20 may be configured by one computer or multiple computers.

[0012] 2 is a diagram showing an example of the hardware configuration of image forming apparatus 10 according to an embodiment of the present invention. In FIG. 2, image forming apparatus 10 includes hardware such as controller 11, scanner 12, printer 13, modem 14, operation panel 15, network interface 16, and SD card slot 17.

[0013] The controller 11 has a CPU 111, a RAM 112, a ROM 113, a HDD 114, an NVRAM 115, etc. The ROM 113 stores various programs, data used by the programs, etc. The RAM 112 is used as a storage area for loading programs, a work area for the loaded programs, etc. The CPU 111 realizes various functions by processing the programs loaded into the RAM 112. The HDD 114 stores programs, various data used by the programs, etc. The NVRAM 115 stores various setting information, etc.

[0014] The scanner 12 is hardware (image reading means) for reading image data from an original. The printer 13 is hardware (printing means) for printing print data on printing paper. The modem 14 is hardware for connecting to a telephone line and is used to send and receive image data via FAX communication. The operation panel 15 is hardware equipped with input means such as buttons for receiving input from a user and display means such as an LCD panel. The LCD panel may have a touch panel function. In this case, the LCD panel also functions as an input means. The network interface 16 is hardware for connecting to a network (whether wired or wireless) such as a LAN. The SD card slot 17 is used to read programs stored on the SD card 80. In other words, in the image forming apparatus 10, not only programs stored in the ROM 113 but also programs stored on the SD card 80 can be loaded into the RAM 112 and executed. Note that the SD card 80 may be replaced with another recording medium (e.g., a CD-ROM or a USB (Universal Serial Bus) memory). That is, the type of recording medium corresponding to the position of the SD card 80 is not limited to a predetermined one. In this case, the SD card slot 17 may be replaced by hardware that corresponds to the type of recording medium.

[0015] Fig. 3 is a diagram showing an example of the hardware configuration of a server device 20 according to an embodiment of the present invention. The server device 20 in Fig. 3 includes a drive device 200, an auxiliary storage device 202, a memory device 203, a processor 204, and an interface device 205, which are all interconnected via a bus B.

[0016] A program that realizes processing in the server device 20 is provided by a recording medium 201 such as a CD-ROM. When the recording medium 201 storing the program is set in the drive device 200, the program is installed from the recording medium 201 to the auxiliary storage device 202 via the drive device 200. However, the program does not necessarily have to be installed from the recording medium 201, but may be downloaded from another computer via a network. The auxiliary storage device 202 stores the installed program as well as necessary files, data, etc.

[0017] When an instruction to start a program is received, the memory device 203 reads and stores the program from the auxiliary storage device 202. The processor 204 is a CPU or a GPU (Graphics Processing Unit), or a CPU and a GPU, and executes functions related to the server device 20 in accordance with the program stored in the memory device 203. The interface device 205 is used as an interface for connecting to a network.

[0018] Fig. 4 is a diagram showing an example of a functional configuration of an information processing system 1 according to an embodiment of the present invention. In Fig. 4, a server device 20 includes a communication unit 21, a learning unit 22, and a correction information estimation model 23. These units are realized by a process in which one or more programs installed in the server device 20 are executed by a processor 204. The server device 20 also uses a learning data storage unit 211. The learning data storage unit 211 can be realized using, for example, the auxiliary storage device 202, or a storage device connectable to the server device 20 via a network.

[0019] The correction information estimation model 23 is a machine learning model (for example, a neural network) that receives image data including a correction instruction as input and outputs (an estimated result of) the correction information indicated by the correction instruction.

[0020] The communication unit 21 communicates with the image forming apparatus 10. For example, the communication unit 21 receives learning data for the correction information estimation model 23 from the image forming apparatus 10. That is, in the present embodiment, the learning data is generated in the image forming apparatus 10. As described above, the input to the correction information estimation model 23 is learning data, and the output from the correction information estimation model 23 is correction information. Therefore, the learning data is a pair of image data (manuscript data in the present embodiment) as input and correction information as a correct answer for the manuscript data. The communication unit 21 stores the received learning data in the communication unit 21. The communication unit 21 also functions as a second receiving unit that receives image data including a correction instruction from the image forming apparatus 10. The communication unit 21 further functions as a second transmitting unit that transmits the correction information output by the correction information estimation model 23 to the image forming apparatus 10.

[0021] The learning unit 22 learns the correction information estimation model 23 using the learning data stored in the learning data storage unit 211. For each piece of learning data, the learning unit 22 learns the correction information estimation model 23 so that the output from the correction information estimation model 23 to which manuscript data included in the learning data has been input approaches the correction information included in the learning data. In this way, the correction information estimation model 23 learns the correspondence between manuscript data including correction instructions and correction information.

[0022] The image forming apparatus 10 has a display control unit 121, a reception unit 122, a processing control unit 131, a reading control unit 132, an image generation unit 133, and a communication unit 134. Each of these units is realized by processing that is executed by the CPU 111 of one or more programs installed in the image forming apparatus 10. The image forming apparatus 10 also uses an image storage unit 150.

[0023] The image storage unit 150 can be realized using, for example, the HDD 114 or a storage device connectable to the image forming apparatus 10 via a network.

[0024] The reading control unit 132 controls the scanner 12 to acquire image data (document data) read from the document by the scanner 12. In this embodiment, the document data of the document to which correction instructions have been added is acquired.

[0025] The image generation unit 133 generates a first corrected image by correcting the manuscript data based on the first correction information output by the correction information estimation model 23, which has received as input manuscript data including an image read from a manuscript to which correction instructions have been added. When the first correction information is corrected to second correction information by the user, the image generation unit 133 also generates a second corrected image by correcting the manuscript data based on the second correction information.

[0026] The display control unit 121 displays a screen for receiving instructions from a user for the image forming apparatus 10, processing results by the image forming apparatus 10, etc. on the operation panel 15. For example, the display control unit 121 displays a corrected image generated by the image generation unit 133 for the manuscript data, etc.

[0027] The receiving unit 122 receives input from a user. For example, when generating learning data for the correction information estimation model 23, the receiving unit 122 receives, from the user, input of correction information as a correct answer for manuscript data used as learning data. The receiving unit 122 also receives, from the user, input of second correction information for the manuscript data when the user determines, during inference after learning of the correction information estimation model 23, that the first corrected image generated by the image generation unit 133 based on the first correction information output by the correction information estimation model 23 to which manuscript data has been input, is inappropriate.

[0028] The display control unit 121 and the reception unit 122 constitute an input / output unit 120 .

[0029] The process control unit 131 controls processing in response to the input received by the receiving unit 122. The process control unit 131 also controls processing for transmitting manuscript data to the server device 20 and receiving, from the server device 20, revision information that is output when the revision information estimation model 23 of the server device 20 receives the manuscript data.

[0030] The communication unit 134 relays communication between the process control unit 131 and the server device 20.

[0031] The image storage unit 150 stores image data that the user has determined to be appropriate from among the corrected images generated by the image generation unit 133.

[0032] The processing procedure executed in the information processing system 1 will be described below.

[0033] First, we will explain the processing procedure executed when generating learning data for the correction information estimation model 23. Fig. 5 is a flowchart for explaining an example of the processing procedure for generating learning data for the correction information estimation model 23. At the start of the processing procedure in Fig. 5, a document to which correction information has been added is set in the scanner 12 of the image forming apparatus 10.

[0034] In step S101, the display control unit 121 displays image data (manuscript data) of a manuscript to which correction instructions have been added, which image data has been read by the reading control unit 132 using the scanner 12, on the operation panel 15. From step S102 onwards, processing is executed to accept annotations (meanings) from the user regarding the correction instructions (revised portions) included in the displayed manuscript data. Annotation refers to specifying the correction information indicated by the correction instructions. In other words, annotation refers to specifying what kind of correction the correction instructions intend.

[0035] In step S102, the receiving unit 122 receives the specification of the correction information from the user. In this embodiment, the correction information includes one or more of a correction symbol, a portion to be cancelled, and a portion to be corrected.

[0036] FIG. 6 is a diagram for explaining the designation of the correction symbol, the portion to be cancelled, and the portion to be corrected.

[0037] In (1) of FIG. 6, the state where the manuscript data is displayed on the touch panel of the operation panel 15 in step S101 is shown. Here, an example is shown where a double line is added (written in addition) to the character "考", and the character "老" is added (written in addition) above the "考" to which the double line is added.

[0038] A correction symbol refers to information (such as a symbol, character, mark, or figure) indicating a correction method. In the example of FIG. 6, the double line added to "考" corresponds to the correction symbol.

[0039] A cancellation part refers to the part (portion) that is deleted (cancelled) by the correction indicated by the correction symbol. In the example of FIG. 6, the character "考" corresponds to the cancellation part. Note that the correction symbol may not only be a symbol indicating deletion but also a symbol indicating insertion, movement, etc.

[0040] A correction part refers to the part (portion) that is the target of movement (correction target). In the example of FIG. 6, the user who gave the correction instruction intends to replace the character "老" with the position of "考", that is, to move the character "老" to the position of "考". Therefore, in this example, the character "老" corresponds to the correction part. Note that in FIG. 6, an example where the added character is the correction part is shown, but an existing character can also be the target of movement (that is, the correction part). For example, when an existing character string is to be moved to another position, the character string can be the correction part.

[0041] (2) of FIG. 6 shows the state where the correction symbol is specified in step S102. For example, the user draws an inscribed rectangle for the correction symbol by a touch operation on the touch panel using a touch pen or the like. In other words, the user draws a rectangle surrounding the correction symbol. In (2) of FIG. 6, the double line is surrounded by a rectangle.

[0042] (3) in FIG. 6 shows how the part to be cancelled is designated in step S102. For example, the user draws an inscribed rectangle for the part to be cancelled by touching the touch panel with a touch pen or the like. In other words, the user draws a rectangle that surrounds the part to be cancelled. In (3) in FIG. 6, "thought" is surrounded by a rectangle.

[0043] (4) in Fig. 6 shows how the part to be corrected is designated in step S102. For example, the user draws an inscribed rectangle for the part to be corrected by touching the touch panel with a touch pen or the like. In other words, the user draws a rectangle that surrounds the part to be corrected. In (4) in Fig. 6, "old" is surrounded by a rectangle.

[0044] For each rectangle, for example, the user may specify two points, and the rectangle with the two points as diagonal vertices may be automatically generated and displayed by the reception unit 122. Furthermore, whether the input by the user is a correction symbol, a cancellation point, or a correction point may be selected from a menu or the like displayed on the operation panel 15 together with the manuscript data.

[0045] The receiving unit 122 stores coordinate information (for example, coordinate information of the top left and top right vertices) capable of specifying the position and shape of a rectangle for each correction symbol, cancellation portion, and correction portion in the RAM 112. The coordinate information refers to coordinate values ​​(X coordinate value, Y coordinate value) in the coordinate system of the manuscript data. The coordinate system of the manuscript data is, for example, a two-dimensional coordinate system with one of the vertices of the manuscript data (for example, the top left vertex) as the origin and pixels as units.

[0046] Following step S102, the receiving unit 122 receives from the user a designation of the arrangement position (destination) of the correction portion after the correction (S103).

[0047] 7 is a diagram for explaining the designation of the destination of the correction portion. When the destination of the correction portion is designated, the reception unit 122 hides the correction symbol and the correction portion designated in step S102. (5) in FIG. 7 shows this state.

[0048] 7(6) shows how the destination of the correction portion is received. The user moves (e.g., drags or drag-and-drops) the correction portion (as a rectangular area) to a desired destination by a touch operation. When the user inputs an instruction to complete the movement, the receiving unit 122 records the coordinate information of the destination of the correction portion in the RAM 112.

[0049] This completes the process of accepting annotations of correction information from the user. Therefore, the correction information has, for example, the following structure.

[0050] · Coordinate information of correction symbols - Coordinate information of the cancellation area Coordinate information of the area to be corrected (before moving) and coordinate information of the area to be corrected (destination) Next, the reception unit 122 transmits the set of manuscript data and correction information to the server device 20 via the communication unit 134 (S104). The communication unit 21 of the server device 20 stores the received set of manuscript data and correction information in the learning data storage unit 211 as learning data for the correction information estimation model 23.

[0051] It should be noted that the correction instructions are not limited to those shown in Figures 6 and 7. Figure 8 is a diagram showing another example of a correction instruction.

[0052] In Figure 8, examples of correction instructions (1) and (2) are shown in the upper row, and correction information (1') corresponding to correction instruction (1) and correction information (2') corresponding to correction instruction (2) are shown in the lower row.

[0053] Correction instruction (1) is an example in which an irregular shape (trajectory) is used to indicate the deletion of characters. In this case, as shown in (1'), the irregular shape is designated as the correction symbol, the characters filled with the irregular shape are designated as the deletion point, and the characters written above the deletion point are designated as the correction point. The destination of the correction point is also specified.

[0054] Correction instruction (2) is an instruction to insert a string. In this case, a symbol representing insertion is specified as the correction symbol, and the string above the correction symbol ("Ebina") and the string whose position will be shifted by the insertion of the character string ("Izumi 2-7-2") are specified as the correction points. The destination of each correction point is also specified.

[0055] The above correction instructions are merely examples, and various other correction instructions may be added.

[0056] Furthermore, multiple correction instructions can be included in one piece of manuscript data, and multiple correction locations can be specified for one correction instruction (correction symbol).

[0057] By storing the correction information based on such various correction instructions and manuscript data in the learning data storage unit 211, the learning unit 22 can learn the correction information estimation model 23 that can respond to various correction instructions. The learning unit 22 learns the correction information estimation model 23 for each learning data as follows.

[0058] The learning unit 22 inputs the manuscript data included in the learning data to the correction information estimation model 23 .

[0059] The learning unit 22 calculates the loss between the output from the correction information estimation model 23 and the correction information included in the learning data.

[0060] The learning unit 22 updates the parameters of the correction information estimation model 23 so that the loss is reduced (that is, so that the output from the correction information estimation model 23 approaches the correction information).

[0061] Through such learning processing, the correction information estimation model 23 learns the correspondence between manuscript data including correction instructions and correction information (tendency of correction information corresponding to correction instructions). Due to the effect of generalization of the correction information estimation model 23 through learning, it can be expected that it will be possible to estimate correction information even for correction instructions that are not included in the learning data.

[0062] Next, the inference process (correction of manuscript data using the learned correction information estimation model 23) will be described.

[0063] Fig. 9 is a flowchart for explaining an example of the processing procedure for correcting manuscript data. Fig. 10 is a sequence diagram for explaining an example of the processing procedure for correcting manuscript data. Fig. 10 shows the processing procedure of Fig. 9 at a functional configuration level. In Fig. 9 and Fig. 10, steps with the same first two digits of the step number correspond to the same processing.

[0064] 9, in a state in which a document with correction instructions added thereto is set in the scanner 12, the image forming apparatus 10 receives an input of a document correction request from the user via the operation panel 15. More specifically, in step S211 in Fig. 10, upon receiving the input of the document correction request, the receiving unit 122 notifies the process control unit 131 of the document correction request (S212). The document correction request may be input by pressing a predetermined button, etc.

[0065] Next, the image forming apparatus 10 reads image data (document data) from the document (S220 in FIG. 9). More specifically, in step S221 in FIG. 10, the process control unit 131 instructs the reading control unit 132 to read the document. The reading control unit 132 causes the scanner 12 to read the image data (document data) from the document and acquires the document data (S222). The reading control unit 132 transmits the document data to the process control unit 131 (S223).

[0066] Next, the image forming apparatus 10 acquires first correction information indicated by the correction instruction included in the manuscript data from the trained correction information estimation model 23 (FIG. 9: S230). More specifically, in step S231 of FIG. 10, the process control unit 131 transmits (inputs) the manuscript data to the server apparatus 20 via the communication unit 134 (first transmission unit). When the communication unit 21 (second reception unit) of the server apparatus 20 receives the manuscript data, the correction information estimation model 23 inputs the manuscript data and outputs first correction information corresponding to the correction instruction included in the manuscript data based on the trained parameters (S232). The communication unit 21 (second transmission unit) transmits the first correction information to the image forming apparatus 10 (S233). The process control unit 131 receives the first correction information via the communication unit 134 (first reception unit).

[0067] Next, the image forming apparatus 10 generates a first corrected image by correcting the copy of the manuscript data based on the first correction information (FIG. 9: S240). More specifically, in step S241 of FIG. 10, the process control unit 131 transmits a request for generating a corrected image, including the manuscript data and the first correction information, to the image generation unit 133. In response to the generation request, the image generation unit 133 applies corrections according to the first correction information to the copy of the manuscript data (hereinafter referred to as "duplicate data") to generate a first corrected image (S242). As described above, the correction information includes coordinate information for the correction symbol, the cancellation portion, the correction portion, and the destination of the correction portion. The image generation unit 133 first erases, from the duplicate data, drawing elements included in the correction symbol and the cancellation portion included in the first correction information. A drawing element is a collection of pixels of a color different from the background that constitutes a character, symbol, figure, image, etc. The image generation unit 133 also moves the content (a set of pixels) of the correction portion included in the first correction information to the destination of the correction portion included in the first correction information. Moving the correction portion means copying the correction portion to the destination and deleting the correction portion in the original position. As a result, the correction of the duplicated data based on the first correction information is completed, and the duplicated data becomes a first corrected image. The image generation unit 133 transmits the first corrected image to the process control unit 131 (S243).

[0068] In addition, if the characters in the original document are typed and the corrections are one or more characters added by hand, the image generation unit 133 may convert the characters at the corrections to characters with the same font and size as the surrounding characters at the destination of the corrections, before or after moving the corrections. This reduces the unnaturalness of the corrections after the movement in the first correction image. For example, the image generation unit 133 converts the handwritten characters at the corrections to text using character recognition technology. The image generation unit 133 also uses character recognition technology to determine the font and size of characters within a predetermined range at the destination of the corrections. The image generation unit 133 converts the characters at the corrections in text form into an image with the determined font and size, thereby making the font and size of the characters the same as the surrounding characters at the destination of the movement.

[0069] Next, the image forming apparatus 10 displays the first corrected image on the operation panel 15 (FIG. 9: S250). More specifically, in step S251 of FIG. 10, the process control unit 131 transmits a display request for the first corrected image together with the first corrected image and the first correction information to the display control unit 121. In response to the display request, the display control unit 121 displays the first corrected image on the operation panel 15 (S252).

[0070] FIG. 11 is a diagram showing a display example of the first correction image. As shown in FIG. 11, the display control unit 121 displays, on the operation panel 15, a query screen 510 for querying the user about the accuracy of the first correction image g1, along with the first correction image g1. The accuracy of the first correction image g1 refers to whether the corrections made to the manuscript data are appropriate (correct). The accuracy of the corrections made to the manuscript data refers to whether the corrections have been made to the manuscript data in accordance with the correction instructions. The query screen 510 includes a YES button 511, a position correction button 512, and a NO button 513. The YES button 511 is a button for receiving an input indicating that the corrections are appropriate. The NO button 513 is a button for receiving an input indicating that the corrections are not appropriate. The position correction button 512 will be described later.

[0071] Next, the image forming apparatus 10 accepts an input from the user regarding the accuracy of the first correction image g1 (S260 in FIG. 9). More specifically, in step S261 in FIG. 10, the accepting unit 122 accepts a selection from the user of any one of the YES button 511, the position correction button 512, and the NO button 513 via the inquiry screen 510. In response to the selection of any one of the buttons, the accepting unit 122 erases the inquiry screen 510 and notifies the process control unit 131 of information indicating the selected button (S262).

[0072] Next, the process branches depending on whether the user inputs that the information is valid (selects the YES button 511) or invalid (selects the position correction button 512 or the NO button 513) (Figure 9: S270).

[0073] 11, the position of the "old" in "Ebina" is shifted upward, and therefore the first corrected image g1 cannot be said to be appropriate. Therefore, the user selects the position correction button 512 or the NO button 513.

[0074] When the position correction button 512 or the NO button 513 is selected (FIG. 9: No in S270), the image forming apparatus 10 executes a process for accepting correction of the first correction information. The first correction image g1 was invalid because the first correction information output by the correction information estimation model 23 was incorrect, and it is therefore necessary for the user to input correct correction information (second correction information).

[0075] However, the procedure for correcting the first correction information differs depending on whether the NO button 513 is selected or the position correction button 512 is selected. When the NO button 513 is selected, all of the first correction information is corrected, whereas when the position correction button 512 is selected, only part of the first correction information is corrected.

[0076] Specifically, when the NO button 513 is selected (Yes in S271 in FIG. 9), the image forming apparatus 10 displays the manuscript data on the operation panel 15 (S280). The manuscript data is image data that includes correction instructions in the state that it was read from the manuscript in step S220. More specifically, in step S281 in FIG. 10, the process control unit 131 transmits the manuscript data together with a request to display the manuscript data to the display control unit 121. The display control unit 121 displays the manuscript data on the operation panel 15 (S282).

[0077] 12 is a diagram showing an example of displaying manuscript data, which, for convenience, shows manuscript data including correction instructions similar to those in (1) of FIG.

[0078] Next, image forming apparatus 10 accepts input of second correction information from the user via the manuscript data (S290 in FIG. 9). More specifically, in step S291 in FIG. 10, accepting unit 122 accepts the second correction information from the user. Accepting unit 122 transmits the input second correction information to process control unit 131 (S292).

[0079] The procedure for inputting the second correction information is the same as the procedure for inputting the correction information described in steps S102 and S103 of Fig. 5. That is, the user specifies the correction symbol, the portion to be cancelled, and the portion to be corrected by touch operation or the like, as shown in (2) to (4) of Fig. 6. As a result, the manuscript data is displayed in a state in which the correction symbol and the portion to be cancelled have been erased ((5) of Fig. 7), and the user specifies the destination of the portion to be corrected ((6) of Fig. 7). In step S292 of Fig. 10, the second correction information based on such input is transmitted to the process control unit 131.

[0080] On the other hand, the input procedure for the second correction information when the position correction button 512 is selected is different from the above. For example, in the first correction image g1 shown in FIG. 11, the position of the character "老" which was the correction location is only slightly shifted upward, and a proper correction image can be obtained as long as the position of "老" is correctly shifted. Such a shift occurred because only the destination of movement of the correction location was incorrect in the first correction information output by the correction information estimation model 23. In other words, it is considered that the correction information estimation model 23 correctly estimated the correction symbol, cancellation location, and correction location. In such a case, it is troublesome for the user to specify the correction symbol, cancellation location, and correction location again. Therefore, in the present embodiment, the position correction button 512 is provided as an option when only the destination of movement of the correction location is to be corrected.

[0081] Specifically, when the position correction button 512 is selected (No in FIG. 9: S271), since step S280 is not executed, the original manuscript data is not displayed and the first correction image remains displayed. In this state, the reception unit 122 receives the second correction information from the user by having a part of the first correction information corrected by the user via the first correction image g1. At this time, in order to reduce the operation burden of the user, the display control unit 121 may superimpose and display a rectangle surrounding the correction location on the first correction image g1 based on the coordinate information of the correction location included in the first correction information received from the processing control unit 131 in step S251 of FIG. 10. Specifically, the display control unit 121 may display a rectangle surrounding "老" in the first correction image g1. The user corrects the destination of movement of the correction location (the character "老") by shifting it to a desired position (downward) (FIG. 10: S291). The reception unit 122 generates the second correction information by correcting the destination of movement of the correction location to the coordinate information corresponding to the input by the user with respect to the first correction information received by the display control unit 121 in step S251, and transmits the second correction information to the processing control unit 131 in step S292 of FIG. 10.

[0082] Next, the image forming apparatus transmits a set of the manuscript data and the second correction information received in step S290 to the server device 20 as learning data (S300). More specifically, in step S301 of FIG. 10 , the process control unit 131 controls a process for storing learning data including the manuscript data and the second correction information received from the receiving unit 122 in step S292 in the server device 20. That is, the process control unit 131 transmits the learning data to the server device 20. Upon receiving the learning data, the communication unit 21 of the server device 20 stores the learning data in the learning data storage unit 211.

[0083] 9 are then repeated based on the second correction information. In this case, if the user has correctly input the second correction information, it is highly likely that the image generation unit 133 will be able to correctly generate the second corrected image. Therefore, in step S250, it is highly likely that the second corrected image will be displayed as intended by the user.

[0084] Fig. 13 is a diagram showing a display example of a second correction image. The second correction image g2 shown in Fig. 13 shows the result of correcting the manuscript data in accordance with the correction instructions.

[0085] When the YES button 511 is selected on the inquiry screen 510 that is displayed again together with the newly generated second edited image g2 (Yes in S270 in FIG. 9), the processing control unit 131 of the MPF stores the second edited image in the image storage unit 150 (S310 in FIGS. 9 and 10). As a result, the user can obtain an edited image with the desired edits made. For example, the user can obtain the desired document by printing the edited image.

[0086] 9, if the second correction information is not input correctly due to an erroneous operation by the user or the like (if the first correction information is not corrected correctly), the second correction image g2 will be generated with content not intended by the user. In this case, the user can correct the second correction information by again selecting the position correction button 512 or the NO button 513. In this case, in step S300, learning data including correction information different from the second correction information for the same manuscript data is transmitted to the server device 20. However, the communication unit 21 determines that the previously received learning data is incorrect, deletes the previously received learning data from the learning data storage unit 211, and stores the newly received learning data in the learning data storage unit 211.

[0087] In the above, an example was described in which the first correction information output from the correction information estimation model 23 was incorrect, but if the first correction information output from the correction information estimation model 23 is correct, the first correction image g1 is stored in the image storage unit 150.

[0088] Thereafter, when a predetermined number of unlearned learning data or more has been accumulated, the learning unit 22 of the server device 20 additionally learns the correction information estimation model 23 using the unlearned learning data. The learning method for the additional learning is the same as the initial learning method. The additional learning can improve the estimation accuracy of the correction information estimation model 23. As a result, it is possible to increase the possibility that the first corrected image will be stored in the image storage unit 150 as being valid.

[0089] As described above, according to this embodiment, the first correction image is displayed on the operation panel 15. Therefore, the user can visually check whether the correction is appropriate. If the correction is not appropriate, the user can input second correction information via the manuscript data or the first correction image. Therefore, it is possible for the user to make corrections to the automatically corrected image.

[0090] Furthermore, by having the correction information estimation model 23 learn the correspondence between image data including correction instructions and correction information using correction instructions for a large number of manuscripts generated in the user environment, the user can write correction instructions that he or she is familiar with into the manuscript without being forced to use specific correction symbols, etc.

[0091] Although the above example shows a document containing character strings as a correction target, a document containing drawing information other than characters, such as pictures and illustrations, may also be a correction target.

[0092] The information processing device is not limited to the image forming device 10. The information processing device may be, for example, a PJ (Projector), an IWB (Interactive White Board: a white board with an electronic blackboard function capable of intercommunication), an output device such as digital signage, a HUD (Head Up Display) device, industrial machinery, an imaging device, a sound collection device, medical equipment, a network home appliance, a notebook PC (Personal Computer), a mobile phone, a smartphone, a tablet terminal, a game console, a PDA (Personal Digital Assistant), a digital camera, a wearable PC, or a desktop PC. In this case, the information processing device may execute the processing of the present embodiment on the document data read by the image forming device 10.

[0093] The image generation unit 133 may be a model generated by the learning effect of machine learning.

[0094] In this embodiment, machine learning refers to a technology for enabling a computer to acquire human-like learning capabilities, in which the computer autonomously generates algorithms necessary for judgments such as data classification from learning data that is previously acquired, and applies these to new data to make predictions. The learning method for machine learning may be any of supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, and deep learning, or may be a combination of these learning methods; the learning method for machine learning is not limited.

[0095] Each function of the present embodiment can be realized by one or more processing circuits. The term "processing circuit" as used herein includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), a conventional circuit module, or other device designed to execute each function described above.

[0096] Additionally, the devices described above represent only one of several computing environments for practicing the present embodiments.

[0097] In some embodiments, image forming apparatus 10 includes multiple computing devices, such as a server cluster, configured to communicate with each other over any type of communications link, including a network, shared memory, etc., to perform the processes disclosed herein. Similarly, server apparatus 20 may include multiple computing devices configured to communicate with each other.

[0098] 9 and 10 may be configured to be shared in various combinations. For example, the image forming apparatus 10 may have the correction information estimation model 23. In this case, the image forming apparatus 10 may execute all steps.

[0099] Although the embodiments of the present invention have been described in detail above, the present invention is not limited to such specific embodiments, and various modifications and variations are possible within the scope of the gist of the present invention as described in the claims.

[0100] For example, aspects of the present invention are as follows. <1> an image generation unit that generates a first corrected image by correcting manuscript data based on first correction information output from a correction information estimation model that receives input of manuscript data including an image read from a manuscript to which correction instructions have been added; a display control unit that displays the first corrected image; a receiving unit that receives input of second correction information for the manuscript data from a user; and The correction information estimation model is a machine learning model that has learned a correspondence relationship between image data including a correction instruction and correction information indicating correction content in accordance with the correction instruction. 1. An information processing device comprising: <2> the image generation unit generates a second corrected image by correcting the manuscript data based on the second correction information received by the reception unit. Characterized by <1> The information processing device described. <3> The display control unit further displays the manuscript data, the accepting unit accepts input of the second correction information from a user via the displayed manuscript data. Characterized by <1> or <2> The information processing device described. <4> the receiving unit receives input of the second correction information from a user via the displayed first correction image; Characterized by <1> ~ <3> Any of the information processing devices described above. <5> The first modification information is information including one or more of a modification symbol, a portion to be deleted by the modification indicated by the modification symbol, a portion to be moved, and a destination of the portion to be moved. Characterized by <1> ~ <4> Any of the information processing devices described above. <6> The correction instructions are handwritten additions to the manuscript. Characterized by <1> ~ <5> Any of the information processing devices described above. <7> a processing control unit that controls a process for storing the manuscript data and the second correction information in a learning data storage unit as learning data for the machine learning model; characterized in that it has <1> ~ <6> Any of the information processing devices described above. <8> An information processing system including an information processing device and a server device, The information processing device includes: a first transmission unit that transmits document data including an image read from a document to which correction instructions have been added to the server device; a first receiving unit that receives first modification information from the server device; an image generating unit that generates a first corrected image by correcting the manuscript data based on the first correction information; a display control unit that displays the first corrected image; a receiving unit that receives input of second correction information for the manuscript data from a user; and The server device a second receiving unit that receives the document data; a correction information estimation model that inputs the manuscript data received by the second receiving unit and outputs the first correction information; a second transmission unit that transmits the first modification information to the information processing device; and The correction information estimation model is a machine learning model that has learned a correspondence relationship between image data including a correction instruction and correction information indicating correction content in accordance with the correction instruction. An information processing system comprising: <9> an image generation step of generating a first corrected image by correcting manuscript data based on first correction information output from a correction information estimation model to which manuscript data including an image read from a manuscript to which correction instructions have been added is input; a display control step of displaying the first corrected image; a receiving step of receiving input of second correction information for the manuscript data from a user; The computer executes The correction information estimation model is a machine learning model that has learned a correspondence relationship between image data including a correction instruction and correction information indicating correction content in accordance with the correction instruction. 1. An information processing method comprising: <10> an image generation step of generating a first corrected image by correcting manuscript data based on first correction information output from a correction information estimation model to which manuscript data including an image read from a manuscript to which correction instructions have been added is input; a display control step of displaying the first corrected image; a receiving step of receiving input of second correction information for the manuscript data from a user; on the computer, The correction information estimation model is a machine learning model that has learned a correspondence relationship between image data including a correction instruction and correction information indicating correction content in accordance with the correction instruction. A program characterized by: [Explanation of symbols]

[0101] 1. Information Processing Systems 10 Image forming device 11 Controller 12 Scanner 13 Printers 14 Modem 15 Operation panel 16 Network Interfaces 17 SD card slot 20 Server device 21 Communications Department 22 Learning Department 23 Modified information estimation model 80 SD cards 111 CPU 112 RAM 113 ROM 114 HDD 115 NVRAM 120 Input / output section 121 Display control unit 122 Reception 131 Processing control section 132 Reading control unit 133 Image Generation Unit 134 Communications Department 150 Image storage unit 200 Drive Device 201 Recording Media 202 Auxiliary storage device 203 Memory Device 204 processors 205 Interface Device 211 Learning data storage unit B Bus [Prior art documents] [Patent documents]

[0102] [Patent Document 1] Japanese Patent Application Publication No. 2018-019299

Claims

1. an image generating unit that generates a first corrected image by correcting manuscript data based on first correction information output from a correction information estimation model that receives input of manuscript data including an image read from a manuscript to which correction instructions have been added; a display control unit that displays the first corrected image; a receiving unit that receives input of second correction information for the manuscript data from a user; and The correction information estimation model is a machine learning model that has learned a correspondence relationship between image data including a correction instruction and correction information indicating correction content in accordance with the correction instruction.

1. An information processing device comprising:

2. the image generation unit generates a second corrected image by correcting the manuscript data based on the second correction information received by the reception unit; 2. The information processing apparatus according to claim 1, wherein:

3. The display control unit further displays the manuscript data, the accepting unit accepts input of the second correction information from a user via the displayed manuscript data; 3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

4. the accepting unit accepts input of the second correction information from a user via the displayed first correction image; 3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

5. The first correction information is information including at least one of a correction symbol, a portion to be deleted by the correction indicated by the correction symbol, a portion to be moved, and a destination of the portion to be moved.

3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

6. The correction instructions are handwritten additions to the manuscript.

3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

7. a processing control unit that controls a process for storing the manuscript data and the second correction information in a learning data storage unit as learning data for the machine learning model; 3. The information processing apparatus according to claim 1, further comprising:

8. An information processing system including an information processing device and a server device, The information processing device includes: a first transmitting unit that transmits document data including an image read from a document to which correction instructions have been added to the server device; a first receiving unit that receives first modification information from the server device; an image generating unit that generates a first corrected image by correcting the original data based on the first correction information; a display control unit that displays the first corrected image; a receiving unit that receives input of second correction information for the manuscript data from a user; and The server device a second receiving unit that receives the document data; a correction information estimation model that inputs the manuscript data received by the second receiving unit and outputs the first correction information; a second transmission unit that transmits the first correction information to the information processing device; and The correction information estimation model is a machine learning model that has learned a correspondence relationship between image data including a correction instruction and correction information indicating correction content in accordance with the correction instruction. An information processing system comprising:

9. an image generation procedure for generating a first corrected image by correcting manuscript data based on first correction information output from a correction information estimation model to which manuscript data including an image read from a manuscript to which correction instructions have been added is input; a display control step of displaying the first modified image; a receiving step of receiving input of second correction information for the manuscript data from a user; The computer executes The correction information estimation model is a machine learning model that has learned a correspondence relationship between image data including a correction instruction and correction information indicating correction content in accordance with the correction instruction.

1. An information processing method comprising:

10. an image generation procedure for generating a first corrected image by correcting manuscript data based on first correction information output from a correction information estimation model to which manuscript data including an image read from a manuscript to which correction instructions have been added is input; a display control step of displaying the first modified image; a receiving step of receiving input of second correction information for the manuscript data from a user; on the computer, The correction information estimation model is a machine learning model that has learned a correspondence relationship between image data including a correction instruction and correction information indicating correction content in accordance with the correction instruction. A program characterized by:

Images