Image processing system, image processing device, and control method

Abstract

To provide an image processing device capable of identifying a top and a bottom of a document without containing a character.SOLUTION: An image processing device (101) includes: an operation part (140) displaying information; and a scanner (10) reading a document, and inputs an image read with a scanner into a leaned model learned on the basis of the image without including a character and direction information of the image (S652), and makes the information on the image read with the scanner on the basis of at least an output result of the leaned model to a display part (S657).SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to an image processing apparatus that performs top-bottom estimation of a scanned image, and is used as a copying machine, a scanner, or a combination machine thereof. [Background technology]

[0002] Conventionally, image processing devices such as scanners that read image information from a document are known. In such image processing devices, a situation may occur in which an image is read with the document upside down due to a user's operational mistake. Patent Document 1 discloses a technology that uses OCR (Optical character recognition) technology to detect the upside down of an image and use it to correct the image orientation. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2010-219895 A Summary of the Invention [Problem to be solved by the invention]

[0004] According to the technology of Patent Document 1, if a document contains characters, it is possible to correct the top and bottom using OCR, but it is not possible to identify the top and bottom of a document that does not contain characters and is made up of only images.

[0005] The object of the present invention is to provide a method for identifying the top and bottom of a document that does not contain text. The process to be performed on images having an incorrect orientation among the images read by the reading unit can be set in advance according to the type of image. The object of the present invention is to provide an image processing device. [Means for solving the problem]

[0006] The present invention provides an image processing device comprising: a display unit for displaying information; a reading unit for reading an original; a means for inputting an image read by the reading unit into a trained model trained based on an image not including characters and orientation information of the image; and a means for displaying information of the image read by the reading unit on the display unit based at least on an output result of the trained model. A means for setting in advance a process to be performed on an image having an incorrect orientation among the images read by the reading unit according to the type of the image. The present invention is characterized in that it has the following features. Effect of the Invention

[0007] According to the present invention, it is possible to distinguish the top and bottom of a document that does not contain characters. The process to be performed on images having an incorrect orientation among the images read by the reading unit can be set in advance according to the type of image. It is possible to provide an image processing device. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating a system configuration. [Diagram 2] FIG. 2 is a diagram illustrating a hardware configuration of an image forming apparatus. [Diagram 3] Fig. 3(a) is a diagram showing the hardware configuration of a learning server, Fig. 3(a) is a diagram showing the hardware configuration of a data server, Fig. 3(a) is a diagram showing the hardware configuration of a general-purpose computer. [Figure 4] FIG. 2 is a diagram illustrating a software configuration of the system. [Diagram 5] FIG. 1 is a diagram illustrating a learning model. [Figure 6] FIG. 11 is a diagram illustrating a usage sequence. [Figure 7] FIG. 13 is a diagram showing learning data. [Figure 8] FIG. 13 is a diagram showing teacher data. [Figure 9] FIG. 13 is a diagram showing a process flow relating to learning. [Figure 10] Fig. 10(a) is a diagram showing a list screen of scanned images, Fig. 10(b) is a diagram showing a screen presenting a correction suggestion for scanned images, and Fig. 10(c) is a diagram showing a list screen of scanned images. [Figure 11]Fig. 11(a) is a diagram showing a flow of the reading process, and Fig. 11(b) is a diagram showing a flow of the estimation process. [Figure 12] Fig. 12(a) is a diagram showing an example of a landscape image, Fig. 12(b) is a diagram showing an example of a landscape image, Fig. 12(c) is a diagram showing an example of a landscape image, and Fig. 12(d) is a diagram showing an example of a landscape image. [Figure 13] Fig. 13(a) is a diagram showing a setting screen for upside-down estimation, and Fig. 13(b) is a diagram showing a list screen of scanned images. [Figure 14] FIG. 13 is a diagram showing a flow of an estimation process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the embodiment of the present invention will be described in detail with reference to the drawings. Note that the scope of the present invention is not limited to the configurations described in the embodiments. Modifications such as replacing part of the configuration with an equivalent or omitting part of the processing may be made within the scope of obtaining the same effect.

[0010] Example 1 <System configuration> 1 is a diagram showing an image processing system 100. The image processing system 100 includes an image forming apparatus 101 and a machine learning server 102.

[0011] The image forming apparatus 101 is equipped with an AI function for estimating the top and bottom of a scanned image, and can output a result based on the top and bottom estimation in response to a document input from a user. A trained model for realizing this AI function is received from a machine learning server 102.

[0012] The machine learning server 102 performs learning processing of a learning model to realize an AI function. The data required for this learning processing is transmitted from the data server 105 on each occasion.

[0013] The data server 105 collects and generates data to be provided to the machine learning server 102. For example, the data server 105 also receives data provided from image forming apparatuses at any time.

[0014] FIG. 6 is a diagram illustrating a system usage sequence.

[0015] In step 601 (hereinafter referred to as S601), the data server 105 collects data that can be used for learning from devices on the network, including the image forming apparatus 101. Examples of data that can be used for learning include the results of a user's correction to the upside-down estimation process described below, and the image. In addition, the results of the image formed by the image forming apparatus 101 converting PDL data sent from the general-purpose computer 103 into an image may also be used as data that can be used for learning. In this case, all images based on the PDL data are treated as images with the correct upside-down orientation.

[0016] In S602, the data server 105 transmits to the machine learning server 102 data suitable for learning purposes from among the collected data.

[0017] In S603, the machine learning server generates learning data based on the data received from the data server 105, and uses the generated learning data to train the learning model. Note that the learning data may be generated in advance on the data server 105 side.

[0018] In S604, the image forming apparatus 101 is instructed to update the trained model used for orientation estimation.

[0019] In S605, the image forming apparatus 101 requests the machine learning server 102 for the latest trained model.

[0020] In S606, the machine learning server 102 transmits the trained model to the image forming apparatus 101.

[0021] In S651, the user places an original on the original reading section of the image forming apparatus and issues a reading instruction.

[0022] In S652, the image forming apparatus 101 executes a process of reading image information from the set document.

[0023] In S653, the image forming apparatus 101 performs top / bottom estimation processing on the acquired image.

[0024] In S654, the image forming apparatus 101 presents the estimation result to the user.

[0025] In S655, the user issues an instruction to correct the image based on the result of orientation estimation.

[0026] In S656, the image forming apparatus 101 executes a job after correcting the image. Here, the job can be a copy job, a save job, or a transmission job. In a copy job, an image is printed using the printer 20. In a save job, an image is stored in the HDD 204, which is an internal storage of the apparatus, or in an external storage such as a USB memory. In a transmission job, an image is sent to an external destination such as a fax destination, an e-mail destination, a shared folder, or a cloud service.

[0027] In S657, the image forming apparatus 101 provides a deliverable to the user. For example, in the case of a copy job, a printed sheet is provided, and in the case of a save job or a send job, the result of the process is presented to the user.

[0028] <Image forming device> FIG. 2 is a diagram showing the hardware configuration of the image forming apparatus 101. As shown in FIG.

[0029] The image forming apparatus 101 is a multifunction peripheral (image processing apparatus) having an image forming function for forming an image on a sheet and a reading function for reading an original. The image forming apparatus 101 is communicably connected to a machine learning server and a data server 105 constituting an image processing system via a network 104. The image forming apparatus 101 can also receive print data from a general-purpose computer via the network 104 and execute image forming processing based on the print data. In this embodiment, the image forming apparatus 101 will be described as an electrophotographic apparatus. However, the image forming apparatus 101 may be an apparatus of another image forming method, such as an inkjet method.

[0030] The image forming apparatus 101 includes an operation unit 140 , a scanner 10 , a printer 20 , and a controller 200 .

[0031] The operation unit 140 is an operation device for the user to perform various operations. The operation unit 140 includes a display as a display unit that displays information, and a touch panel or hard keys as an input unit that accepts information input by the user.

[0032] The scanner 10 is a reading unit that reads image information from an original document. The scanner 10 includes a CPU that controls each component of the scanner 10, and an illumination lamp and a scanning mirror (not shown) for reading an original document.

[0033] The printer 20 is an image forming unit for forming an image on a sheet. The printer 20 includes a CPU for controlling each configuration of the printer unit, a photoconductor drum for performing image formation and fixing, and a fixing unit (not shown).

[0034] The controller 200 is a control unit that performs overall control of the operation of the image forming apparatus. The controller 200 has a configuration connected to a system bus 207 and a configuration connected to an image bus 212, and the system bus 207 and the image bus 212 are connected to each other so as to be able to communicate with each other via an Image Bus I / F 205 that functions as a bus bridge.

[0035] The controller 200 includes a GPU 291 , a CPU 201 , a RAM 202 , a ROM 203 , a HDD 204 , an operation unit I / F 206 , a scanner / printer communication I / F 208 , a network I / F 210 , a wireless communication I / F 270 , and a modem 211 , which are connected to a system bus 207 .

[0036] The CPU 201 is a processor that performs various arithmetic processing. The RAM 202 is a system work memory for the CPU 201 to operate, and is also an image memory for temporarily storing image data. The ROM 203 is a memory that stores a boot program executed by the CPU 201. The HDD 204 is a data storage that stores system software, image data, software counter values, and the like. The controller 200 records and manages the history of job execution, such as user name, number of copies to be printed, color printing, and other output attribute information, as job log information, in the HDD 204 or the RAM 202 when a print or copy job is executed. The operation unit I / F 206 has a role of transmitting information input by a user who uses this image forming apparatus from the operation unit 140 to the CPU 201. The network I / F 210 is connected to the network 104, and communicates (transmits and receives) with the general-purpose computer 103 and other computer terminals (not shown) on the network 104. The modem 211 is connected to a public line, and communicates (transmits and receives) data with an external facsimile device (not shown). The wireless communication I / F 270 wirelessly connects to the network 104. The internal communication I / F 208 communicates with the scanner 10 and the printer 20.

[0037] The GPU 291 is a processor capable of parallel processing of more data. The GPU 291 can be effectively used for applications in which learning is performed multiple times, such as deep learning. In this embodiment, the GPU 291 is used in addition to the CPU 201 for processing using a learning model. Specifically, when executing a learning program including a learning model, the CPU 201 and the GPU 291 cooperate to perform calculations to perform learning. The processing of the machine learning unit 414 may be performed only by the CPU 201 or the GPU 291. The processing of the estimation processing unit 405 may also be performed by the GPU 291, as in the processing of the machine learning unit 414.

[0038] The controller 200 includes, as components connected to an image bus 212 , a RIP 260 , a scanner image processing unit 280 , a printer image processing unit 290 , an image rotation unit 230 , an image compression unit 240 , and a device I / F 220 .

[0039] The RIP 260 is a processor for converting PDL code into a bitmap image. The scanner image processing unit 280 corrects, processes, and edits image data input from the scanner 10. The image rotation unit 230 rotates the image data. The printer image processing unit 290 corrects, converts the resolution, and so on, the image data to be output (printed) by the printer 20.

[0040] The image compression unit 240 performs compression and expansion processing for multi-value image data in JPEG format and for binary image data in JBIG, MMR, or MH format. The device I / F 220 communicates with the scanner 10 and the printer 20.

[0041] <Machine learning server> FIG. 3(a) is a diagram showing the configuration of the machine learning server.

[0042] The machine learning server 102 is a server that performs learning and distribution of a learning model.

[0043] The machine learning server 102 includes a CPU 301, a ROM 302, a RAM 304, a storage 305, an input device 306, a display device 307, a network I / F 308, and a GPU 309. Each component can transmit and receive data to and from each other via a system bus 303.

[0044] The CPU 301 is a processor for performing calculations that control the entire machine learning server 102. The CPU 301 provides various functions by reading and executing programs such as an OS (Operating System) and application software from storage 305. The RAM 304 is a system work memory when the CPU 301 executes the programs. The ROM 302 stores a BIOS (Basic Input Output System), programs for starting the OS, and setting files. The storage 305 is a storage unit that stores information such as a hard disk drive, and stores system software and the like.

[0045] The input device 306 is an input device such as a mouse, a keyboard, etc. The display device 307 is a display device that displays information, such as a display.

[0046] The GPU 309 is an image processing processor. The GPU 309 can be effectively used for applications in which learning is performed multiple times, such as deep learning. Therefore, in the first embodiment, the GPU 309 is used in addition to the CPU 301 for processing by the machine learning unit 414. Specifically, when a learning program including a learning model is executed, the CPU 301 and the GPU 309 cooperate to perform calculations to perform learning. The processing of the machine learning unit 414 may be performed only by the CPU 301 or the GPU 309. The estimation processing unit 405 may also use the GPU 309 like the machine learning unit 414. Calculation processing such as learning may be performed not only by the machine learning server 102 but also in cooperation with other devices.

[0047] <Data servers, general-purpose computers> FIG. 3(b) is a diagram showing the configuration of a general-purpose computer.

[0048] The general-purpose computer 103 includes a CPU 331, a ROM 332, a RAM 334, a storage 335, an input device 336, a display device 337, a network I / F 338, and a GPU 339. Each component can transmit and receive data to and from each other via a system bus 333. Each component is similar to the configuration of the machine learning server 102, and therefore a description thereof will be omitted.

[0049] FIG. 3(b) is a diagram showing the configuration of the data server.

[0050] The data server 105 includes a CPU 361, a ROM 362, a RAM 364, a storage 365, an input device 366, a display device 367, a network I / F 368, and a GPU 369. Each component can transmit and receive data to and from each other via a system bus 363. Each component is similar to the component of the machine learning server 102, and therefore a description thereof will be omitted.

[0051] <System software configuration> Fig. 4 is a diagram showing a software configuration (400) realized by using the hardware resources and programs shown in the hardware configuration diagram of Fig. 2. The programs for realizing the software configuration shown in the diagram are stored in storage for each component, and are read into RAM and executed by the CPU. For example, in the image forming apparatus 101, a program stored in the HDD 204 and read into the RAM 202 is executed by the CPU 201. The same applies to the machine learning server 102 and the data server 105.

[0052] The software configuration in Fig. 4 is for enabling the system to realize a function for performing upside-down error estimation processing using learning data read by the image forming apparatus 101. The functional roles of the components in Fig. 4 will be described.

[0053] The software of the image forming apparatus 101 is composed of a data storage unit 402, a JOB control unit 403, a UI display unit 401, an image reading unit 404, and an estimation processing unit 405. The software of the machine learning server 102 is composed of a learning data generation unit 413, a machine learning unit 414, and a data storage unit 415. The software of the data server 105 is composed of a data collection and provision unit 410 and a data storage unit 412.

[0054] The data storage unit 402 has the functional role of recording data input and output by the image forming apparatus 101, such as image data, learning data, and learning models, in the RAM 202 and HDD 204 in the hardware configuration shown in FIG.

[0055] The JOB control unit 403 has a central functional role of executing basic functions of the image forming apparatus 101, such as copying, faxing, and printing, based on user instructions, and of transmitting and receiving instructions and data between other software components in conjunction with the execution of these basic functions.

[0056] The UI display unit 401 has a functional role of accepting operation settings from the user on a screen provided in the operation unit 140 via the operation unit I / F 206 shown in Fig. 2 and providing the operation acceptance screen. The UI display unit 401 has a functional role of displaying a notification screen for notifying the user of messages such as an estimation result by the estimation processing unit 405 in the system.

[0057] The image reading unit 404 has a functional role of performing an operation of optically reading an original by the scanner 10 shown in FIG.

[0058] 2 and has a functional role of performing estimation processing, classification processing, and the like to realize an AI function for data input and output by the image forming apparatus 101. This estimation processing is performed based on instructions from a JOB control unit 403, and the result output from the estimation processing unit 405 is transmitted to the JOB control unit 403, making it possible to provide feedback to the user, such as by displaying a notification message on the UI display unit.

[0059] The data collection and provision unit 410 has a functional role of collecting and providing learning data for learning in the machine learning server 102. In the system, it has a functional role of receiving learning data including operation information of the image forming device 101 from the image forming device 101 and providing it to the machine learning server 102. In addition, the data may be collected from other image forming devices other than the image forming device 101, the general-purpose computer 103, or other data servers, making it possible to collect data necessary for performing the desired machine learning. The data storage unit 412 has a functional role of recording and managing the collected learning data.

[0060] The learning data generation unit 413 has a functional role of optimizing the learning data by processing the data received from the data server 105 into a form that can obtain effective learning results, such as removing unnecessary data that becomes noise, in order to obtain a desired learning effect. This is executed by the CPU 301 shown in FIG.

[0061] The data storage unit 415 temporarily records data received from the data server 105, generated learning data, and a learned model in the machine learning unit 414 in the RAM 304 or storage 305 shown in FIG.

[0062] The machine learning unit 414 performs machine learning using the learning data generated by the learning data generation unit 413 as input, utilizing the hardware resources GPU 309 and CPU 301 shown in FIG. 3 and a learning method using the learning model shown in FIG.

[0063] <Learning model> 5 is a conceptual diagram showing an input / output structure using a learning model in the machine learning unit 414 of the first embodiment, and illustrates a learning model using a neural network as an example. As an example for explaining the features of the system, learning data X related to the generation of a learning model for predicting upside-down errors using information of a scanned document as an input by this neural network is shown. The learning data X is divided into documents containing only images, documents containing only text, and documents containing both images and text. As input data, the orientation and composition of an object, text, etc. are input.

[0064] Specific algorithms for machine learning include, in addition to neural networks, nearest neighbor methods, naive Bayes methods, decision trees, and support vector machines. In addition, deep learning, which uses neural networks to generate features and connection weighting coefficients for learning, can also be used. Any of the above algorithms that can be used can be used as appropriate and applied to this embodiment.

[0065] The learning model may include an error detection unit and an update unit. The error detection unit obtains an error between the output data Y(4) output from the output layer of the neural network according to the input data X(2) input to the input layer and the teacher data T. Then, the error detection unit calculates a loss (L) representing the error between the output data from the neural network and the teacher data using a loss function.

[0066] The update unit updates the connection weighting coefficients between the nodes of the neural network, etc., based on the loss (L) obtained by the error detection unit so as to reduce the loss. The update unit updates the connection weighting coefficients, etc., using, for example, an error backpropagation method. The error backpropagation method is a technique for adjusting the connection weighting coefficients between the nodes of each neural network so as to reduce the above-mentioned error.

[0067] For the learning model (W), a large amount of training data is prepared that consists of a set of "input data with known correct values" and "correct values." Then, the weighting coefficients in the learning model (W) are adjusted so that the output when input data corresponding to this correct value is input is as close as possible to the correct value, thereby obtaining a highly accurate learning model (W). This is called the learning process, and a learning model that has been adjusted through the learning process is called a trained model. The training data to be prepared (a set of "input data with known correct values" and "correct values") will be as follows: A. Teacher data Input data (X): Objects, composition, and text information of manuscript data with the correct orientation Expected value (T): A (no top-bottom error) = 1, B (top-bottom error) = 0 The expected value (T) is the output (Y) value that indicates the "correct value" when "input data with a known correct value" is input.

[0068] If the orientation is correct, it is set to Y1, and if it is not, it is set to Y2. Also, in this embodiment, only the case where the document is upside down is described, but it goes without saying that the present invention can also be applied to a document rotated by 90 degrees.

[0069] Input data (X) for each training data set is input, and the calculated output (Y) and expected value (T) are used to calculate the loss (L) according to a specified loss function.

[0070] In this embodiment, the loss function is A. Teacher data Loss (L) = 1 - "Estimated probability of A for output (Y)" Let us assume that.

[0071] The weighting between each layer of the intermediate learning model (W) is adjusted so that this loss (L) approaches zero.

[0072] The adjusted learning model (W) is treated as a trained model and is implemented in the machine learning unit 414.

[0073] <Input data, training data> Fig. 7 illustrates an example of the configuration of the input data shown in Fig. 5. The types of input documents are roughly divided into three types: documents containing only text, documents containing only images, and documents containing both text and images (c). In this embodiment, images of all these types are trained in advance so that the upside-down orientation of all these images can be determined with one learning model.

[0074] Manuscript No. 2 is an example of a manuscript that contains only text and is upside down. No. 4 is an example of a manuscript that contains only images and is upside down. No. 6 is an example of a manuscript that contains both text and images and is upside down. No. 8 is the same as No. 4. In order to be able to distinguish between upside down for images that contain only images, two broad types of images are input and trained. One is a type of image that contains a subject (object), such as a car. By learning the structural characteristics of a car, such as having tires on the underside, and shading, it becomes possible to distinguish between upside down. The other is a type of image that does not contain objects, such as a landscape. For these types of images, training is carried out so that it can be judged from the composition.

[0075] FIG. 8 is a diagram illustrating an example of the training data.

[0076] An arbitrary page is called "manuscript data a" as the input (X), and the expected value (T) is "T1 (no upside-down error) = 1, T2 (upside-down error) = 0." The learning items are the object orientation, composition, and characters. By using such training data as information corresponding to the input data, the learning model can be trained.

[0077] <Machine learning server processing flow> 9, a process flow of the machine learning unit 414 and the estimation processing unit 405 will be described. This process flow is realized by the CPU 301 loading a program stored in the ROM 302 or the storage 305 into the RAM 304 and executing it.

[0078] The machine learning unit 414 periodically checks whether the learning data has been updated. When the machine learning unit 414 determines that the learning data has been updated, the machine learning unit 414 starts the learning phase. First, the machine learning unit 414 receives the learning data (S901). The learning data is the manuscript data described above. Then, the machine learning unit 414 performs machine learning based on the received data (S902). Then, the machine learning unit 414 determines whether the machine learning is complete (S903). If the machine learning is not complete, the machine learning unit 414 repeats the process of S902. If the machine learning is complete, the machine learning unit 414 notifies the estimation processing unit 405 of an update of the trained model (S904). Then, the machine learning unit 414 determines whether there is a transmission request for the trained model (S905). If there is not, the machine learning unit 414 repeats the process of S905 and waits for a transmission request from the estimation processing unit 405. If there is a transmission request, the machine learning unit 414 transmits the trained model to the estimation processing device (S906).

[0079] <Processing flow of the image forming device> Fig. 11(a) is a diagram showing a flow of the reading process, and Fig. 11(b) is a diagram showing a flow of the estimation process.

[0080] The flow of a scan-related job will be described with reference to the flowcharts of Figures 11(a) and 11(b). This flow is realized by the CPU 201 loading a program stored in the ROM 203 or HDD 204 into the RAM 202 and executing it.

[0081] The JOB control unit 403 judges whether the operation unit has received a command from the user to start a scan job (S1101). If no operation has been received, the process waits until an operation is received. If an operation is received, the process proceeds to S1102. The scanner 10 reads an original placed on the ADF or the platen glass (S1102). The JOB control unit 403 judges whether the scanner 10 has finished reading all the originals. If not, the process repeats until all the originals have been read. After all the originals have been read, the JOB control unit 403 stores the read data in the data storage unit (S1104). After temporarily storing the read data, the CPU 201 performs an estimation process to determine whether the page order is incorrect (S1105). Details of the estimation process will be described later. After the estimation process is completed, the process after reading (printing, data storage, data transmission, etc.) is executed (S1107), and the process ends.

[0082] The estimation process in S1105 will be described in detail.

[0083] The estimation processing unit 405 inputs the images read in S1102 into the trained model, starting from the first page (S1153). The estimation processing unit 405 acquires information (output result) indicating whether the upside-down of the document is incorrect from the trained model. If the upside-down is incorrect, that is, if correction is required (S1154: YES), the estimation processing unit 405 adds the target image to the correction candidate (S1155). If the upside-down is not incorrect, that is, if correction is not required (S1154: NO), the estimation processing unit 405 skips the process of S1155. After that, the estimation processing unit 405 checks whether all pages have been input into the trained model (S1156). If there are pages that have not yet been input (S1156: NO), the estimation processing unit 405 proceeds to the process of S1157. In S1157, the estimation processing unit 405 inputs the next image of the processed page into the trained model, and returns the process to S1154. If input for all pages has been completed (S1156: YES), the estimation processing unit 405 advances the process to S1158. In S1158, the estimation processing unit 405 displays on the operation unit 140 an operation screen that takes into account the estimation result.

[0084] <Screen after estimation> Fig. 10(a) is a diagram showing a list screen of scanned images, Fig. 10(b) is a diagram showing a screen presenting a correction suggestion for scanned images, and Fig. 10(c) is a diagram showing a list screen of scanned images.

[0085] In S1158, for example, a screen such as that shown in FIG.

[0086] A screen 1000 is a list screen of scanned images, and includes information 1001, a list 1002, an icon 1003, a selection frame 1004, a button 1005, a button 1006, and a button 1007.

[0087] List 1002 is a list of images read by the reading process. In this embodiment, multiple pages of images are displayed in thumbnail format, but they may be displayed in list format. In this embodiment, an icon 1003 is added to an image designated as a correction candidate by the estimation process. Icon 1003 is an icon indicating a correction candidate for the top and bottom. In this embodiment, when displaying list 1002, the correction candidate is pre-selected. The user is notified by a message in information 1001 that a correction candidate is in a selected state. A selection frame 1004 is superimposed on a selected image. Each time a selection operation is performed, each image is switched between a selected state and a non-selected state.

[0088] The button 1005 is a selection object for correcting the selected images collectively with the proposed correction obtained by the estimation process. When the button 1005 is selected while a candidate image for correction is selected, the screen transitions to the screen in FIG. 10(c).

[0089] A button 1006 is a selection object for transitioning to a screen for confirming a proposed correction for the selected image. When the button 1006 is selected with a candidate correction selected, transition is made to a screen for confirming the proposed correction.

[0090] A button 1007 is a selection object for continuing the process without making any corrections. When the button 1007 is selected, the screen transitions to a screen (not shown) for completing the job.

[0091] Screen 1010 is a screen for confirming the proposed revisions. Here, a screen for confirming the second image among the revision candidates is shown as an example. When the revision candidates are selected and button 1006 is selected, screen 1010 is displayed after the first image among the revision candidates has been confirmed.

[0092] Screen 1010 includes information 1011 , information 1012 , information 1013 , button 1014 , button 1015 , and button 1016 .

[0093] Information 1011 is a message that prompts the user to correct the document orientation.

[0094] Information 1012 is an information area for presenting a revision suggestion. In the information 1012, an image that has been rotated based on the output result of the trained model is arranged as a revision suggestion.

[0095] Information 1013 is information indicating the progress of the task of checking correction candidates, and in the illustrated example indicates that the second of four correction candidates is currently being checked.

[0096] Button 1014 is a selection object for deciding to correct the image to the proposed correction suggested in information 1012. When button 1014 is selected, a screen is displayed for confirming the third image among the correction candidates. Similarly, when the third and fourth images among the correction candidates have been confirmed, a screen is displayed as shown in FIG. 10(c).

[0097] Button 1015 is a selection object for deciding to modify in a way different from the proposed modification presented in information 1012. When button 1015 is selected, an editing screen (not shown) is displayed that allows editing such as rotating the diagram.

[0098] The button 1016 is a selection object for deciding not to correct the proposed correction.

[0099] A screen 1020 displays a list of scanned images after correction is reflected. The screen 1020 includes information 1021, a list 1022, an icon 1023, a button 1024, and a button 1025.

[0100] Information 1021 is a message that asks the user whether or not to end the correction and complete the job.

[0101] A list 1022 is a list of images after the correction process has been performed. In this embodiment, an icon 1023 is added to each image that has been subjected to the correction process.

[0102] An icon 1023 indicates that the top and bottom have been corrected.

[0103] A button 1024 is a selection object for redoing corrections. When the button 1024 is selected, a process for returning to the state of the screen 1000 is performed.

[0104] Button 1025 is a selection object for completing a job. When button 1025 is selected, the job is executed with the drawing modifications reflected. For example, in the case of a job to save an image, the modified image is saved. In the case of a job to transmit an image, the modified image is transmitted. In the case of a job to print an image, the modified image is printed.

[0105] <Notes> As described above, according to the system of the present embodiment, when the orientation of a scanned image is estimated to be incorrect, the user can be notified of this. Furthermore, the user can easily correct the image by simply checking the proposed correction. The corrected image can then be used for various purposes such as printing, saving, and transmitting. Furthermore, icons are attached to images estimated to have an incorrect orientation and to corrected images, making them easily identifiable from the list.

[0106] Example 2 In the first embodiment, an example was described in which one trained model is used to estimate the top and bottom of a text document and an image document, and the top and bottom are uniformly set as correction candidates. In the second embodiment, an example is described in which documents are classified by type, estimation is performed for each document type, and processing is performed for each document type. Note that each configuration of the system is the same as in the first embodiment, except for the configuration related to the above-mentioned key points. Therefore, the same reference numerals are used for the same configurations, and detailed description thereof is omitted.

[0107] <Estimated accuracy> 12(a) to (d) are diagrams showing examples of landscape images.

[0108] It has been confirmed that when trying to estimate the top and bottom of a document, the accuracy of the estimation varies depending on the type of document. For example, the estimation accuracy is high for documents that contain text, but tends to be low for documents that do not contain text. In particular, the estimation accuracy is low for documents that contain only images, such as landscape photographs, among documents that do not contain text.

[0109] Fig. 12(a) is a diagram showing an example of a landscape image, Fig. 12(b) is a diagram showing its mirror image, Fig. 12(c) is a diagram showing an example of a landscape image, Fig. 12(d) is a diagram showing its mirror image.

[0110] In the case of landscape photographs with a composition in which light is reflected on the surface of the water, as in Figures 12(a) and 12(c), there is little material to distinguish between top and bottom, so the estimation accuracy tends to be low. If learning is performed to estimate that items with high saturation, brightness, and hue are at the top, it is possible to estimate the top and bottom in Figures 12(a) and 12(b), but it is difficult to estimate the top and bottom in Figures 12(c) and 12(d).

[0111] <Type-specific settings> FIG. 13A shows a setting screen for upside-down estimation.

[0112] A screen 1300 is a setting screen for orientation estimation.

[0113] The screen 1300 includes information 1301 , a setting area 1302 , a setting area 1303 , a setting area 1304 , and a button 1305 .

[0114] Information 1301 is information indicating a guideline for the estimation accuracy. According to the information 1301, documents such as text and tables have a high estimation accuracy, non-landscape documents have the next highest estimation accuracy, and landscape documents have the lowest estimation accuracy. Based on this information, the user decides how to handle the estimation error.

[0115] Setting areas 1302, 1303, and 1304 are areas for setting the processing to be performed after top / bottom estimation for each document type. There are three setting items: "Estimate," "Automatically Select," and "Do Nothing." In Fig. 13(a), "Correct" is set in setting area 1302, "Automatically Select" is set in setting area 1303, and "Do Nothing" is set in setting area 1304.

[0116] When the orientation estimation process is performed in the state set as shown in FIG. 13(a), a screen such as that shown in FIG. 13(b) is displayed.

[0117] A screen 1310 is a screen for displaying a list of scanned images. The screen 1310 includes information 1311 and a list 1312.

[0118] List 1312 is a list of images read by the reading process. If an image becomes a correction candidate through the estimation process and is a target for "correct" processing, an icon corresponding to icon 1023 is added. If an image becomes a correction candidate through the estimation process and is not a target for "correct" processing, an icon corresponding to icon 1003 is added. Furthermore, in this embodiment, when displaying list 1312, if an image is a correction candidate and a target for "auto-select" processing, the image is automatically selected and selection frame 1004 is superimposed. In other words, by changing the settings of screen 1300, the processing state of the correction candidate when displaying screen 1310 can be changed.

[0119] <Processing flow of the image forming device> FIG. 14 is a diagram showing a flow of the estimation process.

[0120] In S1105, in the second embodiment, the estimation process is performed as follows.

[0121] The estimation processing unit 405 inputs the scanned images scanned in S1102 into the trained model (classification) in sequence, starting from the first page (S1401). The trained model (classification) is a trained model trained by inputting various images, similar to the trained model described in Fig. 5, but trained to be able to classify image types. Therefore, it differs in that label information such as image types "text", "non-landscape", "landscape", and "other" is used as training data.

[0122] The estimation processing unit 405 acquires label information corresponding to the image from the trained model (classification).

[0123] If the label information of "character" is obtained as a result of inputting the image (S1402: YES), the estimation processing unit 405 further inputs the read image to the trained model (character). The trained model (character) is a trained model trained based on various images including characters and orientation information. If a result indicating that there is an error in the top and bottom is obtained from the trained model (character) (S1404: YES), the estimation processing unit 405 links the processing settings for character images set on the screen 1300 to the read image. If a result indicating that the top and bottom is correct is obtained (S1404: NO), the estimation processing unit 405 proceeds to the process of S1414.

[0124] When the label information of "non-landscape" is acquired as a result of inputting the scanned image (S1402: NO, S1406: YES), the estimation processing unit 405 further inputs the scanned image to the trained model (non-landscape). The trained model (non-landscape) is a trained model trained based on various non-landscape images and orientation information. When a result indicating that there is an error in the upside-down direction is acquired from the trained model (non-landscape) (S1407: YES), the estimation processing unit 405 links the processing setting for non-landscape images set on the screen 1300 to the scanned image. When a result indicating that the upside-down direction is correct is acquired (S1407: NO), the estimation processing unit 405 proceeds to the process of S1414.

[0125] When the label information of "landscape" is acquired as a result of inputting the scanned image (S1402: NO, S1406: NO, S1410: YES), the estimation processing unit 405 further inputs the scanned image to the trained model (landscape). The trained model (landscape) is a trained model trained based on various landscape images and orientation information.

[0126] If a result indicating that the orientation is incorrect is obtained from the trained model (landscape) (S1404: YES), the estimation processing unit 405 links the processing settings for landscape images set on the screen 1300 to the scanned image. If a result indicating that the orientation is correct is obtained (S1404: NO), the estimation processing unit 405 proceeds to the process of S1414.

[0127] When the label information of "other" is acquired as a result of inputting the image (S1402: NO, S1406: NO, S1410: NO, S1410: NO), the estimation processing unit 405 advances the process to S1414.

[0128] After that, the estimation processing unit 405 checks whether all pages have been input to the trained model (S1414).

[0129] If there are pages that have not yet been input (S1414: NO), the estimation processing unit 405 advances the process to S1415. In S1415, the estimation processing unit 405 inputs the next scanned image of the processed page into the trained model (classification) and returns the process to S1402. If input of all pages has been completed (S1415: YES), the estimation processing unit 405 advances the process to S1416.

[0130] In S1416, the estimation processing unit 405 displays on the operation unit 140 an operation screen that takes into account the estimation result.

[0131] <Notes> As described above, according to the system of this embodiment, when the scanned image contains an image that is estimated to be in the wrong orientation, this can be presented to the user. In particular, since the document is classified by type and the top and bottom are estimated using a learning model for each document type, it is possible to improve the estimation accuracy. In addition, since the post-estimation processing can be set for each document type, it is easy to respond according to the estimation accuracy.

[0132] (Other Examples) The present invention is not limited to the above-mentioned embodiments, and various modifications (including organic combinations of the embodiments) are possible based on the spirit of the present invention, and are not excluded from the scope of the present invention. In other words, the present invention includes all configurations that combine the above-mentioned embodiments and their modifications.

[0133] In the first and second embodiments, examples have been described in which each screen is displayed on the operation unit 140, but the display destination of the information is not limited to the operation unit 140. For example, a similar screen may be displayed on a mobile terminal or a PC using VNC. Also, by providing a WEB page accessible from a WEB browser of a mobile terminal or a PC, the same information may be displayed on a mobile terminal or a PC.

[0134] In the second embodiment, three classification methods, "text", "non-landscape" and "landscape", have been described as methods for classifying images read from a document, but other classification methods, numbers of classifications and combinations may be used. For example, simple classification as "text" and "image" may be used. In this case, classification may be performed such that if the OCR result shows that text is included, it is classified as "text", and if not, it is classified as "image".

[0135] In the second embodiment, a trained model (characters) is used. However, an existing OCR technique may be used to determine the image orientation in which the most characters are detected by OCR as the correct orientation.

[0136] In addition, in the second embodiment, a trained model (characters) is used. However, an existing OCR technique may be used, and the image orientation in which the most characters are detected by OCR may be determined to be the correct orientation.

[0137] The present invention can also be realized by a process in which a program for realizing one or more functions of the above-mentioned embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for realizing one or more functions.

[0138] The present invention may be applied to a system consisting of multiple devices, or to an apparatus consisting of a single device. For example, a part of the software module may be configured to be executed on an external server, and the function may be realized by obtaining the results of processing on the external server.

[0139] In the embodiment, the role of the software configuration 400 is shared among three devices: the image forming device 101, the machine learning server 102, and the data server 105. However, the software configuration 400 may be realized by one device, or the role of the software configuration 400 may be realized by any other number of devices.

[0140] The abbreviations appearing in the examples have the following meanings:

[0141] ADF stands for Auto Document Feeder. ASIC stands for Application Specific Integrated Circuit. BIOS stands for Basic Input / Output System. CPU stands for Central Processing Unit. FAX stands for FACSIMILE. OS stands for Operating System. IF stands for Interface. GPU stands for Graphics Processing Unit. HDD stands for Hard Disk Drive. JBIG stands for Joint Bi-level Image Experts Group. JPEG stands for Joint Photographic Experts Group. MFP stands for Multi Function Peripheral. MH stands for Modified Huffman. MMR stands for Modified Modified Read. OCR stands for Optical character recognition. OS stands for Operating System. PC stands for Personal Computer. PDL stands for Page Description Language. RAM stands for Random-Access Memory. RIP stands for Raster Image Processor. ROM stands for Read Only Memory. UI stands for User Interface. USB stands for Universal Serial Bus. VNC stands for Virtual Network Computing. [Explanation of symbols]

[0142] 10. Scanner 20 Printers 101 Image forming device 102 Machine Learning Server 400 Software Configuration 405 Estimation processing unit 414 Machine Learning Department

Claims

1. A display unit for displaying information; A reading unit that reads a document; A means for inputting an image read by the reading unit into a trained model trained based on an image not including characters and orientation information of the image; A means for displaying information of an image read by the reading unit on the display unit based at least on an output result of the trained model; 13. An image processing apparatus comprising: a means for setting in advance a process to be performed on an image having an incorrect orientation among images read by said reading section, in accordance with a type of image.

2. 2. The image processing apparatus according to claim 1, further comprising: means for displaying an image rotated based on the output result on the display unit.

3. 3. The image processing apparatus according to claim 2, further comprising means for displaying the image rotated based on the output result and the image before the rotation side by side.

4. 4. The image processing apparatus according to claim 2, wherein the image rotated based on the output result is an image for presenting a revision proposal.

5. 5. The image processing apparatus according to claim 1, further comprising: a means for selecting information of at least one image on a screen displaying a list of information of the images read by the reading unit, based on the output result.

6. 6. The image processing device according to claim 1, further comprising: a means for rotating and arranging at least one image on a screen displaying a list of images read by the reading unit based on the output result.

7. 6. The image processing apparatus according to claim 1, further comprising: a means for associating a predetermined icon with at least one image on a screen displaying a list of images read by the reading unit, based on the output result.

8. 8. The image processing apparatus according to claim 7, wherein the predetermined icon is an icon for identifying an image having an incorrect image orientation.

9. 8. The image processing apparatus according to claim 7, wherein the predetermined icon is an icon for identifying an image whose orientation has been rotated based on the output result.

10. 10. The image processing apparatus according to claim 1, further comprising a means for collectively correcting images having an error in image orientation on a screen displaying a list of images read by the reading section.

11. 11. The image processing device according to claim 1, further comprising a further trained model for classifying the type of image.

12. The image processing device according to claim 11 , wherein the further trained model is capable of classifying at least one of images including text, landscape images, and non-landscape images.

13. 13. The image processing apparatus according to claim 1, wherein the image read by the reading unit is an image read for the purpose of storing it.

14. 13. The image processing apparatus according to claim 1, wherein the image read by the reading unit is an image read for the purpose of transmitting it to an external device.

15. 13. The image processing apparatus according to claim 1, wherein the image read by the reading section is an image read for the purpose of printing.

16. 16. The image processing apparatus according to claim 1, further comprising an image forming unit that forms an image on a sheet.

17. A method for controlling an image processing apparatus having a display unit that displays information and a reading unit that reads an original, comprising: A step of inputting an image read by the reading unit into a trained model trained based on an image not including characters and orientation information of the image; A step of displaying information of an image read by the reading unit on the display unit based at least on an output result of the trained model; A control method comprising a step of setting in advance a process to be performed on an image having an incorrect orientation among images read by the reading unit, in accordance with a type of the image.

18. A display unit for displaying information; A reading unit that reads a document; A means for inputting an image read by the reading unit into a trained model trained based on an image not including characters and orientation information of the image; A means for displaying information of an image read by the reading unit on the display unit based at least on an output result of the trained model; 2. An image processing system comprising: a unit for setting in advance a process to be performed on an image having an incorrect orientation among images read by said reading unit, in accordance with a type of image.

Images