Information processing system. image processing system, information processing method, and program

Abstract

To simplify user's setting operation related to a plurality of kinds of job executed by an apparatus which inputs image data.SOLUTION: An information processing system has an estimation part which estimates a job to be executed on the basis of second image data newly inputted by an apparatus, by using a learned model obtained by learning a relation between image data and the job on the basis of learning data in which first image data inputted by the apparatus is associated with a job executed on the first image data on the basis of an instruction by a user among two or more kinds of jobs executable by the apparatus.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to an information processing system, an image processing apparatus, an information processing method, and a program.

Background Art

[0002] In response to the multifunctionalization of devices that process images, such as multifunction printers, copiers, and printers, the number of setting items when operating the devices is also increasing. The increase in setting items may lead to an increase in the operation burden on the user.

[0003] Conventionally, based on a dataset in which image information is associated with operation information representing a user operation related to the image information, based on a learned model obtained by machine learning the relationship between the image information and the operation information, for the received image information as input, a technique for determining recommended operation information representing a recommended operation has been proposed (Patent Document 1).

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, the technique of Patent Document 1 determines a recommended operation for the image information selected by the user when printing is performed. Therefore, it is difficult to reduce the operation burden on the user across multiple types of jobs including jobs other than printing (printing, FAX transmission, email transmission, cloud storage, etc.).

[0005] The present invention has been made in view of the above points, and an object thereof is to simplify the user's setting work related to a plurality of types of jobs executed by a device that inputs image data.

Means for Solving the Problems

[0006] Therefore, in order to solve the above problems, an information processing system uses a learned model that has learned the relationship between image data and jobs based on learning data in which first image data input by a device is associated with a job executed on the first image data based on an instruction by a user among two or more types of jobs that the device can execute, and estimates a job to be executed based on second image data newly input by the device, and has an estimation unit.

Effect of the Invention

[0007] It is possible to simplify the user's setting work related to a plurality of types of jobs executed by a device that inputs image data.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Embodiments for Carrying Out the Invention

[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a diagram showing a configuration example of the information processing system in the first embodiment. In FIG. 1, the information processing system includes one or more image processing apparatuses 10 and an information processing apparatus 20. The image processing apparatus 10 is connected to the information processing apparatus 20 via a network such as a LAN (Local Area Network) or the Internet.

[0010] The image processing apparatus 10 is a device that inputs image data and can execute a job related to any one of a plurality of types of functions on the input image data (hereinafter referred to as "input image"). That is, the image processing apparatus 10 can execute two or more types of jobs. As an example of the plurality of functions, there are FAX transmission, email transmission, printing, OCR processing, and other image processing. The image processing apparatus 10 does not necessarily have to be able to execute jobs for all of these functions, as long as it can execute jobs for two or more types of functions. Note that a job refers to a process executed on an image, and the type of job depends on the type of function. Therefore, the fact that the image processing apparatus 10 can execute jobs related to two or more types of functions is equivalent to the fact that the image processing apparatus 10 can execute two or more types of jobs. Also, the unit of "function" may be equivalent to the unit of application. For example, when applications are deployed for each unit such as FAX transmission, email transmission, printing, OCR processing, and other image processing, these applications may be grasped as units of functions.

[0011] In the present embodiment, the image processing apparatus 10 executes a job estimated to be suitable for the input image by a machine learning model (hereinafter referred to as "job estimation model"). The job estimation model is, for example, a neural network including a CNN (Convolutional Neural Network). However, a learnable model other than the neural network may be used as the job estimation model.

[0012] The information processing apparatus 20 is one or more computers that perform learning of the job estimation model and estimation of jobs using the job estimation model for the input image in the image processing apparatus 10.

[0013] In addition, in the present embodiment, it is assumed that the image processing apparatus 10 is used in an organization such as a company for a specific purpose (e.g., business). Therefore, the image processing apparatus 10 inputs image data from a plurality of types of paper documents used for the purpose. However, the image processing apparatus 10 may be used by general consumers such as in a convenience store.

[0014] FIG. 2 is a diagram showing a hardware configuration example of the image processing apparatus 10 in the first embodiment. In FIG. 2, the image processing apparatus 10 includes hardware such as a controller 11, a scanner 12, a printer 13, a modem 14, an operation panel 15, a network interface 16, and an SD card slot 17.

[0015] The controller 11 includes a CPU 111, a RAM 112, a ROM 113, an HDD 114, an NVRAM 115, and the like. The ROM 113 stores various programs and data used by the programs. The RAM 112 is used as a storage area for loading programs and a work area for the loaded programs. The CPU 111 realizes various functions by processing the programs loaded into the RAM 112. The HDD 114 stores programs and various data used by the programs. The NVRAM 115 stores various setting information and the like.

[0016] Scanner 12 is hardware (image reading means) for reading image data from a document. Printer 13 is hardware (printing means) for printing print data on printing paper. Modem 14 is hardware for connecting to a telephone line and is used to execute transmission and reception of image data by FAX communication. Operation panel 15 is hardware provided with input means such as buttons for receiving input from a user, display means such as a liquid crystal panel, etc. The liquid crystal panel may have a touch panel function. In this case, the liquid crystal panel also serves as the function of input means. Network interface 16 is hardware for connecting to a network such as a LAN (regardless of whether it is wired or wireless). SD card slot 17 is used to read a program stored in SD card 80. That is, in image processing apparatus 10, not only the program stored in ROM 113 but also the program stored in SD card 80 can be loaded into RAM 112 and executed. Note that SD card 80 may be replaced by another recording medium (for example, a CD-ROM or a USB (Universal Serial Bus) memory, etc.). That is, the type of recording medium corresponding to the position of SD card 80 is not limited to a predetermined one. In this case, SD card slot 17 may be replaced by hardware according to the type of recording medium.

[0017] FIG. 3 is a diagram showing an example of the hardware configuration of information processing apparatus 20 in the first embodiment. Information processing apparatus 20 in FIG. 3 includes a drive device 200, an auxiliary storage device 202, a memory device 203, a processor 204, and an interface device 205, etc., which are mutually connected by bus B.

[0018] The program that realizes the processing in the information processing apparatus 20 is provided by a recording medium 201 such as a CD-ROM. When the recording medium 201 storing the program is set in the drive device 200, the program is installed from the recording medium 201 via the drive device 200 into the auxiliary storage device 202. However, the installation of the program does not necessarily have to be performed from the recording medium 201, and it may be downloaded from another computer via a network. The auxiliary storage device 202 stores the installed program and also stores necessary files, data, etc.

[0019] When an activation instruction for the program is given, the memory device 203 reads out and stores the program from the auxiliary storage device 202. The processor 204 is a CPU, a GPU (Graphics Processing Unit), or both a CPU and a GPU, and executes the functions related to the information processing apparatus 20 according to the program stored in the memory device 203. The interface device 205 is used as an interface for connecting to a network.

[0020] FIG. 4 is a diagram showing a functional configuration example of the information processing system in the first embodiment. In FIG. 4, the image processing apparatus 10 includes an authentication unit 121, an instruction reception unit 122, an image input unit 123, a blank paper removal unit 124, a job estimation request unit 125, an inquiry unit 126, a job execution unit 127, a job information recording unit 128, and an image recording unit 129. Each of these units is realized by processing executed by one or more programs installed in the image processing apparatus 10 by the CPU 111. The image processing apparatus 10 also uses a user information storage unit 151. The user information storage unit 151 can be realized, for example, using an HDD 114 or a storage device that can be connected to the image processing apparatus 10 via a network.

[0021] The authentication unit 121 authenticates the user who uses the image processing apparatus 10. The user information storage unit 151 is referred to during authentication. The user information storage unit 151 stores information necessary for authenticating a user who can use the image processing apparatus 10, the attribute information of the user, etc.

[0022] The instruction reception unit 122 receives an instruction to execute a job from the user. The instruction to execute a job refers to an instruction including an instruction to read image data from a document and a specification regarding the job to be executed on the image data. After the learning of the job estimation model, the instruction reception unit 122 receives an instruction to read image data from a document from the user. That is, after the learning of the job estimation model, the instruction reception unit 122 does not necessarily need to receive a specification regarding the job from the user.

[0023] The image input unit 123 controls the scanner 12 in accordance with setting information (hereinafter referred to as "scan setting") regarding the input (reading) of image data included in the instruction to execute a job or the instruction to read an image, and inputs (reads) image data from the paper document set on the scanner 12.

[0024] The blank page removal unit 124 removes blank pages from the image data input by the image input unit 123 after the learning of the job estimation model.

[0025] The job estimation request unit 125 requests the estimation unit 23 of the information processing apparatus 20 to estimate the job to be executed with respect to the image data input by the image input unit 123 (when the blank page removal unit 124 removes blank pages, the image data with the blank pages removed). Further, the job estimation request unit 125 receives the estimation result of the job transmitted from the estimation unit 23.

[0026] The inquiry unit 126 inquires the user whether the job estimated by the estimation unit 23 can be executed.

[0027] When an instruction to execute a job is input by the user, the job execution unit 127 executes the job related to the execution instruction, and after the learning of the job estimation model, executes the job estimated by the estimation unit 23.

[0028] During the learning of the job estimation model, the job information recording unit 128 records the job information of the job executed by the job execution unit 127 based on the job execution instruction input by the user in the job information storage unit 212 of the information processing apparatus 20. The job information is the information set by the user regarding the job.

[0029] During the learning of the job estimation model, the image recording unit 129 records the image data input by the image input unit 123 in the image storage unit 211 of the information processing apparatus 20.

[0030] The information processing apparatus 20 includes a learning data generation unit 21, a learning unit 22, and an estimation unit 23. Each of these units is realized by the processing executed by the processor 204 by one or more programs installed in the information processing apparatus 20. The information processing apparatus 20 also uses the image storage unit 211, the job information storage unit 212, and the model storage unit 213. Each of these storage units can be realized using, for example, an auxiliary storage device 202 or a storage device that can be connected to the information processing apparatus 20 via a network.

[0031] The learning data generation unit 21 generates learning data for the job estimation model by associating the image data stored in the image storage unit 211 with the job information stored in the job information storage unit 212 for each job. The image storage unit 211 stores the image data input by the image processing apparatus 10. The job information storage unit 212 stores the job (job information) executed on the image data based on the instruction of the user among two or more types of jobs that the image processing apparatus 10 can execute. Therefore, the learning data generated by the learning data generation unit 21 is learning data that associates the image data input by the image processing apparatus 10 with the job executed on the image data based on the instruction of the user among two or more types of jobs that the image processing apparatus 10 can execute.

[0032] The learning unit 22 causes the job estimation model to learn the relationship between the image data and the job based on the learning data generated by the learning data generation unit 21. The learning of the job estimation model means updating the parameters (model parameters) of the job estimation model. The learning unit 22 records the updated parameters in the model storage unit 213.

[0033] The estimation unit 23 estimates the job to be executed based on the image data newly input by the image processing apparatus 10 (image data related to the request from the job estimation request unit 125 of the image processing apparatus 10) using the job estimation model that has learned the relationship between the image data and the job.

[0034] FIG. 5 is a sequence diagram for explaining an example of the processing procedure executed by the information processing system during the learning of the job estimation model.

[0035] In step S101, the authentication unit 121 of the image processing apparatus 10 receives a login operation from the user via the login screen displayed on the operation panel 15. By this login operation, information necessary for user authentication, such as a user ID and a password, is input as login information. Note that information used by other known authentication methods, such as biometric information, may be input as login information.

[0036] Subsequently, the authentication unit 121 authenticates the user by comparing the input login information with the information stored in the user information storage unit 151 (S102).

[0037] FIG. 6 is a diagram showing a configuration example of the user information storage unit 151. As shown in FIG. 6, the user information storage unit 151 stores a user ID, which is user identification information, a password, a name, an affiliation, etc. for each user permitted to use the image processing apparatus 10.

[0038] The user ID is the identification information of the user. The password is the correct password registered for the user related to the user ID. The name is the name of the user related to the user ID. The affiliation is information indicating the organization to which the user related to the user ID belongs (for example, department name, etc.).

[0039] If the authentication unit 121 determines that the record including the pair of user ID and password that matches the input login information is stored in the user information storage unit 151, it determines that the authentication is successful, and identifies the user related to the user ID of the login information as the logged-in user. In this case, the authentication unit 121 records the content of the record including the user ID in the user information storage unit 151 as the logged-in user information in the RAM 112, and erases the login screen. As a result, the home screen is displayed on the operation panel 15. The home screen is a screen for receiving a designation of a job to be executed from among two or more types of jobs from the user. If the record including the pair of user ID and password that matches the input login information is not stored in the user information storage unit 151, the authentication unit 121 determines that the authentication has failed. In this case, the steps after step S103 are not executed.

[0040] Subsequently, the instruction reception unit 122 receives a job execution instruction from the user via the home screen (S103). In the job execution instruction, scan settings which are setting information regarding the input (reading) of image data, and setting information (hereinafter referred to as "processing settings") regarding the process (job) to be applied to the input image data (hereinafter referred to as "input image") are specified. Therefore, the operation for the job execution instruction includes one or more input operations for inputting the scan settings and the processing settings. The processing settings include information such as the application name of the application corresponding to the job to be applied to the input image, the type of process in the application (hereinafter referred to as "process type"), and the setting information (hereinafter simply referred to as "setting information") for the process by the application. The process type in the processing settings refers to the process selected by the user from among the processes executable by the application, and can also be said to be information for identifying the operation by the user for the application.

[0041] Subsequently, the image input unit 123 controls the scanner 12 according to the scan settings included in the job execution instruction, and inputs (reads) the image data (input image 9) from the paper document set on the scanner 12 (S104).

[0042] Subsequently, the job execution unit 127 executes a process (job) on the input image according to the process settings included in the job execution instruction (S105). For example, image processing is executed on the input image, data obtained by processing the input image or the input image is stored in a storage destination according to the process settings, or data obtained by processing the input image or the input image is transmitted to a transmission destination according to the process settings. If the content of the process settings is simply copying, copying is executed on the input image. Thus, the jobs executed on the input image vary according to the process settings.

[0043] Subsequently, the job execution unit 127 outputs information indicating the execution result of the job to the operation panel 15 (S106). The information may be information indicating the success or failure of the job, or may be information including data generated by the execution of the job (hereinafter referred to as "output data").

[0044] Subsequently, the image recording unit 129 associates the job ID with the input image and records them in the image storage unit 211 of the information processing apparatus 20 (S107). At this time, the image recording unit 129 may also record the output data generated by the execution of the job in the image storage unit 211 in association with the job ID and the input image. The job ID is identification information assigned to each job.

[0045] FIG. 7 is a diagram showing a configuration example of the image storage unit 211. As shown in FIG. 7, the image storage unit 211 stores image information including, for each job, a job ID, the image data of each page of the input image of the job, and the data of each page of the output data of the job (for example, image data) in association with each other.

[0046] For example, the image information of the first job (record No. 1) indicates that the first page of the output page is the result of aggregating and inverting the first and second pages of the three-page input image, and the second page of the output page is the result of inverting the third page of the input image as it is.

[0047] Subsequently, the job information recording unit 128 associates the job ID with the job information and records it in the job information storage unit 212 of the information processing apparatus 20 (S108). The job information refers to information related to the job, for example, information indicating "in what situation, with what settings, and which application was used to execute the job". Specifically, the job information includes information indicating the situation in which the job was executed (hereinafter referred to as "situation information") and processing settings, etc. Note that the situation information corresponding to "in what such situation" is, for example, information indicating "who, when".

[0048] FIG. 8 is a diagram showing a configuration example of the job information storage unit 212. As shown in FIG. 8, the job information storage unit 212 stores job information including a job ID, a login user ID which is user identification information, an execution date and time, an application name, a processing type, setting information, and an end state, etc. for each job. The execution date and time is the date and time when the job was executed. The application name is the name of the application that executed the processing on the input image. The processing type is the type of the job as described above. The setting information is the setting information for the job. The end state is information indicating in what state the job ended. For example, the end state is "normal end", "abnormal end", "aborted". "Normal end" indicates that the job ended normally. "Abnormal end" indicates that an abnormality occurred during the execution of the job and the job was not executed normally. "Aborted" indicates that the execution of the job was aborted according to an instruction by the user.

[0049] In such job information, the login user ID and the execution date and time constitute the situation information, and the combination of the application name, the processing type, and the setting information constitutes the processing setting.

[0050] Subsequently, the job information recording unit 128 transmits an update notification indicating that the image storage unit 211 and the job information storage unit 212 have been updated to the learning data generation unit 21 of the information processing apparatus 20 (S109).

[0051] In response to receiving the update notification (or every time the update notification is received a predetermined number of times), the learning data generation unit 21 executes steps S110 to S112.

[0052] In step S110, the learning data generation unit 21 acquires unlearned (not used in the learning process) image information (such as job ID, input image, and output data) from the image storage unit 211. Subsequently, the learning data generation unit 21 acquires unlearned job information from the job information storage unit 212 (S111). Subsequently, the learning data generation unit 21 generates learning data by combining the image information acquired from the image storage unit 211 and the job information acquired from the job information storage unit 212 based on the job ID (S112).

[0053] FIG. 9 is a diagram showing a configuration example of learning data. As shown in FIG. 9, the learning data is data obtained by combining image information (FIG. 7) and job information (FIG. 8) having a common job ID. However, the learning data generation unit 21 does not generate learning data including job information with an "end state" of "abnormal end" or "aborted" and image information having the same job ID as the job information. This is because the processing settings included in such job information are likely to be incorrect settings. Note that one line of data in FIG. 9 corresponds to one piece of learning data.

[0054] Note that the learning data in FIG. 9 also includes an "affiliation" that does not include job information. The affiliation is the organization or department to which the user belongs, which is stored in the user information storage unit 151 (FIG. 6) for the login user ID. In this way, the learning data generation unit 21 can include not only the data included in the job information or the image information, but also the data that can be obtained or derived from the data included in the job information or the image information in the learning data. Note that the "affiliation" is information regarding the user who executed the job and is information constituting the situation information.

[0055] Subsequently, the learning data generation unit 21 transmits the generated set of learning data (hereinafter referred to as the "learning data set") to the learning unit 22 (S113). The learning unit 22 executes learning processing of the job estimation model using the learning data (S114). In this learning processing, the learning unit 22 causes the job estimation model to learn the relationship between the data corresponding to the input and the data corresponding to the output in the learning data. Here, the data corresponding to the output in the learning data is the processing setting (application name, processing type, setting information), and is used as the correct data.

[0056] The learning unit 22 inputs the data corresponding to the input in the learning data into the job estimation model, updates the parameters of the job estimation model based on the comparison between the output from the job estimation model and the correct data in the learning data, and stores the updated parameters in the model storage unit 213 (S115). The learning unit 22 calculates an error (loss) with respect to the correct data regarding the output from the job estimation model based on the comparison. For example, if the job estimation model outputs a probability distribution of the setting values for each item constituting the processing setting, the learning unit 22 calculates the error between the probability distribution and the probability distribution of the setting values of each item corresponding to the correct data. The probability distribution of the settings of each item corresponding to the correct data refers to, for example, a probability distribution in which the probability of the correct value is 1 and the probability of other values is 0 for the setting values that each item constituting the processing setting can take. The learning unit 22 updates the parameters of the job estimation model using a known technique such as the error backpropagation method based on the error.

[0057] On the other hand, there are multiple variations (hereinafter referred to as "input patterns") for the input to the job estimation model.

[0058] Input pattern 1 is a pattern that takes only the input image as input. In this case, the job estimation model learns the relationship between the input image and the output (processing settings). A correlation is recognized between the content of the paper document that is the source of the input image and the processing that the user wants to perform on the input image. Specifically, depending on the type of document (contract, invoice, development specification, presentation material, etc.), the processing that the user wants to perform (the purpose of using the image processing apparatus 10) tends to be determined.

[0059] The processing settings are determined by the processing to be performed. The job estimation model for input pattern 1 learns the relationship between input and output based on such a tendency. Note that when the color in the document does not affect the processing settings, the learning unit 22 may use the binarized input image as the input to the job estimation model. This also applies to the following other patterns.

[0060] Input pattern 2 is a pattern that takes, in addition to the input image, the login user ID, which is user identification information among the situation information, as input. In input pattern 2, for example, it is effective when the setting of some parameters of the processing settings varies depending on the user's preference.

[0061] Input pattern 3 is a pattern that takes, in addition to the input image, the organization (department) to which the input image belongs among the situation information as input. For each department in a company, the type of document handled tends to be different, and the processing that the user wants to execute on the document handled according to the department tends to be different. In other words, the processing that a certain department causes the image processing apparatus 10 to execute tends to be almost constant. The job estimation model for input pattern 3 will learn such a tendency.

[0062] Input pattern 4 is a pattern that takes, in addition to the input image, the execution date and time (execution timing) when the input image was input among the situation information as input. In input pattern 3, for example, it is an effective pattern when the timing (time zone) of executing the job affects the processing settings.

[0063] The input pattern 5 is a combination of two or more patterns among the input patterns 2 to 4.

[0064] Note that the learning unit 22 may learn the job estimation model for each group according to user ID, affiliation, execution date and time (time zone), or for each group according to a combination of two or more of user ID, affiliation, and execution date and time (time zone) (that is, for each situation where the job is executed). In this case, the learning unit 22 classifies the learning data for each group, and learns the job estimation model corresponding to the group using the learning data classified into the group for each group. Therefore, the parameters of the job estimation model are stored in the model storage unit 213 for each group. In this case, the data (user ID, affiliation, or execution date and time) serving as the grouping criterion may not be included in the input to the job estimation model. This is because the job estimation model is learned with the data distinguished.

[0065] Next, the estimation of the job using the learned job estimation model will be described. The estimation of the job means estimating the processing setting for the input image.

[0066] FIG. 10 is a sequence diagram for explaining an example of the processing procedure executed by the information processing system at the time of job estimation. In FIG. 10, the same step numbers are assigned to the same steps as in FIG. 5, and the description thereof will be omitted as appropriate.

[0067] From the user's login (S101) to the input of the image (S104), it is substantially the same as the content described in step S101, but step S103 is replaced with S103a.

[0068] In step S103a, the instruction reception unit 122 receives from the user an instruction to read an image from the document. In the image reading instruction, it is sufficient to input the scan settings. That is, the input of the processing settings is not required. In step S104, an image is input from the paper document based on the scan settings. Hereinafter, the image will be referred to as the "target image".

[0069] Subsequently, the blank page removal unit 124 executes blank page removal processing on the target image (S201). The blank page removal processing refers to the processing of removing pages that are blank pages. Whether a certain page is a blank page can be determined by, for example, whether the ratio of pixels with luminance values less than a certain value in the page is equal to or greater than a threshold value, or whether all pixel values of the image corresponding to the page indicate white or are constant. By the blank page removal processing, the user can always set both sides in step S103a without being aware of whether the paper document is single-sided or double-sided, which can reduce the user's operation burden. However, the blank page removal processing may not be executed.

[0070] Subsequently, the job estimation request unit 125 sends a job estimation request to the information processing apparatus 20 (S202). The job estimation request includes input data corresponding to the input pattern of the job estimation model. When the input pattern of the job estimation model is input pattern 1, the estimation request includes the target image as the input data. When the input pattern of the job estimation model is input pattern 2, the estimation request includes the target image and the login user ID as the input data. When the input pattern of the job estimation model is input pattern 3, the estimation request includes the target image and the affiliation of the logged-in user as the input data. The affiliation is specified in step S102. When the input pattern of the job estimation model is input pattern 4, the estimation request includes the target image and the execution date and time (current date and time) when the target image was input as the input data. When the input pattern of the job estimation model is input pattern 5, the estimation request includes the data input in the input pattern 5 as the input data.

[0071] Also, when the job estimation model is generated for each user ID, each affiliation, each group by execution date and time (time zone), or each group by a combination of two or more of user ID, affiliation, and execution date and time (time zone) (i.e., for each situation), in addition to the target image, the estimation request includes, as a situation identifier, data necessary for distinguishing groups (distinguishing situations) (login user ID, the affiliation of the login user or the current date and time, or a combination of two or more of these).

[0072] When the estimation unit 23 of the information processing apparatus 20 receives the estimation request, it acquires the learned parameters from the model storage unit 213 and constructs (loads into the memory device 203) the job estimation model with the parameters set (i.e., the learned job estimation model) (S203). When the job estimation model is generated for each group, the estimation unit 23 may acquire the parameters corresponding to the situation identifier included in the estimation request from among the parameters stored for each group.

[0073] Subsequently, the estimation unit 23 estimates the processing settings for the input data using the job estimation model (S204). Specifically, the estimation unit 23 inputs the input data to the job estimation model. The estimation unit 23 estimates the processing settings based on the output from the job estimation model. For example, if the output from the job estimation model is the probability distribution of the set values of each item constituting the processing settings, the estimation unit 23 sets the set of the set values with the highest probability for each item as the estimated value of the processing settings.

[0074] Subsequently, the estimation unit 23 transmits the processing settings as the estimation result to the job estimation request unit 125 (S205).

[0075] When the job estimation request unit 125 receives the estimation result, the inquiry unit 126 inquires of the user whether the job can be executed based on the estimation result by displaying the estimation result on the operation panel 15 (S206). At this time, the inquiry unit 126 may display, for example, a message for notifying the user that the job is about to be executed based on the automatically estimated processing settings. The inquiry unit 126 may also display a screen corresponding to the application related to the application name included in the estimation result (that is, the screen on which the user makes settings when using the application), and the screen in a state where the processing settings included in the estimation result are set. By doing so, the user can check the processing settings by referring to the familiar screen. At this time, the setting values of some items may be modified by the user. When a modification is made, the inquiry unit 126 reflects the modification content in the estimation result. By doing so, when the estimation result is different from the user's intention, it is possible to avoid the execution of a job different from the user's intention. Note that even when the estimation result is not completely correct, the setting operation by the user can be limited to some items, so that the user's setting work can be simplified.

[0076] Thereafter, when an execution instruction for a job based on the processing settings related to the estimation result or the processing settings in which the modification to the estimation result is reflected is input by the user (S207), the job execution unit 127 executes a process (job) on the input image according to the processing settings (S208). Subsequently, the job execution unit 127 outputs information indicating the execution result of the job to the operation panel 15 in the same manner as step S106 in FIG. 5 (S209).

[0077] As described above, according to the first embodiment, based on the image data input by the image processing apparatus 10, the information processing system automatically estimates a job to be executed using a job estimation model in which the relationship between the image data and a plurality of types of jobs is learned. Therefore, the setting work related to the job by the user can be simplified. That is, the setting work of the user related to a plurality of types of jobs executed by the device that inputs the image data can be simplified.

[0078] Next, a second embodiment will be described. In the second embodiment, differences from the first embodiment will be described. Therefore, points not particularly mentioned may be the same as those in the first embodiment.

[0079] FIG. 11 is a diagram showing a functional configuration example of the information processing system in the second embodiment. In FIG. 11, the same reference numerals are given to the same components as in FIG. 4, and the description thereof is omitted.

[0080] In FIG. 11, the image processing apparatus 10 further includes a notification unit 130. The notification unit 130 is realized by processing that causes the CPU 111 to execute one or more programs installed in the image processing apparatus 10.

[0081] In the second embodiment, even after the learning of the job estimation model, it is assumed that the user designates a job (inputs a process setting) of his or her own will. Even in such a case, the estimation unit 23 of the information processing apparatus 20 estimates the job to be executed with respect to the image data input by the image processing apparatus 10. The notification unit 130 outputs a predetermined notification based on a comparison between the job designated by the user with respect to the image data and the job estimated by the estimation unit 23. For example, the notification unit 130 gives a predetermined notification when the job designated by the user is different from or deviated from the job estimated by the estimation unit 23. The predetermined notification may be, for example, a warning indicating that there may be an error in the designation (process setting) by the user.

[0082] FIG. 12 is a diagram for explaining an example of a processing procedure executed by the information processing system in the second embodiment. In FIG. 12, the same step numbers are given to the same steps as in FIG. 5 or FIG. 10, and the description thereof is appropriately omitted. It is assumed that the job estimation model has been learned at the start of the processing procedure in FIG. 12.

[0083] First, similar to FIG. 5, from the user's login to the reading of image data is executed (S101 to S104). At this time, in step S103, a job execution instruction (job specification) is performed in the same manner as in FIG. 5. That is, the specification of the process settings is also performed by the user.

[0084] Subsequently, similar to FIG. 10, steps S202 to S205 are executed, and a job (process setting) for the image data is estimated.

[0085] Subsequently, the notification unit 130 compares the job (process setting) specified by the user in step S103 with the job (process setting) estimated by the estimation unit 23 in step S204, and determines whether there is a difference that satisfies a predetermined condition between the two (S301). The predetermined condition may be, for example, that the two do not exactly match, or that for one or more items including setting values with high probability (threshold or more) in the probability distribution of the setting values of each item constituting the process setting output by the job estimation model, the two are different. Other conditions may be set.

[0086] If there is a difference that satisfies the predetermined condition, the notification unit 130 issues a predetermined notification to the operation panel 15 (S302). In this case, the subsequent processing procedure may not be executed. That is, the job may not be executed.

[0087] If there is no difference that satisfies the predetermined condition, for example, by executing S105 to S115 in FIG. 5, the job estimation model may be additionally learned.

[0088] According to the second embodiment, for example, when the process setting by the user deviates from the normal business (for example, usually, invoices are sent to the General Affairs Department, but when trying to accumulate them in the business folder), the error can be detected and the execution of an incorrect job can be avoided.

[0089] Next, a third embodiment will be described. In the third embodiment, differences from the first embodiment will be described. Therefore, points not particularly mentioned may be the same as those in the first embodiment.

[0090] FIG. 13 is a diagram showing a functional configuration example of the information processing system in the third embodiment. In FIG. 13, the same parts as those in FIG. 4 are denoted by the same reference numerals, and the description thereof is omitted.

[0091] In FIG. 13, the image processing apparatus 10 has an estimation unit 23. Further, the image processing apparatus 10 has a model storage unit 152. The model storage unit 152 stores a copy of the parameters stored in the model storage unit 213. Therefore, in the third embodiment, job estimation is performed on the side of the image processing apparatus 10. At the time of job estimation, the estimation unit 23 uses the parameters stored in the model storage unit 152.

[0092] Note that in the third embodiment, the image processing apparatus 10 may have the notification unit 130 shown in FIG. 11.

[0093] Next, a fourth embodiment will be described. In the fourth embodiment, differences from the first embodiment will be described. Therefore, points not particularly mentioned may be the same as those in the first embodiment.

[0094] FIG. 14 is a diagram showing a functional configuration example of the information processing system in the fourth embodiment. In FIG. 14, the same parts as those in FIG. 4 are denoted by the same reference numerals, and the description thereof is omitted.

[0095] In FIG. 14, the image processing apparatus 10 has all the functions that the information processing apparatus 20 had in FIG. 4. That is, the image processing apparatus 10 alone constitutes the information processing system. Therefore, in the fourth embodiment, learning of the job estimation model and job estimation are performed on the side of the image processing apparatus 10.

[0096] Note that in the fourth embodiment, the image processing apparatus 10 may have the notification unit 130 shown in FIG. 11.

[0097] Note that each function of the embodiments described above can be realized by one or more processing circuits. Here, the "processing circuit" in this specification refers to a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, an ASIC (Application Specific Integrated Circuit) designed to execute each function described above, a DSP (digital signal processor), an FPGA (field programmable gate array), and devices such as conventional circuit modules.

[0098] Also, the device groups described in the above embodiments merely represent one of a plurality of computing environments for implementing the embodiments disclosed in this specification.

[0099] In one embodiment, the information processing apparatus 20 includes a plurality of computing devices such as a server cluster. The plurality of computing devices are configured to communicate with each other via an arbitrary type of communication link including a network or a shared memory, and perform the processing disclosed in this specification. Similarly, the image processing apparatus 10 can include a plurality of computing devices configured to communicate with each other.

[0100] As described above in detail regarding the embodiments of the present invention, the present invention is not limited to such specific embodiments, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims.

[0101] Aspects of the present invention are, for example, as follows. <1> An estimation unit that estimates a job to be executed based on second image data newly input by the device, using a learned model that has learned the relationship between image data and jobs based on learning data in which first image data input by the device is associated with a job executed on the first image data based on an instruction by a user among two or more types of jobs executable by the device. An information processing system characterized by having the above. <2> A job execution unit that executes the job estimated by the estimation unit. The information processing system according to <1>, characterized by having the above. <3> The estimation unit estimates a job to be executed based on the second image data newly input by the device and the information at the time when the second image data is input, using a learned model that has learned the relationship between image data, the information, and jobs based on learning data in which the first image data is associated with information indicating a situation where a job has been executed on the first image data and the job executed on the first image data. The information processing system according to <1> or <2>, characterized by the above. <4> The information indicating the situation is identification information of a user who uses the device. The information processing system according to <3>, characterized by the above. <5> The information indicating the situation is information regarding an organization to which a user who uses the device belongs. The information processing system according to <3>, characterized by the above. <6> The information indicating the situation is information regarding the time when the second image data is input. The information processing system according to <3>, characterized by the above. <7> It has a blank page removal unit that removes blank pages from the second image data. The estimation unit estimates a job to be executed based on the second image data from which blank pages have been removed, using the learned model. An information processing system according to any one of <1> to <3>, characterized in that... <8> Based on the learned model in which the estimation unit has learned the relationship between image data and jobs for each situation in which the job is executed, from the learned data in which the first image data and the job executed on the first image data are associated, the estimation unit estimates the job to be executed based on the second image data newly input by the device and the learned model corresponding to the situation when the second image data is input. An information processing system according to any one of <1> to <4>, characterized in that... <9> An inquiry unit that inquires the user about the executability of the job estimated by the estimation unit. An information processing system according to any one of <1> to <5>, characterized by having... <10> A notification unit that outputs a predetermined notification based on a comparison between the job specified by the user for the second image data and the job estimated by the estimation unit. An information processing system according to any one of <1> to <6>, characterized by having... <11> An information processing system including an image processing device and an information processing device, wherein... The information processing device... Using a learned model that has learned the relationship between image data and jobs based on learned data in which the first image data input by the image processing device and the job executed on the first image data among two or more types of jobs executable by the image processing device based on an instruction from the user are associated, an estimation unit estimates the job to be executed based on the second image data newly input by the image processing device. And has... The image processing device... A job execution unit that executes the job estimated by the estimation unit. An information processing system characterized by having... <12> Based on the learning data associating the first image data input by the device with the job executed on the first image data based on an instruction from the user among two or more types of jobs executable by the device, a learning unit 22 that learns the relationship between the image data and the job in a model An information processing system characterized by having the above <13> A model storage unit that stores a learned model that has learned the relationship between image data and jobs based on learning data associating the first image data input by the device with the job executed on the first image data based on an instruction from the user among two or more types of jobs executable by the device An estimation unit that estimates the job to be executed based on the second image data newly input by the device using the learned model An image processing apparatus characterized by having the above <14> An estimation procedure for estimating the job to be executed based on the second image data newly input by the device using the learned model that has learned the relationship between image data and jobs based on learning data associating the first image data input by the device with the job executed on the first image data based on an instruction from the user among two or more types of jobs executable by the device An information processing method characterized in that a computer executes the above <15> An estimation procedure for estimating the job to be executed based on the second image data newly input by the device using the learned model that has learned the relationship between image data and jobs based on learning data associating the first image data input by the device with the job executed on the first image data based on an instruction from the user among two or more types of jobs executable by the device A program characterized in that it causes a computer to execute the above

Explanation of Signs

[0102] 10 Image processing apparatus 11 Controller 12 Scanner 13 Printer 14 Modem 15 Operation Panel 16 Network Interface 17 SD Card Slot 20 Information Processing Device 21 Learning Data Generation Unit 22 Learning Unit 23 Estimation Unit 80 SD Card 111 CPU 112 RAM 113 ROM 114 HDD 115 NVRAM 121 Authentication Unit 122 Instruction Reception Unit 123 Image Input Unit 124 Blank Paper Removal Unit 125 Job Estimation Request Unit 126 Inquiry Unit 127 Job Execution Unit 128 Job Information Recording Unit 129 Image Recording Unit 130 Notification Unit 151 User Information Storage Unit 152 Model Storage Unit 200 Drive Device 201 Recording Medium 202 Auxiliary Storage Device 203 Memory Device 204 Processor 205 Interface Device 211 Image Storage Unit 212 Job Information Storage Unit 213 Model Storage Unit B Bus

Prior Art Documents

Patent Documents

[0103]

Patent Document 1

Claims

1. An estimation unit that estimates a job to be executed based on second image data newly input by the device, using a learned model that has learned the relationship between image data and jobs based on learning data in which first image data input by the device is associated with a job executed on the first image data based on an instruction by a user among two or more types of jobs executable by the device. An information processing system, characterized by comprising the same.

2. A job execution unit that executes the job estimated by the estimation unit. The information processing system according to claim 1, characterized by comprising the same.

3. The estimation unit estimates a job to be executed based on the second image data newly input by the device and the information at the time of input of the second image data, using a learned model that has learned the relationship between the image data and the information and the job based on learning data in which the first image data is associated with information indicating a situation in which a job has been executed on the first image data and the job executed on the first image data. The information processing system according to claim 1, characterized by the same.

4. The information indicating the situation is identification information of a user who uses the device. The information processing system according to claim 3, characterized by the same.

5. The information indicating the situation is information regarding an organization to which the user who uses the device belongs. The information processing system according to claim 3, characterized by the same.

6. The information indicating the situation is information regarding the time when the second image data was input. The information processing system according to claim 3, characterized by the same.

7. It has a blank page removal unit that removes blank pages from the second image data. The estimation unit estimates a job to be executed based on the second image data from which blank pages have been removed, using the learned model. The information processing system according to claim 1, characterized by the same.

8. The estimation unit estimates a job to be executed based on the second image data newly input by the device and the situation at the time of input of the second image data, from among learned models that have learned the relationship between image data and jobs based on learning data in which the first image data is associated with the job executed on the first image data, for each situation in which the job has been executed. The information processing system according to claim 1, characterized by the same.

9. An inquiry unit that inquires the user about the feasibility of executing the job estimated by the estimation unit; The information processing system according to claim 1, characterized by comprising the same.

10. A notification unit that outputs a predetermined notification based on a comparison between the job specified by the user for the second image data and the job estimated by the estimation unit; The information processing system according to claim 1, characterized by comprising the same.

11. An information processing system including an image processing apparatus and an information processing apparatus, The information processing apparatus, Based on learning data in which the first image data input by the image processing apparatus is associated with the job executed on the first image data based on an instruction from the user among two or more types of jobs executable by the image processing apparatus, using a learned model that has learned the relationship between image data and jobs, an estimation unit that estimates the job to be executed based on the second image data newly input by the image processing apparatus; Comprising, The image processing apparatus, A job execution unit that executes the job estimated by the estimation unit; An information processing system characterized by comprising the same.

12. A learning unit that causes a model to learn the relationship between image data and jobs based on learning data in which the first image data input by the device is associated with the job executed on the first image data based on an instruction from the user among two or more types of jobs executable by the device; An information processing system characterized by comprising the same.

13. A model storage unit that stores a learned model that has learned the relationship between image data and jobs based on learning data in which the first image data input by the device is associated with the job executed on the first image data based on an instruction from the user among two or more types of jobs executable by the device; An estimation unit that estimates the job to be executed based on the second image data newly input by the device using the learned model; An image processing apparatus characterized by comprising the same.

14. An estimation procedure for estimating the job to be executed based on the second image data newly input by the device using a learned model that has learned the relationship between image data and jobs based on learning data in which the first image data input by the device is associated with the job executed on the first image data based on an instruction from the user among two or more types of jobs executable by the device; An information processing method characterized in that a computer executes the same.

15. An estimation procedure for estimating a job to be executed based on second image data newly input by the device, using a learned model that has learned the relationship between image data and a job based on learning data in which first image data input by the device is associated with a job executed on the first image data based on an instruction by a user among two or more types of jobs executable by the device, A program, characterized in that it causes a computer to execute the same.

Images