Information processing device, information processing method, and computer readable recording medium

Abstract

The present invention provides an information processing device, an information processing method and a computer-readable recording medium, wherein the information processing device comprises: an estimation unit, which estimates the content of parameters set by a user by analyzing the following image, which is obtained by photographing the user operating a first device determined as an object of recording operation in a time series; and a storage unit, which stores the estimated content of the parameters as a record of the operation.

Description

Technical Field

[0001] The present disclosure relates to an information processing device, an information processing method, and a computer-readable recording medium. Background Art

[0002] For example, in an image forming device that forms an image on paper or other recording media, a function of registering repeatedly used operations or parameter values ​​as a work record is prepared as a function to assist users. The settings registered as a work record can be called up with one key. In addition, the hierarchy of the screen displayed in the process of a series of operations can also be stored in the work record (for example, refer to Japanese Patent Publication No. 2016-76746).

[0003] However, the work log is a function that stores which part of the screen has been operated, and does not store how the content of the parameters in the screen has been changed. In addition, the work log is inseparable from the image forming device, and there is no assumption that it can be transferred to other image forming devices. Therefore, when using other image forming devices, a series of operations must be performed manually.

[0004] However, as the multifunctionality progresses, the structure and content of the screen for the user interface prepared in the image forming apparatus have also become complicated. As a result, there are cases where the desired setting item cannot be found or the time required to find the desired setting item becomes longer. In addition, even for products of the same manufacturer, the structure and content of the user interface screen may differ depending on the time of release and the market. Summary of the invention

[0005] An object of the present disclosure is to facilitate the operation of an unfamiliar device, compared to a case where the device is operated only based on information obtained from an unfamiliar operation screen.

[0006] According to the first scheme of the present disclosure, there is provided an information processing device comprising: an estimation unit which estimates the content of parameters set by a user by analyzing an image obtained by capturing, in time series, a situation in which the user operates a first device determined as an object of recording the operation; and a storage unit which stores the estimated content of the parameters as a record of the operation.

[0007] According to the second aspect of the present disclosure, a screen for confirming the content of the estimated parameter is shown to the user.

[0008] According to the third aspect of the present disclosure, a change in the content of the estimated parameter is accepted on the screen.

[0009] According to the fourth aspect of the present disclosure, the estimation unit extracts a difference between an image after a change occurs on the screen that accepts a user operation and an image before the change occurs, and estimates the user operation based on a parameter corresponding to the difference.

[0010] According to the fifth aspect of the present disclosure, when the difference is a change in the display mode of the option or a change in the display related to the selection state of the option, the estimation unit estimates the character string of the option whose selection state has changed as the content of the parameter set by the user.

[0011] According to the sixth scheme of the present disclosure, the information processing device also has a display unit, which displays the result to the user when accepting the execution of the following function, which is a function of confirming whether the content of the saved parameters can be set in the second device different from the first device, and the result is the result of confirmation.

[0012] According to the seventh aspect of the present disclosure, the result of checking whether or not the setting can be performed is displayed for each parameter content.

[0013] According to the 8th scheme of the present disclosure, the information processing device also has a display unit, which, when accepting the execution of the following function, displays the operation of setting the content of the parameter according to the operation screen, wherein the operation screen is the operation screen of the second device different from the first device, and the function is to assist the operation for setting the content of the parameter through the operation screen.

[0014] According to the 9th aspect of the present disclosure, instructions for implementing the operation of setting the content of the parameter are displayed in an augmented reality or mixed reality manner through a terminal worn by the user.

[0015] According to the 10th scheme of the present disclosure, a computer-readable recording medium is provided, which stores a program, which enables a computer to perform a process having the following steps: estimating the content of parameters set by a user by analyzing the following image, which is obtained by photographing the user operating a first device determined as an object of recording operation in a time series; and saving the content of the estimated parameters as a record of the operation.

[0016] According to the 11th scheme of the present disclosure, there is provided an information processing method, which has the following steps: estimating the content of the parameters set by the user by analyzing the following image, which is obtained by photographing the situation in which the user operates the first device determined as the object of the recording operation in a time series; and saving the content of the estimated parameters as a record of the operation.

[0017] (Effect)

[0018] According to the first aspect, the unfamiliar device operation becomes easier than when the device is operated based only on information obtained from an unfamiliar operation screen.

[0019] According to the second aspect, the contents of the estimated parameters can be confirmed before registration.

[0020] According to the third aspect, parameters can be registered with correct contents.

[0021] According to the fourth aspect, no other operations are required except for operating the operation screen that is used by the user.

[0022] According to the fifth aspect, no operation is required other than operating the operation screen that is used habitually.

[0023] According to the sixth aspect, whether or not an instruction is possible can be confirmed before operating the actual machine.

[0024] According to the seventh aspect, whether or not an instruction can be given can be confirmed for each parameter.

[0025] According to the eighth aspect, necessary settings can be efficiently performed even on an unfamiliar operation screen.

[0026] According to the ninth aspect, operations can be performed while confirming the operation screen of the actual machine.

[0027] According to the tenth aspect, the operation of an unfamiliar device becomes easier than in a case where the device is operated based only on information obtained from an unfamiliar operation screen.

[0028] According to the eleventh aspect, the operation of an unfamiliar device becomes easier than in a case where the device is operated based only on information obtained from an unfamiliar operation screen. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a diagram for explaining the conceptual structure of an information processing system used in the embodiment.

[0030] Figure 2 This is a diagram showing a configuration example of a terminal device used in the embodiment.

[0031] Figure 3 This is a diagram illustrating a functional structure implemented by a control unit of a terminal device.

[0032] Figure 4 This is a diagram for explaining a configuration example of a parameter setting assisting server used in the embodiment.

[0033] Figure 5 This is a diagram for explaining a functional structure implemented by a control unit of a parameter setting assisting server.

[0034] Figure 6 This is a sequence diagram that explains the registration action of the operation.

[0035] Figure 7 This is a diagram for explaining a situation in which an operation screen is photographed by a terminal device.

[0036] Figure 8 This is a diagram for explaining a situation in which an operation screen is determined by the parameter setting support server.

[0037] Fig. 9 The figures are diagrams for explaining a method of estimating parameters by a parameter estimation module, wherein (A) shows a transition mode 1 and (B) shows a transition mode 2.

[0038] Fig.10 This is a diagram for explaining the transition of the screen from the screen for accepting the instruction of the registration operation to the screen at the time of estimating the parameters.

[0039] Fig.11 It is a diagram showing an example of a screen requesting confirmation of an estimation result.

[0040] Fig.12 This is a diagram showing an example of a screen used for checking a history in which operations are recorded.

[0041] Fig.13 This is a sequence diagram that explains the actions related to transfer confirmation.

[0042] Fig.14 This is a diagram showing an example of screen transition when a button requesting transfer confirmation is operated.

[0043] Fig.15 This is a sequence diagram for explaining the operation related to the set assistance (support).

[0044] Fig.16 It is a diagram showing an example of transition of the auxiliary setting screen.

[0045] Fig.17 This is a diagram of an image for explaining operations when the terminal device is a glasses-type wearable terminal.

[0046] Fig.18 This is a diagram of an image for explaining operations when the terminal device is a glasses-type wearable terminal. DETAILED DESCRIPTION

[0047] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0048] <Implementation Method>

[0049] <Overall structure of the system>

[0050] Figure 1 The diagram is a diagram for explaining the conceptual structure of an information processing system 1 used in the embodiment. The information processing system 1 includes the Internet 10 , an image forming apparatus 20A installed at a location A, an image forming apparatus 20B installed at a location B, a terminal device 30 carried by a user, and a parameter setting support server 40 .

[0051] The terminal device 30 in the present embodiment is connected to the parameter setting support server 40 via the Internet 10 .

[0052] exist Figure 1 In the example shown in FIG. 1 , one image forming apparatus 20A, one image forming apparatus 20B, and one terminal device 30 are shown respectively; however, there may be plural image forming apparatuses 20A, 20B, and plural terminal devices 30.

[0053] In the present embodiment, it is assumed that the image forming apparatus 20A at the location A is a device that is commonly used by the user who operates the terminal device 30. Therefore, the image forming apparatus 20A is an example of the first device.

[0054] In the present embodiment, "accustomed use" means that the parameters and properties (hereinafter collectively referred to as "parameters") that define the operation of the device can be smoothly set. In addition, only one parameter that is used habitually needs to be set.

[0055] In addition, the commonly used image forming apparatus 20A is not limited to a single model, and may be a plurality of models. In addition, the commonly used image forming apparatus 20A may be manufactured by a plurality of companies.

[0056] In the present embodiment, the image forming apparatus 20B at the assumed location B is an apparatus that the user operating the terminal device 30 is not accustomed to using. Therefore, the image forming apparatus 20B is an example of the second device.

[0057] Basically, except for the image forming apparatus 20A that is used frequently, there are image forming apparatuses 20B that are not used frequently or operated frequently. Therefore, the difference in the location of installation has nothing to do with whether it is used frequently or not. In this embodiment, for the sake of convenience of explanation, the image forming apparatus 20A installed at the location A is called the device that is used frequently, and the image forming apparatus 20B installed at the location B is called the device that is not used frequently. Therefore, even at the same location A, the image forming apparatus 20A that is used frequently and the image forming apparatus 20B that is not used frequently can be set.

[0058] A user who is not used to operating the image forming device 20B as the operation object needs to find buttons corresponding to the parameters that he wants to set on the operation screen and operate them one by one. For example, even if the function names used in the operation screen of the image forming device 20B and the function names used in the operation screen of the image forming device 20A are the same, if the layout of the operation screen is different, the user's operation will still become difficult. In addition, if the function names are different, it is also difficult to confirm whether they are the same function.

[0059] In addition, the same is true for the attribute of the parameter. Although it is "1" in the image forming apparatus 20A, it may be "2" in the image forming apparatus 20B. For example, this may occur when the density imparting method is different. In this way, it is not always easy to reproduce the same operation as the image forming apparatus 20A that is used frequently in the image forming apparatus 20B that is not used frequently.

[0060] The image forming devices 20A and 20B in this embodiment have the functions of copying, scanning, and faxing. Therefore, the image forming devices 20A and 20B have an image reading unit that reads the image of the original, an image forming unit that records the image on a paper as an example of a recording medium, an image processing unit that applies color correction or grayscale correction to the image represented by the image data, a control unit that controls the operation of the entire device; a storage unit that stores image data, etc.; a display unit for displaying a user interface screen, etc.; an operation receiving unit that receives user operations; and a communication interface (=communication IF) that realizes communication via a LAN, etc. Of course, these device structures are examples.

[0061] The parameter setting support server 40 cooperates with the terminal device 30 operated by the user to support so that the settings of the parameters registered in the image forming device 20A which have been registered in advance can be easily reproduced in the image forming device 20B.

[0062] In the present embodiment, the terminal device 30 is a portable terminal. Portable terminals include, for example, smartphones, wearable terminals, and notebook computers. Wearable terminals include, for example, wrist-worn wearable terminals and glasses-type wearable terminals worn on the head.

[0063] Among the glasses-type terminals, there are so-called transmission-type terminals that use a transparent plate in which a holographic optical element is assembled. The holographic optical element has the property of transmitting light incident from the front and refracting the image light propagating inside the transparent plate and outputting it in the direction of the user's eyeball. Therefore, the user wearing the glasses-type terminal can visually confirm the image after superimposing a virtual image or text on the actual object.

[0064] <Structure of Each Device>

[0065] <Structure of terminal device>

[0066] Figure 2 It is a diagram showing a configuration example of a terminal device 30 used in the embodiment.

[0067] Figure 2 The terminal device 30 shown includes a control unit 301 for controlling the overall operation of the device, a storage unit 302 for storing image data, etc., a display unit 303 for displaying a user interface screen, etc., an operation acceptance unit 304 for accepting user operations, a camera 305 for taking images, and a communication interface 306.

[0068] The camera 305 here can realize shooting based on still images or moving images.

[0069] The control unit 301 in this embodiment includes: a CPU (=Central Processing Unit) 311; a ROM (=Read Only Memory) 312, which stores firmware, BIOS (=Basic Input Output System), etc.; and a RAM (=Random Access Memory) 313, which is used as a work area. The control unit 301 functions as a so-called computer.

[0070] The storage unit 302 is composed of, for example, a nonvolatile rewritable semiconductor memory, etc. In the storage unit 302, for example, image data captured by the camera 305 and the like are stored.

[0071] The display unit 303 is composed of, for example, a liquid crystal display or an organic EL display. The display unit 303 displays information that assists the user's operation.

[0072] The operation accepting unit 304 is composed of a touch sensor, a switch, a button, and the like arranged on the surface of the display unit 303 .

[0073] In addition, the control unit 301 and each unit are connected via a bus 307 or a signal line (not shown).

[0074] Figure 3 This is a diagram for explaining a functional configuration implemented by the control unit 301 of the terminal device 30 . Figure 3 The functional modules shown are controlled by the CPU 311 (refer to Figure 2 ) to execute the program. In addition, Figure 3 The functional modules shown are examples of functional modules provided by the control unit 301 .

[0075] The control unit 301 in the embodiment functions as the following modules: an operation object determination module 321 that determines the operation object, i.e., the model; a transition candidate detection module 322 that detects the candidate for transition of the operation screen; a candidate screen storage module 323 that stores the candidates for the operation screen before and after the transition; a candidate screen transmission module 324 that transmits the candidate for the operation screen; an estimation result display module 325 that displays the estimated result of the operation screen generated by the parameter setting auxiliary server 40 (see Figure 1 ) estimated result; whether the transfer display module 326 shows the result after confirming whether the settings of the registered operation can be transferred to the specified model; and the setting auxiliary screen display module 327 shows the following screen, which assists the operation of implementing the same settings as the registered operation.

[0076] The operation target determination module 321 in this embodiment is executed when the user instructs to set the record of the operation. In this embodiment, the operation target determination module 321 determines the image forming apparatus 20A or the like as the operation target using at least one of the various methods described below.

[0077] One method is to read the image captured by the camera 305 (refer to Figure 2 ) is a method for determining the model of a device by looking at a label and other text or numerical columns captured by a computer. In this method, text recognition is performed on the captured image and converted into character codes. In addition to being arranged on the housing of the device to be operated, the label and other text or numerical columns can also be displayed on the operation screen.

[0078] Another method is to determine the model based on the appearance image of the image forming device 20A or the like captured by the camera 305. The operation object determination module 321 determines one or more candidates by comparing with pre-prepared appearance photos, photos of characteristic parts, etc. If multiple candidates match, the information of the device selected by the user is obtained.

[0079] Another method is a method of acquiring information specifying a model through a logical or physical connection with a device that is an operation target.

[0080] The operation target, that is, the model is determined by, for example, the user selecting one from a list.

[0081] In the case of the present embodiment, the transition candidate detection module 322 is executed after the operation target determination module 321 determines the device.

[0082] The transition candidate detection module 322 processes the image data of the operation of the operation screen, and detects the occurrence of events with a high possibility of transition of the operation screen. The so-called events with a high possibility of transition of the operation screen refer to, for example, changes in the overall layout of the operation screen, the movement of the position of radio buttons, etc., and changes in the display color or brightness of the confirmation button. For example, a button that changes from a non-selected state to a selected state will change to a display color different from other buttons. In addition, a button that changes from a non-selected state to a selected state changes to a brightness higher than the brightness of other buttons.

[0083] Here, the end time of the image data captured by the user is also a target in the detection of transition candidates of the operation screen by the transition candidate detection module 322. In the case of this embodiment, the end time of the image data is, for example, the time when the user instructs to end the shooting, the time when a sound notifying the end of the setting such as a mechanical sound is detected, or the time when a predetermined time has passed while no change is detected in the operation screen.

[0084] In the present embodiment, the detection of the transition of the operation screen is performed by the parameter setting support server 40 (see Figure 1 Therefore, the processing performed by the transition candidate detection module 322 is limited to detecting candidates with a high possibility of transition in the operation screen.

[0085] For example, when the operation situation is captured as a dynamic image, the transition candidate detection module 322 compares the previous frame and the next frame in sequence to determine whether the difference between the frames exceeds a predetermined threshold. If the difference between the frames exceeds the predetermined threshold, the transition candidate detection module 322 of this embodiment determines that the possibility of the operation screen transitioning is high.

[0086] In addition, it is preferable to ignore the changes caused by the finger or hand captured during the operation. However, even if the changes caused by the captured finger or hand are detected as changes in the operation screen, the final detection of the transition of the operation screen is also performed by the parameter setting auxiliary server 40 (see Figure 1 ) is executed.

[0087] The candidate picture saving module 323 stores the candidate picture in the storage unit 302 (see Figure 2 ) stores the operation screens corresponding to the time before and after the time detected as the candidate for transition. The operation screens stored in the storage unit 302 also record the attached data such as the shooting time.

[0088] The candidate screen sending module 324 sends the operation screen saved as a candidate for transition according to the information indicating the sending destination of the operation screen. In the case of this embodiment, as the information indicating the sending destination, the parameter setting auxiliary server 40 is recorded on the Internet 10 (refer to Figure 1 ) on the Internet. Information indicating the destination is provided when the application is installed or updated. For example, a URL (=Uniform Resource Locator) is used.

[0089] The estimation result display module 325 displays the result that the setting has been estimated by the user's operation on the display unit 303 (see Figure 2 ). The estimation result is provided by the combination of parameters. The estimation result is provided from the parameter setting assistance server 40.

[0090] The transferability indicating module 326 is activated in the following case: it is necessary to confirm in advance whether the same setting as the operation registered in the image forming device 20A can be realized in other devices. In the case of this embodiment, the transferability indicating module 326 provides information on the model to which the user wants to transfer to the parameter setting auxiliary server 40, and receives the verification result of whether the same setting as the specified operation can be performed in the specified model. The verification result is provided for each parameter. For example, it is notified that the setting of a certain parameter can be performed, while the setting of a certain parameter cannot be performed. The transferability indicating module 326 here is an example of a indicating unit.

[0091] The setting assistance screen display module 327 displays the following operation steps and operation contents based on the operation screen actually operated, and the operation steps and operation contents are used to realize the same setting as the registered operation specified by the user in the model operated by the user. The setting assistance screen display module 327 performs processing by linkage with the parameter setting assistance server 40. The setting assistance screen display module 327 here is an example of a display unit.

[0092] <Structure of parameter setting auxiliary server>

[0093] Figure 4 It is a diagram for explaining a configuration example of the parameter setting support server 40 used in the embodiment.

[0094] Figure 4 The parameter setting support server 40 shown includes a control unit 401 that controls the operation of the entire device, a storage unit 402 that stores data used for supporting parameter setting, and a communication interface 403 .

[0095] The control unit 401 in this embodiment includes a CPU 411, a ROM 412 storing firmware, BIOS, etc., and a RAM 413 used as a work area. The control unit 401 functions as a so-called computer.

[0096] The storage unit 402 is composed of, for example, a hard disk device (HDD: Hard Disk Drive) as a non-volatile storage device. The storage unit 402 stores parameters related to operations registered by the user, data received from the terminal device 30 (see Figure 1 ) The operation screen sent by the user, the estimated parameters used for user operation, the operation screen of each model of the device serving as an auxiliary object, etc.

[0097] In addition, the control unit 401 and each unit are connected via a bus 404 and a signal line (not shown).

[0098] Figure 5 1 is a diagram for explaining a functional configuration implemented by the control unit 401 of the parameter setting assisting server 40 . Figure 5 The functional modules shown are controlled by CPU 411 (refer to Figure 4 ) to execute the program. In addition, Figure 5 The functional modules shown are examples of functional modules provided by the control unit 401 .

[0099] The control unit 401 in the implementation method has: a model information acquisition module 421, which acquires information related to the model of the device operated by the user; a screen transition storage module 422, which stores the transition of the operation screen accompanying the user's operation; a parameter estimation module 423, which analyzes the transition of the operation screen and estimates the parameters of the user's operation; a parameter storage module 424, which stores the estimated parameters in association with the operation; a parameter confirmation module 425, which requests the user to confirm the parameters; a transfer confirmation module 426, which confirms whether the registered parameters and other settings can be transferred to the device specified by the user; and a setting assistance module 427, which assists in performing the following operations: the operation is used to achieve the same settings as the registered parameters, etc.

[0100] The model information acquisition module 421 in this embodiment acquires information related to the model of the device that is the user's operation target from the terminal device 30. The model information acquired when the operation is registered is associated with a record of parameters and the like obtained by analyzing the user operation. The model information acquisition module 421 acquires the label and other text or numerical column images arranged on the housing of the image forming device 20A and the like, the appearance image of the image forming device 20A and the like, and determines the model.

[0101] The screen transition storage module 422 in this embodiment estimates the operation screen currently operated by the user from the operation screens with a high possibility of transition received from the terminal device 30 and stores it. The screen transition storage module 422 in this embodiment estimates the operation screen operated by the user by comparing the operation screen with a high possibility of transition with the list of operation screens of the model as the operation object. The estimated operation screens are stored in the order of operation. That is, the estimated operation screens are stored in time series. The history of the operation screens stored in the order of operation indicates the transition of the operation screens accompanying the operation of the user.

[0102] The parameter estimation module 423 analyzes the transition of the operation screen and estimates the content of the operation performed by the user as the cause of the transition. In other words, the parameter estimation module 423 estimates the content of the parameter set by the user. In the setting of the parameter by the user, for example, there is the selection of the parameter itself, the selection or input of the attribute related to the specific parameter, etc. The parameter estimation module 423 here is an example of an estimation unit.

[0103] The parameter storage module 424 stores the estimated parameters in association with one operation. In addition, the operation here is also associated with the device (eg, the image forming device 20A) that is the operation target. The parameter storage module 424 here is an example of a storage unit.

[0104] The parameter confirmation module 425 requests the user who is the operator of the operation to confirm the contents of the estimated parameters. Specifically, the parameter confirmation module 425 receives a request from the terminal device 30 (see Figure 1 ) Accepts approval or correction of the estimated parameter content. When the user's confirmation is completed, the estimated parameter is saved as the content of the operation registered by the user. The difference from the work record is that the content of the operation parameter is also saved.

[0105] The transferability confirmation module 426 is activated when a request is made from the terminal device 30 to confirm whether a registered operation can be transferred. The transferability confirmation module 426 receives the designation of a registered operation to be transferred and the designation of a model to be transferred.

[0106] The list of registered operations is stored for each model in the parameter setting support server 40. In addition, the management information of the registered operations may also include information for identifying the user who registered the operation. In addition, the registered operations registered by other users are also the objects of the list display.

[0107] Methods for specifying a model as a transfer destination include the following methods: a method of selecting from a list of models, a method of determining by reading a label from an actual machine, a method of determining based on an image of the actual machine's appearance, and a method of obtaining information to determine the model through a connection with the actual machine.

[0108] When a registered operation for which confirmation of transferability is required and a transfer destination model are specified, the transferability confirmation module 426 confirms whether the same setting as the registered operation can be realized for each parameter in the transfer destination model.

[0109] In addition, the names and expressions of the parameters displayed on the operation screen may not necessarily be the same between the model used for registration of the operation and the model used as the transfer destination. However, in the parameter setting auxiliary server 40 in this embodiment, a list of correspondences between parameters based on different models is recorded. In this list, correspondences between attributes based on different models are also recorded. In addition, it is also possible that a parameter in a certain model corresponds to a group of multiple parameters in other models. The same is true for attributes.

[0110] The transferability confirmation module 426 confirms whether the transfer is possible for each set parameter, and notifies the confirmation result to the terminal device 30. The transferability confirmation module 426 here is an example of a means.

[0111] The setting support module 427 is activated when support for setting the same parameters as the registered operation is requested from the terminal device 30. The setting support module 427 receives designation of the registered operation to be supported and designation of the actual device to be supported.

[0112] The list of registered operations is the same as that of the transferability confirmation module 426.

[0113] The following methods are available for specifying the model to be assisted: selecting from a list of models, identifying by reading a label from an actual machine, identifying based on an image of the actual machine's appearance, and obtaining information to identify the model by connecting to the actual machine.

[0114] When the registered operation and the actual machine to be operated are specified, the setting auxiliary module 427 forms a screen that specifically instructs which part of the operation screen to perform which operation in the setting of the same parameters as the registered operation, and notifies the terminal device 30. In the case of this embodiment, the setting auxiliary module 427 is also provided with a function of image recognition of the operation screen displayed on the actual machine, and specifically instructs how to perform which operation on which part of the recognized operation screen. In the instruction, for example, guidance based on sound is used in the display of highlighting or virtual frame.

[0115] In the case of this embodiment, the assistance provided by the setting assistance module 427 ends at the following times: when the user instructs to end the assistance; when a sound notifying the end of the setting, such as a mechanical sound, is detected; when a predetermined time has passed without detecting a change in the operation screen, etc. The setting assistance module 427 here is an example of a display unit.

[0116] <Specific operation examples>

[0117] Next, a description is given of a method of using the terminal device 30 (see Figure 1 ) and the parameter setting auxiliary server 40 (refer to Figure 1 ) specific operation example.

[0118] <Operation when registering an operation is selected>

[0119] Figure 6 is a sequence diagram that illustrates the registration actions of the operation. Figure 6 In the case of , the symbol S represents a step.

[0120] First, a user who wishes to register an operation operates terminal device 30 to obtain information identifying image forming apparatus 20A for which the operation is to be registered (step 1). Information identifying the model is performed by, for example, reading characters printed on the surface of the housing.

[0121] Next, the user activates the camera 305 (see Figure 2 ), and the shooting operation starts (step 2). In addition, the activation of the camera 305 can also be executed by the control of the terminal device 30. Thereafter, the user starts to operate the image forming device 20A. The shooting continues until the operation ends (step 3).

[0122] Figure 7 30 is a diagram for explaining a situation in which the terminal device 30 captures an operation screen. In the case of the present embodiment, the object of operation is the image forming device 20A installed at the location A (see FIG. 20A ). Figure 1 ).

[0123] "Screen 1", "Screen 2", "Screen 3", and "Screen 4" in the figure correspond to operation screens that appear in time series as the user operates. When the images captured by the camera 305 are still images, the user captures "Screen 1", "Screen 2", "Screen 3", and "Screen 4" in sequence. When the images captured by the camera 305 are moving images, the user captures images continuously from the start of the registration operation to the end.

[0124] return Figure 6 Description.

[0125] exist Figure 6 In the case of the transition candidate, the terminal device 30 processes the captured image data and detects the transition candidate (step 4). Next, the terminal device 30 sends information identifying the image forming device 20A and the operation screens before and after the transition candidate is detected (step 5). The transmission destination is the parameter setting support server 40.

[0126] The parameter setting support server 40 identifies the transition of the operation screen accompanying the user's operation by comparing with the list of operation screens stored for the target device (step 6 ).

[0127] Figure 8 This is to explain the parameter setting auxiliary server 40 (refer to Figure 1 ) is a diagram showing a situation in which the operation screen is determined. The operation screen is determined by the screen transition storage module 422 (refer to Figure 5 )implement.

[0128] exist Figure 8 In the middle section, terminal devices 30 are arranged in time sequence (refer to Figure 1 ) is detected as a candidate for transition. Figure 8 In the example, the dialog box is displayed only when the operation screen is switched, but candidates for transition are also detected between these moments. Fig. 9 )The corresponding picture changes are detected as transition candidates.

[0129] The screen transition storage module 422 compares the operation screen received from the terminal device 30 with the list of operation screens prepared for the device as the operation target, and estimates the transition of the operation screen. Specifically, it extracts four screens that match the list, namely "Screen 1", "Screen 2", "Screen 3", and "Screen 4".

[0130] return Figure 6 Description.

[0131] exist Figure 6In the case of, the parameter setting support server 40 estimates the content of the parameter that is the cause of the transition (step 7). The content of the set parameter can be estimated based on the relationship between the previous and next operation screens.

[0132] Fig. 9 This is to illustrate the parameter estimation module 423 (refer to Figure 5 ) are diagrams showing a method of estimating parameters by using (A) to estimate the parameters of the transition mode 1 and (B) to estimate the parameters of the transition mode 2.

[0133] Mode 1 is an example where the operation screen switches from "Screen 1" to "Screen 2". Fig. 9 In the case of mode 1, "Screen 1" is a parameter list screen, while "Screen 2" is switched to a setting screen for properties related to the color mode. This change occurs with the switching of the layout of the operation screen itself. In the case of mode 1, it is estimated that the transition of the operation screen is caused by selecting "Color Mode" in the list of parameters displayed on "Screen 1".

[0134] Mode 2 corresponds to the case where the operation screen remains "Screen 1" but the attributes are changed. Fig. 9 In the example of , in the "color mode", the settable attributes are "color" and "black and white". "Color" is set in "Screen 1", and "black and white" is set in "Screen 2". In addition, even if the user's setting changes between "color" and "black and white" multiple times, the registered setting is still the attribute finally selected by the user. Therefore, in the case of mode 2, the parameter estimation module 423 registers the content of the final setting as a record of the operation.

[0135] return Figure 6 .

[0136] If the parameter estimation for the operation to be registered is completed, the parameter confirmation module 425 (see Figure 5 ) sends the estimation result to the terminal device 30 (step 8).

[0137] In the terminal device 30, the received estimation result is displayed (step 9).

[0138] Then, the terminal device 30 receives the user's approval of the estimated result or the correction made by the user (step 10).

[0139] In the case of this embodiment, approval of the estimated result is also notified to the parameter setting support server 40. Therefore, the approved estimated result is stored in both the terminal device 30 and the parameter setting support server 40 (steps 11 and 12).

[0140] <Example of screen displayed on terminal device>

[0141] Here, an example of a screen displayed on the terminal device 30 in conjunction with registration of an operation will be described.

[0142] Fig.10 This is a diagram for explaining the transition of screens from a screen that accepts an instruction to register an operation to a time when parameters are estimated.

[0143] The screen corresponding to time T1 is an initial screen that can instruct to record the operation. Fig.10 In the initial screen shown, three options are available: "Record the operation of the image forming device", "Operate another image forming device", and "View the recorded data".

[0144] At time T1, the user selects “record operation of image forming apparatus”.

[0145] When "record operation of image forming apparatus" is selected at time T1, a screen corresponding to time T2 is displayed. Fig.10 In the example, it is requested to identify the image forming apparatus to be registered for operation.

[0146] The screen corresponding to time T3 is displayed after the confirmation of the image forming apparatus is completed. Fig.10 In the example of , the operation to be recorded is urged to start. In addition, the recording is not started until the screen corresponding to the time T3 is displayed based on the camera 305 (refer to Figure 2 ) or start shooting based on camera 305 (refer to Figure 2 ) was filmed.

[0147] The screen corresponding to time T4 is displayed so that the user can confirm whether the user's operation is captured.

[0148] The screen corresponding to time T5 is the following screen, which shows that the auxiliary server 40 (see Figure 1 )Analyze the captured operation.

[0149] Fig.11 It is a diagram showing an example of a screen requesting confirmation of an estimation result.

[0150] When the analysis by the parameter setting support server 40 is completed, the terminal device 30 displays Fig.11 The screen shown.

[0151] exist Fig.11 In the case of "normal copy", the name "normal copy" is added to the operation to be registered. This name can be entered while the estimated parameters are stored. Fig.11In the case of , the estimated parameters include "color" for "color mode", "A4" for "paper size", "text / photo" for "document type" ... and "set value" for "parameter n". The symbol " / " here means "or".

[0152] When the Save button on the upper right of the operation diagram is pressed, the estimated parameters are confirmed. In addition, if the estimated result is different from the actual operation, Fig.11 The corrections are made on the screen shown. The corrections may include partial changes, additions, deletions, etc.

[0153] Fig.12 This is a diagram showing an example of a screen used for checking a history in which operations are recorded.

[0154] The screen corresponding to time T11 is Fig.10 At time T11, the user selects "Observe recorded data".

[0155] The screen corresponding to time T12 is an example of a screen that displays the history of recorded operations. Fig.12 In the list, the names of registered operations are displayed.

[0156] exist Fig.12 In the example, "Normal Copy", "Black and White Copy", ... "Enlarged Copy" are shown. For example, if "Normal Copy" is selected, the following is displayed: Fig.11 The screen shown. In the confirmation screen, the Save button is not displayed.

[0157] <Action when transfer confirmation is selected>

[0158] Fig.13 is a sequence diagram that illustrates the actions related to transfer confirmation. Fig.13 In this case, the symbol S also represents a step.

[0159] First, the user who wishes to confirm the transfer operates the terminal device 30 and specifies the registered operation (step 21). Fig.13 In the case of the specified registered operation, the information of the specified registered operation is notified from the terminal device 30 to the parameter setting support server 40. The parameter setting support server 40 reads out the parameter of the specified registered operation (step 22).

[0160] Next, the user who wishes to confirm the transfer operates the terminal device 30 to select the device n to be transferred (step 23). The selection here includes a case of selecting from a list of models and a case of specifying a real machine. When a real machine is specified, the device configuration of the real machine is read to the terminal device 30 through communication with the real machine. However, when the parameter setting auxiliary server 40 can read the device configuration of the real machine, it is not necessary for the terminal device 30 to read the device configuration.

[0161] The parameter setting auxiliary server 40, which receives the notification of the information of the device n as the transfer destination of the registered operation, reads out the information that can be set in the device n (step 24). When the device n is specified from the list of models, the parameter setting auxiliary server 40 reads out the read information that can be set in the device n to the maximum extent. On the other hand, when a real machine is specified as the device n, the parameter setting auxiliary server 40 reads out the information that can actually be set in the specified real machine. For example, when a page binder can be installed in the specified model, the parameter setting auxiliary server 40 reads the settable information on the premise that the page binder is installed. On the other hand, when the page binder is not installed in the real machine, the parameter setting auxiliary server 40 will not use the functions related to the page binder as the settable information. The page binder refers to, for example, binding processing, hole processing and other finishing.

[0162] Next, the parameter setting support server 40 checks whether the parameters of each registered operation can be transferred (step 25). The parameter names may differ depending on the model, so a pre-stored name correspondence list is used. The parameter setting support server 40 notifies the terminal device 30 of whether the confirmation is possible.

[0163] The terminal device 30 that has received the notification displays whether or not the transfer is possible for each registered operation parameter (step 26 ).

[0164] Fig.14 This is a diagram showing an example of screen transition when a button requesting transfer confirmation is operated.

[0165] When the analysis by the parameter setting support server 40 is completed, the terminal device 30 displays Fig.11 The screen shown.

[0166] The screen shown at time T21 shows a state where the “transfer confirmation” button is operated in the confirmation screen of the registered operation.

[0167] The screen shown at time T22 shows a list used for selecting a model name. At time T22, the model name "58020056" is selected.

[0168] The screen shown at time T23 shows whether the settings of the various parameters constituting the registered operation can be performed. In the case of time T23, the attributes of the color mode are highlighted. The highlighted display means that the setting cannot be performed. For example, if the designated machine model is a monochrome printer, color printing cannot be set.

[0169] In addition to highlighting, you can also display explanatory text for parameters that cannot be set, and you can also change the font to display it, and you can also change the font color to display it.

[0170] <Actions when setting assistance (support) is selected>

[0171] Fig.15 This is a sequence diagram that explains the actions related to the set assistance (support). Fig.15 In this case, the symbol S also represents a step.

[0172] First, a user who wishes to be assisted in setting operates the terminal device 30 and specifies a registered operation (step 31). Fig.15 In the case of the specified registered operation, the information of the specified registered operation is notified from the terminal device 30 to the parameter setting support server 40. The parameter setting support server 40 reads out the parameter of the specified registered operation (step 32).

[0173] Next, the terminal device 30 sends the information of the actual machine to the parameter setting auxiliary server 40 (step 33). Various methods are prepared for inputting information for specifying the actual machine. For example, a method of specifying by reading a barcode, a QR (Quick Response) code (registered trademark) or other codes, a method of specifying by photographing an image forming device, or a method of specifying by tapping an image forming device.

[0174] When the information of the actual machine to be set is notified, the parameter setting support server 40 obtains the correspondence between the registered operation parameters and the parameters that can be set in the actual machine (step 34 ).

[0175] Then, the parameter setting support server 40 specifies the operation position and the like for each operation screen of the actual machine (step 35 ). The specified operation position and the like are transmitted from the parameter setting support server 40 to the terminal device 30 .

[0176] In the terminal device 30, the received operation position and the like are sequentially indicated (step 36).

[0177] Fig.16 2 is a diagram showing an example of the transition of the auxiliary setting screen. Fig.16In the embodiment, it is assumed that the terminal device 30 is a portable terminal such as a smart phone. Specifically, it is assumed that the terminal device 30 is held with one hand and the image forming device 20B is operated with one hand (see Figure 1 ) operation screen.

[0178] The screen corresponding to time T31 shows a state where the “Assist” button is operated in the confirmation screen of the registered operation.

[0179] The screen corresponding to time T32 indicates a screen for accepting a designation of the image forming apparatus 20B to be operated. Fig.16 In the example, three methods are shown: "Specify by QR code", "Specify by photographing the image forming device", and "Specify by clicking the image forming device". The user selects any one of the methods to obtain information of the information processing device such as the image forming device 20 that is the object of operation.

[0180] The screen corresponding to time T33 is a screen that shows the user that the information of the image forming apparatus 20B is being acquired.

[0181] The screens corresponding to time T34 and time T35 show examples of screens that assist the operation position and the like.

[0182] In the screen corresponding to time T34, the operation screen is "Screen 1". Fig.16 In the example of , the item that the user should set is indicated by a sentence such as "Please click ○○". In addition, a screen that clearly shows the position to be operated in screen 1 to the user may also be prepared.

[0183] In the screen corresponding to time T35, the operation screen is switched to "Screen 2." The method of notifying the user of the position to be operated is the same as that of "Screen 1."

[0184] By displaying this auxiliary screen, even when operating an unfamiliar image forming apparatus 20B, the user can easily perform the same settings as those of the familiar image forming apparatus 20A.

[0185] Fig.17 FIG. 2 is a diagram for explaining an operation image when the terminal device 30 is a glasses-type wearable terminal. Fig.17 In this case, the user wearing the terminal device 30 on the head can visually recognize a virtual frame 51 and a character string 52 that do not actually exist in the space in addition to the image forming device 20B that actually exists in front of the user. The display in the dialog box represents the appearance of the space visually recognized by the user.

[0186] Here, the frame 51 is used to notify the user of the device recognized as the operation target by the terminal device 30. In addition, the character string 52 describes the action to be requested of the user in words. Fig.17 In the case of "Please observe the image forming device you want to operate" is displayed floating in the air.

[0187] This technology of superimposing computer-generated virtual visual information on real objects is called augmented reality (AR) or mixed reality (MR).

[0188] Fig.18 It is a diagram of an image for explaining an operation when the terminal device 30 is a glasses-type wearable terminal.

[0189] The screen corresponding to time T41 shows that the data stored in frame 51 (see Fig.17 ) is displayed. In the example at time T41, the explanation text 53 is displayed in a manner overlapping with the image forming apparatus 20B that actually exists.

[0190] The screen corresponding to time T42 shows an example of auxiliary information displayed when the user is viewing the operation screen of the image forming device 20B. In this example, while an explanation text 54 such as "Please tap 'black and white'" is superimposed on "Screen 1", a frame 55 is superimposed on the position to be tapped in "Screen 1". This frame 55 is also a virtual image. By displaying the frame 55, the user can easily find the position to be operated even in the unfamiliar "Screen 1".

[0191] The screen corresponding to time T43 shows the screen after the user has performed the operation. At time T43, the display of the explanation text 54 remains, but the display of the frame 55 disappears because the operation of the "black and white" button has been confirmed.

[0192] <Other embodiments>

[0193] The embodiments of the present disclosure are described above, but the technical scope of the present disclosure is not limited to the scope described in the above embodiments. As can be seen from the description of the claims, various changes or improvements to the above embodiments are also included in the technical scope of the present disclosure.

[0194] In the above-described embodiment, in the image forming apparatus 20A (refer to Figure 1 ) and the user who wishes to register the image forming apparatus 20B (refer to Figure 1) The differences and similarities between users who apply registered operations are not described, but the user who operates the image forming apparatus 20A and the user who operates the image forming apparatus 20B are not necessarily the same person.

[0195] In the above embodiment, the terminal device 30 (see Figure 1 ) and the parameter setting auxiliary server 40 (refer to Figure 1 ) to provide the above-mentioned auxiliary services, but all processing may be performed by only the terminal device 30, or may be performed by only the parameter setting auxiliary server 40. In addition, even when all processing is performed by the parameter setting auxiliary server 40, the terminal device 30 is required to capture images related to the actual machine or input information. Here, the terminal device 30 and the parameter setting auxiliary server 40 are both examples of information processing devices.

[0196] In addition, in the above-mentioned embodiment, as an example of the cooperation operation between the terminal device 30 and the parameter setting support server 40, Figure 6 , Fig.13 , Fig.15 , but the manner in which the processing is shared between the terminal device 30 and the parameter setting support server 40 is not limited to this.

[0197] In the above embodiment, the first device and the second device are both image forming apparatuses, but the first device and the second device are not limited to image forming apparatuses, and may be, for example, information processing apparatuses such as computers, washing machines, electronic cookers, and other white appliances.

Claims

1. An information processing device, comprising: an estimating unit that estimates the content of the parameter set by the user by analyzing an image obtained by capturing, in time series, changes in an operation screen as the user operates the first device determined as a target of the recorded operation; a storage unit that stores the estimated parameters as a record of the operation; as well as A display unit, which, when accepting the execution of the following function, displays the operation of setting the content of the parameter according to the following operation screen, wherein the operation screen is the operation screen of the second device different from the first device, and the function is to assist the operation for setting the content of the parameter through the operation screen.

2. The information processing device according to claim 1, in, A screen for confirming the estimated contents of the parameter is shown to the user.

3. The information processing device according to claim 2, in, A change in the estimated content of the parameter is accepted on the screen.

4. The information processing device according to claim 1, in, The estimation unit extracts a difference between an image after a change occurs on a screen that accepts a user's operation and an image before the change occurs, and estimates the user's operation based on a parameter corresponding to the difference.

5. The information processing device according to claim 4, in, When the difference is a change in the display mode of the option or a change in the display related to the selection state of the option, the estimation unit estimates the character string of the option whose selection state has changed as the content of the parameter set by the user.

6. The information processing device according to claim 1, in, The information processing apparatus further includes a display unit that displays a confirmation result to a user when accepting execution of a function of confirming whether the content of the stored parameter can be set in a second device different from the first device.

7. The information processing device according to claim 6, in, The result of checking whether the setting can be performed is displayed for each parameter content.

8. The information processing device according to claim 1, in, An indication of the operation of implementing the setting of the content of the parameter is shown through the terminal worn by the user in an augmented reality or mixed reality manner.

9. A computer-readable recording medium storing a program for causing a computer to execute a process having the following steps: estimating the content of the parameter set by the user by analyzing the following images, the images being obtained by capturing changes in the operation screen as the user operates the first device determined as the target of the recorded operation in a time series; saving the estimated contents of said parameters as a record of the operation; and When the following functions are accepted, the following operation screen is displayed to implement the setting of the parameter contents: in, The operation screen is an operation screen of a second device different from the first device, and the function is a function of assisting an operation for setting the contents of a parameter through the operation screen.

10. An information processing method comprising the following steps: estimating the content of the parameter set by the user by analyzing the following images, the images being obtained by capturing changes in the operation screen as the user operates the first device determined as the target of the recorded operation in a time series; saving the estimated contents of said parameters as a record of the operation; and When the following functions are accepted, the following operation screen is displayed to implement the setting of the parameter contents: in, The operation screen is an operation screen of a second device different from the first device, and the function is a function of assisting an operation for setting the contents of a parameter through the operation screen.

11. A computer program product comprising a computer program, the computer program causing a computer to execute a process having the following steps: estimating the content of the parameter set by the user by analyzing the following images, the images being obtained by capturing changes in the operation screen as the user operates the first device determined as the target of the recorded operation in a time series; saving the estimated contents of said parameters as a record of the operation; and When the following functions are accepted, the following operation screen is displayed to implement the setting of the parameter contents: in, The operation screen is an operation screen of a second device different from the first device, and the function is a function of assisting an operation for setting the contents of a parameter through the operation screen.

Images