Information processing device

Abstract

To provide an information processor capable of automatically adjusting a content of operation instructions provided to a user according to an operation status of a user to ensure that the operating instructions of a portion to be checked to not be overlooked while reducing an operational burden on the user to display the operation instructions for the portion to be confirmed.SOLUTION: An information processor with a display unit includes: a status detection unit that detects a status of a user using the information processor; and an operation explanation execution unit that performs operation to provide the user with predetermined operating instruction information by means of including the display on the display unit to assist the user with the operation to be performed. The operation explanation execution unit adjusts the operation to provide instructional information corresponding to the status of the detected user.SELECTED DRAWING: Figure 11

Description

Technical Field

[0001] This disclosure relates to an information processing apparatus, and more particularly, to an information processing apparatus having a function of displaying operation instructions related to operations such as troubleshooting and component replacement operations for eliminating troubles that occur in the information processing apparatus.

Background Art

[0002] Conventionally, image forming apparatuses (also called multifunction printers) having a number of functions such as a printing function, a document reading function, a transmission function of read image data, a FAX transmission / reception function, an e-mail transmission / reception function, an image data acquisition function, a character recognition function, etc. have been used. In an image forming apparatus, when it is necessary to replace consumables such as a toner cartridge or when a problem such as a conveyance trouble of printing paper occurs, it is necessary to perform a consumable replacement operation or an operation to eliminate the trouble.

[0003] Such consumable replacement operations and troubleshooting operations are generally often performed by the person in charge of managing the image forming apparatus, but may also be performed by a person who is not familiar with the replacement operation. Therefore, in order to enable such inexperienced persons to perform operations such as consumable replacement operations, operation instructions (operation guidance) for a series of replacement operations and the like are displayed in an easy-to-understand manner in the order of operations.

[0004] Furthermore, Patent Document 1 describes an information processing device in which, after specifying the parts to be operated on to restore the fault based on the diagnostic content from a fault diagnosis device, a work content observation device detects the operations performed by the worker based on predetermined observation data, and if it is determined from the observation data of the start of work that work has started, it creates a description of all parts, infers the fault content, creates work content to restore the fault, provides support for the work by voice or display according to that work content, observes the worker's current work content to determine whether the current work has been completed or not, and if it has not been completed, it creates a description of the current work part, checks the consistency between the work content defined in the work procedure and the current work content, and provides support for the work again, wherein the next work content is displayed sequentially according to the work content performed by the worker. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 8-202577 [Overview of the project] [Problems that the invention aims to solve]

[0006] However, if the operation instructions (operation guidance) for replacement work are displayed as a video, the person in charge of managing the information processing equipment or other personnel performing the replacement work may perform the prescribed replacement work while watching the video displayed on the screen. Furthermore, for example, the location of the screen displaying the operating instructions may be far from the location where the replacement work is performed, or the user may have to take their eyes off the display screen while working. Thus, when workers performed tasks without being able to view the instructional videos on the display screen, they sometimes missed the parts of the instructional video they wanted to check.

[0007] If you missed the video of the scene you wanted to check, you had to either watch the explanation video again from the beginning, or, if the video was looping, wait for the video of the scene you wanted to check to appear. Furthermore, when explanatory videos are played and stopped for each step of the process, it is necessary to perform a specific input operation to switch screens in order to display the screen for the step that you want to check. However, the video for the scene you want to check may not be displayed immediately, and the operation to call up the video for the scene you want to check is also cumbersome.

[0008] Even when, as in Patent Document 1, videos of the next task are displayed sequentially according to the task performed by the worker, if the worker has to take their eyes off the display screen to perform the task, they may miss the video of the scene they wanted to check, and it may be difficult to quickly retrieve the video of the scene they wanted to check.

[0009] Therefore, this disclosure was made in consideration of the circumstances described above, and aims to enable the automatic adjustment of the content of the operating instructions provided to workers in response to the work status of workers performing tasks such as replacing consumables and other parts, or dealing with equipment troubles, particularly whether or not the workers are looking at the operating instructions (operating guidance), so that the operating instructions for the parts that workers want to check are not overlooked. Another challenge is to reduce the burden on operators who need to display instructions for the parts they want to check. [Means for solving the problem]

[0010] This disclosure provides an information processing device equipped with a display unit, comprising: a state detection unit that detects the state of a user using the information processing device; and an operation instruction execution unit that performs an operation to provide the user with predetermined operation instruction information by means including displaying it on the display unit in order to support the user's operations, wherein the operation instruction execution unit adjusts its operation to provide the operation instruction information in accordance with the detected state of the user.

[0011] Furthermore, the system is further equipped with a shooting unit that captures images of the space in the vicinity of the information processing device, and the state detection unit is characterized by comprising a human body detection unit that detects a person present in the vicinity of the information processing device and acquires user detection information indicating the presence or absence of a person, and an image recognition unit that performs image recognition on the captured image data captured by the shooting unit and acquires predetermined recognized image information through image recognition.

[0012] Furthermore, the system includes a user status determination unit that determines the state of a user using the information processing device, and the user status determination unit uses the user detection information and the recognized image information to determine whether or not the user is viewing the operation instruction information displayed on the display unit.

[0013] Furthermore, if the user status determination unit determines that the user is not viewing the displayed operation instructions while the operation instruction execution unit is performing the operation to provide the operation instruction information, the operation instruction execution unit will stop performing the operation to provide the operation instruction information.

[0014] Furthermore, if the operation instruction execution unit has stopped providing the operation instruction information, and the user status determination unit determines that the user is viewing the displayed operation instruction information, the operation instruction execution unit will resume providing the operation instruction information.

[0015] Furthermore, when resuming the operation of providing the operation instruction information, the provision of the operation instruction information is restarted from a position a certain time before the position where the operation of providing the operation instruction information was stopped.

[0016] Furthermore, the recognized image information acquired by the image recognition unit through image recognition is characterized by including the position of the head, the position of the eyes, and the direction of the gaze.

[0017] Furthermore, if the user detection information acquired by the human body detection unit indicates that a user is in the vicinity of the information processing device, and if it is determined that the user may be viewing the operation instruction information based on the position of the head, the position of the eyes, or the direction of the gaze in the recognized image information, the user state determination unit determines that the user is viewing the displayed operation instruction information.

[0018] Furthermore, if the user detection information acquired by the human body detection unit indicates that the user is not in the vicinity of the information processing device, or if it is determined that the user may not be looking at the operation instruction information based on the position of the head, the position of the eyes, or the direction of the gaze in the recognized image information, the user state determination unit determines that the user is not looking at the displayed operation instruction information.

[0019] Furthermore, the operation instruction information consists of a plurality of pieces of information that are set in advance in association with the state of the information processing device for each of the plurality of operations to be performed in succession, and when the user performs any of the plurality of operations, the operation instruction execution unit switches the operation instruction information displayed on the display unit to the operation instruction information set in association with the performed operation and the current state of the information processing device.

[0020] Furthermore, providing the user with operational instructions may include outputting the instructions as audio, and providing the user with operational instructions may also include automatically setting whether or not to output the instructions as audio and automatically adjusting the volume of the output audio in accordance with the ambient noise. Furthermore, the operation instruction information is characterized in that it includes video footage that is displayed on the display unit when a malfunction occurs in the information processing device or when it becomes necessary to replace a part of the information processing device. Furthermore, the information processing device is characterized by being an image forming apparatus having multiple functions.

[0021] Also, this disclosure provides a guidance playback adjustment method for an information processing apparatus, comprising: a state detection step in which a control unit provided in the information processing apparatus detects the state of a user who uses the information processing apparatus; and an operation explanation execution step in which, in order to assist an operation to be performed by the user, means including display provides the operation explanation information to the user. In the operation explanation execution step, the operation of providing the operation explanation information is adjusted in accordance with the detected state of the user.

[0022] Further, the state detection step includes: a human body detection step in which a person present near the information processing apparatus is detected and user detection information indicating the presence or absence of the person is acquired; an image recognition step in which image recognition is performed on captured image data of the space near the information processing apparatus and predetermined recognition image information is acquired by the image recognition. After the state detection step, a user state determination step is further provided, in which it is determined whether or not the user is in a state of viewing the displayed operation explanation information by using the user detection information and the recognition image information. In the operation explanation execution step, when it is determined in the user state determination step that the user is not in a state of viewing the displayed operation explanation information while the operation of providing the operation explanation information is being performed, the operation of providing the operation explanation information is stopped.

Advantages of the Invention

[0023] According to this disclosure, in accordance with the detected state of the user, the operation explanation execution unit adjusts the operation of providing predetermined operation explanation information to the user in order to assist an operation to be performed by the user. For example, it is possible to automatically provide appropriate operation explanation content corresponding to the working situation of an operator performing troubleshooting work or component replacement work on the information processing apparatus, etc., and prevent the operator from missing the operation explanation of the part that the operator wants to check.

[0024] Furthermore, if the operation instruction information consists of multiple pieces of information pre-set in association with the state of the information processing device for each of the multiple operations to be performed consecutively, when the user performs any of the multiple operations, the operation instruction execution unit switches the operation instruction information displayed on the display unit to the operation instruction information set in association with the performed operation and the current state of the information processing device. This reduces the burden on the user to perform operations such as displaying the operation instructions for the part they want to check. [Brief explanation of the drawing]

[0025] [Figure 1] This is a block diagram of one embodiment of the image forming apparatus disclosed herein. [Figure 2] This is an explanatory diagram illustrating one embodiment of the information stored in the memory unit of the image forming apparatus disclosed herein. [Figure 3] This is an explanatory diagram illustrating one embodiment of the information stored in the memory unit of the image forming apparatus disclosed herein. [Figure 4] This is an explanatory diagram of one embodiment of a screen (malfunction warning screen) displayed on the display unit of the image forming apparatus disclosed herein. [Figure 5] This diagram illustrates one embodiment of a screen (parts replacement request warning screen) displayed on the display unit of the image forming apparatus disclosed herein. [Figure 6] This diagram illustrates one embodiment of the screen (initial screen for preparing to play back operation instructions) displayed on the display unit of the image forming apparatus disclosed herein. [Figure 7] This is an explanatory diagram of one embodiment of a screen (operation instruction display screen) displayed on the display unit of the image forming apparatus disclosed herein. [Figure 8] This is an explanatory diagram of one embodiment of the operation instruction video information corresponding to the paper jam condition of the image forming apparatus disclosed herein. [Figure 9] This is an explanatory diagram of one embodiment of the display screen for operation instructions corresponding to the status of the image forming apparatus disclosed. [Figure 10] This is a flowchart of one embodiment of the guidance playback adjustment process of the image forming apparatus disclosed herein. [Figure 11]This is a flowchart of one embodiment of the guidance playback adjustment process of the image forming apparatus disclosed herein. [Figure 12] This is a flowchart of one embodiment of the guidance playback adjustment process of the image forming apparatus disclosed herein. [Modes for carrying out the invention]

[0026] Embodiments of this disclosure will be described below with reference to the drawings. However, the description of the following embodiments does not limit this disclosure.

[0027] The information processing device disclosed herein is a device that has a function to explain the content of predetermined operations and tasks performed on the information processing device. For example, it has a function to inform workers, such as managers, of the operations to be performed during tasks such as replacing consumables and other replacement parts attached to the information processing device, or during maintenance work or replacement of faulty parts when troubleshooting problems that occur in the information processing device.

[0028] Furthermore, the information processing device shall have pre-stored operation instruction information (hereinafter referred to as operation guidance, or simply guidance) that explains the content of operations performed on the information processing device. For example, the operation instruction information includes pre-stored information about the operations to be performed for each replacement task, such as the replacement of replacement parts (operation location, operation content, operation sequence, operation images, etc.).

[0029] In the following embodiments, an image forming apparatus corresponding to an information processing apparatus, which has multiple functions, will be described. However, the information processing device is not limited to an image forming apparatus; it can be any electronic device with multiple functions, such as a personal computer, scanner, display device, MMR, ATM, or digital signage.

[0030] <Configuration of an image forming apparatus> Figure 1 shows a block diagram of one embodiment of the image forming apparatus disclosed herein. A multifunction peripheral (MFP) is a device that processes image data, primarily performing tasks such as input, formation, output, storage, and transfer of image data. The image forming apparatus 1 is also called a multifunction printer, or simply an MFP. Furthermore, the image forming apparatus 1 is an electronic device equipped with functions for processing image data, such as a copying function, a printing function, a document reading (scanning) function, a document editing function, a document storage function, a document transmission / reception function (fax transmission / reception, scanner document transmission), an email transmission / reception function, a file transfer function, and a communication function.

[0031] In the following embodiments, the image forming apparatus 1 of this disclosure will be described in particular as having a copying function, a scanning function, and a communication function, but it may also have other functions.

[0032] In Figure 1, the disclosed image forming apparatus (MFP) 1 mainly comprises a control unit 11, an operation unit 12, a display unit 13, an image processing unit 14, a communication unit 15, a malfunction detection unit 21, a parts replacement detection unit 22, an imaging unit 23, a human body detection unit 24, an image recognition unit 25, an operation instruction display control unit 26, a user status determination unit 27, an operation instruction execution unit 28, and a storage unit 40.

[0033] Here, the image processing unit 14, as will be described later, mainly consists of an image input unit, an image forming unit, and an image output unit. Furthermore, the human body detection unit 24 and the image recognition unit 25 correspond to a state detection unit that detects the state of the user using the image forming apparatus.

[0034] The control unit 11 controls the operation of each component, such as the display unit 13 and the image processing unit 14, and is mainly implemented by a microcomputer consisting of a CPU, ROM, RAM, I / O controller, timer, etc. The CPU operates various hardware components organically based on control programs pre-stored in non-volatile memory such as ROM, thereby executing the image formation function, image recognition function, and other functions disclosed herein.

[0035] Furthermore, among the above components, the user status determination unit 27, the operation instruction execution unit 28, etc., are functional blocks in which the CPU performs its respective processing based on a predetermined program.

[0036] The operation unit 12 is an input device for the user of the image forming apparatus 1 to perform predetermined input operations. For example, it is the part for inputting information such as characters or selecting functions, and a keyboard, mouse, or touch panel may be used. User-operated keys include the start key, function selection key, and settings key. For example, users can initiate document scanning by pressing a touch panel or a key to start the scanning operation, or initiate the transmission of information such as image data to a designated destination by pressing a transmission start key. Furthermore, while the operation instructions (operation guidance) are displayed, you can switch to the next screen or return to the previous screen by performing a predetermined input operation.

[0037] The display unit 13 is the part that displays information, and it displays information necessary for the execution of each function, as well as the results of the execution of the function, in order to inform the user. For example, if an LCD or organic EL display is used and a touch panel is used as the operation unit 12, the display unit 13 and the touch panel are arranged to overlap each other. The display unit 13 displays information such as settings for printing and other operations of the image forming apparatus, information necessary for executing functions such as document scanning, a screen for selecting the function to be executed (function selection screen), an operation screen for the selected function, and a screen for requesting replacement of consumables, using characters, symbols, figures, images, icons, animations, videos, etc.

[0038] Furthermore, this disclosure specifically mentions that, as described later, the display unit 13 displays a screen for when a malfunction occurs (malfunction warning screen), a screen requesting the replacement of consumables or other parts (parts replacement request warning screen), a screen for selecting whether or not to view an operation instruction video (initial operation instruction playback preparation screen), and an operation instruction display screen (operation instruction display screen).

[0039] The image processing unit 14 is the part that performs the image forming function, which is the main function of the image forming apparatus 1, and mainly consists of an image input unit, an image forming unit, and an image output unit. Primarily, the image input unit is responsible for inputting predetermined image data, the image forming unit is responsible for converting the input image data into information that can be printed, etc., and the image output unit is responsible for outputting the formed print information, etc., onto printing paper, etc.

[0040] The image input section is where image data of a document containing images, text, graphics, etc., such as print data intended for printing, is input. For example, it is the section that reads a document placed on a document glass or the like. For image input, a scanner (reading device) that reads documents containing information is used. The image forming apparatus 1 includes a document tray on which a document is placed and a document cover that holds the document in place, in order to read the document. Furthermore, the image forming apparatus 1 may be equipped with an automatic document feeder (ADF) that can take multiple documents and automatically transport and scan them one by one.

[0041] There are various methods for inputting image information, but for example, a document containing images can be scanned, and the image data of the document (hereinafter referred to as input image data) can be stored in the storage unit 40.

[0042] Furthermore, an interface for connecting external storage media, such as a USB memory stick, corresponds to the image input section. Electronic data files, such as image information to be input, may be saved on an external storage medium such as a USB memory stick. By connecting the USB memory stick to an input interface such as a USB terminal and performing a predetermined input operation on the operation unit 12, the desired electronic data files saved on the USB memory stick can be read and stored in the storage unit 40 as input image data.

[0043] When printing print data onto a recording medium, the image forming unit generally performs the following steps in sequence: charging, exposure, development, transfer, cleaning, static discharge, and fixing, to form the print data on the recording medium. In the development process, toner is supplied from the toner cartridge to the developing device, and the electrostatic latent image formed on the surface of the charged photoreceptor drum is developed, forming a toner image corresponding to the electrostatic latent image. The toner image formed on the surface of the photoreceptor drum is transferred onto the recording medium by a transfer device, and then fixed onto the recording medium by heating by a fixing device. The image forming unit also converts the input image data into information in a format that can be transferred and displayed.

[0044] The image output unit is the part that outputs the formed input image data, and is equivalent to, for example, a printer. However, the output of input image data is not limited to printing; it also includes storing input image data of scanned documents and sending input image data of scanned documents via fax. For example, storing scanned input image data on an external storage medium such as a USB memory stick, transmitting input image data to other information processing devices or servers via a network such as the internet, and classifying and saving it in a specific save folder (document filing) are also considered equivalent to image output.

[0045] The communication unit 15 is the part that communicates data with other communication devices via the network 2. Network 2 shall be able to utilize any existing communication lines, such as WANs (Internet), LANs, telephone lines, and dedicated lines. The connection to Network 2 can utilize any existing communication method, and the communication method can be either wired or wireless. For example, the image forming apparatus (MFP) 1 may transmit information regarding operation instructions (such as guidance videos), malfunctions, or the need for parts replacement to a mobile device held by a manager via the network 2.

[0046] The malfunction detection unit 21 is the part that detects when some kind of malfunction occurs while the image forming apparatus is performing its functions. For example, if the copying function is running, it can detect if the paper jam occurs in the transport path. Furthermore, when using the automatic document feeder (ADF) to scan multiple documents, the system can detect if a document jams in the transport path. Alternatively, it can detect malfunctions such as abnormalities during paper ejection. For example, a paper jam in the transport path is detected by sensors installed along the transport path that detect the failure to pass the paper. If a malfunction is detected, a malfunction warning screen will be displayed, as described below.

[0047] The parts replacement detection unit 22 is the part that detects whether or not a part needs to be replaced. For example, it can detect the need to replace toner cartridges, replenish printing paper, and replace parts such as ink ribbons and roll paper in dye-sublimation printers. The need to replace a toner cartridge can be detected by receiving a signal from the cartridge indicating that the toner has run out. The need to replenish printing paper can be detected by checking signals from sensors or switches located in the paper tray that indicate the presence or absence of paper. If it is detected that a part needs to be replaced, a part replacement request warning screen will be displayed, as described below.

[0048] The imaging unit 23 is the part that captures images of the space in the vicinity of the image forming apparatus, and corresponds to a camera. For example, a camera is attached to the top surface of the main body of the image forming apparatus or to the space above it to photograph the image forming apparatus itself or the space around the image forming apparatus, and captured image data 42 is obtained. In this disclosure, the imaging unit 23 specifically captures the posture and work status of users who perform input operations using the operation unit 12 to cause the image forming apparatus to execute a desired function, as well as management personnel performing maintenance work such as replacing parts like toner cartridges. Furthermore, while the imaging unit 23 only requires at least one camera to be attached to the image forming apparatus, it is preferable to attach multiple cameras to positions such as the front, diagonally above, top, and side of the image forming apparatus in order to capture the operator's work status and other information related to the image forming apparatus.

[0049] Alternatively, the imaging unit 23 may be mounted not directly on the image forming apparatus, but on the surrounding walls or ceiling, or on other peripheral equipment (ATM, MMK, etc.) in order to photograph the work status of the supervisor from all angles. Image data captured by the camera unit 23 is stored in the memory unit 40 as captured image data 42. Furthermore, the captured image data 42 is used for image recognition, and as will be described later, the position of the person's head, the position of their eyes, and the direction of their gaze are obtained from the captured image data. Furthermore, by recognizing people from captured image data, it can also be used for detecting human bodies.

[0050] The human body detection unit 24 is a part that detects a person present in the vicinity of the image forming apparatus and acquires user detection information 45 indicating whether or not a person is present. For example, it can detect the presence of a person operating an image forming apparatus to perform a specific function, or a person performing maintenance work on a malfunction, or a person replacing parts. Whether or not a person has been detected will be stored in the memory unit 40 as user detection information 45, as described later. To primarily detect the presence or absence of a person in the vicinity of the image forming apparatus, for example, an infrared sensor, an acoustic sensor, or an underfloor weight sensor can be mounted on the upper part of the frame of the front panel of the image forming apparatus. Furthermore, if it is necessary to identify individuals, determine their posture, or detect the tasks they are performing, the aforementioned captured image data can be used for image recognition.

[0051] The image recognition unit 25 is the part that performs predetermined image recognition on the captured image data 42 captured by the shooting unit 23, and acquires predetermined recognized image information 46 through image recognition. The system recognizes the posture and condition of a person performing a specified task, such as replacing parts, and in particular recognizes captured image data 42 acquired when operation instruction information 43 is displayed. From this captured image data, it obtains recognition image information indicating the condition of a person performing an operation or checking the operation instructions.

[0052] For example, the system recognizes whether or not an image corresponding to a person is present in the captured image data 42. If it is recognized that an image corresponding to a person is present, it stores information indicating that the presence of a person has been detected. Furthermore, if it is detected that an image of a person is included, the system will then obtain the position of the person's head and eyes from that image, and from the partial image corresponding to the eye position, it will obtain the direction in which the person is looking, i.e., the direction of their gaze.

[0053] Furthermore, the system may recognize the person's posture and hand movements from the image, and then recognize the operation or work performed on the image forming apparatus. Furthermore, by examining the facial portion of a person's image, it may be possible to determine whether a person near the image forming apparatus is the person in charge of managing the image forming apparatus. Alternatively, the system may recognize, based on the image of the person and the image of the main body of the image forming apparatus, whether the person is in a position where they can perform input operations on the image forming apparatus, or whether the person is in a position away from the image forming apparatus. The information recognized by the image recognition unit 25 is acquired as recognized image information 46 and stored in the storage unit 40. In the following embodiment, the recognized image information 46 mainly includes the position of the head, the position of the eyes, and the direction of the gaze, which are obtained from the person image.

[0054] The operation instruction display control unit 26 is the part that displays operation instruction information (operation guidance) related to specific tasks to be performed on the image forming apparatus on the display unit 13. When displaying operation instructions (operation guidance), the instructions and tasks that administrators should perform should be explained using text, symbols, graphics, still images, videos, photographs, etc. For example, when it's time to replace a consumable part of an image forming machine, the system will display operational instructions (operation guidance) regarding the replacement process. Furthermore, in the case of any malfunction in the image forming apparatus, such as a paper transport problem, operation instructions (operation guidance) will be displayed to resolve the problem.

[0055] Furthermore, in cases where the operations or tasks to be explained are complex, the instruction information will be displayed using not only text and still images, but also videos. However, operational instructions (operational guidance) may be output not only as a display but also as audio data. In particular, when a supervisor is not nearby, voice data may be used to first notify the user of requests for parts replacement or the occurrence of a problem, before displaying operation instructions (operation guidance).

[0056] The user status determination unit 27 is the part that determines the status of the user (person) using the image forming apparatus. The user's status refers to the presence or absence of the user, the user's posture, the user's actions, the user's work, etc. Determining the user's state means, for example, determining whether a user (person) near the image forming apparatus is viewing the operation instruction information (for example, an operation instruction video) displayed on the display unit 13. Alternatively, the system can determine whether the front panel door of the image forming apparatus is open, whether a paper jam is being removed, or whether the door has been closed. Furthermore, the user's condition may also be assessed during tasks such as unpacking consumables.

[0057] The user's status is determined using the user detection information 45 and the recognized image information 46 described above, and the result of this determination is set in the user status information 47, which will be described later. User status information 47 may include, for example, whether the user has confirmed the explanation or not, as will be described later.

[0058] For example, as will be described later, if the user detection information 45 acquired by the human body detection unit 24 indicates that the user is in the vicinity of the information processing device, and if it is determined that the user may be looking at the operation instruction information 43 based on the position of the head, the position of the eyes, or the direction of the gaze in the recognized image information 46, the user state determination unit 27 determines that the user is looking at the displayed operation instruction information 43.

[0059] Furthermore, if the user detection information 45 acquired by the human body detection unit 24 indicates that the user is not in the vicinity of the image forming apparatus, or if it is determined that the user may not be looking at the operation instruction information based on the head position, eye position, or gaze direction of the recognized image information 46, the user state determination unit 27 determines that the user is not looking at the displayed operation instruction information 43.

[0060] The operation instruction execution unit 28 is a part that performs the operation of providing predetermined operation instruction information to the user by means including display on the display unit 13, in order to support the operations that the user should perform. Furthermore, the operation instruction execution unit 28 adjusts its operation to provide operation instruction information in accordance with the user's state detected by the state detection unit. The actions that provide operating instructions include displaying the operating instructions and outputting the operating instructions audibly.

[0061] For example, if a malfunction occurs, the operation instruction execution unit 28 displays a video on the display unit 13 explaining the operations and procedures to resolve the malfunction. Furthermore, if consumables such as toner cartridges need to be replaced, an instructional video to assist with the replacement process will be displayed on the display unit 13.

[0062] Furthermore, in adjusting the operation that provides the aforementioned instructional information, the display state of the instructional video is adjusted using the user status information 47. Adjusting the video display state means, for example, starting, pausing, resuming the display of an explanatory video from a certain point in time, changing the playback position of the video, resuming from a certain point in time, resuming from a position corresponding to the user's actions, zooming in on a part of the video corresponding to the user's actions, changing the playback speed of the video, and rewinding the video during playback. The adjustment of operational instruction information is not limited to video display; it also includes features such as enlarged still images of the most important locations during replacement work, highlighting with flashing lights, and voice prompts for key work points.

[0063] Regarding the adjustment of the above-mentioned operations, for example, if the operation instruction execution unit 28 is performing the operation of providing operation instruction information, and the user status determination unit 27 determines that the user is not viewing the displayed operation instruction information 43, the operation instruction execution unit 28 will stop the operation of providing operation instruction information.

[0064] Furthermore, if the operation instruction execution unit 28 has stopped providing operation instruction information, and the user status determination unit 27 determines that the user is viewing the displayed operation instruction information 43, the operation instruction execution unit 28 will resume providing the operation instruction information.

[0065] Furthermore, as will be described later, when resuming the operation of providing instructional information, the provision of instructional information will be restarted from a position a certain time before the position where the operation of providing instructional information was stopped.

[0066] Furthermore, as will be explained in the later-described embodiment of display screen switching, the operation instruction information 43 consists of a plurality of pieces of information pre-set in association with the state of the image forming apparatus for each of the plurality of operations to be performed consecutively. When the user performs any of the plurality of operations, the operation instruction execution unit 28 switches the operation instruction information 43 displayed on the display unit 13 to the operation instruction information 43 that has been set in association with the performed operation and the current state of the information processing device.

[0067] The memory unit 40 is the part that stores information and programs necessary to perform each function of the image processing apparatus MFP disclosed, and uses semiconductor memory elements such as ROM, RAM, and flash memory, storage devices such as HDDs and SSDs, and other storage media. The memory unit 40 stores, for example, input image data 41, captured image data 42, operation instruction information 43, detection operation response information 44, user detection information 45, recognized image information 46, user status information 47, and guide stop position information 48. Figures 2 and 3 illustrate an example of the information stored in the memory unit 40 of the image forming apparatus.

[0068] The input image data 41 is an image input to the image forming apparatus, for example, image data input when a copy function is performed, or image data input when a read function is performed.

[0069] The captured image data 42 is image data captured by the imaging unit 23. The captured image data 42 may be a still image or a video.

[0070] The operation instruction information 43 is information (operation guidance) that explains a series of operations to the user or manager of the image forming apparatus. It mainly includes information that is displayed on the display unit 13, such as a video, when a malfunction occurs in the image forming apparatus or when it becomes necessary to replace a part of the image forming apparatus, and is stored in advance in the storage unit 40. For example, when a user wants to perform a desired function and wants to know how to perform that function, the explanation of the operation method displayed on the display unit 13 corresponds to the operation instruction information 43.

[0071] Furthermore, when replacing consumables such as toner cartridges, the instructions on how to replace them (toner cartridge replacement guidance) displayed on the display unit 13 correspond to the operation instruction information 43. Alternatively, if a paper transport problem occurs, instructions on how to remove jammed paper (paper transport error response guidance) correspond to operation instruction information 43. If the instructions for operation consist of multiple steps, the instructions will be written according to the order of those steps, and the instructions will be displayed sequentially in the order of the written steps by performing a predetermined input operation. Furthermore, as mentioned above, the operation instruction information 43 is generally stored in the form of still images or videos displayed on the display unit 13, but it may also be stored in the form of audio data.

[0072] The detection operation correspondence information 44 is information that associates predetermined detection conditions with operation instruction actions (guidance actions) that are performed when those detection conditions are met, and is stored in the storage unit 40 in advance. Figure 2 shows an explanatory diagram of one embodiment of the detection operation correspondence information 44 stored in the memory unit 40 of the image forming apparatus. In the detection operation correspondence information 44 in Figure 2, the detection conditions consist of user detection and image recognition results. Furthermore, the guidance function is described as consisting of playing and stopping video display, and outputting and stopping audio. The areas marked with a circle in Figure 2 indicate that the condition is met and that action is performed. However, the detection conditions are not limited to these two conditions, and the guidance operation is not limited to video and audio.

[0073] The detection condition "User detected" indicates the content stored in the current user detection information 45, which is the detection result of the human body detection unit 24, and is either the presence or absence of a user. The "image recognition result" in the detection conditions refers to the content of the recognized image information 46 acquired by the image recognition unit 25, and here it refers to the position of the head, the position of the eyes, and the direction of the gaze, which are obtained by image recognition of the detected person. Here, it is assumed that the position of the person's head, the position of their eyes, and the direction of their gaze can be used to determine whether or not the person is likely looking at the display screen showing the explanatory video.

[0074] Generally, if a person is looking at the screen displaying the explanatory video, it is acceptable to continue displaying the video. However, if they are not looking at the screen displaying the explanatory video, they may want to watch parts of the video they missed, and it may be better to stop playback. If you need to watch a video you missed, you will need to rewind the video playback, or if there is no rewind function, play the video again from the beginning. Therefore, if there is a possibility that the person is not looking at the screen displaying the explanatory video, the video display will be stopped.

[0075] For example, if "User Detection" detects that there are no users, it is determined that there is a possibility that the person is not viewing the explanatory video screen. If "User Detection" detects that there are users, it is determined that there is a possibility that the person is viewing the explanatory video screen.

[0076] Furthermore, if the position of the person's head is recognized as facing forward relative to the position of the image forming apparatus, and the distance between the head and a specific position on the image forming apparatus (for example, the position of the top panel) is within a predetermined distance (for example, 50 cm), it is determined that the person may be looking at the display screen of the explanatory video. On the other hand, if the position of a person's head is recognized as being oriented laterally to the position of the image forming apparatus and not facing forward, and the distance between the head and a specific position on the image forming apparatus (for example, the position of the top panel) is greater than a predetermined distance (for example, 50 cm), it is determined that the person may not be looking at the screen displaying the explanatory video. Alternatively, if a person's head is facing downwards, it may be possible to conclude that the person is not looking at the screen displaying the explanatory video.

[0077] Furthermore, if the position of a person's eyes is recognized as facing directly towards the position of the image forming apparatus, and the distance between the eyes and a specific position on the image forming apparatus (for example, the position of the top panel) is within a predetermined distance (for example, 50 cm), it is determined that the person may be looking at the display screen of the explanatory video. On the other hand, if the position of a person's eyes is recognized as being oriented laterally to the position of the image forming apparatus and not facing forward, and the distance between the eyes and a specific position on the image forming apparatus (for example, the position of the top panel) is greater than a predetermined distance (for example, 50 cm), it is determined that the person may not be looking at the display screen of the explanatory video. Alternatively, if a person's eyes are looking downwards, it may be possible to conclude that the person is not looking at the screen displaying the explanatory video.

[0078] Furthermore, even if a person's eyes are facing directly towards the image forming apparatus, their gaze may not be directed towards the display screen. The direction of gaze can be detected, for example, by extracting a partial image of the eye's position from the captured image data and analyzing the orientation of the cornea (the black part of the eye) in that partial image. Thus, if the detected gaze direction is directed towards the display screen, it is determined that the person may be looking at the display screen of the explanatory video. If the detected gaze direction is not directed towards the display screen, it is determined that the person may not be looking at the display screen of the explanatory video.

[0079] The detection operation correspondence information 44 in Figure 2 is a set of guidance operations that correspond to the judgment results of the detection conditions described above. In Figure 2, the "x" marks next to the head position, eye position, and gaze direction indicate that, in the respective recognition results, it was determined that the person may not be looking at the explanatory video screen, while the "o" marks indicate that it was determined that the person may be looking at the explanatory video screen.

[0080] For example, if the user detection condition is "not present," the system is configured to stop the video display and output audio as a guidance action, regardless of the image recognition result. In this case, it means there is no person near the image forming apparatus, so there is no need to play the video. If a video is already playing, it will be paused. Furthermore, when video playback is paused, it is preferable that the audio corresponding to the video also stops. The audio output at this time should, for example, say, "Video playback has been paused. To resume the video, please look at the displayed screen."

[0081] Next, if the user detection condition is "present," the video display of the guidance operation will differ depending on the image recognition result. For example, if the user detection condition is "present," and the judgments regarding head position, eye position, and gaze direction all indicate that the person may not be looking at the explanatory video screen, then if the video is currently playing, the video playback will be paused. In this case as well, audio will be output, but for example, the audio corresponding to the currently playing video will be stopped, and an audio message such as "Video playback has been paused. Please look at the screen to resume playback" will be output.

[0082] On the other hand, if the user detection condition is "present," and the judgment of head position, eye position, and gaze direction all indicate that there is a possibility that the person is looking at the explanatory video screen, then the video playback will start, and if the video is already playing, it will continue playing. In this case, if you want to output audio, output the audio that corresponds to the video being played. Furthermore, as will be described later, if video playback has been paused, when resuming playback, it may be resumed from the paused position. However, to make it easier for the worker to review the video, the playback position may be set to a position (rewind position) a certain amount of time before the current paused position before resuming playback. For example, by rewinding the video by about 10 seconds, you can be sure to check any parts of the instructions you missed, making the content of the instructions easier to understand.

[0083] Furthermore, if the user detection condition is "present," and if any one of the head position, eye position, or gaze direction judgments indicates that the person may be viewing the explanatory video screen, the video playback will start. If the video is already playing, it will continue playing, and if audio is to be output, the audio corresponding to the currently playing video will be output.

[0084] Furthermore, if the user detection condition is "present," and any two of the judgments regarding head position, eye position, and gaze direction indicate that there is a possibility that the person is looking at the explanatory video screen, the video playback may be started, and if the video is already playing, it may continue playing, and if audio is to be output, the audio corresponding to the video being played may be output.

[0085] However, if, based on the judgment of head position, eye position, or gaze direction, one of these factors indicates that the person may be looking at the explanatory video screen, and conversely, the other two factors indicate that the person may not be looking at the explanatory video screen, then, if it is deemed appropriate to stop the video display, then, contrary to the guidance action shown in the diagram, a setting that means pausing the video playback may be used.

[0086] The user detection information 45 is information indicating the detection status of a person using the image forming apparatus, and is stored by the human body detection unit 24 described above. Figure 3 shows an explanatory diagram of one embodiment of the user detection information 45. In the user detection information 45 in Figure 3, H1 means "human body detected," and H0 means "human body not detected."

[0087] The recognized image information 46 is information recognized by the image recognition unit 25. In this disclosure, in particular, information regarding the head, eyes, and gaze of a person in an image is obtained from the captured image data 42 as recognized image information 46. For example, as shown in Figure 3, the recognition image information 46 includes information about the position of the head, the position of the eyes, and the direction of the gaze.

[0088] User status information 47 is information indicating the current status of the user of the image forming apparatus. For example, it shows the current state of a person who is near the image forming apparatus or a person who is performing some kind of operation. As already mentioned above, user status information 47 is set using user detection information 45 and recognized image information 46. Figure 3 shows an explanatory diagram of one embodiment of the user status information 47. Here, the results determined using user detection information 45 and recognized image information 46 are shown, where ST1 means "explanation confirmed" and ST0 means "explanation not confirmed".

[0089] "Explanation Confirmed State" refers to a state where a user of the image forming apparatus, such as the administrator of the image forming apparatus, is viewing the operation instruction screen, while "Explanation Not Confirmed State" refers to a state where a user of the image forming apparatus, such as the administrator of the image forming apparatus, is not viewing the operation instruction screen.

[0090] Furthermore, as already explained, it is preferable for the instructional video to play when the user is viewing the instruction screen (instruction confirmation state). However, if the user is not viewing the instruction screen (instruction not confirmed), continuing video playback may cause the user to miss the video they wanted to watch. Therefore, to prevent the user from missing the instructional video and to allow them to easily watch any missed parts of the video without having to perform an operation to recall the desired section, the instructional video playback is paused.

[0091] Guide stop position information 48 indicates the position where the instructional video was paused. Guide stop position information 48 is stored when the instructional video is paused and is used when playback of the video is resumed. For example, when displaying a 5-minute instructional video, if the video is paused 2 minutes after playback begins, the system stores information identifying the video being played (for example, the video's name or file name) and the paused position ("2 minutes") in the guide stop position information 48, associating them with each other. Subsequently, when resuming playback of a paused video, the time information stored in the guide stop position information 48 is used to find the position where the video should resume.

[0092] The point at which the video resumes can be the same as the stored time information, but as mentioned above, it is preferable to set it to a time that has been rewound by a certain amount (for example, 10 seconds before the paused position). For example, if the video was paused at "2 minutes" and the restart point is set to 10 seconds before the pause, the video will resume playback from 1 minute and 50 seconds after the start of playback.

[0093] <Example of display screen> Figure 4 shows an explanatory diagram of one embodiment of the screen (malfunction warning screen) displayed on the display unit of an image forming apparatus. The malfunction warning screen (G1) is displayed when a malfunction occurs, such as a paper jam or the need to replace consumable parts. Users who see this screen may resolve the problem themselves if they know how, but it is preferable to inform a store employee, especially in places like convenience stores.

[0094] In Figure 4, the malfunction warning screen (G1) shows an area labeled "OK (Confirm)." This area is where you select and input information after confirming the content displayed on this warning screen. For example, when the person in charge of managing the image forming machine checks the content displayed on the warning screen, they select and input "OK (Confirm)". If "OK (Confirm)" is selected, the initial screen for preparing to play the operation instructions, as shown in Figure 6 below, will be displayed. However, the "OK (Confirmation)" display area is optional, and the display of this screen may be removed after a certain period of time. Note that the content displayed on the malfunction warning screen is not limited to what is shown in Figure 4.

[0095] Figure 5 shows an explanatory diagram of one embodiment of the screen (parts replacement request warning screen) displayed on the display unit of an image forming apparatus. The parts replacement request warning screen (G2) is displayed when the printer runs out of paper, the toner cartridge runs out of toner, or when parts such as ink ribbons or roll paper in dye-sublimation printers need to be replaced. Users who see this screen may resolve the problem themselves if they know how, but it is preferable to inform a store employee, especially in places like convenience stores. In Figure 5, the parts replacement request warning screen (G2) shows an area labeled "OK (Confirm)." Similar to Figure 4, this area is for selecting and inputting information after confirming the content displayed on this warning screen. Also, similar to Figure 4, the "OK (Confirmation)" display area is optional, and the display of this screen may be removed after a certain period of time. The content displayed on the parts replacement request warning screen is not limited to what is shown in Figure 5.

[0096] Figure 6 shows an explanatory diagram of one embodiment of the screen displayed on the display unit of an image forming apparatus (initial screen for preparing to play back operation instructions). The initial screen for preparing to play the operation instructions (G3) is displayed before the operation instructions (guidance) video or other video is shown, and it is also a screen for selecting whether or not to display the operation instructions. For example, as mentioned above, if the "OK (Confirm)" display area is selected in the warning screens shown in Figures 4 and 5, the screen shown in Figure 6 will be displayed.

[0097] If the person handling the problem (for example, the administrator) or the person replacing the parts is unfamiliar with or unaware of the troubleshooting process, it is preferable to display operating instructions (guidance). However, if the person handling troubleshooting (for example, the administrator) already knows and is familiar with the troubleshooting process, it may not be necessary to display the operating instructions (guidance). Therefore, before displaying the instructional video (guidance), the initial screen for preparing to play the instructional video is displayed on G3, allowing the user to select whether or not to view the instructional video. Figure 6 shows the initial screen (G3) for preparing to play the instruction manual, with display areas for "Watch instruction manual video" and "Do not watch instruction manual video."

[0098] If someone unfamiliar with troubleshooting is handling the issue, they should select "Watch the instructional video." If "Watch instructional video" is selected, the instructional video will start playing. On the other hand, if the person handling the issue is familiar with the operation, such as troubleshooting, they can simply select "Do not watch the operation instruction video." If "Do not watch instructional video" is selected, the instructional video will not play. However, if the system is set to always display instructional videos (guidance), the instructional videos may be displayed immediately without showing the initial instructional video playback preparation screen G3.

[0099] Figure 7 shows an explanatory diagram of one embodiment of the screen (operation instruction display screen) displayed on the display unit of an image forming apparatus. Figure 7 shows the operation instruction display screen (G4), which displays the operation instructions (guidance) when the printer runs out of paper. This indicates that a video demonstrating the procedure for replenishing the depleted printing paper will be displayed in the center of the screen. For example, images showing the location of the paper tray for replenishing the paper, pulling out the paper tray, placing the paper into the paper tray, and returning the paper tray to its original position will be displayed in this order.

[0100] Furthermore, when displaying multiple images simultaneously, there may be operations that the operator does not understand. Therefore, it is acceptable to pause the video after each video finishes displaying, and then, once the operator performs the required input operation, proceed to the next image or return to the previous image. Figure 7 shows the operation instruction screen (G4), which displays the areas for "Next video" and "Return to previous video." In this case, if "Next Video" is selected, the next video will be displayed, and if "Previous Video" is selected, the previous video will be displayed.

[0101] <Example of video information for operation instructions displayed when a malfunction occurs> This section describes several video examples of instructional video information displayed when a malfunction occurs, specifically focusing on the video information displayed when a paper jam occurs. Similarly, when replacing consumables or other parts, the operating instructions consist of multiple video videos.

[0102] Figure 8 shows an explanatory diagram of one embodiment of operation instruction video information corresponding to a paper jam in an image forming apparatus. Figure 8 divides the state of the image forming apparatus (MFP) into four stages (M1 to M4). In Figure 8, the state of the image forming apparatus (MFP) consists of M1 when a paper jam occurs, M2 after the MFP door is opened, M3 after the paper is removed, and M4 after the MFP door is closed. These states correspond to the sequence of operations required to resolve a paper jam.

[0103] Figure 8 also shows an overview of the video content of the operation instructions, which are displayed in accordance with the status of the image forming apparatus (MFP). For example, when a paper jam occurs, M1 indicates that the paper jam removal procedure will be displayed, or a video explaining how to open the door will be shown.

[0104] After the MFP door is opened, M2 indicates that a video will be displayed showing the paper removal position inside the door and the operation for removing the paper. After paper removal, M3 indicates that a video will be displayed showing how to return the inside of the door to its pre-removal state and how to close the door (door closing operation). After the MFP door is closed, M4 indicates that a video will be displayed showing that paper removal is complete.

[0105] Figure 9 shows an explanatory diagram of one embodiment of the display screen for operation instructions corresponding to the status of the image forming apparatus. Figure 9 shows an example of a display screen for operation instructions corresponding to the four paper jam states (M1 to M4) shown in Figure 8. Figure 9 shows the M1 display screen when a paper jam occurs, indicating that the door should be opened to remove the paper, and a video demonstrating the door opening procedure is displayed. When this video ends, playback will stop, and if the "Next" area on the screen is selected, the next screen showing M2 after the MFP door opens will be displayed.

[0106] Figure 9 shows the display screen for M2 after the MFP door is opened, indicating that paper in the paper transport path will be removed, the paper removal location will be shown, and a video related to the paper removal operation will be displayed. When this video finishes, playback will stop, and if the "Next" area on the screen is selected, the next screen showing M3 after paper removal will be displayed. Alternatively, if the "Back" display area on the screen is selected, the display screen for the previous paper jam (M1) will be shown. Furthermore, the video of M2 after the MFP door opens may be automatically and repeatedly displayed until the "Next" display area is selected. Although not shown in the diagram, the "Play Same Video" display area may also be displayed to play the video of M2 after the MFP door opens again.

[0107] Figure 9 shows the display screen for M3 after paper removal, indicating that it displays videos related to closing the door, returning to the state before removal, and the door closing operation. When this video ends, playback will stop, and if the "Next" area on the screen is selected, the next screen showing M4 after the MFP door closes will be displayed. Alternatively, if the "Back" display area on the screen is selected, the display screen for the previous MFP door opening (M2) will be shown.

[0108] Figure 9 shows the display screen for M4 after the MFP door is closed, indicating that the paper removal process is complete and the unit can resume operation, and also displays a video confirming that the paper removal process is complete. When this video ends, playback of the video will stop, and if the "End" display area on the screen is selected, the display of the operation instructions will end, and the system will return to a state where the predetermined operation of the image forming apparatus can be resumed. Alternatively, if the "Back" button on the screen is selected, the previous screen showing M3 after paper removal will be displayed.

[0109] As described above, switching between operation instruction screens can be done by the operator's selection input, but to reduce the operator's workload, the operation instruction screen may be automatically switched by detecting the operator's status. For example, a sensor that detects the open / closed state (door open / close sensor) is attached to the front panel of the image forming apparatus to detect the operator's door operation. Additionally, a sensor (paper passage detection sensor) is installed in the paper transport path to detect whether or not the paper has been removed by the operator. Alternatively, a camera attached to the image forming apparatus may be used to photograph the operator's actions, and the operator's actions may be detected by image recognition.

[0110] (Example 1 of switching display screens) Regarding the four scenarios in which a paper jam occurs, as shown in Figures 8 and 9, if the door is not opened, the video of the paper jam (M1) will be displayed. However, if the operator opens the door, that is, if the door opening / closing sensor detects that the door has been opened, the display screen for the operation instructions will automatically switch to show the video of the MFP door after it has been opened (M2). If the operator opens the door, there is no need to display the M1 video regarding door opening in case of a paper jam. Therefore, it is preferable to proceed to the next MFP door opening display screen (M2) without waiting for the operator to input "Next".

[0111] (Example 2 of switching display screens) Furthermore, for example, if the paper passage detection sensor detects that jammed paper has been removed, and the door opening / closing sensor detects that the door has been closed, the display of the MFP door opening (M2) and paper removal (M3) videos will be omitted, and the MFP door closing (M4) video will be displayed instead. In this case, if the operator removes the paper and closes the door, there is no need to keep displaying the MFP door open (M2) and paper removal (M3) videos. Therefore, it is preferable to proceed to the MFP door closed (M4) display screen without waiting for the operator to input "Next". Alternatively, if the paper has been removed, the malfunction has been resolved, and the door has been closed, indicating that the image forming apparatus is ready to resume processing, the video playback of the operation instructions may be terminated without transitioning to the M4 display screen after the MFP door is closed.

[0112] Furthermore, door opening and closing operations can also be detected by image recognition of image data captured by a camera. In this way, by using sensors and image recognition attached to the image forming apparatus to detect the operator's operation status and automatically switching the display screen for operation instructions, the operator does not need to manually switch the display screen, thus reducing the operator's workload.

[0113] Alternatively, suppose that while the MFP door is open and the M2 video is displayed, the operator removes the paper and, in order to watch the next video, selects "Next" on the MFP door open M2 display screen. If the paper has been removed successfully, it is preferable to proceed to the M3 video after the paper has been removed. However, even if the operator believes they have removed the paper, if the paper passage detection sensor detects that paper is actually still present, the paper removal process must be repeated. Therefore, it is undesirable to proceed to the M3 video after paper removal related to door closure.

[0114] In other words, even if the operator selects "Next" on the M2 display screen after the MFP door is opened, if the paper passage detection sensor detects that paper is still present, the system will not proceed to the M3 video after the paper is removed, but will continue to display the M2 video after the MFP door is opened. This allows for the display of an appropriate screen informing the operator of the necessary actions to take in the event of a paper jam in the image forming machine, based on the operator's actions and work status, and the malfunction status of the image forming machine.

[0115] <Operation Instructions (Guidance): Example of Playback Adjustment Process> Figures 10 to 12 show flowcharts of one embodiment of the guidance playback adjustment process of an image forming apparatus. As described above, if a malfunction occurs in the image forming apparatus or if parts need to be replaced, the initial screen for preparing to play the operation instructions (G3) will be displayed, and if "Watch the instructional video" is selected, the operation instructions display screen (G4) will be displayed.

[0116] However, the system uses the detection operation response information 44 to adjust the playback operation of the operation instructions (guidance) in accordance with the current user detection status and image recognition results. For example, if it is determined that the operator is not looking at the display screen, the guidance operation will be temporarily suspended. Subsequently, if it is determined that the operator is now looking at the display screen while the system is paused, the guidance operation will resume from a position a certain time prior to the pause point.

[0117] In step S1 of Figure 10, the operation of the image forming apparatus MFP is checked. In other words, check whether any of the above-mentioned problems (malfunctions) such as paper jams are occurring. For example, a paper jam is detected by the paper passage detection sensor mentioned above. If a malfunction is detected in step S2, proceed to step S3; otherwise, proceed to step S4.

[0118] In step S4, check whether parts need to be replaced. For example, signals from the toner cartridge can be used to check for the presence of toner in the cartridge and determine whether the toner cartridge needs to be replaced. In step S5, if a part needs to be replaced, proceed to step S6; otherwise, return to step S1.

[0119] In step S3, a warning screen indicating a malfunction of the image forming apparatus (malfunction warning screen: G1) is displayed on the display unit 13, and the process proceeds to step S7. In step S6, a warning screen indicating a parts replacement request (parts replacement request warning screen: G2) is displayed on the display unit 13, and the process proceeds to step S7.

[0120] In step S7, prepare the display of the operating instructions. For example, the initial screen G3 for preparing to play the operation instructions, as shown in Figure 6, is displayed on the display unit 13. Here, we check whether "View" or "Do not view" is selected in the display area G3 of screen 6. If the user enters "View" in step S8, proceed to step S10; otherwise, proceed to step S9. If the user enters "Do not view" in step S9, proceed to step S12; otherwise, return to step S8.

[0121] In step S10, since "Watch operation instruction video" was selected, the user status information 47 is initially set to "Confirmation status" ST1. In step S11, playback of the operating instructions (guidance) begins. Here, among the operation instruction information 43 stored in the memory unit 40, the operation instruction (guidance) related to the malfunction or parts replacement that occurred is read out, and playback of that operation instruction is started. Then, proceed to step S21 in Figure 11.

[0122] In step S12, "Do not watch the instruction video" was selected, so the initial screen G3 for preparing to play the instruction video is cleared and the process is terminated. In this case, the worker performs troubleshooting or parts replacement without referring to the operating instructions. Alternatively, after step S12, the process may proceed to step S51 in Figure 12.

[0123] In step S21 of Figure 11, the human body detection unit 24 performs a human body detection check. If a human body is detected in step S22, proceed to step S23; otherwise, proceed to step S41. If a human body is detected, it means that a person, such as a user or operator of the image forming apparatus, is present in the vicinity of the image forming apparatus, and their image is taken.

[0124] In step S23, the captured image data 42 output from the camera is acquired. In step S24, image recognition processing is performed on the captured image data 42 to obtain recognized image information 46. In image recognition processing, as described above, the position of the head, the position of the eyes, and the direction of the gaze are obtained.

[0125] In step S25, the user status determination unit 27 uses the recognized image information 46 to determine the user's status. Here, it is determined whether the user of the image forming apparatus (a person near the image forming apparatus) is looking at the display screen (whether they are checking it or not). In step S26, if the user is currently reviewing the display screen, the process proceeds to step S27; otherwise, i.e., if the user is in a state of not having reviewed the explanation (ST2), the process proceeds to step S41.

[0126] In step S41, playback of the guidance is paused. At this time, the position information of the paused video (guide stop position information 48) is stored. In step S42, the user is notified by voice that the guidance playback has been paused. In step S43, the user status information 47 is set to the "Explanation not confirmed" state (ST0), and the process proceeds to step S51 in Figure 12.

[0127] In step S27, if the current user status information 47 is in the explanation confirmation state (ST1), proceed to step S28; otherwise, i.e., if the user status information 47 is in the explanation unconfirmed state (ST2), proceed to step S29. In step S28, since the user is currently viewing the display screen, the operation instructions (guidance) will continue to play, and the process will proceed to step S51 in Figure 12.

[0128] If the process proceeds to step S29, according to the user status information 47, the user was not looking at the display screen. Since the user has changed from not looking at the display screen to looking at the display screen, the process from step S29 onward will resume displaying the operation instructions (guidance). In step S29, the stored stop position information (guide stop position information 48) is read out.

[0129] In step S30, the playback position of the guidance is set to a position a certain amount of time before the read stop position. The set position is called the rewind position. For example, set it to a position about 10 seconds before the stopping point. In step S31, playback of the guidance is resumed from the set guidance playback position (rewind position). In step S32, the user status information 47 is set to explanation confirmation state (ST1), and the process proceeds to step S51 in Figure 12.

[0130] In step S51 of Figure 12, the operation of the image forming apparatus is checked, similar to step S1. In step S52, the need for part replacement is checked, similar to step S4. If it is detected in step S53 that the malfunction has been resolved, the process proceeds to step S54; otherwise, the process proceeds to step S55. If it is detected in step S54 that the parts replacement has been completed, the process proceeds to step S56; otherwise, the process proceeds to step S55.

[0131] If the user performs an input operation in step S55 that indicates the end of guidance, proceed to step S56; otherwise, return to step S21 in Figure 11. In step S56, the playback of the operation instructions (guidance) is terminated, the operation instructions display screen is cleared, and the process is terminated.

[0132] As described above, by adjusting the display state of the operation instructions (guidance) provided to the user depending on the user's status, and in particular by determining whether the user is looking at the display screen or not, and adjusting the display state of the operation instructions (guidance) to play, pause, or resume, it is possible to ensure that the user does not miss any part of the operation instructions (guidance) that they want to check. [Explanation of symbols]

[0133] 1. Image forming apparatus, 2 networks, 11 Control unit, 12 Operation section, 13 Display section, 14 Image Processing Unit, 15 Communications Department, 21 Malfunction detection unit, 22. Part replacement detection unit, 23 Photography Department, 24 Human body detection unit, 25 Image recognition unit, 26 Operation Instruction Display Control Unit, 27 User status determination unit, 28 Operation Instruction Execution Unit, 40 storage section, 41 Input image data, 42. Image data taken during filming. 43. Operation Instructions Information, 44 Detection operation response information, 45. User detection information, 46 Recognition image information, 47. User status information, 48 Guide Stop Position Information

Claims

1. In an information processing device equipped with a display unit, A human body detection unit detects a person in the vicinity of the information processing device and acquires user detection information indicating whether or not a person is present. A camera unit that photographs the surrounding space of the information processing device, An image recognition unit performs image recognition on captured image data captured by the aforementioned shooting unit and acquires predetermined recognized image information through image recognition. An operation instruction execution unit performs an operation to provide the user with predetermined operation instruction information by means including displaying it on the display unit, in order to assist the user in performing the operations that the user should perform. The system includes a user status determination unit that uses the user detection information and the recognized image information to determine whether or not the user is viewing the operation instruction information displayed on the display unit. An information processing device characterized in that, when the user status determination unit determines that the user is not viewing the displayed operation instruction information while the operation instruction execution unit is providing the operation instruction information, the operation instruction execution unit stops providing the operation instruction information, and when the user status determination unit determines that the user is viewing the displayed operation instruction information while the operation instruction execution unit has stopped providing the operation instruction information, the operation instruction execution unit resumes providing the operation instruction information.

2. The information processing device according to claim 1, characterized in that when the operation of providing the operation instruction information is resumed, the provision of the operation instruction information is resumed from a position a certain amount of time prior to the position where the operation of providing the operation instruction information was stopped.

3. The information processing device according to claim 1, characterized in that the recognized image information acquired by the image recognition unit through image recognition includes the position of the head, the position of the eyes, and the direction of the gaze.

4. The information processing device according to claim 3, characterized in that if the user detection information acquired by the human body detection unit indicates that a user is in the vicinity of the information processing device, and if it is determined that the user may be looking at the operation instruction information based on the position of the head, the position of the eyes, or the direction of the gaze in the recognized image information, the user state determination unit determines that the user is looking at the displayed operation instruction information.

5. The information processing device according to claim 3, characterized in that if the user detection information acquired by the human body detection unit indicates that the user is not in the vicinity of the information processing device, or if it is determined that the user may not be looking at the operation instruction information based on the position of the head, the position of the eyes, or the direction of the gaze in the recognized image information, the user state determination unit determines that the user is not looking at the displayed operation instruction information.

6. The information processing device according to claim 1, wherein the operation instruction information consists of a plurality of pieces of information that are set in advance in association with the state of the information processing device for each of a plurality of operations to be performed in succession, and when any of the plurality of operations is performed by the user, the operation instruction execution unit switches the operation instruction information displayed on the display unit to the operation instruction information set in association with the performed operation and the current state of the information processing device.

7. The information processing device according to any one of claims 1 to 6, characterized in that, in providing the user with operational instruction information, it includes automatically setting whether or not to output the operational instruction information as audio, and automatically adjusting the volume of the output audio, in accordance with the ambient noise.

8. The information processing apparatus according to any one of claims 1 to 6, characterized in that the operation instruction information includes a video that is displayed on the display unit when a malfunction occurs in the information processing apparatus or when it becomes necessary to replace a part of the information processing apparatus.

9. The control unit of the information processing device, A human body detection step that detects a person in the vicinity of an information processing device and obtains user detection information indicating whether or not a person is present, Image recognition step: Using the imaging unit, image recognition is performed on the image data captured of the surrounding space of the information processing device, and predetermined recognized image information is obtained through image recognition. An operation instruction execution step, which provides the user with operation instruction information by means including a display in order to assist the user in performing the operation to be performed, A user state determination step that uses the user detection information and the recognized image information to determine whether or not the user is viewing the displayed operation instruction information, If it is determined during the execution of the operation instruction execution step that the user is not viewing the displayed operation instruction information, the step of stopping the provision of the operation instruction information, A method for adjusting the guidance playback of an information processing device, characterized in that, when the provision of the aforementioned operation instruction information is stopped, the device determines that the user is viewing the displayed operation instruction information, it performs the step of resuming the provision of the aforementioned operation instruction information.

Images