Operation input control method, operation input device, and image processing device

Abstract

To identify and detect multiple types of operation inputs by detecting simple movements of the user's eyes. [Solution] A CPU (51) displays a menu screen including a plurality of operation icons, a cursor, and a plurality of type-specific icons on a display device (3a). The CPU (51) periodically switches the plurality of type-specific icons one by one to an active display state. When an eye movement detection device (8a) detects a blink, the CPU (51) generates an operation input event corresponding to the combination of the cursor position and the type of operation corresponding to one of the plurality of type-specific icons in the active display state.

Description

[Technical Field]

[0001] The present invention relates to an operation input control method, an operation input device, and an image processing device that can detect an operation input by detecting a line of sight and a blink. [Background technology]

[0002] 2. Description of the Related Art Image processing devices such as printers, copiers, facsimile machines, and multifunction peripherals are equipped with an operation input device that detects operation inputs related to image processing.

[0003] The operation input device may have a function of detecting a user's gaze and blinking to detect an operation input in a hands-free manner, and the operation input device moves a cursor in response to the detection result of the user's gaze movement, and generates an operation event corresponding to the position of the cursor in response to the detection result of the user's blinking.

[0004] For example, it is known that an image forming device detects multiple types of operation input by identifying and detecting multiple operating states of a user's eyelids (see, for example, Patent Document 1). The multiple operating states of the eyelids include a blink in which both eyelids are closed for a long time, a blink in which both eyelids are closed for a short time, a blink of only one eyelid, and a state in which the eyes are narrowed. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2022-143954 Summary of the Invention [Problem to be solved by the invention]

[0006] Incidentally, if the operation input device can distinguish and detect a plurality of types of operation inputs by detecting one type of blinking action, operability based on the movement of the user's eyes will be improved.

[0007] An object of the present invention is to provide an operation input control method, an operation input device, and an image processing device that can distinguish and detect multiple types of operation inputs by detecting simple eye movements of a user. [Means for solving the problem]

[0008] An operation input control method according to one aspect of the present invention is a method for controlling an operation input device. The operation input device includes a display device and an eye movement detection device that detects changes in the gaze direction of an operator looking at a screen of the display device and blinks of the operator. The operation input control method includes a processor causing the display device to display a menu screen including a plurality of operation icons each serving as an operation target, a cursor pointing to a position on the screen, and a plurality of type-specific icons corresponding to a plurality of types of operations, including an icon selection operation for selecting one of the plurality of operation icons. The operation input control method further includes, while the menu screen is displayed, the processor changing the display position of the cursor in accordance with the detection result of the change in the gaze direction by the eye movement detection device. The operation input control method further includes, while the menu screen is displayed, the processor periodically switching the plurality of type-specific icons one by one to an active display state. The operation input control method further includes, when the blink is detected by the eye movement detection device while the menu screen is being displayed, the processor generating an operation input event corresponding to a combination of the cursor position and a type of operation corresponding to one of the plurality of type-specific icons in the active display state.

[0009] According to another aspect of the present invention, there is provided an operation input device including the display device, the eye movement detection device, and the processor, wherein the processor detects an operation input by implementing the operation input control method and generates an event according to the detection result of the operation input.

[0010] According to another aspect of the present invention, an image processing device includes the operation input device and one or more image processing units that perform the image processing in response to the event. [Effects of the Invention]

[0011] According to the present invention, it is possible to provide an operation input control method, an operation input device, and an image processing device that can distinguish and detect multiple types of operation inputs by detecting simple eye movements of the user. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram showing the configuration of an image processing apparatus including an operation input device according to an embodiment. [Figure 2] FIG. 2 is a flowchart showing an example of a procedure for non-contact operation control in the operation input device according to the embodiment. [Figure 3] FIG. 3 is a flowchart illustrating an example of a procedure for blink handling processing in the operation input device according to the embodiment. [Figure 4] FIG. 4 is a flowchart illustrating an example of a procedure of a selection process in the operation input device according to the embodiment. [Figure 5] FIG. 5 is a flowchart showing an example of a procedure of a scroll process in the operation input device according to the embodiment. [Figure 6] FIG. 6 is a flowchart illustrating an example of a procedure of a cancellation process in the operation input device according to the embodiment. [Figure 7] FIG. 7 is a diagram showing an example of a menu screen in the operation input device according to the embodiment. [Figure 8] FIG. 8 is a diagram showing an example of switching of the display state of the type-specific icon group in the operation input device according to the embodiment. [Figure 9] FIG. 9 is a diagram showing an example of a menu screen in the scroll mode in the operation input device according to the embodiment. [Figure 10]FIG. 10 is a diagram showing an example of a scrollable menu screen in the operation input device according to the first modified example. [Figure 11] FIG. 11 is a diagram showing an example of switching of the display state of the type-specific icon group in the operation input device according to the second modified example. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that the following embodiment is an example of the present invention, and does not limit the technical scope of the present invention.

[0014] [Configuration of image processing device 10] The operation input device 8 according to the embodiment constitutes a part of the image processing device 10.

[0015] The operation input device 8 has a function of detecting operation input without contact by detecting the user's gaze and blinking. The image processing device 10 can perform various image processes such as image formation process, image reading process, and image transmission process. For example, the image processing device 10 is a copier, facsimile machine, or multifunction peripheral.

[0016] In the example shown in FIG. 1, an image processing apparatus 10 includes an image reading device 1, an image forming device 2, a user interface device 3, a communication device 4, a control device 5, and a secondary storage device 6.

[0017] The image reading device 1 executes the image reading process. The image reading process is a process of reading an image from an original 91 and outputting data of the read image. The read image is an image read from the original 91.

[0018] The image forming apparatus 2 performs the image forming process by a predetermined method such as an electrophotographic method or an inkjet method. The image forming process is a process of forming an image on a sheet 92.

[0019] The image forming device 2 includes a sheet transport mechanism that transports a sheet 92 and a printing device that forms an image on the sheet 92 .

[0020] The user interface device 3 includes a display device 3a and a touch panel 3b. The display device 3a has a screen capable of displaying information. For example, the display device 3a includes a display panel such as a liquid crystal display panel.

[0021] The touch panel 3b is formed on the screen of the display device 3a and detects a touch operation by a person. For example, the touch panel 3b is a capacitive or resistive panel.

[0022] The communication device 4 is a communication interface device that communicates with other devices such as a host device via a network. The control device 5 performs all data transmission and reception between the control device 5 and the other devices through the communication device 4.

[0023] The image forming device 2 executes the image forming process based on the scanned image data or the received print data, which is data included in a print request received from the host device via the communication device 4.

[0024] The communication device 4 is also capable of executing the image transmission process, which is a process of transmitting the data of the scanned image to a specified destination via a network.

[0025] The control device 5 executes various calculations, data processing, and control of various electrical devices included in the image processing device 10. The control device 5 includes a CPU 51, a RAM (Random Access Memory) 52, and the like.

[0026] The secondary storage device 6 is a computer-readable non-volatile storage device. The secondary storage device 6 can store computer programs and various data. For example, one or both of an SSD (Solid State Drive) and a hard disk drive may be used as the secondary storage device 6.

[0027] The secondary storage device 6 stores the computer programs executed by the CPU 51 and data referenced by the CPU 51. The CPU 51 is an example of a processor.

[0028] The CPU 51 is a processor that executes the computer programs stored in the secondary storage device 6 to perform various data processing and control operations.

[0029] It is also conceivable that another processor such as a DSP may perform the data processing and control instead of the CPU 51.

[0030] The RAM 52 is a computer-readable volatile storage device that temporarily stores the computer programs executed by the CPU 51 and data output and referenced during the execution of the computer programs by the CPU 51.

[0031] The CPU 51 includes a plurality of processing modules that are realized by executing the computer programs, including a UI control unit 5a and a job control unit 5b.

[0032] The UI control unit 5a controls the user interface device 3. The UI control unit 5a controls the display screen of the display device 3a and generates an operation event in response to a touch operation detected by the touch panel 3b.

[0033] For example, the UI control unit 5a detects an operation input related to the image processing and generates an operation event according to the detection result of the operation input.

[0034] The job control unit 5b controls the image forming device 2, the image reading device 1, and the communication device 4. The job control unit 5b causes some or all of the image forming device 2, the image reading device 1, and the communication device 4 to execute a job corresponding to a request input through the touch panel 3b or the communication device 4.

[0035] The job includes some or all of the multiple types of image processing, and in this embodiment, the job includes an image scanning job, a print job, and a copy job.

[0036] The image reading job includes the image reading process executed by the operation of the image reading device 1. The print job includes the image forming process executed by the operation of the image forming device 2. The copy job includes the image reading process and the image forming process executed by the operations of the image reading device 1 and the image forming device 2. In other words, the job includes one or more types of image processing.

[0037] For example, when the operation event relating to the image processing occurs, the job control unit 5b executes the image processing corresponding to the operation event.

[0038] In this embodiment, the user interface device 3 further includes a camera 3c and an image processing circuit 3d. The camera 3c captures an image of the face of an operator standing in front of the image processing device .

[0039] The image processing circuit 3d detects changes in the operator's line of sight and blinks by performing image processing on the image obtained by the camera 3c. For example, part or all of the image processing circuit 3d is realized by one or both of an ASIC (Application Specific Integrated Circuit) and a DSP (Digital Signal Processor).

[0040] For example, the image processing circuit 3d detects the positions of the inner corner of the eye and the iris of the operator's eye 9 by image processing of the image captured by the camera 3c. Furthermore, the image processing circuit 3d calculates the direction and amount of change in the position of the iris relative to the position of the inner corner of the eye as the direction and amount of change in the gaze direction.

[0041] Furthermore, the image processing circuit 3d detects a state in which the iris is temporarily hidden by the eyelid in the intraocular region based on the position of the inner corner of the eye as a closed eye state. The image processing circuit 3d detects the iris again after the duration of the closed eye state exceeds a predetermined blink determination time, and then detects the blink of the operator.

[0042] The blink determination time is sufficiently longer than the time it takes for a person to blink as a physiological phenomenon, so that the image processing circuit 3d can detect blinks performed as an operation on the user interface device 3 and distinguish them from blinks performed as a physiological phenomenon.

[0043] The camera 3c and the image processing circuit 3d are an example of an eye movement detection device 8a that detects the change in the gaze direction and the blinking. Note that the CPU 51 may execute part or all of the processing executed by the image processing circuit 3d.

[0044] The UI control unit 5a executes a process of detecting an operation input to the user interface device 3 based on the change in the line of sight direction and the blink detected by the image processing circuit 3d.

[0045] In this embodiment, the user interface device 3 and the UI control unit 5a of the CPU 51 constitute an operation input device 8. The CPU 51 is an example of a processor that controls the operation input device 8. The operation input device 8 detects the operator's line of sight and blinks to detect a non-contact operation input.

[0046] Incidentally, if the operation input device 8 can distinguish and detect a plurality of types of operation inputs by detecting one type of blinking action, the operability based on the movement of the operator's eyes will be improved.

[0047] The operation input device 8 executes non-contact operation control, which will be described later (see FIG. 2). The non-contact operation control includes processing for identifying and detecting multiple types of operation inputs by detecting simple eye movements of the operator. The non-contact operation control is executed in parallel with the detection of operation inputs by the touch panel 3b.

[0048] [Non-contact operation control] An example of the procedure for the non-contact operation control will be described below with reference to the flowchart shown in Fig. 2. The non-contact operation control is an example of a process that realizes an operation input control method for controlling the operation input device 8.

[0049] The UI control unit 5a starts the non-contact operation control when the operation input device 8 is started. In the following description, S101, S102, ... represent identification codes of multiple steps in the non-contact operation control. In the non-contact operation control, the processing of step S101 is executed first.

[0050] <Process S101> In step S101, the UI control unit 5a displays the menu screen G10 on the display device 3a (see FIG. 7).

[0051] The menu screen G10 includes a plurality of operation icons G1, each of which is an object of operation, a cursor G2, and a type-specific icon group G3 (see FIG. 7). The type-specific icon group G3 includes a plurality of type-specific icons G3a to G3d corresponding to a plurality of types of operations.

[0052] The cursor G2 is a movable icon that points to a position on the screen of the display device 3a. In the example shown in Fig. 7, the cursor G2 is an arrow icon. The cursor G2 may also be a circle icon or a cross icon, etc.

[0053] The plurality of types of operations includes an icon selection operation, which is an operation of selecting one of a plurality of operation icons G1. The type-specific icon group G3 includes a selection operation icon G3a corresponding to the icon selection operation.

[0054] In the example shown in FIG. 7, the type-specific icon group G3 includes a selection operation icon G3a, a scroll mode icon G3b, a cancel icon G3c, and an ignore icon G3d.

[0055] The scroll mode icon G3b is an icon corresponding to an operation for transitioning from the normal mode to the screen scroll mode. The display mode when the menu screen G10 is displayed is the normal mode. The processing of steps S102 to S106, which will be described later, is executed under the condition that the display mode is the normal mode.

[0056] The normal mode is a display mode in which the cursor G2 is moved in accordance with the detection result of the change in the line of sight, and the screen scroll mode is a display mode in which the menu screen G10 is scrolled in accordance with the detection result of the change in the line of sight.

[0057] The cancel icon G3c is an icon corresponding to an operation of canceling the most recently executed process, in other words, an operation of redoing the most recently executed process.

[0058] The ignore icon G3d is an icon that indicates that the image processing circuit 3d ignores the detection of the blink.

[0059] In this embodiment, the UI control unit 5a displays the type-specific icon group G3 at equal intervals in the outer edge region of the screen of the display device 3a. In the example shown in Fig. 7, the UI control unit 5a displays the multiple type-specific icons G3a to G3d at equal intervals along a circle.

[0060] It should be noted that the plurality of type-specific icons G3a to G3d may be displayed at equal intervals along an ellipse, a rectangle, or a regular polygon.

[0061] After executing the process of step S101, the UI control unit 5a executes the process of step S102.

[0062] The processing of steps S102 to S107 shown below is executed under the condition that the menu screen G10 is displayed on the display device 3a.

[0063] <Process S102> In step S102, the UI control unit 5a acquires line of sight change data, which is the detection result of the change in line of sight direction, from the image processing circuit 3d, and determines whether the line of sight change data indicates a line of sight movement state or a line of sight stationary state.

[0064] The gaze change data represents a change direction and an amount of change in the gaze direction of the operator detected by the image processing circuit 3d. The UI control unit 5a determines that the gaze change data represents the gaze moving state when the amount of change represented by the gaze change data exceeds a predetermined lower limit. On the other hand, the UI control unit 5a determines that the gaze change data represents the gaze stopped state when the amount of change represented by the gaze change data falls below the lower limit.

[0065] The UI control unit 5a executes the process of step S103 when it determines that the line-of-sight change data indicates the line-of-sight moving state, whereas the UI control unit 5a executes the process of step S105 when it determines that the line-of-sight change data indicates the line-of-sight stopping state.

[0066] The image processing circuit 3d may execute the process of step S102. In this case, the UI control unit 5a acquires, from the image processing circuit 3d, gaze state data indicating either the gaze movement state or the gaze stop state.

[0067] <Process S103> In step S103, the UI control unit 5a stops the active switching process, which controls the display state of the type-specific icon group G3. The initial state of the UI control unit 5a is a state in which the active switching process has been stopped. The active switching process will be described later (see step S105).

[0068] After executing the process of step S103, the UI control unit 5a executes the process of step S104.

[0069] <Process S104> In step S104, the UI control unit 5a moves the cursor G2 in accordance with the direction and amount of change in the line of sight indicated by the line of sight change data (see FIG. 7). That is, the UI control unit 5a changes the display position of the cursor G2 in accordance with the line of sight change data.

[0070] After executing the process of step S104, the UI control unit 5a repeats the processes from step S102 onwards.

[0071] <Process S105> On the other hand, in step S105, the UI control unit 5a starts the active switching process.

[0072] The active switching process is a process of periodically switching the type-specific icons G3a to G3d one by one to an active display state (see FIG. 8). For example, the active display state may be a state in which the icons are displayed in a specific color, a state in which they blink, or a state in which they are surrounded by a specific frame.

[0073] The UI control unit 5a maintains the type-specific icon group G3 in the all-negative state ST0 while the active switching process is stopped (see FIG. 8). In the all-negative state ST0, all of the type-specific icons G3a to G3d are in a negative display state.

[0074] During the active switching process, the UI control unit 5a sequentially switches the display state of the type-specific icon group G3 between four states: a first display state ST1, a second display state ST2, a third display state ST3, and a fourth display state ST4 (see FIG. 8).

[0075] In the first display state ST1, only the selection operation icon G3a is in an active display state. In the second display state ST2, only the scroll mode icon G3b is in an active display state. In the third display state ST3, only the cancel icon G3c is in an active display state. In the fourth display state ST4, only the ignore icon G3d is in an active display state.

[0076] For example, the UI control unit 5a sequentially switches the display state of the type-specific icon group G3 to each of the four states every 1 to 1.5 seconds. In the example shown in Fig. 8, the initial state of the type-specific icon group G3 when the active switching process is started is the first display state ST1.

[0077] The UI control unit 5a continues the active switching process until the active switching process is stopped in step S103, etc. Stopping the active switching process means stopping the periodic switching of the display state of the type-specific icon group G3.

[0078] After executing the process of step S105, the UI control unit 5a repeats the processes from step S106 onwards.

[0079] <Process S106> Meanwhile, in step S106, the UI control unit 5a acquires blink data indicating whether a blink has been detected from the image processing circuit 3d, and selects the next process depending on whether the blink data indicates detection or non-detection.

[0080] If the blink data indicates no detection, the UI control unit 5a repeats the processes from step S102 onwards. On the other hand, if the blink data indicates presence of detection, the UI control unit 5a executes the process of step S107.

[0081] <Process S107> In step S107, the UI control unit 5a executes blink handling processing, which will be described later (see FIG. 3).

[0082] After executing the process of step S107, the UI control unit 5a repeats the processes from step S102 onwards.

[0083] [Blink response processing] Next, an example of the procedure of the blink response processing will be described with reference to the flowchart shown in Fig. 3. The UI control unit 5a executes the blink response processing when the image processing circuit 3d detects the blink of the operator (see steps S106 and S107).

[0084] In the following description, S201, S202, . . . represent identification codes of a plurality of steps in the blink response processing.

[0085] The processes of steps S201 to S204 in the blink response process are executed under the condition that the menu screen G10 is displayed. The processes of steps S201, S202, and S204 in the blink response process are executed under the condition that the display mode is the normal mode. In the blink response process, the process of step S201 is executed first.

[0086] <Process S201> In step S201, the UI control unit 5a selects the next process depending on which of the multiple type-specific icons G3a to G3d is the active icon.

[0087] The active icon is one of the type-specific icons G3a to G3d that is in the active display state when the blink is detected.

[0088] The UI control unit 5a executes the process of step S202 when the active icon is the selection operation icon G3a, and executes the process of step S203 when the active icon is the scroll mode icon G3b.

[0089] Furthermore, the UI control unit 5a executes the process of step S204 when the active icon is the cancel icon G3c, and ends the blink response process without performing any process when the active icon is the ignore icon G3d.

[0090] <Process S202> In step S202, the UI control unit 5a executes a selection process, which will be described later (see FIG. 4). After executing the process of step S202, the UI control unit 5a ends the blink response process.

[0091] <Process S203> In step S203, the UI control unit 5a executes a scroll process, which will be described later (see FIG. 5). After executing the process of step S203, the UI control unit 5a ends the blink response process.

[0092] <Process S204> In step S204, the UI control unit 5a executes a cancellation process, which will be described later (see FIG. 6). After executing the process of step S204, the UI control unit 5a ends the blink response process.

[0093] [Selection Processing] Next, an example of the procedure of the selection process will be described with reference to the flowchart shown in Fig. 4. The UI control unit 5a executes the selection process when the active icon when the blink is detected is the selection operation icon G3a (see steps S201 and S202).

[0094] In the following description, S301, S302, ... represent identification codes of a plurality of steps in the selection process. In the selection process, the process of step S301 is executed first.

[0095] <Process S301> In step S301, the UI control unit 5a determines whether the position of the cursor G2 when the blink is detected is in the icon area or the background area, and selects the next process depending on the determination result.

[0096] The icon area is an area corresponding to an area in the entire area of ​​the menu screen G10 where each of the multiple operation icons G1 is displayed. The icon area includes at least an area overlapping each of the multiple operation icons G1. The icon area may also include an area near each of the multiple operation icons G1. The background area is an area other than the icon area in the entire area of ​​the menu screen G10.

[0097] The UI control unit 5a ends the selection process when it determines that the position of the cursor G2 is in the background area. On the other hand, the UI control unit 5a executes the process of step S302 when it determines that the position of the cursor G2 is in the icon area.

[0098] <Process S302> In step S302, the UI control unit 5a generates an operation input event corresponding to a designated icon, which is one of the plurality of operation icons G1. The designated icon is one of the plurality of operation icons G1 that corresponds to the position of the cursor G2.

[0099] If the operation input event corresponds to a process related to the image processing, the job control unit 5b executes the process corresponding to the operation input event. For example, the job control unit 5b causes one or more of the image reading device 1, the image forming device 2, and the communication device 4 to execute the image processing corresponding to the operation input event.

[0100] The image reading device 1, the image forming device 2, and the communication device 4 are each an example of an image processing unit that executes the image processing in response to the operation input event.

[0101] After executing the process of step S302, the UI control unit 5a executes the process of step S303.

[0102] <Process S303> In step S303, the UI control unit 5a causes the display device 3a to display a new menu screen G10 corresponding to the operation input event that occurred in step S302.

[0103] Furthermore, the UI control unit 5a stops the active switching control in step S303.

[0104] After executing the process of step S303, the UI control unit 5a ends the selection process.

[0105] [Scrolling] Next, an example of the procedure of the scroll process will be described with reference to the flowchart shown in Fig. 5. The UI control unit 5a executes the scroll process when the active icon when the blink is detected is the scroll mode icon G3b (see steps S201 and S203 in Fig. 3).

[0106] In the following description, S401, S402, ... represent identification codes of a plurality of steps in the scroll processing. In the scroll processing, the processing of step S401 is executed first.

[0107] <Process S401> In step S401, the UI control unit 5a selects the next process depending on whether the currently displayed menu screen G10 is a scrollable screen or not. Fig. 9 shows an example of a scrollable menu screen G10.

[0108] If the currently displayed menu screen G10 is a scrollable screen, the UI control unit 5a executes the process of step S402. On the other hand, if the currently displayed menu screen G10 is a non-scrollable screen, the UI control unit 5a ends the scrolling process.

[0109] <Process S402> In step S402, the UI control unit 5a transitions the display mode from the normal mode to the scroll mode. In the scroll mode, the UI control unit 5a ignores operations on each of the operation icons G1. In other words, in the scroll mode, the UI control unit 5a does not generate the operation input event corresponding to each of the operation icons G1.

[0110] For example, in the scroll mode, the UI control unit 5a causes the display device 3a to display a screen in which a scroll direction icon G4 is added to the menu screen G10 (see FIG. 9). The scroll direction icon G4 is an icon that indicates the scroll direction of the menu screen G10.

[0111] After executing the process of step S402, the UI control unit 5a executes the process of step S403. The processes of steps S403 to S405 shown below are executed in the scroll mode.

[0112] <Process S403> In step S403, the UI control unit 5a acquires the gaze change data from the image processing circuit 3d and determines whether the gaze change data represents the gaze moving state or the gaze stopping state. The process in step S403 is the same as the process in step S102 (see FIG. 2).

[0113] The UI control unit 5a executes the process of step S404 when it determines that the line-of-sight change data indicates the line-of-sight moving state, and executes the process of step S405 when it determines that the line-of-sight change data indicates the line-of-sight stopping state.

[0114] On the other hand, when the UI control unit 5a determines that the line-of-sight change data indicates the line-of-sight stopped state, the UI control unit 5a displays the scroll direction icon G4 in a state indicating that scrolling has stopped.

[0115] The image processing circuit 3d may execute the processing of step S403.

[0116] <Process S404> In step S404, the UI control unit 5a scrolls the menu screen G10 in accordance with the direction and amount of change in the line of sight direction represented by the line of sight change data (see FIG. 9).

[0117] Furthermore, the UI control unit 5a displays the scroll direction icon G4 in a state indicating the scroll direction.

[0118] The relationship between the direction of change of the line of sight and the direction of scrolling of the menu screen G10 is set in advance.

[0119] After executing the process of step S404, the UI control unit 5a executes the process of step S405.

[0120] <Process S405> In step S405, the UI control unit 5a acquires the blink data from the image processing circuit 3d, and selects the next process depending on whether the blink data indicates detection or non-detection.

[0121] If the blink data indicates no detection, the UI control unit 5a repeats the processes from step S403 onwards. On the other hand, if the blink data indicates presence of detection, the UI control unit 5a executes the process of step S406.

[0122] <Process S406> In step S406, the UI control unit 5a transitions the display mode from the scroll mode to the normal mode. For example, the UI control unit 5a erases the scroll direction icon G4 from the menu screen G10 when transitioning from the scroll mode to the normal mode.

[0123] After executing the process of step S406, the UI control unit 5a ends the scrolling process.

[0124] As described above, when the menu screen G10 is displayed and the display mode is the screen scroll mode, the UI control unit 5a executes the processes of steps S403 to S405. In steps S403 to S404, the UI control unit 5a scrolls the menu screen G10 in accordance with the detection result of the change in the line of sight by the image processing circuit 3d.

[0125] [Cancellation process] Next, an example of the procedure of the cancellation process will be described with reference to the flowchart shown in Fig. 6. The UI control unit 5a executes the cancellation process when the active icon when the blink is detected is the cancellation icon G3c (see steps S201 and S204 in Fig. 3).

[0126] In the following description, S501, S502, ... represent identification codes of a plurality of steps in the cancellation process. In the cancellation process, the process of step S501 is executed first.

[0127] <Process S501> In step S501, the UI control unit 5a generates a cancel event representing the cancellation of the most recent process as the operation input event.

[0128] For example, if the most recently executed process is switching the display of the menu screen G10, the UI control unit 5a causes the display device 3a to display the original menu screen G10 before switching in response to the occurrence of the cancellation event.

[0129] Furthermore, if the most recently executed process was executed by the job control unit 5b, the job control unit 5b executes a process for canceling the most recently executed process in response to the occurrence of the cancellation event.

[0130] For example, if the most recently executed process is a process for changing parameters related to the image processing, the job control unit 5b returns the most recently changed parameters to their original states.

[0131] After executing the process of step S501, the UI control unit 5a executes the process of step S502.

[0132] <Process S502> In step S502, the UI control unit 5a selects the next process depending on whether the display of the menu screen G10 has been changed or not in step S501.

[0133] If the display of the menu screen G10 has been changed in step S501, the UI control unit 5a executes the process of step S503. On the other hand, if the display of the menu screen G10 has not been changed in step S501, the UI control unit 5a executes the process of step S504.

[0134] <Process S503> In step S503, the UI control unit 5a stops the active switching control, and after executing the process of step S503, the UI control unit 5a ends the cancellation process.

[0135] <Process S504> On the other hand, in step S504, the UI control unit 5a sets the ignore icon G3d to the active display state, ie, the UI control unit 5a displays the type-specific icon group G3 in the fourth display state ST4.

[0136] After the process of step S504 is executed, the UI control unit 5a sets the active display state of the ignore icon G3d as the initial state, and periodically switches the type-specific icons G3a to G3d to the active display state.

[0137] That is, when the cancel event occurs, the UI control unit 5a retries the active switching control with the active display state of the ignore icon G3d as the initial state (see steps S501 and S504). This allows the operator to get used to the display state of the display device 3a changed by the cancel process during the period until the ignore icon G3d switches from the active display state to the negative display state.

[0138] As described above, when the menu screen G10 is displayed and the image processing circuit 3d detects a blink, the UI control unit 5a executes the blink response process (see steps S106 to S107 and steps S201 to S204).

[0139] In the blink response processing, the UI control unit 5a generates the operation input event corresponding to the combination of the position of the cursor G2 and the type of operation corresponding to one of the type-specific icons G3a to G3d in the active display state (see steps S301 and S302).

[0140] The operation input device 8 can distinguish and detect multiple types of operation inputs by detecting a simple eye movement consisting of a change in the line of sight and one type of blink, thereby improving operability based on the operator's eye movement.

[0141] [First Modification] Next, a first modified example of the operation input device 8 will be described with reference to Fig. 10. In Fig. 10, components common to the screen shown in Fig. 9 are denoted by the same reference numerals.

[0142] In this modification, the UI control unit 5a may cause the display device 3a to display a scrollable menu screen G10 in step S101 or step S303 of the non-touch operation control. In this case, the UI control unit 5a causes the display device 3a to display the menu screen G10 including the scroll icon G1a.

[0143] The scroll icon G1a is one of the plurality of operation icons G1. The scroll icon G1a is an icon corresponding to the transition process from the normal mode to the screen scroll mode.

[0144] In a situation where the menu screen G10 including the scroll icon G1a is displayed and the display mode is the normal mode, the UI control unit 5a executes a first scroll mode monitoring process and a second scroll monitoring process.

[0145] The first scroll monitoring process is a process of monitoring whether or not the cursor G2 is displayed in the scroll icon area, and when the cursor G2 is displayed in the scroll icon area, setting the selection operation icon G3a or the scroll mode icon G3b to the active display state.

[0146] The scroll icon region is a region corresponding to the scroll icon G1a, includes at least a region overlapping the scroll icon G1a, or may include a region near the scroll icon G1a.

[0147] The second scroll mode monitoring process is a process of monitoring whether or not the blink is detected by the image processing circuit 3d, and if the blink is detected and the cursor G2 is displayed in the scroll icon area, transitioning the display mode from the normal mode to the screen scroll mode.

[0148] For example, if the selection operation icon G3a is in the active display state when the blink is detected, the UI control unit 5a transitions the display mode from the normal mode to the screen scroll mode as processing of the operation input event that occurs in step S302 (see Figure 4).

[0149] On the other hand, if the scroll mode icon G3b is in the active display state when the blink is detected, the UI control unit 5a transitions the display mode from the normal mode to the screen scroll mode as processing in step S402 (see FIG. 5).

[0150] By adopting this modified example, when the cursor G2 moves to the scroll icon area in accordance with the detection result of the line of sight direction, the operator can quickly switch the display mode to the screen scroll mode by blinking, i.e., the operator does not need to wait until the selection operation icon G3a or the scroll mode icon G3b is switched to the active display state by the active switching control.

[0151] In the first scroll monitoring process, if the selection operation icon G3a is set to the active display state when the cursor G2 is displayed in the scroll icon area, the display of the scroll mode icon G3b may be omitted.

[0152] [Second Modification] Next, a second modified example of the operation input device 8 will be described with reference to Fig. 11. In Fig. 11, components common to the type-specific icon group G3 shown in Fig. 8 are given the same reference numerals.

[0153] The UI control unit 5a in this modified example causes the display device 3a to display a menu screen G10 including a type-specific icon group G3 consisting of three type-specific icons G3a to G3c.

[0154] The three type-specific icons G3a to G3c include a selection operation icon G3a, a scroll mode icon G3b, and a cancel icon G3c.

[0155] During the active switching process, the UI control unit 5a sequentially switches the display state of the type-specific icon group G3 between four states: a first display state ST1, a second display state ST2, a third display state ST3, and a fourth display state ST4 (see FIG. 11).

[0156] In the fourth display state ST4 of this modified example, all of the three type-specific icons G3a to G3c are in the negative display state.

[0157] When the type-specific icon group G3 is in the fourth display state ST4 shown in FIG. 11, the UI control unit 5a in this modified example executes the same processing as when the fourth display state ST4 shown in FIG. 8 is adopted.

[0158] When this modified example is adopted, the same effects as when the operation input device 8 is adopted can be obtained.

[0159] [Notes on the Invention] The following is a summary of the invention extracted from the above-described embodiment. Note that the configurations and processing functions described in the following supplementary notes can be selected and combined as desired.

[0160] <Appendix 1> An operation input control method for controlling an operation input device including a display device and an eye movement detection device that detects a change in the line of sight direction of an operator looking at a screen of the display device and a blink of the operator, comprising: a processor causing the display device to display a menu screen including a plurality of operation icons each of which is an object of operation, a cursor indicating a position on the screen, and a plurality of type-specific icons corresponding to a plurality of types of operations including an icon selection operation for selecting one of the plurality of operation icons; While the menu screen is being displayed, the processor changes the display position of the cursor in accordance with the detection result of the change in the line of sight direction by the eye movement detection device; While the menu screen is being displayed, the processor periodically switches the plurality of type-specific icons one by one to an active display state; and when the blink is detected by the eye movement detection device while the menu screen is being displayed, the processor generates an operation input event corresponding to a combination of the cursor position and a type of operation corresponding to one of the plurality of type-specific icons in the active display state.

[0161] <Appendix 2> The operation input control method described in Appendix 1 includes, when the menu screen is displayed, the processor stops switching the display state of the plurality of type-specific icons when the amount of change in the gaze direction detected by the eye movement detection device exceeds a predetermined lower limit value.

[0162] <Appendix 3> the plurality of type-specific icons include a selection operation icon corresponding to the icon selection operation and a scroll mode icon corresponding to an operation of transitioning from a normal mode to a screen scroll mode, When the menu screen is displayed and the display mode is the normal mode, the processor changes the display position of the cursor in accordance with a detection result of the change in the line of sight direction by the eye movement detection device; In a situation where the menu screen is being displayed and the display mode is the normal mode, the processor transitions the display mode to the screen scroll mode when the scroll mode icon is in the active display state when the blink is detected by the eye movement detection device; The operation input control method described in Appendix 1 or Appendix 2 includes, when the menu screen is displayed and the display mode is the screen scroll mode, scrolling the menu screen in accordance with the detection result of the change in the gaze direction by the eye movement detection device.

[0163] <Appendix 4> the plurality of operation icons includes a scroll icon corresponding to a transition process from the normal mode to the screen scroll mode, In a situation where the menu screen is being displayed and the display mode is the normal mode, the processor sets the selection operation icon or the scroll mode icon to the active display state when the cursor is displayed in an area corresponding to the scroll icon; The operation input control method described in Appendix 3 includes, under a situation where the menu screen is displayed and the display mode is the normal mode, transitioning the display mode to the screen scroll mode when the cursor is displayed in an area corresponding to the scroll icon when the blink is detected by the eye movement detection device.

[0164] <Appendix 5> the plurality of operation icons includes a scroll icon corresponding to a transition process from a normal mode to a screen scroll mode, When the menu screen is displayed and the display mode is the normal mode, the processor changes the display position of the cursor in accordance with a detection result of the change in the line of sight direction by the eye movement detection device; under a condition that the menu screen is being displayed and the display mode is the normal mode, the processor sets a selection operation icon corresponding to the icon selection operation to the active display state when the cursor is displayed in an area corresponding to the scroll icon; In a situation where the menu screen is being displayed and the display mode is the normal mode, the processor transitions the display mode to the screen scroll mode when the cursor is displayed in an area corresponding to the scroll icon when the blink is detected by the eye movement detection device; The operation input control method described in Appendix 1 or Appendix 2 includes, when the menu screen is displayed and the display mode is the screen scroll mode, scrolling the menu screen in accordance with the detection result of the change in the gaze direction by the eye movement detection device.

[0165] <Appendix 6> the plurality of type-specific icons include a selection operation icon corresponding to the icon selection operation and a cancel icon corresponding to an operation for canceling a process, The operation input control method according to any one of Supplementary Note 1 to Supplementary Note 5, further comprising: when the menu screen is displayed and the blink is detected by the eye movement detection device and the cancel icon is in the active display state, the processor generates a cancel event representing the cancellation of the most recent process as the operation input event.

[0166] <Appendix 7> the plurality of type-specific icons include an ignore icon representing ignoring the detection of the blink by the eye movement detection device, 7. The operational input control method of claim 6, further comprising: when the cancellation event occurs, the processor sets the ignore icon to the active display state and then periodically switches the plurality of type-specific icons to the active display state.

[0167] <Appendix 8> The operation input control method according to any one of Supplementary Note 1 to Supplementary Note 7, wherein the processor displays the plurality of type-specific icons at equal intervals in an outer edge region of the screen of the display device.

[0168] <Appendix 9> a display device; an eye movement detection device that detects a change in the line of sight of an operator looking at a screen of the display device and a blink of the operator; a processor that detects an operation input by implementing the operation input control method according to any one of Supplementary Note 1 to Supplementary Note 8, and generates an event according to the detection result of the operation input.

[0169] <Appendix 10> the operation input device according to Supplementary Note 9, which detects an operation input related to image processing and generates an event according to a detection result of the operation input; and one or more image processing units that perform the image processing in response to the event. [Explanation of symbols]

[0170] 3: User interface device 3a:Display device 3b: Touch panel 3c: Camera 3d: Image processing circuit 5: Control device 8: Operation input device 8a: Eye movement detection device 10: Image processing device G1: Operation icon G10: Menu screen G1a: Scroll icon G2: Cursor G3: Type-specific icons G3a: Selection operation icon (type specific icon) G3b: Scroll mode icon (type specific icon) G3c: Cancel icon (type-specific icon) G3d: Ignore icon (type specific icon) G4: Scroll direction icon

Claims

1. 1. An operation input control method for controlling an operation input device including a display device and an eye movement detection device that detects a change in the line of sight direction of an operator looking at a screen of the display device and a blink of the operator, comprising: a processor causing the display device to display a menu screen including a plurality of operation icons each to be an operation target, a cursor pointing to a position on the screen, and a plurality of type-specific icons corresponding to a plurality of types of operations including an icon selection operation for selecting one of the plurality of operation icons; While the menu screen is being displayed, the processor changes the display position of the cursor in accordance with the detection result of the change in the line of sight direction by the eye movement detection device; While the menu screen is being displayed, the processor periodically switches the plurality of type-specific icons one by one to an active display state; and when the blink is detected by the eye movement detection device while the menu screen is being displayed, the processor generates an operation input event corresponding to a combination of the cursor position and a type of operation corresponding to one of the plurality of type-specific icons in the active display state.

2. 2. The operation input control method according to claim 1, further comprising: when the menu screen is displayed, the processor stops switching the display state of the plurality of type-specific icons when an amount of change in the gaze direction detected by the eye movement detection device exceeds a predetermined lower limit value.

3. the plurality of type-specific icons include a selection operation icon corresponding to the icon selection operation and a scroll mode icon corresponding to an operation of transitioning from a normal mode to a screen scroll mode, When the menu screen is displayed and the display mode is the normal mode, the processor changes the display position of the cursor in accordance with a detection result of the change in the line of sight direction by the eye movement detection device; In a situation where the menu screen is being displayed and the display mode is the normal mode, the processor transitions the display mode to the screen scroll mode when the scroll mode icon is in the active display state when the blink is detected by the eye movement detection device; 3. The operation input control method according to claim 1, further comprising: when the menu screen is displayed and the display mode is the screen scroll mode, the processor scrolls the menu screen in accordance with a detection result of the change in the line of sight by the eye movement detection device.

4. the plurality of operation icons includes a scroll icon corresponding to a transition process from the normal mode to the screen scroll mode, In a situation where the menu screen is being displayed and the display mode is the normal mode, the processor sets the selection operation icon or the scroll mode icon to the active display state when the cursor is displayed in an area corresponding to the scroll icon; 4. The operation input control method according to claim 3, further comprising: when the menu screen is displayed and the display mode is the normal mode, the processor transitions the display mode to the screen scroll mode if the cursor is displayed in an area corresponding to the scroll icon when the blink is detected by the eye movement detection device.

5. the plurality of operation icons includes a scroll icon corresponding to a transition process from a normal mode to a screen scroll mode, When the menu screen is displayed and the display mode is the normal mode, the processor changes the display position of the cursor in accordance with a detection result of the change in the line of sight direction by the eye movement detection device; under a condition that the menu screen is being displayed and the display mode is the normal mode, the processor sets a selection operation icon corresponding to the icon selection operation to the active display state when the cursor is displayed in an area corresponding to the scroll icon; In a situation where the menu screen is being displayed and the display mode is the normal mode, the processor transitions the display mode to the screen scroll mode when the cursor is displayed in an area corresponding to the scroll icon when the blink is detected by the eye movement detection device; 3. The operation input control method according to claim 1, further comprising: when the menu screen is displayed and the display mode is the screen scroll mode, the processor scrolls the menu screen in accordance with a detection result of the change in the line of sight by the eye movement detection device.

6. the plurality of type-specific icons include a selection operation icon corresponding to the icon selection operation and a cancel icon corresponding to an operation for canceling a process, 3. The operation input control method according to claim 1, further comprising: when the menu screen is displayed, the processor generates a cancel event representing the cancellation of the most recent process as the operation input event when the cancel icon is in the active display state when the blink is detected by the eye movement detection device.

7. the plurality of type-specific icons include an ignore icon representing ignoring the detection of the blink by the eye movement detection device, 7. The operation input control method of claim 6, further comprising: when the cancellation event occurs, the processor sets the ignore icon to the active display state and then periodically switches the plurality of type-specific icons to the active display state.

8. 3. The operation input control method according to claim 1, wherein the processor displays the plurality of type-specific icons at equal intervals in an outer edge region of the screen of the display device.

9. a display device; an eye movement detection device that detects a change in the line of sight of an operator looking at a screen of the display device and a blink of the operator; 3. An operation input device comprising: a processor that detects an operation input by implementing the operation input control method according to claim 1 or 2, and that generates an event according to the detection result of the operation input.

10. an operation input device according to claim 9, which detects an operation input related to image processing and generates an event according to a result of the detection of the operation input; and one or more image processing units that perform the image processing in response to the event.

Images