Information processing device, information processing method, and storage medium

Abstract

To improve input operability or usability of a device.SOLUTION: An information processing device comprises: a registration information acquisition unit which acquires registration information in which operation determined from an eye and the shape of the periphery of the eye is associated with processing; an image acquisition unit which acquires an image of the eye and the periphery of the eye; a processing selection unit which selects the processing included in the registration information on the basis of the registration information and the image; and a control unit which controls so that the processing selected by the processing selection unit is performed.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present technology relates to an information processing device, an information processing method, and a storage medium, and in particular to the technical field of a user interface. [Background technology]

[0002] For example, in the field of imaging devices, there are known devices that recognize facial expressions such as smiling and actions such as closing the eyes as shutter operations and perform shutter processing. Patent Document 1 listed below discloses an information processing device that allows a user to perform input operations without using their hands. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2013-3647 A Summary of the Invention [Problem to be solved by the invention]

[0004] When taking pictures with an imaging device (camera), there are situations where you cannot use both hands, such as holding it with one hand. For example, with a single-lens reflex camera, the camera body is generally held with the left hand and operated with the right hand, so taking pictures is difficult if you cannot use both hands. Additionally, people with disabilities who are unable to use one hand also have difficulty operating a camera. As in Patent Document 1, by using facial expressions or closing the eyes as controls, it is possible to, for example, operate the shutter even in situations where one cannot use one's hands, but it is difficult to realize a wide variety of operations and processes. Although there is technology that can recognize operations via voice input, there are many situations where you cannot speak when taking a picture.

[0005] Therefore, the present technology proposes a technology that enables a variety of processes to be executed by selecting processing based on the shape of the user's eyes and the area around the eyes obtained as an image. [Means for solving the problem]

[0006] The information processing device according to the present technology includes a registration information acquisition unit that acquires registration information linked to a processing and an action determined from the shape of the eye and the area around the eye, an image acquisition unit that acquires an image of the eye and the area around the eye, a processing selection unit that selects a processing included in the registration information based on the registration information and the image, and a control unit that controls the processing selected by the processing selection unit to be performed. Images of the eyes and the area around the eyes are acquired, and the movement expressed by the eyes and the area around the eyes is determined. This movement includes not only the movement of the eyes themselves, but also the movement of the area around the eyes (for example, eyebrows and wrinkles) and changes in eye position due to neck movement. Then, a process is selected from the registered information that links the movement and the process, and the process is controlled to be executed. [Brief description of the drawings]

[0007] [Figure 1] 1 is an explanatory diagram of an example of an application of an information processing device according to an embodiment of the present technology; [Diagram 2] 1 is a perspective view of an imaging device according to an embodiment; [Diagram 3] 1 is a block diagram of an imaging apparatus according to an embodiment; [Figure 4] FIG. 1 is a block diagram of an information processing apparatus according to an embodiment. [Diagram 5] FIG. 2 is a block diagram of a functional configuration of a calculation unit according to the embodiment. [Figure 6] FIG. 4 is an explanatory diagram of registration information according to an embodiment. [Figure 7] FIG. 4 is an explanatory diagram of registration information according to an embodiment. [Figure 8] 4 is a flowchart of a processing example according to the first embodiment. [Figure 9] FIG. 2 is an explanatory diagram of eye movements recognized in the embodiment. [Figure 10] 11 is an explanatory diagram of an example of determination based on the degree of agreement of actions according to the embodiment; FIG. [Figure 11] 11 is an explanatory diagram of an example of determination based on the degree of agreement of actions according to the embodiment; FIG. [Figure 12] 11 is an explanatory diagram of an example of determination based on the degree of agreement of actions according to the embodiment; FIG. [Figure 13] 13 is a flowchart of a processing example according to the second embodiment. [Figure 14] 13 is a flowchart of a processing example according to the third embodiment. [Figure 15] 13 is a flowchart of a processing example according to the fourth embodiment. [Figure 16] 13 is a flowchart of a processing example according to the fifth embodiment. [Figure 17] FIG. 11 is an explanatory diagram of an operation example according to the embodiment. [Figure 18] FIG. 11 is an explanatory diagram of a notification example according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] The embodiments will be described below in the following order. <1. Applicable equipment examples> <2. Configuration example of imaging device> <3. Configuration example of information processing device> <4. Functional configuration of the calculation unit and registration information> <5. First embodiment> <6. Second embodiment> <7. Third embodiment> <8. Fourth embodiment> <9. Fifth embodiment> <10. Operation and Processing in Each Embodiment> <11. Summary and variations>

[0009] In this disclosure, the term "image" refers to both moving images and still images. When there is a particular need to distinguish between them, the terms "still image" and "moving image" will be used. Additionally, "image" refers to the image that is actually displayed on the screen, but "image" in the signal processing process and transmission path leading up to its display on the screen refers to image data.

[0010] <1. Applicable equipment examples> First, a specific example of an information processing device according to the present disclosure will be described. The information processing device of the present disclosure is a device capable of performing information processing, specifically, a device equipped with a microprocessor or the like, which is capable of selecting the processing to be performed based on an image captured of the user's eyes and the area around the eyes, and executing the selected processing. In the following description, an image of the user's eyes and the area around the eyes is also referred to as an “eye-area image.” In the embodiments, the “eye-area image” refers to an image including the eyes, wrinkles around the eyes, eyebrows, and the like.

[0011] As a specific example of selecting a process to be executed based on an image of the area around the eyes, a "movement" of the eyes and the area around the eyes is first determined from the image, and then a "process" associated with the determined "movement" is selected as the process to be executed. For example, a processor such as a CPU or DSP that performs such processing, or a device equipped with such a processor, is the information processing device referred to in this disclosure.

[0012] FIG. 1 illustrates imaging devices 1 and 1A and a terminal device 100 as specific examples of devices that correspond to information processing devices. The imaging device 1 is exemplified as a camera equipped with a viewfinder such as an EVF (Electric Viewfinder) 5, while the imaging device 1A is exemplified as a so-called compact type camera that does not have a viewfinder. A smartphone and a PC (personal computer) are illustrated as examples of the terminal device 100. Other examples of the terminal device 100 include a tablet device.

[0013] In these various devices, an imaging device, referred to for the sake of explanation as an "eye sensor camera 42," is mounted at a required position so as to capture an image of the area around the user's eyes and obtain an image of the area around the eyes. In the case of the imaging device 1, a configuration is assumed in which the eye sensor camera 42 is provided near the EVF 5 and an image of the eyes and the area around the eyes of a user looking into the EVF 5 is captured.

[0014] In the case of the imaging device 1A without a viewfinder, for example, an eye sensor camera 42 may be provided at a peripheral position of the rear monitor 4, and an image of the eyes and the surroundings of the eyes of a user looking at the rear monitor 4 may be captured.

[0015] In the case of a terminal device 100 such as a smartphone or a PC, a configuration is assumed in which an eye sensor camera 42 is provided at a peripheral position of the screen, and an image of the eyes and the area around the eyes of a user looking at the screen is captured.

[0016] For example, a configuration in which the eye sensor camera 42 is mounted in various devices is assumed. In this case, an internal processor device acquires an image captured by the eye sensor camera 42, i.e., an image around the eye, and selects a process included in the registration information using the image around the eye and the registration information, and executes the selected process.

[0017] The devices shown in Fig. 1 are merely examples. The technology of the present disclosure can be applied to devices that capture images of the area around the eyes and perform processing based on the images. There are a wide variety of devices that correspond to the information processing device of the present disclosure, such as television receivers, game devices, personal computers, workstations, head-mounted display devices, robots, monitoring devices, and sensor devices.

[0018] <2. Configuration example of imaging device> An example of the configuration of an imaging device 1 as an example of an information processing device according to the present disclosure will be described. 2 is a perspective view of the imaging device 1 as seen from the rear side. For the sake of explanation, the subject side is the front (front side) and the photographer side is the rear (rear side).

[0019] The imaging device 1 includes a camera housing 2 and a lens barrel 3 that is detachable from the camera housing 2 and attached to a front surface 2a. Note that the lens barrel 3 being detachable as a so-called interchangeable lens is just one example, and the lens barrel 3 may be a lens barrel that cannot be removed from the camera housing 2.

[0020] A rear monitor 4 is disposed on the rear portion 2b of the camera housing 2. On the rear monitor 4, a live view image, a reproduced image of a recorded image, and the like are displayed. The rear monitor 4 is configured by a display device such as a liquid crystal display (LCD) or an organic EL (Electro-Luminescence) display.

[0021] The EVF 5 is disposed on the top surface 2c of the camera housing 2. The EVF 5 includes an EVF monitor 5a and a frame-shaped surrounding portion 5b that protrudes rearward so as to surround the upper and left and right sides of the EVF monitor 5a. The EVF monitor 5a is formed using an LCD, an organic EL display, etc. Note that an optical view finder (OVF) may be provided instead of the EVF monitor 5a. As described above, the eye sensor camera 42 is disposed near the EVF 5, for example, inside the enclosure 5b, and captures an image of the area around the eyes of the user looking into the EVF 5.

[0022] Various types of controls 6 are provided on the rear surface 2b and the top surface 2c. For example, these include a shutter button, a playback menu start button, a decision button, a cross key, a cancel button, a zoom key, and a slide key. These controls 6 include various types of controls, such as a button, a dial, and a composite control that can be pressed and rotated. The various types of controls 6 enable, for example, a shutter operation, a menu operation, a playback operation, a mode selection / switching operation, a focus operation, a zoom operation, and parameter selection / setting of the shutter speed, F-number, etc.

[0023] The internal configuration of the imaging device 1 is shown in FIG. The imaging device 1 has, for example, a lens system 11, an imaging element unit 12, a camera signal processing unit 13, a recording control unit 14, a display unit 15, a communication unit 16, an operation unit 17, a camera control unit 18, a memory unit 19, a driver unit 22, a gaze detection device unit 41, an eye sensor camera 42, and a sensor unit 43.

[0024] The lens system 11 includes lenses such as a zoom lens and a focus lens, an aperture mechanism, etc. Light (incident light) from a subject is guided by the lens system 11 and collected on the imaging element unit 12.

[0025] The imaging element unit 12 includes an image sensor 12a (imaging element) such as a complementary metal oxide semiconductor (CMOS) type or a charge coupled device (CCD) type. The image sensor unit 12 performs, for example, CDS (Correlated Double Sampling) processing, AGC (Automatic Gain Control) processing, etc., on the electric signal obtained by photoelectrically converting the light received by the image sensor 12a, and further performs A / D (Analog / Digital) conversion processing. Then, the image signal as digital data is output to the downstream camera signal processor 13 and camera controller 18.

[0026] The camera signal processing unit 13 is configured as an image processor, for example, a DSP (Digital Signal Processor) etc. The camera signal processing unit 13 performs various signal processing on the digital signal (captured image signal) from the imaging element unit 12. For example, as camera processes, the camera signal processing unit 13 performs pre-processing, synchronization processing, YC generation processing, resolution conversion processing, etc.

[0027] The camera signal processing unit 13 then performs compression encoding, formatting, and generation and addition of metadata for recording or communication on the image data that has been subjected to the various processes described above, to generate files for recording or communication. For example, still image files may be generated in formats such as JPEG (Joint Photographic Experts Group), TIFF (Tagged Image File Format), GIF (Graphics Interchange Format), etc. It may also be possible to generate image files in the MP4 format used for recording video and audio conforming to MPEG-4. It is also possible to generate an image file as raw image data.

[0028] The recording control unit 14 performs recording and playback on a recording medium, such as a non-volatile memory. The recording control unit 14 performs processing for recording image files, such as moving image data and still image data, and metadata, such as thumbnail images, on the recording medium. The actual form of the recording control unit 14 may be various. For example, the recording control unit 14 may be configured as a flash memory built into the imaging device 1 and a write / read circuit for the memory. The recording control unit 14 may also be in the form of a card recording / playback unit that performs recording / playback access to a recording medium that can be detached from the imaging device 1, such as a memory card (such as a portable flash memory). The recording control unit 14 may also be realized as an HDD (Hard Disk Drive) built into the imaging device 1.

[0029] The display unit 15 is a display unit that displays various information to the photographer, and is, for example, a display device such as a liquid crystal panel (LCD: Liquid Crystal Display) or an organic EL (Electro-Luminescence) display, such as a rear monitor 4 or EVF 5 arranged on the housing of the imaging device 1. The display unit 15 executes various displays on the display screen based on instructions from the camera control unit 18. For example, the display unit 15 displays a reproduced image of image data read from a recording medium by the recording control unit 14. Further, image data of the captured image that has been resolution-converted for display by the camera signal processing unit 13 is supplied to the display unit 15, and the display unit 15 may perform display based on the image data of the captured image in response to an instruction from the camera control unit 18. This causes a so-called through image (live view image of the subject), which is an image captured while checking the composition or recording a movie, to be displayed. Further, based on instructions from the camera control unit 18, the display unit 15 executes display of various operation menus, icons, messages, etc., that is, GUI (Graphical User Interface) on the screen.

[0030] The communication unit 16 performs data communication and network communication with external devices via wired or wireless communication. For example, it transmits and outputs captured image data (still image files and video files) and metadata to external information processing devices, display devices, recording devices, playback devices, etc. The communication unit 16 also serves as a network communication unit, and can communicate via various networks such as the Internet, a home network, and a LAN (Local Area Network), and can transmit and receive various data between servers, terminals, and the like on the network. In addition, the imaging device 1 may be capable of communicating information between, for example, a PC, a smartphone, a tablet terminal, headphones, earphones, a headset, etc., via the communication unit 16, for example, via short-range wireless communication such as Bluetooth (registered trademark), Wi-Fi communication, NFC, etc., or infrared communication. Furthermore, the imaging device 1 and other devices may be capable of communicating with each other via wired connection communication.

[0031] The operation unit 17 collectively indicates input devices for a user to input various operations. Specifically, the operation unit 17 indicates various operators (keys, dials, touch panel, touch pad, etc.) provided on the housing of the imaging device 1. For example, it is assumed that the touch panel is provided on the surface of the rear monitor 4. An operation performed by the user is detected by the operation unit 17, and a signal corresponding to the input operation is sent to the camera control unit .

[0032] The gaze detection unit 41 is a device for detecting the gaze of the user, and is configured, for example, by an infrared LED that irradiates the user's eyes with infrared rays, an infrared camera that captures an image of the user's eyes, and the like. 2 and sending an image captured by the infrared camera to the camera control unit 18, it is possible to detect the gaze direction of a user looking into the EVF 5. Alternatively, such gaze detection device unit 41 may be arranged near the rear monitor 4 so as to capture an image for detecting the gaze direction of a user looking at the rear monitor 4. As an embodiment, a configuration example not including the gaze detection device unit 41 is also contemplated.

[0033] As described above, the eye sensor camera 42 is provided, for example, in the EVF 5 and captures an image of the user's eyes and the area around the eyes. The eye sensor camera 42 has, for example, a visible light image sensor and an image signal processing circuit, and obtains still image data or video data as, for example, a color image or a monochrome image. In other words, these still images and videos are images of the area around the eyes. The images of the area around the eyes captured by the eye sensor camera 42 are sent to the camera control unit 18.

[0034] The camera control unit 18 is configured by a microcomputer equipped with a CPU (Central Processing Unit). The memory unit 19 stores information and the like used for processing by the camera control unit 18. The illustrated memory unit 19 collectively represents, for example, a Read Only Memory (ROM), a Random Access Memory (RAM), a flash memory, and the like. The memory unit 19 may be a memory area built into a microcomputer chip serving as the camera control unit 18, or may be configured by a separate memory chip.

[0035] The camera control unit 18 executes a program stored in the ROM, flash memory, or the like of the memory unit 19 to control the entire imaging device 1 . For example, the camera control unit 18 controls the operation of each necessary unit with respect to controlling the shutter speed of the image sensor unit 12, instructing various signal processing in the camera signal processing unit 13, imaging operations and recording operations in response to user operations, playback operations of recorded image files, operations of the lens system 11 such as zoom, focus, and aperture adjustment in the lens barrel, user interface operations, etc.

[0036] Furthermore, when the gaze detection device unit 41 is provided, the camera control unit 18 can perform processing for detecting the gaze direction of the user based on an infrared image captured by the gaze detection device unit 41. The camera control unit 18 can also perform various controls based on the detected gaze direction of the user. For example, it is possible to set a focus area so that a subject in the gaze direction is in just focus according to the gaze direction, and to perform aperture adjustment control according to the brightness of a subject in the gaze direction.

[0037] The camera control unit 18 has a function as a calculation unit 40 by an application program. The calculation unit 40 performs a process of acquiring registration information in which a process is associated with an action determined from the shape of the eye and the area around the eye. The calculation unit 40 also performs a process of acquiring an image of the area around the eye from the eye sensor camera 42. The calculation unit 40 also performs a process of selecting a process included in the registration information based on the registration information and the image of the area around the eye. The calculation unit 40 also controls so that the selected process is executed. The processing performed by the calculation unit 40 will be described in detail later.

[0038] The camera control unit 18 may perform so-called AI (artificial intelligence) processing for the processing of the calculation unit 40 and other processing.

[0039] The RAM in the memory unit 19 is used as a working area for various data processing by the CPU of the camera control unit 18, for temporarily storing data, programs, and the like. The ROM and flash memory (non-volatile memory) in memory unit 19 are used to store the OS (Operating System) that the CPU uses to control each part, content files such as image files, application programs for various operations, firmware, various setting information, etc. The various types of setting information include communication setting information, setting information relating to imaging operations such as exposure settings, shutter speed settings, and mode settings, and setting information relating to image processing such as white balance settings, color settings, and settings relating to image effects.

[0040] The memory unit 19 also stores programs for processing using the eye peripheral image and various calibration processes. The memory unit 19 also stores data used for these processes. For example, the memory unit 19 can also function as a database for selecting processing using the eye peripheral image and registered information. For example, the registered information described below is stored.

[0041] The driver unit 22 includes, for example, a motor driver for a zoom lens drive motor, a motor driver for a focus lens drive motor, a motor driver for a diaphragm mechanism motor, and the like. These motor drivers apply drive currents to the corresponding drivers in response to instructions from the camera control unit 18, thereby moving the focus lens and zoom lens, opening and closing the aperture blades of the aperture mechanism, and so on.

[0042] The sensor unit 43 collectively represents various sensors mounted on the imaging device 1. As the sensor unit 43, for example, an IMU (inertial measurement unit) may be mounted. With the IMU, for example, angular velocity can be detected by a three-axis angular velocity (gyro) sensor for pitch, yaw, and roll, and acceleration can be detected by an acceleration sensor. Thereby, the attitude of the imaging device 1 with respect to the gravitational direction and the like can be detected. Also, as the sensor unit 43, for example, a pressure sensor, a touch sensor, a position information sensor, an illuminance sensor, a distance measurement sensor, etc. may be mounted.

[0043] <3. Configuration Example of Information Processing Device> Next, with reference to FIG. 4, a configuration example of the terminal device 100 as a PC or a smartphone shown in FIG. 1 will be described as an example of the information processing device of the present disclosure.

[0044] The CPU 71 of the terminal device 100 executes various processes according to programs stored in the ROM 72 or the non-volatile memory unit 74 such as an EEP-ROM (Electrically Erasable Programmable Read-Only Memory), or programs loaded from the storage unit 79 to the RAM 73. The RAM 73 also appropriately stores data and the like necessary for the CPU 71 to execute various processes. The CPU 71, ROM 72, RAM 73, and non-volatile memory unit 74 are interconnected via a bus 83. An input / output interface 75 is also connected to this bus 83.

[0045] Since the terminal device 100 is also assumed to perform image processing and AI (artificial intelligence) processing, instead of or together with the CPU 71, a GPU (Graphics Processing Unit), GPGPU (General-purpose computing on graphics processing units), an AI dedicated processor, etc. may be provided.

[0046] An input unit 76 consisting of operators and operation devices is connected to the input / output interface 75. For example, the input unit 76 may be various operators and operation devices such as a keyboard, a mouse, a key, a dial, a touch panel, a touch pad, a remote controller, or the like. An operation by the user is detected by the input unit 76, and a signal corresponding to the input operation is interpreted by the CPU 71. A microphone may also be used as the input unit 76. Voice uttered by the user may also be input as operation information. In addition, various sensing devices such as an image sensor (imaging unit), an acceleration sensor, an angular velocity sensor, a vibration sensor, an air pressure sensor, a temperature sensor, and an illuminance sensor are also envisioned as input units.

[0047] A display unit 77 including an LCD or an organic EL panel, and an audio output unit 78 including a speaker are connected to the input / output interface 75 either integrally or separately. The display unit 77 is a display unit that performs various displays, and is configured, for example, by a display device provided in the housing of the terminal device 100, a separate display device connected to the terminal device 100, or the like. The display unit 77 displays images for various types of image processing, moving images to be processed, etc., on the display screen based on instructions from the CPU 71. Furthermore, the display unit 77 displays various operation menus, icons, messages, etc., that is, as a GUI (Graphical User Interface), based on instructions from the CPU 71.

[0048] The input / output interface 75 may be connected to a storage unit 79 configured with a hard disk or solid-state memory, or a communication unit 80 configured with a modem or the like. Various programs, data files, etc. are stored in the storage unit 79. A database may also be constructed. The communication unit 80 performs communication processing via a transmission path such as the Internet, and communication with various devices via wired / wireless communication, bus communication, and the like.

[0049] A drive 81 is also connected to the input / output interface 75 as required, and a removable recording medium 82 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory is appropriately mounted thereon. The drive 81 allows data files such as image files and various computer programs to be read from the removable recording medium 82. The read data files are stored in the storage unit 79, and images and sounds contained in the data files are output on the display unit 77 and the sound output unit 78. Furthermore, the computer programs and the like read from the removable recording medium 82 are installed in the storage unit 79 as necessary.

[0050] In this terminal device 100, for example, software for the processing of this embodiment can be installed via network communication by the communication unit 80 or via a removable recording medium 82. Alternatively, the software may be stored in advance in the ROM 72, the storage unit 79, etc.

[0051] The input / output interface 75 may also be connected to a gaze detection device unit 41. As described in the configuration of the imaging device 1 in Fig. 3, the gaze detection device unit 41 is a device for detecting the gaze of a user, and is configured, for example, by an infrared LED that irradiates infrared rays to the user's eyes, an infrared camera that captures an image of the user's eyes, or the like.

[0052] The eye sensor camera 42 is connected to the input / output interface 75. As described in Fig. 3, the eye sensor camera 42 also includes, for example, a visible light image sensor and an image signal processing circuit, and captures an image of the area around the eye. The captured image of the area around the eye is used for processing by the CPU 71.

[0053] There is a case where a sensor unit 43 is connected to the input / output interface 75. As the sensor unit 23, various sensing devices are assumed, similar to those described in the configuration of the imaging device 1 in FIG.

[0054] In the terminal device 100, the CPU 71 is provided with a function as the calculation unit 40 by an application program. The calculation unit 40 has the same processing function as the calculation unit 40 described in the imaging device 1 of FIG. The non-volatile memory unit 74, the ROM 72, and the storage unit 79 store information necessary for the processing of the calculation unit 40. For example, registration information, which will be described later, is stored.

[0055] <4. Functional configuration of the calculation unit and registration information> The calculation unit 40 in the above-mentioned imaging device 1 and terminal device 100 will be described. 5 shows the functional configuration of the calculation unit 40. The calculation unit 40 includes an image acquisition unit 51, a process selection unit 52, a registration information acquisition unit 53, and a control unit 54 as software functional configurations realized by application programs.

[0056] The image acquisition unit 51 has a function of acquiring an image captured by the eye sensor camera 42 as a processing target. The eye sensor camera 42 continuously performs an image capturing operation at, for example, a predetermined frame rate, and supplies image data of each frame as an eye peripheral image to the camera control unit 18 or the CPU 71. The image acquisition unit 51 sequentially captures the image data of each frame (or intermittent frames) as a processing target.

[0057] The registration information acquisition unit 53 acquires, from the storage unit 55, registration information in which a motion determined from the shape of the eye and the area around the eye is associated with a process.

[0058] The storage unit 55 is configured as, for example, a storage area of ​​the memory unit 19 in the imaging device 1 of Fig. 2, or as a storage area of ​​the non-volatile memory unit 74 or storage unit 79 in the terminal device 100 of Fig. 3. The storage unit 55 stores registration information.

[0059] An example of the registration information is shown in FIG. The registration information is information that is registered in advance by a registration operation by a user or as a preset registration by the manufacturer of the imaging device 1 or the terminal device 100. The registered information stores the associations between "actions" and "processes". An "action" is an action expressed by the shape of the eyes and the area around the eyes. For the sake of explanation, such an action expressed by the shape of the eyes and the area around the eyes is also referred to as an "eye gesture". And specific aspects as eye gestures are registered in the registered information. The actions registered in the registered information are, for the sake of explanation, called "registered actions". For example, as actions expressed by eye gestures, in the example of FIG. 6, it shows that "closing eyes tightly", "winking", "being surprised", "smiling", and "frowning eyebrows" are regarded as registered actions.

[0060] In the registered information, processes are associated with these registered actions. The associated processes are, for the sake of explanation, called "registered processes". In the example of FIG. 6, for each registered action such as "closing eyes tightly", various operation processes of "shutter operation", "AF (auto focus) operation", "AE (auto exposure) operation", "peaking assist operation", and "zoom in operation" are associated as registered processes. Note that peaking assist is a process of highlighting and presenting the in-focus position within the through image.

[0061] Also, the registered information may be provided for each user as shown in FIG. 7. FIG. 7 shows an example where registered information is stored for each individual user U1 and U2.

[0062] For example, the registered information regarding user U1 is associated with the personal identification information of user U1, registered actions such as "closing eyes tightly", "winking", "being surprised", "smiling", and "frowning eyebrows", and registered processes such as "shutter operation", "AF operation", "AE operation", "peaking assist", and "zoom in operation". On the other hand, for example, the registered information regarding user U2 is associated with the personal identification information of user U2, registered actions such as "narrowing eyes", "frowning eyebrows", "closing eyes", "putting force on eyes (staring)", and "opening eyes wide", and registered processes such as "decision operation", "shutter operation", "continuous shooting operation", "zoom in operation", and "zoom out operation".

[0063] For example, as in the examples of Figs. 6 and 7, the registration action and the registration process are linked as the registration information.

[0064] The storage unit 55 also stores data for determining whether an eye-periphery image is a registered motion, such as shape data corresponding to a motion, a data set of a DNN (Deep Neural Network) learning model, and pattern data used for object recognition by semantic segmentation. Furthermore, the storage unit 55 may also store calibration data that corresponds to individual differences in the shape of the area around the eyes and individual differences in movements of each user. The memory unit 55 may also store data for personal recognition when using individual registration information as shown in FIG. 7, such as code information that encodes an iris pattern or pattern data of the shape of the eyes.

[0065] The registration information acquiring unit 53 in FIG. 5 acquires data on the registration operation from the storage unit 55 as a function of the registration operation acquiring unit 53a. Furthermore, the registration information acquiring unit 53 acquires data on the registration process from the storage unit 55 as a function of the registration process acquiring unit 53b.

[0066] The process selection unit 52 is a function that selects the registration process included in the registration information based on the registration information and the eye surrounding image. In this example, the process selection section 52 performs processes as an encoding section 52a, a coincidence calculation section 52b, and a selection section 52c.

[0067] The encoding unit 52a encodes the frame of the eye-periphery image acquired by the image acquisition unit 51. This is a process of performing calculations using a machine learning model for one frame of the eye-periphery image, detecting the "state" expressed by the user's eyes and the area around the eyes, and obtaining encoded data indicating the state. The "state" refers to the state of the shape of the eyes and the area around the eyes at that moment, or the state of emotion and action estimated from the shape. A specific example will be described later. Examples of algorithms for encoding include image processing such as class classification using a machine learning model, edge detection, and extraction of differences from a previous image. This allows the "state" to be detected on a frame-by-frame basis.

[0068] The matching degree calculation unit 52b calculates the degree of matching between the actions represented in the face peripheral image and each registered action. For example, information on a continuous "state" for a certain period of time is taken as the "action" of the face peripheral image. In other words, the matching degree calculation unit 52b takes the "state" obtained by the encoding unit 52a during a predetermined frame period from the present to the past as the "action," and calculates the degree of matching between the "action" and each registered action. The matching degree calculation unit 52b may calculate the degree of matching between the moving image data itself as a plurality of frames and each registered motion, without encoding each frame as described above.

[0069] The selection unit 52c selects a registration process based on the degree of coincidence calculated by the degree of coincidence calculation unit 52b. For example, the selection unit 52c determines the registration action with the highest degree of coincidence, and selects the registration process associated with the registration action.

[0070] The process selection unit 52 selects one registration process by, for example, the processes of the encoding unit 52a, the coincidence calculation unit 52b, and the selection unit 52c.

[0071] The control unit 54 controls the imaging device 1 and the terminal device 100 so that the registration process selected by the process selection unit 52 is executed in the imaging device 1 and the terminal device 100. For example, when "shutter operation" is selected, the control unit 54 controls the operation of the imaging device 1 and the terminal device 100 so that the shutter process (process of capturing and recording a still image) is executed.

[0072] 6 and 7, the registration process is exemplified as a process for executing a user operation, but the registration process is not limited to a process corresponding to the operation. Specific examples will be described later, but various auxiliary processes include notification to the user, processes related to the display screen, and processes suitable for the shooting situation. By setting such processes other than the operation as the registration process, the control unit 54 may also perform control other than control corresponding to the operation.

[0073] <5. First embodiment> An example of processing by the calculation unit 40 in the first embodiment will be described. The determination of the registered motion may be performed for each face surrounding image as a video of multiple frames, or a "state" may be identified for each image (one frame), and a match determination may be performed with the registered motion for each combination of the "states" as "motions." In the first embodiment, an example of identifying a "state" for each frame, which is a relatively light processing load, will be described.

[0074] FIG. 8 shows an example of processing by the calculation unit 40. In step S101, an application that executes the functions of the calculation unit 40 is started. That is, a process based on an application program for determining an eye gesture based on an eye peripheral image obtained by the eye sensor camera 42 and selecting and executing a corresponding process is started. By this application program, the processes of the functions of the image acquisition unit 51, the process selection unit 52, the registration information acquisition unit 53, and the control unit 54 shown in FIG. 5 are executed in the processor as the calculation unit 40.

[0075] The calculation unit 40 then repeats the processes shown as steps S102 to S109 until it is determined in step S110 that the application has ended.

[0076] In step S102, the calculation unit 40 acquires one frame of an image of the area around the user's eyes captured by the eye sensor camera . Note that there are cases where the captured image of one frame does not include an image of the area around the eyes. For example, in the case of the imaging device 1, an image of the area around the eyes is obtained when the user is looking into the EVF 5, but an image of the area around the eyes is not obtained when the user is not looking into the EVF 5. Although not shown in the flowchart, if the acquired frame is an image that does not show the eyes, the calculation unit 40 will not perform the processes from step S103 onwards, and will proceed to acquire the next frame.

[0077] In step S103, the calculation unit 40 obtains the "state" of the user by processing the obtained eye-periphery image by the above-mentioned encoding unit 52a. That is, as described above, the "state" represented by the shape of the user's eyes and the areas around the eyes is detected by a method such as class classification using a machine learning model.

[0078] FIG. 9 shows specific examples of the "state" that can be determined from the image around the eyes. From one frame of the image around the eyes, it is possible to determine the shape of an individual's eyes and whether they are left or right eyes. In other words, shape information for personal identification of a user and information indicating the left or right eye are examples of "status" information. In addition, from one frame of the image around the eyes, it is possible to determine the degree to which the eyes are open at that time: wide open, normal, closed, tightly closed, etc. These are also examples of "state" information.

[0079] Also, the position, orientation, and rotation state of the eyes can be determined from one frame of the eye surrounding image. For example, the position of the user's eyes looking into the EVF 5, that is, information on the position and orientation of the eyes relative to the EVF 5. Such information includes the normal position, a rotation state shifted in the X direction (horizontal direction) or Y direction (vertical direction), and a state shifted in the Z direction (the perspective direction relative to the EVF 5). In addition, there is a state when the user looks into the EVF 5 from above, as shown as "downward," a state when the user looks into the EVF 5 from below, as shown as "upward," a state when the user looks into the EVF 5 from the left or right, as shown as "sideways." Furthermore, depending on the attitude of the imaging device 1, that is, whether the user holds the imaging device 1 vertically or horizontally, the length and width change, and there is also a state like "holding vertically" in the figure. These are also examples of "state" information.

[0080] Further, although not shown, the "condition" may include information such as the shape of wrinkles around the eyes, and the shape, position, and angle of eyebrows, which are also examples of "condition" information.

[0081] In step S104 of Fig. 8, the calculation unit 40 updates the "action" of the user. For example, information on a certain continuous "state", such as the "state" from the current frame to the past N frames, is set as the "action" at the current time. That is, a new "state" is added for each frame, and any "state" from a certain time ago is deleted, and the multiple "states" during that certain time are treated as an "action." If a result that is clearly different from the judgment results of the "state" before and after is detected, it is desirable to treat it as an outlier and correct it.

[0082] Since an action is understood as a succession of "states" as shown in Figure 9, there are actions that are expressed by changes in the "state" and actions that are expressed by the continuation of the "state."

[0083] For example, as an "action" resulting from the continuation of a "state," there are actions that are classified as the same as the "state," such as "opening one's eyes wide" and "closing one's eyes tightly." For example, if the state for a certain period of time is always "opening one's eyes wide," the "action" is also determined to be "opening one's eyes wide." In addition, the "actions" expressed by changes in the "state" include "continuous winking," "winking at regular intervals," "head movement," and "facial expressions." "Continuous winks" and "winks at regular intervals" are determined by detecting the "state" of "eyes open" or "eyes closed" over a certain period of time. "Head movement" includes, for example, "horizontal movement," "approaching," "moving away," and "rotation," and these can be determined by changes in position, orientation, and rotation as "states" over a certain period of time.

[0084] In addition, there is "facial expression" as an "action" that can be determined by either the continuation or change of a "state." Examples of facial expressions include "surprise," "smile," "frown," "glare," and "contempt," and these can be determined by the continuation or change of "states" over a certain period of time, such as the degree of eye opening, the degree of wrinkles, and the shape and angle of eyebrows.

[0085] The calculation unit 40 determines the "motion" as in the above example from the "state" of each of the multiple frames of the face peripheral image.

[0086] In step S105, the calculation unit 40 calculates the degree of match between the updated "motion" and all registered motions. Note that all registered actions refer to all registered actions stored when registration is not performed for each individual, as in Fig. 6. When registration information is prepared for each individual, as in Fig. 7, the user is identified from the eye surrounding image, and all registered actions are registered for that individual. Also, calculating the degree of similarity with all registered actions is just one example, and it is also possible to determine several candidate registered actions that are inferred from a certain "action," i.e., a "state" over a certain period of time, and then perform a degree of similarity calculation for the candidate registered actions.

[0087] After calculating the degree of match between the "movement" determined from the eye peripheral image and, for example, all registered movements, the calculation unit 40 selects the registered movement with the highest degree of match that is equal to or greater than a threshold value. For this purpose, the calculation unit 40 first checks in step S106 whether or not there is a registered movement with a degree of match that is equal to or greater than a threshold value.

[0088] If there is no registered motion (eye gesture), it is determined that no motion corresponding to the registered motion is currently being performed, and the process returns to step S102 via step S110. In other words, in this case, processing control based on eye gesture is not initiated.

[0089] If there is a registered motion whose degree of matching is equal to or greater than the threshold, the calculation unit 40 checks in step S107 whether there is one registered motion whose degree of matching is the highest. If the multiple registered movements have the highest degree of match, the movement cannot be clearly determined, and the calculation unit 40 returns to step S102 via step S110. In this case, the process control based on the eye gesture is not initiated.

[0090] If there is one registration action with the highest degree of match, the calculation unit 40 proceeds to step S108 and selects the registration process associated with that registration action.

[0091] Then, in step S109, the calculation unit 40 performs control so that the selected registration process is executed. For example, if the selected process is a shutter operation process, the calculation unit 40 performs control so that a still image capture and recording process in response to the shutter operation is executed.

[0092] The process of FIG. 8 is repeated until it is determined in step S110 that the application is to be terminated. When the user makes an eye gesture at any time while the application is running, a corresponding operation process or the like is executed.

[0093] In step S108, the calculation unit 40 may notify the user of the action determination result or the process selection result by displaying the result on the screen as an icon or text. The result may be notified by sound effects or voice. If the user shakes his / her head in response to such a notification, the selection may be cancelled. For example, when a process that does not match the user's intention is about to be executed, the process of step S110 may be cancelled without being executed.

[0094] An example of selection of processing depending on the coincidence calculation in the processing of FIG. 8 will be described. For example, consider the case where the calculation result of the degree of match is as shown in Fig. 10. The example in Fig. 10 shows the degree of match calculated for each of the registered actions "close eyes tightly," "wink," "surprise," "smile," and "frown" when the registered information includes such actions. In step S106, the calculation unit 40 compares the degree of coincidence with a threshold value.

[0095] In the example of Fig. 10, the threshold is different for each registration action. This is to adjust the ease of initiation as a registration process linked to the registration action. For example, a registration action that corresponds to a registration process that does not cause any problems if it is accidentally activated, such as a shutter operation, has a lower threshold to make it easier to activate. On the other hand, if there is a registration process such as an image deletion operation, the threshold for the registration action linked to it is set higher to carefully determine the eye gesture. Of course, this is just one example, and the threshold may be the same for all registration operations.

[0096] In the example of Fig. 10, the degree of match for "surprise" exceeds the threshold. Therefore, the calculation unit 40 proceeds to step S107. In this case, the only registered motion with the highest degree of match is "surprise." Therefore, the calculation unit 40 proceeds to steps S108 and S109, and performs control so that a registration process corresponding to "surprise," for example, a process corresponding to an AE operation according to the example of the registration information in Fig. 6, is performed.

[0097] 11 is a case where there are multiple registered actions such as "close eyes tightly" and "frown" whose matching degree exceeds the threshold, and these matching degrees are the same at the maximum value. In such a case, the calculation unit 40 returns from step S107 to step S102 and does not initiate the process.

[0098] 12, there is a case where the registration action with the highest degree of matching is "smile", but there is no registration action with a degree of matching exceeding the threshold value. In such a case, the calculation unit 40 returns from step S106 to step S102 and does not initiate the process.

[0099] Note that the above process is an example. For example, without providing the threshold determination in step S106, if there is a registration action with the highest degree of matching as shown in FIG. 12, control of the registration process linked to that registration action may be executed. Also, when there are multiple registered actions with the highest degree of matching as shown in Fig. 11, one of them may be selected and the execution control of the corresponding process may be performed. For example, it is possible to prioritize the processes and select one of them based on the priority.

[0100] <6. Second embodiment> The second embodiment is an example in which a user is notified when there is no registered action with a degree of matching equal to or greater than a threshold value, or when there are multiple registered actions with a degree of matching equal to the maximum value, as shown in Figures 11 and 12 above.

[0101] 13 shows an example of processing by the calculation unit 40. In the flowcharts of the following embodiments, the same processes as those already described are given the same step numbers to avoid duplicated explanations.

[0102] In the processing example of FIG. 13, if it is determined in step S106 that there is no registered motion with a degree of matching equal to or greater than the threshold, the calculation unit 40 proceeds to step S120 and performs processing to notify information regarding the motion with the highest degree of matching. For example, in the case of Fig. 12, "smile" has the highest degree of agreement, but since it is lower than the threshold, the certainty of the eye gesture is low. Therefore, the calculation unit 40 displays a message such as "Please squint your eyes and smile" on the display unit 15 such as the EVF 5, and notifies the user to smile more clearly. This is expected to encourage the user to make clearer eye gestures in response to the notification.

[0103] If it is determined in step S107 that there are a plurality of registered actions with the highest degree of match, the calculation unit 40 proceeds to step S121 and performs control to display information for distinguishing between the actions. For example, if the degree of match for both "closing your eyes tightly" and "frowning" reaches its maximum value, as shown in Figure 11, it would be possible to control the display so that a notification such as "Please close your eyes longer or move your eyebrows a bit more" is displayed using text or an icon. This is also expected to encourage users to make clearer eye gestures.

[0104] <7. Third embodiment> An example of processing in the third embodiment is shown in Fig. 14. Fig. 14 is a modified example of the processing between step S102 and step S106 in Fig. 6 or Fig. 13 described above. It is assumed that the registration information is registered for each individual as shown in FIG.

[0105] After acquiring the frame of the eye peripheral image in step S102, the calculation unit 40 determines whether the user is a registered user based on the eye peripheral image in step S130 of Fig. 14. For example, the shape of the eyes in the eye peripheral image is compared with the shape of the eyes of a registered user. It may also be determined whether the user is a registered user by iris authentication.

[0106] If it is determined that the image is not an eye-periphery image of a registered user, the calculation unit 40 returns to step S102 from step S131 via step S110. In other words, if the user is not a registered user, processing according to the eye gesture determination will not be initiated.

[0107] On the other hand, if it is determined that the user is registered, the calculation unit 40 determines the "state" and updates the "action" in steps S104 and S105, and acquires registration information corresponding to the currently determined individual user in step S132. Then, in step S106A, the degree of agreement between each of the registered actions included in the registration information of the corresponding user and the "action" determined in step S105 is calculated. Then, from step S106 onwards, the calculation unit 40 performs the process described in FIG. 6 or FIG. 13.

[0108] According to the processing example of FIG. 14, processing by eye gesture can be initiated only for registered users.

[0109] <8. Fourth embodiment> A processing example of the fourth embodiment is shown in Fig. 15. Fig. 15 shows an example in which processing is added between step S109 and step S110 in Fig. 6 or Fig. 13 above.

[0110] When the control of the process selected in step S109 is executed, the calculation unit 40 determines a negative action of the user in step S150. For example, within a predetermined time (e.g., about 2 seconds) immediately after a process such as a shutter operation, the presence or absence of a motion that can be presumed to be negative to the processing operation, such as frowning, surprise, etc., is determined from the image around the user's face. If the user shakes his or her head from side to side, this may be determined to be a negative motion.

[0111] If no particular negative action is detected, the calculation unit 40 returns to step S102 via step S110. On the other hand, if a negative action is detected, the calculation unit 40 proceeds to step S152 and selects a corresponding action according to the execution process.

[0112] When the user indicates a negative action, it is assumed that the process executed in response to the eye gesture was not in line with the user's intention. In this case, a response process is performed to recover the executed process. However, since the appropriate response process differs depending on the type of the executed process, a response process is selected for each process. For example, when a still image capture and recording process is performed in response to a shutter operation, it is considered that there will be no significant damage to the user, so only a notification will be issued. On the other hand, when an image deletion process is performed, it is a relatively serious malfunction, so the deletion will be canceled (deleted image will be restored) and a notification will be issued. When an AF process or the like is performed, it may be sufficient to only cancel it (cancel AF).

[0113] If the selection of the corresponding process in step S152 is to cancel the process, the calculation unit 40 proceeds from step S153 to step S154 to perform the cancellation process. Note that the cancellation process in step S154 may be configured to perform an inquiry notification as to whether or not to cancel, and to perform the cancellation process when the user performs an operation (including an eye gesture) that allows the cancellation.

[0114] If a notification such as a message is to be executed by the selection of the corresponding process in step S152, the calculation unit 40 proceeds from step S155 to step S156 and performs a necessary notification process for the user. For example, the calculation unit 40 notifies the user that a shutter process has been performed by a process using an eye gesture.

[0115] According to the processing example of FIG. 15, recovery can be performed when it is presumed that initiating a process by eye gesture was not appropriate for the user.

[0116] <9. Fifth embodiment> As a fifth embodiment, an example of calculating the degree of coincidence using a moving image of multiple frames is shown in FIG.

[0117] After starting an application in step S201, the calculation unit 40 repeats the processes from step S202 to step S207 until the application is terminated in step S208.

[0118] In step S202, the calculation unit 40 acquires a moving image as an image of the area around the eye. In this case, a plurality of frames capturing an action over a certain period of time are processed. Then, in step S203, the calculation unit 40 performs DNN processing or the like on the video of multiple frames and calculates the degree of match with the registered motion.

[0119] Steps S204 to S207 are similar to the processes in steps S106 to S109 in FIG.

[0120] In this way, a processing example for determining the degree of match with the registered motion in units of multiple frames is also conceivable. However, in this case, the processing load is larger than the processing for determining the "state" for each frame as shown in Fig. 8, so it is desirable to apply this to a device with ample processing capacity.

[0121] <10. Operation and Processing in Each Embodiment> The following describes operations and processes that can be applied to the processing examples of the first to fifth embodiments described above.

[0122] As described above, by providing the registration information, acquiring an image of the area around the eyes, determining the movement, and selecting a process, various processes according to the eye gesture can be realized in the imaging device 1 and the terminal device 100. For example, the following processes are possible.

[0123] When you close your eyes tightly, the shutter is operated and a still image is captured and recorded. · AF and AE are performed when your eyes are wide open. - When you strain your eyes (glare), zooming and peaking assistance will occur. When you laugh or get surprised, the shutter is operated and a still image is captured and recorded. When previewing an image immediately after capturing and recording a still image, if the user frowns, the preview screen will immediately close. If it is determined that the user is smiling while viewing an image preview, the image will be tagged or protected. If the user has their eyes closed when a setting change notification is sent, the notification will be sent again after the user opens their eyes. When changing a value using the dial, if you frown while looking at the results, it will assume that you have changed the value too far and will reset the value. -If the user squints due to glare, the exposure will be reduced and / or the brightness of the EVF5 will be reduced.

[0124] The above is a specific example of processing corresponding to an action as an eye gesture. Such combinations of operations and processes are set in advance by the user or manufacturer and registered as registration information. These exemplary operations and processes will be described below.

[0125] Examples of combinations of operations and processes include the operation and processing of an operation system and the processing of an operation assistance system. First, examples of the operation and processing of the operation system will be described.

[0126] The action of closing the eyes tightly can be processed as, for example, a shutter operation or a confirmation operation. It may also be processed as a continuous shooting operation depending on the duration of closing the eyes.

[0127] It is possible to associate AF processing and AE processing with the action of opening the eyes wide. In one example, AF processing and AE processing are performed based on the position of the cursor (mouse cursor) in the through image displayed on the EVF5. Furthermore, in the case where the line of sight detection device 41 is provided and line of sight detection is possible, the AF processing and AE processing may be performed on a subject ahead of the line of sight on the display of the EVF 5 or the like.

[0128] For example, zoom processing or peaking assist processing may be associated with the action of glaring with narrowed eyes. There is also an example where this is performed for the position of the cursor (mouse cursor) on the image displayed on the EVF 5. In addition, if the gaze detection device unit 41 is provided and gaze detection is possible, peaking assist processing may be performed for a subject in front of the gaze on the display of the EVF 5, etc.

[0129] Furthermore, as an operation of the operation system, for example, a composite operation using eye gestures may be registered as an operation. For example, combine the action of "squinting your eyes" with the action of "moving your head." Fig. 17A shows a state in which a preview image 30 is being viewed on the EVF 5. With such preview image 30 displayed, if the user narrows their eyes as shown in Fig. 17B and moves their head up, down, left and right as shown in Fig. 17C, the calculation unit 40 recognizes this as a drag operation of the preview image 30 and moves the image. Also, as shown in FIG. 17D, when the user narrows his / her eyes and moves his / her head back and forth, the calculation unit 40 may recognize this as an enlargement / reduction operation of the preview image 30 and perform enlargement / reduction control of the image. Furthermore, as shown in FIG. 19E, when the user narrows his / her eyes and shakes his / her head from side to side, the calculation unit 40 may recognize this as an operation to skip the preview image 30 to the next image, and perform image scrolling control on the display.

[0130] Next, an example of processing of the operation assistance system will be described. For example, the calculation unit 40 performs processing to respond to a situation estimated from a movement determined as an eye gesture.

[0131] It is conceivable that processing may be performed to call up data stored in accordance with personal identification, whether the eye is left or right, whether the imaging device 1 is held horizontally or vertically, and so on. For example, when the registration information is stored according to the individual as described above, the registration information is selected and called up according to the individual identification. In addition, since the shape of the eyes and the areas around the eyes differ from person to person and depending on whether the eye is left or right, it is assumed that some degree of calibration will be performed. When storing such calibration data, the calibration data will be called up according to the recognition of the person and the recognition of the left or right eye. It is appropriate to select registration information and calibration data depending on whether the device is held horizontally or vertically.

[0132] Auxiliary processing can also be performed depending on the head position. For example, the position at which the head is peering can be detected, and the GUI, live view image, and playback image of recorded images displayed on the EVF5, etc. can be made to track the position of the head. Furthermore, in the case where the gaze detection device unit 41 is provided, the position of the infrared LED that is turned on may be changed depending on the head position in order to improve detection accuracy.

[0133] Auxiliary processing can also be performed by detecting blinking. For example, if a setting value is changed and it is detected that the user has their eyes closed at the time of notifying the user of the change, the notification will be made again after the user opens their eyes.

[0134] In response to squinting, the camera can provide additional processing to deal with glare situations, for example lowering the exposure or reducing the brightness of the EVF5 when the user squints.

[0135] There are also cases where auxiliary processing is performed by determining emotions such as smiling. For example, if the user smiles or looks surprised when previewing a captured image, the image may be tagged as a favorite or protected.

[0136] If a negative emotion behavior, such as furrowing the brow, is detected, the following processing example may be performed. For example, if a negative emotion is detected in response to a system notification (such as a pop-up), the notification will be immediately terminated, or a setting change will be made so that no further notifications will be sent. Additionally, if a negative emotional gesture is detected during image preview, the image preview will immediately end. In addition, if negative emotional behavior is detected during the process of inputting various values, operating dials, etc., the value settings will be reset.

[0137] Moreover, if the user performs an action indicating "difficulty in seeing," such as squinting, it is determined that the display is small and difficult to see, and processing is performed to enlarge or simplify the display to make it easier to see. Depending on the action indicating "difficulty in seeing," the age and appropriate size may be set during calibration.

[0138] The relationship between the actions and processes as exemplified above is stored in advance as registration information. An example of a procedure for registering actions and processes will now be described.

[0139] First, the user selects a process that can be executed on a device such as the imaging device 1 or the terminal device 100. Note that, even for the same operation, different processing may be set for each target (subject, shooting mode, preview time, etc.).

[0140] Next, the user performs a motion (eye gesture) that is associated with the action. It is appropriate to perform the eye gesture multiple times and have the system perform a calibration and learning process. In this case, the shape of the eye and its surroundings observed from the eye-periphery image and information indicating changes in the shape are registered. Furthermore, the user sets a threshold value for the degree of matching for the registration action (see S106).

[0141] Here, when performing an eye gesture as an input for a registration operation, it is preferable to first register the user's straight face. The user then performs eye gestures in response to the emotions presented by the system (laughter, surprise, anger, etc.). This is done multiple times, and the system calibrates and learns. This allows the system to register motion information that is suited to the individual.

[0142] By allowing an individual user to perform registration as described above, a registration action that matches the eye gesture of the user is stored, and the registration process desired by the user is linked to the gesture. Note that even if the user does not perform registration by himself, the device may store registration information based on a combination of a registration operation and a registration process as preset data.

[0143] It is desirable to register eye and eye-related movements (eye gestures) on the device that is actually used. The registered information may also be stored on the cloud so that it can be accessed on another device.

[0144] The registration action may be set as a combination of multiple actions. By setting a combination of multiple actions, it is suitable for preventing erroneous operation. For example, opening your eyes wide and then closing them tightly is assumed to be a shutter operation.

[0145] Eye gestures can be combined with other operations (button input, voice input). For example, saying "shutter" and closing your eyes will operate the shutter.

[0146] It is desirable to improve the accuracy of the registration operation by saving settings for each user and by performing repeated calibration. Calibration may be performed in the background to update accuracy after each successful operation in use. When an unregistered user uses a device, instead of not invoking the process as in FIG. 14, the function of this example may be invoked by utilizing the registration information of other users. Furthermore, when the imaging device 1 or the like is held vertically and horizontally, for example, information on the registration action when held vertically may be rotated and used as data on the registration action when held horizontally.

[0147] In addition, in outdoor situations where the sunlight is too bright and wrinkles form between the eyebrows even with a straight face, it is possible to prevent malfunction by changing the threshold value for determining movement. Neutral face data suited to such a situation may be stored separately. Also, the neutral face may be updated and the registration operation may be automatically corrected. Furthermore, when biometric information such as heart rate, sweating, and goose bumps can be obtained from an image around the eyes or from other sensors, a registration operation combined with this information may be stored.

[0148] <11. Summary and variations> According to the above embodiment, the following effects can be obtained. The information processing device according to the embodiment is realized as the imaging device 1 or the terminal device 100. Alternatively, it is realized as a processor having the functions of the calculation unit 40. An information processing device of such an embodiment includes a registration information acquisition unit 53 that acquires registration information linked to an action determined from the shape of the eye and the area around the eye, an image acquisition unit 51 that acquires an image of the area around the eye, a processing selection unit 52 that selects a registration processing included in the registration information based on the registration information and the image of the area around the eye, and a control unit 54 that controls the processing selected by the processing selection unit 52 (the processing linked to the registration action) to be performed. That is, the calculation unit 40 acquires an eye peripheral image, determines an eye and eye peripheral movement (eye gesture), selects a registration process associated with a registration movement that matches the eye gesture in the registration information, and controls to execute the registration process.

[0149] This allows the user to perform operations such as taking pictures using eye gestures even when their hands are full. Also, the information processing device can predict the user's intention and perform operation assistance processing accordingly. In particular, eye movements are determined by not only the eyes but also the area around the eyes. For example, not only the degree to which the eyes are open, but also the movement of wrinkles around the eyes is used for identification. This makes it possible to distinguish whether the eyes are closed tightly or lightly, even if the same eyes are closed. Also, there is a lot of information that can be obtained from things other than the degree to which the eyes are open. This makes it possible to identify a wider variety of movements and link them to processing. In addition, by detecting the area around the eyes, eye gestures can be detected more accurately, regardless of the various eye shapes of people. For example, eye movements can be detected with high accuracy regardless of whether the person has large or small eyes. In addition, by comparing eye gestures with registered actions, it is possible to distinguish between eye gestures that are valid as operations, etc. and eye gestures that are invalid (unregistered) as operations, etc.

[0150] In the embodiment, an example was given in which the processing selection unit 52 calculates the degree of similarity between the action appearing in the eye-periphery image and the registered action in the registration information, and selects the registration processing linked to the registered action whose degree of similarity meets a certain numerical value (threshold) (see Figures 8, 13, and 16). The degree of match between the motion observed in multiple frames of a moving image or still image and each registered motion is determined, and registered motions with a degree of match equal to or greater than a certain threshold are determined. Then, a registration process corresponding to the registered motion is selected. As a result, registered motions that do not meet the threshold are not selected, and therefore motions with a high degree of likelihood can be determined from among various motions.

[0151] In the embodiment, an example has been described in which the processing selection unit 52 calculates the degree of similarity between the action appearing in the eye-periphery image and all registered actions in the registration information, and selects the processing associated with the action that has the highest degree of similarity among the registered actions (see Figures 8, 13, and 16). Calculating the degree of agreement between the captured image of the area around the eyes and all registered actions is effective in preventing erroneous determination of actions due to the original shape of the user's eyes. For example, the shape, size, thinness, eyelid condition, wrinkles, etc. of the eyes differ from user to user, but by comparing with all registered actions, it is possible to determine the most likely registered action according to the shape of the area around the user's eyes.

[0152] In the first embodiment, the processing selection unit 52 calculates the degree of similarity with registered actions by using the encoded information of the state over a certain period of time, which is obtained by determining and encoding the state of the eyes and the area around the eyes for each frame of the eye-periphery image (see FIG. 8 ). The "state" of the eyes and the area around the eyes is determined for each frame using processing such as class classification using a machine learning model, and then encoded. The state (change in state) at each point in time is then treated as a "movement." The likely movement can be determined by comparing the "movement" with registered movements. By encoding each still image as a "state" in this way, the processing burden can be significantly reduced compared to inputting multiple frames of images into a DNN or similar device to determine the movement.

[0153] In the second embodiment, an example has been described in which the calculation unit 40 controls the notification to the user when a plurality of candidates for an action are obtained from among the registered actions (see FIG. 13). For example, a notification may be given to the user to encourage a clearer eye gesture, thereby improving the accuracy of operation recognition. A notification may be given to prompt the user to select one of a number of options.

[0154] In the second embodiment, an example was described in which the calculation unit 40 controls notification to the user when the degree of matching between the actions appearing in the eye-periphery image and the registered actions is calculated and the degree of matching does not satisfy a certain numerical value (threshold value) for any of the registered actions (see FIG. 13). For example, the user may be notified to make a clearer eye gesture, thereby improving the accuracy of the operation recognition. Alternatively, the user may be notified to select the highest match, even if the match is low.

[0155] In the embodiment, an example has been given in which the registration information includes personal identification information for identifying the user (see FIG. 7). Images of the eyes and the areas around the eyes enable personal identification. If personal identification information is included as registration information, it becomes possible to respond according to the personal identification of the user.

[0156] An example has been given in which the processing selection unit 52 determines whether the acquired eye-periphery image is an eye-periphery image of a registered user based on personal identification information, and does not select processing if the image is not of a registered user (see Figure 14). That is, the eye gesture processing is not activated unless the user is registered. This allows only registered specific users to perform eye gesture operations, etc. In other words, it is possible to prevent a user who is unfamiliar with the eye gesture function from being confused by activating the function when using the device.

[0157] In the third embodiment, the registration information is associated with a motion determined from the shape of the eyes and the area around the eyes in accordance with the personal identification information, and the processing selection unit 52 uses the information on the registered motion and registered processing corresponding to the user identified based on the personal identification information in the registration information to determine the motion and select the processing (see Figs. 7 and 14). This allows the eye gestures represented by the eye peripheral images for each user to be registered as actions, improving the accuracy of judgment. Calibration data can also be accumulated for each user. In addition, any operation can be registered as an eye gesture for each user. It is also possible to set auxiliary processing required for each user. As a result, even in a device used by multiple users, eye gesture processing suitable for each user is possible.

[0158] In the embodiment, the eye-periphery image is an image including the shape of eyebrows or the shape of wrinkles around the eyes. By including the shape of eyebrows and wrinkles in the image of the eyes and the area around the eyes, eye gestures can be determined more accurately than simply based on the state of the eyes. In particular, since eye shapes vary from person to person, including the shape of eyebrows and wrinkles makes it easier to determine the degree to which the eyes are open or closed, emotional expression, etc.

[0159] In the embodiment, the registration process selected by the process selection unit 52 includes an operation response process corresponding to a user operation. That is, the eye gesture is recognized as an "operation" such as a shutter operation, and the control unit 54 executes a process corresponding to the operation. This enables the user to perform an intentional operation by an eye gesture.

[0160] The registration processes selected by the process selection unit 52 include operation corresponding processes for the recording operation of the captured image, the zoom-in operation, and the zoom-out operation of the camera. For example, an image capturing operation can be performed as an eye gesture. The image capturing operation can be a shutter operation for capturing a still image, a recording start operation for capturing a moving image, etc. Alternatively, a zoom operation such as zooming in and out can be performed as an eye gesture. Since these are operations that are performed relatively frequently, making them possible with eye gestures is effective in improving operability.

[0161] In the embodiment, the registration process selected by the process selection unit 52 includes a process corresponding to an operation related to a specific operation image or an image specified by gaze detection. For example, specific operation images such as mouse cursors and icons, icons identified by gaze detection, operations related to images such as still images, such as dragging operations and zooming operations, can be made possible by eye gestures. This makes it possible to perform a variety of operations as GUI operations by eye gestures.

[0162] In the embodiment, the registration process selected by the process selection unit 52 includes a process for responding to a situation estimated from a determined motion. For example, as shown in the various examples of the operation assistance system, the usability and convenience of the device can be improved by automatically responding to the situation estimated from the determined actions. For example, the display brightness can be lowered in response to the user's actions that seem dazzling. This can be said to be processing that also responds to the user's unconscious eye gestures.

[0163] In the fourth embodiment, an example was given in which, when the image around the eyes determines that the processing selection unit 52 is performing a negative action in relation to the execution of control by the control unit 54, the processing selection unit 52 selects a corresponding processing, and the control unit 54 controls so that the corresponding processing is performed (see FIG. 15). For example, when performing some kind of operation processing or operation assistance processing using eye gestures, the processing may not be what the user intended. In such cases, the system will determine the negative intention from the image of the eyes and the area around the eyes, and respond by notifying the user or canceling the operation. This allows recovery from the case where the user recognizes the operation as unintended.

[0164] Modifications and applications of the technology of the present disclosure will be further described below. In the case of an imaging device 1 or a terminal device 100 equipped with a gaze detection device unit 41, it is possible to identify left / right, portrait orientation, and the individual based on the shape of the eyes determined from the image around the eyes, and automatically switch calibration data for gaze detection.

[0165] Gaze detection involves finding a "reference point" and a "moving part (moving point)" of the eye, and then detecting the gaze from the position of the moving point relative to the reference point. For example, the corneal reflection method involves photographing the eye with a point light source shining on the cornea, and measuring the center of corneal curvature found from this corneal reflection image (Purkinje image) as the reference point and the moving point as the pupil. The gaze direction is then calculated from the positions of the moving point and the reference point. To carry out this type of gaze detection processing, calibration according to individual differences is necessary. Therefore, identifying individuals from images around the eyes and automatically switching calibration data for gaze detection is effective in improving gaze detection accuracy.

[0166] Also, it is advisable to notify the user of this switching using text or an icon. For example, notifications such as "The right eye was detected, so the data was switched to the right eye," or "Portrait orientation was detected, so the data was switched" may be given. If the user reacts negatively to the notification, such as frowning, the switch may be cancelled.

[0167] If your gaze is difficult to detect due to poor viewing position, a guide notification can be displayed to help you move your head to the correct position. For example, in Fig. 18A, the user is looking into the EVF 5 of the imaging device 1 from above, making it difficult to detect the line of sight. Such a head position can be determined from the image around the eyes. Therefore, the EVF 5 display, as shown in Fig. 18B, notifies the user to look into the EVF 5 from the front. This is expected to encourage the user to correct the posture of their head, as shown in Fig. 18C. Such notification may be made during calibration of gaze detection.

[0168] In the embodiment, the imaging device 1 and the terminal device 100 are given as examples of information processing devices of the present technology. The present technology can also be realized in the terminal device 100 such as a smartphone or a PC by being configured to be able to capture an image of the area around the eyes. In addition, since the camera that captures the user's image on a smartphone or other device is often wide-angle, a face detection function is used to cut out only the image around the eyes, and the processing of this embodiment is performed using the cut-out image around the eyes.

[0169] When the technology disclosed herein is applied to a smartphone, it is difficult to operate the smartphone when taking a selfie because the device is held in one hand. Therefore, usability can be improved by registering actions and processes, such as closing one eye tightly and performing the shutter process after the action is completed.

[0170] In the case of a PC, people often use both hands to operate the device, so gesture input using the eyes is effective. For example, opening your eyes wide and then closing them tightly while typing can take a screenshot.

[0171] It may also be combined with gaze detection in a PC or the like. For example, if you glare at a pop-up notification, the notification will disappear instantly. Additionally, if you are surprised by a pop-up notification, the details of the notification will be displayed. Also, if you close your eyes tightly while looking at a disabled window, it will become enabled.

[0172] The eye sensor camera 42 in the imaging device 1 or the terminal device 100 has an angle of view that covers the eyebrows from the center of the eye, and may have any configuration or arrangement as long as it can capture the image. A camera for other purposes, such as eye contact detection or gaze detection, may be used as the eye sensor camera 42. Furthermore, a distance sensor, a pressure sensor, a touch sensor, or the like may be provided in the device such as the imaging device 1 or the terminal device 100 to detect the movement of facial muscles and use the detected movement for determining an eye gesture.

[0173] Also, a switch for enabling / disabling input by eye gestures may be provided as a physical operator or as an operator on the system.

[0174] Regarding eye gesture input, if a certain registration action / registration process is determined and then invalidated by the user, it is appropriate to determine that the determination is incorrect and to perform a learning process.

[0175] The program of the embodiment is a program that causes a processor such as a CPU or a DSP, or a device including these, to execute the processes shown in the above-mentioned FIGS. 8, 13, 14, 15, and 16. In other words, the program of the embodiment is a program that causes an information processing device to execute the following steps: acquiring registration information linked to an action determined from the shape of the eye and the area around the eye, acquiring an image of the area around the eye, selecting a registration process included in the registration information based on the registration information and the image of the area around the eye, and controlling the selected process to be performed.

[0176] By using such a program, an information processing device that executes the processing of the calculation unit 40 described above can be realized by various types of computer devices.

[0177] Such a program can be pre-recorded in a HDD as a recording medium built into a device such as a computer device, or in a ROM in a microcomputer having a CPU. Also, such a program can be temporarily or permanently stored (recorded) in a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disk, a DVD (Digital Versatile Disc), a Blu-ray Disc (registered trademark), a magnetic disk, a semiconductor memory, or a memory card. Such a removable recording medium can be provided as a so-called package software. Furthermore, such a program can be installed in a personal computer or the like from a removable recording medium, or can be downloaded from a download site via a network such as a LAN (Local Area Network) or the Internet.

[0178] In addition, such a program is suitable for providing a wide range of information processing devices according to the embodiment. For example, by downloading the program to a personal computer, a communication device, a mobile terminal device such as a smartphone or tablet, a mobile phone, a game device, a video device, a PDA (Personal Digital Assistant), or the like, these devices can function as the information processing device of the present disclosure.

[0179] It should be noted that the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0180] The present technology can also be configured as follows. (1) a registration information acquisition unit that acquires registration information in which a motion determined from the shape of the eye and the area around the eye is associated with a process; An image acquisition unit for acquiring an image of the eye and the area around the eye; a process selection unit that selects a process included in the registration information based on the registration information and the image; A control unit that controls the process selected by the process selection unit to be performed. Information processing device. (2) The processing selection unit A degree of match between the action appearing in the image and the action in the registered information is calculated, and a process associated with the action whose degree of match satisfies a certain numerical value is selected. The information processing device according to (1) above. (3) The processing selection unit A degree of match between the action appearing in the image and all actions in the registration information is calculated, and a process associated with the action that has the highest degree of match among the registered actions is selected. The information processing device according to (1) or (2) above. (4) The processing selection unit The coded information of the state during a certain period obtained by determining and coding the state of the eyes and the surroundings of the eyes for each frame of the image is used as information of the movement appearing in the image, and a degree of coincidence with the movement in the registered information is calculated, the movement is determined based on the degree of coincidence, and a process associated with the determined movement is selected. An information processing device according to any one of (1) to (3) above. (5) In a case where the processing selection unit determines a motion based on a degree of coincidence between the motion appearing in the image and the motion in the registration information and selects a processing associated with the determined motion, If multiple action candidates are found, control the notification to the user. An information processing device according to any one of (1) to (4) above. (6) In a case where the processing selection unit determines a motion based on a degree of coincidence between a motion appearing in the image and a certain motion in the registration information and selects a processing associated with the determined motion, If the degree of match for any of the actions in the registered information does not meet a certain numerical value, notification to the user is controlled. An information processing device according to any one of (1) to (5) above. (7) The registration information includes personal identification information that identifies the user. An information processing device according to any one of (1) to (6) above. (8) The processing selection unit Based on the personal identification information, it is determined whether the image is an image of a registered user, and if the image is not an image of a registered user, no selection of processing is performed. The information processing device according to (7) above. (9) The registration information associates a motion determined from the shape of the eyes and the surroundings of the eyes with a process in correspondence with the personal identification information, The processing selection unit determines an action and selects a processing by using information on an action and a processing corresponding to a user identified based on the personal identification information in the registration information. The information processing device according to (7) or (8) above. (10) The image of the eye and the area around the eye is an image including the shape of the eyebrows or the shape of the wrinkles around the eye. An information processing device according to any one of (1) to (9) above. (11) The process selected by the process selection unit includes an operation response process in response to a user operation. An information processing device according to any one of (1) to (10) above. (12) The process selected by the process selection unit includes a process corresponding to at least one of a recording operation of a captured image, a zoom-in operation, and a zoom-out operation of a camera. An information processing device according to any one of (1) to (11) above. (13) The process selected by the process selection unit includes a process corresponding to an operation on a specific operation image or an image identified by gaze detection. An information processing device according to any one of (1) to (12) above. (14) The process selected by the process selection unit includes a process for responding to a situation estimated from the determined motion. An information processing device according to any one of (1) to (13) above. (15) The process selection unit, when determining a negative action against the execution of control by the control unit based on the image, selects a corresponding process; The control unit controls the corresponding process to be performed. An information processing device according to any one of (1) to (14) above. (16) An information processing device, Acquire registration information that associates a process with a motion determined from the shape of the eyes and the area around the eyes; Obtaining an image of the eye and the area surrounding the eye; selecting a process included in the registration information based on the registration information and the image; Controlling the selected process to be performed Information processing methods. (17) A step of acquiring registration information in which a motion determined from the shape of the eyes and the surroundings of the eyes is associated with a process; acquiring an image of the eye and the area surrounding the eye; selecting a process included in the registration information based on the registration information and the image; A control procedure for performing the selected process; A computer-readable storage medium storing a program for causing an information processing device to execute the above. [Explanation of symbols]

[0181] 1. Imaging device 5. EVF 18 Camera control unit 19 Memory section 40 Arithmetic section 41 Line of Sight Detection Device 42 Eye Sensor Camera 43 Sensor section 51 Image acquisition unit 52 Processing selection section 52a Encoding section 52b Matching calculation part 52b Selection section 53 Registration Information Acquisition Department 53a Registration operation acquisition unit 53b Registration process acquisition unit 54 Control section 55 Storage section 71 CPU 100 Terminal Equipment

Claims

1. a registration information acquisition unit that acquires registration information in which a motion determined from the shape of the eye and the surrounding area of ​​the eye is associated with a process; An image acquisition unit for acquiring an image of the eye and the area around the eye; a process selection unit that selects a process included in the registration information based on the registration information and the image; A control unit that controls the process selected by the process selection unit to be performed. Information processing device.

2. The processing selection unit A degree of match between the action appearing in the image and the action in the registered information is calculated, and a process associated with the action whose degree of match satisfies a certain numerical value is selected. The information processing device according to claim 1 .

3. The processing selection unit A degree of match between the action appearing in the image and all actions in the registration information is calculated, and a process associated with the action that has the highest degree of match among the registered actions is selected. The information processing device according to claim 1 .

4. The processing selection unit The coded information of the state during a certain period obtained by determining and coding the state of the eyes and the surroundings of the eyes for each frame of the image is used as information of the movement appearing in the image, and a degree of coincidence with the movement in the registered information is calculated, the movement is determined based on the degree of coincidence, and a process associated with the determined movement is selected. The information processing device according to claim 1 .

5. In a case where the processing selection unit determines a motion based on a degree of coincidence between the motion appearing in the image and the motion in the registration information and selects a processing associated with the determined motion, If multiple action candidates are found, control the notification to the user. The information processing device according to claim 1 .

6. In a case where the processing selection unit determines a motion based on a degree of coincidence between a motion appearing in the image and a certain motion in the registration information and selects a processing associated with the determined motion, If the degree of match for any of the actions in the registered information does not meet a certain numerical value, notification to the user is controlled. The information processing device according to claim 1 .

7. The registration information includes personal identification information that identifies the user. The information processing device according to claim 1 .

8. The processing selection unit Based on the personal identification information, it is determined whether the image is an image of a registered user, and if the image is not an image of a registered user, no selection of processing is performed. The information processing device according to claim 7.

9. The registration information associates a motion determined from the shape of the eyes and the surroundings of the eyes with a process in correspondence with the personal identification information, The processing selection unit determines an action and selects a processing by using information on an action and a processing corresponding to a user identified based on the personal identification information in the registration information. The information processing device according to claim 7.

10. The image of the eye and the area around the eye is an image including the shape of the eyebrows or the shape of the wrinkles around the eye. The information processing device according to claim 1 .

11. The process selected by the process selection unit includes an operation response process in response to a user operation. The information processing device according to claim 1 .

12. The process selected by the process selection unit includes a process corresponding to at least one of a recording operation of a captured image, a zoom-in operation, and a zoom-out operation of a camera. The information processing device according to claim 1 .

13. The process selected by the process selection unit includes a process corresponding to an operation on a specific operation image or an image identified by gaze detection. The information processing device according to claim 1 .

14. The process selected by the process selection unit includes a process for responding to a situation estimated from the determined motion. The information processing device according to claim 1 .

15. The process selection unit, when determining a negative action against the execution of control by the control unit based on the image, selects a corresponding process; The control unit controls the corresponding process to be performed. The information processing device according to claim 1 .

16. An information processing device, Acquire registration information that associates a process with a motion determined from the shape of the eyes and the area around the eyes; Obtaining an image of the eye and the area surrounding the eye; selecting a process included in the registration information based on the registration information and the image; Controlling the selected process to be performed Information processing methods.

17. A step of acquiring registration information in which a motion determined from the shape of the eyes and the surroundings of the eyes is associated with a process; acquiring an image of the eye and the area surrounding the eye; selecting a process included in the registration information based on the registration information and the image; A control procedure for performing the selected process; A computer-readable storage medium storing a program for causing an information processing device to execute the above.

Images