Imaging device, imaging method, and imaging control program
The imaging device uses eye movement detection to lock onto and control imaging conditions based on user gaze, addressing camera shake issues and ensuring accurate focus on intended subjects.
Patent Information
- Application Number
- JP2020198120
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2020-11-30
- Publication Date
- 2024-11-13
- Estimated Expiration
- 2040-11-30
AI Technical Summary
Existing imaging systems, such as cameras with monitors, face challenges in accurately controlling imaging conditions due to camera shake, leading to unintended subjects being selected for shooting mode control.
An imaging device with an optical system, image sensor, and eye movement detection unit that specifies and locks onto areas gazed at by the user for a predetermined time, allowing control of imaging conditions based on eye movements to maintain focus on the desired subject.
Enables precise control of imaging conditions on the desired subject, preventing unintended subjects from being selected due to camera shake, without the user needing to remove their gaze from the monitor.
Smart Images

Figure 0007581803000001 
Figure 0007581803000002 
Figure 0007581803000003
Abstract
Description
[Technical field]
[0001] The present invention relates to an imaging apparatus, an imaging method, and an imaging control program, and more particularly to an imaging apparatus, an imaging method, and an imaging control program suitable for controlling imaging conditions of a desired subject with high precision. [Background technology]
[0002] Patent document 1 discloses a video surveillance system comprising a first camera that captures an image of an area to be monitored, a display device that displays the image captured by the first camera, a second camera that captures an image of a user looking at the image displayed on the display device, a detection means that detects the user's eye movements based on the image captured by the second camera, and a control means that controls the manner in which the first camera captures images in accordance with the user's eye movements. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] Patent No. 6276954 Summary of the Invention [Problem to be solved by the invention]
[0004] However, when the structure of Patent Document 1 is applied to a camera with a monitor that displays a subject image (including a camera mounted on a mobile terminal such as a smartphone), the influence of camera shake makes it impossible to identify which subject image on the monitor the user is looking at, and there is a possibility that an unintended subject will be selected as the subject for which the shooting mode is to be controlled. In other words, the related technology has a problem in that it is not possible to precisely control the shooting conditions of a desired subject.
[0005] The present invention has been made in consideration of the above-mentioned problems, and has an object to provide an imaging device, an imaging method, and an imaging control program suitable for controlling imaging conditions of a desired subject with high precision. [Means for solving the problem]
[0006] An imaging device according to one aspect of this embodiment includes an imaging unit that images a subject and generates image data, a display unit that displays an image represented by the image data, a detection unit that detects a user's eye movement, an identification unit that identifies an area of the image displayed on the display unit that is gazed upon by the user for a predetermined period of time or more, and a control unit that controls imaging conditions for a subject included in an image of the area identified by the identification unit in accordance with the predetermined eye movement of the user detected by the detection unit.
[0007] An imaging method according to one aspect of this embodiment images a subject to generate image data, displays an image represented by the image data on a display unit, detects the movement of a user's eyes, identifies an area of the image displayed on the display unit that is gazed upon by the user for a predetermined period of time or longer, and controls the imaging conditions of the subject included in the image of the identified area in accordance with the predetermined eye movement of the user.
[0008] An imaging control program according to one aspect of this embodiment causes a computer to perform the following processes: capturing an image of a subject and generating image data; displaying an image represented by the image data on a display unit; detecting a user's eye movement; identifying an area of the image displayed on the display unit that is gazed upon by the user for a predetermined period of time or longer; and controlling the imaging conditions of a subject included in the image of the identified area in accordance with a predetermined eye movement of the user. Effect of the Invention
[0009] According to the present embodiment, it is possible to provide an imaging device, an imaging method, and an imaging control program suitable for controlling imaging conditions of a desired subject with high precision. [Brief description of the drawings]
[0010] [Figure 1]1 is a block diagram showing an example of a configuration of an imaging device according to a first embodiment. [Diagram 2] 2 is a flowchart showing an example of the operation of the imaging apparatus shown in FIG. [Diagram 3] 2 is a diagram showing an example of an image displayed on a display unit mounted on the imaging device shown in FIG. 1. [Figure 4] 2 is a diagram showing an example of an image displayed on a display unit mounted on the imaging device shown in FIG. 1. [Diagram 5] 10 is a flowchart showing another example of the operation of the imaging apparatus shown in FIG. [Figure 6] 2 is a block diagram showing an example of a hardware configuration of an imaging control function of the imaging device according to the first embodiment. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0011] In the following, specific embodiments to which the present invention is applied will be described in detail with reference to the drawings. In each drawing, the same elements are given the same reference numerals, and for clarity of explanation, duplicate explanations will be omitted as necessary.
[0012] <Embodiment 1> FIG. 1 is a block diagram showing a configuration example of an imaging device 1 according to the first embodiment. The imaging device 1 is a camera with a monitor (including a camera mounted on a mobile terminal such as a smartphone or tablet) that displays an image of a subject to be imaged, and controls the imaging conditions of a subject included in an image of a monitor area gazed upon by a user (image-taker) for a predetermined period of time according to a predetermined eye movement. Thereby, the imaging device 1 can prevent a subject unintended by the user from being selected as a control target of the imaging conditions due to the influence of camera shake. In addition, the imaging device 1 can control the imaging conditions of a desired subject without taking the line of sight off the display unit by controlling the imaging conditions according to the eye movement. That is, the imaging device 1 according to the first embodiment can accurately control the imaging conditions of a desired subject without being influenced by camera shake. A specific description will be given below.
[0013] As shown in FIG. 1, the imaging device 1 includes an imaging unit 11, a display unit 12, a detection unit 13, an identification unit 14, and a control unit 15.
[0014] The imaging unit 11 is a unit that captures an image of a subject and generates image data. Specifically, the imaging unit 11 at least includes an optical system 111, an imaging element 112, and an image processing unit 113.
[0015] The optical system 111 is made up of a focus lens, a zoom lens, an aperture, a shutter, and the like.
[0016] The imaging element 112 is, for example, a CMOS image sensor or a CCD image sensor, and captures a subject image formed via the optical system 111 to generate imaging data. Here, the imaging element 112 generates imaging data captured by a user or an automatic shutter operation.
[0017] The image processing unit 113 generates image data by performing a predetermined process on the imaging data generated by the imaging element 112. The image data generated by the image processing unit 113 is stored in a storage unit (not shown).
[0018] The display unit (monitor) 12 displays an image indicated by image data generated by the image processing unit 113 or image data stored in a storage unit (not shown).
[0019] The detection unit 13 detects the eye movement of a user looking at the display unit 12. For example, the detection unit 13 is a camera mounted on the imaging device 1 so as to be able to capture an image of the user looking at the display unit 12, and detects the eye movement of the user from an image captured by the camera.
[0020] The identification unit 14 identifies an area of the image displayed on the display unit 12 (monitor area) that is gazed at by the user for a predetermined period (e.g., 3 seconds) or more as a control target area for the imaging conditions. This makes it possible to prevent an area unintended by the user from being identified as a control target area for the imaging conditions due to the influence of camera shake. The identification unit 14 estimates the area gazed at by the user from, for example, eye movement (gaze) detected by the detection unit 13.
[0021] The control unit 15 performs overall control of the imaging device 1. Furthermore, the control unit 15 controls the imaging conditions of a subject included in an image of an area identified by the identification unit 14 among the areas of the image displayed on the display unit 12, in accordance with the eye movement detected by the detection unit 13. This allows the user to control the imaging conditions of a desired subject without taking his / her eyes off the monitor.
[0022] In this way, the imaging device 1 according to the present embodiment can prevent a subject unintended by the user from being selected as a control subject for imaging conditions due to the influence of camera shake by setting the subject gazed at for a predetermined period or more as a control subject for imaging conditions. Also, the imaging device 1 according to the present embodiment can control the imaging conditions of a desired subject without taking the line of sight off the display unit by controlling the imaging conditions according to the movement of the eyes. That is, the imaging device 1 according to the present embodiment 1 can accurately control the imaging conditions of a desired subject without being affected by camera shake.
[0023] (flowchart) Next, the operation of the imaging device 1 will be described with reference to FIGS. Fig. 2 is a flowchart showing an example of the operation of the imaging device 1. Fig. 3 and Fig. 4 are diagrams showing an example of an image displayed on the display unit 12 mounted on the imaging device 1. Fig. 3 shows an image on the display unit 12 before the imaging conditions are controlled, and Fig. 4 shows an image on the display unit 12 after the imaging conditions are controlled.
[0024] First, a user (photographer) holds the imaging device 1 toward a subject (step S101). At this time, an image of the subject is displayed on the display unit 12 mounted on the imaging device 1. In the example of Fig. 3, an image including subject A is displayed on the display unit 12.
[0025] After that, the user stares at an image of a desired subject among the images displayed on the display unit 12 (step S102). In the example of Fig. 3, the user stares at the image of subject A. The identification unit 14 estimates the area stared at by the user from the movement of the eyes (line of sight) detected by the detection unit 13, for example.
[0026] When the user continues to gaze at a desired subject for a predetermined period of time or more, the identification unit 14 identifies an area including the desired subject as an area to be controlled for the imaging conditions (YES in step S102 to S103). In other words, the identification unit 14 locks the gaze target. The locked area may be highlighted, for example, by being surrounded by a white frame.
[0027] 3, since the user continues to gaze at the subject A for a predetermined period or longer, the identification unit 14 identifies the area B including the subject A as the area to be controlled in the imaging conditions. In other words, the identification unit 14 locks the area B including the subject A.
[0028] If the user's gaze period is less than the predetermined period, the gaze target is not locked (NO in step S102).
[0029] Thereafter, the imaging conditions of the locked area are controlled according to the eye movement of the user detected by the detection unit 13 (step S104). Note that how the imaging conditions are controlled according to the eye movement is determined in advance, but can be changed by the user, and the number of settings can be freely increased. The user's settings are compiled in a correspondence table and stored in a storage device (not shown).
[0030] For example, when the user moves his / her eyes up and down, the control unit 15 performs focus control of the locked area using a focus lens provided in the optical system 111 (YES in step S104→S105). When the user moves his / her eyes left and right, the control unit 15 performs zoom control of the locked area using a zoom lens provided in the optical system 111 (YES in step S104→S106). When the user rotates his / her eyes, the control unit 15 performs aperture control (brightness control) of the locked area using an aperture provided in the optical system 111 (YES in step S104→S107). This allows the user to control the imaging conditions of the locked area without taking his / her eyes off the monitor.
[0031] 4, the user moves his / her eyes left and right, thereby performing zoom control on area B including subject A (i.e., area B including subject A is enlarged).
[0032] If the user does not move his / her eyes, the imaging conditions of the locked area are not controlled (NO in step S104).
[0033] Thereafter, when an image is captured by operating the shutter, the lock on the gaze target is released (YES in step S108 to S109), but when the shutter is not operated, the lock is not released and the image capturing conditions of the locked area are controlled again according to the user's eye movement (NO in step S108). Note that in this example, the lock is released when the shutter is operated, but this is not limited to this. The settings may be changed so that the lock is released by another action (for example, the action of looking away from the monitor) instead of the shutter operation. This makes it possible to release the lock without capturing an image.
[0034] Thereafter, when the power is turned off, the operation of the imaging device 1 ends (YES in step S110). If the power is not turned off, the process returns to step S102 (NO in step S110).
[0035] In this way, the imaging device 1 according to the present embodiment can prevent a subject unintended by the user from being selected as a control subject for imaging conditions due to the influence of camera shake by setting the subject gazed at for a predetermined period or more as a control subject for imaging conditions. Also, the imaging device 1 according to the present embodiment can control the imaging conditions of a desired subject without taking the line of sight off the display unit by controlling the imaging conditions according to the movement of the eyes. That is, the imaging device 1 according to the present embodiment 1 can accurately control the imaging conditions of a desired subject without being affected by camera shake.
[0036] In the present embodiment, the case where the lock is released every time the imaging conditions are controlled has been described as an example, but the present invention is not limited to this. The imaging device 1 may be configured so that the lock is not released until the user's line of sight is removed from the display unit 12. A brief description will be given below with reference to FIG. 5.
[0037] FIG. 5 is a flowchart showing another example of the operation of the imaging device 1. 5, after the imaging conditions are controlled, it is determined whether or not the user's line of sight has left the display unit 12 (monitor) (step S208). This determination is made based on, for example, the movement of the user's eyes detected by the detection unit 13.
[0038] When the user's line of sight moves away from the display unit 12, the lock is released (YES in step S208 to S109), but when the user's line of sight does not move away from the display unit 12, the lock is not released and the imaging conditions of the locked area are controlled again in accordance with the movement of the user's eyes (NO in step S208). Therefore, the user can continuously capture images of a desired subject while changing the imaging conditions without moving his / her line of sight away from the desired subject. The continuous imaging may be performed manually by the user or automatically.
[0039] Other operations shown in the flowchart of FIG. 5 are similar to those shown in the flowchart of FIG. 3, and therefore description thereof will be omitted.
[0040] (Hardware configuration of imaging control function of imaging device 1) The imaging control process performed by the imaging device 1 can be realized by a general-purpose computer system, which will be briefly described below with reference to FIG.
[0041] 6 is a block diagram showing an example of a hardware configuration of an imaging control function. The computer 100 includes, for example, a CPU (Central Processing Unit) 101, which is a control device, a RAM (Random Access Memory) 102, and a ROM (Read Only Memory) 103. The computer 100 further includes an IF (Interface) 104, which is an interface with the outside, and an HDD (Hard Disk Drive) 105, which is an example of a non-volatile storage device. Furthermore, the computer 100 may include input devices such as a keyboard and a mouse, and a display device such as a display, as other components not shown.
[0042] The HDD 105 stores an OS (Operating System) (not shown) and an imaging control program 106. The imaging control program 106 is a computer program in which imaging control processing according to the present embodiment is implemented.
[0043] The CPU 101 controls various processes in the computer 100, and access to the RAM 102, ROM 103, IF 104, and HDD 105. In the computer 100, the CPU 101 reads and executes an OS and an imaging control program 106 stored in the HDD 105. In this way, the computer 100 realizes an imaging control function of the imaging device 1 according to this embodiment.
[0044] The invention made by the inventor has been specifically described above based on the embodiments, but it goes without saying that the present invention is not limited to the embodiments already described, and various modifications are possible without departing from the gist of the invention.
[0045] In the above embodiment, the present invention has been described as a hardware configuration, but the present invention is not limited to this. The present invention can also be realized by having a CPU (Central Processing Unit) execute a computer program to perform any process. The above-mentioned program can be stored using various types of non-transitory computer readable media and supplied to a computer. The non-transitory computer readable media includes various types of tangible storage media. Examples of the non-transitory computer readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROM (Read Only Memory), CD-R, CD-R / W, and semiconductor memory (e.g., mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM (Random Access Memory)). The program may also be supplied to a computer by various types of transitory computer readable media. Examples of the transitory computer readable media include electric signals, optical signals, and electromagnetic waves. The temporary computer-readable medium can supply the program to the computer via a wired communication path such as an electric wire or an optical fiber, or via a wireless communication path.
[0046] Although the present invention has been described above with reference to the embodiment, the present invention is not limited to the above. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the invention. [Explanation of symbols]
[0047] 1. Imaging device 11 Imaging unit 12 Display section 13 Detection section 14 Specific part 15 Control section 100 Computers 101 CPU 102 RAM 103 ROM 104 IF 105 HDD 106 Imaging control program 111 Optical system 112 Image sensor 113 Image Processing Unit
Claims
1. an imaging unit that captures an image of a subject and generates image data; A display unit that displays an image represented by the image data; A detection unit that detects the movement of a user's eyes; an identification unit that identifies an area of the image displayed on the display unit that is gazed at by the user for a predetermined period of time or more; a control unit that controls an imaging condition of a subject included in an image of the area specified by the specification unit in accordance with a predetermined eye movement of the user detected by the detection unit; Equipped with The control unit performs focus control of the area specified by the determination unit when the eye movement is an up-down movement, performs zoom control of the area specified by the determination unit when the eye movement is a left-right movement, and performs aperture control of the area specified by the determination unit when the eye movement is a rotational movement. Imaging device.
2. The identification unit is configured to estimate an area gazed at by the user from the eye movement of the user detected by the detection unit. The imaging device according to claim 1 .
3. An image of a subject is captured to generate image data; Displaying an image represented by the image data on a display unit; Detect the user's eye movements, Identifying an area of the image displayed on the display unit that is gazed at by the user for a predetermined period of time or more; controlling an imaging condition of a subject included in an image of the specified area in accordance with a predetermined eye movement of the user; 1. An imaging method, comprising: In the control of the imaging conditions, when the eye movement is an up-down movement, focus control of the identified area is performed, when the eye movement is a left-right movement, zoom control of the identified area is performed, and when the eye movement is a rotational movement, aperture control of the identified area is performed. Imaging method.
4. A process of capturing an image of a subject and generating image data; A process of displaying an image represented by the image data on a display unit; Detecting the user's eye movements; A process of identifying an area of the image displayed on the display unit that is gazed at by the user for a predetermined period of time or more; A process of controlling an image capturing condition of a subject included in an image of the specified area in accordance with a predetermined eye movement of the user; An imaging control program for causing a computer to execute the following: In the process of controlling the imaging conditions, when the eye movement is an up-down movement, focus control of the identified area is performed, when the eye movement is a left-right movement, zoom control of the identified area is performed, and when the eye movement is a rotational movement, aperture control of the identified area is performed. Imaging control program.
Citation Information
Patent Citations
Communication controller
JP1987076954A
Information processing apparatus and program
JP2013238814A
Visual line direction detection device and visual line direction detection method
JP2015022700A
Video monitoring system
JP2015065514A
Fixation position recognition device, imaging device, display device, fixation position recognition method, and program
WO2018016209A1
