Information processing device, information processing method, information processing program

Abstract

Provided are an information processing device, an information processing method, and an information processing program capable of imaging according to an intent of a camera user in response to movement of the gaze of the camera user, while giving the impression that an imaging device understands the intent of the camera user.　Provided is an information processing device comprising a control object position determination unit that determines a control object position on the basis of: reference gaze information including a locus on a display unit of the imaging device; and newest gaze information including the locus of the gaze of the camera user with respect to the display unit during imaging.

Description

Technical Field

[0001] This technology relates to an information processing apparatus, an information processing method, and an information processing program.

Background Art

[0002] Some cameras can perform autofocus control using the photographer's line of sight. Also, a technology has been proposed to switch the autofocus mode between continuous AF (Auto Focus) that continuously focuses on the subject and one-shot AF where the focus position is fixed, depending on whether the pattern of the photographer's line-of-sight trajectory matches the trajectory of the subject's movement (Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In conventional AF using the line of sight, the focus may not be set on the intended subject, or it may take time to focus on the intended subject.

[0005] For example, when focusing using line-of-sight information during still-image shooting with a wide setting of continuous AF, the focus may shift to an unintended subject, or when the line of sight returns to the original position after looking at another position, the focus may not be set and it may take time to achieve focus.

[0006] This technology was developed in light of these considerations, and aims to provide an information processing device, an information processing method, and an information processing program that can take images in accordance with the photographer's intentions while giving the impression that the imaging device is understanding the photographer's intentions in response to the photographer's gaze movements. [Means for solving the problem]

[0007] To solve the above-mentioned problems, the first technology includes a control target position determination unit that determines the control target position based on reference line of sight information including the trajectory on the display unit of the imaging device and the latest line of sight information including the trajectory of the photographer's gaze toward the display unit at the time of shooting. The reference gaze information includes the trajectory of the photographer's gaze toward the display unit during a shoot prior to the time of shooting, and the gaze position, which is a position on the gaze trajectory where the photographer's gaze was aligned for a predetermined period of time or longer. The control target position determination unit determines the gaze position as the control target position if part or all of the reference gaze information and the latest gaze information match, and furthermore, if the photographer's gaze trajectory in the latest gaze information is directed toward the gaze position included in the reference gaze information. It is an information processing device.

[0008] Furthermore, the second technology determines the control target position based on reference line-of-sight information, including the trajectory on the display unit of the imaging device, and the latest line-of-sight information, including the trajectory of the photographer's gaze toward the display unit during shooting. This is an information processing method in which the reference gaze information includes the trajectory of the photographer's gaze toward the display unit during a shoot prior to the time of shooting, and the gaze position, which is a position on the gaze trajectory where the photographer's gaze was aligned for a predetermined period of time or longer. If the reference gaze information and the latest gaze information match in part or all, and furthermore, if the photographer's gaze in the latest gaze information's trajectory is directed toward the gaze position included in the reference gaze information, the gaze position is determined as the control target position. It is an information processing method.

[0009] Furthermore, the third technology determines the controlled target position based on reference line-of-sight information, including the trajectory on the display unit of the imaging device, and the latest line-of-sight information, including the trajectory of the photographer's gaze relative to the display unit during shooting. This is an information processing method in which the reference gaze information includes the trajectory of the photographer's gaze toward the display unit during a shoot prior to the time of shooting, and the gaze position, which is a position on the gaze trajectory where the photographer's gaze was aligned for a predetermined period of time or longer. If the reference gaze information and the latest gaze information match in part or all, and furthermore, if the photographer's gaze in the latest gaze information's trajectory is directed toward the gaze position included in the reference gaze information, the gaze position is determined as the control target position. It is an information processing program that instructs a computer to execute information processing methods. [Brief explanation of the drawing]

[0010] [Figure 1] This is a block diagram showing the configuration of the imaging device 100. [Figure 2] This is a block diagram showing the configuration of the information processing device 200. [Figure 3] This flowchart shows the process for registering reference line-of-sight information. [Figure 4] This is an explanatory diagram of the registration process for reference line-of-sight information. [Figure 5] This is a flowchart showing the process for determining the position of the controlled object. [Figure 6]It is a flowchart showing the control target positioning process in the line-of-sight focus fixation priority control. [Figure 7] It is a flowchart showing the control target positioning process in the line-of-sight focus movement priority control. [Figure 8] It is a flowchart showing the photographing-related control. [Figure 9] It is a diagram showing the first specific example of the line-of-sight focus fixation priority control. [Figure 10] It is a diagram showing the second specific example of the line-of-sight focus fixation priority control. [Figure 11] It is a diagram showing the first specific example where the line-of-sight focus fixation priority control is not performed. [Figure 12] It is a diagram showing the second specific example where the line-of-sight focus fixation priority control is not performed. [Figure 13] It is a diagram showing the third specific example where the line-of-sight focus fixation priority control is not performed. [Figure 14] It is a diagram showing a specific example of the line-of-sight focus movement priority control.

Embodiments for Carrying Out the Invention

[0011] Hereinafter, embodiments of the present technology will be described with reference to the drawings. The description will be made in the following order. <1. Embodiment> [1-1. Configuration of Imaging Device 100] [1-2. Configuration of Information Processing Device 200] [1-3. Processing and Control by Information Processing Device 200] [1-3-1. Registration Process of Line-of-Sight Information] [1-3-2. Control Target Positioning and Photographing-Related Control] [1-3-3. Specific Examples of Control] <2. Variation>

[0012] <1. Embodiment> [1-1. Configuration of Imaging Device 100] Referring to FIG. 1, the configuration of the imaging device 100 will be described. The imaging device 100 includes a control unit 101, an optical imaging system 102, a lens driving driver 103, an imaging element 104, an image signal processing unit 105, a camera processing unit 106, a storage unit 107, an interface 108, an input unit 109, a display unit 110, a microphone 111, a subject recognition unit 112, an AF processing unit 113, a gaze detection unit 114, and an information processing device 200.

[0013] The control unit 101 is composed of a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), etc. The CPU executes various processes according to the programs stored in the ROM and issues commands to control the entire imaging device 100 and each part thereof.

[0014] The optical imaging system 102 includes an imaging lens for condensing light from a subject onto the imaging element 104, a driving mechanism for moving the imaging lens for focusing and zooming, a shutter mechanism, an iris mechanism, etc. These are driven based on control signals from the control unit 101 and the lens driving driver 103. The optical image of the subject obtained through the optical imaging system 102 is formed on the imaging element 104.

[0015] The lens driving driver 103 is composed of, for example, a microcomputer, etc., and performs autofocus so as to focus on a target subject by moving the imaging lens along the optical axis direction by a predetermined amount based on the focus control information supplied from the AF processing unit 113 or the information processing device 200. Also, according to the control from the control unit 101, it controls the operations of the driving mechanism, shutter mechanism, iris mechanism, etc. of the optical imaging system 102. Thereby, adjustments such as exposure adjustment and aperture value (F value) adjustment are made.

[0016] The image sensor 104 converts the incident light from the subject obtained through the imaging lens into an electric charge and outputs an imaging signal. The image sensor 104 then outputs the pixel signal to the image signal processing unit 105. The image sensor 104 can be a CCD (Charge Coupled Device), CMOS (Complementary Metal Oxide Semiconductor), or the like.

[0017] The image sensor 104 has normal pixels such as R (Red), G (Green), and B (Blue), as well as phase-difference detection pixels for phase-difference detection. Phase-difference AF (Auto Focus) can be performed using the phase-difference information output from these phase-difference detection pixels. Note that the phase-difference detection pixels may function only as phase-difference detection pixels and not as normal pixels, or a single pixel may be composed of two independent photodiodes that function for both imaging and phase-difference detection. Note that phase-difference detection may be performed by an AF sensor dedicated to phase-difference detection.

[0018] The image signal processing unit 105 processes the imaging signal output from the image sensor 104, performing sample-and-hold (CDS) processing to maintain a good signal-to-noise ratio (S / N), AGC (Auto Gain Control) processing, and A / D (Analog / Digital) conversion to create an image signal. Furthermore, the image signal processing unit 105 applies recording processing to image signals intended for recording and display processing to image signals intended for display.

[0019] The camera processing unit 106 applies signal processing to the image signal, such as white balance adjustment, color correction, gamma correction, Y / C conversion, and AE (Auto Exposure) processing.

[0020] The storage unit 107 is, for example, a large-capacity storage medium such as a hard disk or flash memory. The video data and image data processed by the image signal processing unit 105 and the camera processing unit 106 are stored in a compressed or uncompressed state according to a predetermined standard. In addition, EXIF ​​(Exchangeable Image File Format), which includes additional information such as information about the stored data, imaging location information indicating the imaging location, and imaging time information indicating the date and time of imaging, is also stored associated with the data.

[0021] Interface 108 is an interface between the system and other devices or the internet. Interface 108 may include wired or wireless communication interfaces. More specifically, wired or wireless communication interfaces may include cellular communication such as 3TTE, Wi-Fi, Bluetooth®, NFC (Near Field Communication), Ethernet®, HDMI® (High-Definition Multimedia Interface), USB (Universal Serial Bus), etc. Furthermore, if the imaging device 100 and the information processing device 200 are connected in hardware, interface 108 may include connection terminals between the devices and buses within the devices (hereinafter, these will also be referred to as internal device interfaces). Furthermore, if the imaging device 100 and the information processing device 200 are implemented in a distributed manner across multiple devices, interface 108 may include different types of interfaces for each device. For example, interface 108 may include both communication interfaces and internal device interfaces.

[0022] The input unit 109 is for the user to give various instructions to the imaging device 100. When the user provides input to the input unit 109, a control signal corresponding to that input is generated and supplied to the control unit 101. The control unit 101 then performs various processing corresponding to that control signal. The input unit 109 can include a shutter button for shutter input, physical buttons for various operations, a touch panel, and a touchscreen integrated with a display unit 110.

[0023] The display unit 110 is a display device such as a display or EVF (Electronic View Finder) that displays through images, which are display image signals that have been processed for display, images / videos that have been processed for recording and stored in the storage unit 107, a GUI (Graphical User Interface), etc.

[0024] Microphone 111 is a sound-collecting device used to record audio when shooting video.

[0025] The subject recognition unit 112 recognizes a specific subject (such as a person's face or an object) from image data generated by the capture using known subject recognition processing. Known subject recognition techniques may include template matching, matching based on the brightness distribution information of the subject, methods based on skin-colored areas or human facial features contained in the image, and methods using artificial intelligence. Furthermore, these methods may be combined to improve recognition accuracy.

[0026] The AF processing unit 113 performs AF detection using phase difference information supplied from the image sensor 104 to determine the focus position and calculate the amount of defocus. Based on the amount of defocus, it generates focus control information indicating where within the field of view to focus on the target subject, and how much the lens drive driver 103 of the imaging device 100 should move the lens. The autofocus processing is then performed by supplying the focus control information to the lens drive driver 103.

[0027] Furthermore, the AF processing unit 113 acquires the photographer's gaze information from the gaze detection unit 114 and performs autofocus control (sometimes referred to as gaze input AF) to automatically focus on the position where the photographer's gaze is aligned on the display unit 110 detected by the gaze detection unit 114. If the control related to shooting performed by the information processing device 200 is AF control, the shooting-related control unit 203 may perform AF control by controlling the AF processing unit 113, or the shooting-related control unit 203 itself may operate as the AF processing unit.

[0028] The gaze detection unit 114 consists of an infrared LED, an infrared-compatible image sensor, etc. It detects the photographer's gaze on the display unit 110 based on an image taken by illuminating the photographer's face with the infrared LED and capturing the photographer's eyeballs with the infrared-compatible image sensor. Based on the position of the pupil relative to the position of the corneal reflection of the eyeball, it can detect the direction of the photographer's gaze and determine which part of the display unit 110 the photographer is fixated on. The gaze detection unit 114 can detect the movement of the photographer's gaze by continuously detecting the photographer's gaze. The gaze detection unit 114 supplies the detected photographer's gaze to the information processing device 200 as coordinate data in a two-dimensional XY plane with a predetermined position on the display surface of the display unit 110 as the origin.

[0029] The imaging device 100 is configured as described above. The imaging device 100 may be a device specializing in camera functions such as a digital camera, SLR camera, camcorder, professional camera, or professional-grade photographic equipment, or it may be a smartphone, tablet, wearable device, or portable game console equipped with camera functions.

[0030] In the above description, the imaging device 100 is equipped with an AF function using on-sensor phase-detection AF, but other AF methods such as phase-detection AF or contrast AF may also be used.

[0031] [1-2. Configuration of the Information Processing Device 200] Next, the configuration of the information processing device 200 will be described with reference to Figure 2. The information processing device 200 is composed of a gaze information management unit 201, a control target position determination unit 202, a shooting-related control unit 203, and a display control unit 204.

[0032] The gaze information management unit 201 analyzes coordinate data indicating the movement of the photographer's gaze relative to the display unit 110, detected by the gaze detection unit 114, to generate gaze information. In predetermined cases, it registers, stores, and manages this gaze information as reference gaze information. The reference gaze information includes the trajectory of the photographer's gaze relative to the display unit 110 in photographs taken prior to the most recent photograph. Furthermore, the reference gaze information includes a gaze position, which is a location on the photographer's gaze trajectory included in the reference gaze information where the photographer's gaze was aligned for a predetermined period of time or longer, and a gaze duration, which is the time the photographer's gaze was aligned at that gaze position. The gaze information includes the gaze trajectory, gaze position, and gaze duration. In addition, the gaze information management unit 201 also stores and manages coordinate data and the latest gaze information. The gaze information management unit 201 itself may be equipped with a storage medium to store the reference gaze information, coordinate data, and latest gaze information, or it may process the data to be stored in the storage unit 107.

[0033] A trajectory is formed by connecting at least two coordinates, indicated by coordinate data, with line segments based on the order in which the coordinate data was detected, and represents the path of the line of sight's movement.

[0034] The gaze position is a position on the trajectory of the photographer's line of sight to the display unit 110 of the imaging device 100 where the gaze is fixed (stared at) for a predetermined period of time or longer.

[0035] The fixation time is the length of time that the photographer focuses on a specific point along the trajectory of their gaze.

[0036] The gaze information management unit 201 may have its own storage medium and store the reference gaze information itself, or it may use the storage unit 107 provided by the imaging device 100 and process the data so that the reference gaze information is stored in the storage unit 107.

[0037] The control target position determination unit 202 determines the control target position, which is the position within the field of view displayed on the display unit 110 that is subject to control by the shooting-related control unit 203.

[0038] The shooting-related control unit 203 performs various shooting-related controls using the control target position determined by the control target position determination unit 202. For example, if the shooting-related control is AF control, the shooting-related control unit 203 may perform AF control by controlling the AF processing unit 113 of the imaging device 100, or the shooting-related control unit 203 itself may operate as the AF processing unit.

[0039] In this embodiment, a first control and a second control will be described as controls that utilize the position of the target object. The first control is a gaze-focus-fixed priority control that keeps the focus fixed on the target object even if the photographer's gaze moves. The second control is a gaze-focus-movement priority control that, when the photographer's gaze moves, adjusts the focus to the target object located at the destination of the gaze.

[0040] In this embodiment, the shooting control that utilizes the gaze by operating the information processing device 200 in the imaging device 100 is referred to as the gaze utilization mode. When taking a picture, the photographer must turn on the gaze utilization mode to operate the information processing device 200, and furthermore, must pre-select and set whether to have the information processing device 200 execute the gaze focus fixed priority mode, which prioritizes gaze focus fixed control, or the gaze focus movement priority mode, which prioritizes gaze focus movement control. Note that the information processing device 200 can also automatically set whether to execute the gaze focus fixed priority mode or the gaze focus movement priority mode.

[0041] The display control unit 204 displays a user interface on the display unit 110 that corresponds to the processing by the shooting-related control unit 203, and presents information regarding the controlled position to the photographer within that user interface. The information regarding the controlled position includes an AF frame that indicates the position where autofocus is set. A specific example of the user interface will be described later.

[0042] The information processing device 200 is configured as described above. The information processing device 200 may also be configured as a dedicated device using hardware that has the function. If the information processing device 200 is configured as hardware, the information processing device 200 may include a gaze detection unit 114.

[0043] Furthermore, the information processing device 200 may not only be configured as a dedicated device, but may also be configured by a program, and the imaging device 100 may function as the information processing device 200 by executing that program. The program may be pre-installed on the imaging device 100, or it may be distributed via download or storage media so that the user can install it on the imaging device 100 themselves.

[0044] [1-3. Processing and control by the information processing device 200] [1-3-1. Registration process for eye-tracking information] Next, referring to Figures 3 and 4, we will explain the process of registering gaze information, which shows the movement of the photographer's gaze, as reference gaze information. Figure 4 shows the through image displayed on the display unit 110 of the imaging device 100 and the photographer's gaze toward the display unit 110.

[0045] As shown in Figure 3, first, in step S101, if the gaze-based mode in which the information processing device 200 operates is turned on in the imaging device 100, the process proceeds to step S102 (Yes in step S101). Next, in step S102, the gaze information management unit 201 queries the gaze detection unit 114 to obtain coordinate data indicating the movement of the photographer's gaze detected by the gaze detection unit 114. The gaze detection unit 114 detects coordinate data by continuously detecting the photographer's gaze. The gaze detection unit 114 may acquire coordinate data at a predetermined period and actively supply the coordinate data to the gaze information management unit 201 regardless of whether an query is made.

[0046] Next, in step S103, the gaze information management unit 201 analyzes the coordinate data. By analyzing the coordinate data, the trajectory of the gaze, the gaze position, and the gaze duration can be obtained. While the gaze detection unit 114 and the information processing device 200 are operating, coordinate data indicating the movement of the photographer's gaze is constantly detected, and the gaze information management unit 201 acquires gaze information by analyzing the coordinate data.

[0047] The system detects a gaze position as a position on the trajectory of the gaze information where the gaze is directed for a predetermined time, which is a first threshold, and further detects the length of time the gaze is directed at that gaze position as the gaze time. Since the gaze position is determined by the gaze time relative to a position on the trajectory, there may be one gaze position or multiple gaze positions on a single trajectory. If there are multiple gaze positions, the system may obtain a comparison result of the gaze times of the multiple gaze positions. The first threshold is a predetermined length of time for detecting a gaze position. The first threshold may be set by default in the information processing device 200, set by the photographer, pre-set by default, or automatically set according to the shooting settings, etc.

[0048] Next, in step S104, the gaze information management unit 201 stores the gaze information and coordinate data. The gaze information management unit 201 itself may be equipped with a storage medium and store the gaze information and coordinate data, or it may be processed to cause the storage unit 107 to store the gaze information and coordinate data.

[0049] Next, in step S105, the gaze information management unit 201 checks whether the gaze information stored in step S104, namely the trajectory, gaze position, and gaze time, has been repeated with a predetermined frequency. The predetermined frequency is, for example, twice, but is not limited to a specific frequency. If it has been repeated with the predetermined frequency, the process proceeds to step S106 (Yes in step S105). The predetermined frequency of repetition may be continuous or discontinuous, and the trajectory may be a reciprocal movement or a non-reciprocal movement.

[0050] Then, in step S106, the gaze information management unit 201 registers and stores gaze information, which consists of the trajectory of the gaze, the gaze position, and the gaze duration repeated at a predetermined frequency, as reference gaze information. However, coordinate data does not need to be stored together with the reference gaze information. In the following description, the process of registering gaze information as reference gaze information includes the process of storing that reference gaze information.

[0051] Reference gaze information may be stored in the storage unit 107 of the imaging device 100 through storage control by the gaze information management unit 201, or the information processing device 200 may be equipped with a storage unit for such storage. Any storage medium or storage method is acceptable as long as it can be used in processing after being stored in either the imaging device 100 or the information processing device 200.

[0052] In the case of non-reciprocating motion, for example, if the photographer moves their gaze toward the display unit 110 of the imaging device 100, then moves their gaze away from the field of view displayed on the display unit 110, and then returns their gaze to the field of view and performs the same movement again, the gaze information is registered as reference gaze information.

[0053] On the other hand, if the gaze information in step S105— namely the trajectory, gaze position, and gaze duration—is not repeated at a predetermined frequency, the process returns to step S101 (step S105 No.).

[0054] Then, steps S101 to S106 are repeated until it is confirmed in step S101 that the gaze utilization mode is off, and the reference gaze information is stored.

[0055] The gaze information management unit 201 may, after registering reference gaze information, register the different gaze information as new reference gaze information if the gaze usage mode remains on and the photographer repeats gaze movements indicated by different gaze information at a predetermined frequency, while retaining the already registered reference gaze information. In this case, the number of reference gaze information will increase. However, even if the gaze information is different, it may be registered as new reference gaze information only if certain conditions are met, such as when the controlled target position is the same or when part of the gaze information is the same. Alternatively, the reference gaze information may be deleted and the different gaze information may be registered as new reference gaze information.

[0056] This is how the eye-tracking information is registered. The photographer may register the eye-tracking information as reference eye-tracking information in advance before taking the picture, or they may have the eye-tracking information taken during the picture registered as reference eye-tracking information.

[0057] The eye-tracking information management unit 201 always performs the process shown in Figure 3 when acquiring reference eye-tracking information from coordinate data indicating the movement of the photographer's gaze. Therefore, even after registering reference eye-tracking information, eye-tracking information is acquired from new coordinate data obtained from the eye-tracking detection unit 114, and when the trajectory, gaze position, and gaze duration of that eye-tracking information are repeated at a predetermined frequency, it is registered as reference eye-tracking information. In other words, even after the registration of reference eye-tracking information, new reference eye-tracking information will be registered. When registering new reference eye-tracking information, the previous reference eye-tracking information may be retained as history or deleted.

[0058] Figure 4 shows a specific example of reference gaze information registered by the gaze information management unit 201 described in Figure 3. Figure 4 shows the through image displayed on the display unit 110 of the imaging device 100 and the photographer's gaze toward the display unit 110.

[0059] First, let's assume that the photographer's line of sight, as shown in Figure 4A, is focused on a person's face at position A for a first threshold or more, then moves to position B as shown in Figure 4B, then to position C where the animal is located as shown in Figure 4C, then moves again to position B as shown in Figure 4B, and finally returns to position A as shown in Figure 4A, in a back-and-forth motion.

[0060] In the through-image shown in Figure 4, the rectangular frames represent AF frames. Solid AF frames indicate that the focus is set to that position by eye-input AF, while dashed AF frames indicate that the AF frame is moving and the focus is not set to that position.

[0061] The focus is initially set to position A, where the gaze is directed, and then to position C, where the gaze is directed. Therefore, the AF frame displayed on the display unit 110 is displayed over position A where the person's face is located, then passes through position B, and is displayed over position C where the animal is located. Finally, the AF frame passes through position B and is displayed again over position A where the person's face is located.

[0062] If this eye movement is repeated at a predetermined frequency, the eye-tracking information management unit 201 registers the eye-tracking information obtained from the coordinate data indicating that eye movement as reference eye-tracking information. Here, it is assumed that the eye gaze is fixed on position A by a first threshold or more, and that position A is registered as the gaze position in the reference eye-tracking information.

[0063] In the example in Figure 4, position A, which is the point of fixation, is the endpoint of the line of sight in the trajectory of the reference line of sight information. However, this is merely an example, and the point of fixation is not necessarily limited to the endpoints of the trajectory. Furthermore, the trajectory is not necessarily straight; it can be curved or meandering. Moreover, the trajectory is not limited to returning to the starting point of movement as shown in Figure 4.

[0064] [1-3-2. Determination of the controlled object's position and image capture-related control] Next, with reference to Figures 5 to 8, the determination of the control target position by the control target position determination unit 202 and the shooting-related control by the shooting-related control unit 203 will be explained. These are performed during shooting after the registration process of the reference line-of-sight information shown in Figures 3 and 4.

[0065] First, the determination of the controlled object position will be explained with reference to Figures 5, 6, and 7. First, in step S201 of the flowchart in Figure 5, if the line-of-sight utilization mode in which the information processing device 200 operates is turned on in the imaging device 100, the process proceeds to step S202 (Yes in step S201).

[0066] Next, in step S202, the control target position determination unit 202 obtains gaze information (referred to as the latest gaze information) by querying the gaze information management unit 201, which has analyzed coordinate data showing the latest movement of the photographer's gaze at the time of shooting after the registration of the reference gaze information. The gaze information management unit 201 supplies the latest gaze information in response to the inquiry from the control target position determination unit 202 and also stores it as history. Note that the latest gaze information is not stored as reference gaze information.

[0067] Thus, even when shooting after registering reference gaze information, the gaze information management unit 201 analyzes the coordinate data to acquire gaze information and stores it as history, and the control target position determination unit 202 acquires the latest gaze information from the gaze information management unit 201.

[0068] The latest gaze information is used for processing by the control target position determination unit 202, as well as for processing by the gaze information management unit 201. If it is repeated at a predetermined frequency, it is registered as reference gaze information. This reference gaze information is then used for the control target position determination process in subsequent imaging.

[0069] Next, in step S203, the control target position determination unit 202 reads and acquires the reference gaze information stored by the gaze information management unit 201 in the process shown in Figure 3.

[0070] Next, in step S204, if the operating mode of the information processing device 200 is the gaze focus fixed priority mode, the process proceeds to step S205 (Yes in step S204).

[0071] Next, in step S205, the control target position determination unit 202 compares and determines whether at least a portion of the reference gaze information matches the latest gaze information. In this comparison, in addition to comparing whether the trajectories of the reference gaze information and the latest gaze information match, it also compares whether the gaze positions match and whether the gaze times match. If there are multiple gaze positions, it may also compare whether the relative lengths of the gaze times for each gaze position match.

[0072] It should be noted that while it is stated that at least a portion of the reference gaze information and the latest gaze information must match, this does not exclude cases where all of them match. Cases where all of them match may be processed in the same way as cases where only a portion matches.

[0073] Furthermore, when comparing the trajectory of the reference gaze information with the trajectory of the latest gaze information, agreement may be included not only when the trajectories completely overlap and match, but also when the difference in angle, length, etc., of the trajectories is within a predetermined tolerance. In addition, when comparing the gaze position of the reference gaze information with the gaze position of the latest gaze information, agreement may be included when the difference in gaze position is within a predetermined tolerance. Moreover, when comparing the gaze time of the reference gaze information with the gaze time of the latest gaze information, agreement may be included when the difference in gaze time is within a predetermined tolerance.

[0074] If a portion of the reference gaze information and the latest gaze information match, the process proceeds to step S206 (Yes in step S205). Here, "partially matching" means, for example, that the trajectory of the reference gaze information and the trajectory of the latest gaze information match to a predetermined extent, the gaze position of the latest gaze information is included within a predetermined tolerance range that includes the gaze position of the reference gaze information, and the gaze time of the latest gaze information is included within a tolerance range set by upper and lower threshold limits for the gaze time of the reference gaze information.

[0075] Then, in step S206, the control target position determination unit 202 performs the control target position determination process in the gaze focus fixed priority mode. The control target position determination process in the gaze focus fixed priority mode will be described later with reference to Figure 6.

[0076] On the other hand, if the reference gaze information and the latest gaze information do not match, the process proceeds to step S207 (No. of step S205), and the control target position determination unit 202 determines the control target position based on the latest gaze information. Note that in the first cycle of the process, if the latest gaze information is fixated on only one coordinate position (such as the starting point of the trajectory) and that coordinate position is fixed on below the first threshold, then the latest gaze information and the reference gaze information do not partially match, so the process in Figure 5 proceeds from step S205 to step S207 (No. of step S205). Then, in step S207, the coordinate position indicated by the latest gaze information is determined as the control target position. Therefore, when the process in Figure 5 is started, the control target position is always determined first in step S207.

[0077] The explanation returns to step S204. If the operating mode of the information processing device 200 in step S204 is not the gaze focus fixed priority mode, the process proceeds to step S208 (No in step S204). If the operating mode is not the gaze focus fixed priority mode, it means that the operating mode is the gaze focus movement priority mode. Step S204 may also be a process to confirm whether the operating mode is the gaze focus movement priority mode. In that case, if Yes, the process proceeds to step S208, and if No, the process proceeds to step S205.

[0078] Next, in step S208, a comparison is performed to determine whether a portion of the reference line of sight information and the latest line of sight information match. The comparison method is the same as the method described in step S205.

[0079] If a portion of the reference gaze information matches the latest gaze information, the process proceeds to step S209 (Yes in step S208). In step S209, the control target position determination unit 202 performs the control target position determination process in gaze focus movement priority mode. The control target position determination process in gaze focus movement priority mode will be described later with reference to Figure 7.

[0080] If the reference line-of-sight information and the latest line-of-sight information do not match in part, the process proceeds to step S207 (No. of step S208), and the controlled target position determination unit 202 determines the controlled target position based on the latest line-of-sight information.

[0081] The process shown in Figure 5 is carried out in the manner described above.

[0082] Next, referring to the flowchart in Figure 6, the process of determining the position of the controlled object in the gaze focus-fixed priority mode by the controlled object position determination unit 202 will be explained. This process corresponds to step S206 in the flowchart in Figure 5. The gaze focus-fixed priority mode keeps the focus fixed at the gaze position on the trajectory of the reference gaze information as long as a portion of the reference gaze information and the latest gaze information match.

[0083] In the explanation below, "the coordinate position indicated by the latest gaze information" refers to the latest position where the photographer's gaze is aligned along the trajectory of the gaze included in the latest gaze information.

[0084] First, in step S301, it is determined whether the coordinate position indicated by the latest gaze information has been gazed upon for a time equal to or greater than the second threshold. If it has not been gazed upon for a time equal to or greater than the second threshold, the process proceeds to step S302 (No. of step S301).

[0085] Then, in step S302, it is determined whether the current controlled target position is the gaze position of the reference gaze information and whether the gaze time for that gaze position is greater than or equal to the second threshold. If it is greater than or equal to the second threshold, the process proceeds to step S303 (Yes in step S302). Then, in S303, the controlled target position is determined to be the same as the current position (the controlled target position is not changed).

[0086] On the other hand, if in step S301 the coordinate position indicated by the latest gaze information has been gazed upon for a time equal to or greater than the second threshold, or if in step S302 the current controlled target position is the gaze position of the reference gaze information and the gaze time for that gaze position is less than the second threshold, the process proceeds to step S304 (Yes in step S301 or No in step S302). Then, in step S304, the controlled target position determination unit 202 determines the coordinate position indicated by the latest gaze information on the trajectory of the reference gaze information as the controlled target position. The information of the determined controlled target position is stored either by the information processing device 200 itself or by storing it in the storage unit 107 of the imaging device 100.

[0087] Next, referring to the flowchart in Figure 7, the process of determining the control target position in the eye-tracking focus movement priority mode by the control target position determination unit 202 will be explained. This process corresponds to step S209 in the flowchart in Figure 5. The eye-tracking focus movement priority mode anticipates the movement of the photographer's gaze during shooting and adjusts the focus to the control target position.

[0088] First, in step S401, if the trajectory of the reference gaze information and the trajectory of the latest gaze information match by a predetermined percentage or more, the process proceeds to step S402 (Yes in step S401).

[0089] Next, in step S402, if the gaze is directed towards the fixation position on the trajectory of the reference gaze information in the trajectory of the latest gaze information, the process proceeds to step S403 (Yes in step S402).

[0090] Next, in step S403, if there are multiple gaze positions on the trajectory of the reference line of sight information, the process proceeds to step S404 (Yes in step S403).

[0091] Next, in step S404, if the gaze time for the gaze position on the trajectory of the reference gaze information, which was determined in step S402 to be where the photographer's gaze was directed, is greater than or equal to the second threshold, the process proceeds to step S405 (Yes in step S404). This gaze time is the gaze time included in the reference gaze information.

[0092] Then, in step S405, the control target position determination unit 202 determines the gaze position to which the gaze is directed in the trajectory of the latest gaze information as the control target position. The information of the determined control target position is stored either by the information processing device 200 itself or by storing it in the storage unit 107 of the imaging device 100.

[0093] If, in step S401, the trajectory of the reference gaze information and the trajectory of the latest gaze information do not match by a predetermined percentage or more (No. in step S401), the process proceeds to step S406. In step S406, the control target position determination unit 202 determines the control target position based on the latest gaze information. The same applies if, in step S402, the movement of the gaze in the latest gaze information is not directed toward the gaze position (No. in step S402), or if, in step S404, the gaze time toward the gaze position on the trajectory of the reference gaze information, which the photographer's gaze is deemed to be directed toward, is not equal to or greater than the second threshold (No. in step S403).

[0094] Next, the processing performed by the image capture control unit 203 will be explained with reference to the flowchart in Figure 8. The processing performed by the image capture control unit 203 is carried out after the control target position has been determined by the control target position determination unit 202.

[0095] First, in step S501, it is checked whether the control function is enabled. If the control function of the information processing device 200 is enabled, the process proceeds to step S502 (Yes in step S501).

[0096] Next, in step S502, if the gaze-based mode in which the information processing device 200 operates is turned on in the imaging device 100, the process proceeds to step S503 (Yes in step S502). Next, in step S503, the imaging-related control unit 203 reads out the control target position determined by the control target position determination unit 202.

[0097] Next, in step S504, the shooting-related control unit 203 performs control using the control target position. On the other hand, if the gaze-based mode is not turned on in step S502, the process proceeds to step S504 (No. of step S501), and control is performed without using the control target position.

[0098] In this embodiment, the first and second controls, which utilize the control target position by the shooting-related control unit 203, will be described. The first control is a gaze-focus-fixed priority control that keeps the focus fixed on the control target position even if the photographer's gaze moves. The second control is a gaze-focus-movement priority control that, when the photographer's gaze moves, adjusts the focus to the control target position located at the destination of the gaze.

[0099] [1-3-3. Specific Examples of Control] Next, with reference to Figures 9 to 14, the determination of the control target position by the control target position determination unit 202 and the control by the shooting-related control unit 203 will be explained based on specific examples. Figures 9 to 14 show the through image displayed on the display unit 110 of the imaging device 100 and the photographer's line of sight to the display unit 110 during shooting. As a prerequisite, it is assumed that the line of sight of the photographer to the display unit 110 is pre-stored as reference line of sight information by the line of sight information management unit 201.

[0100] First, let's explain the first example of the first control with reference to Figure 9. The first control is a control in the gaze focus-fixing priority mode, which keeps the focus fixed to the control target position. Here, as described above in Figure 4, the trajectory of the reference gaze information is the trajectory in which the gaze moves in the order of position A, position B, position C, position B, position A, and positions A and C are the gaze positions.

[0101] Then, assume that the photographer's gaze moves from position A shown in Figure 9A to position B shown in Figure 9B during the shooting after the registration of the reference gaze information. This movement of the photographer's gaze during the shooting after the registration of the reference gaze information is the latest gaze information. If the trajectory of the gaze in the latest gaze information matches a part of the trajectory of the gaze in the reference gaze information, the process in Figure 5 proceeds from step S205 to step S206, and the control target position determination process in Figure 6 is performed. Assume that position A is the gaze position in the latest gaze information.

[0102] In the latest gaze information, at the point when the gaze moves from position A to position B, the coordinate position indicated by the latest gaze information is position B. Even if the gaze moves from position A to position B, if position B, which is the coordinate position indicated by the latest gaze information, has not been gazed at for a time equal to or greater than the first threshold, the process in Figure 6 proceeds from step S301 to step S302 (No in step S301). Then, in step S302, it is determined whether position A, which is the current controlled target position, is the gaze position of the reference gaze information, and whether the gaze time for that gaze position is equal to or greater than the second threshold. If the gaze time in the reference gaze information for position A is equal to or greater than the second threshold, the process proceeds to step S303 (Yes in step S302), and the controlled target position determination unit 202 determines the controlled target position to be the same as the current one (the controlled target position is not changed). Therefore, if the controlled target position has already been determined to be position A before the photographer's gaze moves to position B, the controlled target position remains at position A. Once position A, which is the gaze position, is determined to be the control target position, the shooting-related control unit 203 performs gaze focus fixing priority control so that it continues to focus on position A, the control target position, even if the photographer's gaze moves from position A to position B.

[0103] If, in the latest gaze information, the gaze has moved from position A to position B, and the trajectory of the gaze in the latest gaze information does not match a part of the trajectory of the gaze in the reference gaze information, or if the gaze is fixed on position B for a period of time equal to or greater than the second threshold, or if the gaze time in the reference gaze information for position A is less than the second threshold, the coordinate position indicated by the latest gaze information is determined to be the control target position (step S207 or step S304).

[0104] Next, a second example of the first control will be explained with reference to Figure 10. As shown in Figure 10A, assume that the photographer's gaze in the photograph taken after the registration of the reference gaze information moves from position A to position B shown in Figure 10B, then to position C shown in Figure 10C, and then back to position B shown in Figure 10B. This movement of the photographer's gaze in the photograph taken after the registration of the reference gaze information is the latest gaze information.

[0105] First, referring to Figure 10, we will explain an example of maintaining focus on position A, where the gaze time for position A is greater than or equal to the second threshold in the reference gaze information, and the gaze time for position C is less than the second threshold.

[0106] First, let's explain the case where the gaze moves from position A to position B in the latest gaze information. In this case, if the reference gaze information and the latest gaze information partially match, and the gaze is fixed on position B, which is the coordinate position indicated by the latest gaze information, for less than the first threshold, the process proceeds from step S205 to step S206 (Yes in step S205). Since the gaze has not been fixed on position B for a time greater than the second threshold, the process proceeds further from step S301 to step S302 (No in step S301). Next, if position A, which is the current target of control, is the fixation position in the reference gaze information, and the fixation time for position A in the reference gaze information is greater than or equal to the second threshold, the process proceeds from step S302 to step S303 (Yes in step S302). Then, in step S303, the target of control position determination unit 202 determines position A as the target of control, and the shooting-related control unit 203 controls the camera to continue fixing the focus on position A.

[0107] Next, we will explain the case where the gaze moves from position A to position B, and then from position B to position C in the latest gaze information. In this case, if the reference gaze information and the latest gaze information partially match, and the gaze is fixed on position C, which is the coordinate position indicated by the latest gaze information, for less than the second threshold, the process proceeds from step S205 to step S206 (Yes in step S205), and then from step S301 to step S302 (No in step S301). Next, if position A, which is the current target of control, is the fixation position in the reference gaze information, and the fixation time for position A in the reference gaze information is greater than or equal to the second threshold, the process proceeds from step S302 to step S303 (Yes in step S302). Then, in step S303, the target of control position determination unit 202 determines position A as the target of control, and the shooting-related control unit 203 controls the system to keep the focus fixed on position A.

[0108] Next, we will explain the case where the gaze moves from position A to position B, from position B to position C, and then from position C back to position B in the latest gaze information. In this case, if the reference gaze information and the latest gaze information partially match, and the gaze is fixed on position B, which is the coordinate position indicated by the latest gaze information, for less than the first threshold, the process proceeds from step S205 to step S206 (Yes in step S205). Since the gaze has not been fixed for a time greater than the second threshold, the process proceeds further from step S301 to step S302 (No in step S301). Next, if position A, which is the current target of control, is the fixation position in the reference gaze information, and the fixation time for position A in the reference gaze information is greater than or equal to the second threshold, the process proceeds from step S302 to step S303 (Yes in step S302). Then, in step S303, the target of control position determination unit 202 determines position A as the target of control, and the shooting-related control unit 203 controls the system to keep the focus fixed on position A.

[0109] Next, we will explain the case where the reference gaze information and the latest gaze information do not partially match. The case where the reference gaze information and the latest gaze information do not partially match is when the gaze trajectory in the latest gaze information does not match the gaze trajectory in the reference gaze information, or when the gaze trajectory matches, but the latest gaze information shows that position B was gazed at for a first threshold or position C was gazed at for a second threshold or more. In this case, the process proceeds from step S205 to step S207 (step S205 no.), and in step S207, the control target position determination unit 202 determines the coordinate position indicated by the latest gaze information (position B, position C, or any other coordinate position) as the control target position. Then, the shooting-related control unit 203 releases the focus lock at position A and controls the camera to focus on the new coordinate position which is the control target position.

[0110] Next, as an example of a change in the focus fixed position from position A to position C, we will explain the case where the gaze time for positions A and C in the reference gaze information is greater than or equal to the second threshold.

[0111] If, in the latest gaze information, the gaze moves from position A to position B, and the reference gaze information and the latest gaze information partially coincide, and the gaze is fixed on position B, which is the coordinate position indicated by the latest gaze information, below the first threshold, then it is the same as in the example above where the focus remains fixed on position A, but the gaze moves from position A to position B.

[0112] Furthermore, if the gaze moves from position A to position B, and then from position B to position C, and the reference gaze information and the latest gaze information partially coincide, and the gaze is fixed on position C, which is the coordinate position indicated by the latest gaze information, below the second threshold, then it is the same as in the example above where the gaze moves from position A to position B, and then from position B to position C, while maintaining focus on position A.

[0113] Furthermore, if the gaze moves from position A to position B in the latest gaze information, or from position A to position B, and then from position B to position C, and if the reference gaze information and the latest gaze information do not partially match, the process proceeds from step S205 to step S207 (step S205 No.). Next, in step S207, the control target position determination unit 202 determines the coordinate position indicated by the latest gaze information as the control target position. Then, the shooting-related control unit 203 releases the focus lock at position A and controls the system to focus on the new coordinate position which is the control target position.

[0114] Next, referring to Figure 10, we will explain the case where the focus fixed position changes from position A to position C. First, if the latest gaze information and the reference gaze information partially match, and in the latest gaze information the gaze moves from position A to position B, and then from position B to position C, and fixes on position C for more than the second threshold, the process proceeds from step S301 to step S304 (Yes in step S301). Then, in step S304, the control target position determination unit 202 determines position C as the control target position, and the shooting-related control unit 203 controls the system to release the fixation of position A and focus on position C.

[0115] Next, we will explain the processing after the gaze moves from position A to position B, and from position B to position C in the latest gaze information, and the focus fixed position changes from position A to position C. In the latest gaze information, the gaze moves from position C to position B, and the latest gaze information and the reference gaze information partially match (partial match means that position B is being gazed at for less than the first threshold), and if position B, which is the coordinate indicated by the latest gaze information, is being gazed at for less than the first threshold, then position B is not being gazed at for more than the second threshold, so the process proceeds from step S205 to step S206 (Yes in step S205), and the process proceeds from step S301 to step S302 (No in step S301). Next, if position C, which is the current controlled target position, is the gaze position in the reference gaze information, and the gaze time for position C, which is the gaze position in the reference gaze information, is greater than or equal to the second threshold, the process proceeds from step S302 to step S303 (Yes in step S302). Then, in step S303, the control target position determination unit 202 determines position C as the control target position, and the shooting-related control unit 203 controls the system to keep the focus fixed on position C.

[0116] Next, we will explain the case where the reference gaze information and the latest gaze information do not partially match. This includes cases where the gaze trajectory in the latest gaze information does not match the gaze trajectory in the reference gaze information, and cases where the gaze trajectories match, but the latest gaze information shows that position B is being gazed at above the first threshold. In this case, the process proceeds from step S205 to step S207 (step S205 No.), and in step S207, the control target position determination unit 202 determines the coordinate position indicated by the latest gaze information as the control target position. Then, the shooting-related control unit 203 releases the focus lock at position C and controls the camera to focus on the new coordinate position which is the control target position.

[0117] This prevents the focus from shifting to a different position against the photographer's intention. Furthermore, it gives the photographer the impression that the imaging device 100 understands their intentions. This gaze-focus priority control is particularly effective in video recording, where the focus often needs to remain fixed at the same position. However, this control can also be used in still image shooting.

[0118] Furthermore, if the trajectory of the latest gaze information after registration of the reference gaze information matches the reference gaze information trajectory only in a certain part of the way, without including the starting point, gaze focus lock priority control will not be performed. For example, if the trajectory of the reference gaze information is position A, position B, position C, position B, position A, and the trajectory of the latest gaze information does not start from position A, but matches only from position B to position C.

[0119] Next, referring to Figures 11 to 13, we will explain an example in which the gaze focus lock priority control is not performed because the trajectory of the latest gaze information deviates from the trajectory of the reference gaze information. Figures 11 to 13 show the through image displayed on the display unit 110 of the imaging device 100 and the photographer's gaze toward the display unit 110 during shooting. Here, as described above in Figure 4, the trajectory of the reference gaze information is the trajectory in which the gaze moves in the order of position A, position B, position C, position B, position A, with positions A and C being the gaze positions, and the gaze time at position A is greater than or equal to the second threshold, and the gaze time at position C is less than the second threshold.

[0120] As shown in Figure 11A, after registering the reference gaze information, the latest gaze information shows that the photographer's gaze movement aligns with position A and the focus is set to position A. Then, as shown in Figure 11B, the gaze moves to position B, then to position C as shown in Figure 11C, and then to position D as shown in Figure 11D. In this case, because the gaze has moved to position D, the reference gaze information in Figure 4 and the latest gaze information no longer match. Therefore, in the process shown in Figure 5, the process proceeds from step S205 to step S207 (step S205 No.), and the control target position determination unit 202 determines position D, which is the position of the latest gaze, as the control target position. The shooting-related control unit 203 then controls the camera to focus on position D, which is a new control target position that is not on the trajectory of the reference gaze information. Since position D is not on the trajectory of the reference gaze information, the camera exits the gaze focus fixed priority control.

[0121] In the example shown in Figure 11, when the photographer's gaze moves to position D in the latest gaze information after the registration of the reference gaze information, gaze focus lock priority control is not performed, and control is performed to focus on position D. However, until the gaze moves to position D, gaze focus lock priority control is performed to keep the focus on position A.

[0122] Furthermore, as shown in Figure 12A, after registering the reference gaze information, the latest gaze information shows that the photographer's gaze movement aligns with position A and the focus is set to position A. Then, as shown in Figure 12B, the gaze moves to position E. In this case, because the gaze is moving towards position E, the reference gaze information in Figure 4 and the latest gaze information no longer match. Therefore, in the process shown in Figure 5, the process proceeds from step S205 to step S207 (step S205 no.), and the control target position determination unit 202 determines position E, which is the position of the latest gaze, as the control target position. The shooting-related control unit 203 then controls the camera to focus on position E, which is a new control target position that is not on the trajectory of the reference gaze information. Since the camera is not on the trajectory of the reference gaze information when it moves towards position E, it exits the focus lock operation.

[0123] Furthermore, as shown in Figure 13A, after registering the reference gaze information, the latest gaze information shows that the photographer's gaze movement aligns with position A and the focus is set to position A. Then, as shown in Figure 13B, suppose the photographer's gaze moves to position G in order to change the subject from the person on the right to the person on the left. In this case, because the photographer has moved towards position G, the reference gaze information and the latest gaze information no longer match. Therefore, in the process shown in Figure 5, the process proceeds from step S205 to step S207 (step S205 No.), and the control target position determination unit 202 determines position G, which is the position of the latest gaze, as the control target position. The shooting-related control unit 203 then controls the camera to focus on position G, which is the new control target position and is not on the trajectory of the reference gaze information. Since the camera is no longer on the trajectory of the reference gaze information when it moves towards position G, the camera exits the focus lock operation.

[0124] Next, the second control will be explained with reference to Figure 14. The second control is a control in the gaze focus movement priority mode, which anticipates the movement of the photographer's gaze and moves the focus to the target position. Here, as described above in Figure 4, the trajectory of the reference gaze information is the trajectory in which the gaze moves in the order of position A, position B, position C, position B, position A, and positions A and C are assumed to be the gaze positions.

[0125] As shown in Figure 14, in the latest gaze information after the registration of the reference gaze information, the photographer's gaze movement is assumed to be from position A to position B shown in Figure 14B, then to position C shown in Figure 14C, and then back to position B shown in Figure 14B.

[0126] First, if the latest gaze information shows that the movement of the gaze is from position A through position B towards position C, and the reference gaze information and the latest gaze information match by a predetermined ratio, the process in Figure 7 proceeds from step S401 to step S402 (Yes in step S401). Next, since the movement of the gaze in the latest gaze information is toward position C, which is the gaze position of the reference gaze information, the process proceeds from step S402 to step S403 (Yes in step S402). Next, since there are multiple gaze positions at positions A and C in the reference gaze information, the process proceeds from step S403 to step S404 (Yes in step S403). Then, if the gaze time at position C, which is the gaze position toward in the latest gaze information, is greater than or equal to the second threshold, the process proceeds to step S405 (Yes in step S404), and position C, toward which the gaze is toward in the latest gaze information, is determined to be the control target position. When position C, which is the gaze position, is determined to be the control target position, the shooting-related control unit 203 performs gaze focus movement priority control to move the focus to position C, which is the control target position, before the photographer's gaze reaches position C, and focuses on position C.

[0127] Subsequently, if the latest gaze information shows that the movement of the gaze is from position C through position B towards position A, and the reference gaze information and the latest gaze information match by a predetermined ratio, the process in Figure 7 proceeds from step S401 to step S402 (Yes in step S401). Since the movement of the gaze in the latest gaze information is toward position A, which is the gaze position of the reference gaze information, the process proceeds from step S402 to step S403 (Yes in step S402). Since there are multiple gaze positions at positions A and C in the reference gaze information, the process proceeds from step S403 to step S404 (Yes in step S403). Then, if the gaze time at position A, which is the gaze position toward in the latest gaze information, is greater than or equal to the second threshold, the process proceeds to step S405 (Yes in step S404), and position A, toward which the gaze is toward in the latest gaze information, is determined to be the control target position. When position A, which is the gaze position, is determined to be the control target position, the shooting-related control unit 203 performs gaze focus movement priority control to move the focus to position A, which is the control target position, before the photographer's gaze reaches position A, and focuses on position A.

[0128] Furthermore, if the latest gaze information does not match the reference gaze information by a predetermined percentage when the gaze movement is from position A to position B and then to position C (No. in step S401), or if the gaze time at position C, which is the gaze position to which the gaze is directed in the latest gaze information, is not equal to or greater than the second threshold (No. in step S404), the process determines the controlled target position based on the latest gaze information in step S406.

[0129] Furthermore, if the latest gaze information does not match the reference gaze information by a predetermined percentage when the gaze movement is from position C to position A via position B, or if the gaze time at position C, which is the gaze position to which the gaze is directed in the latest gaze information, is not equal to or greater than the second threshold (No. in step S404), the process determines the control target position based on the latest gaze information in step S406.

[0130] In this way, the camera can anticipate the photographer's intentions and focus on the optimal position. Because the camera anticipates the photographer's intentions and focuses on position A before the photographer's gaze aligns with position A, the photographer can take a picture immediately. Furthermore, the photographer can be given the impression that the imaging device 100 is understanding their intentions. Eye-tracking focus shift priority control is particularly effective in still image shooting, where the focus position often changes. However, this control may also be used in video shooting.

[0131] The photographer may choose whether the shooting-related control unit 203 prioritizes eye-tracking focus locking or eye-tracking focus movement, or the information processing device 200 may automatically set which control to perform based on the operating mode of the imaging device 100. For example, the system may automatically set eye-tracking focus locking priority control to be performed in video shooting mode and eye-tracking focus movement priority control to be performed in still image shooting mode.

[0132] Alternatively, the system may automatically set which control to use based on the autofocus mode of the imaging device 100. For example, it may be set to prioritize eye-tracking focus movement in single AF mode and eye-tracking focus lock priority in continuous AF mode.

[0133] Single AF is an autofocus mode where the autofocus works when the shutter button is half-pressed, and once focus is achieved, the focus is locked in. This mode is suitable for shooting stationary subjects such as landscapes and snapshots. Continuous AF, on the other hand, is a mode where the autofocus works as long as the shutter button is half-pressed, maintaining focus. This mode is suitable for shooting moving subjects such as sports and railway photography.

[0134] Alternatively, the system may automatically set which control to perform based on the autofocus subject tracking sensitivity. An imaging device 100 with AF functionality generally has an AF tracking sensitivity setting function that allows the user to select from multiple levels (for example, 5 levels) of AF tracking sensitivity when the subject moves out of the focus area during still image shooting.

[0135] For example, if the tracking sensitivity has 5 levels from 1 to 5, the highest sensitivity "5" is "sensitive" and is effective for shooting scenes where you want to successively focus on subjects at different distances. The lowest sensitivity "1" is "sticky" and prevents the focus from shifting to another subject even if another subject crosses in front of the subject you want to photograph, allowing for stable focus tracking. "3" is the sensitivity in between.

[0136] The system is set to prioritize eye-tracking focus lock when the tracking sensitivity is low (sticky), and to prioritize eye-tracking focus movement when the subject tracking sensitivity is high (sensitive).

[0137] Alternatively, the system can automatically determine which control method to use based on the latest gaze information. If the latest gaze information indicates that the photographer's gaze frequently returns to the target position, the system can be set to prioritize gaze focus locking control. On the other hand, if the latest gaze information indicates that the photographer's gaze frequently returns to the target position, the system can be set to prioritize gaze focus movement control.

[0138] Furthermore, if the latest gaze information indicates that the eye movement remains (is fixed on) the target position for a long time, the system is set to prioritize eye focus locking control. On the other hand, if the latest gaze information indicates that the eye movement remains (is fixed on) the target position for a short time, the system is set to prioritize eye focus movement control.

[0139] Furthermore, if the latest gaze information shows a large relative ratio of the time the gaze remains (is fixed on) at the controlled position versus other positions, the system is set to prioritize gaze focus locking control. On the other hand, if the latest gaze information shows a small relative ratio of the time the gaze remains (is fixed on) at the controlled position versus other positions, the system is set to prioritize gaze focus movement control.

[0140] Furthermore, the system may be configured to automatically determine whether to perform the first control or the second control based on the subject recognized by the subject recognition unit 112. If the subject at the position where the photographer's gaze is directed (i.e., the subject they are focusing on) is larger than a predetermined size, the system may be configured to prioritize fixed gaze focus control. On the other hand, if the subject at the position where the photographer's gaze is directed (i.e., the subject they are focusing on) is smaller than a predetermined size, the system may be configured to prioritize shifting gaze focus control.

[0141] Next, we will explain other control methods.

[0142] In the third control example, the shooting-related control unit 203 adjusts the exposure based on the controlled position. The shooting-related control unit 203 adjusts the exposure by sending an exposure adjustment control signal to the lens drive driver 103. By adjusting the exposure, the amount of light captured can be adjusted according to the subject at the controlled position, thereby obtaining videos and still images with optimal brightness. Alternatively, the shooting-related control unit 203 may control the lens drive driver 103 to perform exposure adjustment, or the shooting-related control unit 203 itself may be configured to function as the lens drive driver 103.

[0143] In the fourth control example, the shooting-related control unit 203 adjusts the white balance based on the controlled position. The shooting-related control unit 203 adjusts the white balance by having the camera processing unit 106 send a control signal for white balance adjustment. By adjusting the white balance to match the subject at the controlled position, it is possible to obtain videos and still images with a color tone more desirable to the user. Alternatively, the shooting-related control unit 203 may control the camera processing unit 106 to perform white balance adjustment, or the shooting-related control unit 203 may be configured to function as the camera processing unit 106 that performs white balance adjustment.

[0144] In the fifth control example, the image capture control unit 203 generates metadata related to the controlled position, which is attached to the captured image. This information may include, for example, the type, size, and color of the subject recognized at the controlled position. The metadata may be used within the imaging device 100 or by an external device outside the imaging device 100. The imaging device 100 may be controlled so that the image capture control unit 203 generates the metadata, or the image capture control unit 203 may be configured to generate the metadata itself.

[0145] In the sixth control example, the shooting-related control unit 203 inputs a shutter button when the target position is within the field of view, and controls the imaging device 100 to obtain a video or still image with the target position within the field of view. Furthermore, the shooting-related control unit 203 may also perform recording control to save the image data (still images or video images) obtained from the shooting to the storage unit 107.

[0146] In the seventh control example, the shooting-related control unit 203 determines the object to be recognized by the subject recognition processing performed by the subject recognition unit 112. For example, the object to be recognized is determined to be a subject located at the control target position. The accuracy of autofocus can be improved by performing autofocus to focus on the subject recognized based on the subject recognition information. The shooting-related control unit 203 may control the subject recognition unit 112, or the shooting-related control unit 203 itself may be configured to function as a subject recognition unit.

[0147] When the subject recognition unit 112 recognizes a subject present at the control target position, the control target position is set to the position of the recognized subject on the display unit 110. As a result, when the subject moves, the control target position can be made to follow the movement of the subject. Therefore, even if the subject moves, the control target position will also move in accordance with the subject's position. Consequently, even if the control target position moves, the shooting-related control unit 203 can continue to perform control, such as maintaining focus on the control target position.

[0148] The processing in this technology is carried out as described above. According to this technology, the imaging device 100 can take pictures in accordance with the photographer's intentions while giving the impression that it is understanding the photographer's intentions in response to the photographer's gaze movements.

[0149] <2. Variant> Although embodiments of this technology have been described in detail above, this technology is not limited to the embodiments described above, and various modifications based on the technical concept of this technology are possible.

[0150] In the control target position determination process in the embodiment, the comparison between the reference gaze information and the latest gaze information includes a comparison of the gaze position and gaze time. However, control may be performed by comparing only the trajectory and gaze position without including the gaze time. Alternatively, control may be performed by comparing only the trajectory without including the gaze position and gaze time.

[0151] Furthermore, if the imaging device 100 has a function that allows the AF frame to be fixed at an arbitrary position based on input from the photographer (such as half-pressing the shutter), the control target position determination unit 202 may determine the position specified by the user as the control target position.

[0152] Furthermore, the half-press shutter function of the imaging device 100 may be used for processing by the information processing device 200. The first specific example is that the gaze position that the photographer was looking at when the shutter was half-pressed while their line of sight was aligned with the display unit 110 is treated as a gaze position of equivalent importance to one that was looked at for a time exceeding the second threshold.

[0153] A second specific example involves determining whether or not the shutter was half-pressed, instead of, or in addition to, determining whether or not the process in step S105 of the process in Figure 3 is repeated at a predetermined frequency.

[0154] Furthermore, a third specific example involves replacing step S302 of the processing in Figure 6 with a determination of whether the current controlled target position lies on the trajectory of the reference line of sight information and coincides with the gaze position where the shutter was half-pressed.

[0155] In this embodiment, the information processing device 200 is described as operating within the imaging device 100, but this technology may also be implemented by connecting the information processing device 200, which is configured on a server, to the imaging device 100 via a network.

[0156] In the embodiment, it was explained that the shooting-related control unit 203 performs predetermined control by comparing reference gaze information with the latest gaze information. However, instead of reference gaze information, the gaze information management unit 201 may store the time the gaze is aligned to a specific position and / or the frequency of repeated gaze movements, and if the gaze movement during shooting after storage matches the stored content, the shooting-related control unit 203 may perform predetermined control.

[0157] In this embodiment, the reference gaze information was described as the movement of the photographer's gaze relative to the display unit 110, but the reference gaze information may be something other than that. For example, it may be preset data showing the movement of the gaze that has been prepared in advance before shooting. This preset data consists of position data, trajectory data, gaze position data, etc., that show the movement of the gaze. This preset data may be prepared in advance as a preset by the manufacturer of the imaging device 100 and stored in the information processing device 200, or it may be created by the photographer or another user and stored in the information processing device 200. The preset data can also be created by referring to the photographer's gaze movement habits obtained by analyzing the photographer's gaze movement, or to the gaze movements common to multiple photographers obtained by analyzing the gaze movements of multiple photographers.

[0158] In this embodiment, a vector may be used instead of the direction of movement in step S402 of Figure 7.

[0159] In this embodiment, control was performed based on the photographer's eye movement information as input, but control may also be performed based on other input methods as long as they reflect the photographer's intentions.

[0160] This technology can also be configured as follows: (1) The system includes a control target position determination unit that determines the control target position based on reference line-of-sight information including the trajectory on the display unit of the imaging device, and the latest line-of-sight information including the trajectory of the photographer's gaze toward the display unit during shooting. Information processing device. (2) The information processing device according to (1), wherein the reference line of sight information includes the trajectory of the photographer's line of sight to the display unit during a photograph taken prior to the time of the photograph. (3) The information processing device according to (2), wherein the reference gaze information includes a gaze position which is a position on the trajectory where the photographer's gaze was aligned for a predetermined period of time or longer. (4) The information processing device according to (3), wherein the reference gaze information includes a gaze time, which is the time the photographer's gaze was aligned with the gaze position. (5) The information processing device according to (3), wherein the control target position determination unit determines the gaze position included in the reference gaze information as the control target position when part or all of the reference gaze information matches the latest gaze information. (6) The information processing device according to (5), wherein the control target position determination unit determines the control target position based on a comparison result between the gaze time, which is the time the photographer's line of sight was aligned with the gaze position, and a predetermined threshold. (7) The information processing device according to (5), wherein the control target position determination unit determines the gaze position as the control target position if the gaze is directed toward the gaze position included in the reference gaze information in the trajectory of the latest gaze information. (8) An information processing apparatus according to any one of (1) to (7), comprising an imaging-related control unit that performs control related to imaging by the imaging device based on the control target position. (9) The information processing device according to (8), wherein the shooting-related control unit performs autofocus control to maintain focus on the target position or to move the focus position to the target position. (10) The information processing apparatus according to (8), wherein the shooting-related control unit performs autofocus control to maintain focus on the target position when the imaging device is shooting a video, and performs autofocus control to move the focus position to the target position when the imaging device is shooting a still image. (11) The information processing device described in (8) is an information processing device that performs control to adjust the exposure based on the control target position, wherein the shooting-related control unit performs control. (12) The information processing device described in (8) is an information processing device that performs control to adjust the white balance based on the control target position of the shooting-related control unit. (13) The information processing device according to (8), wherein the imaging-related control unit performs both or either control to output information about the controlled target position and control to generate information about the controlled target position as metadata. (14) The information processing apparatus described in (8) above, which performs shooting control so that the position to be controlled is within the field of view. (15) The information processing apparatus described in (8) is an information processing apparatus that controls the recording of image data generated by the shooting. (16) An information processing apparatus according to any one of (1) to (15), comprising a display control unit that displays information relating to the controlled position on the display unit. (17) The reference line-of-sight information is prepared in advance as data before the time of the photograph, as described in any of (1) to (16) of the information processing device. (18) The information processing device according to any one of (1) to (17), wherein the control target position determination unit determines the position specified by the input to the imaging device by the photographer as the control target position. (19) The controlled target position is determined based on reference line-of-sight information, including the trajectory on the display unit of the imaging device, and the latest line-of-sight information, including the trajectory of the photographer's gaze toward the display unit during shooting. Information processing methods. (20) The controlled target position is determined based on reference line-of-sight information, including the trajectory on the display unit of the imaging device, and the latest line-of-sight information, including the trajectory of the photographer's gaze toward the display unit during shooting. An information processing program that instructs a computer to execute information processing methods. [Explanation of symbols]

[0161] 100... Imaging device 110...Display section 200... Information Processing Device 201...Gaze information management department 202...Control target position determination unit 203...Shooting-related control unit

Claims

1. The system includes a control target position determination unit that determines the control target position based on reference line-of-sight information including the trajectory on the display unit of the imaging device and the latest line-of-sight information including the trajectory of the photographer's gaze toward the display unit during shooting. The aforementioned reference gaze information includes the trajectory of the photographer's gaze toward the display unit during a photograph taken prior to the time of the photograph, and the gaze position which is a position on the trajectory of the gaze where the photographer's gaze was aligned for a predetermined period of time or longer. The control target position determination unit determines the gaze position as the control target position if part or all of the reference gaze information and the latest gaze information match, and furthermore, if the gaze in the trajectory of the photographer's gaze in the latest gaze information is directed toward the gaze position included in the reference gaze information. Information processing device.

2. The aforementioned reference gaze information includes a gaze duration, which is the time the photographer's gaze was aligned with the gaze position. The information processing apparatus according to claim 1.

3. The control target position determination unit determines the control target position based on a comparison between the gaze time, which is the time the photographer's line of sight was aligned with the gaze position, and a predetermined threshold. The information processing apparatus according to claim 2.

4. The system includes a shooting-related control unit that performs control related to shooting by the imaging device based on the controlled target position. The information processing apparatus according to claim 1.

5. The aforementioned shooting-related control unit performs autofocus control to either maintain focus on the target position or move the focus position to the target position. The information processing apparatus according to claim 4.

6. The aforementioned shooting-related control unit performs autofocus control to maintain focus on the target position when the imaging device is shooting video, and performs autofocus control to move the focus position to the target position when the imaging device is shooting still images. The information processing apparatus according to claim 4.

7. The aforementioned shooting-related control unit performs control to adjust the exposure based on the controlled target position. The information processing apparatus according to claim 4.

8. The aforementioned shooting-related control unit performs control to adjust the white balance based on the controlled target position. The information processing apparatus according to claim 4.

9. The aforementioned shooting-related control unit performs both or either control that outputs information regarding the controlled target position and control that generates information regarding the controlled target position as metadata. The information processing apparatus according to claim 4.

10. The aforementioned shooting-related control unit performs shooting control so that the controlled target position is within the field of view. The information processing apparatus according to claim 4.

11. The aforementioned shooting-related control unit performs recording control of the image data generated by the shooting. The information processing apparatus according to claim 4.

12. The system includes a display control unit that displays information regarding the controlled position on the display unit. The information processing apparatus according to claim 1.

13. The aforementioned reference line-of-sight information was prepared as data in advance before the time of the photograph. The information processing apparatus according to claim 1.

14. The control target position determination unit determines the position specified by the input from the photographer to the imaging device as the control target position. The information processing apparatus according to claim 1.

15. This is an information processing method for determining the position of a target to be controlled based on reference line-of-sight information including the trajectory on the display unit of an imaging device, and the latest line-of-sight information including the trajectory of the photographer's gaze toward the display unit at the time of shooting. The aforementioned reference gaze information includes the trajectory of the photographer's gaze toward the display unit during a photograph taken prior to the time of the photograph, and the gaze position which is a position on the trajectory of the gaze where the photographer's gaze was aligned for a predetermined period of time or longer. If the reference gaze information and the latest gaze information match in part or in whole, and furthermore, the gaze in the photographer's gaze trajectory in the latest gaze information is directed toward the gaze position included in the reference gaze information, then the gaze position is determined as the control target position. Information processing methods.

16. This is an information processing method for determining the position of a target to be controlled based on reference line-of-sight information including the trajectory on the display unit of an imaging device, and the latest line-of-sight information including the trajectory of the photographer's gaze toward the display unit at the time of shooting. The aforementioned reference gaze information includes the trajectory of the photographer's gaze toward the display unit during a photograph taken prior to the time of the photograph, and the gaze position which is a position on the trajectory of the gaze where the photographer's gaze was aligned for a predetermined period of time or longer. If the reference gaze information and the latest gaze information match in part or in whole, and furthermore, the gaze in the photographer's gaze trajectory in the latest gaze information is directed toward the gaze position included in the reference gaze information, then the gaze position is determined as the control target position. An information processing program that instructs a computer to execute information processing methods.

Images