Imaging apparatus, method of controlling an imaging apparatus, and medium

Abstract

Provided is an imaging device having a display image generation unit that performs a line-of-sight correspondence display image generation process of generating a display image in which superimposed information is superimposed on a captured image, based on a superimposed information display region for displaying the superimposed information and a gaze region designated based on line-of-sight information.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to an imaging apparatus, a method of controlling an imaging apparatus, and a program. BACKGROUND

[0002] In an imaging apparatus equipped with an auto focus (AF) function, a focus position is generally displayed with a frame such as a rectangle (hereinafter appropriately referred to as an AF frame). The AF frame is displayed by so-called on screen display (OSD) superimposed on a captured image.

[0003] LIST OF CITATIONS

[0004] PATENT LITERATURE

[0005] Patent Literature 1: Japanese Patent Application Laid-Open No. 2016-218106 SUMMARY

[0006] PROBLEMS TO BE SOLVED BY THE INVENTION

[0007] In imaging, a point at which a user gazes at an image is generally a point desired to be imaged, but if an AF frame is displayed at the point, the user has difficulty in confirming a subject at a point at which the AF frame is superimposed. On the other hand, if the AF frame is not displayed, there is a problem that the user has difficulty in understanding whether it is in focus.

[0008] An object of the present disclosure is to provide an imaging apparatus capable of appropriately displaying superimposed information such as an AF frame, a method of controlling the imaging apparatus, and a program.

[0009] SOLUTION TO PROBLEM

[0010] For example, the present disclosure provides an imaging apparatus including a display image generation unit that performs a line-of-sight correspondence display image generation process of generating a display image in which superimposed information and a captured image are superimposed, based on a superimposed information display region for displaying the superimposed information and a gazed region designated based on line-of-sight information.

[0011] Further, for example, the present disclosure provides a method of controlling an imaging apparatus, the method including causing a display image generation unit to perform a line-of-sight correspondence display image generation process of generating a display image in which superimposed information and a captured image are superimposed, based on a superimposed information display region for displaying the superimposed information and a gazed region designated based on line-of-sight information.

[0012] Further, for example, the present disclosure provides a program for causing a computer to execute a control method of causing a display image generation unit to perform a line-of-sight correspondence display image generation process of generating a display image in which superimposed information and a captured image are superimposed, based on a superimposed information display region for displaying the superimposed information and a gazed region designated based on line-of-sight information. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 is a block diagram for explaining a configuration example of an imaging apparatus according to an embodiment.

[0014] Figure 2 is a figure referred to in a process of designating a gazing region.

[0015] Figure 3 is a figure referred to in a process of a display image generation unit according to an embodiment.

[0016] Figure 4 is a figure referred to in a process of a display image generation unit according to an embodiment.

[0017] Figure 5 is a figure referred to in a process of a display image generation unit according to an embodiment.

[0018] Figure 6 is a figure referred to in a process of a display image generation unit according to an embodiment.

[0019] Figure 7 is a figure referred to in a process of a display image generation unit according to an embodiment.

[0020] Figure 8 is a figure referred to in a process of a display image generation unit according to an embodiment.

[0021] Figure 9 is a figure referred to in a process of a display image generation unit according to an embodiment.

[0022] Figure 10 is a flowchart illustrating a flow of a process executed by an imaging apparatus according to an embodiment.

[0023] Figure 11 is a figure for explaining a first example of determining whether to execute a line-of-sight correspondence display image generation process according to an imaging mode or the like.

[0024] Figure 12 is a flowchart for explaining a first example of determining whether to execute a line-of-sight correspondence display image generation process according to an imaging mode or the like.

[0025] Figure 13 is a figure for explaining a second example of determining whether to execute a line-of-sight correspondence display image generation process according to an imaging mode or the like.

[0026] Figure 14 is a flowchart for explaining a second example of determining whether to execute a line-of-sight correspondence display image generation process according to an imaging mode or the like.

[0027] Figure 15 is a diagram for explaining a third example of determining whether to perform the line-of-sight correspondence display image generation processing in accordance with an imaging mode or the like.

[0028] Figure 16 is a flowchart for explaining the third example of determining whether to perform the line-of-sight correspondence display image generation processing in accordance with an imaging mode or the like.

[0029] Figure 17 is a diagram for explaining a modification variant. DETAILED DESCRIPTION

[0030] Hereinafter, an embodiment of the present disclosure or the like will be described with reference to the drawings. Note that the description will be given in the following order.

[0031] <First Embodiment>

[0032] <Second Embodiment>

[0033] <Modification Variant>

[0034] The following described embodiment or the like is a preferable specific example of the present disclosure, and the content of the present disclosure is not limited to these embodiments or the like.

[0035] <First Embodiment>

[0036] [Configuration Example of Imaging Apparatus]

[0037] First, a configuration example of an imaging apparatus (imaging apparatus 100) according to an embodiment will be described with reference to Figure 1 The imaging apparatus 100 includes a control unit 101, an optical imaging system 102, a lens driver 103, an imaging element 104, an image signal processing unit 105, a codec unit 106, a storage unit 107, an interface 108, an input unit 109, a display unit 110, a microphone 111, a detection unit 112, an AF control unit 113, and a line-of-sight detection unit 114. The control unit 101 includes a gaze region pointing unit 101A and a display image generation unit 101B as functional blocks.

[0038] The control unit 101 includes a central processing unit (CPU), a random access memory (RAM), a read only memory (ROM), and the like. The CPU performs various processing and issues a command according to a program stored in the ROM, thereby comprehensively controlling the entire imaging device 100 and each unit. The gaze region specifying unit 101A specifies a gaze region on which the user gazes on the display unit 110 on the basis of the line-of-sight information. Further, the display image generating unit 101B performs line-of-sight correspondence display image generation processing of generating a display image in which the superimposition information and the captured image are superimposed on the basis of a superimposition information display region for displaying the superimposition information and the gaze region specified on the basis of the line-of-sight information. Note that details of the line-of-sight correspondence display image generation processing performed by the display image generating unit 101B will be described later.

[0039] The optical imaging system 102 includes an imaging lens for focusing light from a subject on the imaging element 104, a driving mechanism for moving the imaging lens to perform focusing and zooming, a shutter mechanism, an aperture mechanism, and the like. These are driven on the basis of a control signal from the control unit 101 and the lens driver 103. An optical image of the subject obtained by the optical imaging system 102 is formed on the imaging element 104.

[0040] The lens driving driver 103, for example, includes a microcomputer, and moves the imaging lens by a predetermined amount along the optical axis direction on the basis of focus control information supplied from the AF control unit 113 or the information processing device 200, thereby performing auto focus to focus on a target subject. Further, the operations of the driving mechanism, the shutter mechanism, the aperture mechanism, and the like of the optical imaging system 102 are controlled under the control of the control unit 101. Therefore, an exposure time (shutter speed) and an aperture value (F value) are adjusted.

[0041] The imaging element 104 photoelectrically converts light incident from a subject and obtained through the imaging lens into an electric charge amount and outputs an imaging signal. Then, the imaging element 104 outputs a pixel signal to the image signal processing unit 105. As the imaging element 104, a charge coupled device (CCD), a complementary metal oxide semiconductor (CMOS), or the like is used.

[0042] The imaging element 104 includes a red (R) pixel, a green (G) pixel, and a blue (B) pixel as normal pixels, and a phase difference detection pixel that performs phase difference detection. It is possible to perform so-called phase difference auto focus (AF) using phase difference information output from the phase difference detection pixel. Note that the phase difference detection pixel can function only as a phase difference detection pixel and can not function as a normal pixel, or can constitute one pixel by two independent photodiodes to perform imaging and phase difference detection. Note that the phase difference detection can be performed by an AF sensor dedicated to phase difference detection.

[0043] The image signal processing unit 105 performs sampling and hold to maintain a high signal-to-noise ratio (S / N) by correlated double sampling (CDS) processing, automatic gain control (AGC) processing, analog / digital (A / D) conversion, and the like on an imaging signal output from the imaging element 104 to create an image signal. Further, the image signal processing unit 105 performs recording processing on an image signal for recording and display processing on an image signal for display.

[0044] The codec unit 106 performs, for example, encoding processing for recording and communication on an image signal subjected to predetermined processing.

[0045] The storage unit 107 is a large-capacity storage medium such as a hard disk or a flash memory, for example. Video data and image data processed by the image signal processing unit 105 and the codec unit 106 are stored in a compressed state or an uncompressed state based on a predetermined standard. Further, an exchangeable image file format (EXIF) including additional information such as information on the stored data, imaging position information indicating an imaging position, and imaging time information indicating an imaging date and time is also stored in association with the data.

[0046] The interface 108 is an interface with another device, the Internet, and the like. The interface 108 can include a wired or wireless communication interface. Further, more specifically, the wired or wireless communication interface can include cellular communication such as 3TTE, Wi-Fi, Bluetooth (registered trademark), near field communication (NFC), Ethernet (registered trademark), high-definition multimedia interface (HDMI) (registered trademark), universal serial bus (USB), and the like.

[0047] Note that the imaging device 100 can include a communication unit capable of connecting to the Internet, other devices, and the like, such as a wireless local area network (LAN), a wide area network (WAN), or wireless fidelity (WiFi). Further, communication between the imaging device 100 and an external device can be short-range wireless communication such as near field communication (NFC) or ZigBee (registered trademark), or a tethering connection such as a WiFi tether, a universal serial bus (USB) tether, or a Bluetooth (registered trademark) tether.

[0048] Various instructions are given to the imaging device 100 by a user using the input unit 109. When the user makes an input to the input unit 109, a control signal corresponding to the input is generated and supplied to the control unit 101. Then, the control unit 101 performs various processing corresponding to the control signal. Examples of the input unit 109 include a shutter button for shutter input, a physical button for various operations, a touch panel, a touch screen configured integrally with a display as the display unit 110, and the like.

[0049] The display unit 110 is a display device such as a display that displays a pass-through image that is a display image signal that has undergone display processing, an image / video that has undergone recording image processing and is stored in the storage unit 107, a graphical user interface (GUI), and the like.

[0050] The microphone 111 is a sound collecting device for recording sound at the time of recording.

[0051] The detection unit 112 performs AF detection for determining a focus position using phase difference information supplied from the imaging element 104 and calculates a defocus amount. The defocus amount is supplied to the AF control unit 113.

[0052] The AF control unit 113 generates focus control information indicating which position in the angle of view to focus on (for example, XY coordinate information) and how much to move the lens of the imaging device 100 by the lens driver 103 to focus on a target subject based on the defocus amount calculated by the detection unit 112. The focus control information is information for performing automatic focus control in the imaging device 100.

[0053] The line-of-sight detection unit 114 generates a line-of-sight detection result by detecting the line of sight of the user. The generated line-of-sight detection result is supplied from the line-of-sight detection unit 114 to the control unit 101.

[0054] [About Line-of-Sight Information and Gaze Area]

[0055] Next, a specific example of a process of detecting the line of sight of the user by the line-of-sight detection unit 114 will be described. For example, the line-of-sight detection unit 114 images the eyes of the user (image capturer) and detects the line-of-sight direction of the user using the image of the eyes. That is, the line-of-sight detection unit 114 includes, for example, a camera unit that images the eyes of the user and a unit that detects the line-of-sight direction of the user. The camera unit can include a light emission unit that emits infrared light or the like.

[0056] As a method of detecting the line-of-sight direction of the user, a known method can be applied. For example, it is possible to apply a corneal reflection method that emits infrared light or the like and uses reflection from the cornea to detect the line-of-sight direction of the user based on the position of the pupil. Further, for example, a method of recognizing a point such as the inner or outer canthus that does not move by image recognition and estimating the line-of-sight direction from the position of the iris of the eye can be applied.

[0057] The line-of-sight detection unit 114 is provided, for example, in the viewfinder of the imaging device 100. The line-of-sight detection unit 114 can be provided in the housing of the imaging device 100. For example, the line-of-sight detection unit 114 can be provided on the surface of the housing of the imaging device 100 on which the display unit 110 is provided.

[0058] The gaze direction detected by the gaze detection unit 114 is supplied to the control unit 101.

[0059] The gaze region specifying unit 101A of the control unit 101 generates gaze information from the gaze detection result supplied from the gaze detection unit 114. The gaze information is, for example, information indicating the distribution of the gaze detection result. Specifically, as shown in Figure 2 Figure 2 The gaze region specifying unit 101A obtains a histogram of the values (hereinafter appropriately referred to as gaze levels) of the distribution of the gaze detection result PA corresponding to both the X-axis direction and the Y-axis direction on the display unit 110, and specifies one or more regions AR corresponding to the peaks thereof as the gaze region GA. Further, the gaze region specifying unit 101A can obtain a variance value of each point by dividing the histogram at a predetermined resolution, and specify a region in which the variance value is equal to or greater than a threshold value as the gaze region GA. Further, a gaze region having an area determined to be a predetermined value and a region extending from a region with a high gaze level to an area in which the area becomes the predetermined value can be set as the gaze region GA. Further, a region in which the gaze level is equal to or greater than a threshold value can be set as the gaze region GA.

[0060] Note that the gaze information can be information indicating the trajectory of the gaze detection result PA. For example, in a case where the trajectory of the gaze detection result PA falls within a certain region for a predetermined time of about several seconds, the gaze region specifying unit 101A can also determine the region as the gaze region GA.

[0061] [Specific Example of the Process of the Display Image Generation Unit]

[0062] Next, a specific example of the process performed by the display image generation unit 101B will be described. In the present example, an AF frame displayed based on focus information indicating the focus alignment in the captured image will be described as an example of the superimposition information.

[0063] ​Note that the overlay information can be divided into information associated with the display content of the display unit 110 or the state of the imaging device 100 (hereinafter appropriately referred to as dynamic overlay information) and information that can change the display position to an appropriate position without affecting the overall movement, although the value is changed (hereinafter appropriately referred to as fixed overlay information). Examples of dynamic overlay information, in addition to the AF frame mentioned above, include zebra stripe displays indicating bright areas equal to or higher than a set value, peaking displays indicating areas with high-frequency components (the areas in focus), and horizontal indicators indicating the tilt of the imaging device 100. The AF frame, peaking displays, and zebra stripe displays can also be image state information indicating the state of the image captured in the imaging device 100. Examples of fixed overlay information include the remaining battery capacity, the number of images that can be captured, the shutter speed, and the ISO sensitivity.

[0064] From another perspective, the overlaid information is imaging device status information that indicates the state of the imaging device 100. Imaging device status information is setting information that indicates the settings of the imaging device 100. Examples of setting information include shutter speed, aperture value, ISO sensitivity, mode dial, flash mode, etc. Examples of imaging device status information other than setting information include remaining battery capacity, the number of images that can be captured, etc.

[0065] Figure 3 An example of a through image displayed on display unit 110 is illustrated. On display unit 110, image IMA is displayed, which includes a person SA as a child and two deer SB and SC as subjects.

[0066] Here, consider an example where the gaze region identification unit 101A identifies the area near the center of a person's face (SA) as the gaze region (GA) and displays a rectangular frame (AF) within the identified portion. Note that in Figure 3 In this context, the actual displayed AF frame (hereinafter appropriately referred to as AF frame 21A) is darker than the other AF frames 21. The area including the multiple displayed AF frames 21A corresponds to the overlay information display area AR1 used to display overlay information. Furthermore, although in Figure 3 The gaze area GA is illustrated in the diagram, but the dotted lines defining the gaze area GA are not actually displayed on the display unit 110. However, the gaze area GA can be displayed in a mode that the user can recognize.

[0067] like Figure 3 As shown, when the gaze area GA and the overlay information display area AR1 do not overlap, the AF frame 21A is displayed in normal display mode (rectangular shape). However, as... Figure 4As shown, when the AF frame 21A is displayed in the gaze region GA, it may be difficult to understand the expression of the person SA, and the shutter scene may be missed (this could be the timing of starting to capture a moving image). Therefore, the display image generation unit 101B performs gaze-corresponding display image generation processing to generate a display image based on the overlap state between the gaze region GA and the overlay information display area AR1 used to display the AF frame 21A. In this embodiment, the display image generation unit 101B performs gaze-corresponding display image generation processing based on whether the gaze region GA and the overlay information display area AR1 overlap each other; more specifically, when the gaze region GA and the overlay information display area AR1 overlap each other, the display image generation unit 101B performs gaze-corresponding display image generation processing. Note that the display image generation unit 101B can perform gaze-corresponding display image generation processing if the size or ratio of the area where the gaze region GA and the overlay information display area AR1 overlap each other is equal to or greater than a certain value.

[0068] Specifically, the gaze-corresponding display image generation process performed by the display image generation unit 101B is a process that performs an overlap-corresponding display image generation process. In the case where the overlay information display area AR1 and the gaze area GA overlap, the AF frame (hereinafter appropriately referred to as AF frame 21B) in the overlapping area of ​​the overlay information display area AR1 and the gaze area GA overlaps with the AF frame 21A in the non-overlapping area of ​​other areas is displayed differently in the display image on the display unit 110.

[0069] (First specific example)

[0070] As a first concrete example of overlapping correspondence display image generation processing, such as Figure 5 As shown, the display image generation unit 101B generates a display image, wherein the display mode of AF frame 21B in the overlapping region where the overlay information display area AR1 and the gaze area GA overlap is semi-transparent, and the display mode of AF frame 21A in the non-overlapping region where the overlay information display area AR1 and the gaze area GA do not overlap is set to a transmittance similar to normal transmittance. The generated image is displayed on the display unit 110. By increasing the transmittance of AF frame 21B in the overlapping region via semi-transparency, the user can confirm the subject in the gaze area GA. Note that transmittance can be expressed by the following equation, where transmittance is T.

[0071] Output image = Captured image × T + OSD image × (1-T)

[0072] (Transmittance corresponds to the α value in image processing, and is generally referred to as alpha mixing)

[0073] Note that the translucency is not necessarily 50% transmittance, and can be, for example, transmittance equal to or higher than a predetermined transmittance at which a face of the person SA can be visually recognized.

[0074] In this example, the transmittance of the AF frame 21B can vary depending on the position in the overlapping region. For example, in the overlapping region, the transmittance of the AF frame 21B can vary between a region near the outer edge of the overlapping region and a region other than the region near the outer edge of the overlapping region. Specifically, the transmittance of the AF frame 21B in the region other than the region near the outer edge of the overlapping region can be made lower than the transmittance of the AF frame 21B in the region near the outer edge of the overlapping region.

[0075] (Second Specific Example)

[0076] As a second specific example of the overlapping correspondence display image generation process, as shown in Figure 6 the display image generation unit 101B generates a display image in which the display shape of the AF frame 21B in the overlapping region in which the superimposition information display region AR1 and the gaze region GA overlap each other is different from the display shape of the AF frame 21A in the non-overlapping region in which the superimposition information display region AR1 and the gaze region GA do not overlap each other. For example, the AF frame 21B in the overlapping region has a larger rectangular display shape than the AF frame 21A in the non-overlapping region, so that the user can confirm the face of the person SA in the overlapping region. With this process, the user can confirm the subject in the gaze region GA.

[0077] Further, in the present example, as shown in Figure 7 the display image generation unit 101B can generate a display image in which the frame line of the AF frame 21B in the overlapping region is thinned, and the display image can be displayed on the display unit 110. Due to the frame thinning of the AF frame 21B, the user can more easily confirm the image of the portion of the overlapping region. Note that the thickness of the outer edge of the frame can be reduced when the shape of the AF frame 21B is maintained in the default display mode.

[0078] Further, as shown in Figure 8 the shape of the AF frame 21B in the overlapping region can be changed. For example, the AF frame 21B can have a rectangular shape, a circular shape, an elliptical shape, or a polygonal shape. The shape of the AF frame 21B can vary depending on the position in the overlapping region.

[0079] (Third Specific Example)

[0080] A third specific example of the superimposition correspondence display image generation process is an example in which a display image that makes the gaze area GA conspicuous is generated, while the transmittance of the AF frame 21B in the superimposition area and the display shape of the frame are changed as in the first and second specific examples described above. Specifically, as shown in FIG. 19B, the display image generation unit 101B can halftone or the like the luminance of the area outside the superimposition area so that the image of the superimposition area is more conspicuous than the other areas. In this example, a display image in which the display image of the superimposition area is most easily seen, the superimposition information display area AR1 is second most easily seen, and the other parts are most difficult to see can be generated and displayed. Figure 9

[0081] As another specific example, in a case where the superimposition information in the gaze area GA is superimposition information that does not need to be displayed in the gaze area GA, the display position of the superimposition information can be moved outside the gaze area GA. That is, the superimposition information display area can be set to an area that does not overlap with the gaze area. An example of superimposition information that does not need to be displayed in the gaze area GA is fixed superimposition information. Furthermore, in a case where the areas overlap with each other, the superimposition information can be moved so that the area is smaller than the current superimposition area.

[0082] [Flow of the process]

[0083] Figure 10 is a flowchart illustrating the flow of the process executed by the imaging device 100. For example, when the power of the imaging device 100 is turned on, the following process is executed. The process can be started at the stage when the user is in a posture of holding up the imaging device 100 and capturing an image, based on the detection result of the gyro sensor or the pressure-sensitive sensor. Furthermore, the process can be started after the line-of-sight detection unit 114 detects the line of sight. The process can be repeated while the line of sight is detected. In step ST11, the line-of-sight correspondence display image generation process is started. Then, the process proceeds to step ST12.

[0084] In step ST12, based on the focus control information supplied from the AF control unit 113, the control unit 101 determines the superimposition information display area AR1 for displaying the AF frame 21A as an example of superimposition information on the display unit 110. Then, the process proceeds to step ST13.

[0085] In step ST13, the gaze area GA is determined. That is, the gaze area pointing unit 101A points the gaze area GA of the display unit 110 based on the line-of-sight detection result of the line-of-sight detection unit 114. Then, the process proceeds to step ST14.

[0086] ​In step ST14, the control unit 101 determines whether the superimposition information display region AR1 and the gaze region GA overlap. In a case where the determination in step ST14 is "No", the processing proceeds to step ST16.

[0087] In step ST16, since the superimposition information display region AR1 and the gaze region GA do not overlap, the display image generation unit 101B generates a display image in which the AF frame 21A is displayed in a normal display mode (default display mode) such as a rectangle. The generated display image is displayed on the display unit 110 (see Figure 3 ).

[0088] In the determination processing in step ST14, it is determined whether the superimposition information display region AR1 and the gaze region GA overlap. In a case where the determination in step ST14 is "Yes", the processing proceeds to step ST15.

[0089] In step ST15, the display image generation unit 101B performs superimposition correspondence display image generation processing of generating a superimposition correspondence display image. The superimposition correspondence display image is an image in which the transmittance or the display shape of the AF frame 21B in the superimposition region is changed or the superimposition region is highlighted (see Figures 5 to 9 ). The superimposition correspondence display image is displayed on the display unit 110.

[0090] According to the above-described present embodiment, it is possible to prevent the display content of the gaze region in which the user gazes from becoming difficult to view due to the OSD such as the AF frame. Therefore, the display content in the gaze region as a portion in which the user's attention is concentrated becomes easy to view, and thus it is possible to appropriately determine the timing at which the user presses the shutter or the timing at which the user captures a moving image.

[0091] <Second Embodiment>

[0092] Next, a second embodiment will be described. Note that, in the description of the second embodiment, the same reference signs are attached to the configurations that are the same as or similar to those described above, and the repetitive description is appropriately omitted. Further, unless otherwise described, the matters described in the first embodiment can be applied to the second embodiment.

[0093] The second embodiment is an embodiment in which whether or not the line-of-sight correspondence display image generation processing described in the first embodiment is performed is determined in accordance with the imaging mode, the operation of the imaging device 100, and the characteristics of the superimposition information.

[0094] [First Example]

[0095] The first example is an example in which whether or not the line-of-sight correspondence display image generation processing is performed is determined in accordance with the imaging mode. Figure 11 is a table for explaining an outline of the present example. As Figure 11As shown in FIG. 6, in a case where the imaging mode is an AUTO mode and an aperture priority mode, the line-of-sight correspondence display image generation processing is not performed (stopped). On the other hand, in a case where the imaging mode is a PROGRAM mode and a shutter speed priority mode, the line-of-sight correspondence display image generation processing is performed (executed).

[0096] In the AUTO mode, it is generally assumed that the user of the imaging device 100 is a beginner. Therefore, in a case where the imaging mode is the AUTO mode, the line-of-sight correspondence display image generation processing is not performed. That is, in order to assist the beginner user, the AF frame in the superimposition region is displayed in a normal display mode (rectangular shape).

[0097] In the PROGRAM mode (a mode in which some settings such as exposure are reserved), since it is assumed that the user desires to perform the control function of the imaging device 100, the line-of-sight correspondence display image generation processing is performed.

[0098] The aperture priority mode is assumed to be an imaging mode set in a case where an object such as a landscape having less motion is imaged with emphasis on blur and the like. Therefore, in a case where the imaging mode is the aperture priority mode, in order to reliably notify the user of information on whether the AF is accurate or not, the line-of-sight correspondence display image generation processing is not performed, and the AF frame in the superimposition region is displayed in a normal display mode (rectangular shape).

[0099] The shutter speed priority mode is assumed to be an imaging mode set in order to capture a moment of a fast-moving subject. That is, it is assumed that the user wants to confirm the subject in real time. Therefore, in a case where the shutter speed priority mode is set, the line-of-sight correspondence display image generation processing is performed so that the user can easily confirm the display content in the gaze region GA.

[0100] Figure 12 is a flowchart illustrating a detailed flow of the processing according to the present embodiment. In step ST211, the control unit 101 determines whether the information is fixed superimposition information or dynamic superimposition information, and determines the type of the superimposition information. Here, in a case where it is determined that the superimposition information is the above-described dynamic superimposition information, the processing proceeds to step ST212.

[0101] In step ST212, the control unit 101 determines the imaging mode. As a result of the determination, in a case where the imaging mode is the AUTO mode or the aperture priority mode, the processing proceeds to step ST213. In step ST213, the line-of-sight correspondence display image generation processing is not performed.

[0102] When the determination result in the determination processing of step ST212 is the PROGRAM mode or the shutter speed wired mode, the process proceeds to step ST214. In step ST214, the gaze correspondence display image generation processing is executed by the display image generation unit 101B.

[0103] When the determination result in the determination processing of step ST211 is that the superimposition information is the fixed superimposition information, the process proceeds to step ST215.

[0104] In step ST215, the control unit 101 determines whether or not it is necessary to display the fixed superimposition information. Here, when the control unit 101 determines that it is necessary to display the fixed superimposition information, the process proceeds to step ST216, and the process ends without executing the gaze correspondence display image generation processing. When the control unit 101 determines that it is not necessary to display the fixed superimposition information, the process proceeds to step ST216, and the gaze correspondence display image generation processing is executed.

[0105] For example, in step ST215, the control unit 101 determines the necessity of displaying the fixed superimposition information as follows. When the shutter button of the imaging device 100 is in a half-pressed state (hereinafter appropriately referred to as an S1 state), since the user is in a posture of capturing an image, the control unit 101 determines that it is not necessary to display the fixed superimposition information. As another example, when the fixed superimposition information is the battery remaining capacity, and the battery remaining capacity is sufficient (for example, equal to or greater than a threshold value (for example, 70%)), the need to display the battery remaining capacity is low. Therefore, when the remaining capacity of the battery is sufficient, the control unit 101 determines that it is not necessary to display the remaining capacity of the battery. On the other hand, when the remaining capacity of the battery is less than the threshold value, the control unit 101 determines that it is necessary to display the remaining capacity of the battery.

[0106] Further, as another example, the control unit 101 determines the necessity of displaying the fixed superimposition information based on a period in which the gaze area GA overlaps with the superimposition information display area in which the fixed superimposition information is displayed. When the period in which the gaze area GA overlaps with the superimposition information display area in which the fixed superimposition information is displayed is equal to or longer than a threshold value (for example, several 10 seconds), the control unit 101 determines that it is necessary to display the fixed superimposition information, assuming that the user has checked the content of the fixed superimposition information. On the other hand, when the period in which the gaze area GA overlaps with the superimposition information display area in which the fixed superimposition information is displayed is shorter than the threshold value (for example, several 10 seconds), the control unit 101 determines that it is not necessary to display the fixed superimposition information.

[0107] Furthermore, as another example, the control unit 101 determines the necessity of displaying the fixed overlay information based on the degree of overlay between the fixed overlay information and the main subject. The main subject can be identified using known methods, such as identifying the largest of a plurality of subjects as the main subject. When the degree of overlay between the fixed overlay information and the main subject is equal to or greater than a certain level, it is necessary to make the main subject easily visible; therefore, the control unit 101 determines that displaying the fixed overlay information is unnecessary. Conversely, when the degree of overlay between the fixed overlay information and the main subject is less than a certain level, the display of the main subject is not obstructed by the fixed overlay information; therefore, the control unit 101 determines that displaying the fixed overlay information is necessary.

[0108] Note that, as in this example, the type of overlay information is determined because, when the overlay information is fixed overlay information, it is often displayed near the outer edge of the display unit 110 and often does not overlap with the viewing area GA. Therefore, by determining the necessity of displaying fixed overlay information, it can be displayed appropriately.

[0109] [Second Example]

[0110] The second example is an example of determining whether to perform gaze-corresponding image generation processing based on the user's operating state in each imaging mode. Note that repeated descriptions of processes similar to those in the first example according to this embodiment will be appropriately omitted. Furthermore, in this example, the pressed state of the shutter button (S0 state (the state where the shutter button is not pressed) / S1 state (the state where the shutter button is half-pressed)) will be described as an example of the user's operating state.

[0111] Figure 13 This is a table that provides a summary of this example. For example... Figure 13 As shown, in AUTO mode, it is generally assumed that the user of the imaging device 100 is a beginner. Therefore, when the imaging mode is AUTO mode, the eye-tracking image generation process is not performed regardless of the shutter button's state. That is, to help beginner users, the AF frame in the overlapping area is displayed in normal display mode (rectangular shape).

[0112] In PROGRAM mode (a mode that preserves some settings such as exposure), assuming the user expects to perform the control functions of the imaging device 100, the line-of-sight corresponding display image generation process is performed regardless of the state of the shutter button.

[0113] The aperture priority mode is assumed to be an imaging mode set in a case where blurring and the like are valued and an object such as a landscape having less motion is imaged. In the aperture priority mode, in a case where the state is the S0 state, the user is assumed to be in a stage of confirming the subject, and thus the line-of-sight correspondence display image generation processing is executed so as to easily confirm the subject. On the other hand, in a case where the shutter button is in the S1 state, since it is considered that the composition of imaging has been confirmed and imaging is about to be executed, the line-of-sight correspondence display image generation processing is not executed, in other words, the AF frame is displayed in the normal mode, and information about whether the AF is accurate or not is provided to the user.

[0114] The shutter speed priority mode is assumed to be an imaging mode set in order to capture a moment of a fast-moving subject. In a case where the imaging mode is the shutter speed priority mode and the state is the S0 state, the user is considered to be tracking the subject, but the line-of-sight correspondence display image generation processing is stopped so that the user can confirm the focus state of the subject to be tracked. In a case where the state is the S1 state, since it is considered that the user continues to wait for a good timing to capture an image, the line-of-sight correspondence display image generation processing is executed.

[0115] Figure 14 is a flowchart illustrating a detailed flow of the processing according to the present embodiment. Since the processing of determining the type of the superimposition information (processing according to step ST211), the processing of determining the imaging mode (processing according to step ST212), and the processing in a case where the superimposition information is the fixed superimposition information (processing according to steps ST215 to ST217) are similar to those in the above-described first embodiment, a repeated description will be omitted.

[0116] In a case where the imaging mode is the AUTO mode in the determination processing according to step ST212, the processing proceeds to step ST213. In a case where the imaging mode is the AUTO mode, the line-of-sight correspondence display image generation processing is not executed regardless of the state of the shutter button, and the generation processing of the normal display image is executed. Further, in a case where the imaging mode is the PROGRAM mode, the processing proceeds to step ST214, and the line-of-sight correspondence display image generation processing is executed regardless of the state of the shutter button.

[0117] In a case where the imaging mode is the aperture priority mode in the determination processing according to step ST212, the processing proceeds to step ST221. In step ST221, the control unit 101 determines the state of the shutter button. Here, if the state of the shutter button is the S1 state, the line-of-sight correspondence display image generation processing is not executed, and the generation processing of the normal display image is executed. Further, if the state of the shutter button is the S0 state, the line-of-sight correspondence display image generation processing is executed by the display image generation unit 101B.

[0118] In a case where the imaging mode is the shutter speed priority mode in the determination processing according to step ST222, the processing proceeds to step ST222. In step ST222, the control unit 101 determines the state of the shutter button. Here, if the state of the shutter button is the S0 state, the line-of-sight correspondence display image generation processing is not performed, and the normal display image generation processing is performed. Further, if the state of the shutter button is the S1 state, the line-of-sight correspondence display image generation processing is performed by the display image generation unit 101B.

[0119] [Third Example]

[0120] The third example is an example in which whether to perform the line-of-sight correspondence display image generation processing is determined according to the operation state of the user in each imaging mode. Note that the repetitive description of the processing similar to the processing according to the first and second examples of the present embodiment described above will be appropriately omitted. Further, in the present example, in addition to the second example, whether to perform the line-of-sight correspondence display image generation processing is also determined according to whether the button to which the AF-ON function is assigned is pressed as the operation state of the user. Note that AF-ON refers to activation of AF by pressing a predetermined button (generally, a button on the back of the camera that can be pressed with the thumb) in a case where it is desired to separately perform AF and shutter release. Generally, the AF frame is displayed in a state where the shutter button is in the S1 state or in a state where the button to which the AF-ON (that is, the AF-ON function) is assigned is pressed.

[0121] Figure 15 is a table for explaining an outline of the present example. As shown in Figure 15 In the AUTO mode, it is generally assumed that the user of the imaging device 100 is a beginner. Therefore, in a case where the imaging mode is the AUTO mode, the line-of-sight correspondence display image generation processing is not performed regardless of the state of the shutter button. That is, in order to assist the beginner user, the AF frame in the superimposition region is displayed in the normal display mode (rectangular shape).

[0122] In the PROGRAM mode (a mode in which some settings such as exposure are reserved), it is assumed that the user desires to perform the control function of the imaging device 100, and therefore the line-of-sight correspondence display image generation processing is performed regardless of the state of the shutter button.

[0123] The aperture priority mode is assumed to be an imaging mode set in a case where blurring and the like are valued and an object having less motion such as a landscape is imaged. In a case where the mode is the aperture priority mode and the AF-ON or shutter button is in the S0 state, the gaze correspondence display image generation processing is executed so as to easily view a subject while checking the AF frame. Further, in a case where the shutter button is in the S1 state, since it is considered that the composition of imaging has been confirmed and imaging is about to be performed, the gaze correspondence display image generation processing is not executed, in other words, the AF frame is displayed in the normal mode, and information on whether the AF is accurate is provided to the user.

[0124] The shutter speed priority mode is assumed to be an imaging mode set in order to capture a moment of a fast-moving subject. In a case where the imaging mode is the shutter speed priority mode and the AF-ON or shutter button is in the S0 state, the gaze correspondence display image generation processing is not executed and the display of the AF frame is prioritized. Further, in a case where the state of the shutter button is the S1 state, the gaze correspondence display image generation processing is executed in consideration of the user continuing to wait for a good timing to capture an image.

[0125] Figure 16 is a flowchart illustrating a detailed flow of the processing according to the present example. Since the processing of determining the type of the superimposition information (processing according to step ST211), the processing of determining the imaging mode (processing according to step ST212), and the processing in a case where the superimposition information is the fixed superimposition information (processing according to steps ST215 to ST217) are similar to those in the above-described first embodiment, a repeated description will be omitted.

[0126] In a case where the imaging mode is the AUTO mode in the determination processing according to step ST212, the processing proceeds to step ST213. In a case where the imaging mode is the AUTO mode, the gaze correspondence display image generation processing is not executed regardless of the state of the shutter button, and the generation processing of the normal display image is executed. Further, in a case where the imaging mode is the PROGRAM mode, the processing proceeds to step ST214, and the gaze correspondence display image generation processing is executed regardless of the state of the shutter button.

[0127] In a case where the imaging mode is the aperture priority mode in the determination processing according to step ST212, the processing proceeds to step ST231. In step ST231, the control unit 101 determines the state of the shutter button and determines whether the AF-ON is set. Here, if the AF-ON or the shutter button is in the S0 state, the processing proceeds to step ST214, and the gaze correspondence display image generation processing is executed by the display image generation unit 101B. If the state of the shutter button is the S1 state, the gaze correspondence display image generation processing is not executed, and the normal display image generation processing is executed.

[0128] In a case where the imaging mode is the shutter speed priority mode in the determination processing according to step ST212, the processing proceeds to step ST232. In step ST232, the control unit 101 determines the state of the shutter button and determines whether the AF-ON is set. Here, if the state of the shutter button is the S1 state, the gaze correspondence display image generation processing is not executed, and the normal display image generation processing is executed. Further, if the AF-ON or the shutter button is in the S0 state, the gaze correspondence display image generation processing is executed by the display image generation unit 101B.

[0129] <Modification>

[0130] Although the plurality of embodiments of the present disclosure have been specifically described above, the content of the present disclosure is not limited to the above-described embodiments, and various modifications based on the technical idea of the present disclosure are possible.

[0131] In the above-described embodiments, the superimposition correspondence display image generation processing can be processing of determining the display form of the superimposition information in the superimposition region in accordance with the distribution of the gaze level determined on the basis of the gaze information in the display image. For example, the superimposition correspondence display image generation processing can be processing of increasing the difference between the display form of the superimposition information in the superimposition region and the display form of the superimposition information in the non-superimposition region as the gaze level increases.

[0132] Further, the superimposition correspondence display image generation processing can be processing of changing the display form of the superimposition information in the superimposition region in a stepwise manner in accordance with the distribution of the gaze level. For example, as shown in FIG. 31, it is assumed that there are a region AR31 in which the gaze level is higher than a threshold and a region AR32 in which the gaze level is lower than the threshold in the superimposition region. In this case, the display form of the AF frame 21 in the region AR31 and the display form of the AF frame 21 in the region AR32 are changed in a stepwise manner. For example, the AF frame 21 is displayed in the region AR31 having a first transmittance, and the AF frame 21 is displayed in the region AR32 having a second transmittance lower than the first transmittance. That is, the user can easily view the subject in the region having a high gaze level. Figure 17 ​

[0133] Further, as shown in Figure 17 The first display form of the AF frame 21 in the region AR31 (an example of a first region), the second display form of the AF frame 21 in the region AR32 (an example of a second region) in an overlapping region different from the region AR31, and the display form of the AF frame 21 in a non-overlapping region can be different, as shown in

[0134] As the gaze level, the gaze level can be determined in accordance with the trajectory of the line of sight. For example, a gaze region in which the user gazes can be designated in accordance with the trajectory of the line of sight, and the gaze level can be changed in accordance with the distance from the center of gravity of the gaze region. The degree of change (how to change the display form) can be determined in accordance with the distance from the center of gravity of the gaze region, without using the gaze level.

[0135] In a case where the gaze level is a value of a distribution described in the embodiment, the gaze level can be scaled so that the maximum value of the gaze level corresponds to the maximum transmittance of the AF frame. A table of the transmittance corresponding to the gaze level can be provided, and the transmittance of the AF frame can be determined in accordance with the table.

[0136] Further, in the above-described multiple display examples, for the AF frame in the gaze region in the non-overlapping region, processing of forming a display shape similar to the AF frame in the overlapping region or the like can be performed.

[0137] The gaze region is not limited to the current gaze region, and can be a region including a region predicted based on a movement vector based on a gaze trajectory and in which the user will gaze.

[0138] In the processing according to the above-described embodiment, machine learning-based processing, that is, processing using a learning model obtained by learning performed in advance, can be performed.

[0139] The superimposition information display region can be a region in which other information of the superimposition information is displayed.

[0140] The display image generation unit can determine whether to perform the line-of-sight information correspondence display image generation processing in accordance with each of multiple stages of the operation of the imaging device by the user. Examples of the multiple stages include a stage from the S0 state to the S1 state and AF-ON in the S1 state. The line-of-sight information correspondence display image generation processing can be performed in the stage from the S0 state to the S1 state. In the AF-ON stage of the S1 state, the line-of-sight information correspondence display image generation processing can not be performed, and the AF frame can be displayed in a normal display mode.

[0141] The configurations, methods, steps, shapes, materials, numerical values, and the like described in the above embodiments and modified examples are merely examples, and configurations, methods, steps, shapes, materials, numerical values, and the like different from those described above can be used as needed, or the configurations, methods, steps, shapes, materials, numerical values, and the like described in the above embodiments and modified examples can be replaced with known ones. Furthermore, the configurations, methods, steps, shapes, materials, numerical values, and the like in the embodiments and modified examples can be combined with one another within a range in which technical contradiction does not occur.

[0142] Note that the content of the present disclosure should not be construed as being limited by the effects exemplified in the present specification.

[0143] The present disclosure can also take the following configurations. (1)

[0145] An imaging apparatus including

[0146] A display image generation unit performs a line-of-sight correspondence display image generation process of generating a display image in which superimposition information and a captured image are superimposed, on the basis of a superimposition information display region for displaying the superimposition information and a gaze region designated on the basis of line-of-sight information. (2)

[0148] The imaging apparatus according to (1), wherein

[0149] The line-of-sight correspondence display image generation process is a process of generating the display image on the basis of an overlapping state between the superimposition information display region and the gaze region. (3)

[0151] The imaging apparatus according to (2), wherein

[0152] The line-of-sight correspondence display image generation process is a process of generating the display image in accordance with whether the superimposition information display region and the gaze region overlap each other. (4)

[0154] The imaging apparatus according to (3), wherein

[0155] The line-of-sight correspondence display image generation process is a process of performing an overlapping correspondence display image generation process in which, in a case where the superimposition information display region and the gaze region are caused to overlap each other in the display image, the superimposition information in an overlapping region in which the superimposition information display region and the gaze region overlap each other is in a different display form from the superimposition information in a non-overlapping region that is another region. (5)

[0157] The imaging apparatus according to (4), wherein

[0158] The superimposition correspondence display image generation process is a process of making the display form of the superimposition information different between the overlapping region and the non-overlapping region within the superimposition information display region. (6)

[0160] The imaging apparatus according to (5), wherein

[0161] The superimposition correspondence display image generation process is a process of making the display form of the superimposition information semi-transparent in the overlapping region. (7)

[0163] The imaging apparatus according to (5), wherein

[0164] The superimposition correspondence display image generation process is a process of making the display shape of the superimposition information different between the overlapping region and the non-overlapping region within the superimposition information display region. (8)

[0166] The imaging apparatus according to (5), wherein

[0167] The superimposition correspondence display image generation process is a process of determining the display form of the superimposition information in the overlapping region in accordance with a distribution of the gaze level determined on the basis of the line-of-sight information in the display image. (9)

[0169] The imaging apparatus according to (8), wherein

[0170] The superimposition correspondence display image generation process is a process of changing the display form of the superimposition information in the overlapping region in a stepwise manner in accordance with the distribution of the gaze level. (10)

[0172] The imaging apparatus according to (8), wherein

[0173] The superimposition correspondence display image generation process includes a first display form of the superimposition information in a first region, a second display form of the superimposition information in a second region different from the first region and included in the overlapping region, and a third display form of the superimposition information in the non-overlapping region, as the display form of the superimposition information in the overlapping region, and

[0174] In a case where the gaze level in the first region is higher than the gaze level in the second region, a difference between the first display form and the third display form is greater than a difference between the second display form and the third display form. (11)

[0176] The imaging apparatus according to any one of (3) to (10), wherein

[0177] In a case where the superimposition information display region and the gaze region overlap each other, the display image generation unit performs superimposition correspondence display image generation processing of setting the superimposition information display region to a region that does not overlap the gaze region. (12)

[0179] The imaging apparatus according to any one of (1) to (11), wherein

[0180] The superimposition information is imaging apparatus state information indicating a state of the imaging apparatus. (13)

[0182] The imaging apparatus according to any one of (1) to (11), wherein

[0183] The superimposition information is image state information indicating a state of a captured image in the imaging apparatus. (14)

[0185] The imaging apparatus according to (15), wherein

[0186] The image state information is focus information of the captured image. (15)

[0188] The imaging apparatus according to any one of (1) to (14), wherein

[0189] The display image generation unit determines whether to perform the line-of-sight information correspondence display image generation processing in accordance with an imaging mode. (16)

[0191] The imaging apparatus according to any one of (1) to (14), wherein

[0192] The display image generation unit determines whether to perform the line-of-sight information correspondence display image generation processing in accordance with each of a plurality of stages of user operations to the imaging apparatus. (17)

[0194] The imaging apparatus according to any one of (1) to (16), wherein

[0195] The line-of-sight information is information indicating a distribution of a line-of-sight detection result. (18)

[0197] The imaging apparatus according to any one of (1) to (16), wherein

[0198] The line-of-sight information is information indicating a trajectory of a line-of-sight detection result. (19)

[0200] A method of controlling an imaging apparatus, the method including causing a display image generation unit to perform a line-of-sight correspondence display image generation process of generating a display image in which superimposed information and a captured image are superimposed, based on a superimposed information display region for displaying the superimposed information and a gaze region designated based on line-of-sight information. (20)

[0202] A program for causing a computer to execute a control method of causing a display image generation unit to perform a line-of-sight correspondence display image generation process of generating a display image in which superimposed information and a captured image are superimposed, based on a superimposed information display region for displaying the superimposed information and a gaze region designated based on line-of-sight information.

[0203] List of reference numerals

[0204] 100 imaging apparatus

[0205] 101 control unit

[0206] 101A gaze region designation unit

[0207] 101B display image generation unit

[0208] 114 line-of-sight detection unit

Claims

1. An imaging device, comprising: The display image generation unit performs gaze-correspondence display image generation processing to generate a display image that overlays the overlay information and the captured image, based on the overlay information display area for displaying overlay information and the gaze area identified based on gaze information. The gaze-correspondence display image generation process is a process of generating the display image based on the overlap state between the overlay information display area and the gaze area; The gaze-correspondence display image generation process is a process that generates the display image based on whether the overlay information display area and the gaze area overlap. and The gaze-corresponding display image generation process is a process that performs an overlap-corresponding display image generation process. In the case where the overlay information display area and the gaze area overlap each other, the overlay information in the overlapping area where the overlay information display area and the gaze area overlap each other is displayed in a different form than the overlay information in the non-overlapping area which is another area.

2. The imaging device according to claim 1, wherein... The overlapping corresponding display image generation process is a process that makes the overlapping information in the overlapping area and the non-overlapping area of ​​the overlapping information display area have different display forms.

3. The imaging apparatus according to claim 2, wherein... The overlapping corresponding display image generation process is a process that makes the display form of the superimposed information semi-transparent in the overlapping area.

4. The imaging apparatus according to claim 2, wherein The overlapping correspondence display image generation process is a process that makes the display shape of the overlay information different between the overlapping area and the non-overlapping area within the overlay information display area.

5. The imaging apparatus according to claim 2, wherein... The overlapping corresponding display image generation process is a process that determines the display form of the overlay information in the overlapping region based on the distribution of gaze levels determined by the gaze information in the display image.

6. The imaging apparatus according to claim 5, wherein The overlapping corresponding display image generation process is a process that changes the display form of the overlay information in the overlapping region in a stepwise manner according to the distribution of the gaze level.

7. The imaging apparatus according to claim 5, wherein The overlapping corresponding display image generation process includes a first display form of overlay information in a first region, a second display form of overlay information in a second region that is different from the first region but included in the overlapping region, and a third display form of overlay information in the non-overlapping region, as the display form of overlay information in the overlapping region, and When the gaze level in the first region is higher than the gaze level in the second region, the difference between the first display format and the third display format is greater than the difference between the second display format and the third display format.

8. The imaging apparatus according to claim 1, wherein When the overlay information display area overlaps with the gaze area, the display image generation unit performs an overlap correspondence display image generation process to set the overlay information display area as an area that does not overlap with the gaze area.

9. The imaging apparatus according to claim 1, wherein The overlay information is imaging device status information that indicates the state of the imaging device.

10. The imaging apparatus according to claim 1, wherein The overlay information is image state information that indicates the state of the captured image in the imaging device.

11. The imaging apparatus according to claim 10, wherein The image state information is the focus information of the captured image.

12. The imaging apparatus according to claim 1, wherein The display image generation unit determines whether to perform gaze information correspondence display image generation processing based on the imaging mode.

13. The imaging apparatus according to claim 1, wherein The display image generation unit determines whether to perform gaze information correspondence display image generation processing based on each stage of the multiple stages of user operation on the imaging device.

14. The imaging apparatus according to claim 1, wherein The gaze information is information indicating the distribution of gaze detection results.

15. The imaging apparatus according to claim 1, wherein The gaze information is information indicating the trajectory of the gaze detection result.

16. A method for controlling an imaging device, the method comprising: The display image generation unit performs gaze-correspondence display image generation processing to generate a display image that superimposes the superimposed information and the captured image, based on the superimposed information display area for displaying superimposed information and the gaze area identified based on gaze information. The gaze-correspondence display image generation process is a process of generating the display image based on the overlap state between the overlay information display area and the gaze area; The gaze-correspondence display image generation process is a process that generates the display image based on whether the overlay information display area and the gaze area overlap. and The gaze-corresponding display image generation process is a process that performs an overlap-corresponding display image generation process. In the case where the overlay information display area and the gaze area overlap each other, the overlay information in the overlapping area where the overlay information display area and the gaze area overlap each other is displayed in a different form than the overlay information in the non-overlapping area which is another area.

17. A computer storage medium having instructions stored thereon, the instructions causing a processor to perform the method according to claim 16 when executed by a processor.

Images