Control device, display device, control method, and program

Abstract

To provide a control device that can realize viewpoint shift when some areas of an image are displayed without a user contacting an operation member, a display device, a control method, and a program.SOLUTION: The control device is for controlling a display device that displays a partial image based on a portion of an area of an image on a display surface. The control device has a generating unit for generating a partial image by applying a transformation process to a portion of an area of the image, a detection unit for detecting a first position of a user's line of sight on the display surface, and a control unit for causing the generating unit to generate a first partial image on the basis of the first position and for displaying the first partial image on the display surface when a predetermined time has elapsed during which the first position is located within the predetermined area.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to a control device, a display device, a control method, and a program. [Background technology]

[0002] VR (Virtual Reality) content is mainly viewed using a non-transparent HMD (Head Mounted Display) that covers the user's field of vision with a display unit. Also, photos and videos for VR are captured using an imaging device (see Patent Document 1) that can capture a wide-angle image such as a spherical image in one go. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6897268 Summary of the Invention [Problem to be solved by the invention]

[0004] When taking VR photos or videos and it is not possible to bring an HMD to the shooting location, the VR photos or videos can be viewed with a perspective projection display on the imaging device itself, so that they can be viewed with a view close to that of the naked eye. With perspective projection display, a portion of the image is displayed according to the user's viewing angle (the entire screen is not displayed at once), so the viewpoint must be moved to check the surroundings. If viewpoint movement is performed using a touch panel or cross key, there is a risk that the subject will move from the intended position due to contact between the user and the operating parts of the imaging device.

[0005] The present invention has an object to provide a control device, a display device, a control method, and a program that can realize viewpoint movement when a partial area of ​​an image is displayed without the user having to touch an operating member. [Means for solving the problem]

[0006] A control device as one aspect of the present invention is a control device for controlling a display device that displays a partial image based on a partial region of an image on a display surface, and is characterized in having a generation unit that generates the partial image by applying a conversion process to the partial region of the image, a detection unit that detects a first position of the user's gaze on the display surface, and a control unit that, when a predetermined time has elapsed during which the first position is located within a predetermined region, causes the generation unit to generate a first partial image based on the first position and displays the first partial image on the display surface. Effect of the Invention

[0007] According to the present invention, it is possible to provide a control device, a display device, a control method, and a program that are capable of realizing viewpoint movement when a partial area of ​​an image is displayed without the user having to touch an operating member. [Brief description of the drawings]

[0008] [Figure 1] 1 is a block diagram of a digital camera that is an example of a display device according to an embodiment of the present invention. [Diagram 2] FIG. 1 is an explanatory diagram of a method for generating a hemispherical image. [Diagram 3] FIG. 13 is a diagram showing correspondence between a fisheye image and a hemisphere in a three-dimensional virtual space. [Figure 4] FIG. 2 is a diagram showing the positions of a virtual camera in a three-dimensional virtual space and an area where perspective projection transformation is performed in a hemispherical image. [Diagram 5] 11 is a flowchart showing a viewpoint moving operation during perspective projection display in the first embodiment. [Figure 6] FIG. 2 is a diagram illustrating a perspective projection transformation in a hemispherical image in the first embodiment. [Figure 7] FIG. 11 is an explanatory diagram of the difference between absolute position specification and relative position specification. [Figure 8] 13 is a flowchart showing a viewpoint moving operation during perspective projection display in the second embodiment. [Figure 9] FIG. 13 is a diagram showing perspective projection display by relative position specification in the second embodiment. [Figure 10]13A to 13C are diagrams showing perspective projection displays in a gyro mode and a combined mode in the third embodiment. [Figure 11] 13 is a flowchart showing a viewpoint moving operation during perspective projection display in the third embodiment. [Figure 12] 13 is a flowchart showing a viewpoint moving operation during perspective projection display in the fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In each drawing, the same reference numerals are given to the same members, and duplicated explanations will be omitted.

[0010] FIG. 1 is a block diagram of a digital camera 100 which is an example of a display device according to an embodiment of the present invention.

[0011] The imaging unit 22 is an imaging element configured with a CCD, CMOS element, or the like that converts an optical image into an electrical signal.

[0012] The image processing unit (generation unit) 24 performs perspective projection transformation capable of converting a partial area of ​​a distorted wide-angle image, such as a fisheye image captured by the imaging unit 22, into a partial image. Examples of distorted wide-angle images include images with a viewing angle exceeding 180 degrees and images of projection methods such as equidistant projection, central projection, equistereographic projection, and orthogonal projection. Note that, since the perspective projection transformation is performed by setting a viewing angle, a partial image is generated by converting a partial area of ​​the wide-angle image.

[0013] The memory control unit 15 controls transmission and reception of data between the image processing unit 24 and the memory 32. Output data from the imaging unit 22 is written into the memory 32 via the image processing unit 24 and the memory control unit 15. Alternatively, the output data from the imaging unit 22 is written into the memory 32 via the memory control unit 15 without going through the image processing unit 24.

[0014] The memory 32 stores image data obtained by the imaging unit 22 and image data to be displayed on the display unit 28 and the EVF 29. The memory 32 has a storage capacity sufficient to store a predetermined number of still images and a predetermined period of moving images and audio. The memory 32 also serves as a memory for displaying images (video memory).

[0015] The display image data written in the memory 32 is displayed by the display unit 28 or EVF 29 via the memory control unit 15. The display unit 28 and EVF 29 each perform display on a display device such as an LCD or an organic EL in response to a signal from the memory control unit 15. A live view display (LV) can be performed by sequentially transferring and displaying data stored in the memory 32 to the display unit 28 or EVF 29. Hereinafter, an image displayed in the live view display is referred to as a live view image (LV image).

[0016] The gaze detection unit 160 detects the user's gaze at the eyepiece 16 of the finder. The gaze detection unit 160 includes a dichroic mirror 162, an imaging lens 163, a gaze detection sensor 164, a gaze detection circuit 165, and an infrared light emitting diode 166. Note that since the system control unit (control unit) 50 can also execute a predetermined process in response to detection of the gaze, the gaze detection unit 160 can also be said to be a part of the operation unit 70.

[0017] The gaze detection unit 160 detects the gaze by a method called the corneal reflex method. The corneal reflex method is a method for detecting the direction and position of the gaze from the positional relationship between the reflected light of infrared light emitted from an infrared emitting diode 166 from the cornea of ​​the eyeball 161 and the pupil of the eyeball 161. Note that a method other than the corneal reflex method may be used as long as it is possible to detect the direction and position of the gaze.

[0018] The infrared light emitting diode 166 is a light emitting element for detecting the direction and position of the user's line of sight in the viewfinder, and irradiates the user's eyeball 161 with infrared light. The infrared light emitted from the infrared light emitting diode 166 is reflected by the eyeball 161, and the infrared reflected light reflected by the eyeball 161 reaches the dichroic mirror 162. The dichroic mirror 162 reflects only the infrared light and transmits visible light. The infrared reflected light whose optical path has been changed forms an image on the imaging surface of the line of sight detection sensor 164 via the imaging lens 163. The imaging lens 163 is an optical member that constitutes the line of sight detection optical system. The line of sight detection sensor 164 is constituted by an imaging device such as a CCD image sensor.

[0019] The gaze detection sensor 164 photoelectrically converts the incident reflected infrared light into an electric signal and outputs it to the gaze detection circuit 165. The gaze detection circuit 165 detects the gaze position of the user from the movement of the eyeball 161 based on the output signal of the gaze detection sensor 164, and outputs the detection information to the system control unit 50.

[0020] The system control unit 50 determines that the user is gazing at a specific area when the time that the user's gaze position is fixed on the specific area exceeds a predetermined time based on the detection information acquired from the gaze detection circuit 165. Therefore, the specific area can be said to be the position at which the user gazes (gaze position). Note that "the gaze position is fixed on a specific area" means, for example, that the average position of the gaze movement is within the specific area until the predetermined time has elapsed, and the variation (dispersion) is less than a predetermined value. The predetermined time can be arbitrarily changed by the system control unit 50.

[0021] The non-volatile memory 56 is an electrically erasable and recordable memory, such as a flash ROM. Constants and programs for the operation of the system control unit 50 are recorded in the non-volatile memory 56. Here, the programs refer to programs for executing various flowcharts described later in this embodiment.

[0022] The system control unit 50 is a control unit consisting of at least one processor or circuit, and controls the entire digital camera 100. The system control unit 50 executes programs recorded in the nonvolatile memory 56 to realize each process of this embodiment, which will be described later. The system memory 52 is, for example, a RAM, and the system control unit 50 loads constants and variables for the operation of the system control unit 50, and programs and the like read from the nonvolatile memory 56 into the system memory 52. ​​The system control unit 50 also performs display control by controlling the memory 32, the display unit 28, and the like.

[0023] The recording medium 200 is a recording medium such as a memory card for recording captured images, and is composed of a semiconductor memory, a magnetic disk, or the like.

[0024] The communication unit 54 transmits and receives video signals and audio signals to and from an external device connected wirelessly or via a wired cable. The communication unit 54 can also connect to a wireless LAN (Local Area Network) or the Internet. The communication unit 54 can also communicate with an external device via Bluetooth (registered trademark) or Bluetooth Low Energy. The communication unit 54 can transmit images (including LV images) captured by the imaging unit 22 and images recorded on the recording medium 200, and can receive image data and various other information from the external device.

[0025] The attitude detection unit 55 detects the attitude of the digital camera 100 with respect to the direction of gravity. An acceleration sensor, a gyro sensor, or the like can be used as the attitude detection unit 55. It is also possible to detect the inclination and movement of the digital camera 100 (panning, tilting, lifting, whether or not it is stationary, etc.) using an acceleration sensor or a gyro sensor. Furthermore, a spirit level based on the detected inclination may be displayed on the display unit 28 or the EVF 29.

[0026] The eyepiece detection unit 57 is, for example, an infrared proximity sensor, and detects the approach (approach) and departure (away) of the eyeball (object) 161 to the eyepiece unit 16 of the finder incorporating the EVF 29. When an object approaches, infrared rays projected from the light projecting unit of the eyepiece detection unit 57 are reflected by the object and received by the light receiving unit. Depending on the amount of infrared rays received, the eyepiece detection unit 57 can also determine how close the object is to the eyepiece unit 16 (eyepiece distance). In this way, the eyepiece detection unit 57 performs eyepiece detection to detect the proximity of the object to the eyepiece unit 16. The eyepiece detection unit 57 detects that an object has been placed near the eyepiece unit 16 when an object approaching within a predetermined distance from a non-eyepiece state (non-approaching state) is detected. The eyepiece detection unit 57 also detects that an object that was detected as approaching the eyepiece unit 16 has been moved away from the eyepiece state (approaching state) by more than a predetermined distance. The threshold for detecting eye contact and the threshold for detecting eye removal may be different, for example, by providing hysteresis. After eye contact is detected, the eye is considered to be in the in-eye state until eye removal is detected. After eye removal is detected, the eye is considered to be in the non-in-eye state until eye contact is detected. Note that the infrared proximity sensor is just one example, and other sensors may be used as eye contact detection unit 57 as long as they can detect the approach of an eye or object that can be considered as eye contact.

[0027] The system control unit 50 switches between display (display state) / non-display (non-display state) of the display unit 28 and the EVF 29 depending on the state detected by the eyepiece detection unit 57. Specifically, at least in a shooting standby state and when the display destination switching is automatic switching, when the eye is not placed in contact with the camera, the display is turned on with the display unit 28 as the display destination and the EVF 29 is not displayed. Also, when the eye is placed in contact with the camera, the display is turned on with the EVF 29 as the display destination and the display unit 28 is not displayed.

[0028] The system control unit 50 can detect the following operations or states of the eyepiece unit 16 by using detection information obtained from the line of sight detection circuit 165 and by controlling the eyepiece detection unit 57. (1) The line of sight, which had not been directed toward the eyepiece 16, is now directed toward the eyepiece 16. In other words, eye-gaze input begins. (2) The gaze is being input into the eyepiece 16. (3) The subject is gazing at the eyepiece 16. (4) The line of sight directed toward the eyepiece 16 is averted, i.e., the eye-gaze input ends. (5) A state in which no gaze input is made to the eyepiece 16.

[0029] These operations and states, as well as the position (direction) in which the gaze is directed toward the eyepiece unit 16, are notified to the system control unit 50 via the internal bus, and the system control unit 50 determines what kind of operation (gaze operation) has been performed on the eyepiece unit 16 based on the notified information.

[0030] The operation unit 70 accepts various operations. The operations accepted by the operation unit 70 can be transmitted to the image processing unit 24 via the internal bus. In addition, the display range, position, and the like of the generated partial image can be adjusted or set by the user's operation via the operation unit.

[0031] In this embodiment, the image processing unit 24, the system control unit 50, and the line of sight detection unit 160 constitute a control device for controlling the display device.

[0032] Hereinafter, a method for generating a hemispherical image used when viewing VR (Virtual Reality) in this embodiment will be described with reference to FIG.

[0033] Fig. 2(a) shows an image captured when a fisheye lens is used with the digital camera 100. As shown in Fig. 2(a), the fisheye image becomes a distorted image cut into a circle when captured by the imaging unit 22. The image processing unit 24 uses, for example, a two-dimensional / three-dimensional computer graphics library such as Open GL (Open Graphics Library) to draw the hemisphere shown in Fig. 2(b) and pastes the fisheye image so as to cover its inside.

[0034] Specifically, the fisheye image is first associated with a coordinate system consisting of a vertical angle θ with the zenith direction of the captured image as an axis, and a horizontal angle φ (with the circumferential direction as an axis) around the axis of the zenith direction, as shown in FIG. 3. In this case, when the viewing angle range of the fisheye image is 180°, the vertical angle θ and the horizontal angle φ are in the range of −90° to 90°. The position (θ, φ) which is the coordinate value of the fisheye image can be associated with each point on the spherical surface representing the hemispherical image shown in FIG. 2(b). As shown in FIG. 2(b), when the center of the hemisphere is 0 and the three-dimensional coordinates on the spherical surface are (X, Y, Z), the three-dimensional coordinates (X, Y, Z) can be expressed as the two-dimensional coordinates of the fisheye image by the following formulas (1) to (3).

[0035] X = rc osθ sinφ (1) Y = rsinθ (2) Z = rcosθcosφ (3) Here, r is the radius of the hemisphere. By pasting the fisheye image inside the hemisphere based on the coordinate correspondence expressed by formulas (1) to (3), a hemispherical image can be generated in the three-dimensional virtual space.

[0036] Fisheye images 180° in front of and 180° behind the photographer are acquired, and each half-spherical image is generated using the method described above. A 360° full-spherical image can then be generated by stitching these together.

[0037] As described above, since a celestial sphere image and a hemispherical image are images pasted to cover a spherical surface, the images will look unnatural if they are used as is. Therefore, a predetermined projection transformation is performed on a partial area of ​​the image, and an image that does not look unnatural can be generated by displaying the image as a partial image with less distortion.

[0038] In the following, perspective projection transformation will be described as an example of projection transformation, but other transformation processes may be used as long as they enable partial display of a celestial sphere or hemisphere image.

[0039] 4 is a diagram showing the positions of a virtual camera in a three-dimensional virtual space in a hemispherical image and an area where perspective projection conversion is performed. The virtual camera corresponds to the position of the viewpoint of a user who views a celestial sphere image displayed as a three-dimensional sphere. The area where perspective projection conversion is performed is determined by information (θ, φ) indicating the direction of the virtual camera and the angle of view, and an image of this area is displayed on the display unit 28 or the EVF 29.

[0040] The viewpoint movement operation during perspective projection display of digital camera 100 in each embodiment will be described below. EXAMPLES

[0041] Fig. 5 is a flowchart showing the viewpoint movement operation during perspective projection display of digital camera 100 of this embodiment. Each process in the flowchart of Fig. 5 is realized by system control unit 50 expanding a program stored in non-volatile memory 56 into system memory 52, executing it, and controlling each functional block. Fig. 6 is a diagram showing perspective projection transformation of a hemispherical image of this embodiment.

[0042] As shown in Fig. 6(a), it is assumed that a partial image obtained by perspective projection conversion of a part of the hemispherical image by the image processing unit 24 is displayed on the EVF 29. The process of the flowchart in Fig. 5 is started when the system control unit 50 detects that the user has placed his / her eye near the eyepiece unit 16, as shown in Fig. 6(b).

[0043] In step S501, the system control unit 50 acquires information (θ,φ) indicating the direction of the user's gaze from the gaze detection unit 160 by the corneal reflex method. Here, θ is the elevation angle of the gaze, and φ is the angle in the circumferential direction of the gaze. The system control unit 50 acquires the user's gaze position (X,Y,Z) (first position) on the spherical surface of the hemispherical image shown in FIG. 6(b) from the information (θ,φ) using formulas (1) to (3). In this embodiment, the system control unit 50 and the gaze detection unit function as a detection unit that detects the position of the user's gaze on the display surface. In the initial state before the gaze position is determined in step S502 described later, the information (θ,φ) is set to (0,0), and the gaze position (X,Y,Z) is set.

[0044] In step S502, the system control unit 50 determines whether the time for gazing at the gaze position acquired in step S501 (gazing time; the time for which the gaze position is fixed (located within a predetermined area)) has exceeded a predetermined time. If the gaze time has exceeded the predetermined time, the process of step S503 is executed, and if it is determined not to have exceeded the predetermined time, the process of step S501 is executed.

[0045] In step S503, the system control unit 50 controls the image processing unit 24 to perform perspective projection transformation centered on the gaze position acquired in step S501. As a result, a partial image (first partial image) centered on the gaze position is generated, as shown in FIG. 6(c).

[0046] As described above, according to the configuration of this embodiment, the user can move the viewpoint when a portion of a wide-angle image is being displayed without touching the operating member.

[0047] In this embodiment, the absolute position is specified as described in the second embodiment, but a relative position may be specified instead of the absolute position. EXAMPLES

[0048] In the first embodiment, as shown in Fig. 7(a), when there is gaze at a specific position, perspective projection conversion is performed with the actual gazed position as the center. However, since there is a limit to the angle at which the user can direct their gaze, it is possible that the image is difficult to see or cannot be seen depending on the gaze angle. Therefore, in this embodiment, the method of specifying the perspective projection display position is switched depending on the gaze angle.

[0049] When the angle of the user's line of sight is equal to or greater than a certain value, the range in which perspective projection is performed in the gaze direction is shifted by a predetermined amount for each predetermined time unit, as shown in Fig. 7(b). For convenience, the method of designating in which perspective projection is performed centered on the actual gaze position, as explained in the first embodiment, is referred to as absolute position designation, and the method of designating in which the perspective projection display range is continually shifted in the gaze direction, as explained in this embodiment, is referred to as relative position designation.

[0050] Fig. 8 is a flowchart showing the viewpoint movement operation during perspective projection display in this embodiment. The processes of steps S801 and S802 in Fig. 8 are similar to the processes of steps S501 and S502 in Fig. 5, respectively, and therefore will not be described.

[0051] In step S803, the system control unit 50 determines whether the absolute value of the line of sight elevation angle θ acquired in step S801 exceeds a first predetermined angle, or the absolute value of the line of sight circumferential angle φ exceeds a second predetermined angle. The first and second predetermined angles are set to angles at which the user performing the line of sight input feels uncomfortable to see. The "angle at which the user feels uncomfortable to see" is, for example, an angle greater than 60°. If it is determined that the absolute value of the line of sight elevation angle θ exceeds the first predetermined angle, or the absolute value of the line of sight circumferential angle φ exceeds the second predetermined angle, the process of step S805 is executed, and if it is determined that the absolute value does not exceed the second predetermined angle, the process of step S804 is executed.

[0052] In step S804, the system control unit 50 executes the perspective projection display by absolute position specification, which is the same process as in step S503 in FIG.

[0053] In step S805, the system control unit 50 executes perspective projection display by specifying a relative position based on the line of sight position determined by the line of sight elevation angle θ and the line of sight circumferential angle φ acquired in step S801.

[0054] FIG. 9 is a diagram showing perspective projection display by relative position specification. The initial perspective projection display position is as shown in FIG. 9(a) by using information (θ 0 ,φ 0 ) is determined by the information (θ 0 ,φ 0 ) is moved a predetermined amount based on the reference position (θ, φ). Here, the information (θ', φ') can be expressed by the following formulas (4) and (5) using the information (θ, φ) acquired in step S801.

[0055] θ´=θ-θ 0 (4) φ´=φ-φ 0 (5) In addition, if the value obtained by dividing the information (θ', φ') by a predetermined value n is (Δθ', Δφ'), then the information (θ 0 +Δθ´,φ 0 +Δφ´) on the spherical surface of the hemispherical image (X 1 ,Y 1 ,Z 1 ) (second position) can be obtained. Then, as shown in FIG. 9(c), the three-dimensional coordinates (X 1 ,Y 1 ,Z 1 After a certain time, the perspective projection is performed with the position (θ 0 +2Δθ´,φ 0 +2Δφ´) 2 ,Y 2 ,Z 2 ) is the center of perspective projection display.

[0056] The above process is repeated until it is determined in step S806 described later that the process has been performed n times. When the process has been performed n times, the position of the perspective projection display coincides with the three-dimensional coordinates corresponding to the acquired information (θ, φ).

[0057] In step S806, the system control unit 50 determines whether the process of step S805 has been executed n times. If it is determined that the process has been executed n times, the process of step S807 is executed, and if it is determined that the process has not been executed n times, the process of step S805 is executed.

[0058] In step S807, the system control unit 50 divides the acquired information (θ, φ) by a predetermined value m to obtain a value (Δθ, Δφ), which is added to the angle of the previous perspective projection position as shown in Figure 9(e), and performs perspective projection display centered on the three-dimensional position corresponding to that angle.

[0059] In step S808, the system control unit 50 determines whether the angle of the perspective projection display position is a constant value. Here, the constant value is a value that exceeds the display range of the semi-spherical image, for example, an angle of ±90° in the zenith direction or the circumferential direction. If it is determined that the angle of the perspective projection display position is a constant value, the process of step S809 is executed, and if it is determined that the angle is not a constant value, the process of step S807 is executed.

[0060] In step S809, the system control unit 50 fixes the position of the perspective projection display at a three-dimensional position corresponding to the angle determined in step S808, as shown in FIG. 9(f), so as not to display a portion beyond the display range of the hemispherical image.

[0061] As described above, according to the configuration of this embodiment, when a part of a wide-angle image is displayed, even if it is desired to check an area that is difficult to see on the periphery of the image, the viewpoint can be moved without difficulty. EXAMPLES

[0062] In the first embodiment, the viewpoint can be moved only by gaze input, but in the present embodiment, the viewpoint can be moved by gaze input, gyro, or a combination of both during playback image display. Note that, during shooting, the viewpoint cannot be moved by gyro because the subject will move from the position intended by the user if the viewpoint is moved by gyro.

[0063] In the viewpoint movement operation using the gyro, the position of the perspective projection display (third position) is determined according to the tilt of the main body, as shown in Fig. 10(a). The tilt and pan, which are the tilt amounts of the digital camera 100 detected by the attitude detection unit 55, correspond to the vertical angle θ and horizontal angle φ in Fig. 4, respectively, so that the three-dimensional coordinates in the hemispherical image can be obtained from the formulas (1) to (3). The viewpoint movement operation using the gyro becomes possible by performing the perspective projection display centered on the obtained three-dimensional coordinates.

[0064] In the viewpoint movement operation using a combination of eye-gaze input and gyro, the position of the perspective projection display is determined by adding up the inclination of the digital camera 100 and the angle of the line of sight, as shown in FIG. 10(b). Here, the inclination of the digital camera 100 detected by the orientation detection unit 55 is used as information (θ 1 ,φ 1 ), and the information detected by the line of sight detection unit 160 is (θ 2 ,φ 2 ) At this time, the sum of these information (θ 1 +θ 2 ,φ 1 +φ 2 ) determines the position of the perspective projection display (fourth position). For convenience, below, viewpoint movement operation by eye-gaze input will be referred to as eye-gaze mode, viewpoint movement operation by tilting the camera body will be referred to as gyro mode, and viewpoint movement operation by combining these will be referred to as combined mode.

[0065] FIG. 11 is a flow chart showing the viewpoint movement operation during perspective projection display in this embodiment.

[0066] In step S1101, the system control unit 50 acquires a mode for performing a viewpoint movement operation selected by a user through various operations performed on the operation unit 70.

[0067] In step S1102, the system control unit 50 determines whether or not playback image display is being performed. If playback image display is determined to be performed, the process of step S1103 is performed. If not, that is, if LV image display is determined to be performed, the process of step S1106 is performed.

[0068] The processes in steps S1103 to S1105 in FIG. 11 are similar to the processes in steps S501 to S503 in FIG. 5, respectively, and therefore will not be described.

[0069] In step 1106, the system control unit 50 determines whether the mode acquired in step S1101 is the line-of-sight mode. If it is determined that the acquired mode is the line-of-sight mode, the process of step S1103 is executed, and if it is determined that the acquired mode is not the line-of-sight mode, that is, if it is determined that the acquired mode is the gyro mode or the combined mode, the process of step S1107 is executed.

[0070] In step S1107, the system control unit 50 selects the center position of the perspective projection display range according to the tilt of the digital camera 100 detected by the orientation detection unit 55. Then, the process of step S1105 is executed. In step S1105, perspective projection transformation is performed around the three-dimensional coordinates in the hemispherical image corresponding to the selected position, and perspective projection display is performed around the selected position, as shown in FIG. 10(a).

[0071] In step S1108, the system control unit 50 determines whether the acquired mode is the gyro mode. If it is determined that the selected mode is the gyro mode, this flow ends. If it is determined that the acquired mode is not the gyro mode, that is, the combined mode, the process of step S1103 is executed. In the process of step S1103, the position is determined by adding up the inclination of the camera body and the line of sight angle, and a perspective projection display is performed with the determined position as the center.

[0072] As described above, according to the configuration of this embodiment, when a portion of a wide-angle image is displayed, the user can select a suitable viewpoint movement operation method. EXAMPLES

[0073] VR photos and images include those that can be viewed stereoscopically using the parallax between the eyes. At this time, a wide-angle image for the right eye and a wide-angle image for the left eye are acquired. However, the digital camera 100 can only perform perspective projection display for one of the wide-angle images. Therefore, in this embodiment, the right eye image and the left eye image (images captured from multiple viewpoints) displayed on the EVF 29 are switched based on the user's intentional blinking. In this embodiment, the gaze detection unit 160 can determine whether the user is intentionally blinking.

[0074] FIG. 12 is a flow chart showing the viewpoint movement operation during perspective projection display in this embodiment.

[0075] In step S1201, the system control unit 50 controls the gaze detection unit 160 to determine whether the user's blink time in the EVF 29 exceeds a first predetermined time. Since there is a possibility that an erroneous determination will occur due to an unintentional blink by the user, it is necessary to determine whether the user has intentionally blinked. An "intentional blink" is, for example, a change from an open eye state to a closed eye state, and the closed eye state continues for a first predetermined time (e.g., 300 ms) or more. If it is determined that the blink time exceeds the first predetermined time, the process of step S1202 is executed, and if it is determined that the blink time does not exceed the first predetermined time, the process of step S1203 is executed.

[0076] In step S1202, the system control unit 50 controls the image processing unit 24 to switch between the right eye image and the left eye image. If a perspective projection display is performed on the wide-angle image for the right eye at first, the switching can be performed by applying the above-described hemispherical image generation method again to the wide-angle image for the left eye.

[0077] The processes in steps S1203 to S1205 in FIG. 12 are similar to the processes in steps S501 to S503 in FIG. 5, respectively, and therefore will not be described.

[0078] As described above, according to the configuration of this embodiment, when a portion of a wide-angle image is displayed, the user can switch between a right eye image and a left eye image without touching an operating member.

[0079] Although the condition for switching between the right eye image and the left eye image is that the blinking time exceeds a first predetermined time, other conditions may be used as long as they can determine whether the user intentionally blinks. For example, it may be used whether the number of blinks exceeds a predetermined number. Specifically, the condition is that the eyes change from an open state to a closed state, the closed state continues for a second predetermined time (e.g., 100 ms) or more, and the same condition is met again within a fourth predetermined time (e.g., 1 s) (two consecutive blinks).

[0080] Furthermore, the condition is not limited to a condition that a blinking action has occurred, but may be any condition that can be intentionally established without the user having to make any contact.

[0081] In the present embodiment, the present invention is applied to the digital camera 100, but the present invention is not limited thereto. The present invention is applicable to any display device that has an image processing function unit and a line of sight detection function and displays a partial image based on a partial area of ​​an image on a display surface. That is, the present invention is applicable to a head mounted display (HMD) that is capable of line of sight detection. By reading a recording medium on which a wide-angle image is recorded and generating a full celestial sphere or semi-celestial sphere image by an image processing function, perspective projection display can be performed when displaying a reproduced image. In addition, by connecting the HMD to a communication unit of an imaging device wirelessly or with a wired cable, perspective projection display can be performed on the HMD even during LV display. [Other Examples] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-mentioned embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0082] The disclosure of this embodiment includes the following configurations and methods.

[0083] (Configuration 1) A control device for controlling a display device to display a partial image based on a partial region of an image on a display surface, a generating unit that generates the partial image by performing a conversion process on a partial area of ​​the image; A detection unit that detects a first position of a user's line of sight on the display surface; A control device characterized by having a control unit that, when a predetermined time has elapsed during which the first position is located within a predetermined area, causes the generation unit to generate a first partial image based on the first position and displays the first partial image on the display surface. (Configuration 2) The control device according to configuration 1, wherein the control unit causes the generation unit to generate the first partial image centered on the first position. (Configuration 3) The control device described in configuration 1 is characterized in that the control unit generates a second partial image centered on a second position moved a predetermined amount from the original position toward the first position at predetermined time intervals, and displays the second partial image on the display surface. (Configuration 4) 4. The control device according to any one of configurations 1 to 3, wherein the image is an image recorded by a recording unit and reproduced. (Configuration 5) The control device described in configuration 4 is characterized in that, when the control device is set to a mode in which the partial image is generated based on a third position on the display surface corresponding to the tilt amount of the display device, the control device generates a third partial image centered on the third position. (Configuration 6) The control device described in configuration 4 or 5, characterized in that when the control device is set to a mode in which the partial image is generated based on the first position and a third position on the display surface corresponding to the tilt amount of the display device, the control device causes the generation unit to generate a fourth partial image centered on a fourth position based on the first position and the third position. (Configuration 7) The control device according to any one of configurations 1 to 6, wherein the image is a live view image captured by an imaging device. (Configuration 8) 8. The control device according to any one of configurations 1 to 7, wherein the transformation process is a perspective projection transformation. (Configuration 9) The control device according to any one of configurations 1 to 8, wherein the image is a spherical image. (Configuration 10) 9. The control device according to any one of configurations 1 to 8, wherein the image is a hemispherical image. (Configuration 11) The control device according to any one of configurations 1 to 10, wherein the detection unit is capable of detecting an angle of the user's line of sight corresponding to the first position. (Configuration 12) The control device described in configuration 11, characterized in that when the angle is smaller than a predetermined amount, the control unit causes the generation unit to generate the first partial image centered on the first position, and when the angle is greater than the predetermined amount, causes the generation unit to generate a second partial image centered on a second position moved a predetermined amount from the original position toward the first position at predetermined time intervals. (Configuration 13) 13. The control device according to any one of configurations 1 to 12, wherein the generation unit is capable of converting coordinates of the first position into coordinates in the image. (Configuration 14) A control device described in any one of configurations 1 to 13, characterized in that the partial image is an image generated by applying a conversion process to a partial area of ​​an image from one of images captured from multiple viewpoints. (Configuration 15) 15. The control device according to configuration 14, wherein the images captured from a plurality of viewpoints are a right eye image and a left eye image. (Configuration 16) 16. The control device according to configuration 14 or 15, wherein the control unit is capable of switching the partial images. (Configuration 17) 17. The control device according to any one of configurations 14 to 16, wherein the detection unit is capable of determining whether the user is intentionally blinking. (Configuration 18) The detection unit is capable of determining whether a user intentionally blinks, 18. The control device according to any one of configurations 14 to 17, wherein the control unit switches the partial image when a user intentionally blinks for a predetermined period of time. (Configuration 19) The detection unit is capable of determining whether a user intentionally blinks, 18. The control device according to any one of configurations 14 to 17, wherein the control unit switches the partial image when the number of times the user intentionally blinks exceeds a predetermined number. (Configuration 20) A control device according to any one of configurations 1 to 19; A display device comprising: (Method 1) 1. A control method for controlling a display device to display a partial image based on a partial region of an image on a display surface, comprising: generating the partial image by performing a conversion process on a partial area of ​​the image; detecting a first position of a user's gaze on the display surface; A control method characterized by comprising a step of generating a first partial image based on the first position and displaying the first partial image on the display surface when the time during which the first position is located within a specified area has elapsed. (Configuration 21) A program for causing a computer to execute the control method according to method 1.

[0084] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0085] 24 Image processing unit (generation unit) 50 System control unit (detection unit, control unit) 100 Digital camera (display device) 160 Line of sight detection unit (detection unit)

Claims

1. A control device for controlling a display device that displays a partial image based on a partial region of an image on a display surface, a generating unit that generates the partial image by performing a conversion process on a partial area of the image; a detection unit that detects a first position of the user's line of sight on the display surface; A control device characterized by having a control unit that, when a predetermined time has elapsed during which the first position is located within a predetermined area, causes the generation unit to generate a first partial image based on the first position and display the first partial image on the display surface, and generates a second partial image centered on a second position moved a predetermined amount from the original position toward the first position every predetermined time, and displays the second partial image on the display surface.

2. The control device according to claim 1 , wherein the control unit causes the generation unit to generate the first partial image centered on the first position.

3. 3. The control device according to claim 1, wherein the image is an image recorded by a recording unit and then reproduced.

4. The control device described in claim 3, characterized in that when the control device is set to a mode in which the partial image is generated based on a third position on the display surface corresponding to the tilt amount of the display device, the control device generates a third partial image centered on the third position.

5. The control device described in claim 3, characterized in that when the control device is set to a mode in which the partial image is generated based on the first position and a third position on the display surface corresponding to the tilt amount of the display device, the control device causes the generation unit to generate a fourth partial image centered on a fourth position based on the first position and the third position.

6. 3. The control device according to claim 1, wherein the image is a live view image captured by an imaging device.

7. 3. The control device according to claim 1, wherein the transformation process is a perspective projection transformation.

8. The control device according to claim 1 or 2, wherein the image is a spherical image.

9. 3. The control device according to claim 1, wherein the image is a hemispherical image.

10. The control device according to claim 1 or 2, wherein the detection unit is capable of detecting an angle of the user's line of sight corresponding to the first position.

11. The control device described in claim 10, characterized in that when the angle is smaller than a predetermined amount, the control unit causes the generation unit to generate the first partial image centered on the first position, and when the angle is greater than the predetermined amount, causes the generation unit to generate a second partial image centered on a second position moved a predetermined amount from the original position toward the first position at predetermined time intervals.

12. The control device according to claim 1 or 2, wherein the generating unit is capable of converting the coordinates of the first position into coordinates in the image.

13. 3. The control device according to claim 1, wherein the partial image is an image generated by performing a conversion process on a partial area of an image captured from one of a plurality of viewpoints.

14. 14. The control device according to claim 13, wherein the images captured from a plurality of viewpoints are a right-eye image and a left-eye image.

15. The control device according to claim 13 , wherein the control unit is capable of switching the partial images.

16. The control device according to claim 13, wherein the detection unit is capable of determining whether the user is intentionally blinking.

17. The detection unit is capable of determining whether the user intentionally blinks, The control device according to claim 13 , wherein the control unit switches the partial image when a predetermined period of time has elapsed since the user intentionally blinked.

18. The detection unit is capable of determining whether the user intentionally blinks, The control device according to claim 13 , wherein the control unit switches the partial image when the number of times the user intentionally blinks exceeds a predetermined number.

19. The control device according to claim 1 or 2; A display device comprising:

20. 1. A control method for controlling a display device that displays a partial image based on a partial region of an image on a display surface, comprising: generating the partial image by performing a conversion process on a partial region of the image; detecting a first position of a user's gaze on the display surface; A control method characterized by comprising the steps of: when a predetermined time has elapsed during which the first position is located within a predetermined area, generating a first partial image based on the first position and displaying the first partial image on the display surface; and generating a second partial image centered on a second position moved a predetermined amount from the original position toward the first position every predetermined time, and displaying the second partial image on the display surface.

21. A program causing a computer to execute the control method according to claim 20.

Images