Control device, display device, control method, and program

Abstract

To provide a control device that can realize viewpoint shift when some areas of an image are displayed without a user contacting an operation member, a display device, a control method, and a program.SOLUTION: The control device is for controlling a display device that displays a partial image based on a portion of an area of an image on a display surface. The control device has a generating unit for generating a partial image by applying a transformation process to a portion of an area of the image, a detection unit for detecting a first position of a user's line of sight on the display surface, and a control unit for causing the generating unit to generate a first partial image on the basis of the first position and for displaying the first partial image on the display surface when a predetermined time has elapsed during which the first position is located within the predetermined area.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present invention relates to a control device, a display device, a control method, and a program.

Background Art

[0002] VR (Virtual Reality) content is mainly viewed using a non-transparent HMD (Head Mounted Display) that covers the user's field of view with a display unit. In addition, VR photographs and videos are captured using an imaging device capable of capturing wide-angle images such as omnidirectional images at one time (see Patent Document 1).

Prior Art Literature

Patent Literature

[0003]

Patent Document 1

Summary of the Invention

Problem to be Solved by the Invention

[0004] If an HMD cannot be brought to the shooting site when capturing VR photographs or videos, perspective projection display on the imaging device body can be used to check the VR photographs or videos with an appearance close to that of the naked eye. In perspective projection display, only a partial region matching the user's viewing angle is displayed (the entire screen is not displayed at once), so viewpoint movement is required to check the surroundings. If viewpoint movement is performed via operation using a touch panel or cross key, contact between the user and the operating member of the imaging device may cause the subject to shift from the intended position.

[0005] An object of the present invention is to provide a control device, a display device, a control method, and a program that can realize viewpoint movement when a partial region of an image is displayed without the user contacting an operating member.

Means for Solving the Problem

[0006] One aspect of the present invention is a control device for controlling a display device that displays a partial image based on a portion of an image on a display surface, comprising: a generation unit that generates a partial image by applying a transformation process to a portion of an image; and a first position of the user's line of sight on the display surface. And the angle of the user's gaze corresponding to the first position. A detection unit detects the first position, and when a predetermined time has elapsed since the first position was located within a predetermined area, If the angle is smaller than the specified angle, First position in the generation section Centered around The first partial image is generated, and the first partial image is displayed on the display surface. If the angle is greater than the specified angle, Return to the original position at set intervals. ra A second partial image is generated centered on a second position, which is moved by a predetermined amount in the direction of the first position. ,table Display surface ni It has a control unit that displays a partial image. Furthermore, the angle of view is defined as 0 degrees when the first position is at the center of the partial image. It is characterized by the following. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide a control device, a display device, a control method, and a program that enable viewpoint movement when a portion of an image is displayed, without the user having to touch an operating member. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram of a digital camera, which is an example of a display device according to an embodiment of the present invention. [Figure 2] This is an explanatory diagram of the method for generating hemispherical images. [Figure 3] This figure shows the correspondence between a fisheye image and a hemisphere in a three-dimensional virtual space. [Figure 4] This figure shows the positions of the virtual camera and the region where perspective projection transformation is performed in the three-dimensional virtual space of the hemispherical image. [Figure 5] This flowchart shows the viewpoint movement operation during perspective projection display in Example 1. [Figure 6] This figure shows the perspective projection transformation in the hemispherical image of Example 1. [Figure 7]It is an explanatory diagram showing the difference between absolute position specification and relative position specification. [Figure 8] It is a flowchart showing the viewpoint movement operation during perspective projection display in Example 2. [Figure 9] It is a diagram showing perspective projection display by relative position specification in Example 2. [Figure 10] It is a diagram showing perspective projection display in the gyro mode and combined mode of Example 3. [Figure 11] It is a flowchart showing the viewpoint movement operation during perspective projection display in Example 3. [Figure 12] It is a flowchart showing the viewpoint movement operation during perspective projection display in Example 4. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In each drawing, the same members are denoted by the same reference numerals, and overlapping explanations are omitted.

[0010] FIG. 1 is a block diagram of a digital camera 100, which is an example of a display device according to an embodiment of the present invention.

[0011] An imaging unit 22 is an imaging element constituted by a CCD, a CMOS element, or the like that converts an optical image into an electrical signal.

[0012] An image processing unit (generation unit) 24 performs perspective projection conversion capable of converting a partial region of a distorted wide-angle image such as a fisheye image captured by the imaging unit 22 into a partial image. The distorted wide-angle image is, for example, an image with an angle of view exceeding 180 degrees, or an image in a projection method such as equidistant projection, central projection, stereographic projection, or orthographic projection. Since perspective projection conversion is performed by setting a viewing angle, a partial region of the wide-angle image is converted to generate a partial image.

[0013] The memory control unit 15 controls transmission and reception of data between the image processing unit 24 and the memory 32. Output data from the imaging unit 22 is written to the memory 32 via the image processing unit 24 and the memory control unit 15. Alternatively, the output data from the imaging unit 22 is written to the memory 32 via the memory control unit 15 without passing through the image processing unit 24.

[0014] The memory 32 stores image data obtained by the imaging unit 22, and image data to be displayed on the display unit 28 and the EVF 29. The memory 32 has a storage capacity sufficient to store a predetermined number of still images, moving images and audio of a predetermined duration. The memory 32 also serves as an image display memory (video memory).

[0015] Display image data written to the memory 32 is displayed by the display unit 28 or the EVF 29 via the memory control unit 15. Each of the display unit 28 and the EVF 29 performs display in accordance with signals from the memory control unit 15 on a display device such as an LCD or an organic EL. Live view display (LV) can be implemented by sequentially transferring data accumulated in the memory 32 to the display unit 28 or the EVF 29 for display. Hereinafter, an image displayed in live view display is referred to as a live view image (LV image).

[0016] The line-of-sight detection unit 160 detects a user's line of sight at the eyepiece unit 16 of the finder. The line-of-sight detection unit 160 includes a dichroic mirror 162, an imaging lens 163, a line-of-sight detection sensor 164, a line-of-sight detection circuit 165, and an infrared light emitting diode 166. Since the system control unit (control unit) 50 can also execute predetermined processing in accordance with detected line of sight, it can also be said that the line-of-sight detection unit 160 is a part of the operation unit 70.

[0017] The line-of-sight detection unit 160 detects line of sight by a method called the corneal reflection method. The corneal reflection method is a method of detecting the direction and position of line of sight from the positional relationship between the reflected light of infrared light emitted from the infrared light emitting diode 166 from the cornea of the eyeball 161 and the pupil of the eyeball 161. A method other than the corneal reflection method may be used as long as the direction and position of line of sight can be detected.

[0018] The infrared light-emitting diode 166 is a light-emitting element for detecting the direction and position of the user's gaze within the viewfinder, and it illuminates the user's eyeball 161 with infrared light. The infrared light emitted from the infrared light-emitting diode 166 is reflected by the eyeball 161, and the infrared reflected light from the eyeball 161 reaches the dichroic mirror 162. The dichroic mirror 162 reflects only infrared light and transmits visible light. The infrared reflected light, whose optical path has been altered, is imaged onto the imaging surface of the gaze detection sensor 164 via the imaging lens 163. The imaging lens 163 is an optical component that constitutes the gaze detection optical system. The gaze detection sensor 164 is composed of an imaging device such as a CCD type image sensor.

[0019] The gaze detection sensor 164 converts the incident infrared reflected light into an electrical signal via photoelectric conversion and outputs it to the gaze detection circuit 165. Based on the output signal from the gaze detection sensor 164, the gaze detection circuit 165 detects the user's gaze position from the movement of the eyeball 161 and outputs the detection information to the system control unit 50.

[0020] The system control unit 50 determines that the user is fixated on a specific area if the time the user's gaze position remains fixed in that area exceeds a predetermined time, based on the detection information obtained from the gaze detection circuit 165. Therefore, the specific area can be said to be the position the user is fixated on (the fixation position). Note that "the gaze position is fixed in a specific area" means, for example, that for the duration of the predetermined time, the average position of the gaze movement is within the specific area, and the variation (variance) is less than a predetermined value. The predetermined time can be arbitrarily changed by the system control unit 50.

[0021] The non-volatile memory 56 is an electrically erasable and recordable memory, such as a Flash-ROM. Constants for the operation of the system control unit 50 and programs are stored in the non-volatile memory 56. Here, the program refers to a program for executing various flowcharts described later in this embodiment.

[0022] The system control unit 50 is a control unit consisting of at least one processor or circuit, and controls the entire digital camera 100. The system control unit 50 implements each of the processes described in this embodiment by executing a program recorded in the non-volatile memory 56. The system memory 52 is, for example, RAM, and the system control unit 50 loads constants, variables for the operation of the system control unit 50, and the program read from the non-volatile memory 56 into the system memory 52. ​​The system control unit 50 also performs display control by controlling the memory 32, the display unit 28, etc.

[0023] The recording medium 200 is a recording medium such as a memory card for recording captured images, and is composed of semiconductor memory, magnetic disks, etc.

[0024] The communication unit 54 transmits and receives video and audio signals to and from external devices connected wirelessly or via wired cables. The communication unit 54 can also connect to wireless LAN (Local Area Network) and the internet. Furthermore, the communication unit 54 can communicate with external devices using Bluetooth® and Bluetooth Low Energy. The communication unit 54 can transmit images (including LV images) captured by the imaging unit 22 and images recorded on the recording medium 200, and can receive image data and other various information from external devices.

[0025] The attitude detection unit 55 detects the attitude of the digital camera 100 relative to the direction of gravity. An accelerometer, gyroscope, or the like can be used as the attitude detection unit 55. It is also possible to use an accelerometer or gyroscope to detect the tilt and movement of the digital camera 100 (pan, tilt, lift, and whether it is stationary or not). Furthermore, a level indicator based on the detected tilt may be displayed on the display unit 28 or EVF 29.

[0026] The eyepiece detection unit 57 is, for example, an infrared proximity sensor that detects the approach (eye-catching) and departure (eye-separation) of an eyeball (object) 161 to the eyepiece 16 of the viewfinder which has a built-in EVF 29. When an object approaches, infrared light emitted from the light-emitting unit of the eyepiece detection unit 57 is reflected by the object and received by the light-receiving unit. Based on the amount of infrared light received, the eyepiece detection unit 57 can also determine how close the object is to the eyepiece 16 (eye-catching distance). In this way, the eyepiece detection unit 57 performs eye-catching detection to detect the proximity distance of an object to the eyepiece 16. The eyepiece detection unit 57 detects that an object has been caught in the eye when it is detected approaching the eyepiece 16 within a predetermined distance from a non-eye-catching state (non-approaching state). Furthermore, the eyepiece detection unit 57 detects that an object has been separated from the eye when it has moved away from the eye-catching state (approaching state) beyond a predetermined distance. The threshold for detecting eye contact and the threshold for detecting eye separation may be different, for example, by providing hysteresis. After detecting eye contact, the device is considered to be in an eye-contact state until eye separation is detected. After detecting eye separation, the device is considered to be in a non-eye-contact state until eye contact is detected again. Note that the infrared proximity sensor is just one example; other sensors may be used as the eye-contact detection unit 57 as long as they can detect the approach of an eye or object that can be considered as eye contact.

[0027] The system control unit 50 switches the display (display state) / hidden (hidden state) of the display unit 28 and EVF 29 according to the state detected by the eyepiece detection unit 57. Specifically, at least in the shooting standby state and when the display destination switching is automatic, when not using an eyepiece, the display destination is set to the display unit 28 and the display is turned on, and the EVF 29 is hidden. When using an eyepiece, the display destination is set to the EVF 29 and the display is turned on, and the display unit 28 is hidden.

[0028] The system control unit 50 can detect the following operations or states on the eyepiece unit 16 by using detection information acquired from the gaze detection circuit 165 or by controlling the eyepiece detection unit 57. (1) The line of sight, which was not previously directed towards the eyepiece 16, is now directed towards the eyepiece 16. In other words, eye-line input begins. (2) The eyepiece 16 is in a state where eye-line input is being received. (3) The eyepiece 16 is being fixed on. (4) The line of sight that was directed towards the eyepiece 16 is taken away. In other words, the eye-line input is terminated. (5) No eye-line input is being sent to the eyepiece 16.

[0029] These operations and states, as well as the position (direction) of the line of sight directed towards the eyepiece 16, are notified to the system control unit 50 via the internal bus. Based on the notified information, the system control unit 50 determines what kind of operation (line of sight operation) was performed on the eyepiece 16.

[0030] The control unit 70 accepts various operations. Operations received by the control unit 70 can be transmitted to the image processing unit 24 via the internal bus. In addition, the user can adjust and set the display range and position of the generated partial image through operations performed via the control unit.

[0031] In this embodiment, the image processing unit 24, the system control unit 50, and the gaze detection unit 160 constitute a control device for controlling the display device.

[0032] The method for generating the hemispherical image used when viewing VR (Virtual Reality) in this embodiment will be explained below with reference to Figure 2.

[0033] Figure 2(a) shows an image captured when a fisheye lens is used with the digital camera 100. As shown in Figure 2(a), the fisheye image, when captured by the imaging unit 22, is a distorted image cropped into a circle. The image processing unit 24 uses, for example, a 2D / 3D computer graphics library such as OpenGL (Open Graphics Library) to draw the hemisphere shown in Figure 2(b) and pastes the fisheye image so as to cover the inside of it.

[0034] Specifically, the fisheye image is first mapped to a coordinate system consisting of a vertical angle θ with the zenith direction as the axis of the captured image, as shown in Figure 3, and a horizontal angle φ around the zenith direction axis (with the circumferential direction as the axis). In this case, if the field of view of the fisheye image is 180°, the vertical angle θ and the horizontal angle φ will be in the range of -90° to 90°. The position (θ,φ), which is the coordinate value of the fisheye image, can be mapped to each point on the sphere representing the hemispherical image shown in Figure 2(b). As shown in Figure 2(b), when the center of the hemisphere is 0 and the three-dimensional coordinates on the sphere are (X,Y,Z), the three-dimensional coordinates (X,Y,Z) can be expressed in relation to the two-dimensional coordinates of the fisheye image by the following equations (1) to (3).

[0035] X = rcosθsinφ (1) Y=rsinθ (2) Z = rcosθcosφ (3) Here, r is the radius of the hemisphere. Based on the coordinate correspondence expressed by equations (1) to (3), a hemispherical image can be generated in a three-dimensional virtual space by pasting a fisheye image onto the inside of a hemisphere.

[0036] By acquiring fisheye images of 180° in front of and behind the photographer, generating hemispherical images using the method described above, and stitching them together, a 360° full-sphere image can be generated.

[0037] As mentioned above, since full-sphere and hemispherical images are images pasted over a sphere, they can appear unnatural if left as they are. Therefore, by applying a predetermined projection transformation to a portion of the image and displaying it as a partially distorted image, it is possible to generate an image that does not appear unnatural.

[0038] Below, we will explain perspective projection transformation as an example of projection transformation, but other transformation processes may be used as long as they allow for partial display of full-sphere and hemispherical images.

[0039] Figure 4 shows the positions of the virtual camera and the region where perspective projection transformation is performed in the three-dimensional virtual space of the hemispherical image. The virtual camera corresponds to the viewpoint of the user viewing the 360-degree image, which is displayed as a three-dimensional sphere. The region where perspective projection transformation is performed is determined by the information (θ,φ) indicating the direction of the virtual camera and the field of view, and the image of this region is displayed on the display unit 28 or EVF 29.

[0040] The following describes the viewpoint movement operation of the digital camera 100 during perspective projection display in each embodiment. [Examples]

[0041] Figure 5 is a flowchart showing the viewpoint movement operation during perspective projection display of the digital camera 100 in this embodiment. Each process in the flowchart of Figure 5 is realized by the system control unit 50 loading the program stored in the non-volatile memory 56 into the system memory 52 and executing it, thereby controlling each functional block. Figure 6 is a diagram showing the perspective projection transformation in a hemispherical image in this embodiment.

[0042] As shown in Figure 6(a), a partial image of a hemispherical image, transformed by the image processing unit 24 into a perspective projection image, is displayed on the EVF 29. The flowchart in Figure 5 is initiated when the system control unit 50 detects that the user has placed their eyepiece 16, as shown in Figure 6(b).

[0043] In step S501, the system control unit 50 acquires information (θ,φ) indicating the direction of the user's gaze from the gaze detection unit 160 using the corneal reflection method. Here, θ is the elevation angle of the gaze, and φ is the circumferential angle of the gaze. The system control unit 50 uses equations (1) to (3) to acquire the user's gaze position (X,Y,Z) (first position) on the spherical surface of the hemispherical image shown in Figure 6(b) from the information (θ,φ). In this embodiment, the system control unit 50 and the gaze detection unit function as detection units that detect the position of the user's gaze on the display surface. In the initial state before the gaze position is determined in step S502, which will be described later, the information (θ,φ) is set to (0,0), and the gaze position (X,Y,Z) is set accordingly.

[0044] In step S502, the system control unit 50 determines whether a predetermined time has elapsed for the gaze position acquired in step S501 (gaze time; the time for which the gaze position is fixed (located within a predetermined area)). If the gaze time has elapsed, the process in step S503 is executed; otherwise, the process in step S501 is executed.

[0045] In step S503, the system control unit 50 controls the image processing unit 24 to perform a perspective projection transformation centered on the line of sight position acquired in step S501. As a result, a partial image (first partial image) centered on the line of sight position is generated, as shown in Figure 6(c).

[0046] As described above, the configuration of this embodiment allows the user to move their viewpoint when displaying a portion of a wide-angle image without having to touch the operating member.

[0047] In this embodiment, absolute position specification is performed as described in Embodiment 2, but relative position specification may be used instead of absolute position specification. [Examples]

[0048] In Example 1, as shown in Figure 7(a), when a user fixates on a specific location, perspective projection transformation is performed centered on the actual location of the fixation. However, since there are limits to the angle at which a user can direct their gaze, it is possible that the image may be difficult or impossible to see depending on the angle of the gaze. Therefore, in this example, the method of specifying the perspective projection display position is switched according to the angle of the gaze.

[0049] If the user's line of sight angle is above a certain level, the range of perspective projection in the direction of gaze is continuously shifted by a predetermined amount at predetermined time intervals, as shown in Figure 7(b). Hereinafter, for convenience, the method of specifying that perspective projection is centered on the actual gaze position, as explained in Example 1, will be referred to as absolute position specification, and the method of specifying that the perspective projection display range is continuously shifted in the direction of gaze, as explained in this embodiment, will be referred to as relative position specification.

[0050] Figure 8 is a flowchart showing the viewpoint movement operation during perspective projection display in this embodiment. The processes in steps S801 and S802 in Figure 8 are the same as those in steps S501 and S502 in Figure 5, respectively, so their explanation is omitted.

[0051] In step S803, the system control unit 50 determines whether the absolute value of the elevation angle θ of the line of sight, obtained in step S801, exceeds a first predetermined angle, or whether the absolute value of the circumferential angle φ of the line of sight exceeds a second predetermined angle. The first and second predetermined angles are set to angles that the user performing the eye input finds difficult to see. An "angle that the user finds difficult to see" is, for example, an angle greater than 60°. If it is determined that the absolute value of the elevation angle θ of the line of sight exceeds the first predetermined angle, or the absolute value of the circumferential angle φ of the line of sight exceeds the second predetermined angle, the process in step S805 is executed; otherwise, the process in step S804 is executed.

[0052] In step S804, the system control unit 50 performs perspective projection display by absolute position specification, which is the same process as in step S503 in Figure 5.

[0053] In step S805, the system control unit 50 performs perspective projection display by relative position specification based on the position of the line of sight, which is determined by the elevation angle θ of the line of sight and the circumferential angle φ of the line of sight, acquired in step S801.

[0054] Figure 9 shows a perspective projection display with relative position specification. Assume that the initial perspective projection display position is determined by information (θ0, φ0), as shown in Figure 9(a). Figure 9(b) shows the information (θ', φ') when the information (θ0, φ0) is moved by a predetermined amount relative to the original information (θ', φ0). Here, the information (θ', φ') can be expressed by the following equations (4) and (5), using the information (θ, φ) acquired in step S801.

[0055] θ' = θ - θ0 (4) φ' = φ - φ0(5) Furthermore, if we let (Δθ´,Δφ´) be the value obtained by dividing the information (θ´,φ´) by a predetermined value n, then from equations (1) to (3), we can obtain the three-dimensional coordinates (X1,Y1,Z1) (second position) on the sphere of the hemispherical image corresponding to the information (θ0+Δθ´,φ0+Δφ´). Then, as shown in Figure 9(c), a perspective projection display is performed centered on the three-dimensional coordinates (X1,Y1,Z1). Next, after a predetermined time, as shown in Figure 9(d), a perspective projection display is performed centered on the three-dimensional coordinates (X2,Y2,Z2) corresponding to the position (θ0+2Δθ´,φ0+2Δφ´).

[0056] The above process is performed until it is determined in step S806, described later, that it has been executed n times. When this process is executed n times, the position of the perspective projection display will match the three-dimensional coordinates corresponding to the acquired information (θ,φ).

[0057] In step S806, the system control unit 50 determines whether the process in step S805 has been executed n times. If it is determined that it has been executed n times, the process in step S807 is executed; otherwise, the process in step S805 is executed.

[0058] In step S807, the system control unit 50 adds the value (Δθ, Δφ) obtained by dividing the acquired information (θ, φ) by a predetermined value m to the angle of the previous perspective projection position, as shown in Figure 9(e), and performs a perspective projection display centered on the three-dimensional position corresponding to that angle.

[0059] In step S808, the system control unit 50 determines whether the angle of the perspective projection display position is a constant value. Here, a constant value is a value that exceeds the display range of the hemispherical image, for example, an angle of ±90° in the zenith direction or circumferential direction. If it is determined that the angle of the perspective projection display position is a constant value, the process in step S809 is executed; otherwise, the process in step S807 is executed.

[0060] In step S809, the system control unit 50 fixes the position of the perspective projection display at a three-dimensional position corresponding to the angle determined in step S808, as shown in Figure 9(f), so as not to display any portion of the hemispherical image that exceeds the display range.

[0061] As described above, with the configuration of this embodiment, even when displaying only a portion of a wide-angle image, it is possible to easily shift the viewpoint to check hard-to-see areas around the edges of the image. [Examples]

[0062] In Example 1, viewpoint movement was limited to eye-tracking input. However, in this embodiment, viewpoint movement can be performed using eye-tracking input, a gyroscope, or a combination of both during playback image display. Note that during shooting, viewpoint movement using the gyroscope is not possible because it would cause the subject to shift from the position intended by the user.

[0063] In gyro-based viewpoint movement operations, as shown in Figure 10(a), the position of the perspective projection display (third position) is determined according to the tilt of the main unit. Since the tilt and pan values ​​of the digital camera 100 detected by the attitude detection unit 55 correspond to the vertical angle θ and horizontal angle φ in Figure 4, respectively, the three-dimensional coordinates in the hemispherical image can be obtained from equations (1) to (3). By performing perspective projection display centered on the obtained three-dimensional coordinates, viewpoint movement operations using the gyro become possible.

[0064] In viewpoint movement operations using both eye-tracking input and gyroscope, as shown in Figure 10(b), the position of the perspective projection display is determined by adding the tilt of the digital camera 100 and the angle of the line of sight. Here, the tilt of the digital camera 100 detected by the attitude detection unit 55 is denoted as information (θ1, φ1), and the information detected by the eye-tracking detection unit 160 is denoted as (θ2, φ2). At this time, the position of the perspective projection display (fourth position) is determined by the sum of these pieces of information (θ1+θ2, φ1+φ2). Hereafter, for convenience, viewpoint movement operations using eye-tracking input will be referred to as the eye-tracking mode, viewpoint movement operations using the tilt of the camera body as the gyroscope mode, and viewpoint movement operations using both in combination as the combined mode.

[0065] Figure 11 is a flowchart showing the viewpoint movement operation during perspective projection display in this embodiment.

[0066] In step S1101, the system control unit 50 acquires the mode for performing the viewpoint movement operation selected by the user through various operations performed on the operation unit 70.

[0067] In step S1102, the system control unit 50 determines whether it is a playback image display. If it is determined to be a playback image display, the process in step S1103 is executed. If it is determined not to be a playback image display, i.e., if it is determined to be an LV image display, the process in step S1106 is executed.

[0068] The processes in steps S1103 to S1105 in Figure 11 are the same as the processes in steps S501 to S503 in Figure 5, so their explanation is omitted.

[0069] In step 1106, the system control unit 50 determines whether the mode acquired in step S1101 is the gaze mode. If it is determined that the acquired mode is the gaze mode, the process in step S1103 is executed. If it is determined that the mode is not the gaze mode, i.e., if it is determined that the acquired mode is the gyro mode or the combined mode, the process in step S1107 is executed.

[0070] In step S1107, the system control unit 50 selects the center position of the perspective projection display range according to the tilt of the digital camera 100 detected by the attitude detection unit 55. Then, the process in step S1105 is executed. In step S1105, a perspective projection transformation is performed around the three-dimensional coordinates in the hemispherical image corresponding to the selected position, and a perspective projection display centered on the selected position is performed, as shown in Figure 10(a).

[0071] In step S1108, the system control unit 50 determines whether the acquired mode is gyro mode. If it is determined that the selected mode is gyro mode, this flow is terminated. If it is determined that the acquired mode is not gyro mode, i.e., a combined mode, the process in step S1103 is executed. In the process in step S1103, the position is determined by adding the tilt of the camera body and the angle of the line of sight, and a perspective projection display is performed centered on the determined position.

[0072] As described above, according to the configuration of this embodiment, when a portion of a wide-angle image is displayed, the user can select a preferred viewpoint movement operation method. [Examples]

[0073] VR photos and videos include those that can be viewed stereoscopically by utilizing binocular parallax. In this case, a wide-angle image for the right eye and a wide-angle image for the left eye are acquired separately. However, the digital camera 100 can only perform perspective projection display on one of the wide-angle images. Therefore, in this embodiment, the right-eye image and the left-eye image (images captured from multiple viewpoints) displayed on the EVF 29 are switched based on the user's intentional eye closing. In this embodiment, the gaze detection unit 160 can determine the user's intentional eye closing.

[0074] Figure 12 is a flowchart showing the viewpoint movement operation during perspective projection display in this embodiment.

[0075] In step S1201, the system control unit 50 controls the gaze detection unit 160 to determine whether the user's eye-closing time in the EVF 29 exceeds a first predetermined time. Since unintentional blinking by the user may cause a false determination, it is necessary to determine if the user is intentionally closing their eyes. "Intentional eye-closing" refers, for example, to a change from an open eye state to a closed eye state, where the closed eye state continues for a first predetermined time (e.g., 300ms) or longer. If it is determined that the eye-closing time exceeds the first predetermined time, the process in step S1202 is executed; otherwise, the process in step S1203 is executed.

[0076] In step S1202, the system control unit 50 controls the image processing unit 24 to switch between the right-eye image and the left-eye image. If perspective projection was initially performed on the wide-angle image for the right eye, the switch can be performed by applying the hemispherical image generation method described above again to the wide-angle image for the left eye.

[0077] The processes in steps S1203 to S1205 in Figure 12 are the same as the processes in steps S501 to S503 in Figure 5, so their explanation is omitted.

[0078] As described above, according to the configuration of this embodiment, when a portion of a wide-angle image is displayed, the user can switch between the right-eye image and the left-eye image without touching the operating member.

[0079] The condition for switching between the right and left eye images is set to the duration of eye closure exceeding a first predetermined time. However, other conditions may be used as long as they can determine the user's intentional eye-closing action. For example, whether the number of times the eyes are closed exceeds a predetermined number may be used. Specifically, this means that the eyes change from an open state to a closed state, the closed state continues for a second predetermined time (e.g., 100 ms) or more, and the same condition is met again within a fourth predetermined time (e.g., 1 s) (two consecutive eye closures).

[0080] Furthermore, the condition is not limited to the condition that the user closes their eyes; any condition that can be intentionally met without user interaction is acceptable.

[0081] In this embodiment, the present invention has been described using the application of a digital camera 100 as an example, but the present invention is not limited thereto. It can be applied to any display device having an image processing function and a gaze detection function, which displays a partial image based on a part of an image on its display surface. In other words, it can also be applied to a head-mounted display (HMD) capable of gaze detection. By reading a recording medium on which a wide-angle image has been recorded and generating a full-spherical or hemispherical image using the image processing function, perspective projection display can be performed when displaying a playback image. Furthermore, by connecting the HMD to the communication unit of the imaging device wirelessly or via a wired cable, perspective projection display can be performed on the HMD even during LV display. [Other examples] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0082] This embodiment includes the following configurations and methods.

[0083] (Composition 1) A control device for controlling a display device that displays a partial image based on a portion of an image on its display surface, A generation unit that generates the partial image by applying a transformation process to a portion of the aforementioned image, A detection unit for detecting the first position of the user's line of sight on the display surface, A control device characterized by having a control unit that, when the time for which the first position is located within a predetermined area has elapsed, causes the generation unit to generate a first partial image based on the first position, and displays the first partial image on the display surface. (Configuration 2) The control device according to configuration 1, characterized in that the control unit causes the generation unit to generate the first partial image centered on the first position. (Composition 3) The control device according to configuration 1, characterized in that the control unit generates a second partial image centered on a second position which is moved by a predetermined amount from the original position in the direction of the first position at predetermined intervals, and displays the second partial image on the display surface. (Composition 4) The control device according to any one of configurations 1 to 3, characterized in that the aforementioned image is an image recorded and reproduced by the recording unit. (Composition 5) The control device according to configuration 4, characterized in that, when the control device is set to a mode that generates the partial image based on a third position on the display surface corresponding to the amount of tilt of the display device, it generates a third partial image centered on the third position. (Composition 6) The control device according to configuration 4 or 5, characterized in that, when the control device is set to a mode that generates the partial image based on the first position and a third position on the display surface corresponding to the tilt amount of the display device, the generation unit generates a fourth partial image centered on a fourth position based on the first position and the third position. (Composition 7) The control device according to any one of configurations 1 to 6, characterized in that the aforementioned image is a live view image captured by an imaging device. (Composition 8) The control device according to any one of configurations 1 to 7, characterized in that the conversion process is a perspective projection conversion. (Composition 9) The control device according to any one of configurations 1 to 8, characterized in that the aforementioned image is a 360-degree spherical image. (Composition 10) The control device according to any one of configurations 1 to 8, characterized in that the aforementioned image is a hemispherical image. (Composition 11) The control device according to any one of configurations 1 to 10, characterized in that the detection unit is capable of detecting the angle of the user's line of sight corresponding to the first position. (Composition 12) The control device according to configuration 11, characterized in that when the angle is smaller than a predetermined amount, the control unit causes the generation unit to generate a first partial image centered on the first position, and when the angle is larger than a predetermined amount, it causes the generation unit to generate a second partial image centered on a second position moved by a predetermined amount from the original position in the direction of the first position at predetermined intervals. (Composition 13) The control device according to any one of configurations 1 to 12, characterized in that the generation unit is capable of converting the coordinates of the first position into coordinates in the image. (Composition 14) The control device according to any one of configurations 1 to 13, characterized in that the aforementioned partial image is an image generated by applying a transformation process to a portion of an image taken from one viewpoint among images taken from multiple viewpoints. (Composition 15) The control device according to configuration 14, characterized in that the images captured from multiple viewpoints are right-eye images and left-eye images. (Composition 16) The control device according to configuration 14 or 15, characterized in that the control unit is capable of switching the partial image. (Composition 17) The control device according to any one of configurations 14 to 16, characterized in that the detection unit is capable of determining whether the user is intentionally closing their eyes. (Composition 18) The detection unit is capable of determining whether the user is intentionally closing their eyes. The control device according to any one of configurations 14 to 17, characterized in that the control unit switches the partial image when the user's intentional eye-closing time has elapsed. (Composition 19) The detection unit is capable of determining whether the user is intentionally closing their eyes. The control device according to any one of configurations 14 to 17, characterized in that the control unit switches the partial image when the number of times the user intentionally closes their eyes exceeds a predetermined number of times. (Composition 20) A control device described in any one of configurations 1 to 19, A display device characterized by having a display unit. (Method 1) A control method for controlling a display device that displays a partial image based on a portion of an image on its display surface, The steps include generating the partial image by applying a transformation process to a portion of the aforementioned image, The steps include detecting the first position of the user's line of sight on the display surface, A control method characterized by comprising the steps of: generating a first partial image based on the first position and displaying the first partial image on the display surface when a predetermined time has elapsed while the first position is located within a predetermined area. (Composition 21) A program characterized by causing a computer to execute the control method described in Method 1.

[0084] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of its gist. [Explanation of Symbols]

[0085] 24 Image Processing Unit (Generation Unit) 50 System Control Unit (Detection Unit, Control Unit) 100 Digital Cameras (Display Devices) 160 Eye-line detection unit (detection unit)

Claims

1. A control device for controlling a display device that displays a partial image based on a portion of an image on its display surface, A generation unit that generates the partial image by applying a transformation process to a portion of the aforementioned image, A detection unit that detects a first position of the user's line of sight on the display surface and the angle of the user's line of sight corresponding to the first position, If the time during which the first position is located within the predetermined area elapses, If the angle is smaller than a predetermined angle, the generation unit generates a first partial image centered on the first position, and displays the first partial image on the display surface. The system includes a control unit that, when the angle is greater than the predetermined angle, generates a second partial image centered on a second position which is moved by a predetermined amount from the original position in the direction of the first position at predetermined intervals, and displays the second partial image on the display surface. The control device is characterized in that the angle of sight is defined as 0 degrees when the first position is at the center of the partial image.

2. The control device according to claim 1, characterized in that the control unit causes the generation unit to generate the first partial image centered on the first position.

3. The control device according to claim 1 or 2, characterized in that the aforementioned image is an image recorded and reproduced by the recording unit.

4. The control device according to claim 1 or 2, characterized in that the aforementioned image is a live view image captured by an imaging device.

5. The control device according to claim 1 or 2, characterized in that the conversion process is a perspective projection conversion.

6. The control device according to claim 1 or 2, characterized in that the aforementioned image is a 360-degree spherical image.

7. The control device according to claim 1 or 2, characterized in that the aforementioned image is a hemispherical image.

8. The control device according to claim 1 or 2, characterized in that the generation unit is capable of converting the coordinates of the first position into coordinates in the image.

9. The control device according to claim 1 or 2, characterized in that the aforementioned partial image is an image generated by applying a transformation process to a portion of an image taken from one viewpoint among images taken from multiple viewpoints.

10. The control device according to claim 9, characterized in that the images captured from multiple viewpoints are a right-eye image and a left-eye image.

11. A control device according to claim 1 or 2, A display device characterized by having a display unit.

12. A control method for controlling a display device that displays a partial image based on a portion of an image on its display surface, The steps include generating the partial image by applying a transformation process to a portion of the aforementioned image, A step of detecting a first position of the user's line of sight on the display surface and the angle of the user's line of sight corresponding to the first position, If the time during which the first position is located within the predetermined area elapses, If the angle is smaller than a predetermined angle, a first partial image centered on the first position is generated, and the first partial image is displayed on the display surface. If the angle is greater than the predetermined angle, the system includes the steps of generating a second partial image centered on a second position moved by a predetermined amount from the original position in the direction of the first position at predetermined intervals, and displaying the second partial image on the display surface. The control method is characterized in that the angle of sight is defined as 0 degrees when the first position is at the center of the partial image.

13. A program characterized by causing a computer to execute the control method described in claim 12.

Images