Electronic device, control method of electronic device and storage medium

Abstract

The present invention relates to an electronic device, a control method of the electronic device, and a storage medium. The electronic device includes: an acquisition unit configured to acquire an image set including a first image and a second image having parallax with each other; and a control unit configured to control so that: 1) in a first display mode in which the first image is to be displayed but the second image is not to be displayed, the first image is displayed on the display in a manner in which the orientation of the first image is corrected to point to a predetermined orientation, and 2) in a second display mode in which both the first image and the second image are to be displayed, the first image and the second image are displayed on the display in a manner in which the orientation of the first image and the second image is not corrected to point to a predetermined orientation.

Description

Technical Field

[0001] The present invention relates to an electronic device for controlling the display of a VR image, a control method of the electronic device, and a storage medium. Background Art

[0002] A technology for displaying a stereoscopic virtual reality (VR) image by acquiring wide-angle images with parallax using two optical systems and mapping and displaying these images on a virtual sphere is known. A dual-lens VR camera for photographing images with parallax includes two optical systems facing the same direction. Therefore, the dual-lens VR camera photographs two images with parallax through one image photographing operation. As a dual-lens VR camera, in particular, a camera that photographs a wide range of at least 180° (hemispherical, 90° in all directions from the center of the image) vertically and horizontally in each optical system is known.

[0003] In addition, as a VR display method, a "single-lens VR display" is known, which transforms a VR image to be mapped on a virtual sphere and displays the one image. Another known VR display method is a "dual-lens VR display" which displays a VR image for the left eye and a VR image for the right eye side by side in the left area and the right area, respectively.

[0004] Japanese Patent Laid-Open No. 2019-121858 proposes an electronic device that controls zenith correction (correction of the pitch angle and roll angle of the VR camera) and correction of the yaw angle of the VR camera when displaying a VR image captured by the VR camera so that a user can easily observe the VR image. Here, the electronic device controls zenith correction and the like according to conditions such as the installation position, movement speed, and shake of the VR camera.

[0005] If zenith correction is not performed, the zenith in the VR image and the user's perception of it deviate, which makes it difficult to recognize and observe the VR image. In some cases, unless zenith correction is performed, it becomes difficult to operate using the VR image.

[0006] When a VR image is captured in a state where a dual-lens VR camera is tilted, the VR image is captured in a state where the height of the left optical system and the height of the right optical system are different. Therefore, parallax in the direction of gravity (vertical direction) is generated between the left and right subject images formed on the sensor. Performing zenith correction on the VR image (dual-lens VR image) captured in this way can correct the tilt of the VR image, but cannot eliminate the parallax in the direction of gravity, which can cause interference in the recognition of stereoscopic objects. As a result, the user perceives these VR images as double images. Such VR images that make the user perceive a double image may cause the user to experience symptoms such as headaches, dizziness, and nausea, symptoms of the so-called VR disease. In this way, if zenith correction is performed when a dual-lens VR display is performed for a dual-lens VR image captured in a tilted state, a double image that causes discomfort may be generated. This problem is not considered in Japanese Patent Laid-Open No. 2019-121858. Summary of the invention

[0007] The present invention is to provide an electronic device capable of appropriately displaying VR images captured by a plurality of optical systems.

[0008] One aspect of the present invention is:

[0009] An electronic device, comprising:

[0010] An acquisition unit configured to acquire an image set including a first image and a second image having parallax with each other; and

[0011] A control unit configured to control:

[0012] 1) in the case of a first display mode in which the first image is to be displayed but the second image is not to be displayed, displaying the first image on a display in such a manner that the orientation of the first image is corrected to point to a predetermined orientation, and

[0013] 2) In the case of the second display mode in which both the first image and the second image are to be displayed, the first image and the second image are displayed on the display without correcting their orientations to point to the predetermined orientation.

[0014] Another aspect of the present invention is:

[0015] A method for controlling an electronic device, the method comprising:

[0016] An acquisition step for acquiring an image set including a first image and a second image having a parallax with each other;

[0017] A first control step for controlling so that: in a first display mode in which the first image is displayed but the second image is not displayed, the first image is displayed on a display in such a manner that the orientation of the first image is corrected to point to a predetermined orientation; and

[0018] The second control step is used to control so that: in the second display mode of displaying both the first image and the second image, the first image and the second image are displayed on the display in a manner not correcting the orientation of the first image and the second image to point to the predetermined orientation.

[0019] Another aspect of the present invention is:

[0020] A computer-readable storage medium stores a program for causing a computer to execute the control method.

[0021] Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1A and Figure 1B is a diagram for explaining a VR camera according to an embodiment;

[0023] FIG. 2A to FIG. 2C is a diagram for explaining a display control device according to an embodiment;

[0024] FIG. 3A to FIG. 3C is a diagram for explaining zenith correction according to the embodiment;

[0025] 4A to 4F is a diagram for explaining a display mode according to an embodiment; and

[0026] Figure 5 is a flowchart depicting a process of display control according to an embodiment. DETAILED DESCRIPTION

[0027] Embodiments of the present invention will be described with reference to the accompanying drawings. First, a VR image will be described, and then a display control device according to the embodiment will be described.

[0028] VR Graphics

[0029] VR images are images that can be VR displayed (displayed in a "VR view" display mode). VR images include omnidirectional images (spherical images) taken by an omnidirectional camera (spherical camera), and panoramic images whose image range (effective image range) is wider than the display range of images that can be displayed on the display unit at one time. VR images are not only still images, but also include moving images and live view images (images acquired from the camera in almost real time). VR images have a field of view of up to 360° in the vertical direction (vertical angle, angle relative to the zenith, elevation angle, depression angle, attitude angle, pitch angle) and an image range (effective image range) of up to 360° in the horizontal direction (horizontal angle, azimuth angle, yaw angle).

[0030] Even if the field of view is less than 360° in the vertical direction and less than 360° in the horizontal direction, a VR image includes an image having a wider angle of view (field of view) than the angle of view that can be photographed (photographed) by a standard camera, or an image range (effective image range) wider than the display range that can be displayed on the display unit at one time. For example, an image photographed by an omnidirectional camera that can photograph an image of a subject with a field of view (field of view) of 360° in the horizontal direction (horizontal angle, azimuth angle) and a vertical angle of 210° with the zenith as the center is a VR image. In addition, for example, an image photographed by a camera that can photograph an image of a subject with a field of view (field of view) of 180° in the horizontal direction (horizontal angle, azimuth angle) and a vertical angle of 180° with the horizontal direction as the center is a VR image. In other words, an image range in a field of view of at least 160° (±80°) in the vertical and horizontal directions, respectively, and an image range wider than the range that humans can visually recognize at one time is a VR image.

[0031] When the VR image is VR displayed (displayed in the "VR view" display mode), the seamless all-round image in the horizontal direction (horizontal rotation direction) can be observed (watched) by changing the posture of the display device (display device displaying the VR image) in the horizontal rotation direction. In the vertical direction (vertical rotation direction), the all-round image can be observed within the range of ±105° from directly above (zenith), but the range beyond 105° from directly above is a blank area where no image exists. The VR image can be regarded as "an image whose image range is at least a part of the virtual space (VR space)".

[0032] VR display (VR view) is a display method for displaying an image in a field of view corresponding to the posture of the display device in a VR image, and is a display method (display mode) in which the display range can be changed. In the case where a head-mounted display (HMD) as a display device is mounted on the head to observe an image, an image in a field of view corresponding to the user's facial direction is displayed. For example, it is assumed that at a specific timing, an image with a field of view angle (viewing angle) of 0° (a specific orientation, such as north) in the horizontal direction and 90° (90° from the zenith, i.e., horizontal) in the vertical direction is displayed in the VR image. If, in this state, the posture of the display device is reversed front and back (for example, the display surface is changed from facing south to facing north), within the same VR image, the display range is changed to an image with a field of view angle of 180° (opposite orientation, such as south) in the horizontal direction and 90° (horizontal) in the vertical direction at the center. In the case where the user is observing the HMD, if the user turns his face from north to south (i.e., if the user looks back), the image displayed on the HMD also changes from an image facing north to an image facing south. Through this VR display, the user can be provided with a visual sensation (immersion) as if the user is in the VR image (VR space). A smartphone mounted in VR goggles (head mounted adapter) can be regarded as a type of HMD.

[0033] The display method of VR images is not limited to the above method. The display range may be moved (scrolled) not by a change in posture but by a user operation on a touch panel and a direction button, etc. In VR display (in the display mode "VR view"), the display range can be changed not only by a change in posture but also by a touch movement on the touch panel, a drag operation using a mouse, and pressing a direction button, etc.

[0034] An example of a camera (imaging device) that can capture an image of a subject in a field of view (angle of view) that is 180° in the horizontal direction (horizontal angle, azimuth angle) and 180° in the vertical angle with the horizontal direction as the center is Figure 1A The VR camera 100 shown in FIG. 1 is a VR camera 100. When the user presses the shutter button 103, the VR camera 100 can capture a 180° image in front of the camera via the lenses 101 and 102. If the VR camera 100 is used, images of the subject can be captured from two positions (two optical systems) of the lenses 101 and 102, so two VR images (as a set) with parallax can be acquired simultaneously. Here, in the state of the VR camera 100 where the lenses 101 and 102 are horizontally positioned, as shown in FIG. Figure 1A As shown in FIG. 1 , the vertical direction 104 of the VR camera 100 is consistent with the gravity direction 105. However, in a state where the lens 101 and the lens 102 are not horizontally positioned in the VR camera 100, as shown in FIG. Figure 1BAs shown in FIG. 1 , the vertical direction 104 of the VR camera 100 is not consistent with the gravity direction 105. Figure 1B If two VR images captured in the state shown are subjected to zenith correction and dual-lens VR display, a double image may be generated during the image capture due to the influence of parallax in the gravity direction 105 .

[0035] Display Controls

[0036] Figure 2A 1 is an appearance diagram of an example of a display control device 200 as an electronic device. The display control device 200 can acquire a VR image from the VR camera 100 and display the VR image. The display 205 is a display unit that displays images and various information. The display 205 is integrated with a touch panel 206a described later. The display control device 200 can detect a touch operation on the display surface of the display 205. The display control device 200 can perform VR display of a VR image (VR content) on the display 205.

[0037] The operation unit 206 includes a touch panel 206a and operation units 206b, 206c, 206d, and 206e. The operation unit 206b is a power button that receives an operation to switch on / off the power of the display control device 200. The operation unit 206c and the operation unit 206d are volume buttons to increase / decrease the volume of the sound output from the sound output unit 212. The operation unit 206e is a main button to display the main screen on the display 205.

[0038] The sound output terminal 212a is an earphone jack (component) for outputting sound to earphones, an external speaker, etc. The speaker 212b is an internal speaker of the electronic device for outputting sound.

[0039] Figure 2B 2 is a block diagram showing an example of the structure of the display control device 200. The display control device 200 of the present embodiment can be configured using a display device such as a smartphone. Here, a CPU 201, a memory 202, a nonvolatile memory 203, an image processing unit 204, a display 205, an operation unit 206, a storage medium interface 207, an external interface 209, and a communication interface 210 are connected to the internal bus 250. In addition, a sound output unit 212 and a posture detection unit 213 are also connected to the internal bus 250. The units connected to the internal bus 250 are configured so that data can be exchanged with each other via the internal bus 250.

[0040] The CPU 201 is a control unit that controls the display control device 200 as a whole. The CPU 201 includes at least one processor or circuit. The memory 202 includes, for example, a RAM (for example, a volatile memory composed of a semiconductor element). The CPU 201 controls each unit of the display control device 200 according to a program stored in the nonvolatile memory 203, for example. In this case, the CPU 201 uses the memory 202 as a working memory. The nonvolatile memory 203 stores data such as image data and sound data, and various programs for the CPU 201 to operate. The nonvolatile memory 203 is composed of, for example, a flash memory or a ROM.

[0041] Based on the control of the CPU 201, the image processing unit 204 performs various types of image processing on the image stored in the nonvolatile memory 203 or the storage medium 208, the video signal obtained via the external interface 209, and the image obtained via the communication interface 210. The image processing performed by the image processing unit 204 includes A / D conversion processing, D / A conversion processing, encoding processing of image data, compression processing, decoding processing, enlargement / reduction processing (resizing), noise reduction processing, and color conversion processing. The image processing unit 204 also performs various types of image processing including panoramic expansion, mapping processing, and conversion processing on VR images that are wide-range images (including both omnidirectional images and non-omnidirectional images) with wide-range data. The image processing unit 204 can be composed of a dedicated circuit block for performing specific image processing. Depending on the type of image processing, the CPU 201 can perform image processing according to a program without using the image processing unit 204.

[0042] The display 205 displays a GUI screen for configuring an image and a graphical user interface (GUI) based on the control of the CPU 201. The CPU 201 controls each unit of the display control device 200 by generating and outputting a display control signal according to a program. Therefore, each unit of the display control device 200 is controlled to generate a video signal for displaying an image on the display 205 and output the image signal to the display 205. Then, the display 205 displays an image based on the output video signal. The display control device 200 may include only a structure up to an interface for outputting a video signal for displaying an image on the display 205. In this case, the display 205 may be an external monitor (e.g., TV).

[0043] The operation unit 206 is an input device for receiving user operations. The operation unit 206 includes a text information input device (e.g., a keyboard), a pointing device (e.g., a mouse, a touch panel), a button, a dial, a joystick, a touch sensor, a touch pad, etc. The touch panel is a flat device superimposed on the display 205. The touch panel is an input device configured to output coordinate information according to a touch position.

[0044] In the storage medium interface 207, a storage medium 208 (e.g., a memory card, a CD, a DVD) can be installed. Based on the control of the CPU 201, the storage medium interface 207 reads data from the installed storage medium 208 or writes data to the storage medium 208. The external interface 209 is an interface for inputting / outputting video signals and audio signals by connecting an external device (e.g., a VR camera 100) wirelessly or via a cable. The communication interface 210 is an interface for sending / receiving various data such as files and commands by communicating with an external device and the Internet 211.

[0045] The sound output unit 212 outputs sounds of moving images and music data, operation sounds, ring tones, various notification sounds, etc. The sound output unit 212 includes a sound output terminal 212a for connecting headphones, etc., and a speaker 212b. The sound output unit 212 can output sounds via wireless communication.

[0046] The posture detection unit 213 detects (senses) the posture of the display control device 200 relative to the gravity direction and the inclination of the posture relative to each axis of yaw, roll, and pitch. Based on the posture detected by the posture detection unit 213, it can be determined, for example, whether the display control device 200 is kept horizontal or vertical, whether the display control device 200 is turned upward, turned downward, or in a tilted posture. For the posture detection unit 213, at least one of an acceleration sensor, a gyro sensor, a geomagnetic sensor, an orientation sensor, and an altitude sensor, etc., or a combination of a plurality of these sensors can be used.

[0047] The operation unit 206 includes a touch panel 206a. The CPU 201 can detect the following operations on the touch panel 206a or the states thereof.

[0048] A finger or pen that is not touching the touch panel 206a touches the touch panel 206a, that is, touch-down (hereinafter referred to as touch-down) starts.

[0049] A finger or a pen is touching the touch panel 206a (hereinafter referred to as Touch-On).

[0050] The finger or pen is moving while touching the touch panel 206a (hereinafter referred to as touch-move).

[0051] The finger or pen touching the touch panel 206a is released from the touch panel 206a, that is, the touch ends (hereinafter referred to as touch-up).

[0052] Nothing is touching the touch panel 206a (hereinafter referred to as no touch (Touch-Off)).

[0053] When a touch is detected, touch continuation is also detected. Unless touch cessation is detected after a touch, touch continuation is usually detected. When touch movement is also detected, touch continuation is detected. Even when touch continuation is detected, touch movement is not detected unless the touch position is moving. When touch cessation is detected for all fingers or pens, untouch is detected.

[0054] These operations, states, and the position coordinates of the finger or pen touching the touch panel 206a are notified to the CPU 201 via the internal bus. Then, based on the notified information, the CPU 201 determines the type of operation (touch operation) performed on the touch panel 206a. For touch movement, the CPU 201 can also determine the moving direction of the finger or pen on the touch panel 206a based on the change in the position coordinates, the vertical component and the horizontal component of the touch panel 106a, respectively. If it is detected that the touch movement has been performed for at least a predetermined distance, the CPU 201 determines that a sliding operation has been performed.

[0055] The operation of quickly moving a finger on the touch panel 206a for a certain distance and releasing the finger in a touch state is called "flick". In other words, flicking is an operation of quickly moving and releasing the finger on the touch panel 206a. If it is detected that the touch movement is carried out at a predetermined speed or faster for at least a predetermined distance, and then the touch is detected to stop, the CPU 201 determines that a flick has been performed (determined that a flick has been performed after a sliding operation). In addition, a touch operation of touching multiple points (for example, two points) at the same time and moving these touch positions closer together is called "pinch-In", and a touch operation of moving the two positions farther apart is called "pinch-Out". Pinch-Out and pinch-in are collectively referred to as pinch-out operations (or simply "pinch-out").

[0056] For the touch panel 206a, various types of touch panels can be used, such as a resistive film type, a capacitive type, a surface acoustic wave type, an infrared type, an electromagnetic induction type, an image recognition type, and an optical sensor type. There are a type that detects a touch when the touch panel is actually contacted, and a type that detects a touch when a finger or a pen approaches the touch panel, but any type can be used.

[0057] Figure 2C 2 is an external view of a VR goggles (head mounted adapter) 230 to which the display control device 200 can be mounted. The display control device 200 can be used as a head mounted display (HMD) by being mounted in the VR goggles 230.

[0058] The insertion slot 231 is a slot for inserting the display control device 200. The entire display control device 200 can be inserted into the VR goggles 230 so that the display surface of the display 205 faces the headband 232 for fixing the VR goggles 230 to the user's head (in other words, the display surface faces the user). Therefore, the user can observe the display 205 of the display control device 200 in a state where the VR goggles 230 are worn on his head without holding the display control device 200 with his hands. In this case, as the user moves his head or body, the posture of the display control device 200 changes. The posture detection unit 213 detects the posture change of the display control device 200, and the CPU 201 performs VR display processing based on the posture change. In this case, the posture detection unit 213 "detects the posture of the display control device 200" is equivalent to "detecting the posture of the user's head (the direction in which the user's eyes are facing)". It should be noted that the display control device 200 itself may be an HMD with a mounting unit that can be mounted on the user's head without VR goggles.

[0059] Zenith Correction

[0060] The display control device 200 can control the image processing unit 204 through the CPU 201 to perform zenith correction for correcting the display direction (display area) and display the VR image after the zenith correction, so that the user can observe the image more easily. It should be noted that the display control device 200 can perform display after zenith correction, but can also perform display without zenith correction. Now, reference will be made to FIG. 3A to FIG. 3C To illustrate the zenith correction performed when displaying VR images.

[0061] Figure 3A 1 is a schematic diagram showing a state when video recording (image capture) is performed using the VR camera 100. Figure 3A, the VR camera 100 captures a VR image (wide-range image) of a house 301 and a dog 302 as subjects. Here, the house 301 is located in a direction D (e.g., north) when viewed from the VR camera 100, and the dog 302 is located in a direction 90° to the right of the direction D (e.g., east) when viewed from the VR camera 100. The VR camera 100 is held slightly tilted to the right with respect to the gravity direction 105, so the gravity direction 105 and the vertical direction 104 (vertical axis) of the VR camera 100 are not parallel. It is assumed that in this state, the user directs the imaging lens of the VR camera 100 toward the house 301 to capture a VR image (a moving image or a still image), and the VR camera 100 records the imaging result as image A.

[0062] Figure 3B The display control device 200 is shown in FIG. Figure 3A The display control device 200 (CPU 201) acquires the image A from the storage medium of the VR camera 100 or from the storage medium 208 such as a memory card installed in the display control device 200 wirelessly or by file transfer via a cable.

[0063] Figure 3B Also shown is a display example of the display 205 in the case where a partial area of ​​the image A is VR displayed (VR view) according to the gesture of the display control device 200. Figure 3B , the VR goggles 230 on which the display control device 200 is installed are installed on the user's head, and the user is observing image A in the VR display. Image A is a VR image captured by the VR camera 100 in a tilted state. Therefore, if zenith correction is not performed, a subject (image) tilted to the left (the opposite direction in which the VR camera 100 is tilted to the right with respect to the gravity direction 105 during imaging) is displayed on the display 205. In other words, without zenith correction, the house 301 in image A is displayed in a tilted state. Specifically, in the VR display, image A is displayed so that the direction in image A corresponding to the vertical direction 104 of the VR camera 100 during imaging is parallel to the gravity direction 303 (the direction of the axis passing through the zenith) detected by the posture detection unit 213 of the display control device 200. It should be noted that if the display method is not VR display, image A is displayed so that the direction in image A corresponding to the vertical direction 104 of the VR camera 100 during imaging is parallel to the vertical direction of the display 205 of the display control device 200 .

[0064] Figure 3B The right side of the indicator indicates that the user Figure 3BAn example of display of the display 205 when the state on the left side is turned horizontally to the right. Based on the change in the posture of the user when turning to the right, the display range (direction) of the image A is changed, and the range (direction) in which the dog 302 is photographed is displayed. In this state, the dog 302 is also displayed in a tilted state. If the position to which the user turns changes to the left or right in the horizontal direction, the display control device 200 changes the display range by rotating the display range of the moving image A around the vertical direction 104 of the VR camera 100 during shooting as an axis. Therefore, in the display observed by the user, the position of the dog 302 in the vertical direction, which should appear to be at the same height as the house 301, deviates.

[0065] Figure 3C is a schematic diagram showing a situation where the display control device 200 displays (reproduces) the image A after performing zenith correction. Figure 3A Image A is captured in the state of Figure 3B same, Figure 3C This is the case when the display method is VR display (VR view). Figure 3C On the left side, VR goggles 230 equipped with the display control device 200 are mounted on the user's head, and the user is observing image A in the VR display.

[0066] exist Figure 3C In the embodiment of the present invention, the orientation (display) of image A is corrected (zenith correction) so that the gravity direction 105 detected by the VR camera 100 when capturing image A becomes parallel to the vertical direction when reproducing the image. Specifically, image A is corrected and displayed so that the direction in image A corresponding to the gravity direction 105 during imaging (the gravity direction detected by the VR camera 100) is parallel to the gravity direction 303 detected by the posture detection unit 213 of the display control device 200. In other words, if the zenith correction is performed, the home 301 is displayed without tilt in image A. It should be noted that if the display method is not VR display, image A is displayed so that the direction in image A corresponding to the gravity direction 105 during imaging is parallel to the vertical direction of the display 205 of the display control device 200. In either case, the display orientation is corrected to offset the amount of deviation between the vertical direction 104 of the VR camera 100 during shooting, which is tilted relative to the gravity direction 105 (the direction of the axis passing through the zenith), and the gravity direction 105 detected by the VR camera 100 when image A is captured.

[0067] Figure 3C The right side of the Figure 3CAn example of display of the display 205 when the user turns right horizontally from the state on the left side. Based on the change in the posture when the user turns right, the display range (direction) of the image A is changed, and the area where the dog 302 is photographed is displayed. In this state, the dog 302 is also displayed without tilt. In addition, in the VR display, the dog 302 appears to be at the same height as the house 301 as in the actual view.

[0068] Display Mode

[0069] The display control device 200 determines whether to perform zenith correction on the VR image to be displayed according to the display mode (display method). In this embodiment, there are four display modes: Figure 4A Thumbnail display mode shown; Figure 4B The plane image display mode shown; Figure 4C The single-lens VR display mode shown; and Figure 4E and Figure 4F The dual-lens VR display mode shown in FIG. 4A to 4F The display of a set (image set) of two VR images (left eye image and right eye image) having parallax in the gravity direction and the horizontal direction captured by the VR camera 100 is described. In the thumbnail display mode, the plane image display mode, and the single-lens VR mode, it is assumed that the user observes the display of the entire screen with both the left eye and the right eye. On the other hand, in the dual-lens VR display mode, it is assumed that the right eye sees the right image of the two images arranged side by side on the left and right sides of the screen area, and the left eye sees the left image of the two images.

[0070] Figure 4A The thumbnail display mode shown is a display mode in which multiple images are arranged on one screen. The thumbnail display mode is a mode for the user to select a VR image (image set) to be displayed for observation. Each of the multiple images displayed in the thumbnail display mode is one of the VR images included in an image set recorded in the storage medium 208. Screen 410 is a screen of the display 205 in thumbnail display mode. Images 411 to 414 indicate a part or all of the area of ​​the recorded VR image. The user can select the VR image (image set) to be observed by touching any one of the images 411 to 414. The VR image (image set) to be displayed in the thumbnail display mode can be changed by performing a flick operation. Button 415 is a software button for the user to operate. By selecting button 415, the user can end the display of the VR image. In this mode, images 411 to 414 after the zenith operation are performed are displayed. Since both the left and right eyes of the user see the same display, no double image is produced in stereoscopic vision, and zenith correction makes it easier for the user to observe the image.

[0071] Figure 4B The plane image display mode shown is a mode in which the entire image of one of the VR images included in the image set (the image set of one VR image selected by the user in the thumbnail mode) is displayed on one screen. Screen 420 is a screen of the display 205 that displays the VR image selected by the user in the thumbnail display mode on one screen (plane image). Buttons 421 to 424 are software buttons for the user to perform various operations. By selecting one of the buttons 421 to 424 by the user, the function assigned to the selected button can be executed. Specifically, if the user touches button 421, the display returns to the previous display. If the user touches button 422, the currently displayed VR image is displayed as a single-lens VR display image (the display mode is changed to a single-lens VR display mode). Other functions are assigned to buttons 423 and 424. For example, a function for sharing a VR image with an external device (other users) is assigned to button 423. And, for example, a function for deleting a VR image from the storage medium 208 is assigned to button 424. In this mode, an image 411 after zenith correction is performed is displayed. Since in this mode both the left and right eyes of the user see the same display, no double image is produced in stereoscopic vision, and zenith correction makes it easier for the user to observe the image.

[0072] Figure 4C The single-lens VR display mode shown is a mode in which a VR image (one of the VR images included in the image set of VR images previously displayed in the planar image display) is mapped on a virtual sphere and a partial area of ​​the transformed image is displayed. In the single-lens VR display mode, the display range can be changed according to the user's display range change operation. Screen 430 is a screen of the display 205 in the single-lens VR display. Button 432 is a software button for the user to operate. By the user touching button 432, the currently displayed VR image is displayed as a dual-lens VR display image (the display mode is changed to the dual-lens VR display mode). In this mode, a partial area of ​​the image 411 after zenith correction is displayed. Since in this mode, both the left eye and the right eye of the user see the same display, no double image is generated in stereoscopic vision, and the zenith correction makes it easier for the user to observe the image.

[0073] Figure 4D 440 is a screen displayed when the user instructs dual-lens VR display, and indicates a wizard display screen 441 for zenith correction. Wizard display 441 is a display for asking (confirming) the user whether execution of zenith correction in dual-lens VR display is enabled or disabled. Buttons 442 and 443 are software buttons for the user to operate. If the user touches button 442, zenith correction in dual-lens VR display is enabled, and a “Zenith Correction” is displayed on display 205. Figure 4E On the other hand, if the user touches button 443, the zenith correction in the dual-lens VR display is disabled, and the display 205 is displayed. Figure 4F The screen in .

[0074] Figure 4E and Figure 4F The dual-lens VR display mode shown is a mode in which images in an image set (an image set of VR images previously displayed in a single-lens VR display) are respectively mapped on a virtual sphere, and partial areas of each transformed image are displayed side by side. In the dual-lens VR display mode, the display range can be changed according to the user's display range change operation. Screens 450a and 450b are screens of the display 205 in the dual-lens VR display, respectively.

[0075] Figure 4E The images 451a and 452a on the screen 450a shown are the same VR images captured by the left optical system of the VR camera 100 and subjected to zenith correction. Figure 4E In the dual-lens VR display mode shown, one of the VR images included in the image set is displayed side by side. Image 451a is a VR image for the user's left eye, and image 452 is a VR image for the user's right eye. For images 451a and 452a, VR images captured by the right optical system of the VR camera 100 can be used. Here, images 451a and 452a are the same image, so no parallax is generated. Figure 4E In the case of the dual-lens VR display shown, two images without parallax are displayed, so the user does not have stereoscopic vision. However, since both the left and right eyes of the user see the same image, a double image caused by the parallax between the two VR images is not generated. Therefore, zenith correction makes it easier for the user to observe the image.

[0076] Figure 4F The image 451b on the screen 450b shown is a VR image captured by the left optical system of the VR camera 100, and the image 452b on the screen 450b is a VR image captured by the right optical system of the VR camera 100. Figure 4EIn the dual-lens VR display mode shown, two VR images of the image set are displayed side by side. In other words, one image (image 451b) on screen 450b is a VR image for the user's left eye to observe, and the other image (image 452b) is a VR image for the user's right eye to observe. The two images have parallax but do not receive zenith correction. Since zenith correction is not performed, no parallax in the direction of gravity is generated between the two images when observed, and problems related to stereoscopic vision (such as double images) are unlikely to occur. In addition, since the two images have parallax with each other, stereoscopic vision is possible.

[0077] As described above, except for the case of the dual-lens VR display mode among the four display modes, the display control device 200 performs zenith correction on the VR image to be displayed. In the case of the dual-lens VR display mode, the display control device 200 allows the user to select whether to perform zenith correction on the VR image to be displayed. Specifically, in the present embodiment, in a case where the left eye and the right eye of the user see the same display image, the display control device 200 performs zenith correction on the VR image. On the other hand, in a case where the left eye and the right eye of the user see different display images from each other, the display control device 200 does not perform zenith correction on the VR image.

[0078] Display control processing

[0079] Will refer to Figure 5 The display control process of the display control device 200 is described with reference to the flowchart in FIG. When the display of the VR image is instructed to the operation unit 206 of the display control device 200 by a predetermined user operation, Figure 5 The processing in the flowchart starts. Figure 5 Each processing step in the flowchart is implemented by the CPU 201 expanding the program stored in the nonvolatile memory 203 in the memory 202 and executing the program. In this embodiment, it is assumed that the display control device 200 controls the display of the image set (image file) captured (acquired) by the VR camera 100.

[0080] In S501, the CPU 201 reads (acquires) a plurality of image sets (image files) taken by the VR camera 100 from the storage medium 208. In this embodiment, the CPU 201 reads (acquires) four image sets from the storage medium 208 to display Figure 4A Thumbnails shown.

[0081] In S502, the CPU 201 reads (acquires) detailed information about each of the plurality of image sets read in S501 from the storage medium 208. In this case, the detailed information is, for example, tilt information about the image set, the gravity direction of the VR camera 100 during imaging, and the vertical direction of the VR camera 100 during imaging. For example, the tilt of the vertical direction 104 of the VR camera 100 during imaging relative to the gravity direction 105 (tilt information about the VR camera 100 during imaging) is attached to the image set by the VR camera 100 as tilt information about the image set, and is stored in the storage medium 208. The VR camera 100 can acquire the tilt of the vertical direction 104 of the VR camera 100 during imaging relative to the gravity direction 105 by a gyro sensor of the VR camera 100 or the like. The tilt information about the image set can be regarded as tilt information about the direction connecting the two optical systems (lenses 101 and 102) of the VR camera 100 relative to the horizontal direction when the image set is captured.

[0082] In S503, for each of the multiple image sets read, the CPU 201 determines whether each image set is an inclined image set based on the tilt information. Here, an "inclined image set" refers to an image set (VR image) shot in a state where the gravity direction 105 and the vertical direction 104 of the VR camera 100 are not parallel, or an image set (VR image) shot in a state where the horizontal direction is not parallel to the direction of the two optical systems (lenses 101 and 102) connecting the VR camera 100. Therefore, the CPU 201 can determine whether the image set is inclined by determining whether the gravity direction 105 during the shooting of the VR camera 100 is parallel to the vertical direction 104 of the VR camera 100 during the shooting. If the image set in the multiple image sets is an inclined image set, the process enters S504, otherwise it enters S505.

[0083] The CPU 201 can determine the tilt information of the image set based on the features of the VR image (image set). For example, the CPU 201 uses a known straight line extraction method to extract the features of the straight lines in the VR image and calculates the coordinates of two points where multiple straight lines intersect. Then, the CPU 201 obtains the tilt of the VR camera based on the position and tilt of the line segment connecting the two points.

[0084] In S504, the CPU 201 controls the image processing unit 204 to perform zenith correction on a VR image included in the tilted image set determined in S503, and displays the image as a thumbnail on the display 205. In the thumbnail display in S504, a distorted image is generated by equidistant cylindrical projection, such as Figure 4AIn the image 411, the entire VR image is displayed (framed), so the display range cannot be changed. The image 411 may be an image within a predetermined square range extracted from the center of a rectangular VR image drawn by equidistant cylindrical projection, rather than the entire VR image framed in the screen.

[0085] In S505, the CPU 201 displays a VR image included in the non-tilt image set determined in S503 as a thumbnail on the display 205. Here, the CPU 201 does not perform zenith correction on the VR image. For example, in the thumbnail display of S505, for example, Figure 4A Images 412 to 414 in FIG.

[0086] In other words, the processing steps in S503 to S505 are independently performed for each image set in the plurality of image sets. Specifically, in S503, for each image set in the plurality of image sets read in S501, the CPU 201 determines whether the image set is a tilted image set. Then, among the plurality of image sets, the tilted image set is zenith corrected in S504 and displayed as one image in the thumbnail display. The non-tilted image set is not zenith corrected and is displayed as one image in the thumbnail display.

[0087] In S506, the CPU 201 determines whether the user instructs to end the thumbnail display mode. Figure 4A In the case of pressing the button 415 in the display mode, the CPU 201 determines that the user instructs to end the thumbnail display mode. If the user instructs to end the thumbnail display mode, then Figure 5 All processing steps in the flowchart are completed, otherwise the process enters S507.

[0088] In S507, the CPU 201 determines whether the user has selected a plurality of VR images displayed in the thumbnail display (in Figure 4A In the case of the touch operation, it is a touch operation of selecting any one VR image from images 411 to 414). The touch operation is touch or touch stop. If a touch operation is performed, the process proceeds to S508, otherwise it proceeds to S506. In the following description, the VR image selected by the user in S507 is referred to as a "selected image", and the image set including the selected image is referred to as a "selected image set".

[0089] In S508, the CPU 201 determines whether the selected image set is an oblique image set based on the inclination information related to the selected image set. If the selected image set is an oblique image set, the process proceeds to S509, otherwise to S510.

[0090] In S509, the CPU 201 controls the image processing unit 204 to perform zenith correction on the selected image and to display the selected image on the display 205 as a flat image. Figure 4B As shown. Figure 4B In the , a distorted VR image generated by equidistant cylindrical projection is displayed (without transformation by mapping on a virtual sphere). Here, since the entire VR image is framed in the screen and displayed in this state, the display range cannot be changed.

[0091] In S510, the CPU 201 displays the selected image on the display 205 without performing zenith correction on the image (performs planar image display).

[0092] In S511, the CPU 201 determines whether the user instructs single-lens VR display. Figure 4B When the button 422 in is pressed, the CPU 201 determines that the user has instructed single-lens VR display. If the single-lens VR display is instructed, the process proceeds to S513, otherwise proceeds to S512.

[0093] In S512, the CPU 201 determines whether the user has performed an operation to end the plane image display. Figure 4B In the case where the button 421 in the display is pressed, the CPU 201 determines that an operation to end the plane image display has been performed. If the user has performed an operation to end the plane image display, the process proceeds to S503, otherwise proceeds to S511.

[0094] In S513, the CPU 201 determines whether the selected image set is an oblique image set. If the selected image set is an oblique image set, the process proceeds to S514, otherwise to S515.

[0095] In S514, the CPU 201 controls the image processing unit 204 to perform zenith correction on the selected image. Figure 4C As shown, the CPU 201 displays the selected image after zenith correction on the display 205 as a single-lens VR display. Figure 4C In other words, in Figure 4C In the example, a portion of a VR image after image transformation processing is displayed to remove image distortion.

[0096] In S515 , the CPU 201 does not perform zenith correction on the selected image, and performs single-lens VR display on the display 205 .

[0097] In S516, the CPU 201 determines whether the user has performed a display range change operation. The display range change operation is a flick operation on the touch panel 206a or an operation to change the posture of the display control device 200. If a display range change operation has been performed, the process proceeds to S517, otherwise to S518.

[0098] In S517 , the CPU 201 controls the image processing unit 204 to change the range of the selected image to be displayed on the display 205 .

[0099] In S518, the CPU 201 determines whether the user instructs the dual-lens VR display. Figure 4C When the button 432 in is pressed, the CPU 201 determines that the dual-lens VR display is instructed. If the dual-lens VR display is instructed, the process proceeds to S520, otherwise proceeds to S519.

[0100] In S519, the CPU 201 determines whether the user has performed an operation to end the single-lens VR display. Figure 4C If the button 421 in is pressed, the CPU 201 determines that an operation to end the single-lens VR display has been performed. If the user has performed an operation to end the single-lens VR display, the process proceeds to S508, otherwise it proceeds to S516.

[0101] In S520, the CPU 201 determines whether the selected image set is an oblique image set. If the selected image set is an oblique image set, the process proceeds to S521, otherwise to S524.

[0102] In S521, if Figure 4D As shown, the CPU 201 displays (performs) a guide display 441 on the display 205 to allow the user to select whether to perform zenith correction on the selected image.

[0103] In S522, the CPU 201 determines the content selected by the user in the guide display 441. If the user has selected to perform zenith correction using the operation unit 206, the process proceeds to S523, or if the user has selected not to perform zenith correction, the process proceeds to S524.

[0104] Processing steps S521 and S522 are not essential, and for example, if in S520 the selected image set is an oblique image set ("YES" in S520), the process may proceed to S523.

[0105] In S523, the CPU 201 controls the image processing unit 204 to perform zenith correction on the selected image and perform dual-lens VR display on the display 205 ( Figure 4E). In the dual-lens VR display of S523, the VR images for the left eye and the right eye are the same image without parallax. In S523, since the same image is displayed for the left eye and the right eye, stereoscopic vision cannot be achieved, but the advantage is that a double image that reduces visibility is not generated.

[0106] In S524, the CPU 201 performs dual-lens VR display on the display 205 without performing zenith correction on the two images in the selected image set ( Figure 4F ). In the dual-lens VR display of S524, different VR images that generate parallax between the left eye and the right eye are displayed on the left and right sides. If the VR images are displayed in this way, not only can stereoscopic vision be achieved because the images have parallax, but also because zenith correction is not performed, visibility is not reduced.

[0107] In S525, the CPU 201 determines whether the user has performed a display range change operation. The display range change operation is a flick operation on the touch panel 206a or an operation to change the posture of the display control device 200. If a display range change operation has been performed, the process proceeds to S526, otherwise to S527.

[0108] In S526 , the CPU 201 controls the image processing unit 204 to change the range of the VR image to be displayed on the display 205 .

[0109] In S527, the CPU 201 determines whether the user has performed an operation to end the dual-lens VR display. Figure 4E or Figure 4F In the case where the button 421 in is pressed, the CPU 201 determines that an operation to end the dual-lens VR display has been performed. If the user has performed an operation to end the dual-lens VR display, the process proceeds to S513, otherwise proceeds to S525.

[0110] According to the above control flow, the user chooses whether to perform zenith correction on the VR image to be displayed only when the user indicates the dual-lens VR display mode. Therefore, according to the present embodiment, the VR image captured in a tilted state can be displayed for each reproduction mode, so that the user can observe the image most easily. In the case of the plane image display mode, the thumbnail display mode, and the single-lens VR display mode, the VR image after zenith correction is displayed so that the user can easily observe and operate the image. In the case of the dual-lens VR display mode in which images with parallax are displayed side by side, double images such as those caused by zenith correction can be prevented. As a result, the following display control device (electronic device) can be implemented, which controls the display so that the user can observe the VR image captured in a tilted state most easily for each reproduction scene.

[0111] In the present embodiment, zenith correction is not performed on the VR image only in the case of the dual-lens VR display mode. Therefore, in the dual-lens VR mode, when the user is wearing the VR goggles 230 (HMD), the user's left eye and right eye can observe the corresponding images of the left VR image and the right VR image on the screen. In other words, when the user is wearing the VR goggles 230, zenith correction of the VR image may generate a double image. Therefore, when the display control device 200 or the installation detection unit of the VR goggles 230 detects that the user is wearing the VR goggles 230, the CPU 201 may not perform zenith correction on the VR image. In addition, in other cases, the CPU 201 may perform zenith correction on the VR image. Here, for example, if the distance from the display 205 (display control device 200) to the user's eyes is within a predetermined distance, the installation detection unit can detect that the VR goggles 230 are worn. In addition, for example, if the contact of the VR goggles 230 with the user's head is detected, the installation detection unit can detect that the VR goggles 230 are worn. If the installation detection unit detects that the VR goggles 230 are worn, the CPU 201 may switch the display mode to a dual-lens VR display mode (displaying left and right images with parallax). If the installation detection unit does not detect that the VR goggles 230 are worn, the CPU 201 may switch the display mode to a mode other than the dual-lens VR display mode.

[0112] According to the present invention, VR images captured by a plurality of optical systems can be displayed more appropriately.

[0113] It is assumed that the above-described various controls performed by the CPU 201 may be performed by one hardware component or by a plurality of hardware components (eg, a plurality of processors and circuits) that share the processing for controlling the entire apparatus.

[0114] Although the present invention has been described with reference to the preferred embodiments, the present invention is not limited to these specific embodiments and includes various modes within the scope not departing from the essence of the present invention. The above-mentioned embodiments are merely examples of the present invention and can be combined as needed.

[0115] In the above embodiments, the present invention is applied to a display control device as an example, but the present invention is not limited thereto, but is applicable to any electronic device that controls a display device, wherein the display device can display VR images for the left eye and the right eye of the user, respectively. In other words, the present invention is applicable to personal computers, PDAs, portable telephone terminals, portable image viewers, head-mounted displays, printers including displays, digital photo frames, music players, game consoles, and e-book readers, etc.

[0116] Other embodiments

[0117] The embodiments of the present invention may also be implemented by providing software (program) for performing the functions of the above-described embodiments to a system or device via a network or various storage media, and a computer or a central processing unit (CPU) or a microprocessing unit (MPU) of the system or device reads and executes the program.

[0118] Although the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. An electronic device, comprising: An acquisition unit configured to acquire an image set including a first image and a second image having parallax with each other; a control unit configured to control to display the first image on a display without displaying the second image in a first display mode, or to display both the first image and the second image on the display in a second display mode; as well as a correction unit configured to perform a zenith correction on the first image when the image set is displayed on the display in the first display mode, and not perform the zenith correction on both the first image and the second image when the image set is displayed on the display in the second display mode, The control unit controls to display the first image on the display in the first display mode in a manner in which the orientation of the first image is corrected to a predetermined orientation through the zenith correction.

2. The electronic device according to claim 1, wherein: The correction unit performs the zenith correction based on tilt information indicating a direction of an optical system used to capture the set of images with respect to a horizontal direction.

3. The electronic device according to claim 2, wherein: The image set is captured by a camera device having two optical systems, and The tilt information indicates a tilt of a direction in which the two optical systems are connected relative to the horizontal direction.

4. The electronic device according to claim 3, wherein: The control unit controls to display the first image on the display in the case of the first display mode and in the case that a horizontal direction during imaging is not parallel to a direction in which the two optical systems are connected during the imaging.

5. The electronic device according to claim 1, wherein: The zenith correction is performed to correct the orientation of the first image so that a direction in the first image corresponding to the gravity direction during imaging is parallel to a vertical direction of the display.

6. The electronic device according to claim 1, wherein: The zenith correction is performed to correct the orientation of the first image so that the direction in the first image corresponding to the gravity direction during imaging is parallel to the gravity direction during display of the first image on the display.

7. The electronic device according to claim 1, wherein: The first image and the second image are displayed side by side on the display in the second display mode, and In the second display mode, the first image is an image to be observed by one of a left eye and a right eye of a user, and the second image is an image to be observed by the other of the left eye and the right eye of the user.

8. The electronic device according to claim 1, wherein: The first display mode is a display mode in which display is performed in at least one of a first display mode in which only one first image is displayed and a second display mode in which two identical first images are displayed side by side.

9. The electronic device according to claim 8, wherein: The control unit controls to switch the second display manner and the second display mode according to a user input.

10. The electronic device according to claim 9, wherein: In the case where two images are displayed side by side, the control unit controls to display a guide for the user to select one of the second display manner and the second display mode.

11. The electronic device according to claim 1, wherein: The image to be displayed is generated by mapping an equidistant cylindrical projection onto a virtual sphere.

12. A method for controlling an electronic device, the method comprising: An acquisition step for acquiring an image set including a first image and a second image having a parallax with each other; a control step for displaying the first image on a display without displaying the second image in a first display mode, or displaying both the first image and the second image on the display in a second display mode; as well as a correction step for performing a zenith correction on the first image when the image set is displayed on the display in the first display mode, and for not performing the zenith correction on both the first image and the second image when the image set is displayed on the display in the second display mode, The control step includes displaying the first image on the display in the first display mode in such a manner that the orientation of the first image is corrected to a predetermined orientation through the zenith correction.

13. A computer-readable storage medium storing a program for causing a computer to execute the control method according to claim 12.

Images