Electronic device and control method for the same

Abstract

To make it possible to display a portion that a sharer wants to show to sequentially reproduce a plurality of VR images.SOLUTION: An electronic device has: acquisition means for acquiring a plurality of sequentially captured VR images; and generation means for generating VR images whose reference directions are corrected to directions in which a particular subject is present in the VR images from each of the plurality of VR images. The electronic device also has: acquisition means for acquiring a plurality of sequentially captured VR images; and generation means for generating a plurality of VR images that are corrected to be reproduced sequentially tracking a specific subject.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to an electronic device and a method for controlling an electronic device. [Background technology]

[0002] A technique is known in which, when a spherical image is stored in a server on a network and the spherical image is made viewable (shared) on a display device owned by another user, a sharer of the spherical image allows the viewer to view the image in a specific direction. Patent Document 1 discloses a technique for controlling a display unit to change a display range according to the attitude of the display unit when playing a video, and to display a main subject on the display unit even in different attitudes. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-121918 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, when sharing multiple VR images captured sequentially, such as a VR video, it is difficult to sequentially play back multiple VR images with the portion that the sharer wants to show as the display range, regardless of the orientation of the display unit.

[0005] An object of the present invention is to provide an electronic device that can display the parts that a sharer wants to show and sequentially play back a plurality of VR images. [Means for solving the problem]

[0006] The electronic device of the present invention is characterized by having an acquisition means for acquiring multiple VR images captured sequentially, and a generation means for generating a VR image from each of the multiple VR images in which the reference direction of the VR image is corrected to the direction in which a specific subject is present in the VR image. Effect of the Invention

[0007] According to the present invention, a plurality of VR images can be played back in sequence, with the parts that the sharer wants to show being displayed. [Brief description of the drawings]

[0008] [Figure 1] 1 is an external view and a block diagram of a digital camera according to an embodiment of the present invention. [Diagram 2] 1A and 1B are an external view and a block diagram of a display control device, and an external view of VR goggles. [Diagram 3] FIG. [Figure 4] 11 is a flowchart illustrating an image sharing process. [Diagram 5] 13 is a screen example in an image sharing process. [Figure 6] 11 is a flowchart illustrating an example of an image reproduction process. [Figure 7] 13 is a flowchart illustrating a selected image reproduction process. [Figure 8] 11 is a screen example in image reproduction processing. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Fig. 1(A) is a front perspective view (external view) of a digital camera 100 (imaging device) which is an example of an electronic device. Fig. 1(B) is a rear perspective view (external view) of the digital camera 100. The digital camera 100 has an omnidirectional It is a camera (omnidirectional camera; spherical camera) for capturing images (spherical images).

[0010] Barrier 102a is a protective window for a front camera unit whose shooting range is in front of digital camera 100. The front camera unit is, for example, a wide-angle camera unit whose shooting range is a wide range of 180 degrees or more up, down, left, and right on the front side of digital camera 100. Barrier 102b is a protective window for a rear camera unit whose shooting range is a rear side of digital camera 100. The rear camera unit is, for example, a wide-angle camera unit whose shooting range is a wide range of 180 degrees or more up, down, left, and right on the rear side of digital camera 100.

[0011] The display unit 28 displays various information. The shutter button 61 is an operation unit (operation member) for issuing a shooting instruction. The mode change switch 60 is an operation unit for switching between various modes. The connection I / F 25 is a connector for connecting a connection cable to the digital camera 100, and an external device such as a smartphone, a personal computer, or a television is connected to the digital camera 100 using the connection cable. The operation unit 70 is various switches, buttons, dials, touch sensors, etc. that accept various operations from the user. The power switch 72 is a push button for switching the power on / off.

[0012] The light-emitting unit 21 is a light-emitting member such as a light-emitting diode (LED) that notifies the user of various states of the digital camera 100 by light-emitting patterns and colors. The fixing unit 40 is, for example, a tripod screw hole, and is used to fix and install the digital camera 100 on a fixing device such as a tripod.

[0013] FIG. 1C is a block diagram showing an example of the configuration of the digital camera 100. The barrier 102a covers the imaging system (photographing lens 103a, shutter 101a, imaging unit 22a, etc.) of the front camera unit to prevent the imaging system from becoming dirty or damaged. The photographing lens 103a is a lens group including a zoom lens and a focus lens, and is a wide-angle lens. The shutter 101a is a shutter with an aperture function that adjusts the amount of subject light incident on the imaging unit 22a. The imaging unit 22a is an imaging element (imaging sensor) composed of a CCD or CMOS element that converts an optical image into an electrical signal. The A / D converter 23a converts an analog signal output from the imaging unit 22a into a digital signal. Note that the barrier 102a may not be provided and the outer surface of the photographing lens 103a may be exposed, and the photographing lens 103a may prevent other imaging systems (shutter 101a and imaging unit 22a) from becoming dirty or damaged.

[0014] The barrier 102b covers the imaging system (taking lens 103b, shutter 101b, imaging section 22b, etc.) of the rear camera section to prevent the imaging system from becoming dirty or damaged. The taking lens 103b is a lens group including a zoom lens and a focus lens, and is a wide-angle lens. The shutter 101b is a shutter with an aperture function that adjusts the amount of subject light incident on the imaging section 22b. The imaging section 22b is an imaging element composed of a CCD or CMOS element that converts an optical image into an electrical signal. The A / D converter 23b converts an analog signal output from the imaging section 22b into a digital signal. Note that the barrier 102b may not be provided and the outer surface of the taking lens 103b may be exposed, and the taking lens 103b may prevent other imaging systems (shutter 101b and imaging section 22b) from becoming dirty or damaged.

[0015] VR (Virtual Reality) images are captured by the imaging units 22a and 22b. A VR image is an image that can be displayed in VR (displayed in the display mode "VR view"). VR images include omnidirectional images (spherical images) captured by an omnidirectional camera (spherical camera) and panoramic images that have a wider image range (effective image range) than the display range that can be displayed at one time on the display unit. VR images include not only still images, but also videos and live view images (images acquired from a camera in almost real time). A VR image can be displayed in a wide range of directions (vertical angle, angle from the zenith, elevation angle, depression angle, altitude) up to a maximum of 100 meters. It has a viewing range (effective viewing range) of 360 degrees in the left and right directions (horizontal angle, azimuth angle, yaw angle) and 360 degrees in the right and left directions (horizontal angle, azimuth angle, yaw angle).

[0016] In addition, VR images also include images that have a wider viewing angle (field of view) than the viewing angle that can be captured by a normal camera, or a wider image range (effective image range) than the display range that can be displayed at one time on a display unit, even if the viewing angle is less than 360 degrees up and down and less than 360 degrees left and right. For example, an image captured by a spherical camera capable of capturing an object with a viewing angle (field of view) of 360 degrees left and right (horizontal angle, azimuth angle) and a vertical angle of 210 degrees centered on the zenith is a type of VR image. In addition, an image captured by a camera capable of capturing an object with a viewing angle (field of view) of 180 degrees left and right (horizontal angle, azimuth angle) and a vertical angle of 180 degrees centered on the horizontal direction is a type of VR image. In other words, an image that has a viewing angle of 160 degrees (±80 degrees) or more in the up and down and left and right directions and has a wider image range than the range that a human can see at one time is a type of VR image.

[0017] When this VR image is displayed in VR (displayed in "VR view" display mode), seamless omnidirectional images can be viewed left and right (horizontal rotation direction) by changing the position of the display device (display device displaying the VR image) in the left and right rotation direction. In the up and down directions (vertical rotation direction), seamless omnidirectional images can be viewed within a range of ±105 degrees from directly above (the zenith), but the range beyond 105 degrees from directly above becomes a blank area with no images. A VR image can also be described as "an image whose image range is at least a part of a virtual space (VR space)."

[0018] VR display (VR view) is a display method (display mode) that can change the display range, displaying an image of a VR image with a viewing range according to the orientation of the display device. When viewing by wearing a head-mounted display (HMD), which is a display device, an image with a viewing range according to the orientation of the user's face is displayed. For example, assume that a VR image is displayed with a viewing angle (field of view) centered on 0 degrees left and right (a specific direction, for example, north) and 90 degrees up and down (90 degrees from the zenith, i.e., horizontal) at a certain point in time. From this state, if the orientation of the display device is flipped over (for example, if the display surface is changed from facing south to facing north), the display range is changed to an image with a viewing angle centered on 180 degrees left and right (the opposite direction, for example, south) and 90 degrees up and down (horizontal) in the same VR image. In the case where a user is viewing an HMD, if the user turns his or her face from north to south (i.e., turns around), the image displayed on the HMD will also change from a north image to a south image. This type of VR display can provide users with a sense of immersion, as if they were visually inside the VR image (VR space). A smartphone attached to a VR goggle (head-mounted adapter) can be considered a type of HMD.

[0019] The display method of the VR image is not limited to the above. The display range may be moved according to a user's operation on a touch panel or a direction button, instead of a change in posture. During VR display (when the display mode is "VR view"), in addition to changing the display range due to a change in posture, the display range may be changed according to a touch move on the touch panel, a drag operation on the mouse, pressing a direction button, etc.

[0020] The image processing unit 24 performs predetermined processing (pixel interpolation, resizing such as reduction, color conversion, etc.) on the data from the A / D converters 23a and 23b or the data from the memory control unit 15. The image processing unit 24 also performs predetermined arithmetic processing using the captured image data. The system control unit 50 performs exposure control and distance measurement control based on the calculation results obtained by the image processing unit 24. This allows AF (autofocus) processing, AE (automatic exposure) processing, FE (flash pre-flash) processing, etc. to be performed using a TTL (through-the-lens) method. The image processing unit 24 further performs predetermined arithmetic processing using the captured image data, and outputs the obtained Based on the calculation result, the image processing unit 24 performs TTL-type AWB (auto white balance) processing. The image processing unit 24 also performs basic image processing on the two images (two fisheye images; two wide-angle images) obtained from the A / D converters 23a and 23b, and performs stitching processing to combine the two images that have been subjected to basic image processing, thereby generating a single VR image. The image processing unit 24 also performs image cropping, enlargement, distortion correction, and the like for VR display of a VR image during VR display in live view or during playback, and performs rendering to draw the processing result in a predetermined storage area (VRAM) in the memory 32.

[0021] In the stitching image processing, the image processing unit 24 uses one of the two images as a reference image and the other as a comparison image, calculates the amount of deviation between the reference image and the comparison image for each area by pattern matching processing, and detects a stitching position for stitching the two images based on the amount of deviation for each area. The image processing unit 24 corrects the distortion of each image by geometric transformation in consideration of the detected stitching position and the lens characteristics of each optical system, and converts each image into an image in a celestial sphere format (a celestial sphere image format). The image processing unit 24 generates one celestial sphere image (VR image) by synthesizing (blending) two celestial sphere format images. The generated celestial sphere image is, for example, an image using equirectangular projection, and the position of each pixel of the celestial sphere image can be associated with the coordinates of the surface of a sphere (VR space).

[0022] The output data from the A / D converters 23a, 23b is written to the memory 32 via the image processing unit 24 and the memory control unit 15, or via the memory control unit 15 without via the image processing unit 24. The memory 32 stores image data obtained by the imaging units 22a, 22b and converted into digital data by the A / D converters 23a, 23b, and image data to be output to an external display from the connection I / F 25. The memory 32 has a storage capacity sufficient to store a predetermined number of still images and a predetermined period of moving images and audio.

[0023] The memory 32 also serves as a memory (video memory) for image display. The image display data stored in the memory 32 can be output from the connection I / F 25 to an external display. The VR images captured by the imaging units 22a and 22b, generated by the image processing unit 24, and stored in the memory 32 are sequentially transferred to the external display and displayed, thereby realizing a function as an electronic viewfinder and enabling live view display (LV display). Hereinafter, an image displayed in live view display is referred to as a live view image (LV image). The VR images stored in the memory 32 can also be transferred to a wirelessly connected external device (such as a smartphone) via the communication unit 54 and displayed on the external device, enabling live view display (remote LV display).

[0024] The non-volatile memory 56 is a memory serving as an electrically erasable and recordable recording medium, such as an EEPROM (Electrically Erasable Programmable Read-Only Memory). The non-volatile memory 56 records constants, programs, etc. for the operation of the system control unit 50. The programs referred to here are computer programs for executing various processes.

[0025] The system control unit 50 is a control unit having at least one processor or circuit, and controls the entire digital camera 100. The system control unit 50 realizes each process of this embodiment by executing a program recorded in the nonvolatile memory 56. The system memory 52 is, for example, a RAM (random access memory). The system control unit 50 loads constants and variables for the operation of the system control unit 50, programs read from the nonvolatile memory 56, and the like in the system memory 52. ​​The system control unit 50 also performs display control by controlling the memory 32, the image processing unit 24, and the memory control unit 15. The system timer 53 is a timekeeping unit that measures the time used for various controls and the time of a built-in clock.

[0026] The mode changeover switch 60, the shutter button 61, and the operation unit 70 are used to input various operation instructions to the system control unit 50. The mode changeover switch 60 switches the operation mode of the system control unit 50 to one of a still image recording mode, a video shooting mode, a playback mode, a communication connection mode, and the like. Modes included in the still image recording mode include an auto shooting mode, an auto scene determination mode, a manual mode, an aperture priority mode (Av mode), a shutter speed priority mode (Tv mode), and a program AE mode. In addition, there are various scene modes and a custom mode, which are shooting settings according to shooting scenes. The user can directly switch to one of these modes using the mode changeover switch 60. In addition, the user may switch to a list screen of shooting modes using the mode changeover switch 60, and then select one of the multiple modes displayed on the display unit 28 using other operation members to switch the mode. Similarly, the video shooting mode may also include multiple modes.

[0027] The shutter button 61 includes a first shutter switch 62 and a second shutter switch 64. The first shutter switch 62 is turned on when the shutter button 61 is pressed halfway (instruction to prepare for shooting) during operation and generates a first shutter switch signal SW1. In response to the first shutter switch signal SW1, the system control unit 50 starts shooting preparation operations such as AF (autofocus) processing, AE (auto exposure) processing, AWB (auto white balance) processing, and FE (pre-flash) processing.

[0028] The second shutter switch 64 is turned on when the shutter button 61 is fully pressed (photographing instruction) and generates a second shutter switch signal SW2. The system control unit 50 starts a series of photographing processing operations from reading signals from the imaging units 22a and 22b to writing image data to the recording medium 90 in response to the second shutter switch signal SW2.

[0029] The shutter button 61 is not limited to an operating member that can be operated in two stages, full press and half press, but may be an operating member that can be pressed only in one stage. In this case, pressing it in one stage performs the shooting preparation operation and the shooting process consecutively. This is the same operation as when a shutter button that can be pressed halfway and full press is pressed all the way (when the first shutter switch signal SW1 and the second shutter switch signal SW2 are generated almost simultaneously).

[0030] The operation unit 70 is assigned appropriate functions for each situation by selecting and operating various function icons and options displayed on the display unit 28, and acts as various function buttons. Examples of the function buttons include an end button, a back button, an image forward button, a jump button, a filter button, and an attribute change button. For example, when the menu button is pressed, a menu screen in which various settings can be made is displayed on the display unit 28. The user can intuitively make various settings by operating the operation unit 70 while looking at the menu screen displayed on the display unit 28.

[0031] The power supply control unit 80 is composed of a battery detection circuit, a DC-DC converter, a switch circuit for switching between blocks to which electricity is applied, and the like, and detects whether a battery is attached, the type of battery, the remaining battery level, etc. The power supply control unit 80 also controls the DC-DC converter based on the detection results and instructions from the system control unit 50, and supplies the necessary voltage for the necessary period to each unit including the recording medium 90. The power supply unit 30 is composed of primary batteries such as alkaline batteries and lithium batteries, secondary batteries such as NiCd batteries, NiMH batteries, and Li batteries, an AC adapter, etc.

[0032] The recording medium I / F 18 is an interface with a recording medium 90 such as a memory card or a hard disk. The recording medium 90 is a memory card or the like for recording captured images. The recording medium 90 is composed of a semiconductor memory, an optical disk, a magnetic disk, etc. The recording medium 90 may be an exchangeable recording medium that is detachable from the digital camera 100, or may be a recording medium built into the digital camera 100.

[0033] The communication unit 54 transmits and receives video signals, audio signals, and the like to and from an external device connected wirelessly or via a wired cable. The communication unit 54 can be connected to a wireless LAN (Local Area Network) or the Internet, and can communicate with an external device (such as a server) on the network via the network. The communication unit 54 can also communicate with an external device via Bluetooth (registered trademark) or Bluetooth Low Energy. The communication unit 54 can transmit images (including LV images) captured by the imaging units 22a and 22b and images recorded on the recording medium 90, and can receive images and various other information from external devices.

[0034] The attitude detection unit 55 detects the attitude of the digital camera 100 with respect to the direction of gravity. Based on the attitude detected by the attitude detection unit 55, it is possible to determine whether the images captured by the imaging units 22a and 22b were captured with the digital camera 100 held horizontally or vertically. It is also possible to determine to what degree the images captured by the imaging units 22a and 22b were captured with the digital camera 100 tilted in three axial directions (rotation directions), namely, the yaw direction, pitch direction, and roll direction.

[0035] The system control unit 50 can add orientation information corresponding to the attitude detected by the attitude detection unit 55 to the image file of the VR image captured by the imaging units 22a and 22b, or rotate the image (adjust the orientation of the image to correct the tilt) and record it. One or a combination of sensors selected from an acceleration sensor, a gyro sensor, a geomagnetic sensor, a direction sensor, and an altitude sensor can be used as the attitude detection unit 55. The acceleration sensor, gyro sensor, direction sensor, and the like that constitute the attitude detection unit 55 can also be used to detect the movement of the digital camera 100 (panning, tilting, lifting, whether or not it is stationary, etc.).

[0036] The microphone 20 is a microphone that collects sounds around the digital camera 100 to be recorded as sounds for VR images (VR video), which are videos. The connection I / F 25 is a connection plug to which an HDMI (registered trademark) cable, a USB cable, or the like is connected to connect to an external device to transmit and receive video.

[0037] FIG. 2(A) is a front perspective view of the display control device 200. The display control device 200 is an example of an electronic device, such as a smartphone or a tablet terminal. The display 205 is a display unit that displays images and various information. The display 205 is integrally configured with a touch panel 206a, and is capable of detecting a touch operation on the display surface of the display 205. The display control device 200 is capable of VR display of a VR image (VR content) on the display 205. The operation member 206b is a power button that accepts an operation to switch the power of the display control device 200 on and off. The operation member 206c and the operation member 206d are volume buttons for increasing or decreasing the volume of the sound output from the speaker 212b, or an earphone or an external speaker connected to the audio output terminal 212a. The operation member 206e is a home button for displaying a home screen on the display 205. The audio output terminal 212a is an earphone jack, and is a terminal that outputs an audio signal to the earphone or an external speaker. The speaker 212b is a built-in speaker that outputs audio.

[0038] 2B is a rear perspective view of the display control device 200. The imaging unit 215 is a camera capable of capturing an image of the real space.

[0039] 2(C) is a block diagram showing an example of the configuration of the display control device 200. A CPU 201, a memory 202, a non-volatile memory 203, an image processing unit 204, a display 205, an operation unit 206, a recording medium I / F 207, an external I / F 209, and a communication I / F 210 are connected to an internal bus 250. In addition, an audio output unit 212, an attitude detection unit 213, a self-position / surrounding environment estimation unit 214, and an imaging unit 215 are also connected to the internal bus 250. The units connected to the internal bus 250 can exchange data with each other via the internal bus 250.

[0040] The CPU 201 is a control unit that controls the entire display control device 200, and is composed of at least one processor or circuit. The memory 202 is, for example, a RAM (a volatile memory using semiconductor elements). The CPU 201 controls each unit of the display control device 200 using the memory 202 as a work memory according to a program stored in, for example, the nonvolatile memory 203. The nonvolatile memory 203 stores various information such as image data, audio data, other data, and various programs for the operation of the CPU 201. The nonvolatile memory 203 is, for example, a flash memory or a ROM (Read-Only Memory).

[0041] The image processing unit 204 performs various image processing on the image stored in the non-volatile memory 203 or the recording medium 208, the video signal acquired via the external I / F 209, or the image acquired via the communication I / F 210 under the control of the CPU 201. The various image processing includes A / D conversion processing, D / A conversion processing, image data encoding processing, image data compression processing, image data decoding processing, image data enlargement / reduction processing (resizing), image data noise reduction processing, and image data color conversion processing. The various image processing also includes panoramic development, mapping processing, and conversion of VR images, which are omnidirectional images or wide-range images having a wide-range image even if not omnidirectional. The image processing unit 204 may be a dedicated circuit block for performing specific image processing. Depending on the type of image processing, the CPU 201 can perform image processing according to a program without using the image processing unit 204.

[0042] The display 205 displays images and a GUI screen constituting a GUI (Graphical User Interface) under the control of the CPU 201. The CPU 201 generates a display control signal according to a program, and controls each unit of the display control device 200 to generate a video signal for display on the display 205 and output it to the display 205. The display 205 displays an image based on the generated and output video signal. Note that the configuration of the display control device 200 itself is limited to an interface for outputting a video signal for display on the display 205, and the display 205 may be an external monitor (for example, a television or a head-mounted display).

[0043] The operation unit 206 includes various input units for receiving user operations. For example, the operation unit 106 includes a character information input device (e.g., a keyboard), a pointing device (e.g., a mouse or a touch panel), a button, a dial, a joystick, a touch sensor, and a touch pad. In this embodiment, the operation unit 206 includes a touch panel 206a, operation members 206b, 206c, 206d, and 206e, and a gesture detection unit 206f.

[0044] A recording medium 208 such as a memory card, CD, or DVD can be detachably attached to the recording medium I / F 207. The recording medium I / F 207 reads data from the attached recording medium 208 and writes data to the recording medium 208 under the control of the CPU 201. The recording medium 108 is a storage unit that stores various data including images to be displayed on the display 105.

[0045] The external I / F 209 is an interface for connecting to an external device by wire or wirelessly and inputting and outputting video signals and audio signals. The communication I / F 210 is an interface for communicating with an external device or the Internet 211 and transmitting and receiving various data such as files and commands. The communication I / F 210 can also communicate with the controller 216.

[0046] The audio output unit 212 outputs the sound of the video or music data played back by the display control device 200, operation sounds, ringtones, various notification sounds, etc. The audio output unit 212 includes an audio output terminal 212a for connecting earphones or an external speaker, and a speaker 212b, but the audio output unit 112 may output audio data to an external speaker via wireless communication.

[0047] The attitude detection unit 213 detects the attitude (tilt) of the display control device 200 with respect to the gravity direction, and the attitude of the display control device 200 with respect to each axis of the yaw direction, pitch direction, and roll direction, and notifies the CPU 201 of the attitude information. Based on the attitude detected by the attitude detection unit 213, it is possible to determine whether the display control device 200 is held horizontally, held vertically, facing up, facing down, or in an oblique attitude. It is also possible to determine the presence or absence and magnitude of the tilt of the display control device 200 in the rotation directions such as the yaw direction, pitch direction, and roll direction, and whether the display control device 200 has rotated in the rotation direction. One sensor or a combination of multiple sensors from among an acceleration sensor, a gyro sensor, a geomagnetic sensor, a direction sensor, and an altitude sensor can be used as the attitude detection unit 213.

[0048] The self-position / surrounding environment estimation unit 214 estimates the self-position and surrounding environment of the display control device 200 or the VR goggles 230 described below.

[0049] The self-position is the position of the display control device 200 or the VR goggles 230 in a space of a predetermined range. The self-position is expressed by three parameters that represent a position in a coordinate system defined by three axes, the X-axis, the Y-axis, and the Z-axis, which are mutually orthogonal at a predetermined position in the space of a predetermined range, so that the predetermined position is the origin. The self-position may be expressed by further using three parameters that represent the attitude (orientation).

[0050] The surrounding environment includes an obstacle region. The obstacle region is a region of objects that exist around the display control device 200 or the VR goggles 230 and that are obstacles to a user holding the display control device 200 or a user wearing the VR goggles 230. For example, the obstacle region is expressed by a plurality of sets of three parameters that represent a position in a coordinate system defined by three axes, an X axis, a Y axis, and a Z axis, which are mutually orthogonal at a predetermined position within a predetermined range of space so that the predetermined position is the origin.

[0051] The imaging unit 215 is a camera capable of capturing an image of the real space. The image captured of the real space can be used for various detection processes, and is used, for example, by the gesture detection unit 206f and the self-position / surrounding environment estimation unit 214. The image captured of the real space can be displayed on the display 205.

[0052] The operation unit 206 includes a touch panel 206a. The touch panel 206a is configured to be planar and overlaid on the display 205, and is an input device that outputs coordinate information according to a touched position. The CPU 201 can detect the following operations or states on the touch panel 206a. The operation object (for example, a finger or a pen) that has not been touching the touch panel 206a has newly touched the touch panel 206a, that is, the start of touching (hereinafter, referred to as touchdown) (called Touch-Down) A state in which the operating object touches the touch panel 206a (hereinafter referred to as Touch-On) The operating object is moved while touching the touch panel 206a (hereinafter referred to as Touch-Move). The operation object that had been touching the touch panel 206a is removed from the touch panel 206a, that is, the touch ends (hereinafter, referred to as "touch-up"). A state in which nothing is touching the touch panel 206a (hereinafter referred to as Touch-Off)

[0053] When touch-down is detected, touch-on is also detected at the same time. After touch-down, touch-on will usually continue to be detected unless touch-up is detected. If touch-move is detected, touch-on is also detected at the same time. Even if touch-on is detected, touch-move will not be detected if the touch position does not move. When it is detected that all operating objects that were touching have touched up, touch-off is detected.

[0054] These operations and states, as well as the position coordinates of the operating object touching the touch panel 206a, are notified to the CPU 201 via an internal bus. The CPU 201 determines what kind of operation (touch operation) has been performed on the touch panel 206a based on the notified information. For touch-move, the moving direction of the operating object moving on the touch panel 206a can also be determined for each of the vertical and horizontal components on the touch panel 206a based on changes in the position coordinates. If a touch-move of a predetermined distance or more is detected, it is determined that a slide operation has been performed.

[0055] An operation in which the operating object is touched on the touch panel 206a, quickly moved a certain distance, and then released is called a flick. In other words, a flick is an operation in which the operating object is quickly traced on the touch panel 206a as if flicking it. When a touch-move of a predetermined distance or more at a predetermined speed or more is detected and a touch-up is detected, it can be determined that a flick has been performed (it can be determined that a flick has occurred following a slide operation).

[0056] Furthermore, a touch operation in which multiple points (for example, two points) are touched simultaneously and the touch positions are brought closer to each other is called pinch in, and a touch operation in which the touch positions are moved away from each other is called pinch out. Pinch out and pinch in are collectively called pinch operation (or simply pinch). The touch panel 206a may be of any of various types such as a resistive film type, a capacitive type, a surface acoustic wave type, an infrared type, an electromagnetic induction type, an image recognition type, and an optical sensor type. There are types that detect a touch by contact with the touch panel, and types that detect a touch by the approach of an operating object to the touch panel, and either type may be used.

[0057] The operation unit 206 includes a gesture detection unit 206f. The gesture detection unit 206f detects a gesture of a user (for example, a user's hand) from an image (an image of a real space) acquired by the imaging unit 215 under the control of the CPU 201. The CPU 201 performs various processes (controls) according to the detected gesture.

[0058] FIG. 2(D) is an external view of VR goggles (head mount adapter) 230 to which the display control device 200 can be attached. The display control device 200 can also be used as a head mount display by attaching it to the VR goggles 230. The insertion port 231 is an insertion port for inserting the display control device 200. The entire display control device 200 is inserted into the VR goggles 230 with the display surface of the display 205 facing the headband 232 side (i.e., the user side) for fixing the VR goggles 230 to the user's head. The user can view the display 205 without holding the display control device 200 with his / her hands while wearing the VR goggles 230 on which the display control device 200 is attached on his / her head. In this case, when the user moves his / her head or whole body, the attitude of the display control device 200 also changes. The attitude detection unit 213 detects the attitude change of the display control device 200 at this time, and the CPU 201 performs processing for VR display (display in the display mode "VR view") based on this attitude change. In this case, the detection of the attitude of the display control device 200 by the attitude detection unit 213 is equivalent to detecting the attitude of the user's head (the direction in which the user's line of sight is facing). The VR display (VR view) is a display method (display mode) in which an image of a field of view range corresponding to the attitude of the display control device 200 is displayed among VR images, and the display range can be changed.

[0059] The display control device 200 itself may be a head-mounted display that can reach the head without VR goggles. The display control device 200 may be capable of detecting the user's line of sight or facial expression, and the user may be able to operate the display control device 200 by the line of sight or facial expression.

[0060] 3A is an external view of controllers 340, 350, which are an example of the controller 216 capable of communicating with the display control device 200. The controller 340 is a grip-type controller that the user operates with the left hand. The user holds a holding portion 341 of the controller 340 with the left hand, and operates an operation member arranged on an operation surface 342 with the fingers (e.g., thumb) of the left hand. The controller 340 notifies the display control device 200 of an operation performed by the user. The controller 350 is a grip-type controller that the user operates with the right hand, and has a similar configuration to the controller 340 (e.g., a form obtained by flipping the controller 340 horizontally).

[0061] Note that controller 360 shown in FIG. 3B may be used as controller 216. Controller 360 is a ring-type controller that is worn on the user's finger and operated. Controller 360 has a ring unit 361 for wearing on user's finger 253, and an operation member 362. Operation member 362 is, for example, a push button, a rotary dial, or an optical track pad. The optical track pad is a type of operation member that can detect contact or approach of a finger.

[0062] 4 is a flowchart illustrating an example of image sharing processing in the display control device 200. The image sharing processing is realized by expanding a program recorded in the non-volatile memory 203 (for example, a program of a specific application for performing VR display and sharing processing of a VR image recorded in the recording medium 208) in the memory 202 and executing the program by the CPU 201. The CPU 201 starts the image sharing processing, for example, when a user performs an operation to display a list of images on the display 205 in order to share a VR image.

[0063] The display control device 200 has a function of performing VR display of a VR image. When performing VR display, a display range that is a part of the VR image is displayed. The display control device 200 also has a function of transmitting an image file of a VR image recorded in a recording medium 208 to a server or the like on a network so that other users can view the VR image. Hereinafter, the function of transmitting a VR image to an external device so that other users can view the VR image is referred to as a sharing function.

[0064] When using the sharing function of a VR image, a user can specify an external device to which the image is to be sent (destination), a display range of the VR image to be shared (display direction, which is the direction from the center of the VR space to the center of the display range), etc. Then, the display control device 200 transmits additional information including orientation information related to the specified display range (display direction) to the external device together with the VR image as metadata, etc. When another user accesses the VR image (shared image) on the external device and changes the VR display, When the orientation information is sent, the display range according to the sent orientation information is displayed when the VR display starts.

[0065] When the image sharing process is started, in step S401, the CPU 201 acquires a list of image data stored in the recording medium 208. In step S402, the CPU 201 instructs the display 205 to display a list of thumbnail images of the image data stored in the recording medium 208 based on the acquired list.

[0066] 5A is an example of a screen displaying a list of thumbnail images on the display 205 of the display control device 200. A plurality of thumbnail images 502 are displayed (arranged) on the display 205. In step S403, the CPU 201 determines whether or not a selection operation has been performed on the operation unit 206 to select one of the thumbnail images displayed in the list. If a thumbnail image has been selected, the CPU 201 proceeds to step S407, where it performs a sharing process for the VR image corresponding to the thumbnail image selected in response to the selection operation. If a thumbnail image has not been selected, the process proceeds to step S404.

[0067] In step S404, the CPU 201 determines whether or not the user has performed other operations (operations other than image selection) on the operation unit 206. If other operations have been performed, the process proceeds to step S405, and if other operations have not been performed, the process proceeds to step S406. In step S405, the CPU 201 executes processing according to the other operations performed in step S404 (other processing). The other operations include, for example, a scroll operation for scrolling the screen display to display a thumbnail image that is not displayed in a list display of thumbnail images, and an operation for duplicating or deleting a VR image corresponding to a thumbnail image.

[0068] In step S406, the CPU 201 determines whether or not an end operation for ending the list display of images has been performed on the operation unit 206. If an end operation has been performed, the image sharing process shown in Fig. 4 ends. If an end operation has not been performed, the process returns to step S402.

[0069] The processes from step S407 to step S422 are an example of a process for sharing a VR image corresponding to the thumbnail image selected in step S403. In step S407, the CPU 201 reads out and acquires, from the recording medium 208, the VR image corresponding to the selected thumbnail image.

[0070] In step S408, the CPU 201 displays the reference range of the VR image acquired in step S407 on the display 205. The reference range is a range displayed on the display 205 with the reference direction facing forward. Specifically, the CPU 201 extracts orientation information (for example, orientation information generated by an imaging device or the like when the VR image is captured and recorded together with the VR image) from additional information of the image file of the acquired VR image. Hereinafter, the orientation (orientation for VR display; display direction) indicated by this orientation information is referred to as the reference direction. The reference direction can be, for example, the optical axis direction of the lens of the imaging device. The CPU 201 determines, as the display range (reference range), a range that is displayed in accordance with the size of the display 205, with the position indicated by the reference direction of the acquired VR image as the center. The CPU 201 displays an image of the determined display range on the display 205.

[0071] FIG. 5(B) is an example of a screen in which a VR image selected from the list of thumbnail images in FIG. 5(A) is displayed on the display 205 of the display control device 200. The display 205 displays a VR image 510. The display 205 also displays a share instruction for sharing the VR image 510 on an external device and a social networking service (SNS). A share button 511 for sharing the image is displayed.

[0072] In step S409, the CPU 201 determines whether or not a change operation (change instruction) for changing the display direction of the VR display has been performed on the operation unit 206. The user may perform a change instruction by performing an operation on the operation unit 206, or may perform a change instruction by another method. For example, as shown in FIG. 5(B), the user may perform a touch-move operation on the touch panel 206a in the direction of the arrow 512, or may perform a change instruction by changing the attitude of the display control device 200. If a change operation has been performed, the process proceeds to step S410, and if no change operation has been performed, the process proceeds to step S411.

[0073] In step S410, the CPU 201 changes the display range of the VR image 510 displayed on the display 205 in response to the change operation in step S409. Fig. 5(C) shows an example of a screen in which the display range has been changed by performing a touch-move operation in the direction of the arrow 512 on the screen shown in Fig. 5(B).

[0074] In step S411, the CPU 201 determines whether or not a sharing instruction has been issued to transmit the displayed VR image to an external device as a shared image. The user can issue an instruction to share the VR image by, for example, touching the share button 511 on the touch panel 206a. If a sharing instruction has been issued, the process proceeds to step S412, and if no sharing instruction has been issued, the process proceeds to step S420.

[0075] In step S412, the CPU 201 instructs the display 205 to display a shared menu screen. Fig. 5(D) is an example of a shared menu screen. When the share button 511 is touched in the state of Fig. 5(B) or Fig. 5(C), the display screen of the display 205 transitions to the screen of Fig. 5(D). In Fig. 5(D), a shared menu screen 520 is displayed superimposed on the VR image 510. The shared menu screen 520 includes a button 521, a button 622, and a button 623.

[0076] Button 521 ("Share") is a button for instructing that the reference direction (reference range) displayed in step S408 be displayed. Button 522 ("Share in this orientation") is a button for instructing that the display direction (display range) after being changed in step S410 be displayed. Button 523 ("Share with any subject") is a button for instructing that the display direction (display range) including a subject specified by the user be displayed. Button 523 is displayed when VR image 510 is a VR video including multiple frames.

[0077] In step S413, CPU 201 determines whether or not button 523, which instructs that a display direction including a subject designated by a user be displayed, has been selected. If button 523 has been selected, the process proceeds to step S414. If button 523 has not been selected and button 521 or button 522 has been selected, the process proceeds to step S419.

[0078] In step S414, the CPU 201 displays a subject selection screen. Fig. 5(E) is an example of the subject selection screen. The display 205 displays a seek bar 530 for indicating the playback position of the video and a slider 531 for indicating the current playback position, superimposed on the VR image 510.

[0079] In step S415, CPU 201 determines whether the user has performed an operation to select a specific subject that the user wishes to show to the viewer. The operation to select a subject includes, for example, an operation to touch an area on touch panel 206a where the desired subject is displayed, or an operation to move a finger so as to surround the desired subject. In FIG. 5(E), the user has selected (specified) The particular subject is indicated by being enclosed in a subject selection frame 532 .

[0080] If the user performs an operation to select a subject, the process proceeds to step S416. If the user has not selected a subject, CPU 201 returns to step S415 and waits until the user performs an operation to select a subject. If the user has not selected a subject even after a predetermined time has elapsed, CPU 201 may proceed to the process of step S419.

[0081] In step S416, CPU 201 determines whether or not to confirm the subject selected in step S415. CPU 201 displays a confirmation screen for confirming with the user whether or not to confirm the selected subject. FIG. 5(F) is an example of the confirmation screen. Display 205 displays confirmation screen 540, YES button 541, and NO button 542 superimposed on VR image 510. When the user presses (tap) YES button 541 to confirm the selected subject, the process proceeds to step S417. When the user presses (tap) NO button 542, CPU 201 returns to step S415 and waits until the subject selection operation is received from the user.

[0082] In step S417, CPU 201 detects the subject from each frame of the VR video based on the data of the subject selected in step S415. CPU 201 generates, for each frame of the VR video, the difference in angle between the reference direction of the frame and the direction of the detection position of the subject as correction data for the reference direction. The reference direction of the frame is set for each frame when the VR video is shot, and is the direction that is the center of the display range. The reference direction of the frame is added to the file of the VR video as additional information.

[0083] In step S418, the CPU 201 changes (corrects) the reference direction of each frame of the VR video based on the correction data generated in step S417. The CPU 201 generates a VR video file in which the reference direction of each frame of the VR video is corrected and re-encoded, and stores the file in the recording medium 208. The CPU 201 can generate a VR video corrected so that a specific subject is tracked and sequentially played in an initial state by correcting the reference direction of each frame of the VR video and re-encoding the VR video. The initial state is a state before the display direction (display range) is changed due to a user operation or a change in the posture of the display control device 200. The CPU 201 may replace the original VR video with the corrected VR video in which the reference direction is corrected and store the corrected VR video, or may generate the corrected VR video as data separate from the original VR video and store the corrected VR video in the recording medium 208. The correction data may be linked to the file of the VR video and stored in the recording medium 208, or may be added to the file of the VR video and stored.

[0084] In step S419, CPU 201 transmits the VR video file (image data) generated in step S418 to an external device, and shares the image data as a shared image. If button 521 ("Share") is selected in step S413, CPU 201 transmits the image data to an external device or the like without changing the reference direction set at the time of shooting. If button 522 ("Share in this orientation") is selected in step S413, CPU 201 corrects the reference direction of each frame to the display direction being displayed (current display direction), and transmits the image data to an external device or the like. The display direction being displayed is the current display direction after the user changes the display range by a touch-move operation or the like.

[0085] In step S420, CPU 201 determines whether or not another operation (an operation other than a sharing instruction) has been performed on operation unit 206. If another operation has been performed, the process proceeds to step S421. If another operation has not been performed, the process proceeds to step S422. In step S421, CPU 201 executes a process (other process) according to the other operation performed in step S420. The other operation may be, for example, changing the color tone of an image. This also includes editing operations such as adjusting brightness, and operations to cut out parts of VR images.

[0086] In step S422, the CPU 201 determines whether or not an end operation has been performed to end the process for sharing the VR image selected in step S403. If an end operation has been performed, the process proceeds to S404, and if an end operation has not been performed, the process returns to step S409.

[0087] Fig. 6 is a flowchart illustrating an image reproduction process in the display control device 200. Fig. 7 is a flowchart illustrating a selected image reproduction process of an image selected from a list of displayed images. Figs. 6 and 7 show an example of a process when a VR image generated in the image sharing process of Fig. 4 is viewed as a shared image on a social networking service (SNS) together with other images. The image reproduction process and the selected image reproduction process can be executed by using the display control device 200.

[0088] 6 and 7 are realized by expanding a program recorded in non-volatile memory 203 (for example, a program of a specific application for VR display of VR images recorded in recording medium 208) in memory 202 and executing the program by CPU 201. CPU 201 starts image playback processing, for example, when a user performs an operation to display a list of shared images on display 205 in order to play back a shared VR image.

[0089] When the image playback process is started, in step S601, the CPU 201 acquires a list of image data stored in the recording medium 208. In step S602, the CPU 201 acquires information on the reference direction of the VR image from among the images read from the recording medium 208. The information on the reference direction of the VR image may be added as metadata to the image data file of the VR image, or may be stored in a separate file associated with the image data file. In step S603, the CPU 201 displays a list of the images (shared images) acquired in step S601.

[0090] 8(A) is an example of a screen displaying a list of the images acquired in step S601 on the display 205 of the display control device 200. The display 205 displays a timeline 700 of an SNS, and the timeline 700 includes still images 701 and omnidirectional still images 702 that are displayed side-by-side in the vertical direction. The user can scroll the images included in the timeline 700 by performing a scroll operation of touch-moving on the touch panel 206a in the direction indicated by the arrow 710a.

[0091] Objects 701a and 702a, which are GUI objects indicating the type of image, are superimposed on still images 701 and 702. Object 701a ("STILL") indicates that the image is a still image that is not a VR image. Object 702a ("360° STILL") indicates that the image is a still image of a VR image (here, an omnidirectional image).

[0092] In step S604, the CPU 201 determines whether or not a selection operation for selecting any of the images displayed in the list has been performed on the operation unit 206. If an image has been selected, the process proceeds to step S605, and if an image has not been selected, the process proceeds to step S606.

[0093] In step S606, the CPU 201 determines whether or not a scroll operation has been performed on the list display of images. If a scroll operation has been performed, the process proceeds to step S607. If a scroll operation has not been performed, the process proceeds to step S615.

[0094] In step S607, CPU 201 determines whether or not the image displayed on display 205 is a VR video among the list of images included in timeline 700. If the image displayed on display 205 is a VR video, processing proceeds to step S611, and if the image displayed on display 205 is not a VR video, processing proceeds to step S608.

[0095] In step S608, the CPU 201 determines whether or not a scroll operation is being performed. If a scroll operation is being performed, the process proceeds to step S609, and if a scroll operation is not being performed, the process proceeds to step S614.

[0096] In step S609, the CPU 201 disables the display direction change operation (predetermined operation). That is, the CPU 201 performs control so as not to change the display range based on the display direction change operation even if the display direction change operation is received from the user while the image list is being scrolled. The reason for disabling the display direction change operation is that while the user is browsing the image list on the timeline 700 while scrolling it, the display range of the VR image is controlled to be in the direction that the sharer of the VR image wants to show.

[0097] The display direction change operation is, for example, an operation of changing the display range by a touch-move operation on the touch panel 206a on the VR image, and an operation of changing the attitude of the display control device 200. When the user changes the attitude of the display control device 200, the CPU 201 performs control so as to change the display range of the VR image based on the position and attitude of the display control device 200 detected by the attitude detection unit 213.

[0098] In step S610, when a still VR image is displayed by a scroll operation on the timeline 700 (image list), the CPU 201 performs pan display to move the display range of the VR image, for example, in the horizontal direction of the display 205. By performing pan display, the CPU 201 can inform the user that the image being displayed is a VR image and that the display range can be changed.

[0099] FIG. 8(B) is an example of a screen in which a still image 702, which is an omnidirectional image, is displayed on the display 205 during a scroll operation of the timeline 700. When the still image 702 is displayed on the display 205 in response to a scroll operation by a touch move in the direction of the arrow 710b, the CPU 201 displays a display range centered on the reference direction of the still image 702, and then performs a pan display in which the display range is moved in the direction of the arrow 711. The reference direction of the VR image of the still image may be the direction set at the time of shooting, or may be the direction specified by the user (sharer). By performing a pan display, the display control device 200 can display the direction that the sharer of the VR image wants the viewer to see, and can also indicate to the user that the displayed image is a VR image.

[0100] In the process of step S610, CPU 201 performs control to automatically change the display range, so if the display range is further changed based on a user operation, the process of changing the display range will be duplicated. In order to avoid unexpected viewpoint movement due to the overlap of the process of changing the display range, CPU 201 disables the user's operation of changing the display range in step S609. Note that if the image displayed on display 205 is not a VR image, CPU 201 does not need to execute the processes of steps S609 and S610.

[0101] In step S611, the CPU 201 automatically starts playing the VR video. FIG. 8(C) is an example of a screen when the timeline 700 is further scrolled downward from the state of FIG. 8(B) by a scroll operation using a touch move in the direction of the arrow 710b. On the screen after scrolling, a video 703, which is an omnidirectional image, is displayed on the display 205. It is being done.

[0102] The display 205 displays an object 703a and a seek bar 703b superimposed on the video 703. The object 703a ("360° MOVIE") is a GUI object indicating the type of image, and indicates that the video 703 is a VR image (here, an omnidirectional image). The seek bar 703b indicates the playback position of the video 703.

[0103] Fig. 8(D) is an example of a screen when the timeline 700 is further scrolled downward by a scroll operation by touch-move in the direction of the arrow 710b from the state of Fig. 8(C). The seek bar 703b indicates that the automatic playback of the video 703 has progressed further than in the state of Fig. 8(C).

[0104] Subject 712 is a specific subject determined in step S416, and in each frame of moving image 703, the reference direction is corrected to the direction in which subject 712 exists. CPU 201 determines the display range in each frame of moving image 703 so that the reference direction is at the center, thereby enabling playback so that subject 712 is displayed in front.

[0105] In step S612, the CPU 201 determines whether or not the VR video is being automatically played back. If the VR video is being automatically played back, the process proceeds to step S613, and if the VR video is not being automatically played back, the process proceeds to step S614.

[0106] In step S613, CPU 201 invalidates the display direction change operation, similarly to step S609. CPU 201 determines the display range of each frame based on the reference direction corrected in steps S417 and S418 of FIG. 4, and plays the VR video. That is, CPU 201 can change the display range during playback of the VR video to a desired range by correcting the reference direction of each frame to a direction that the sharer of the VR video wants the viewer to see (for example, a direction in which a specific subject exists). If the display range is further changed based on a user's operation during playback of the VR video, the display range change process will be duplicated. In order to avoid unexpected viewpoint movement due to the overlap of display range change processes, CPU 201 invalidates the user's operation of changing the display range until playback of the VR video ends.

[0107] In step S614, the invalidation of the display direction change operation in steps S609 and S613 is cancelled. That is, when the CPU 201 accepts a display direction change operation from the user, the CPU 201 changes the display range of the VR image based on the user operation.

[0108] Steps S615 and S616 are similar to steps S409 and S410, respectively, of the image sharing process shown in FIG.

[0109] In step S617, CPU 201 determines whether or not other operations (operations other than the operation of changing the display orientation) have been performed on operation unit 206. If other operations have been performed, the process proceeds to step S618, and if other operations have not been performed, the process proceeds to step S619. In step S618, CPU 201 executes processing according to the other operation performed in step S617 (other processing). Other operations include, for example, operations such as posting a comment or selecting a reaction icon in response to content posted by another person.

[0110] In step S619, the CPU 201 determines whether or not an end operation has been performed to end the image playback process shown in Fig. 6. If an end operation has been performed, the image playback process ends. If an end operation has not been performed, the process returns to step S601.

[0111] Fig. 7 is a flowchart illustrating the details of the selected image playback process of step S605 in Fig. 6. The processes from step S620 to step S626 are single playback processes for displaying and playing back the image selected in step S604 from the image list described with reference to Figs. 8(A) to 8(D) alone.

[0112] Fig. 8(E) is an example of a screen for playing video 703 selected from the image list shown in Fig. 8(D). For example, when a user selects a desired shared image from the image list by a touch operation or the like, CPU 201 switches to the screen of Fig. 8(E) which displays the selected shared image alone. Display 205 displays video 703 alone, and further displays seek bar 703d. Display 205 also displays back button 713 and VR display button 714. Back button 713 is a button (item) for returning to the image list shown in Fig. 8(D). VR display button 714 is a button (item) for VR displaying the image displayed on display 205.

[0113] In step S620, the CPU 201 starts playing the image selected in step S604. In step S621, the CPU 201 determines whether or not the VR display button 714 has been operated (selected). If the VR display button 714 has been operated, the process proceeds to step S627, and if the VR display button 714 has not been operated, the process proceeds to step S622.

[0114] In step S622, the CPU 201 determines whether or not a change operation (change instruction) to change the display direction of the VR display has been performed on the operation unit 206. If a change operation has been performed, the process proceeds to step S623, and if no change operation has been performed, the process proceeds to step S624. In step S623, the CPU 201 changes the display range of the VR image displayed on the display 205 in response to the change operation in step S622.

[0115] In step S624, CPU 201 determines whether or not other operations (operations other than the operation of changing the display orientation) have been performed on operation unit 206. If other operations have been performed, the process proceeds to step S625, and if other operations have not been performed, the process proceeds to step S626. Other operations include, for example, editing operations for the color and brightness of an image. In step S625, CPU 201 executes processing (other processing) according to the other operations performed in step S624.

[0116] In step S626, CPU 201 determines whether or not an end operation has been performed to end the selected image reproduction process shown in Fig. 7. An operation to end the selected image reproduction process is, for example, selecting back button 713. If an end operation has been performed, the selected image reproduction process ends, and the process proceeds to step S606 in Fig. 6. If an end operation has not been performed, the process returns to step S621.

[0117] The processes from step S627 to step S632 are processes for playing back the VR image by VR display for viewing through the VR goggles 230. In step S627, the CPU 201 plays back the VR image by VR display.

[0118] Fig. 8(F) is an example of a screen on which a VR image is played back by VR display. The display 205 displays a left eye image 723a and a right eye image 723b. The display 205 also displays a back button 715 and a single playback button 716. The back button 715 is a button (item) for ending the VR display and returning to the image list shown in Fig. 8(D). The single playback button 716 is a button (item) for ending the VR display of the image displayed on the display 205 and returning to step S622.

[0119] Steps S628 and S629 are similar to steps S622 and S623, respectively. When playing back a VR image by VR display, the CPU 201 changes the display range of the VR image displayed on the display 205 upon receiving a change operation (including an operation to change the attitude of the display control device 200) for changing the display direction of the VR display.

[0120] In step S630, CPU 201 determines whether or not other operations (operations other than the operation of changing the display direction) have been performed on operation unit 206. If other operations have been performed, the process proceeds to step S631, and if other operations have not been performed, the process proceeds to step S632. Other operations include, for example, an operation of moving a slider indicating the playback position of the VR video on a seek bar using controller 340 and controller 350 described in FIG. 3(A). In this case, the screen in FIG. 8(F) may display a seek bar similar to that in FIG. 8(E). In step S631, CPU 201 executes a process (other process) according to the other operation performed in step S630.

[0121] In step S632, the CPU 201 determines whether or not an end operation has been performed to end the VR display of the VR image. The end operation to end the VR display is, for example, a selection operation of the back button 715 or the single playback button 716. If an end operation has been performed, the process proceeds to step S622. Note that if an end operation has been performed, the CPU 201 may proceed to step S606. If an end operation has not been performed, the process returns to step S628.

[0122] Note that the VR image is not limited to a 360° spherical image with 360° in the vertical and horizontal directions, but may be an image with 180° or less in the vertical and horizontal directions. Furthermore, an image compatible with the VR180 standard may be superimposed with a GUI object indicating that the image is compatible with the standard.

[0123] 4 shows an example of a process of correcting the reference direction of the VR video and re-encoding the VR video using the generated correction data, but is not limited thereto. For example, the display control device 200 may store the correction data as a file separate from the VR video file, and record the correction data file and the VR video file in association with each other on the recording medium 208. In this case, in step S419, the display control device 200 may transmit the original VR video file whose reference direction has not been changed and the corresponding correction data file to the external device. Then, when playing the VR video in step S611 of FIG. 6, the display control device 200 plays the VR video while correcting the reference direction of each frame of the original VR video using the correction data.

[0124] Furthermore, when displaying the sharing menu in step S412 in Fig. 4, the display control device 200 may allow the user to select multiple sharing destinations. Furthermore, the display control device 200 may allow the user to select a sharing method from among button 521 ("Share"), button 522 ("Share in this orientation"), and button 523 ("Share with any subject") for each sharing destination to which the VR video is to be transmitted.

[0125] Furthermore, for a frame in which a specific subject specified by the user is not detected (does not exist) in step S417 of FIG. 4, the display control device 200 can correct the reference direction based on the position of the specific subject detected in another frame. The other frames are, for example, frames before and after the frame in which the subject was not detected. A subject that was detected in a VR image until just before may become undetectable, for example, when the subject turns away and a characteristic part such as a face is no longer detected, or when the subject moves away and is no longer detected. By adopting the position of the subject detected in the most recent frame, the display control device 200 can appropriately correct the reference direction even for a frame in which the subject was not detected. .

[0126] In addition, the specific subject to be tracked in step S415 is not limited to a specific person or animal, and may be specified by the type of the tracking target, such as a person, an animal, a car, or an airplane. The specific subject may be specified by a specific subject and the type of the tracking target. By specifying the specific subject by a combination of the specific subject and the type of the tracking target, the accuracy of detection of the specific subject is improved.

[0127] Also, in step S419 of Figure 4, when the corrected VR image (image data of the VR video) is transmitted to an external device and shared, the reference direction of the original VR image stored in the recording medium 208 may be overwritten with the reference direction of the shared VR image.

[0128] 6 may be looped. The operation of changing the display direction is disabled until the video being automatically played is hidden by scrolling the timeline 700. The display control device 200 may cancel the disabling of the operation of changing the display direction when the playback ends once, and may change the display range according to the operation of changing the display direction by the user during the second or subsequent playback. Also, in step S611, only the VR video shared by operating the button 523 ("Share with any subject") may be automatically played.

[0129] The button 523 ("Share with any subject") may also be displayed when the image acquired in step S407 in Fig. 4 is a VR image of a still image. In particular, when a VR image of a plurality of still images captured sequentially by continuous shooting (continuous shooting) or interval shooting at a predetermined time interval is acquired as a set of images of still images, the display control device 200 may display the button 523.

[0130] 6, the process for the display direction change operation differs depending on whether the image displayed on the display 205 is a VR video or a still image (including images other than VR images). Without being limited to this, a video (short video) of a predetermined number of seconds or less among VR videos may be processed for the display direction change operation in the same manner as a still image. For example, since the playback of a short video ends in a predetermined number of seconds or less, the display direction change operation by the user may be disabled during a scroll operation of the timeline 700, but the disablement may be canceled when the scroll operation ends.

[0131] Furthermore, the display control device 200, which is an example of an electronic device according to the present invention, may be configured as a housing integrated with the VR goggles 230. Furthermore, the display control device 200 may be configured as an integrated unit with the digital camera 100.

[0132] In the above embodiment, the display control device 200 corrects the reference direction of each of a plurality of VR images (including a VR video, and a plurality of VR images obtained by interval shooting or continuous shooting) captured in sequence to a direction in which a specific subject that the sharer wants to show is present. By correcting the reference direction, the corrected VR images are played back in sequence in the initial state while tracking the specific subject designated by the sharer. Also, unlike a video generated by cutting out a specific subject, the corrected VR images according to this embodiment allow the user to change the display range in the same way as the original VR image.

[0133] Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to these specific embodiments and includes various forms within the scope of the gist of the present invention. Furthermore, the above-mentioned embodiment is merely one embodiment of the present invention, and various forms can be appropriately combined.

[0134] The present invention can also be realized by executing the following process. That is, software (programs) that realize the functions of the above-mentioned embodiments are supplied to a system or device via a network or various storage media. The computer (or CPU, MPU, etc.) of the system or device reads the supplied programs into memory and executes them. In this case, these programs and the storage media that store the programs constitute the present invention.

[0135] The various controls (processing) described as being executed by the CPU 201 and the system control unit 50 may be executed by a single piece of hardware, or may be executed by sharing the processing among a plurality of pieces of hardware.

[0136] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0137] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) An acquisition means for acquiring a plurality of VR images captured sequentially; a generating means for generating a VR image from each of the plurality of VR images by correcting a reference direction of the VR image to a direction in which a specific subject exists in the VR image; 1. An electronic device comprising: (Configuration 2) The generating means generates, for each of the plurality of VR images, correction data for correcting the reference direction of the VR image based on a difference between the reference direction of the VR image and a direction in which the specific subject exists in the VR image, and corrects the reference direction of each of the plurality of VR images using the correction data. 2. The electronic device according to configuration 1. (Configuration 3) The present invention further includes a transmission means for transmitting the plurality of VR images and the correction data to an external device, or for transmitting the plurality of VR images obtained by correcting the reference direction of each of the plurality of VR images by the generation means to the external device. 3. The electronic device according to configuration 2. (Configuration 4) The method further includes a storage unit that controls the storage of the VR images and the correction data in a recording medium in association with each other. 4. The electronic device according to configuration 2 or 3. (Configuration 5) The acquisition means acquires a VR video including a plurality of frames corresponding to the plurality of VR images, respectively. 5. The electronic device according to any one of configurations 1 to 4. (Configuration 6) The specific subject is specified by the user. 6. The electronic device according to any one of configurations 1 to 5. (Configuration 7) The generating means corrects the reference direction of each of the plurality of VR images and generates the plurality of corrected VR images as data separate from the plurality of VR images. 7. The electronic device according to any one of configurations 1 to 6. (Configuration 8) The generating means corrects the reference direction of each of the plurality of VR images, replaces the reference direction with the plurality of VR images, and stores the result. 7. The electronic device according to any one of configurations 1 to 6. (Configuration 9) The present invention further includes an instruction means for instructing, based on an operation by the user, whether to correct the reference direction of the VR image based on a direction in which the specific subject exists or to correct the reference direction of the VR image based on a direction specified by the user. 9. The electronic device according to any one of configurations 1 to 8. (Configuration 10) The instruction means instructs, for each destination of the plurality of VR images, whether to correct the reference direction of the VR image based on a direction in which the specific subject exists or to correct the reference direction of the VR image based on a direction specified by the user, based on an operation of the user. 10. The electronic device according to configuration 9. (Configuration 11) The generating means corrects the reference direction of the VR image in which the specific subject does not exist, based on the position of the specific subject detected in another VR image. 11. The electronic device according to any one of configurations 1 to 10. (Configuration 12) a switching means for switching between displaying a list of the multiple shared images and displaying one of the multiple shared images individually; a display control means for controlling a change in a display range of each of the plurality of shared images; and The display control means When the shared image is a plurality of VR images after correction generated by the generating means and the shared image is displayed alone, even if a predetermined operation for changing a display direction is received from a user during playback of the shared image, control is performed so that a display range based on the predetermined operation is not changed; When the plurality of shared images are displayed in a list, while the user is performing an operation of scrolling the list of the plurality of shared images, even if the predetermined operation is accepted, control is performed so that the display range is not changed based on the predetermined operation. 12. The electronic device according to any one of configurations 1 to 11. (Configuration 13) The predetermined operation is an operation of changing the attitude of the electronic device. 13. The electronic device according to configuration 12. (Configuration 14) The display control means controls the display range to be changed based on the predetermined operation when the shared image is a plurality of VR images after correction generated by the generation means and the shared image is displayed in VR alone. 14. The electronic device according to configuration 12 or 13. (Configuration 15) The plurality of VR images are a plurality of images captured continuously. 15. The electronic device according to any one of configurations 1 to 14. (Configuration 16) The plurality of VR images are images captured at a predetermined time interval. 15. The electronic device according to any one of configurations 1 to 14. (Configuration 17) An acquisition means for acquiring a plurality of VR images captured sequentially; A generating means for generating a plurality of VR images corrected so that the plurality of VR images are played back in sequence by tracking a specific subject; 1. An electronic device comprising: (method) An acquisition step of acquiring a plurality of VR images captured sequentially; a generating step of generating a VR image from each of the plurality of VR images by correcting a reference direction of the VR image to a direction in which a specific subject exists in the VR image; 13. A method for controlling an electronic device comprising: (program) A program for causing a computer to function as each of the means of the electronic device according to any one of configurations 1 to 17. [Explanation of symbols]

[0138] 200: display control device, 201: CPU

Claims

1. an acquisition means for acquiring a plurality of sequentially captured VR images; a generating means for generating a VR image in which a range displayed on a display is corrected so that a specific subject is included in each of the plurality of VR images; An electronic device comprising:

2. In the generated VR image, the specific subject is displayed in front.

2. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

3. When the VR image is generated with the correction in the direction in which the specific subject is included in the range displayed on the display, even if a first operation for changing the display range is received by a user, the control means further comprises a control means for performing a first control that does not change the display range based on the first operation.

2. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

4. The generating means generates the third VR image by correcting the range displayed on the display so as to include the specific subject, if the specific subject is included in a third VR image at a third time point after the second time point, even if the specific subject is not included in a second VR image at a second time point in the VR image.

2. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

5. The acquisition means acquires a VR video including a plurality of frames corresponding to the plurality of VR images.

5. The electronic device according to claim 1, wherein the first and second electrodes are electrically connected to the first and second electrodes.

6. The specific subject is designated by the user.

5. The electronic device according to claim 1, wherein the first and second electrodes are electrically connected to the first and second electrodes.

7. An item indicating the specific subject designated by the user is superimposed and displayed.

7. The electronic device according to claim 6, wherein the electronic device is a semiconductor device.

8. The generating means corrects the reference direction of each of the plurality of VR images, and Generate multiple corrected VR images as data separate from the image.

5. The electronic device according to claim 1, wherein the first and second electrodes are electrically connected to the first and second electrodes.

9. The generating means corrects the reference direction of each of the plurality of VR images, replaces the reference direction with the plurality of VR images, and stores the VR images.

5. The electronic device according to claim 1, wherein the first and second electrodes are electrically connected to the first and second electrodes.

10. The apparatus further includes an instruction means for instructing, based on an operation by the user, whether to correct the reference direction of the VR image based on the direction in which the specific subject exists or based on a direction specified by the user.

5. The electronic device according to claim 1, wherein the first and second electrodes are electrically connected to the first and second electrodes.

11. The instruction means instructs, for each destination of the plurality of VR images, whether to correct the reference direction of the VR images based on the direction in which the specific subject exists or to correct the reference direction of the VR images based on a direction specified by the user, based on an operation by the user.

11. The electronic device according to claim 10.

12. The electronic device described in any one of claims 1 to 4, characterized in that, for the VR image in which the specific subject does not exist, the generation means corrects the reference direction of the VR image based on the position of the specific subject detected in other VR images.

13. a switching means for switching between displaying a list of the plurality of shared images and displaying one of the plurality of shared images individually; a display control means for controlling a change of a display range of each of the plurality of shared images; and The display control means When the shared image is a plurality of corrected VR images generated by the generation means and the shared image is displayed singly, even if a predetermined operation for changing the display direction is received from a user during playback of the shared image, control is performed so that the display range is not changed based on the predetermined operation; When the plurality of shared images are displayed in a list, while the user is performing an operation to scroll the list of the plurality of shared images, even if the predetermined operation is accepted, control is performed so that the display range is not changed based on the predetermined operation.

5. The electronic device according to claim 1, wherein the first and second electrodes are electrically connected to the first and second electrodes.

14. The predetermined operation is an operation for changing the attitude of the electronic device.

14. The electronic device according to claim 13.

15. The display control means controls the display range to be changed based on the predetermined operation when the shared image is the plurality of VR images after the correction generated by the generation means and the shared image is displayed in VR alone.

14. The electronic device according to claim 13.

16. The plurality of VR images are a plurality of images captured continuously.

5. The electronic device according to claim 1, wherein the first and second electrodes are electrically connected to the first and second electrodes.

17. The plurality of VR images are a plurality of images taken at a predetermined time interval.

5. The electronic device according to claim 1, wherein the first and second electrodes are electrically connected to the first and second electrodes.

18. The VR image is an image with a wider range than the display range that can be displayed at one time on the display. be 5. The electronic device according to claim 1, wherein the first and second electrodes are electrically connected to the first and second electrodes.

19. The VR image is an omnidirectional image having a 360-degree field of view, an image having a 180-degree field of view, or a panoramic image.

20. The electronic device according to claim 18.

20. The VR image is an image obtained through multiple optical systems that input different optical images.

5. The electronic device according to claim 1, wherein the first and second electrodes are electrically connected to the first and second electrodes.

21. The VR image is an image generated via multiple imaging units that capture different image areas.

5. The electronic device according to claim 1, wherein the first and second electrodes are electrically connected to the first and second electrodes.

22. The VR image is an image included in a video, The VR image, an item indicating the playback position of the video, and an item indicating the current playback position are controlled to be displayed on the display.

5. The electronic device according to claim 1, wherein the first and second electrodes are electrically connected to the first and second electrodes.

23. an acquisition step of acquiring a plurality of sequentially captured VR images; a generating step of generating a VR image in which a range displayed on a display is corrected so that a specific subject is included in each of the plurality of VR images; 1. A method for controlling an electronic device, comprising:

24. A program for causing a computer to function as each of the means of the electronic device according to any one of claims 1 to 4.

Images