Information processing device, control method for information processing device, and program

Abstract

This system provides an efficient way to view VR videos. [Solution] Camera 100 plays back a VR video based on the video captured. Camera 100 displays a first VR video, which is an enlarged version of a first sub-region within the range indicated by the playback section, and then displays a second VR video, which is an enlarged version of a second sub-region within the range indicated by the playback section, by sequentially switching the display control for each of the multiple sub-regions.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, a control method for an information processing apparatus, and a program.

Background Art

[0002] In recent years, techniques for displaying Virtual Reality (hereinafter referred to as "VR") images with a stereoscopic effect and a wide viewing angle have become widespread, in which images with parallax and a wide viewing angle are acquired using two optical systems, and are mapped onto a virtual sphere for display. Their convenience is drawing attention in a wide range of fields including games, education, medical care, and tourism, and the production of high-quality VR images has become an important issue. Therefore, in checking the quality of VR images, it is necessary to efficiently check for problems such as reflections, unnatural seams, and mixing of unnecessary objects. In response to this, a technique for efficiently confirming an image with a wide viewing angle is known. For example, in Patent Document 1, a specific object is displayed in an enlarged manner during video playback. This makes it possible to efficiently confirm the quality at the viewing angle at which the specific object is displayed.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problem to be Solved by the Invention

[0004] However, in the above-described Patent Document 1, it is not possible to efficiently confirm an area that does not include a specific object.

[0005] An object of the present invention is to provide a mechanism that enables efficient confirmation of VR videos.

Means for Solving the Problem

[0006] To achieve the above objective, the information processing device of the present invention comprises: playback section determination means for determining a playback section of a VR video; partial region determination means for determining a plurality of partial regions from a single frame image in the VR video; and control means for controlling the display of the VR video on a display means. The control means is characterized in that it sequentially switches the display control for each of the plurality of partial regions, displaying a first VR video which is a first VR video of the range indicated by the playback section of the VR video and is an enlargement of the first partial region of the plurality of partial regions, and then displaying a second VR video which is a second VR video of the range indicated by the playback section of the VR video and is an enlargement of the second partial region of the plurality of partial regions. [Effects of the Invention]

[0007] According to the present invention, VR videos can be viewed efficiently. [Brief explanation of the drawing]

[0008] [Figure 1] This is an external view of the camera used as an information processing device according to this embodiment. [Figure 2] This figure shows an example of the internal configuration of the camera in Figure 1. [Figure 3] This diagram illustrates the loop playback settings screen displayed on the display unit in Figure 1. [Figure 4] This diagram illustrates the loop playback settings screen displayed on the display unit in Figure 1. [Figure 5] This figure shows how the sub-regions are sequentially displayed within the playback interval determined in this embodiment. [Figure 6] This flowchart shows the steps involved in the VR video playback process performed by the camera shown in Figure 1. [Figure 7] Figure 6 is a flowchart showing the procedure for the partial region selection process in S602. [Figure 8] Figure 6 is a flowchart showing the procedure for determining the playback region in S603. [Modes for carrying out the invention]

[0009] Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. In this embodiment, the case in which the information processing device is a digital camera will be described as an example. However, the information processing device is not limited to a digital camera that photographs a subject and generates a right-eye image and a left-eye image with parallax. The information processing device may be any device other than a digital camera that can acquire a right-eye image and a left-eye image with parallax, and display the two images obtained by performing perspective projection transformation on the two acquired images, respectively. Furthermore, the information processing device does not have to have a display means itself. For example, the information processing device may be a device that acquires a right-eye image and a left-eye image with parallax, performs perspective projection transformation on the two acquired images, respectively, and outputs the two images obtained by the perspective projection transformation to another device equipped with a display means.

[0010] Figure 1 is an external view of the camera 100 as an information processing device according to this embodiment. Figure 1(a) is an external perspective view of the camera 100 as seen from the front, and Figure 1(b) is an external perspective view of the camera 100 as seen from the rear.

[0011] Camera 100 has a shutter button 101, a power switch 102, a mode selector switch 103, a main electronic dial 104, a sub electronic dial 105, a video button 106, and an external viewfinder display 107 on its top surface. The shutter button 101 is an operation for preparing to shoot or giving a shooting command. The power switch 102 is an operation for switching the power of camera 100 on and off. The mode selector switch 103 is an operation for switching between various modes. The main electronic dial 104 is a rotary operation for changing settings such as shutter speed and aperture. The sub electronic dial 105 is a rotary operation for moving the selection frame (cursor) and advancing images. The video button 106 is an operation for giving a command to start and stop video recording. The external viewfinder display 107 displays various settings such as shutter speed and aperture.

[0012] The camera 100 also has a display unit 108, a touch panel 109, directional keys 110, a SET button 111, an AE lock button 112, a zoom button 113, a playback button 114, a menu button 115, an eyepiece 116, an eyepiece detection unit 118, and a touch bar 119 on its back. The display unit 108 displays images and various information. The touch panel 109 is an operation unit that detects touch operations on the display surface (touch operation surface) of the display unit 108. The directional keys 110 is an operation unit consisting of keys that can be pressed up, down, left, and right (4-way keys). Operations can be performed according to the position where the directional keys 110 are pressed. The SET button 111 is an operation unit that is mainly pressed when confirming a selection item. The AE lock button 112 is an operation unit that is pressed when fixing the exposure state in shooting standby mode. The zoom button 113 is an operation unit that switches the zoom mode on and off in the live view display (LV display) of the shooting mode. When the magnification mode is on, the live view image (LV image) can be enlarged or reduced by operating the main electronic dial 104. The magnification button 113 is used in playback mode to enlarge the playback image or increase the magnification ratio. The playback button 114 is an operation unit for switching between shooting mode and playback mode. By pressing the playback button 114 in shooting mode, the camera switches to playback mode, and the latest image recorded on the recording medium 227 (described later) can be displayed on the display unit 108.

[0013] The menu button 115 is an operation button that is pressed to display a menu screen on the display unit 108 that allows for various settings. Users can intuitively make various settings using the menu screen displayed on the display unit 108 and the directional keys 110 and SET button 111. The eyepiece section 116 is the part for placing the eye into the eyepiece viewfinder (peep-in type viewfinder) 117. Users can view the image displayed on the internal EVF 217 (Electronic View Finder), which will be described later, through the eyepiece section 116. The eyepiece detection section 118 is a sensor that detects whether or not the user is placing their eye into the eyepiece section 116.

[0014] The touch bar 119 is a line-shaped touch operation area (line touch sensor) that can accept touch operations. The touch bar 119 is positioned so that it can be touched by the right thumb when the grip section 120 is held with the right hand (with the little finger, ring finger, and middle finger) so that the shutter button 101 can be pressed with the right index finger. In other words, the touch bar 119 can be operated when the user is looking through the eyepiece viewfinder 117 with their eyepiece in the eyepiece section 116 and is ready to press the shutter button 101 at any time (shooting posture). The touch bar 119 can accept tap operations (touching and releasing without moving within a predetermined period of time), left and right sliding operations (touching and then moving the touch position while keeping the touch on the touch bar), etc.

[0015] The camera 100 also includes a grip section 120, a thumb rest section 121, a terminal cover 122, a lid 123, a communication terminal 124, a display switching button 125, etc. The grip section 120 is a holding section shaped to be easily gripped by the user with their right hand when holding the camera 100. With the camera 100 held by gripping the grip section 120 with the little finger, ring finger, and middle finger of the right hand, the shutter button 101 and the main electronic dial 104 are positioned to be operated by the index finger of the right hand. Similarly, in the same position, the sub electronic dial 105 and the touch bar 119 are positioned to be operated by the thumb of the right hand. The thumb rest section 121 (thumb standby position) is a grip section located on the back of the camera 100, in a place where it is easy to rest the thumb of the right hand holding the grip section 120 when no other controls are being operated. The thumb rest section 121 is made of a rubber material or the like to enhance the holding force (grip feel). The terminal cover 122 protects connectors such as connection cables that connect the camera 100 to external devices. The cover 123 protects the recording medium 227 and the slot for storing the recording medium 227, which will be described later, by closing the slot. The communication terminal 124 is a terminal for the camera 100 to communicate with the lens unit 200, which will be described later and can be attached and detached. The display switching button 125 is used to switch the display on the display unit 108, etc.

[0016] Figure 2 is a diagram illustrating an example of the internal configuration of the camera 100 in Figure 1. Note that components identical to those in Figure 1 are denoted by the same reference numerals, and descriptions thereof are omitted as appropriate. A lens unit 200 is attached to the camera 100.

[0017] First, the lens unit 200 will be described. The lens unit 200 is a type of interchangeable lens that is detachable from the camera 100. The lens unit 200 is a single-lens unit, and is an example of a normal lens.

[0018] The lens unit 200 includes a diaphragm 201, a lens 202, a diaphragm drive circuit 203, an AF (autofocus) drive circuit 204, a lens system control circuit 205, a communication terminal 206, and the like.

[0019] The diaphragm 201 is configured such that its aperture diameter is adjustable. The lens 202 is composed of a plurality of lens elements. The diaphragm drive circuit 203 adjusts the amount of light by controlling the aperture diameter of the diaphragm 201. The AF drive circuit 204 drives the lens 202 to achieve focusing. The lens system control circuit 205 controls the diaphragm drive circuit 203, the AF drive circuit 204, and the like based on instructions from a system control unit 50, which will be described later. The lens system control circuit 205 controls the diaphragm 201 via the diaphragm drive circuit 203, and adjusts the position of the lens 202 via the AF drive circuit 204 to achieve focusing. The lens system control circuit 205 is capable of communicating with the camera 100. Specifically, communication is performed via the communication terminal 206 of the lens unit 200 and the communication terminal 124 of the camera 100. The communication terminal 206 is a terminal for the lens unit 200 to communicate with the camera 100 side.

[0020] Next, the camera 100 will be described. The camera 100 includes a shutter 210, an imaging unit 211, an A / D converter 212, a memory control unit 213, an image processing unit 214, a memory 215, a D / A converter 216, an EVF 217, a display unit 108, and a system control unit 50. Note that in Figure 2, the A / D converter is denoted as A / D, and the D / A converter is denoted as D / A.

[0021] The shutter 210 is a focal-plane shutter that can freely control the exposure time of the imaging unit 211 based on instructions from the system control unit 50. The imaging unit 211 is an image sensor composed of a CCD or CMOS element, etc., which converts an optical image into an electrical signal. The imaging unit 211 may have an image plane phase difference sensor that outputs defocus amount information to the system control unit 50. The A / D converter 212 converts the analog signal output from the imaging unit 211 into a digital signal. The image processing unit 214 performs predetermined processing (pixel interpolation, resizing such as reduction, color conversion, etc.) on the data from the A / D converter 212 or the data from the memory control unit 213. The image processing unit 214 also performs predetermined calculation processing using the captured image data, and the system control unit 50 performs exposure control and distance measurement control based on the obtained calculation results. This processing enables TTL (through-the-lens) AF processing, AE (automatic exposure) processing, EF (flash pre-flash) processing, etc. Furthermore, the image processing unit 214 performs predetermined calculations using the captured image data and performs TTL-type AWB (auto white balance) processing based on the obtained calculation results.

[0022] Image data from the A / D converter 212 is written to the memory 215 via the image processing unit 214 and the memory control unit 213. Alternatively, image data from the A / D converter 212 is written to the memory 215 via the memory control unit 213 without going through the image processing unit 214. The memory 215 stores image data obtained by the imaging unit 211 and converted into digital data by the A / D converter 212, as well as image data for display on the display unit 108 and EVF 217. The memory 215 has sufficient storage capacity to store a predetermined number of still images, a predetermined amount of video footage, and audio. The memory 215 also serves as a memory for image display (video memory).

[0023] The D / A converter 216 converts the image display data stored in the memory 215 into an analog signal and supplies it to the display unit 108 and EVF 217. Therefore, the display image data written to the memory 215 is displayed on the display unit 108 and EVF 217 via the D / A converter 216. The display unit 108 and EVF 217 perform display according to the analog signal from the D / A converter 216. The display unit 108 and EVF 217 are, for example, displays such as LCDs and OLEDs. The digital signal, which has been A / D converted by the A / D converter 212 and stored in the memory 215, is converted into an analog signal by the D / A converter 216 and sequentially transferred to the display unit 108 and EVF 217 for display, thereby performing live view display.

[0024] The system control unit 50 is a control unit consisting of at least one processor and / or at least one circuit. That is, the system control unit 50 may be a processor, a circuit, or a combination of a processor and a circuit. The system control unit 50 controls the entire camera 100. The system control unit 50 implements each process of the flowchart described later by executing a program recorded in the non-volatile memory 219. The system control unit 50 also performs display control by controlling the memory 215, D / A converter 216, display unit 108, EVF 217, etc.

[0025] The camera 100 also includes a system memory 218, a non-volatile memory 219, a system timer 220, a communication unit 221, a posture detection unit 222, and an eyepiece detection unit 118.

[0026] For example, RAM is used as the system memory 218. The system memory 218 stores constants and variables for the operation of the system control unit 50, programs read from the non-volatile memory 219, etc. The non-volatile memory 219 is an electrically erasable and recordable memory. For example, EEPROM is used as the non-volatile memory 219. The non-volatile memory 219 stores constants and programs for the operation of the system control unit 50, etc. The program here refers to a program for executing the flowchart described later. The system timer 220 is a timing unit that measures the time used for various controls and the time of the built-in clock. The communication unit 221 transmits and receives video signals and audio signals to and from external devices connected by wireless or wired cables. The communication unit 221 can also connect to wireless LAN (Local Area Network) and the internet. In addition, the communication unit 221 can communicate with external devices using Bluetooth® and Bluetooth Low Energy. The communication unit 221 can transmit images (including live images) captured by the imaging unit 211 and images recorded on the recording medium 227, and can receive image data and other various information from external devices. The attitude detection unit 222 detects the attitude of the camera 100 relative to the direction of gravity. Based on the attitude detected by the attitude detection unit 222, it is possible to determine whether the image captured by the imaging unit 211 was taken with the camera 100 held horizontally or vertically. The system control unit 50 can add orientation information corresponding to the attitude detected by the attitude detection unit 222 to the image file of the image captured by the imaging unit 211, or rotate the image before recording. For example, an acceleration sensor or a gyroscope sensor can be used as the attitude detection unit 222. It is also possible to detect the movement of the camera 100 (pan, tilt, lift, whether it is stationary or not, etc.) using the attitude detection unit 222.

[0027] The eyepiece detection unit 118 can detect the approach of any object to the eyepiece section 116 of the eyepiece viewfinder 117, which incorporates the EVF 217. For example, an infrared proximity sensor is used as the eyepiece detection unit 118. When an object approaches, infrared light emitted from the light emitter of the eyepiece detection unit 118 is reflected by the object and received by the light receiver of the infrared proximity sensor. The distance from the eyepiece section 116 to the object can be determined by the amount of infrared light received. In this way, the eyepiece detection unit 118 performs eyepiece detection to detect the proximity distance of an object to the eyepiece section 116. The eyepiece detection unit 118 is an eyepiece detection sensor that detects the approach (eye-catching) and departure (eye-moving) of an eye (object) to the eyepiece section 116 of the eyepiece viewfinder 117. When an object is detected approaching within a predetermined distance from the eyepiece section 116 from a non-eye-catching state (non-approaching state), it is detected that the eye has been caught in the eye. On the other hand, if the object that was detected as approaching moves beyond a predetermined distance from the eyepiece state (close-up state), it is detected that the eye has been removed. The threshold for detecting eyepiece contact and the threshold for detecting eye removal may be different, for example, by providing hysteresis. After eyepiece contact is detected, the eyepiece state is maintained until eye removal is detected. After eye removal is detected, the non-eyepiece state is maintained until eyepiece contact is detected again. The system control unit 50 switches the display (display state) / non-display (non-display state) of the display unit 108 and EVF 217 according to the state detected by the eyepiece detection unit 118. Specifically, at least in the shooting standby state and when the display destination switching setting is set to automatic switching, when the eyepiece is not in use, the display destination is set to the display unit 108 and the display is turned on, and the EVF 217 is hidden. When eyepiece contact is made, the display destination is set to the EVF 217 and the display is turned on, and the display unit 108 is hidden. Furthermore, the eyepiece detection unit 118 is not limited to an infrared proximity sensor; other sensors that can detect a state that can be considered as an eyepiece may be used.

[0028] Furthermore, the camera 100 includes an external viewfinder display unit 107, an external viewfinder display drive circuit 223, a power control unit 224, a power supply unit 225, a recording medium interface 226, an operation unit 228, etc. Note that in Figure 2, the recording medium interface is abbreviated as "I / F".

[0029] The external viewfinder display unit 107 displays various settings of the camera 100, such as shutter speed and aperture, via the external viewfinder display drive circuit 223. The power control unit 224 consists of a battery detection circuit, a DC-DC converter, a switch circuit for switching which blocks are powered, and detects whether a battery is installed, the type of battery, and the remaining battery level. The power control unit 224 also controls the DC-DC converter based on the detection results and instructions from the system control unit 50, supplying the necessary voltage to each part, including the recording medium 227, for the required period. The power supply unit 225 includes primary batteries such as alkaline batteries and lithium batteries, secondary batteries such as NiCd batteries, NiMH batteries and Li batteries, and an AC adapter. The recording medium I / F 226 is an interface with the recording medium 227, such as a memory card or hard disk. The recording medium 227 is a memory card or the like for recording captured images, and consists of semiconductor memory or a magnetic disk. The recording medium 227 may be detachable or built-in.

[0030] The operation unit 228 is an input unit that receives user input (user operation) and is used to input various instructions to the system control unit 50. The operation unit 228 includes a shutter button 101, a power switch 102, a mode selector switch 103, a touch panel 109, and other operating components 229. Other operating components 229 include a main electronic dial 104, a sub electronic dial 105, a video button 106, a directional key 110, a SET button 111, an AE lock button 112, a zoom button 113, a playback button 114, a menu button 115, a touch bar 119, and the like.

[0031] The shutter button 101 has a first shutter switch 230 and a second shutter switch 231. The first shutter switch 230 turns on during the operation of the shutter button 101, so-called half-press (shooting preparation instruction), and generates a first shutter switch signal SW1. The system control unit 50 starts shooting preparation processing such as AF processing, AE processing, AWB processing, and EF processing in response to the first shutter switch signal SW1. The second shutter switch 231 turns on when the operation of the shutter button 101 is completed, so-called full-press (shooting instruction), and generates a second shutter switch signal SW2. The system control unit 50 starts a series of shooting processes from reading the signal from the imaging unit 211 to generating an image file containing the captured image and writing it to the recording medium 227 in response to the second shutter switch signal SW2.

[0032] The mode switch 103 switches the operating mode of the system control unit 50 to one of the following: still image shooting mode, video shooting mode, playback mode, etc. Modes included in the still image shooting mode include auto shooting mode, auto scene detection mode, manual mode, aperture priority mode (Av mode), shutter speed priority mode (Tv mode), and program AE mode (P mode). There are also various scene modes and custom modes that provide shooting settings for different shooting scenes. The user can directly switch to any of the above shooting modes using the mode switch 103. Alternatively, the user can switch to the shooting mode list screen using the mode switch 103, and then selectively switch to one of the displayed modes using the operation unit 228. Similarly, the video shooting mode may also include multiple modes.

[0033] The touch panel 109 is a touch sensor that detects various touch operations on the display surface of the display unit 108 (the operating surface of the touch panel 109). The touch panel 109 and the display unit 108 can be configured as an integrated unit. For example, the touch panel 109 is mounted on the upper layer of the display surface of the display unit 108 so that its light transmittance does not interfere with the display of the display unit 108. By associating the input coordinates on the touch panel 109 with the display coordinates on the display surface of the display unit 108, a GUI (Graphical User Interface) can be configured that makes it appear as if the user can directly operate the screen displayed on the display unit 108. The touch panel 109 can use any of the following methods: resistive, capacitive, surface acoustic wave, infrared, electromagnetic induction, image recognition, or optical sensor. Depending on the method, a touch may be detected when there is contact with the touch panel 109, or when a finger or pen approaches the touch panel 109; either method is acceptable.

[0034] Figures 3 and 4 are diagrams illustrating the loop playback settings screen 300 displayed on the display unit 108 of Figure 1. The loop playback settings screen 300 is a screen for the user to configure settings related to loop playback, which is a function for checking videos generated during video recording, and consists of a playback section setting button 301 and a partial area setting button 302.

[0035] When the user selects the playback interval setting button 301 on the loop playback setting screen 300, the display unit 108 switches to the playback interval specification screen 303. The playback interval specification screen 303 is a user interface for setting the playback interval for loop playback. The user can specify the playback interval for loop playback by specifying the time before and after the frame image at the time the loop playback start operation was performed on the playback interval specification screen 303. Figure 3(b) shows the playback interval for loop playback when a loop playback start request is received during video playback. Based on the frame image 304 that was being played at the time of the loop playback start request, the first frame 305 and the last frame 306 in the interval corresponding to the number of seconds before and after specified on the playback interval specification screen 303 are determined, and the loop playback interval 307 is identified. In this way, in this embodiment, the loop playback interval is set on the playback interval specification screen 303.

[0036] On the other hand, when the user selects the partial area setting button 302 on the loop playback settings screen 300, the display unit 108 switches to the partial area selection screen 400 shown in Figure 4(a). The partial area selection screen 400 is a user interface for setting the area that is enlarged and played back during loop playback.

[0037] Figure 4(a) shows an example of the partial area selection screen 400 displayed on the display unit 108 of Figure 1. The partial area selection screen 400 allows you to select which area of ​​the frame images included in the video to enlarge and play during loop playback. The partial area selection screen 400 consists of a checkbox 401 for setting the surrounding area to check, a checkbox 402 for setting the four corners to check, and a checkbox 403 for setting the overlapping area to be omitted when switching between left and right.

[0038] For example, in the partial area selection screen 400, if only checkbox 401 is checked, as shown in Figure 4(a), the partial area to be played back during loop playback (hereinafter referred to as the "playback area during loop playback") will be the area shown in gray in Figure 4(b). The circles shown in Figure 4(b) represent the left and right image circles of the circular fisheye image obtained by the camera 100. In other words, in this embodiment, if only checkbox 401 is checked, the playback area during loop playback is the area surrounding the center of the above circle (right 5, left 5 in Figure 4(b)). In this embodiment, since the VR image is recorded with the left and right sides optically swapped, the image of the left eye is recorded on the right side and the image of the right eye is recorded on the left side.

[0039] Furthermore, in the partial area selection screen 400, as shown in Figure 4(c), if checkboxes 401 and 403 are checked, the playback area during loop playback will be the gray area shown in Figure 4(d). In other words, in this embodiment, if checkboxes 401 and 403 are checked, the overlapping areas on the left and right of the gray area shown in Figure 4(b) are excluded from the playback area during loop playback. In this embodiment, overlapping areas on the left and right are determined by whether or not they correspond to the gray area shown in Figure 4(e) that does not overlap on the left and right, and based on the result of the determination, the left 1, left 2, left 4, left 7, and left 8 areas are excluded from the playback area during loop playback.

[0040] Furthermore, in the partial area selection screen 400, as shown in Figure 4(f), if only checkbox 403 is checked, the user can select the playback area during loop playback. In this case, to allow the user to select any partial area, after loop playback starts, grid lines are superimposed on the circular fisheye image, which includes the fisheye image of the left eye and the fisheye image of the right eye, as shown in Figure 4(g), and the playback section is played. The user selects the partial area in this display state.

[0041] Figure 5 shows how partial regions are sequentially displayed within the playback interval determined in this embodiment. Figure 5(a) shows a state in which the right eye partial region 501 and the left eye partial region 502 are selected for a circular fisheye image 500 that includes the left eye image and the right eye image. Figure 5(b) shows how playback is performed sequentially with perspective projection transformation while switching between partial regions for each playback interval 503. Specifically, the video within the range of playback interval 503, with the left eye partial region 502 enlarged, is subjected to perspective projection transformation and played back. Subsequently, the video within the range of playback interval 503, with the right eye partial region 501 enlarged, is subjected to perspective projection transformation and played back.

[0042] Figure 6 is a flowchart showing the procedure for VR video playback processing performed by the camera 100 in Figure 1. VR video playback processing is achieved by the system control unit 50 executing a program recorded in the non-volatile memory 219. In this embodiment, the camera 100 holds the left eye video and the right eye video used for VR video playback. The left eye video and the right eye video are videos composed of wide-angle images such as fisheye images, and are videos obtained by the camera 100, or videos obtained by another camera and acquired by the camera 100 from that other camera.

[0043] In Figure 6, first, at S600, the system control unit 50 determines whether or not it has received a loop playback start request from the user. In this embodiment, the user can make a loop playback start request by operating the touch panel 109 or other operating members 229.

[0044] If it is determined in S600 that the user has not requested the start of loop playback, the process proceeds to S615. In S615, the system control unit 50 decodes the corresponding frame image. This is done by the circuit of the image processing unit 214 controlled by the system control unit 50. The image processing unit 214 decodes the image encoded in a format such as JPEG and places the image in a predetermined format such as YUV into the memory 215.

[0045] Next, in S616, the system control unit 50 resizes the decoded frame image to fit the display device and outputs it to the display unit 108, EVF 217, etc. Then, in S617, the system control unit 50 determines whether the frame image displayed in S616 is the final frame image.

[0046] If S617 determines that the frame image displayed in S616 is the final frame image, this process terminates. If S617 determines that the frame image displayed in S616 is not the final frame image, this process proceeds to S618. In S618, the system control unit 50 sets the next frame image. After that, this process returns to S600.

[0047] If it is determined in S600 that a loop playback start request has been received from the user, the process proceeds to S601. In S601, the system control unit 50 identifies (determines) the playback section for loop playback (playback section determination means). Specifically, the system control unit 50 identifies the playback section based on the frame image at the time the loop playback start request was received, and based on the forward playback seconds and backward playback seconds set in the playback section specification screen 303 (see, for example, Figure 3(b)).

[0048] Next, in S602, the system control unit 50 performs the partial region determination process shown in Figure 7, which will be described later, to identify candidates for the regeneration region.

[0049] Next, in S603, the system control unit 50 performs the playback region determination process shown in Figure 8, which will be described later, to determine the playback region from the candidates identified in S602. In this embodiment, although the details will be described later, the unplayed subregions among the subregions selected for the left eye image are determined to be the playback region. Furthermore, once all subregions selected for the left eye image have been played back, the unplayed subregions among the subregions selected for the right eye image are determined to be the playback region.

[0050] Next, in S604, the system control unit 50 determines whether all playback areas have been played back. If it is determined in S604 that all playback areas have been played back, the process proceeds to S615. As a result, loop playback ends, and video playback resumes from the frame image at the end of loop playback.

[0051] If it is determined in S604 that no playback area has been played, the process proceeds to S605. In S605, the system control unit 50 sets the image of the first frame of the playback section identified in S601.

[0052] Next, in S606, the system control unit 50 decodes the frame image set in S605. Then, in S607, the system control unit 50 determines whether or not the user has performed an operation to change the playback area. In this embodiment, the user can change the playback area by operating, for example, the touch panel 109 or the directional keys 110.

[0053] If it is determined in S607 that the user has not performed an operation to change the playback area, this process proceeds to S609, which will be described later. On the other hand, if it is determined in S607 that the user has performed an operation to change the playback area, this process proceeds to S608. In S608, the system control unit 50 changes the playback area based on the amount of movement of the display position determined by the user's operation. The newly identified playback area may replace one of the multiple sub-areas selected in S602, or it may be added.

[0054] Next, in S609, the system control unit 50 performs a perspective projection transformation on the frame image decoded in S606, using the viewpoint of the playback region determined in S603 or S608. For example, the system control unit 50 performs a perspective projection transformation on an image obtained by cutting out the playback region determined in S603 or S608 from the frame image decoded in S606. This generates a VR image of one eye, such as the left or right eye, with a sense of depth, representing the playback region determined in S603 or S608.

[0055] Next, in S610, the system control unit 50 displays the VR image obtained in the processing of S609 on the display unit 108 or EVF 217. Then, in S611, the system control unit 50 determines whether or not it has received a request from the user to end loop playback. In this embodiment, the user can request to end loop playback by pressing a predetermined key for ending the loop playback function, such as the menu button 115, or by touching a predetermined GUI button displayed on the touch panel 109.

[0056] If it is determined in S611 that a request to end loop playback has been received from the user, this process proceeds to S612. In S612, the system control unit 50 prepares for the end of loop playback. Specifically, the system control unit 50 sets the VR image displayed in S610 (hereinafter referred to as the "playback frame image") as the image of the last frame of the playback section. Next, this process proceeds to S613.

[0057] If it is determined in S611 that the user has not requested the end of loop playback, the process proceeds to S613. In S613, the system control unit 50 determines whether the playback frame image is the final frame image in the playback section identified in S601.

[0058] If it is determined in S613 that the playback frame image is not the last frame image in the playback section, the process proceeds to S614. In S614, the system control unit 50 sets the next frame image. After that, the process returns to S606. In this way, in this embodiment, a VR video is played that is an enlarged version of the playback area determined in S603, which is the VR video within the range indicated by the playback section set on the playback section specification screen 303.

[0059] If, in S613, it is determined that the playback frame image is the final frame image in the playback section, the process returns to S603, and the display control of the VR image of the next determined playback area is performed. In this way, in this embodiment, multiple VR videos with different areas magnified are displayed in rotation.

[0060] Figure 7 is a flowchart showing the procedure for the partial region selection process in S602 of Figure 6.

[0061] In Figure 7, at S700, the system control unit 50 determines whether a partial area is set or not. For example, if checkbox 401 or checkbox 402 is selected on the partial area selection screen 400, it is determined that a partial area is set. In this case, this process ends. On the other hand, if neither checkbox 401 nor checkbox 402 is selected on the partial area selection screen 400, it is determined that a partial area is not set. In this case, this process proceeds to S701.

[0062] In S701, the system control unit 50 sets the image of the first frame of the playback section identified in S601. Then, in S702, the system control unit 50 decodes the image of the first frame set in S701.

[0063] Next, in S703, the system control unit 50 overlays grid lines representing the divided sub-regions onto the decoded frame image. For example, as shown in Figure 4(g), grid lines representing the divided sub-regions are overlaid onto the decoded circular fisheye image.

[0064] Next, in S704, the system control unit 50 determines whether the user has performed an operation to select one of the sub-regions displayed divided by grid lines. In this embodiment, the user can confirm their selection, for example, by touching one or more of the sub-regions displayed on the touch panel 109. Alternatively, the user can confirm their selection by moving the selection frame to the position of the sub-region using the directional keys and pressing the SET button 111.

[0065] If it is determined in S704 that the user has not performed an operation to select one of the sub-regions, this process proceeds to S706, which will be described later. If it is determined in S704 that the user has performed an operation to select one of the sub-regions, this process proceeds to S705.

[0066] In S705, the system control unit 50 stores information about the partial region selected by the user in the system memory 218 or the like. Next, in S706, the system control unit 50 determines whether the frame image displayed in S703 is the image of the last frame of the playback section.

[0067] If it is determined in S706 that the above frame image is not the image of the last frame in the playback section, the process proceeds to S707. In S707, the system control unit 50 sets the next frame image. After that, the process returns to S702.

[0068] If it is determined in S706 that the above frame image is the image of the last frame of the playback section, the process proceeds to S708. In S708, the system control unit 50 determines whether or not it has received a request from the user to terminate partial region identification. In this embodiment, the user can terminate partial region identification by pressing a predetermined key associated with the termination of processing, such as the menu button 115.

[0069] If it is determined in S708 that the user has not requested the termination of the partial area identification operation, this process returns to S701. If it is determined in S708 that the user has requested the termination of the partial area identification operation, this process terminates.

[0070] Figure 8 is a flowchart showing the procedure for determining the playback region in S603 of Figure 6.

[0071] In Figure 8, in S800, the system control unit 50 determines whether or not there are any unreproduced subregions among the selected subregions of the left eye image.

[0072] If it is determined in S800 that there is an unreproduced region in the left eye image, the process proceeds to S806. In S806, the system control unit 50 identifies one region from the unreproduced regions in the left eye image and determines it to be the next regenerated region. After that, the process ends.

[0073] If it is determined in S800 that there are no unresolved regions in the left eye image, the process proceeds to S801. In S801, the system control unit 50 determines whether or not there are any unresolved regions among the selected regions in the right eye image.

[0074] If it is determined in S801 that there are no unresolved areas in the right eye image, the process proceeds to S805. In S805, the system control unit 50 determines that there are no unresolved areas in both the left eye image and the right eye image. After that, the process ends.

[0075] If it is determined in S801 that there is an unreproduced portion of the image of the right eye, the process proceeds to S802. In S802, the system control unit 50 determines whether or not to omit the overlapping region when switching between left and right images, based on whether or not the checkbox 403 is set.

[0076] If it is determined in S802 that the overlapping area should not be omitted when switching between left and right, this process proceeds to S806 as described above.

[0077] If it is determined in S802 that the overlapping area should be omitted when switching between left and right, the process proceeds to S803. In S803, the system control unit 50 determines whether the unreproduced portion of the right eye image is an overlapping portion of the left eye image. In this embodiment, overlapping areas between the left and right eyes are determined, as described above, by whether or not they correspond to the gray area shown in Figure 4(e) that does not overlap between the left and right eyes.

[0078] If, in S803, it is determined that the unplayed portion of the right eye image overlaps with the played portion of the left eye image, the process proceeds to S804. In S804, the system control unit 50 changes the unplayed portion that was determined to overlap with the played portion of the left eye image in S803 to a played state. As a result, this unplayed portion is excluded from the playback area during loop playback. After that, the process returns to S801.

[0079] If, in S803, it is determined that the unreconstructed region in the right eye image does not overlap with the reconstructed region in the left eye, the process proceeds to S806 as described above.

[0080] According to the embodiment described above, the display control is performed by sequentially switching for each of the multiple sub-regions, displaying a first VR video which is an enlarged version of a first sub-region within the range indicated by the playback section, and then displaying a second VR video which is an enlarged version of a second sub-region within the range indicated by the playback section. In other words, multiple VR videos with different enlarged regions are displayed in sequence. This allows for efficient viewing of VR videos.

[0081] Furthermore, in the embodiment described above, the region selected by the user from among multiple sub-regions obtained by dividing the frame image in the VR video is determined as the playback region. This allows the user to efficiently view a VR video with an enlarged view of the region they wish to examine.

[0082] Furthermore, in the embodiment described above, the playback area is determined based on the user's settings on the partial area selection screen 400. This allows the user to efficiently view VR videos without having to select each playback area individually.

[0083] Furthermore, in the above-described embodiment, both the first VR video and the second VR video are videos composed of images from one eye, and the first partial region is a different region from the second partial region. This makes it possible to efficiently view multiple different regions in a video composed of images from one eye.

[0084] Furthermore, in the embodiment described above, the first VR video is a video composed of images from one eye, and the second VR video is a video composed of images from the other eye. This allows for efficient viewing of two videos composed of images from different eyes.

[0085] Furthermore, in the embodiment described above, if it is determined that the second partial region overlaps with the first partial region, the display of the second VR video is omitted after the first VR video is displayed. This prevents the VR video of an enlarged version of the already confirmed partial region from being displayed multiple times, thereby allowing for efficient viewing of the VR video.

[0086] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the information processing device.

[0087] For example, in the embodiment described above, a configuration was described in which a perspective projection transformation is performed on an image obtained by cutting out a playback region determined in S603 or S608 from the frame image decoded in S606 in S609. However, the configuration is not limited to this. For example, a configuration in which a perspective projection transformation is performed on the entire frame image decoded in S606 may also be used. In such a configuration, in S610, the VR image obtained by this perspective projection transformation is enlarged and displayed in the playback region determined in S603 or S608. Even in such a configuration, the same effects as in the embodiment described above can be achieved.

[0088] Furthermore, in a configuration where perspective projection transformation is performed on the entire frame image decoded in S606, it is also possible to have a configuration that not only displays the VR image obtained by this perspective projection transformation, but also outputs the VR video composed of this VR image to another device.

[0089] Furthermore, although the above-described embodiment described a configuration in which the VR image obtained by perspective projection conversion is displayed in S610, the configuration is not limited to this. For example, the fisheye image obtained by shooting with camera 100 may be enlarged and displayed in the playback area determined in S603 or S608.

[0090] Furthermore, the embodiments described above represent only one embodiment of the present invention, and it is possible to combine these embodiments as appropriate.

[0091] The present invention may also be a display method comprising each of the processing steps performed in the information processing apparatus described above.

[0092] Furthermore, the present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. The present invention can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0093] Furthermore, the disclosure of this embodiment includes the following configurations and methods. (Configuration 1) An information processing apparatus comprising: a playback section determination means for determining a playback section of a VR video; a sub-region determination means for determining a plurality of sub-regions from a single frame image in the VR video; and a control means for controlling the display of the VR video on a display means, wherein the control means sequentially switches the display control for each of the plurality of sub-regions, after displaying a first VR video which is a first VR video of the range indicated by the playback section of the VR video and is an enlargement of the first sub-region of the plurality of sub-regions, and then displaying a second VR video which is a second VR video of the range indicated by the playback section of the VR video and is an enlargement of the second sub-region of the plurality of sub-regions. (Configuration 2) The information processing apparatus according to Configuration 1, characterized in that the partial region determination means determines as the partial region a region selected by the user from among a plurality of regions obtained by dividing a single frame image in the VR video. (Configuration 3) The information processing apparatus according to Configuration 1 or 2, characterized in that the partial region determination means determines a plurality of partial regions from a single frame image in the VR video based on settings made by the user. (Configuration 4) The information processing device according to any one of Configurations 1 to 3, characterized in that the VR video is composed of an image of the left eye and an image of the right eye, the first VR video is a video composed of the image of one eye, the second VR video is a video composed of the image of one eye, and the first partial region is a region different from the second partial region. (Configuration 5) The information processing device according to any one of Configurations 1 to 4, characterized in that the VR video is composed of an image of the left eye and an image of the right eye, the first VR video is a video composed of the image of one eye, and the second VR video is a video composed of the image of the other eye. (Configuration 6) The information processing apparatus according to Configuration 5, characterized in that, when it is determined that the second partial region is an overlapping region with the first partial region, the control means displays the first VR video and then omits the display of the second VR video. (Configuration 7) An information processing apparatus according to any one of Configurations 1 to 6, further comprising means for photographing a subject and generating an image, and means for performing a perspective projection transformation on the image and reproducing it. (Configuration 8) A control method for an information processing device, comprising: a playback section determination step for determining a playback section of a VR video; a sub-region determination step for determining a plurality of sub-regions from a single frame image in a VR video; and a control step for controlling the display means to display the VR video, wherein the control step sequentially switches the display control for each of the plurality of sub-regions, after displaying a first VR video which is a first VR video of the range indicated by the playback section of the VR video and is an enlargement of the first sub-region of the plurality of sub-regions, and then displaying a second VR video which is a second VR video of the range indicated by the playback section of the VR video and is an enlargement of the second sub-region of the plurality of sub-regions. (Configuration 9) A program that causes an information processing device to execute a playback section determination step of determining a playback section of a VR video, a sub-region determination step of determining a plurality of sub-regions from a single frame image in the VR video, and a control step of controlling a display means to display the VR video, wherein the control step is characterized by sequentially switching the display control for each of the plurality of sub-regions, after displaying a first VR video which is a first VR video of the range indicated by the playback section of the VR video and is an enlargement of the first sub-region of the plurality of sub-regions, and then displaying a second VR video which is a second VR video of the range indicated by the playback section of the VR video and is an enlargement of the second sub-region of the plurality of sub-regions. [Explanation of symbols]

[0094] 50 System Control Unit 100 Cameras 303 Playback section selection screen 400 Partial Area Selection Screen

Claims

1. A means for determining the playback section of a VR video, A subregion determination means for determining multiple subregions from a single frame image in the aforementioned VR video, The display means includes a control means for controlling the display of the VR video, The control means is characterized by sequentially switching the display control for each of the plurality of partial regions, after displaying a first VR video which is a first VR video in the range indicated by the playback section of the VR video and is an enlarged version of the first partial region of the plurality of partial regions, and then displaying a second VR video which is a second VR video in the range indicated by the playback section of the VR video and is an enlarged version of the second partial region of the plurality of partial regions.

2. The information processing apparatus according to claim 1, characterized in that the partial region determination means determines a region selected by the user as the partial region from among a plurality of regions obtained by dividing a single frame image in the VR video.

3. The information processing apparatus according to claim 1, wherein the partial region determination means determines a plurality of partial regions from a single frame image in the VR video based on settings made by the user.

4. The aforementioned VR video consists of an image for the left eye and an image for the right eye. The first VR video is a video composed of images from one eye. The second VR video is a video composed of images from one of the eyes. The information processing apparatus according to claim 1, characterized in that the first subregion is a region different from the second subregion.

5. The aforementioned VR video consists of an image for the left eye and an image for the right eye. The first VR video is a video composed of images from one eye. The information processing apparatus according to claim 1, characterized in that the second VR video is a video composed of images from the other eye.

6. The information processing apparatus according to claim 5, characterized in that, if it is determined that the second sub-region is an overlapping region with the first sub-region, the control means displays the first VR video and then omits the display of the second VR video.

7. The information processing apparatus according to claim 1, further comprising means for photographing a subject and generating an image, and means for performing a perspective projection transformation on the image and reproducing it.

8. The playback section determination process determines the playback section of the VR video, A subregion determination step in which multiple subregions are determined from a single frame image in the VR video, The system includes a control step for controlling the display means to display the VR video, Control method for an information processing device, characterized in that the control step involves sequentially switching the display control for each of the plurality of partial regions, after displaying a first VR video which is a first VR video in the range indicated by the playback section of the VR video and is an enlarged version of the first partial region of the plurality of partial regions, and then displaying a second VR video which is a second VR video in the range indicated by the playback section of the VR video and is an enlarged version of the second partial region of the plurality of partial regions.

9. In an information processing device, The playback section determination process determines the playback section of the VR video, A subregion determination step in which multiple subregions are determined from a single frame image in the VR video, The system performs a control step that controls the display means to display the VR video, The control step is a program characterized by sequentially switching the display control for each of the plurality of partial regions, which involves displaying a first VR video in the range indicated by the playback section of the VR video, which is a first VR video obtained by enlarging the first partial region of the plurality of partial regions, and then displaying a second VR video in the range indicated by the playback section of the VR video, which is a second VR video obtained by enlarging the second partial region of the plurality of partial regions.

Images