Program, display control device, image display system, and control method

Abstract

To provide a program, a display control device, an image display system, and a control method which display a video with high senses of reality and presence on a head-mounted display when video data on live action are reproduced.SOLUTION: A storage control section makes a storage device store imaging data including video data obtained by imaging a real space and imaging direction data corresponding to the video data and indicating an imaging direction of the video data in the real space. A projection section projects a video according to the video data in a virtual space from a virtual viewpoint on the basis of the imaging data. The projection section controls a projection direction of the video according to the video data in the virtual space on the basis of the imaging direction data corresponding to the video data.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] The present invention relates to a program, a display control device, an image display system, and a control method. [Background technology]

[0002] In recent years, head-mounted displays (HMDs) have come into widespread use. Live-action video data of a real space, which is a space in the real world, is stored on a recording medium, and the video data is sometimes played back and displayed on an HMD. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-115122 Summary of the Invention [Problem to be solved by the invention]

[0004] When live-action video data is played back, the live-action video displayed on the HMD is always displayed in the same direction. Here, "same direction" means the same direction with respect to the real space viewed by the user wearing the HMD (such as always facing forward), or the same direction with respect to the HMD screen (such as always the reference direction of the HMD). For example, if the direction in which the user wearing the HMD is facing forward is set as the reference viewing direction of the HMD, the center of the live-action video is always displayed fixed in the reference viewing direction. For this reason, when live-action video data is displayed on an HMD, the image may lack a sense of reality or presence. For example, consider video data taken in a scene where a subject such as a car is driving past a camera in real space, with the camera angle changed to match the subject's movement so that the subject is captured at the center of the image. When this video data is played back and displayed on an HMD, even though the real subject was moving at the time of shooting, the subject is displayed in the same direction as the HMD's field of view (the aforementioned standard field of view), resulting in an image that lacks realism and presence.

[0005] Therefore, one of the objects of the present invention is to display on an HMD images that have a higher sense of reality and presence than conventional images when live-action video data is played back. [Means for solving the problem]

[0006] A program according to one aspect of the present invention causes a computer that executes control to display an image of the field of view of a virtual space viewed in the field of view direction from a virtual viewpoint within the virtual space on a display unit of a head-mounted display to function as: a memory control unit that stores, in a memory device, shooting data including video data of a real space photographed and shooting direction data corresponding to the video data that indicates the shooting direction of the video data in the real space; and a projection unit that projects an image corresponding to the video data from the virtual viewpoint into the virtual space based on the shooting data, and the projection unit controls the projection direction of the image corresponding to the video data within the virtual space based on the shooting direction data corresponding to the video data.

[0007] A display control device according to one aspect of the present invention is a display control device that performs control to display an image of the field of view of a virtual space as viewed in the field of view from a virtual viewpoint within the virtual space on a display unit of a head-mounted display, and includes: a memory control unit that stores shooting data in a memory device, the shooting data including video data of a real space and shooting direction data corresponding to the video data and indicating the shooting direction of the video data in the real space; and a projection unit that projects an image corresponding to the video data from the virtual viewpoint into the virtual space based on the shooting data, and the projection unit controls the projection direction of the image corresponding to the video data within the virtual space based on the shooting direction data corresponding to the video data.

[0008] An image display system according to one aspect of the present invention is an image display system comprising a head-mounted display including a display unit, and a display control device that executes control to cause the display unit to display an image of the field of view of the virtual space as viewed in the field of view from a virtual viewpoint within the virtual space, wherein the display control device includes a memory control unit that stores shooting data in a memory device, the shooting data including video data of a real space and shooting direction data corresponding to the video data that indicates the shooting direction of the video data in the real space, and a projection unit that projects an image corresponding to the video data from the virtual viewpoint into the virtual space based on the shooting data, and the projection unit controls the projection direction of the image corresponding to the video data within the virtual space based on the shooting direction data corresponding to the video data.

[0009] A control method according to one aspect of the present invention is a control method for controlling a computer that executes control to display an image of the field of view of a virtual space viewed in the field of view direction from a virtual viewpoint within the virtual space on a display unit of a head-mounted display, and includes a storage control step of storing, in a storage device, shooting data including video data of a real space photographed and shooting direction data corresponding to the video data that indicates the shooting direction of the video data in the real space, and a projection step of projecting an image corresponding to the video data from the virtual viewpoint into the virtual space based on the shooting data, wherein the projection step controls the projection direction of the image corresponding to the video data within the virtual space based on the shooting direction data corresponding to the video data. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a schematic block diagram showing an example of a hardware configuration of an image display system according to an embodiment of the present invention. [Figure 2] FIG. 10 is a diagram illustrating definitions of directions in a virtual space. [Figure 3] 10 is a diagram for explaining a field of view image of a field of view range viewed in a field of view direction from a virtual viewpoint in a virtual space. FIG. [Figure 4] 1 is a conceptual diagram showing an example in which video data captured in a real space R is played back and projected onto a virtual space V. FIG. [Figure 5] FIG. 10 is a diagram conceptually illustrating an example in which the projection direction TD of the captured image in the virtual space V changes in accordance with a change in the capture direction XC during capture. [Figure 6] 1 is a diagram conceptually showing an example of the relationship between a real space R when video data is captured and a virtual space V when the video data is played back. [Figure 7] FIG. 10 is a diagram showing an example of time-series changes in the projection direction TD in a comparative example in which imaging direction data is not used. [Figure 8] 10A and 10B are diagrams showing an example of time-series changes in the projection direction TD in this embodiment using imaging direction data. [Figure 9]FIG. 2 is a schematic functional block diagram illustrating an example of a functional configuration of a display control device. [Figure 10] FIG. 10 is a diagram illustrating an example of imaging direction data. [Figure 11] FIG. 10 is a diagram showing another example of the imaging direction data. [Figure 12] 10 is a flowchart illustrating an example of processing by a display control device. [Figure 13] FIG. 2 is a diagram showing an example of a screen displayed on a display unit of an HMD. [Figure 14] FIG. 2 is a diagram showing an example of a screen displayed on a display unit of an HMD. [Figure 15] FIG. 2 is a schematic functional block diagram illustrating an example of a functional configuration of a display control device. [Figure 16] FIG. 10 is a diagram showing an example of a screen displayed on the display unit of the HMD when an image projected into a virtual space is not displayed within the field of view. [Figure 17] FIG. 17 is a diagram showing the positional relationship in a virtual space corresponding to the screen shown in FIG. 16. [Figure 18] FIG. 10 is a diagram showing an example of a screen displayed on the display unit of the HMD when an image projected into a virtual space is not displayed within the field of view. [Figure 19] 10 is a flowchart illustrating an example of processing by a display control device. [Figure 20] 10A and 10B are diagrams illustrating an example of an image projected into a virtual space when a roll direction component corresponding to the shooting direction is reflected in the projection direction. [Figure 21] FIG. 21 is a diagram showing an example of an image projected into a virtual space when the same image data as in FIG. 20 is reproduced and the roll direction component corresponding to the shooting direction is not reflected in the projection direction. [Figure 22] 10 is a flowchart illustrating an example of processing by a display control device. [Figure 23] FIG. 2 is a diagram showing an example of a screen displayed on a display unit of an HMD. [Figure 24] FIG. 2 is a diagram showing an example of a screen displayed on a display unit of an HMD. [Figure 25]FIG. 1 is a diagram illustrating an example of an HMD. [Figure 26] 1 is a schematic block diagram showing an example of the hardware configuration of a stand-alone HMD or an information processing device used as an HMD. DETAILED DESCRIPTION OF THE INVENTION

[0011] The program, display control device, image display system, and control method for executing display control of an HMD according to the present embodiment are suitable for playing back live video data captured in real space. Hereinafter, an example of an embodiment of the present invention will be described with reference to the drawings.

[0012] [1. Example of image display system configuration] 1 is a schematic block diagram showing an example of the hardware configuration of an image display system 1 according to an embodiment of the present invention. The image display system 1 includes an HMD 10 and a display control device 20 that controls display on the HMD 10.

[0013] The HMD 10 is worn on the user's head (see FIG. 2) and can display a field of view image of the field of view range as seen in the field of view direction from a virtual viewpoint in a virtual space. The HMD 10 is equipped with a sensor 12, such as a gyro sensor, that detects the movement and tilt of the HMD 10, and detects changes in the movement and tilt of the head of the user wearing the HMD 10, and changes the field of view direction and field of view in response to those changes. For example, the field of view image displayed on the HMD 10 changes to a field of view image of the field of view range to the right in the virtual space when the user's head is turned to the right, and changes to a field of view image of the field of view range to the upward direction in the virtual space when the user's head is turned up, giving the user a sense of immersion as if they were actually there.

[0014] The HMD 10 mainly includes a display unit 11, a sensor 12, a processor 13, a storage device 14, and the like. The display unit 11 is a display that displays a field of view image that represents a field of view range from a virtual viewpoint in a virtual space. The virtual space is also called a virtual reality (VR) space. The display unit 11 displays a stereoscopic image (an image for the right eye and an image for the left eye) using binocular parallax, or a non-stereoscopic image (a 2D image) as the field of view image. The display unit 11 may be a virtual image projection display that forms a virtual image by using, for example, a half mirror. The display unit 11 may also be a retinal projection display that forms a field of view image directly on the retina by using, for example, the crystalline lens of the user's eye.

[0015] The sensor 12 detects posture information such as the rotation angle and tilt of the HMD 10. Detection information regarding the orientation of the HMD 10 can be obtained from the output of this sensor 12. For example, the sensor 12 is an angular velocity sensor (gyro sensor) that detects the angular velocity of an object. The sensor 12 may be a sensor that detects a change in direction, or a sensor that detects the direction itself. For example, the sensor 12 is not limited to a gyro sensor, but may be an acceleration sensor, an angular acceleration sensor, a tilt sensor, a geomagnetic sensor, or the like, and may be realized by appropriately combining these.

[0016] The processor 13 functions as a control center that controls each unit included in the HMD 10. For example, the processor 13 is a CPU (Central Processing Unit). The processor 13 may be configured to include hardware such as a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), or an FPGA (Field Programmable Gate Array) in addition to or instead of the CPU. The processor 13, for example, controls the supply of an image signal received from the display control device 20 to the display unit 11 via an interface. The storage device 14 stores programs executed by the processor 13 and temporarily stores data and parameters processed by the processor 13. The storage device 14 includes, for example, a read-only memory (ROM), a random access memory (RAM), a video random access memory (VRAM), an auxiliary storage device, etc. The auxiliary storage device may be a non-volatile semiconductor memory, a hard disk drive, a solid state drive, etc.

[0017] The HMD 10 may also include a network adapter, etc., so that VR images can be input via a network. The HMD 10 may also include an audio output unit, a GPS (Global Positioning System) receiving unit, etc.

[0018] The display control device 20 is connected to the HMD 10 wirelessly or via a wired connection so as to be able to communicate with the HMD 10, and executes display control for the HMD 10. The display control device 20 is, for example, a stationary or portable game machine. The display control device 20 may also be an arcade (commercial) game machine installed in an amusement facility or the like. Alternatively, the display control device 20 may also be a personal computer, a tablet computer, a smartphone, a mobile phone terminal, a PHS (Personal Handy-phone System) terminal, a personal digital assistant (PDA), a multi-function television receiver with information processing capabilities (a so-called smart TV), or the like.

[0019] The display control device 20 mainly comprises a processor 21, a storage device 22, an operation unit 23, and a communication unit 24, which are interconnected via bus lines including an address bus, a data bus, a control bus, etc. Note that an interface circuit, an image processing unit, a sound processing unit, etc. are interposed between the bus lines and each component as needed, but are not shown here.

[0020] The processor 21 interprets and executes program instructions and controls the entire display control device 20. For example, the processor 21 is a CPU. The processor 21 may be configured to include hardware such as a GPU, DSP, or FPGA in addition to or instead of the CPU. The storage device 22 includes, for example, a ROM, RAM, VRAM, and auxiliary storage device. The ROM stores programs, data, and the like required for basic operational control of the display control device 20. The RAM or VRAM stores various programs and data and ensures a working area for the processor 21. The auxiliary storage device stores programs, various data, and the like, and may be, for example, a non-volatile semiconductor memory, a hard disk drive, a solid state drive, or the like.

[0021] The display control device 20 may also include a recording medium drive. Examples of recording medium drives include DVD-ROM drives, CD-ROM drives, hard disk drives, optical disk drives, flexible disk drives, silicon disk drives, and cassette medium readers. In this case, examples of recording media include DVD-ROMs, CD-ROMs, hard disk drives, optical disks, flexible disks, and semiconductor memories. The recording medium drive reads image data, audio data, and program data from the recording medium and supplies the read data to RAM or the like of the storage device 22 via a decoder.

[0022] For example, shooting data including video data previously captured (photographed) of real space, and programs such as games that include the shooting data, are stored in a storage device 22 (auxiliary storage device, etc.), or are read from a recording medium drive and imported into the display control device 20.

[0023] The operation unit 23 is used by the user to input various operation commands to the display control device 20. For example, the user performs operations to view played video or to operate a game that includes video. Examples of the operation unit 23 include a position input unit with a touch interface (such as a component of a touch panel), physical buttons, a controller, an analog stick, a keyboard, a pointing device, and the like. The operation unit 23 may also be configured as an operation unit that can accept voice input by identifying voice input from a voice input unit such as a microphone. The operation unit 23 may also be configured to perform operations through user gestures.

[0024] The communication unit 24 includes a communication interface (not shown) and has a communication control function for communicating data when playing back video, playing games, etc. Here, the communication control function for data communication includes, for example, an Internet connection function, a wireless LAN (Local Area Network) connection function, and a short-range wireless communication function using a predetermined frequency band (for example, a 2.4 GHz frequency band). The communication unit 24 transmits a connection signal for connecting the display control device 20 to a network based on a command from the processor 21, and also receives information transmitted from a communication partner and supplies the information to the processor 21. For example, it is possible to receive live video from a remote location distributed via a network through the communication unit 24 and display the live video on the HMD 10 in approximately real time.

[0025] The display control device 20 may also be connected to a tracking unit that detects the position of the HMD 10 (or the position of the head of the user wearing the HMD 10). For example, the HMD 10 includes multiple light sources (LEDs, etc.) for tracking, and the tracking unit includes an imaging unit for imaging the HMD 10. The tracking unit, which is fixed at a predetermined position, identifies the position of the HMD 10 based on the positions of the multiple light source units that appear in an image captured by the imaging unit. The display control device 20 may also include a display unit such as a liquid crystal display or an organic EL (Electro-Luminescence) display.

[0026] [2. Direction in virtual space (VR space)] 2 is a diagram showing the definition of directions in a virtual space (VR space) according to this embodiment. In this embodiment, the Z axis is the vertical direction in which a user standing upright wears an HMD 10, which displays a field of view image of a field of view seen in the direction of the field of view from a virtual viewpoint in the virtual space (see FIG. 1). The X axis is the horizontal axis perpendicular to the Z axis and is the line of sight of a user facing forward (the reference direction for the virtual space), and the Y axis is the horizontal axis perpendicular to the Z axis and the X axis.

[0027] For example, when the image display system 1 is started, the visual axis direction of the HMD 10 (the direction perpendicular to the display surface of the display unit 11 of the HMD 10) is set as the reference direction for the virtual space. Also, for example, the reference direction for the virtual space may be adjusted by a user wearing the HMD 10 on his / her head performing a predetermined operation (such as a reset operation in the front direction).

[0028] Here, a change in the rotation direction around the Z axis is called a change in the yaw direction (left-right direction), a change in the rotation direction around the Y axis is called a change in the pitch direction (up-down direction), and a change in the rotation direction around the X axis is called a change in the roll direction. For example, the above-mentioned sensor 12 of the HMD 10 detects the angular velocity or angular acceleration in the rotation direction of each axis (yaw direction, pitch direction, and roll direction). Note that a change in the yaw direction is sometimes called a change in the left-right direction, and a change in the pitch direction is sometimes called a change in the up-down direction.

[0029] FIG. 3 is a diagram illustrating a field of view image of a field of view range viewed in a field of view direction from a virtual viewpoint in a virtual space according to this embodiment. In this diagram, a virtual viewpoint P (user's virtual viewpoint) in virtual space V is the intersection (origin) of the X-axis, Y-axis, and Z-axis. For example, assume that the user is facing forward and the direction of the user's line of sight with respect to virtual space V is along the X-axis (reference direction). In this case, the field of view range viewed in the field of view direction from virtual viewpoint P in virtual space V is a range determined by yaw angle α (the interior angle between dashed lines a and b, and the interior angle between dashed lines c and d) and pitch angle β (the interior angle between dashed lines a and d, and the interior angle between dashed lines b and c) centered on the reference direction (X-axis direction). Here, yaw angle α is the horizontal field of view, and pitch angle β is the vertical field of view, which are preset as the field of view angles of the field of view image displayed on HMD 10 in image display system 1. For example, the horizontal field of view of HMD 10 is 100 degrees.

[0030] For example, when the head of a user wearing the HMD 10 changes in the pitch direction or yaw direction, a change in the orientation and posture of the HMD 10 (a change in the visual axis direction of the HMD 10) is detected by the sensor 12 or the like. Based on the information detected by the sensor 12 or the like, the processor 21 of the display control device 20 determines that the line of sight direction with respect to the virtual space V has changed from the X-axis direction (reference direction) to the pitch direction or the yaw direction. The field of view direction is changed in accordance with this change in the line of sight direction with respect to the virtual space V, thereby changing the range of the field of view image (field of view range) displayed on the display unit 11. Similarly, when the head of a user wearing the HMD 10 changes in the roll direction, the change is detected by the sensor 12 or the like, and the range of the field of view image (field of view range) displayed on the display unit 11 rotates in the roll direction while the line of sight direction remains in the X-axis direction. In this way, the range of the field of view image displayed on the display unit 11 changes in accordance with the orientation (posture) of the HMD 10.

[0031] Furthermore, display objects other than the reproduced video, such as various objects, lines, symbols, and characters, can be placed as needed within the virtual space V. For example, an "object for making it possible to recognize the shooting direction" or an "arrow object," which will be described later, may be placed within the virtual space V and displayed as part of the field of view image.

[0032] When displaying a stereoscopic image using binocular parallax, there are viewing directions corresponding to the virtual viewpoints for the right eye and the left eye, and the field of view image for the right eye and the field of view image for the left eye for each viewing direction are displayed on the display unit 11 of the HMD 10.

[0033] [3. Display Control] In the image display system 1 of the present embodiment, captured data including real-life video data captured in a real space is used for playback. The outline of the display control executed in the image display system 1 or the display control device 20 is as follows.

[0034] In this embodiment, the shooting direction when shooting video is recorded, and when playing back the video, the video is played back in a projection direction based on the shooting direction. That is, shooting direction data indicating the shooting direction when shooting real-life video data is recorded in synchronization with the video data. That is, the shooting data includes video data of a real space and shooting direction data corresponding to the video data indicating the shooting direction of the video data in real space. Then, when playing back the shooting data including the video data, control is performed to change the projection direction of the video in the virtual space based on the shooting direction data corresponding to the video data. That is, in the virtual space, an image based on the video data is projected and displayed in a projection direction based on the shooting direction data corresponding to the video data. A specific example will be described below.

[0035] 4 is a diagram conceptually illustrating an example in which an image obtained by playing back video data captured in real space R is projected onto virtual space V. The diagram illustrating real space R in the lower part of FIG. 4 is a plan view of camera 200 capturing real space R from above, and the diagram illustrating virtual space V in the upper part of the same figure conceptually illustrates an XY cross section of virtual space V. Note that in FIGS. 5 to 8 and 17 described below, the diagrams illustrating virtual space V also conceptually illustrate the XY cross section of virtual space V.

[0036] A 360-degree spherical virtual screen (with a circular cross section) is set in the virtual space V, centered on the virtual viewpoint P, and the image produced by playing back video data is projected from the virtual viewpoint P into the virtual space V. In other words, the image produced by playing back video data and projecting it into the virtual space V is placed (displayed) at the position of the virtual screen in the virtual space V. By changing the projection direction of the image projected from the virtual viewpoint P into the virtual space V, the projection range of the image in the virtual space V (the range of the image placed in the virtual space V) changes (see Figure 5).

[0037] In this embodiment, the horizontal angle of view θH1 of the camera 200 capturing the real space R is set to 150 degrees. The capturing direction XC of the camera 200 in the real space R is the visual axis direction of the camera 200 (≈ the direction of the lens center). For example, when the camera 200 is started up, the visual axis direction of the camera 200 is set as the reference capturing direction XC0 for the real space R. Furthermore, for example, after the camera 200 is started up, the user may perform a predetermined operation to set the reference capturing direction XC0 for the real space R. As illustrated in FIG. 4, an image obtained by playing back "video data captured by the camera 200 in the reference capturing direction XC0 in the real space R" is projected in the virtual space V with the direction of the X axis, which is the reference direction, as the projection direction. In other words, the reference capturing direction XC0 in the real space R when the video data was captured corresponds to the reference direction (reference projection direction TD0) in the virtual space V when the video data was played back.

[0038] The projection range TR (image display range) in virtual space V of an image corresponding to the image data is based on the angle of view of camera 200 that captured the image data. For example, in the case of live-action image data captured by camera 200 with a horizontal angle of view θH1 (e.g., 150 degrees) and a vertical angle of view θV1 (e.g., 120 degrees), the projection range TR of the image in virtual space V is basically an area having a horizontal viewing angle θH2 (= θH1) and a vertical viewing angle θV2 (= θV1). In this embodiment, live-action image captured by camera 200 with a horizontal angle of view of 150 degrees is projected into virtual space V within a projection range TR having a horizontal viewing angle of 150 degrees.

[0039] As a variation, the projection range TR in virtual space V of a live-action image captured by a camera with a horizontal angle of view θH1 and a vertical angle of view θV1 may be narrowed to a horizontal viewing angle θH2 (<θH1) and a vertical viewing angle θV2 (<θV1) when projecting the image into virtual space V. For example, the projection range TR of the image may be narrowed when projecting (placing) the live-action image into virtual space V by not using areas near the edges of the image (areas far from the visual axis of camera 200), where distortion due to distortion aberration in the optical system of camera 200 is likely to occur.

[0040] Here, a case where the shooting direction XC of camera 200 in Fig. 4 is rotated to the right (yaw direction) is shown in Fig. 5. Fig. 5 is a conceptual diagram showing an example in which the projection direction TD of the video captured by camera 200 in virtual space V also changes in accordance with a change in the shooting direction XC when camera 200 captures an image.

[0041] In the example of FIG. 4, in the case of an image captured by camera 200 in reference imaging direction XC0, projection direction TD in virtual space V is reference projection direction TD0 = X-axis direction (reference direction). The horizontal projection range TR of the image in virtual space V is "-75 degrees to 75 degrees," with the X-axis direction being 0 degrees. On the other hand, in the example of FIG. 5, the imaging direction XC of camera 200 is rotated 25 degrees to the right (-25 degrees in the yaw direction) from reference imaging direction XC0 in FIG. 4. In the case of an image captured by camera 200 in FIG. 5, projection direction TD in virtual space V is rotated 25 degrees to the right (-25 degrees in the yaw direction) from reference projection direction TD0 = X-axis direction. As a result, the horizontal projection range TR of the image in virtual space V rotates 25 degrees to the right, changing to "-100 degrees to 50 degrees."

[0042] 6 is a conceptual diagram showing an example of the relationship between the real space R when video data is captured and the virtual space V when the video data is played back. When playing back captured data of the real space R, control is performed to change the projection direction TD of the video in the virtual space V based on the shooting direction data corresponding to the video data. Therefore, basically, the shooting direction XC in the real space R when the video data is captured corresponds to the projection direction TD in the virtual space V when the video data is played back. As the projection direction TD changes, the position of the projection range TR of the video in the virtual space V also changes.

[0043] On the other hand, the field of view direction SD in the virtual space V changes based on a change in the orientation of the HMD 10 (detection information by the sensor 12). As the field of view direction SD changes, the position of the field of view range SR in the virtual space V (the position of the field of view image) also changes.

[0044] Here, it is assumed that in real space R, a scene is taking place in which a car passes in front of the viewer from the left and then drives past to the right, and the camera 200 is turning from left to right to capture the image as the car moves.

[0045] FIG. 7 shows, as a comparative example of this embodiment, an example of time-series changes in the projection direction TD (and projection range TR) of the virtual space V onto which the image captured by the camera 200 is projected when the image capturing direction data (information on changes in the image capturing direction of the camera 200) is not used. When the image capturing direction data is not used, even if the image capturing direction of the camera 200 in the real space R changes, the projection direction TD of the image in the virtual space V does not change from the X-axis direction, which is the reference direction. Therefore, the position of the projection range TR in the virtual space V does not change either. The center of the image projected into the virtual space V (the center of the projection range TR) is always located in the X-axis direction, which is the reference direction. Therefore, when the captured image is displayed on the HMD 10 without using the image capturing direction data, the car in the image is displayed in the same direction relative to the visual axis (reference direction) of the HMD 10, even though it is actually moving, resulting in an image that lacks realism and presence.

[0046] FIG. 8 shows an example of time-series changes in the projection direction TD (and projection range TR) of the virtual space V onto which the image captured by the camera 200 is projected when the image capturing direction data (information on changes in the image capturing direction of the camera 200) is used. When the image capturing direction data is used, the projection direction TD and projection range TR in the virtual space V also rotate in accordance with changes in the image capturing direction XC. In this case, when the captured image is displayed on the HMD 10, an image of a car moving from left to right with respect to the visual axis of the HMD 10 is displayed. Therefore, if a car is located to the left during playback of the image data, the user must turn the HMD 10 to the left in order to see the car (or the car will be visible to the left when the user is facing forward). By turning their head in the direction of the car, the user can clearly see the car, resulting in a more realistic and immersive image.

[0047] [4. Functional configuration of display control device] 9 is a schematic functional block diagram showing an example of the functional configuration of the display control device 20. The display control device 20 executes control to display, on the display unit 11 of the HMD 10, an image of a field of view SR of the virtual space V as seen in a field of view direction SD from a virtual viewpoint P in the virtual space V (field of view image).

[0048] Here, the "virtual space V" is a virtual (pseudo) space generated by a computer (information processing device). For example, the virtual space V is constructed in a storage device (such as a VRAM) by computer calculations. A view range SR of a portion of the virtual space V is displayed on the display unit 11 of the HMD 10. The direction of the HMD 10 is linked to the real space R (real space) based on the detection results of a sensor 12 that detects the movement and tilt of the HMD 10, such as a gyro sensor.

[0049] The "field of view direction SD" is the direction of the HMD 10 in the virtual space V. In other words, the field of view direction SD is the direction that determines the "field of view range SR of virtual space V" displayed on the HMD 10 (see FIG. 6). Therefore, the field of view direction SD is the direction of the center of the "image of the field of view range SR of virtual space V" displayed on the display unit 11 of the HMD 10. The viewing direction SD may or may not match the user's line of sight. For example, in an HMD 10 with an eye tracking function, the viewing direction SD and the line of sight may differ.

[0050] The "viewing range SR" is the range of the virtual space V displayed on the HMD 10, and is the range of the virtual space V as seen from the virtual viewpoint P in the virtual space V in the viewing direction SD (see FIG. 6). The viewing range SR is a range of a predetermined viewing angle based on the viewing direction SD. The viewing range SR can be set according to the viewing angle of the display on the HMD 10.

[0051] Furthermore, the "image of the field of view SR" is an image displayed on the display unit 11 of the HMD 10 and is an image showing the field of view SR of the virtual space V. If an image projected into the virtual space V exists in the field of view SR, an image of the field of view SR of the virtual space V including that image will be displayed on the display unit 11 of the HMD 10. Note that the "image of the field of view SR" may include display objects other than the image projected into the virtual space V. For example, "display objects other than the projected image," such as CG (Computer Graphics), various objects, lines, symbols, and characters, may also be arranged in the virtual space V and superimposed on the image or displayed in the virtual space V without an image. The image of the field of view SR may be a non-stereoscopic image or may be a stereoscopic image (image for the right eye and image for the left eye) that can be viewed stereoscopically using binocular parallax. Furthermore, the image of the field of view SR is basically a moving image, but may also include still images. For example, when the image of the field of view SR is supplied (transmitted) to the display unit 11 of the HMD 10, it may be appropriately converted into a transmission format appropriate for the HMD.

[0052] 25, the "HMD" may be an HMD 50 including an attachment 51 (wearing implement) that can be worn on the user's head and an information processing device 52, such as a smartphone, that is attached to the attachment 51. In this case, a display unit 61 of the information processing device 52 serves as the display unit of the HMD. Furthermore, the "HMD" may be a so-called standalone HMD in which the display control function is integrated with the HMD. Furthermore, the term "HMD" may refer to goggles or glasses that can be worn on the head. Furthermore, the term "HMD" is not limited to the narrow definition of an HMD, but may also refer to, for example, headphones, a headset (headphones with a microphone), a glasses-type camera, an ear-hook camera, a camera hat, or the like that have HMD functionality.

[0053] 9, the display control device 20 includes a control unit 30. The control unit 30 is realized, for example, by the processor 21 executing a program stored in the storage device 22. The control unit 30 includes a memory control unit 31 and a projection unit 32. The storage control unit 31 has a function of storing, in the storage device 22, image data including image data of a real space R captured and image capture direction data corresponding to the image data and indicating the image capture direction XC of the image data in the real space R. Here, the "real space R" is a space in the real world that is not the virtual space V.

[0054] The "photography data" is information recorded when photographing the real space R. The photography data includes "video data" and "photography direction data." For example, the photography data may be included in application software such as a game.

[0055] "Video data" refers to data of live-action video captured in real space R. The format of the video data (or data format, file format, compression method, etc.) is not particularly limited. For example, known formats such as MP4, AVI, WebM, and MKV can be applied. Alternatively, formats to be developed in the future or proprietary formats may be applied. Audio data, subtitle data, etc. may be attached to the video data. The video data may also be referred to as imaging data, etc.

[0056] For example, the video data is obtained by capturing images of the real space R with a camera. Note that the camera itself is not included in the components for controlling the display of the HMD. Here, the term "camera" refers to a real camera that captures (images) the real space R. When a non-stereoscopic image is displayed on the HMD, data captured by at least one camera of the real space can be used as the video data. When a stereoscopic image utilizing binocular parallax is displayed on the HMD, data captured by a left-eye optical system (left-eye camera) that captures images for the left eye and a right-eye optical system (right-eye camera) that captures images for the right eye can be used as the video data. In this case, for example, a single camera may be equipped with an optical system for the left eye and an optical system for the right eye, as in a stereo camera. The left-eye camera and the right-eye camera may be configured separately. A camera that captures images in a single direction and is not binocular may also be used.

[0057] The "shooting direction data" is data recorded in association with video data, and is information that enables identification of the shooting direction XC of the video data in real space R. For example, the shooting direction data is recorded in synchronization with the video data during shooting. For example, the shooting direction data may be obtained by detecting information about the shooting direction XC from an angular velocity sensor (gyro sensor) or the like mounted on the camera during shooting of the real space R, and recording the information in association with the video data. Alternatively, the shooting direction XC may be identified from the captured video data by known image analysis during or after shooting, and the shooting direction data may be recorded in association with the captured video data.

[0058] For example, the shooting direction data is recorded as information that can identify the shooting direction XC of the video data in the real space R, corresponding to each frame of the video data. FIG. 10 is a diagram showing an example of shooting direction data. As illustrated in FIG. 10, the shooting direction data can be a table including fields for "frame" and "shooting direction." The "frame" field stores identification information (e.g., a frame number) for uniquely identifying each frame of video data. The "shooting direction" field includes fields for "yaw," "pitch," and "roll." The "yaw" field stores information on the yaw direction component corresponding to the shooting direction XC of the target frame of video data. Similarly, the "pitch" and "roll" fields store information on the pitch direction and roll direction components corresponding to the shooting direction XC of the target frame of video data, respectively. As illustrated in FIG. 10, shooting direction data may be recorded in association with each frame of video data (for all frames of video data).

[0059] Alternatively, in order to compress the amount of shooting direction data, the shooting direction data may be recorded once for multiple frames of video data (for example, two frames, four frames, etc.). Fig. 11 shows an example in which shooting direction data is recorded in association with every two frames of video data. When shooting direction data is recorded in association with video data for multiple frames, for example, for frames for which corresponding shooting direction data is not recorded, shooting direction data interpolated by linear interpolation or the like can be used when playing back the video data. A known algorithm can be used for the interpolation.

[0060] The shooting direction data may include data that can identify at least one rotational component among the yaw, pitch, and roll components corresponding to the shooting direction XC. The shooting direction data may be any information that can identify the shooting direction XC. For example, the shooting direction data may be information on the absolute rotation angles of the yaw, pitch, and roll components corresponding to the shooting direction XC, as exemplified in FIG. 10 or FIG. 11. Alternatively, the shooting direction data may be information on the amount of change from the shooting direction XC of the previous frame. When shooting direction data is recorded in association with video data for each of multiple frames, the shooting direction data may be information on the amount of change from the shooting direction XC of the frame in which the shooting direction data was previously recorded.

[0061] There is also video data in which the shooting direction XC remains unchanged for a while (i.e., the video is shot without moving the camera position for a while) and then changes. In such video data, shooting direction data may not be recorded for frames in which the shooting direction XC remains unchanged from that of the previous frame (i.e., frames in which the shooting direction XC remains unchanged). Furthermore, shooting direction data may be recorded for each frame or for every set of frames, focusing only on frames in which the shooting direction XC of the previous frame has changed. In this case, shooting direction data is not recorded during a period in which consecutive frames remain unchanged from the shooting direction XC of the previous frame, thereby reducing the amount of data.

[0062] For example, the shooting direction data can be recorded as header information of the shooting data (or video data). For example, when the MP4 file format is adopted, the shooting direction data can be recorded in the "moov area in which metadata can be recorded" included in the MP4 box structure. Alternatively, the shooting data and the video data may be stored in separate files, with the shooting data being recorded as information associated with the video data.

[0063] The "imaging direction XC" is a direction corresponding to the visual axis direction of the camera in the real space R. In other words, the imaging direction XC is a direction corresponding to the orientation of the camera capturing the real space R, and is the front direction (direction pointing toward the subject) of the optical axis of the imaging optical system of the camera. The imaging direction XC is also a direction toward the approximate center of the captured video (captured image).

[0064] The projection unit 32 has a function of projecting an image corresponding to the video data from a virtual viewpoint P into a virtual space V based on the shooting data. Here, the "image corresponding to the video data" is an image obtained by playing back the video data and projected into the virtual space V.

[0065] The projection unit 32 has a function of controlling the projection direction of an image corresponding to the image data in the virtual space V, based on the shooting direction data corresponding to the image data. Here, the "projection direction TD" is the direction in the virtual space V of the image that is displayed (arranged) by being projected into the virtual space V. The projection direction TD is also the direction of the center of the image projected into the virtual space V.

[0066] Furthermore, "controlling the projection direction TD of the image corresponding to the video data in the virtual space V based on the shooting direction data corresponding to the video data" means changing the projection direction TD of the image corresponding to the video data in the virtual space V based on the shooting direction XC of the shooting direction data corresponding to the video data (see Figures 4 to 6 and 8).

[0067] According to the configuration of the present embodiment described above, even when video is played back in the HMD 10 with a limited field of view SR, video with a higher sense of reality and presence than conventional video can be played back. That is, conventionally, the shooting direction XC of video data in the real space R is not taken into consideration when playing back video data. Therefore, as illustrated in FIG. 7 , even when video data in which the shooting direction XC in the real space R changes, the projection direction TD of the video in the virtual space V does not change, and the projection range TR of the video is fixed within the virtual space V. Therefore, the HMD 10 displays video with a lack of realism and presence. In contrast, according to the configuration of the present embodiment, when video data in which the shooting direction XC changes is played back, as illustrated in FIG. 8 , the projection direction TD and projection range TR of the video in the virtual space V change based on the shooting direction data. Therefore, for example, when video data in which the shooting direction XC changes to track a moving subject such as a car is played back, video with a high sense of reality and presence is played back on the HMD 10.

[0068] [5. Processing] Next, an example of processing executed by the display control device 20 of this embodiment will be described below. Fig. 12 is a flowchart showing an example of display control processing of the HMD 10 by the display control device 20. The processing described below is realized by the control unit 30 (processor 21 of the display control device 20) executing a program stored in the storage device 22 (the same applies to the processing referring to the flowcharts of Figs. 19 and 22).

[0069] In this embodiment, a display control process will be described for the case where "captured data" including the above-mentioned video data in which the real space R has been captured and the above-mentioned shooting direction data indicating the shooting direction XC of the video data in the real space R is played back and displayed on the HMD 10. When playing back captured data including video data for the left eye and the right eye that allows stereoscopic viewing using binocular parallax, the following process is executed for each of the left eye and the right eye. When the user inputs a predetermined operation from the operation unit 23 to start the display control process of the HMD 10, the display control device 20 starts the display control process.

[0070] In S100, the control unit 30 initializes the orientation of the HMD 10 and sets a reference direction in the virtual space V. As a result, for example, as shown in FIGS. 2 and 3, the reference direction (X axis) of the virtual space V is initially set corresponding to the visual axis direction of the HMD 10 when the user faces forward.

[0071] In S102, the control unit 30 reads the shooting data to be played back and stores it in the storage device 22. For example, the shooting data is stored on a recording medium, and the control unit 30 reads the shooting data from the recording medium via a recording medium drive and stores it in the storage device 22. Alternatively, the control unit 30 may download the shooting data to be played back via, for example, the Internet, and store it in the storage device 22. Note that if the shooting data has already been stored in the storage device 22 before S100, the processing of S102 can be omitted.

[0072] In S104, the control unit 30 initializes the projection direction TD of the image in the virtual space V when (before) playback of the captured image data including video data starts. That is, the control unit 30 sets a reference projection direction TD0 of the image in the virtual space V. In this embodiment, the control unit 30 sets the reference projection direction TD0 of the image corresponding to the reference capturing direction XC0 in the capturing direction data to the reference direction (X axis) of the virtual space V.

[0073] As a variation, the reference projection direction TD0 of the image corresponding to the reference shooting direction XC0 in the shooting direction data may be set to a direction deviated from the reference direction of the virtual space V (X axis).

[0074] In S106, the control unit 30 specifies a projection direction TD of the image in the virtual space V based on imaging direction data that corresponds to the image data and indicates an imaging direction XC of the image data in the real space R. In S108, the control unit 30 projects the image corresponding to the image data in the virtual space V in the projection direction specified in S106.

[0075] In S110, the control unit 30 acquires detection information regarding the orientation of the HMD 10 from the sensor 12 (for example, an angular velocity sensor) of the HMD 10. In S112, the control unit 30 sets (changes) the viewing direction SD based on the detection information regarding the orientation of the HMD 10 acquired in S110.

[0076] In S114, the control unit 30 generates an image of the field of view SR according to the field of view direction SD and outputs it to the HMD 10. The processes of S106 to S114 are repeatedly executed for each frame of the video until the display ends (YES in S116). As a result, when shooting data including video data in which the shooting direction XC changes is played back, the projection direction TD and projection range TR of the video in the virtual space V change based on the shooting direction data corresponding to the video data (see Figures 5 and 8), and a highly realistic and immersive video is played back.

[0077] [6. Display of projection direction] In this embodiment, the projection direction TD of the image in the virtual space V changes based on the shooting direction data, and therefore the projection range TR of the image may change (see FIGS. 5 and 8). Because the virtual space V that can be displayed on the HMD 10 is a wide space with a 360-degree horizontal and vertical aperture, it may happen that the user becomes confused about the projection direction TD of the image in the virtual space V. Therefore, as shown below, display control may be performed so that the user can recognize the projection direction TD (or the corresponding shooting direction XC).

[0078] [6-1. Displaying an object in an image that allows the direction of shooting or projection to be recognized] Typically, when capturing a real space R with the camera 200, the camera 200 is pointed in a direction that the photographer considers to be noteworthy (important). Therefore, it is considered that there is a lot of important information (such as noteworthy objects) in the capturing direction XC (the visual axis direction of the camera 200). The capturing direction XC of the image captured of the real space R corresponds to the projection direction TD of the image projected into the virtual space V, and corresponds to the direction of approximately the center of the image projected into the virtual space V (the projection range TR of the image). In other words, it is considered that there is a lot of important information in the direction of the center of the image projected into the virtual space V. As described above, in this embodiment, the projection direction TD of the image in the virtual space V changes based on the capturing direction data, so the projection range TR and center of the image may change (see FIGS. 5 and 8). Particularly when the projection range TR of the image projected into the virtual space V is wide, it may be difficult for a user viewing the image with the HMD 10 to determine the center of the image. Therefore, an object may be displayed in the image projected into the virtual space V to enable the user to recognize the capturing direction XC or the projection direction TD corresponding to the capturing direction XC. This is explained below.

[0079] 13 and 14 are diagrams showing an example of a screen displayed on the display unit 11 of the HMD 10. The screen G10 shown in Fig. 13 displays a field of view image of the field of view range in which a frame object OB1 is superimposed on the image to make it possible to recognize the projection direction TD (= shooting direction XC = center of the projection range TR of the image).

[0080] Note that an object for making the projection direction TD recognizable may be displayed directly at the center of the projection range TR of the image, which is the projection direction TD. However, as mentioned above, since the projection direction TD is likely to contain a lot of important information, the object may be visually obstructive. Therefore, as illustrated in FIGS. 13 and 14, a frame object OB1 is displayed around the projection direction TD (= shooting direction XC = center of the projection range TR of the image) so that the projection direction TD and the shooting direction XC can be inferred. The frame object OB1 is composed of four L-shaped frames that indicate the vertices of a rectangle centered on the projection direction TD of the image (= shooting direction XC). The user can recognize that the center of the rectangle of the frame object OB1 is the direction that is likely to require attention.

[0081] The size of the rectangle of the frame object OB1 (in other words, the distance of the four frames of the frame object OB1 from the center of the projection range TR of the image) can be set arbitrarily. However, it is desirable that the size of the rectangle of the frame object OB1 be smaller than the size of the image of the field of view SR.

[0082] The color of the frame object OB1 can also be set arbitrarily. Note that if the color of the video on which the frame object OB1 is superimposed and the color of the frame object OB1 are similar in color, the frame object OB1 may be difficult to recognize. Therefore, the color of the video on which the frame object OB1 is superimposed may be determined using a known image recognition process, and the color of the frame object OB1 may be changed according to the color of the video. For example, the color of the frame object OB1 is basically set to a predetermined default color. Then, if it is determined that the default color makes the frame object OB1 difficult to recognize in relation to the color of the video on which the frame object OB1 is superimposed, the color of the frame object OB1 may be changed to, for example, a color complementary to the default color.

[0083] 13, the field of view direction SD (=center of the field of view image) is shifted from the projection direction TD of the video, and therefore, a frame object OB1 is displayed at a position shifted from the center of the field of view image displayed on the screen G10 of the HMD 10. When a user looking at the screen G10 of FIG. 13 turns his / her head (HMD 10) toward the center of the frame object OB1, the screen G11 displayed on the display unit 11 of the HMD 10 is the field of view image exemplified in FIG. In the display control of the HMD10 of this embodiment, if the projection direction TD of the image in the virtual space V changes based on the shooting direction data, the position of the frame object OB1 also changes accordingly. If the user turns their head (rotates the viewing direction SD) while following the center of the frame object OB1, the viewing direction SD can easily follow the projection direction TD of the image that is considered to be of interest.

[0084] 15, the display control device 20 can be configured to include an object display unit 33. The object display unit 33 has a function of displaying an object in the image corresponding to the image data projected into the virtual space V, for enabling recognition of the shooting direction XC or the projection direction TD of the shooting direction data corresponding to the image data.

[0085] Here, the "object for enabling the user to recognize the shooting direction XC or the projection direction TD" is an object displayed in an image projected into the virtual space V, which allows the user to recognize the shooting direction XC at the time of shooting the image. Alternatively, it is an object for enabling the user to recognize the projection direction TD corresponding to the shooting direction XC. Here, the shooting direction XC of an image is the direction of the approximate center of the image. Furthermore, the projection direction TD corresponding to the shooting direction XC is the direction of the approximate center of the image projected in the virtual space V. Therefore, in other words, the object for enabling the user to recognize the shooting direction XC or the projection direction TD, which is displayed in the image projected into the virtual space V, is an object for enabling the user to recognize the direction of the approximate center of the image.

[0086] The object displayed in the image projected into the virtual space V may be displayed directly in the projection direction TD of the image (= shooting direction XC = center of the projection range TR of the image). Alternatively, the object displayed in the image may be displayed around the projection direction TD of the image, avoiding the projection direction TD of the image, so that the projection direction TD can be inferred. For example, a frame object OB1 (see FIGS. 13 and 14) that is an object displayed in the image and indicates each vertex of a rectangle centered on the projection direction TD of the image is an example of an "object for making the shooting direction XC or the projection direction TD recognizable." Not limited to a rectangle, a frame object that indicates each vertex of a polygon centered on the projection direction TD of the image is an example of an "object for making the shooting direction XC or the projection direction TD recognizable." Furthermore, for example, an object that indicates the circumference (or a part of the circumference) of a circle centered on the projection direction TD of the image is an example of an "object for making the shooting direction XC or the projection direction TD recognizable."

[0087] The object is placed in the virtual space V, for example, superimposed on an image projected into the virtual space V. As a result, if the image projected into the virtual space V is within the field of view SR, the object is superimposed on the image and displayed on the display unit 11 of the HMD 10. The object may be placed in the virtual space V and displayed superimposed on the image only when the image projected into the virtual space V is displayed in the view range SR. Alternatively, the object may be controlled to be placed in the virtual space V regardless of whether the image projected into the virtual space V is displayed in the view range SR.

[0088] In this configuration, an object is displayed in the image to enable the user to recognize the shooting direction XC or the projection direction TD corresponding to the shooting direction XC, so that the user viewing the image on the HMD 10 can recognize the direction that is likely to require attention. In particular, when the projection range TR of the image projected into the virtual space V is wide, it becomes difficult to determine the center direction of the image, so applying this configuration is effective.

[0089] [6-2. Displaying information outside the image that allows the projection direction to be recognized] In this embodiment, the viewing direction SD in the virtual space V and the viewing range SR determined by the viewing direction SD change depending on the orientation of the HMD 10 (i.e., the orientation of the head of the user wearing the HMD 10). Furthermore, the projection direction TD of the reproduced video in the virtual space V changes based on the shooting direction data included in the shooting data. For this reason, a large difference may sometimes occur between the viewing direction SD and the projection direction TD in the virtual space V, and no video may be displayed in the viewing range SR. In this case, the user may lose track of the video projection direction TD. In particular, when an image with a narrow viewing range SR is displayed on the HMD 10, the area outside the viewing range SR inevitably becomes wide, making it difficult for the user to determine which direction is the video projection direction TD (in which direction the video is being displayed). Therefore, as described below, display control may be performed on the area outside the video projection range TR so that the user can recognize the video projection direction TD.

[0090] 16 is a diagram showing an example of a screen G12 displayed on the display unit 11 of the HMD 10 when the image projected into the virtual space V is not displayed in the field of view SR. An arrow object OB2 (an example of information suggesting the projection direction of the image) is displayed on the screen G12. This suggests that the projection direction TD of the image is in the direction indicated by the arrow object OB2 (leftward in FIG. 16).

[0091] The arrow object OB2 is displayed in the field of view SR when the image projected into the virtual space V is not displayed in the field of view SR (in other words, when the difference between the projection direction TD and the field of view direction SD is such that the image is not displayed in the field of view SR). For example, the horizontal viewing angle of the image projection range TR is 150 degrees, and the horizontal viewing angle of the field of view SR is 100 degrees. In this case, if the difference between the projection direction TD and the field of view direction SD (the angle between them) becomes 125 degrees or more in the horizontal direction, the image will not be displayed in the field of view SR. While an example of the horizontal direction is shown here, the same applies to the vertical direction. If the difference between the projection direction TD and the field of view direction SD (the angle between them) becomes a predetermined value or more in the vertical direction, the image will not be displayed in the field of view SR. When the image projected into the virtual space V is not displayed in the field of view SR, the control unit 300 places the arrow object OB2 indicating the projection direction TD in the center of the field of view SR of the virtual space V. In other words, the arrow object OB2 is placed in the field of view direction SD.

[0092] The position where the arrow object OB2 is placed (displayed) is not limited to the center of the view range SR, but may be any position within the view range SR. The color of the arrow object OB2 can also be set arbitrarily as long as it can be recognized in relation to the background color of the screen G12.

[0093] If the user changes the orientation of the HMD 10 (head orientation) in the direction indicated by the arrow object OB2, the viewing direction SD approaches the projection direction TD of the image, and the image is displayed in the viewing range SR at the shortest possible time. Incidentally, since the virtual space V is a 360-degree space in all directions, even if the orientation of the HMD 10 is changed in the opposite direction to the direction indicated by the arrow object OB2, the image will ultimately be displayed in the field of view SR. The direction indicated by the arrow object OB2 here indicates the projection direction TD of the image that will cause the image to be displayed in the field of view SR in the shortest time possible.

[0094] Furthermore, as illustrated in FIG. 16, when an image projected into the virtual space V is not displayed in the field of view SR, the color (background color) of the background BC displayed in the field of view SR may be displayed in a gradation in addition to (or instead of) the arrow object OB2. FIG. 17 is a diagram showing the positional relationship in the virtual space V corresponding to the screen G12 shown in FIG. 16. For example, the gradation of this background BC has its darkest portion in the direction opposite to the projection direction TD and becomes lighter as it approaches the projection direction TD. This allows the user to visually determine the direction of the projection direction TD even when no image is projected in the field of view SR and the background BC is displayed, based on the color gradation of the background BC. In other words, by changing the orientation of the HMD 10 (head orientation) in the direction in which the background BC becomes brighter, the field of view direction SD approaches the projection direction TD of the image, and the image is displayed in the field of view SR in the shortest possible time.

[0095] The projection direction TD and viewing direction SD of the image in the virtual space V can change not only left and right (yaw direction, horizontal direction) but also up and down (pitch direction, vertical direction). Therefore, as shown in FIG. 18, an arrow object OB2 indicating the projection direction TD may indicate a diagonal direction. In this case, the gradation of the background BC becomes brighter as it approaches the projection direction TD indicated by the arrow object OB2. An arrow object OB2 indicating the upward or downward direction may also be displayed.

[0096] As shown in FIG. 15, the display control device 20 can be configured to include a detection information acquisition unit 34, a viewing direction change unit 35, and a projection direction information display unit 36. The detection information acquisition unit 34 has a function of acquiring detection information related to the orientation of the HMD 10. Here, "detection information related to the orientation of the HMD 10" refers to detection information related to the orientation of the HMD 10 that changes when the user changes the orientation of their head while wearing the HMD 10. The "detection information" can be acquired from a detection unit such as the sensor 12 that detects the orientation or changes in orientation of the HMD 10. For example, the detection result of the sensor 12 (such as an angular velocity sensor, an acceleration sensor, or a geomagnetic sensor) built into the HMD 10 is an example of "detection information related to the orientation of the HMD 10." Furthermore, the measurement result of a tracking system that captures an image of the HMD 10 using an imaging unit provided outside the HMD 10 and measures the position and orientation of the HMD 10 is an example of "detection information related to the orientation of the HMD 10." Furthermore, the measurement result of a tracking system that analyzes images of surrounding objects captured by an imaging unit built into the HMD 10 itself and measures the position and orientation of the HMD 10 is an example of "detection information related to the orientation of the HMD 10."

[0097] The viewing direction change unit 35 has a function of changing the viewing direction SD based on the detection information acquired by the detection information acquisition unit 34.

[0098] The projection direction information display unit 36 ​​has a function of displaying information suggesting the projection direction TD of the image projected into the virtual space V in the view range SR when the image is not displayed in the view range SR.

[0099] Here, the "information suggesting the projection direction TD of the image" refers to information that is displayed in the field of view SR when the image projected into the virtual space V is not displayed in the field of view SR, and is information such as an object that enables the user to indirectly determine the projection direction TD of the image. Since the image is not displayed in the field of view, the projection direction TD does not exist in the field of view SR, and therefore the projection direction TD itself cannot be directly indicated, but any information that enables the user to indirectly determine the projection direction TD will suffice.

[0100] The viewing direction SD (and the viewing range SR determined by the viewing direction SD) changes depending on the orientation of the HMD 10, while the projection direction TD of the image changes based on the shooting direction data included in the shooting data. In other words, the viewing direction SD and the projection direction TD change independently. As mentioned above, if a difference of a predetermined amount or more occurs between the viewing direction SD and the projection direction TD, no image will be displayed in the viewing range SR. Therefore, "when the image projected into the virtual space V is not displayed in the viewing range SR" can be said to be a case where a difference of a predetermined amount or more occurs between the viewing direction SD and the projection direction TD.

[0101] For example, an arrow object OB2 (see FIGS. 16 to 18) displayed in the field of view SR where no video is displayed, which indicates approaching the projection direction TD of the video, is an example of "information suggesting the projection direction TD of the video." In this case, if the user turns the HMD 10 (the direction of the head) in the direction indicated by the arrow object OB2, the field of view SD approaches the projection direction TD, and the video is displayed in the field of view SR in the shortest time. Furthermore, instead of or together with the arrow object OB2, text information such as "left," "right," "up," "down," "upper right," or "lower left" may be displayed, and this text information is an example of "information suggesting the projection direction TD of the video."

[0102] Also, for example, the gradation of the background BC displayed in the field of view SR where no image is displayed, which becomes brighter as it approaches the projection direction TD, is an example of "information suggesting the projection direction TD of the image." In this case, if the user changes the orientation of the HMD 10 (head direction) in the direction where the background BC becomes brighter, the field of view SD will approach the projection direction TD, and the image will be displayed in the field of view SR in the shortest time. Note that instead of a gradation of brightness difference (light and dark), a gradation of hue difference or saturation difference may also be used. The arrow object OB2 and the like and the gradation of the background BC may be displayed together, or only one of them may be displayed.

[0103] In this configuration, when an image projected into the virtual space V is no longer displayed in the field of view SR, information suggesting the projection direction TD of the image is displayed in the field of view SR. This allows the user to easily know in which direction to point the HMD 10 (head) so that the image will be displayed in the field of view SR.

[0104] Next, an example of processing for displaying information that enables the above-mentioned projection direction TD to be recognized will be described. Fig. 19 is a flowchart showing an example of processing by the display control device 20. The flowchart in Fig. 19 is a modified example of the flowchart in Fig. 12, and the same processing as in Fig. 12 is assigned the same step numbers and description thereof will be omitted. Also, in Fig. 19, descriptions of S100 to S104 are omitted.

[0105] 12, after executing S100 to S112, control unit 30 determines whether the image projected into virtual space V is displayed in field of view SR (S200). For example, control unit 30 determines that the image will not be displayed in field of view SR when the difference between projection direction TD and field of view direction SD (the angle between them) is equal to or greater than a predetermined value, and determines that the image will be displayed in field of view SR when the difference is less than the predetermined value.

[0106] When the control unit 30 determines that the image projected into the virtual space V is displayed in the field of view SR (YES in S200), it displays (places) a frame object OB1 indicating the projection direction TD within the image (S202). That is, in S202, the control unit 30 places a rectangular frame object OB1 centered in the projection direction TD of the image in the virtual space V, superimposed on the projected image.

[0107] On the other hand, if the control unit 30 determines that the image projected into the virtual space V is not displayed in the field of view SR (NO in S200), it displays (places) an arrow object OB2 indicating the projection direction TD in the center of the field of view SR (field of view direction SD) (S204). In addition, the control unit 30 displays the background BC displayed in the field of view SR with a gradation that indicates the projection direction TD (S206). For example, the control unit 30 places the background BC, which has a gradation that becomes brighter as it approaches the projection direction TD, at least in the field of view SR.

[0108] As shown in FIG. 17, a background BC having a gradation that becomes brighter as it approaches the projection direction TD may be placed not only in the field of view SR but also in the entire area outside the projection range TR of the image.

[0109] After executing S202 or S206, the control unit 30 generates an image of the field of view range SR according to the field of view direction SD and outputs it to the HMD 10. The processing from S106 onwards is repeatedly executed for each frame of the video until the display ends (YES in S116).

[0110] As a result, when the image projected into the virtual space V is displayed in the field of view SR, that is, when the image played back on the HMD 10 is displayed, a frame object OB1 is displayed as shown in Fig. 13 or 14. This allows the user to recognize the direction in the image that is likely to be of interest. On the other hand, when the image projected into the virtual space V is not displayed in the field of view SR, that is, when the image played back on the HMD 10 is not displayed, the arrow object OB2 and the gradation of the background BC are displayed as shown in Fig. 16 or 18. This allows the user to recognize the projection direction TD of the image, and easily know in which direction to point the HMD 10 (head) to display the image.

[0111] [7. Aspects of limiting rotation (change) of projection direction for at least some rotation directions] The projection unit 32 may limit the rotation of the projection direction TD in at least some of the rotation directions of the yaw direction, the pitch direction, and the roll direction. Here, "limiting the rotation of the projection direction TD" means limiting the change in the projection direction TD of an image when the image is projected into the virtual space V. "Limiting the rotation of the projection direction TD" includes preventing the projection direction TD from rotating in the yaw direction, pitch direction, or roll direction. Furthermore, "limiting the rotation of the projection direction TD" includes limiting the range of rotation (range of rotation angle) of the projection direction TD in the yaw direction, pitch direction, or roll direction using a threshold value.

[0112] According to this configuration, by restricting the rotation of the projection direction TD in at least some of the yaw direction, pitch direction, and roll direction, excessive strain on the user's neck and eyes when viewing through the HMD 10 can be reduced. Various examples of limiting the rotation of the projection direction TD are given below.

[0113] [7-1. How to limit part of the rotation direction reflected in the projection direction] The projection unit 32 may not reflect some rotational direction components among the yaw, pitch, and roll direction components corresponding to the imaging direction XC in the projection direction TD. That is, when controlling the projection direction TD of the image in the virtual space V based on the imaging direction data indicating the imaging direction XC of the image data in the real space R, the projection unit 32 has a function of not reflecting some rotational direction components among the yaw, pitch, and roll direction components corresponding to the imaging direction XC in the control of the projection direction TD.

[0114] With this configuration, changes in the projection direction TD of the image in the virtual space V are limited to changes in only one or two of the rotational directions of yaw, pitch, and roll, thereby reducing strain on the user's neck and eyes.

[0115] [7-1-1. How to limit the pitch rotation of the projection direction] The projection unit 32 may not reflect the pitch direction component corresponding to the shooting direction XC in the projection direction TD. That is, the projection unit 32 has a function of not reflecting the pitch direction component corresponding to the shooting direction XC in the control of the projection direction TD when controlling the projection direction TD of the image in the virtual space V based on the shooting direction data indicating the shooting direction XC of the video data in the real space R.

[0116] For example, when controlling the projection direction TD based on the shooting direction data shown in Figure 10 or Figure 11, the projection unit 32 does not reflect (does not use) the shooting direction data in the pitch direction (data in the "pitch" field) in the control of the projection direction TD. In this case, the recording of unused "pitch" field data may be omitted from the shooting direction data (the shooting direction data may not include information on the pitch direction component corresponding to the shooting direction XC). This reduces the storage capacity of the shooting direction data.

[0117] According to this configuration, the image displayed on the display unit 11 of the HMD 10 does not move in the pitch direction (up and down direction) relative to the HMD 10 (i.e., relative to the user's head and eyes), but is limited to movement in the horizontal direction (yaw direction). Generally, when a user moves their neck or line of sight, the pitch direction (up and down direction) places a greater burden on the user than the yaw direction (left and right direction). Therefore, by limiting the movement of the image in the pitch direction with this configuration, the burden on the user's neck and eyes can be effectively reduced.

[0118] Furthermore, the direction of the head of a user wearing the HMD 10 may change depending on the user's posture. For example, if the user is sitting in a reclining chair with a slanted backrest, the user's line of sight will be directed slightly upward. Even in such a case, by limiting the rotation of the projection direction TD in the pitch direction using this configuration, the projection direction TD of the image can be maintained horizontal to the user's eyes, reducing strain on the user's neck and eyes.

[0119] [7-1-2. How to limit the roll direction rotation of the projection direction] The projection unit 32 may not reflect a component in the roll direction corresponding to the shooting direction XC in the projection direction TD. That is, the projection unit 32 has a function of not reflecting a component in the roll direction corresponding to the shooting direction XC in the control of the projection direction TD when controlling the projection direction TD of the video in the virtual space V based on the shooting direction data indicating the shooting direction XC of the video data in the real space R.

[0120] For example, when controlling the projection direction TD based on the shooting direction data shown in Figure 10 or Figure 11, the projection unit 32 does not reflect (does not use) the shooting direction data in the roll direction (data in the "Roll" field) in the control of the projection direction TD. In this case, the recording of unused data in the "roll" field of the shooting direction data may be omitted (the shooting direction data may not include information on the roll direction component corresponding to the shooting direction XC). This reduces the storage capacity of the shooting direction data.

[0121] With this configuration, the roll direction component corresponding to the shooting direction XC is not reflected in the projection direction TD of the image, which causes a change in the difference between the direction of gravity in the real world relative to the HMD 10 (i.e., relative to the user's head) and the direction of gravity in the image. In this case, it may be possible to display an image that is dynamic and impactful, which may be advantageous for image presentation. A specific example is shown below.

[0122] FIG. 20 is a diagram illustrating an example of an image projected into a virtual space V when a roll direction component corresponding to the shooting direction XC is reflected in the projection direction TD. The image in FIG. 20 is an image captured in a real space R by an onboard camera 200 of a motorcycle during a motorcycle race. In this example, the motorcycle leans to the left due to a left corner, and the shooting direction XC of the camera 200 also rolls to the left. As in this example, when playing back video data that leaned to the left (rolled to the left) during shooting in the real space R, if the roll direction component corresponding to the shooting direction XC is reflected in the projection direction TD, the image display will be as illustrated in FIG. 20. In this case, the roll rotation of the shooting direction XC is also reproduced in the image, so that the gravity direction GD1 of the image and the gravity direction GD2 relative to the HMD 10 always coincide and do not change. As such, an image in which the gravity direction GD1 of the image and the gravity direction GD2 relative to the HMD 10 always coincide and do not change has the advantage of being less likely to cause so-called VR sickness. On the other hand, the image tends to lack impact because there is little change in the image. Therefore, it is possible to intentionally limit the rotation of the image projection direction TD in the roll direction.

[0123] 21 is a diagram showing an example of an image projected into a virtual space V when the same image data as in FIG. 20 is played back, but the roll direction component corresponding to the shooting direction XC is not reflected in the projection direction TD. In this way, by intentionally restricting the roll direction rotation of the projection direction TD of the image, the gravity direction GD1 of the image can be changed with respect to the gravity direction GD2 relative to the HMD 10, resulting in an image display that is dynamic and impactful.

[0124] The above-mentioned process of "not reflecting in the projection direction TD some of the rotational direction components among the yaw, pitch, and roll direction components corresponding to the imaging direction XC" is performed, for example, by the following process in S106 of the flowchart in Fig. 12 or 19. That is, the control unit 30 specifies the projection direction TD of the image in the virtual space V using only the "data of the rotational direction components reflected in the projection direction TD" in the imaging direction data (see Fig. 10 or 11).

[0125] [7-1-3. Limiting the rotation of the projection direction by a threshold value] The method of restricting the rotation of the projection direction TD of the image projected into the virtual space V is not limited to the above-mentioned method of preventing a specific rotation direction component corresponding to the shooting direction XC from being reflected in the projection direction TD, but may also be restricted by the degree of rotation (rotation angle) of the projection direction TD.

[0126] For example, if a user wearing the HMD 10 were to look in a direction rotated 180 degrees in the yaw direction from the reference direction (direction directly opposite the user's back) using the direction facing forward as the reference direction, this would place a significant strain on the user's neck and eyes. Therefore, a threshold may be set for rotation directions in which the projection direction TD significantly changes from the reference direction (front direction), so that the projection direction TD does not change at rotation angles exceeding the threshold. For example, thresholds may be set for the rotation directions in which the projection direction TD rotates from the reference direction, such as 90 degrees left and right in the yaw direction and 60 degrees up and down in the pitch direction, and the rotation of the projection direction TD is limited by the thresholds. Furthermore, since the roll direction may also place a strain on the neck, a threshold may be set for the rotation angle in the projection direction TD from the reference direction, such as 45 degrees left and right in the roll direction. The rotation of the projection direction TD may be limited by setting a threshold for at least some of the rotation directions among the yaw direction, the pitch direction, and the roll direction. The thresholds are not limited to the above example and may be set arbitrarily. Furthermore, when the shooting direction XC of the shooting direction data (or the projection direction TD corresponding to the shooting direction XC) exceeds the threshold, the video may be projected into the virtual space V with the threshold as the projection direction TD.

[0127] The projection unit 32 in this embodiment has the function of limiting the rotation of the projection direction TD projection direction for at least some of the rotation directions among the yaw direction, pitch direction, and roll direction so that the rotation angle from the reference direction in the projection direction TD does not exceed a predetermined threshold value. Here, the "reference direction" is a reference direction in the virtual space V, and as in the example described above, can be the direction of the X axis (see Figures 2, 3, etc.) corresponding to the direction in which the user wearing the HMD 10 faces forward. In addition, for rotation in the roll direction, the Y axis or Z axis can be the reference direction (see Figures 2, 3, etc.). Note that the reference direction may be any direction that serves as a reference in the virtual space V, and therefore may be set to a direction deviated from the X axis, Y axis, or Z axis.

[0128] With this configuration, the rotation of the projection direction TD is limited by a threshold value for at least some of the rotation directions of the yaw, pitch, and roll directions (for example, a threshold value of 90 degrees left and right in the yaw direction from the reference direction, and 60 degrees up and down in the pitch direction from the reference direction). This reduces strain on the user's neck and eyes.

[0129] Furthermore, the projection unit 32 in this embodiment has the function of projecting an image in a projection direction TD corresponding to the threshold value when the rotation angle from the reference direction of at least some of the rotational direction components of the yaw direction, pitch direction, and roll direction components corresponding to the shooting direction XC exceeds the threshold value.

[0130] According to this configuration, when the rotation angle (change from the reference direction) of the imaging direction XC (or the projection direction TD corresponding to the imaging direction XC) from the reference direction exceeds a threshold value for at least some of the rotation directions among the yaw, pitch, and roll directions, the image is projected in the projection direction TD corresponding to the threshold value. As a result, if the change from the reference direction of the imaging direction XC is within the threshold value, the projection direction TD of the image changes according to the imaging direction XC, thereby enabling the playback of images with a higher sense of reality and presence than conventional images. Furthermore, when the rotation angle of the imaging direction XC from the reference direction exceeds the threshold value, the image is projected into the virtual space V while remaining in the projection direction TD corresponding to the threshold value. This eliminates the need for the user to move their neck or eyes to follow the movement of the image. This allows the playback of images with a higher sense of reality and presence than conventional images, while reducing the strain on the user's neck and eyes by limiting the range of movement (projection range) of the image in the virtual space V using a threshold value.

[0131] Next, an example of the process of limiting the rotation of the projection direction TD using a threshold value will be described. Fig. 22 is a flowchart showing an example of the process of the display control device 20, and is an example of the "limitation process using a threshold value" of a subroutine executed instead of S106 in Fig. 12 or 19.

[0132] In S300, the control unit 30 identifies the projection direction TD of the image in the virtual space V based on the imaging direction data corresponding to the image data and indicating the imaging direction XC of the image data in the real space R. In S302, the control unit 30 determines whether the rotation angle of the projection direction TD in the yaw direction from the reference direction exceeds a threshold (S304). If the result of S302 is YES, the yaw direction of the projection direction TD is corrected to a direction corresponding to the threshold. On the other hand, if the result of S302 is NO, the process proceeds to S306. In S306, the control unit 30 determines whether the rotation angle of the projection direction TD in the pitch direction from the reference direction exceeds a threshold. If the result of S306 is YES, the pitch direction of the projection direction TD is corrected to a direction corresponding to the threshold (S308). On the other hand, if the result of S306 is NO, the process proceeds to S310. In S310, the control unit 30 determines whether the rotation angle of the projection direction TD in the roll direction from the reference direction exceeds a threshold. If the answer is YES in S310, the roll direction of the projection direction TD is corrected to a direction corresponding to the threshold (S312). On the other hand, if the answer is NO in S310, or after S312 is executed, the subroutine ends.

[0133] 22 shows the process when a threshold is set for each of the rotation directions, yaw, pitch, and roll, but if there is a rotation direction for which a threshold is not set, the process for that rotation direction can be omitted. For example, if a threshold is not set for the roll direction, S310 and S312 can be omitted.

[0134] [7-1-4. User-configurable restrictions on rotation of projection direction] The user may be able to set the content of the restriction on the rotation of the projection direction TD of the image. For example, the user may be able to set whether or not at least some of the rotation directions among the yaw direction, pitch direction, and roll direction are to be not reflected in the projection direction TD, or the user may be able to set the threshold value.

[0135] 15, the display control device 20 can be configured to include a restriction content setting unit 37. This restriction content setting unit 37 has a function of setting the restriction content of the rotation of the projection direction TD based on a user operation. Here, "setting the restriction on the rotation of the projection direction TD" includes setting whether to rotate the projection direction TD in all or part of the yaw, pitch, and roll directions. Also, "setting the restriction on the rotation of the projection direction TD" includes setting a threshold for the range (range of rotation angle) in which the projection direction TD rotates in all or part of the yaw, pitch, and roll directions.

[0136] The restriction content setting unit 37 displays a setting screen for setting the restriction content for the rotation of the projection direction TD on, for example, a display unit provided in the display control device 20, a display unit externally connected to the display control device 20, or the display unit 11 of the HMD 10. Furthermore, the restriction content setting unit 37 accepts the setting of the restriction content on the setting screen based on a user operation on the operation unit 23.

[0137] This configuration allows the user to set the rotation restriction of the projection direction TD of the image, enabling a high degree of freedom in image playback. For example, the user can set a setting that prioritizes reducing strain on the neck and eyes (by increasing the rotation restriction of the projection direction TD), or a setting that prioritizes highly realistic and immersive image playback (by eliminating or reducing the rotation restriction of the projection direction TD).

[0138] The "processing for setting the restriction on the rotation of the projection direction TD based on the user's operation" according to this aspect can be executed, for example, before the start of playback of the captured data. For example, in the flowcharts of Fig. 12 or 19, this process may be executed before S100, S102, or S104.

[0139] Furthermore, the processing according to this aspect may be executable even after the start of playback of the captured image data. For example, when the user performs a predetermined operation during playback of the captured image data, an interrupt process may be generated to interrupt (pause) the playback, allowing the user to set the restriction on the rotation of the projection direction TD (or change the current setting).

[0140] In addition, the setting information of the restriction contents set by the user once may be saved, and the saved restriction contents may be applied to subsequent playback of captured image data until the user changes the setting.

[0141] [8. Methods for preventing VR sickness] It is important to take measures to prevent users of the HMD 10 from experiencing VR sickness. It has been found that a wide field of view SR makes it easier for users to experience VR sickness when viewing an image in the field of view SR displayed on the display unit 11 of the HMD 10. Therefore, VR sickness may be suppressed by narrowing the field of view SR (or narrowing the display range of the image in the field of view SR).

[0142] 23 and 24 are diagrams showing an example of a screen displayed on the display unit 11 of the HMD 10, in which the display range of the image in the field of view SR is narrowed. In the following, narrowing the visibility range SR includes narrowing the display range of the image in the visibility range SR (or widening the non-display range of the image in the visibility range SR).

[0143] As a specific example, the area near the center of the field of view direction SD (a predetermined range from the field of view direction SD in the field of view range SR) is displayed in a normal state, and the area outside the center is darkened as it moves away from the field of view direction SD, thereby narrowing the effective display range of the image in the field of view range SR. To achieve this, for example, a mask process is used in which a mask (a mask for hiding part of the image) for narrowing the display range of the image in the field of view SR is placed in the virtual space V. As described above, if the field of view image is darkened as it moves away from the field of view direction SD, a mask with variable opacity (or transparency) can be placed. When the display range of the image in the field of view SR is narrowed in this way, for example, an image of screen G13 shown in FIG. 23 is displayed on the display unit 11 of the HMD 10. As a result, the peripheral field of view of the HMD 10 other than the field of view near the center becomes darker as it moves away from the center of the field of view, thereby narrowing the field of view.

[0144] In particular, it is desirable to narrow the visibility range SR in the left and right directions. The method of narrowing the field of view SR is not limited to the above. For example, as shown in screen G14 in Fig. 24, the image of predetermined left and right end regions of the field of view SR may simply be cut off. In this case, a mask process may be performed in which a mask is placed on the left and right end regions of the field of view SR, or the size (field of view angle) of the field of view SR itself may be set narrower than the field of view that can be displayed by the HMD 10 (default field of view angle). As exemplified in Fig. 24, only the horizontal field of view angle may be narrowed, or the vertical field of view angle may also be narrowed.

[0145] The narrowing of the field of view range SR may be manually set by the user. For example, the field of view angle that can be displayed by the HMD 10 may be set as the default field of view range (image display range), and the user may operate the field of view range change operation to set a field of view range narrower than the default field of view range. In this case, the size (narrowness) of the field of view range may be set in multiple stages.

[0146] Alternatively, the control unit 30 may automatically adjust the size (narrowness) of the field of view SR. In particular, as described above, in this embodiment, the projection direction TD of the image changes within the virtual space V independently of the user's head movement. Therefore, it is also a preferred embodiment to control the width of the field of view SR in relation to the change in the projection direction TD of the image to prevent VR sickness. As exemplified below, the narrowness of the field of view SR may be automatically set based on the change in the projection direction TD (at least some of the rotation directions of the yaw direction, pitch direction, and roll direction).

[0147] For example, when the projection direction TD of the image in the virtual space V changes by more than a certain amount, the field of view range SR may be automatically set so that it is narrower than normal (compared to when it does not change by more than a certain amount).

[0148] For example, when the projection direction TD starts to rotate from a stopped (not rotating) state, and the rotation angle of the projection direction TD exceeds a predetermined threshold, the field of view range SR may be automatically set to a narrower value.

[0149] Furthermore, it is believed that the faster the rate of change of the projection direction TD, the more likely VR sickness is to occur. Therefore, for example, when the rotation angle (i.e., angular velocity) per unit time in the projection direction TD exceeds a predetermined threshold, the field of view range SR may be automatically narrowed (the field of view range SR may be set narrower than before the threshold was exceeded).

[0150] Alternatively, the view range SR may be set narrower as the angular velocity of the rotation of the projection direction TD increases. In this case, the size of the view range SR is automatically adjusted according to the angular velocity of the projection direction TD.

[0151] Furthermore, it is considered that the more rapidly the projection direction TD changes, the more likely VR sickness is to occur. Therefore, for example, when the angular acceleration of the rotation of the projection direction TD exceeds a predetermined threshold, the field of view SR may be automatically narrowed (the field of view SR may be set narrower than before the threshold was exceeded).

[0152] Alternatively, the field of view SR may be set narrower as the angular acceleration of the rotation of the projection direction TD increases. In this case, the size of the field of view SR is automatically adjusted according to the angular acceleration of the projection direction TD.

[0153] Furthermore, VR sickness can also occur due to changes in the viewing direction SD. Therefore, it is a preferred embodiment to prevent VR sickness by controlling the width of the viewing range SR in relation to changes in the viewing direction SD. As exemplified below, the narrowness of the viewing range SR may be automatically set based on changes in the viewing range SR (at least some of the rotation directions of the yaw direction, pitch direction, and roll direction).

[0154] For example, when the viewing direction SD in the virtual space V changes by more than a certain amount, the viewing range SR may be automatically set so that it is narrower than normal (compared to when it does not change by more than a certain amount). For example, when the viewing direction SD starts to rotate from a stationary (not rotating) state, and the rotation angle of the viewing direction SD exceeds a predetermined threshold, the viewing range SR may be automatically set to a narrower value.

[0155] Furthermore, it is believed that the faster the speed of change of the viewing direction SD, the more likely VR sickness is to occur. Therefore, for example, when the angular velocity of the rotation of the viewing direction SD exceeds a predetermined threshold, the viewing range SR may be automatically narrowed (the viewing range SR may be set narrower than before the threshold was exceeded). Alternatively, the view range SR may be set narrower as the angular velocity of the rotation of the view direction SD increases. In this case, the size of the view range SR is automatically adjusted according to the angular velocity of the view direction SD.

[0156] Furthermore, it is considered that the more rapidly the field of view direction SD changes, the more likely VR sickness will occur. Therefore, for example, when the angular acceleration of the rotation of the field of view direction SD exceeds a predetermined threshold, the field of view range SR may be automatically narrowed (the field of view range SR may be set narrower than before the threshold was exceeded). Alternatively, the field of view range SR may be set narrower as the angular acceleration of the rotation of the field of view direction SD increases. In this case, the size of the field of view range SR is automatically adjusted according to the angular acceleration of the field of view direction SD.

[0157] Furthermore, the aforementioned automatic setting of the narrowness of the field of view range SR based on a change in the field of view range SR and the automatic setting of the narrowness of the field of view range SR based on a change in the field of view direction SD may be used in combination.

[0158] [9. Modifications, etc.] Although the embodiments of the present invention have been described above, the specific configurations are not limited to the above-described embodiments, and the present invention also includes designs that do not deviate from the gist of the present invention. Furthermore, the above-described configurations and aspects can be combined in any manner.

[0159] [9-1] The above mainly describes an example of playing back "shooting data" including "video data" and "shooting direction data." However, the present invention can also be applied to playing back "video data for which shooting direction data has not yet been associated." That is, while playing back video data, changes in the shooting direction may be identified in real time using known image analysis to generate shooting direction data. In this way, the projection direction TD of the image corresponding to the video data may be controlled based on the shooting direction data generated during playback of the video data.

[0160] [9-2] It is also possible to display on the HMD 10 actual images (live images) captured in real time by the camera 200 that captures the real space R. For example, the camera 200 illustrated in FIG. 6 may be connected to the display control device 20 via a wired or wireless connection, and live video captured by the camera 200 may be displayed on the HMD 10 as a VR image. Alternatively, the camera 200 at a remote location and the display control device 20 may be connected via a network such as the Internet to enable data communication. In this case, the live video of the remote location captured by the camera 200 may be displayed on the HMD 10 as a VR image. Alternatively, a server having a distribution service function may be set on the network, and the live video captured by the camera 200 may be distributed via the network via the server. In this case, the display control device 20 may receive the live video of the remote location distributed via the network, and the live video may be played back in approximately real time and displayed on the HMD 10.

[0161] In other words, an image display system including a camera that captures real space R, a display control device with the above-mentioned configuration, and an HMD, or an image display system including the camera, a distribution server, a display control device with the above-mentioned configuration, and an HMD, can be constructed to control the projection direction TD of an image in virtual space V based on shooting direction data corresponding to the video data captured in real space R.

[0162] [9-3] As illustrated in FIG. 1, when the HMD 10 and the display control device 20 both have the configuration and functions of an information processing device (computer) equipped with a processor and a storage device, some of the functions of the control unit 30 described above may be realized by the processor 13 of the HMD 10 that executes the program according to this embodiment, and the remaining functions may be realized by the processor 21 of the display control device 20 that executes the program according to this embodiment.

[0163] [9-4] FIG. 26 is a schematic block diagram showing an example of the configuration of a standalone HMD 60 in which the functions of the display control device 20 are integrated with the HMD 10, or an information processing device 52 such as a smartphone used as the HMD 50 shown in FIG. 25. The HMD 60 or the information processing device 52 includes a display unit 61, a sensor 62, a processor 63, a storage device 64, an operation unit 65, and a communication unit 66. The display unit 61, the sensor 62, the processor 63, the storage device 64, the operation unit 65, and the communication unit 66 have the same configurations as the display unit 11, the sensor 12, the processor 21, the storage device 22, the operation unit 23, and the communication unit 24, respectively, and therefore their description will be omitted. Note that the operation unit 65 or the communication unit 66 may be configured separately, such as externally, or may be omitted. This standalone HMD 60 or the information processing device 52 also has the configuration and functions of an information processing device (computer) equipped with a processor and a storage device, and the functions of the control unit 30 described above are realized by the processor 63 that executes the program according to this embodiment.

[0164] [9-5] Some or all of the functions of the control unit 30 described above may be implemented by an integrated circuit such as an LSI (Large Scale Integration). Also, each of the above functions may be implemented individually as a processor. Alternatively, some or all of the above functions may be integrated into a processor.

[0165] [9-6] The computer-readable program according to this embodiment is recorded on various computer-readable recording media, such as a hard disk, an optical disk (CD-ROM, DVD-ROM, etc.), a flexible disk, or a semiconductor memory, and is read from the recording media and executed by a computer constituting the image display system 1 or the display control device 20. The program may also be provided to a computer via a network, including a communication line such as the Internet, a WAN, a LAN, or a dedicated line. A computer may read a program stored in a file server (online storage). A computer may also receive a program distributed from a distribution server. The recording media may include internal or external recording media accessible from the distribution server for program distribution. The program code stored on the distribution server's recording media does not need to be in a format directly executable by the computer that receives it. In other words, the format of the program stored on the distribution server's recording media is arbitrary, as long as it can be installed in a computer executable manner after being downloaded from the distribution server. A program may also be divided into multiple parts, downloaded at different times, and then combined. Each of the divided programs may be distributed by a different distribution server. The computer-readable recording medium also includes a storage medium that stores the program for a certain period of time, such as a volatile memory such as RAM in a server that transmits the program via a network or in a computer that receives the program. The program may also be a differential program that can achieve the above-mentioned functions in combination with a program already stored in the computer.

[0166] [10. Notes] From the above description, the present invention can be understood, for example, as follows: Note that, to facilitate understanding of the present invention, reference numerals in the accompanying drawings are conveniently placed in parentheses, but this does not mean that the present invention is limited to the illustrated embodiments.

[0167] 1) A program according to one aspect of the present invention causes a computer that executes control to display on a display unit (11) of a head-mounted display (10) an image of a field of view (SR) of a virtual space (V) as viewed from a virtual viewpoint (P) in the virtual space in a field of view (SD). The program causes the computer to function as a storage control unit (31) that stores in a storage device (22) image data of a real space (R) and image capture data including image capture direction data corresponding to the image data and indicating the image capture direction (XC) of the image data in the real space, and a projection unit (32) that projects an image corresponding to the image data from the virtual viewpoint (P) into the virtual space (V) based on the image capture data. The projection unit (32) controls a projection direction (TD) of the image corresponding to the image data in the virtual space (V) based on the image capture direction data corresponding to the image data.

[0168] Here, the "computer" may include at least a processor and a storage device (memory). Here, the processor is, for example, a CPU. The processor may also include hardware such as a GPU, DSP, or FPGA in addition to or instead of the CPU. For example, personal computers, tablet computers, smartphones, stationary or portable game consoles, commercial game consoles, mobile phone terminals, PHS terminals, PDAs, multi-function television receivers with information processing functions, servers, and other devices that include a processor and a storage device are all included in the "computer." Furthermore, a "computer" may be composed of multiple devices that can communicate with each other. For example, a system including a server and a terminal device is also included in the "computer." The HMD itself, including the processor and storage device, is also included in the "computer" category. For example, in the case of an HMD that uses an information processing device such as a smartphone as an attachment, the smartphone is included in the "computer" category. Also, for example, a standalone HMD is included in the "computer" category.

[0169] According to the aspect described in 1) above, even when playing back video within a limited field of view in an HMD, it is possible to play back video with a higher sense of reality and presence than conventionally possible.

[0170] 2) In one aspect of the present invention, in the aspect described in 1) above, the computer is further made to function as an object display unit (33) that displays an object (OB1) in the image corresponding to the video data projected into the virtual space (V) to enable recognition of the shooting direction (XC) or the projection direction (TD) of the shooting direction data corresponding to the video data.

[0171] In the aspect described in 2) above, the shooting direction of the image captured in the real space corresponds to the projection direction of the image projected into the virtual space, which corresponds to the direction of approximately the center of the image. Typically, when capturing a real space with a camera, the camera is pointed in a direction that the photographer considers noteworthy (important), so it is believed that there is a lot of important information (such as noteworthy objects) in the shooting direction. In this aspect, an object is displayed in the image to enable the user to recognize the shooting direction or the projection direction corresponding to the shooting direction, so that the user viewing the image with the HMD can recognize the direction that is believed to be noteworthy. In particular, when the projection range (full field of view) of the image projected into the virtual space is wide, it is difficult to determine the center direction of the image, so applying this configuration is effective.

[0172] 3) In one aspect of the present invention, in the aspect described in 1) or 2) above, the computer is further made to function as a detection information acquisition unit (34) that acquires detection information regarding the orientation of the head-mounted display (10), a field of view direction change unit (35) that changes the field of view direction (SD) based on the detection information acquired by the detection information acquisition unit (34), and a projection direction information display unit (36) that displays information suggesting the projection direction (TD) of the image projected into the virtual space (V) in the field of view range (SR) when the image is not displayed in the field of view range (SR).

[0173] According to the aspect described in 3) above, the viewing direction and the viewing area determined by the viewing direction change depending on the orientation of the HMD (i.e., the orientation of the head of the user wearing the HMD). Furthermore, the projection direction of the played-back image changes based on the shooting direction data included in the shooting data. This can result in a large difference between the viewing direction and the projection direction, and in some cases, no image is displayed in the viewing area. This can cause the user to lose track of the projection direction of the image. In this regard, in this aspect, when an image is no longer displayed in the viewing area, information that enables the user to recognize the projection direction is displayed in the viewing area, making it easy to see in which direction the HMD (head) should be pointed so that the image will be displayed in the viewing area.

[0174] 4) In one aspect of the present invention, in any of the aspects described above in 1) to 3), the projection unit (32) limits the rotation of the projection direction (TD) with respect to at least some of the rotation directions among the yaw direction, pitch direction, and roll direction.

[0175] According to the aspect described in 4) above, by limiting the rotation of at least some of the projection directions among the yaw direction, pitch direction, and roll direction, excessive strain on the user's neck and eyes when viewing with an HMD can be reduced.

[0176] 5) In one aspect of the present invention, in the aspect described in 4) above, the projection unit (32) does not reflect some of the rotational direction components among the yaw direction, pitch direction, and roll direction components corresponding to the shooting direction (XC) in the projection direction (TD).

[0177] According to the aspect described in 5) above, the change in the projection direction is limited to changes in only one or two of the rotational directions of the yaw direction, pitch direction, and roll direction, thereby reducing the strain on the user's neck and eyes.

[0178] 6) In one aspect of the present invention, in the aspect described in 5) above, the projection unit (32) does not reflect the pitch direction component corresponding to the shooting direction (XC) in the projection direction (TD).

[0179] According to the aspect described in 6) above, the image displayed on the display unit of the HMD does not move in the pitch direction (up and down direction) relative to the HMD (i.e., relative to the user's head and eyes), but is limited to movement in the horizontal direction (yaw direction). Generally, when a user moves their neck or line of sight, the pitch direction (up and down direction) places a greater burden on the user than the yaw direction (left and right direction). Therefore, by limiting the movement of the image in the pitch direction according to this aspect, the strain on the user's neck and eyes can be effectively reduced.

[0180] 7) In one aspect of the present invention, in the aspect described in 5) or 6) above, the projection unit (32) does not reflect the roll direction component corresponding to the shooting direction (XC) in the projection direction (TD).

[0181] According to the aspect described in 7) above, the roll direction component corresponding to the shooting direction is not reflected in the projection direction, so there is a change in the difference between the direction of gravity in the real world relative to the HMD (i.e., relative to the user's head) and the direction of gravity in the image. This makes it possible to display images that are full of dynamism and impact.

[0182] 8) In one aspect of the present invention, in any of the aspects described in 4) to 7) above, the projection unit (32) limits the rotation of the projection direction (TD) for at least some of the rotation directions among the yaw direction, the pitch direction, and the roll direction so that the rotation angle in the projection direction (TD) from a reference direction does not exceed a predetermined threshold value.

[0183] According to the aspect described in 8) above, the rotation of the projection direction is limited by a threshold value for at least some of the rotation directions of the yaw direction, pitch direction, and roll direction (for example, a threshold value of 90 degrees left and right in the yaw direction and 60 degrees up and down in the pitch direction from the reference direction). This reduces the strain on the user's neck and eyes.

[0184] 9) In one aspect of the present invention, in the aspect described in 8) above, the projection unit (32) projects the image in the projection direction (TD) corresponding to the threshold value when the rotation angle from the reference direction of at least some of the rotational direction components of the yaw direction, pitch direction, and roll direction components corresponding to the shooting direction (XC) exceeds the threshold value.

[0185] According to the aspect described in 9) above, when the rotation angle (change from the reference direction) of the shooting direction (or the projection direction corresponding to the shooting direction) from the reference direction exceeds a threshold value for at least some of the rotation directions of the yaw direction, pitch direction, and roll direction, the image is projected in the projection direction corresponding to the threshold value. As a result, if the change from the reference direction of the shooting direction is within the threshold value, the projection direction of the image changes according to the shooting direction, allowing for the playback of images with a higher sense of reality and presence than conventional images. Furthermore, when the rotation angle of the shooting direction from the reference direction exceeds the threshold value, the image is projected into the virtual space while remaining in the projection direction corresponding to the threshold value, eliminating the need for the user to move their neck or eyes to follow the movement of the image. As a result, while playing images with a higher sense of reality and presence than conventional images, the strain on the user's neck and eyes can be reduced by setting a threshold value to limit the movement range (projection range) of the image in the virtual space.

[0186] 10) In one aspect of the present invention, in the aspects described in 4) to 9) above, the computer is further made to function as a restriction content setting unit (37) that sets the restriction content for rotation of the projection direction (TD) based on user operation.

[0187] According to the aspect described in 10) above, the user can set the restrictions on the rotation of the projection direction by himself, enabling a high degree of freedom in video playback. For example, the user can set the settings to prioritize reducing strain on the neck and eyes (by increasing the restrictions on the rotation of the projection direction), or set the settings to prioritize highly realistic and immersive video playback (by eliminating or reducing the restrictions on the rotation of the projection direction).

[0188] 11) A program according to one aspect of the present invention causes a computer that executes control to display on a display unit (11) of a head-mounted display (10) an image of a field of view (SR) of a virtual space (V) viewed in a field of view (SD) from a virtual viewpoint (P) in the virtual space (V), to function as a shooting direction identification unit that identifies a shooting direction (XC) in the real space (R) of video data in which the real space (R) is photographed, and a projection unit (32) that controls a projection direction (TD) of the video in the virtual space (V) according to the video data, based on the shooting direction (XC) identified by the shooting direction identification unit. Here, the shooting direction identification unit can identify the shooting direction (or a change in the shooting direction) by reading "shooting direction data indicating the shooting direction of the video data in real space" that is recorded in association with the video data. For example, the shooting direction identification unit can identify the shooting direction by reading "shooting direction data" included in the shooting data stored in the storage control unit (31) described above. Alternatively, the shooting direction identification unit may identify the shooting direction (or a change in the shooting direction) from the video data by image analysis. According to the aspect described in 11) above, the same effects as those of the aspect described in 1) above can be achieved.

[0189] 12) A display control device (20) according to one aspect of the present invention executes control to display on a display unit (11) of a head-mounted display (10) an image of a field of view (SR) of a virtual space (V) as viewed from a virtual viewpoint (P) in the virtual space in a field of view (SD). The display control device (20) includes: a storage control unit (31) that stores, in a storage device (22), image data of a real space (R) captured and image capture direction data corresponding to the image data, indicating an image capture direction (XC) of the image data in the real space; and a projection unit (32) that projects an image corresponding to the image data from the virtual viewpoint (P) into the virtual space (V) based on the image capture data. The projection unit (32) controls a projection direction (TD) of the image corresponding to the image data in the virtual space (V) based on the image capture direction data corresponding to the image data. This achieves the same effect as the aspect described in 1) above.

[0190] 13) An image display system (1) according to one aspect of the present invention includes a head-mounted display (10) including a display unit (11) and a display control device (20) that controls the display unit (11) to display an image of a field of view (SR) of the virtual space (V) as seen from a virtual viewpoint (P) in a field of view (SD). The display control device (20) includes a storage control unit (31) that stores, in a storage device (22), image data obtained by capturing a real space (R) and image capture direction data corresponding to the image data, indicating a capture direction (XC) of the image data in the real space. The display control device (20) also includes a projection unit (32) that projects an image corresponding to the image data from the virtual viewpoint (P) into the virtual space (V) based on the image capture data. The projection unit (32) controls a projection direction (TD) of the image corresponding to the image data in the virtual space (V) based on the image capture direction data corresponding to the image data. This achieves the same effect as the aspect described in 1) above.

[0191] 14) A control method according to one aspect of the present invention is a control method for controlling a computer that executes control to display an image of a field of view (SR) of a virtual space (V) as viewed in a field of view (SD) from a virtual viewpoint (P) in the virtual space on a display unit (11) of a head-mounted display (10), the control method including: a storage control step (S102) for storing, in a storage device (22), image data obtained by capturing a real space (R) and image capture direction data corresponding to the image data, indicating an image capture direction (XC) of the image data in the real space; and a projection step (S108) for projecting an image corresponding to the image data from the virtual viewpoint (P) into the virtual space (V) based on the image capture data, wherein the projection step controls a projection direction (TD) of the image corresponding to the image data in the virtual space (V) based on the image capture direction data corresponding to the image data (S106, S108). This achieves the same effect as the aspect described in 1) above.

[0192] 15) An information storage medium according to one aspect of the present invention is a computer-readable information storage medium having recorded thereon the program according to any one of the aspects 1) to 11). This provides the same effects as those of the aspects 1) to 11). [Explanation of symbols]

[0193] 1...image display system, 10...HMD, 11·61...display unit, 12·62...sensor, 13...processor, 14...storage device, 20...display control device, 21·63...processor, 22·64...storage device, 23·65...operation unit, 24·66...communication unit, 30...control unit, 31...memory control unit, 32...projection unit, 33...object display unit, 34...detection information acquisition unit, 35...view direction change unit, 36...projection direction information display unit, 37... Restriction content setting unit, 60...HMD (information processing device), 200...camera, OB1...frame object, OB2...arrow object, BC...background, R...real space, V...virtual space, P...virtual viewpoint, SD...field of view direction, SR...field of view range, XC...shooting direction, XC0...reference shooting direction, TD...projection direction, TD0...reference projection direction, TR...projection range, GD1...gravity direction of image, GD2...gravity direction relative to HMD, G10 to G14...screen

Claims

1. a computer that executes control to display an image of a field of view range of the virtual space seen in a field of view direction from a virtual viewpoint in the virtual space on a display unit of a head-mounted display; a storage control unit that stores, in a storage device, image data including image data of a real space and image capture direction data that indicates an image capture direction of the image data in the real space and corresponds to the image data; a projection unit that projects an image corresponding to the image data from the virtual viewpoint into the virtual space based on the shooting data; and make it work, The projection unit controls a projection direction of the image in the virtual space according to the image data, based on the image capturing direction data corresponding to the image data. program.

2. and causing the computer to further function as an object display unit that displays, within the image corresponding to the image data projected into the virtual space, an object for enabling recognition of the shooting direction or the projection direction of the shooting direction data corresponding to the image data. The program according to claim 1.

3. a detection information acquisition unit that acquires detection information regarding the orientation of the head mounted display; a view direction change unit that changes the view direction based on the detection information acquired by the detection information acquisition unit; a projection direction information display unit that displays, when the image projected into the virtual space is not displayed in the field of view, information suggesting the projection direction of the image in the field of view; and further functioning the computer. The program according to claim 1.

4. The projection unit limits the rotation of the projection direction with respect to at least some of the rotation directions of a yaw direction, a pitch direction, and a roll direction. The program according to claim 1.

5. The projection unit does not reflect a part of the rotation direction components among the yaw direction, the pitch direction, and the roll direction components corresponding to the imaging direction in the projection direction. The program according to claim 4.

6. The projection unit does not reflect the pitch direction component corresponding to the shooting direction in the projection direction. The program according to claim 5.

7. The projection unit does not reflect the component of the roll direction corresponding to the shooting direction in the projection direction. The program according to claim 5.

8. The projection unit limits the rotation of the projection direction so that a rotation angle of the projection direction from a reference direction does not exceed a predetermined threshold value for at least some of the rotation directions among the yaw direction, the pitch direction, and the roll direction. The program according to claim 4.

9. When the rotation angle from the reference direction of at least a part of the rotation direction components among the yaw direction, the pitch direction, and the roll direction components corresponding to the imaging direction exceeds the threshold, the projection unit projects the image in the projection direction corresponding to the threshold. The program according to claim 8.

10. The computer is further caused to function as a restriction setting unit that sets restrictions on the rotation of the projection direction based on a user operation. The program according to claim 4.

11. A display control device that performs control to display an image of a field of view range of a virtual space viewed in a field of view direction from a virtual viewpoint in the virtual space on a display unit of a head-mounted display, a storage control unit that stores, in a storage device, image data including image data of a real space and image capture direction data that indicates an image capture direction of the image data in the real space and corresponds to the image data; a projection unit that projects an image corresponding to the image data from the virtual viewpoint into the virtual space based on the shooting data; Including, The projection unit controls a projection direction of the image in the virtual space according to the image data, based on the image capturing direction data corresponding to the image data. Display control device.

12. An image display system comprising: a head-mounted display including a display unit; and a display control device that executes control to display on the display unit an image of a field of view of the virtual space as seen in a field of view direction from a virtual viewpoint in the virtual space, The display control device a storage control unit that stores, in a storage device, image data including image data of a real space and image capture direction data that indicates an image capture direction of the image data in the real space and corresponds to the image data; a projection unit that projects an image corresponding to the image data from the virtual viewpoint into the virtual space based on the shooting data; Including, The projection unit controls a projection direction of the image in the virtual space according to the image data, based on the image capturing direction data corresponding to the image data. Image display system.

13. A control method for controlling a computer that executes control to display an image of a field of view range of a virtual space viewed in a field of view direction from a virtual viewpoint in the virtual space on a display unit of a head-mounted display, comprising: a storage control step of storing, in a storage device, image data including image data of a real space captured and image capture direction data corresponding to the image data and indicating an image capture direction of the image data in the real space; a projection step of projecting an image corresponding to the image data from the virtual viewpoint into the virtual space based on the shooting data; Including, The projection step controls a projection direction of the image in the virtual space according to the image data, based on the image capturing direction data corresponding to the image data. Control method.

Images