Program, display control device, image display system, control method, and recording medium

Abstract

In the present invention, a storage control unit causes a storage unit to store captured image data, including video data captured of a real space and image capture direction data corresponding to the video data and indicating the image capture direction of the video data in the real space. A projection unit projects a video corresponding to the video data from a virtual viewpoint into a virtual space on the basis of the captured image data. The projection unit also controls the projection direction of the video corresponding to the video data within the virtual space on the basis of the image capture direction data corresponding to the video data.

Description

Program, display control device, image display system, control method, and recording medium

[0001] This application claims priority based on Japanese Patent Application No. 2024-082942 filed on May 21, 2024 with the Japan Patent Office, the entire contents of which are incorporated herein by reference. The present invention relates to a program, a display control device, an image display system, a control method, and a recording medium.

[0002] In recent years, head-mounted displays (HMDs) have come into wide use. In some cases, live-action video data of a real space, which is a space in the real world, is stored in a recording medium, and the video data is played back and displayed on an HMD.

[0003] JP 2016-115122 A

[0004] When live-action video data is played back, the live-action video displayed on the HMD is always displayed in the same direction. Here, "same direction" refers to the same direction relative to the real space viewed by the user wearing the HMD (e.g., always facing forward), or the same direction relative to the HMD screen (e.g., always the reference direction of the HMD). For example, if the reference viewing direction of the HMD is set to the direction in which the user wearing the HMD is facing forward, the center of the live-action video is always displayed fixedly in the reference viewing direction. Therefore, when live-action video data is displayed on the HMD, the displayed video may lack a sense of realism and presence. For example, consider video data captured in a scene in which a subject, such as a car, passes in front of the camera in real space, with the camera orientation changed to match the subject's movement so that the subject is displayed in the center of the video. When this video data is played back and displayed on the HMD, even though the real subject was moving during the capture, the subject is displayed in the same direction relative to the viewing direction of the HMD (the reference viewing direction), resulting in a video that lacks a sense of realism and presence.

[0005] Therefore, one of the objects of the present invention is to display on an HMD an image that is more realistic and present than conventional images when live video data is reproduced.

[0006] A program according to one aspect of the present invention causes a computer that executes control to display an image of the field of view of a virtual space viewed in the field of view direction from a virtual viewpoint within the virtual space on a display unit of a head-mounted display to function as: a memory control unit that stores, in a memory device, shooting data including video data of a real space photographed and shooting direction data corresponding to the video data that indicates the shooting direction of the video data in the real space; and a projection unit that projects an image corresponding to the video data from the virtual viewpoint into the virtual space based on the shooting data, and the projection unit controls the projection direction of the image corresponding to the video data within the virtual space based on the shooting direction data corresponding to the video data.

[0007] A display control device according to one aspect of the present invention is a display control device that performs control to display an image of the field of view of a virtual space as viewed in the field of view from a virtual viewpoint within the virtual space on a display unit of a head-mounted display, and includes: a memory control unit that stores shooting data in a memory device, the shooting data including video data of a real space and shooting direction data corresponding to the video data and indicating the shooting direction of the video data in the real space; and a projection unit that projects an image corresponding to the video data from the virtual viewpoint into the virtual space based on the shooting data, and the projection unit controls the projection direction of the image corresponding to the video data within the virtual space based on the shooting direction data corresponding to the video data.

[0008] An image display system according to one aspect of the present invention is an image display system comprising a head-mounted display including a display unit, and a display control device that executes control to cause the display unit to display an image of the field of view of the virtual space as viewed in the field of view from a virtual viewpoint within the virtual space, wherein the display control device includes a memory control unit that stores shooting data in a memory device, the shooting data including video data of a real space and shooting direction data corresponding to the video data that indicates the shooting direction of the video data in the real space, and a projection unit that projects an image corresponding to the video data from the virtual viewpoint into the virtual space based on the shooting data, and the projection unit controls the projection direction of the image corresponding to the video data within the virtual space based on the shooting direction data corresponding to the video data.

[0009] A control method according to one aspect of the present invention is a control method for controlling a computer that executes control to display an image of the field of view of a virtual space viewed in the field of view direction from a virtual viewpoint within the virtual space on a display unit of a head-mounted display, and includes a storage control step of storing, in a storage device, shooting data including video data of a real space photographed and shooting direction data corresponding to the video data that indicates the shooting direction of the video data in the real space, and a projection step of projecting an image corresponding to the video data from the virtual viewpoint into the virtual space based on the shooting data, wherein the projection step controls the projection direction of the image corresponding to the video data within the virtual space based on the shooting direction data corresponding to the video data.

[0010] 1 is a schematic block diagram showing an example of a hardware configuration of an image display system according to an embodiment of the present invention. FIG. 1 is a diagram showing definitions of directions in a virtual space. FIG. 2 is a diagram for explaining a field of view image of a field of view range viewed in a field of view direction from a virtual viewpoint in a virtual space. FIG. 3 is a conceptual diagram showing an example in which an image obtained by playing back video data captured in a real space R is projected onto a virtual space V. FIG. 4 is a conceptual diagram showing an example in which a projection direction TD of the captured video in a virtual space V changes according to a change in a shooting direction XC during shooting. FIG. 5 is a conceptual diagram showing an example of a relationship between a real space R at the time of capturing video data and a virtual space V at the time of playing back the video data. FIG. 6 is a diagram showing an example of a time series change in the projection direction TD in a comparative example in which shooting direction data is not used. FIG. 7 is a diagram showing an example of a time series change in the projection direction TD in this embodiment in which shooting direction data is used. FIG. 8 is a schematic functional block diagram showing an example of a functional configuration of a display control device. FIG. 9 is a diagram showing an example of shooting direction data. FIG. 10 is a diagram showing another example of shooting direction data. FIG. 11 is a flowchart showing an example of processing by a display control device. FIG. 12 is a diagram showing an example of a screen displayed on a display unit of an HMD. FIG. 13 is a diagram showing an example of a screen displayed on a display unit of an HMD. 17 is a diagram showing an example of a screen displayed on the display unit of the HMD when an image projected into the virtual space is not displayed in the field of view range. FIG. 18 is a diagram showing a positional relationship in the virtual space corresponding to the screen shown in FIG. 16. FIG. 19 is a diagram showing an example of a screen displayed on the display unit of the HMD when an image projected into the virtual space is not displayed in the field of view range. FIG. 19 is a flowchart showing an example of processing by a display control device. FIG. 20 is a diagram showing an example of an image projected into the virtual space when a roll direction component corresponding to the shooting direction is reflected in the projection direction. FIG. 21 is a diagram showing an example of an image projected into the virtual space when the same image data as in FIG. 20 is played back, but when a roll direction component corresponding to the shooting direction is not reflected in the projection direction. FIG. 22 is a flowchart showing an example of processing by a display control device. FIG. 23 is a diagram showing an example of a screen displayed on the display unit of the HMD. FIG. 24 is a diagram showing an example of a screen displayed on the display unit of the HMD. FIG. 25 is a diagram showing an example of an HMD.FIG. 1 is a schematic block diagram showing an example of the hardware configuration of a stand-alone HMD or an information processing device used as an HMD.

[0011] The program, display control device, image display system, and control method for executing display control of an HMD according to the present embodiment are suitable for playing back live-action video data captured in a real space. An example of an embodiment of the present invention will now be described with reference to the drawings.

[0012] 1 is a schematic block diagram showing an example of the hardware configuration of an image display system 1 according to an embodiment of the present invention. The image display system 1 includes an HMD 10 and a display control device 20 that controls display on the HMD 10.

[0013] The HMD 10 is worn on the user's head (see FIG. 2 ) and can display a field of view image of the field of view range as seen in the field of view direction from a virtual viewpoint in a virtual space. The HMD 10 is equipped with a sensor 12, such as a gyro sensor, that detects the movement and tilt of the HMD 10, and detects changes in the movement and tilt of the head of the user wearing the HMD 10, and changes the field of view direction and field of view in response to these changes. For example, the field of view image displayed on the HMD 10 changes to a field of view image of the field of view range to the right in the virtual space when the user's head is turned to the right, and changes to a field of view image of the field of view range to the upward in the virtual space when the user's head is turned up, giving the user a sense of immersion as if they were actually there.

[0014] The HMD 10 mainly includes a display unit 11, a sensor 12, a processor 13, a storage device 14, etc. The display unit 11 is a display that displays a field of view image that represents the field of view range from a virtual viewpoint in a virtual space. The virtual space is also referred to as a virtual reality (VR) space. The display unit 11 displays a stereoscopic image (an image for the right eye and an image for the left eye) using binocular parallax or a non-stereoscopic image (a 2D image) as the field of view image. The display unit 11 may be, for example, a virtual image projection display that forms a virtual image using a half mirror or the like. Alternatively, the display unit 11 may be, for example, a retinal projection display that forms a field of view image directly on the retina using the lens of the user's eye.

[0015] The sensor 12 detects posture information such as the rotation angle and tilt of the HMD 10. Detection information regarding the orientation of the HMD 10 can be obtained from the output of this sensor 12. For example, the sensor 12 is an angular velocity sensor (gyro sensor) that detects the angular velocity of an object. The sensor 12 may be a sensor that detects a change in direction, or a sensor that detects the direction itself. For example, the sensor 12 is not limited to a gyro sensor, but may be an acceleration sensor, angular acceleration sensor, tilt sensor, geomagnetic sensor, or the like, and may be realized by appropriately combining these.

[0016] The processor 13 functions as a control center that controls each component of the HMD 10. For example, the processor 13 is a central processing unit (CPU). The processor 13 may be configured to include hardware such as a graphics processing unit (GPU), a digital signal processor (DSP), or a field programmable gate array (FPGA) in addition to or instead of the CPU. The processor 13, for example, controls the supply of image signals received from the display control device 20 to the display unit 11 via an interface. The storage device 14 stores programs executed by the processor 13 and temporarily stores data and parameters processed by the processor 13. The storage device 14 includes, for example, read-only memory (ROM), random access memory (RAM), video random access memory (VRAM), and auxiliary storage device. Examples of the auxiliary storage device that can be used include non-volatile semiconductor memory, a hard disk drive, and a solid-state drive.

[0017] The HMD 10 may also include a network adapter or the like so that VR images can be input via a network. The HMD 10 may also include an audio output unit, a GPS (Global Positioning System) receiving unit, and the like.

[0018] The display control device 20 is connected to the HMD 10 wirelessly or via a wired connection so as to be able to communicate with the HMD 10, and executes display control for the HMD 10. The display control device 20 is, for example, a stationary or portable game machine. The display control device 20 may be an arcade (commercial) game machine installed in an amusement facility or the like. Alternatively, the display control device 20 may be a personal computer, a tablet computer, a smartphone, a mobile phone terminal, a PHS (Personal Handy-phone System) terminal, a personal digital assistant (PDA), a multi-function television receiver with information processing capabilities (a so-called smart TV), or the like.

[0019] The display control device 20 mainly comprises a processor 21, a storage device 22, an operation unit 23, and a communication unit 24, which are interconnected via bus lines including an address bus, a data bus, a control bus, etc. Note that an interface circuit, an image processing unit, a sound processing unit, etc. are interposed between the bus lines and each component as needed, but are not shown here.

[0020] The processor 21 interprets and executes program instructions and controls the entire display control device 20. For example, the processor 21 is a CPU. The processor 21 may be configured to include hardware such as a GPU, DSP, or FPGA in addition to or instead of the CPU. The storage device 22 includes, for example, a ROM, a RAM, a VRAM, an auxiliary storage device, etc. The ROM stores programs, data, etc. necessary for basic operational control of the display control device 20. The RAM or VRAM stores various programs and data and ensures a working area for the processor 21. The auxiliary storage device stores programs, various data, etc., and may be, for example, a non-volatile semiconductor memory, a hard disk drive, a solid state drive, etc.

[0021] The display control device 20 may also include a recording medium drive. Examples of recording medium drives include DVD-ROM drives, CD-ROM drives, hard disk drives, optical disk drives, flexible disk drives, silicon disk drives, and cassette medium readers. In this case, the recording medium may include DVD-ROMs, CD-ROMs, hard disks, optical disks, flexible disks, and semiconductor memories. The recording medium drive reads image data, audio data, and program data from the recording medium and supplies the read data to RAM or the like of the storage device 22 via a decoder.

[0022] For example, shooting data including video data previously captured (photographed) in real space, and programs such as games that include the shooting data, are stored in a storage device 22 (auxiliary storage device, etc.), or are read from a recording medium drive and imported into the display control device 20.

[0023] The operation unit 23 is used by the user to input various operation commands to the display control device 20. For example, the user performs operations to view played video or to operate a game that includes video. Examples of the operation unit 23 include a position input unit with a touch interface (such as a component of a touch panel), physical buttons, a controller, an analog stick, a keyboard, a pointing device, etc. The operation unit 23 may also be configured as an operation unit that can accept voice input by identifying voice input from a voice input unit such as a microphone. The operation unit 23 may also be configured to perform operations through user gestures.

[0024] The communication unit 24 includes a communication interface (not shown) and has a communication control function for communicating data when playing back video, playing games, etc. Here, the communication control function for data communication includes, for example, an Internet connection function, a wireless LAN (Local Area Network) connection function, and a short-range wireless communication function using a predetermined frequency band (e.g., the 2.4 GHz frequency band). The communication unit 24 transmits a connection signal for connecting the display control device 20 to a network based on a command from the processor 21, and also receives information transmitted from a communication partner and supplies the information to the processor 21. For example, the communication unit 24 can receive live video from a remote location distributed via a network and display the live video on the HMD 10 in approximately real time.

[0025] The display control device 20 may also be connected to a tracking unit that detects the position of the HMD 10 (or the position of the head of the user wearing the HMD 10). For example, the HMD 10 includes multiple light sources (LEDs, etc.) for tracking, and the tracking unit includes an imaging unit for imaging the HMD 10. The tracking unit, which is fixed at a predetermined position, identifies the position of the HMD 10 based on the positions of the multiple light source units that appear in an image captured by the imaging unit. The display control device 20 may also include a display unit such as a liquid crystal display or an organic EL (Electro-Luminescence) display.

[0026] 2. Directions in Virtual Space (VR Space) Figure 2 is a diagram showing definitions of directions in virtual space (VR space) according to this embodiment. In this embodiment, the Z axis is the vertical direction in which a user wearing an HMD 10, which displays a field of view image of a field of view range seen in the field of view direction from a virtual viewpoint in the virtual space, stands upright (see Figure 1). The X axis is the horizontal axis perpendicular to the Z axis and is the line of sight of a user facing forward (the reference direction for the virtual space), and the Y axis is the horizontal axis perpendicular to the Z axis and the X axis.

[0027] For example, when the image display system 1 is started, the visual axis direction of the HMD 10 (the direction perpendicular to the display surface of the display unit 11 of the HMD 10) is set as the reference direction for the virtual space. Also, for example, the reference direction for the virtual space may be adjusted by a user wearing the HMD 10 on their head performing a predetermined operation (such as a reset operation in the front direction).

[0028] Here, a change in the rotation direction around the Z axis is called a change in the yaw direction (left-right direction), a change in the rotation direction around the Y axis is called a change in the pitch direction (up-down direction), and a change in the rotation direction around the X axis is called a change in the roll direction. For example, the above-mentioned sensor 12 of the HMD 10 detects the angular velocity or angular acceleration in the rotation direction of each axis (yaw direction, pitch direction, and roll direction). Note that a change in the yaw direction is sometimes called a change in the left-right direction, and a change in the pitch direction is sometimes called a change in the up-down direction.

[0029] FIG. 3 is a diagram illustrating a field of view image of a field of view range viewed in the field of view direction from a virtual viewpoint in a virtual space according to this embodiment. In this diagram, a virtual viewpoint P (user's virtual viewpoint) in the virtual space V is the intersection (origin) of the X-axis, Y-axis, and Z-axis. For example, assume that the user is facing forward and the direction of the user's line of sight relative to the virtual space V is along the X-axis (reference direction). In this case, the field of view range viewed in the field of view direction from the virtual viewpoint P in the virtual space V is a range determined by a yaw angle α (the interior angle between dashed lines a and b, and the interior angle between dashed lines c and d) and a pitch angle β (the interior angle between dashed lines a and d, and the interior angle between dashed lines b and c) centered on the reference direction (X-axis direction). Here, the yaw angle α is the horizontal field of view, and the pitch angle β is the vertical field of view, which are preset as the field of view angles of the field of view image displayed on the HMD 10 in the image display system 1. For example, the horizontal field of view of the HMD 10 is 100 degrees.

[0030] For example, when the head of a user wearing the HMD 10 changes in the pitch direction or yaw direction, a change in the orientation and posture of the HMD 10 (a change in the visual axis direction of the HMD 10) is detected by the sensor 12 or the like. Based on the detection information by the sensor 12 or the like, the processor 21 of the display control device 20 determines that the line of sight direction with respect to the virtual space V has changed from the X-axis direction (reference direction) to the pitch direction or the yaw direction. By changing the field of view direction in accordance with this change in the line of sight direction with respect to the virtual space V, the range of the field of view image (field of view range) displayed on the display unit 11 is changed. Similarly, when the head of a user wearing the HMD 10 changes in the roll direction, the change is detected by the sensor 12 or the like, and the range of the field of view image (field of view range) displayed on the display unit 11 rotates in the roll direction while the line of sight direction remains in the X-axis direction. In this way, the range of the field of view image displayed on the display unit 11 is changed in accordance with the orientation (posture) of the HMD 10.

[0031] Furthermore, display objects other than the reproduced video, such as various objects, lines, symbols, and characters, can be placed as needed within the virtual space V. For example, an "object for enabling recognition of the shooting direction" or an "arrow object," which will be described later, may be placed within the virtual space V and displayed as part of the field of view image.

[0032] When displaying a stereoscopic image using binocular parallax, there are viewing directions corresponding to the virtual viewpoints for the right eye and the left eye, and the field of view image for the right eye and the field of view image for the left eye for each viewing direction are displayed on the display unit 11 of the HMD 10.

[0033] 3. Display Control In the image display system 1 according to the present embodiment, captured image data including real-life video data captured in a real space is used for playback. The outline of the display control executed by the image display system 1 or the display control device 20 is as follows.

[0034] In this embodiment, the shooting direction when shooting video is recorded, and when playing back the video, the video is played back in a projection direction based on the shooting direction. That is, shooting direction data indicating the shooting direction when shooting real-life video data is recorded in synchronization with the video data. That is, the shooting data includes video data of a real space and shooting direction data corresponding to the video data indicating the shooting direction of the video data in real space. Then, when playing back the shooting data including the video data, control is performed to change the projection direction of the video in the virtual space based on the shooting direction data corresponding to the video data. That is, in the virtual space, an image based on the video data is projected and displayed in a projection direction based on the shooting direction data corresponding to the video data. A specific example will be described below.

[0035] 4 is a diagram conceptually illustrating an example in which an image obtained by playing back video data captured in real space R is projected onto virtual space V. The diagram illustrating real space R in the lower part of FIG. 4 is a plan view of camera 200 capturing the real space R from above, and the diagram illustrating virtual space V in the upper part of the same figure conceptually illustrates an XY cross section of virtual space V. Note that in FIGS. 5 to 8 and 17 described below, the diagrams illustrating virtual space V also conceptually illustrate the XY cross section of virtual space V.

[0036] A 360-degree spherical virtual screen (with a circular cross section) centered on a virtual viewpoint P is set in the virtual space V, and an image produced by playing back video data is projected from the virtual viewpoint P into the virtual space V. That is, the image produced by playing back video data and projecting it into the virtual space V is placed (displayed) at the position of the virtual screen in the virtual space V. By changing the projection direction of the image projected from the virtual viewpoint P into the virtual space V, the projection range of the image in the virtual space V (the range of the image placed in the virtual space V) changes (see FIG. 5 ).

[0037] In this embodiment, the horizontal angle of view θH1 of the camera 200 capturing the real space R is set to 150 degrees. The capturing direction XC of the camera 200 in the real space R is the visual axis direction of the camera 200 (≒ the direction of the lens center). For example, when the camera 200 is started up, the visual axis direction of the camera 200 is set as the reference capturing direction XC0 with respect to the real space R. Furthermore, for example, after the camera 200 is started up, the user may perform a predetermined operation to set the reference capturing direction XC0 with respect to the real space R. As illustrated in FIG. 4 , an image reproduced from "video data captured by the camera 200 in the reference capturing direction XC0 in the real space R" is projected in the virtual space V with the direction of the X axis, which is the reference direction, as the projection direction. In other words, the reference capturing direction XC0 in the real space R when the video data was captured corresponds to the reference direction (reference projection direction TD0) in the virtual space V when the video data was reproduced.

[0038] The projection range TR (image display range) in virtual space V of an image corresponding to the image data is based on the angle of view of camera 200 that captured the image data. For example, in the case of real-life image data captured by camera 200 with a horizontal angle of view θH1 (e.g., 150 degrees) and a vertical angle of view θV1 (e.g., 120 degrees), the projection range TR of the image in virtual space V is basically an area having a horizontal viewing angle θH2 (= θH1) and a vertical viewing angle θV2 (= θV1). In this embodiment, a real-life image captured by camera 200 with a horizontal angle of view of 150 degrees is projected into virtual space V within a projection range TR having a horizontal viewing angle of 150 degrees.

[0039] As a variation, the projection range TR in virtual space V of a real-life image captured by a camera with a horizontal angle of view θH1 and a vertical angle of view θV1 may be narrowed to a horizontal viewing angle θH2 (<θH1) and a vertical viewing angle θV2 (<θV1) when projecting the real-life image into virtual space V. For example, the projection range TR of the image may be narrowed when projecting (placing) the real-life image into virtual space V by not using areas near the edges of the image (areas far from the visual axis of camera 200), where distortion due to distortion aberration in the optical system of camera 200 is likely to occur.

[0040] Here, a case where the imaging direction XC of the camera 200 in Fig. 4 is rotated to the right (yaw direction) is shown in Fig. 5. Fig. 5 is a conceptual diagram showing an example in which the projection direction TD of the image captured by the camera 200 in the virtual space V also changes in accordance with a change in the imaging direction XC when the camera 200 captures an image.

[0041] In the example of FIG. 4 , in the case of an image captured by camera 200 in reference shooting direction XC0, the projection direction TD in virtual space V is reference projection direction TD0 = X-axis direction (reference direction). The horizontal projection range TR of the image in virtual space V is "-75 degrees to 75 degrees," with the X-axis direction being 0 degrees. Meanwhile, in the example of FIG. 5 , the shooting direction XC of camera 200 is rotated 25 degrees to the right (-25 degrees in the yaw direction) from the reference shooting direction XC0 in FIG. 4 . In the case of an image captured by camera 200 in FIG. 5 , the projection direction TD in virtual space V rotates 25 degrees to the right (-25 degrees in the yaw direction) from the reference projection direction TD0 = X-axis direction. As a result, the horizontal projection range TR of the image in virtual space V rotates 25 degrees to the right, changing to "-100 degrees to 50 degrees."

[0042] 6 is a conceptual diagram showing an example of the relationship between a real space R when video data is captured and a virtual space V when the video data is played back. When playing back captured data of the real space R, control is performed to change the projection direction TD of the video in the virtual space V based on the shooting direction data corresponding to the video data. Therefore, basically, the shooting direction XC in the real space R when the video data is captured corresponds to the projection direction TD in the virtual space V when the video data is played back. As the projection direction TD changes, the position of the projection range TR of the video in the virtual space V also changes.

[0043] On the other hand, the viewing direction SD in the virtual space V changes based on a change in the orientation of the HMD 10 (detection information by the sensor 12). As the viewing direction SD changes, the position of the viewing range SR in the virtual space V (the position of the viewing image) also changes.

[0044] Here, it is assumed that in real space R, a scene is taking place in which a car passes in front of the viewer from the left and then drives past to the right, and the camera 200 is turning from left to right to capture the image as the car moves.

[0045] FIG. 7 shows, as a comparative example of this embodiment, an example of time-series changes in the projection direction TD (and projection range TR) of the virtual space V into which the image captured by the camera 200 is projected when the imaging direction data (information on changes in the imaging direction of the camera 200) is not used. When the imaging direction data is not used, even if the imaging direction of the camera 200 in the real space R changes, the projection direction TD of the image in the virtual space V does not change from the X-axis direction, which is the reference direction. Therefore, the position of the projection range TR in the virtual space V also does not change. The center of the image projected into the virtual space V (the center of the projection range TR) is always located in the X-axis direction, which is the reference direction. Therefore, when the captured image is displayed on the HMD 10 without using the imaging direction data, the car in the image is displayed in the same direction relative to the visual axis (reference direction) of the HMD 10, even though it is actually moving, resulting in an image that lacks realism and presence.

[0046] FIG. 8 shows an example of time-series changes in the projection direction TD (and projection range TR) of the virtual space V onto which the image captured by the camera 200 is projected when using imaging direction data (information on changes in the imaging direction of the camera 200). When imaging direction data is used, the projection direction TD and projection range TR in the virtual space V also rotate in accordance with changes in the imaging direction XC. In this case, when the captured image is displayed on the HMD 10, an image of a car moving from left to right relative to the visual axis of the HMD 10 is displayed. Therefore, if a car is located to the left during playback of the video data, the user must turn the HMD 10 to the left in order to see the car (or the car will be visible to the left when the user is facing forward). By turning their head in the direction of the car, the user can clearly see the car, resulting in a more realistic and immersive image.

[0047] 9 is a schematic functional block diagram showing an example of the functional configuration of the display control device 20. The display control device 20 executes control to display, on the display unit 11 of the HMD 10, an image (field of view image) of the field of view range SR of the virtual space V as seen in a field of view direction SD from a virtual viewpoint P in the virtual space V.

[0048] Here, the "virtual space V" is a virtual (pseudo) space generated by a computer (information processing device). For example, the virtual space V is constructed in a storage device (such as a VRAM) by calculations of the computer. A field of view SR of a portion of the virtual space V is displayed on the display unit 11 of the HMD 10. The direction of the HMD 10 is linked to the real space R (real space) based on the detection results of a sensor 12 that detects the movement and tilt of the HMD 10, such as a gyro sensor.

[0049] The "viewing direction SD" is the direction of the HMD 10 in the virtual space V. In other words, the viewing direction SD is the direction that determines the "viewing range SR of the virtual space V" displayed on the HMD 10 (see FIG. 6). Therefore, the viewing direction SD is the center direction of the "image of the viewing range SR of the virtual space V" displayed on the display unit 11 of the HMD 10. The viewing direction SD may or may not match the user's line of sight. For example, in an HMD 10 with an eye-tracking function, the viewing direction SD and the line of sight may differ.

[0050] The "visible range SR" is the range of the virtual space V displayed on the HMD 10, and is the range of the virtual space V as seen from a virtual viewpoint P in the virtual space V in a visual direction SD (see FIG. 6). The visual range SR is a range of a predetermined visual angle based on the visual direction SD. The visual range SR can be set according to the visual angle of the display on the HMD 10.

[0051] Furthermore, the "image of the field of view SR" is an image displayed on the display unit 11 of the HMD 10 and is an image showing the field of view SR of the virtual space V. If an image projected into the virtual space V exists within the field of view SR, an image of the field of view SR of the virtual space V including that image will be displayed on the display unit 11 of the HMD 10. Display objects other than the image projected into the virtual space V may also be included in the "image of the field of view SR." For example, "display objects other than the projected image," such as CG (computer graphics), various objects, lines, symbols, and characters, may also be placed in the virtual space V and superimposed on the image or displayed in the virtual space V without an image. The image of the field of view SR may be a non-stereoscopic image, or may be a stereoscopic image (image for the right eye and image for the left eye) that can be viewed stereoscopically using binocular parallax. Furthermore, the image of the field of view SR is basically a moving image, but may also include still images. When the image of the field of view SR is supplied (transmitted) to, for example, the display unit 11 of the HMD 10, it may be appropriately converted into a transmission format suitable for the HMD.

[0052] Furthermore, the "HMD" may be, for example, an HMD 50 including an attachment 51 (wearing device) that can be worn on the user's head and an information processing device 52, such as a smartphone, that is attached to the attachment 51, as illustrated in FIG. 25 . In this case, a display unit 61 of the information processing device 52 serves as the display unit of the HMD. The "HMD" may also be a so-called standalone HMD in which a display control function is integrated with the HMD. The "HMD" may also be a goggle-type or eyeglass-type HMD that can be worn on the head. The "HMD" is not limited to an HMD in the strict sense, but may also include, for example, headphones, a headset (headphones with a microphone), an eyeglass-type camera, an ear-hook camera, a camera-equipped hat, or the like that has the function of an HMD.

[0053] 9 , the display control device 20 includes a control unit 30. The control unit 30 is realized, for example, by the processor 21 executing a program stored in the storage device 22. The control unit 30 includes a storage control unit 31 and a projection unit 32. The storage control unit 31 has a function of storing, in the storage device 22, image data obtained by capturing a real space R and image capturing direction data corresponding to the image data and indicating an image capturing direction XC of the image data in the real space R. Here, the "real space R" refers to a space in the real world other than the virtual space V.

[0054] The "photography data" is information recorded when photographing the real space R. The photography data includes "video data" and "photography direction data." For example, the photography data may be included in application software such as a game.

[0055] "Video data" refers to data of real-life video captured in real space R. The format of the video data (or data format, file format, compression method, etc.) is not particularly important. For example, known formats such as MP4, AVI, WebM, and MKV can be applied. Alternatively, a format to be developed in the future or a unique format may be applied. Audio data, subtitle data, etc. may be attached to the video data. The video data may also be referred to as imaging data, etc.

[0056] For example, the video data is obtained by capturing the real space R with a camera. Note that the camera itself is not included in the components for controlling the display of the HMD. Here, the term "camera" refers to a real camera that captures (images) the real space R. When a non-stereoscopic image is displayed on the HMD, the video data can be data captured by at least one camera of the real space. When a stereoscopic image utilizing binocular parallax is displayed on the HMD, the video data can be data captured by a left-eye optical system (left-eye camera) that captures an image for the left eye and a right-eye optical system (right-eye camera) that captures an image for the right eye. In this case, for example, a single camera may be provided with an optical system for the left eye and an optical system for the right eye, as in a stereo camera. Furthermore, the left-eye camera and the right-eye camera may be configured separately. Furthermore, a camera that captures a single direction and is not binocular may be used.

[0057] The "shooting direction data" is data recorded in correspondence with video data, and is information that enables identification of the shooting direction XC of the video data in real space R. For example, the shooting direction data is recorded in synchronization with the video data during shooting. For example, the shooting direction data may be obtained by detecting information about the shooting direction XC from an angular velocity sensor (gyro sensor) or the like mounted on the camera during shooting of the real space R, and recording the information in association with the video data. Alternatively, the shooting direction XC may be identified from the captured video data by known image analysis during or after shooting, and the shooting direction data may be recorded in association with the captured video data.

[0058] For example, the shooting direction data is recorded as information that can identify the shooting direction XC of the video data in the real space R, corresponding to each frame of the video data. FIG. 10 is a diagram showing an example of the shooting direction data. As shown in FIG. 10, the shooting direction data can be a table including fields for "frame" and "shooting direction." The "frame" field stores identification information (e.g., a frame number) for uniquely identifying each frame of the video data. The "shooting direction" field includes fields for "yaw," "pitch," and "roll." The "yaw" field stores information on the yaw direction component corresponding to the shooting direction XC of the target frame of the video data. Similarly, the "pitch" and "roll" fields store information on the pitch direction and roll direction components corresponding to the shooting direction XC of the target frame of the video data, respectively. As shown in FIG. 10, the shooting direction data may be recorded in association with each frame of the video data (for all frames of the video data).

[0059] Alternatively, in order to compress the amount of shooting direction data, shooting direction data may be recorded once for multiple frames of video data (e.g., two frames, four frames, etc.). Figure 11 shows an example in which shooting direction data is recorded in association with every two frames of video data. When shooting direction data is recorded in association with video data for multiple frames, for example, for frames for which corresponding shooting direction data is not recorded, shooting direction data interpolated by linear interpolation or the like can be used when playing back the video data. A known algorithm can be used for the interpolation.

[0060] The shooting direction data may include data capable of identifying at least one rotational component among the yaw, pitch, and roll components corresponding to the shooting direction XC. Furthermore, the shooting direction data may be any information capable of identifying the shooting direction XC. For example, the shooting direction data may be information on the absolute rotation angles of the yaw, pitch, and roll components corresponding to the shooting direction XC, as exemplified in FIG. 10 or FIG. 11 . Alternatively, the shooting direction data may be information on the amount of change from the shooting direction XC of the previous frame. When shooting direction data is recorded in association with video data for each of multiple frames, the shooting direction data may be information on the amount of change from the shooting direction XC of the frame in which the shooting direction data was previously recorded.

[0061] There may also be video data in which the shooting direction XC remains unchanged for a while (i.e., the video is shot without moving the camera position for a while) and then changes. In such video data, shooting direction data may not be recorded for frames in which the shooting direction XC remains unchanged from that of the previous frame (i.e., frames in which the shooting direction XC remains unchanged). Then, shooting direction data may be recorded for each frame or for every several frames, focusing only on frames in which a change occurs from the shooting direction XC of the previous frame. In this case, shooting direction data is not recorded during a period in which consecutive frames remain unchanged from the shooting direction XC of the previous frame, thereby reducing the amount of data.

[0062] For example, the shooting direction data can be recorded as header information of the shooting data (or video data). For example, when the MP4 file format is adopted, the shooting direction data can be recorded in a "moov area in which metadata can be recorded" included in the MP4 box structure. Alternatively, the shooting data and the video data can be stored in separate files, with the shooting data being recorded as information associated with the video data.

[0063] The "imaging direction XC" is a direction corresponding to the visual axis direction of the camera in the real space R. In other words, the imaging direction XC is a direction corresponding to the orientation of the camera that captures the real space R, and is the front direction (direction pointing toward the subject) of the optical axis of the imaging optical system of the camera. Furthermore, the imaging direction XC is a direction toward approximately the center of the captured video (captured image).

[0064] The projection unit 32 has a function of projecting an image corresponding to the image data from a virtual viewpoint P into a virtual space V based on the shooting data. Here, the "image corresponding to the image data" refers to an image obtained by playing back the image data and projected into the virtual space V.

[0065] The projection unit 32 has a function of controlling the projection direction of an image corresponding to the image data in the virtual space V, based on the shooting direction data corresponding to the image data. Here, the "projection direction TD" is the direction in the virtual space V of the image that is displayed (arranged) by being projected into the virtual space V. The projection direction TD is also the direction of the center of the image projected into the virtual space V.

[0066] Furthermore, "controlling the projection direction TD of the image corresponding to the video data in the virtual space V based on the shooting direction data corresponding to the video data" means changing the projection direction TD of the image corresponding to the video data in the virtual space V based on the shooting direction XC of the shooting direction data corresponding to the video data (see Figures 4 to 6 and Figure 8).

[0067] According to the configuration of the present embodiment described above, even when the HMD 10 plays back video within a limited field of view SR, it is possible to play back video with a higher sense of reality and presence than conventional systems. That is, conventionally, the shooting direction XC of the video data in the real space R is not taken into consideration when playing back the video data. Therefore, as illustrated in FIG. 7 , even when the video data has a changing shooting direction XC in the real space R, the projection direction TD of the video in the virtual space V does not change, and the projection range TR of the video is fixed within the virtual space V. Therefore, the HMD 10 displays video with a lack of realism and presence. In contrast, according to the configuration of the present embodiment, when video data with a changing shooting direction XC is played back, the projection direction TD and projection range TR of the video in the virtual space V change based on the shooting direction data, as illustrated in FIG. 8 . Therefore, for example, when video data with a changing shooting direction XC is played back to track a moving subject such as a car, a video with a high sense of reality and presence is played back on the HMD 10.

[0068] [5. Processing] Next, an example of processing executed by the display control device 20 of this embodiment will be described below. Fig. 12 is a flowchart showing an example of display control processing of the HMD 10 by the display control device 20. The processing described below is realized by the control unit 30 (processor 21 of the display control device 20) executing a program stored in the storage device 22 (the same applies to the processing referring to the flowcharts of Figs. 19 and 22).

[0069] In this embodiment, a display control process will be described for the case where "shooting data" including the above-mentioned video data in which the real space R has been shot and the above-mentioned shooting direction data indicating the shooting direction XC of the video data in the real space R is played back and displayed on the HMD 10. When playing back shooting data including video data for the left eye and the right eye that allows stereoscopic viewing using binocular parallax, the following process is executed for each of the left eye and the right eye. When the user inputs a predetermined operation from the operation unit 23 to start the display control process of the HMD 10, the display control device 20 starts the display control process.

[0070] In S100, the control unit 30 initializes the orientation of the HMD 10 and sets a reference direction in the virtual space V. As a result, for example, as shown in Figures 2 and 3, the reference direction (X axis) of the virtual space V is initially set corresponding to the visual axis direction of the HMD 10 when the user faces forward.

[0071] In S102, the control unit 30 reads the shooting data to be played back and stores it in the storage device 22. For example, the shooting data is stored on a recording medium, and the control unit 30 reads the shooting data from the recording medium via a recording medium drive and stores it in the storage device 22. Alternatively, the control unit 30 may download the shooting data to be played back, for example, via the Internet, and store it in the storage device 22. Note that if the shooting data has already been stored in the storage device 22 before S100, the processing of S102 can be omitted.

[0072] In S104, the control unit 30 initializes the projection direction TD of the image in the virtual space V when (before) playback of the captured image data including the video data starts. That is, the control unit 30 sets a reference projection direction TD0 of the image in the virtual space V. In this embodiment, the control unit 30 sets the reference projection direction TD0 of the image corresponding to the reference capturing direction XC0 in the capturing direction data to the reference direction (X axis) of the virtual space V.

[0073] As a variation, the reference projection direction TD0 of the image corresponding to the reference shooting direction XC0 in the shooting direction data may be set to a direction deviated from the reference direction of the virtual space V (X axis).

[0074] In S106, the control unit 30 specifies a projection direction TD of the image in the virtual space V based on imaging direction data that corresponds to the image data and indicates an imaging direction XC of the image data in the real space R. In S108, the control unit 30 projects the image corresponding to the image data in the virtual space V in the projection direction specified in S106.

[0075] In S110, the control unit 30 acquires detection information regarding the orientation of the HMD 10 from the sensor 12 (e.g., an angular velocity sensor) of the HMD 10. In S112, the control unit 30 sets (changes) the viewing direction SD based on the detection information regarding the orientation of the HMD 10 acquired in S110.

[0076] In S114, the control unit 30 generates an image of the field of view range SR according to the field of view direction SD and outputs it to the HMD 10. The processes of S106 to S114 are repeatedly executed for each frame of the video until the display ends (YES in S116). As described above, when shooting data including video data in which the shooting direction XC changes is played back, the projection direction TD and projection range TR of the video in the virtual space V change based on the shooting direction data corresponding to the video data (see FIGS. 5 and 8), and a highly realistic and immersive video is played back.

[0077] [6. Aspects of Displaying Projection Direction] In this embodiment, the projection direction TD of the image in the virtual space V changes based on the shooting direction data, and therefore the projection range TR of the image may change (see FIGS. 5 and 8). Because the virtual space V that can be displayed on the HMD 10 is a wide space that extends 360 degrees left and right and up and down, it may happen that the user becomes confused about the projection direction TD of the image in the virtual space V. Therefore, as shown below, display control may be performed so that the user can recognize the projection direction TD (or the corresponding shooting direction XC).

[0078] 6-1. Aspects of Displaying an Object with a Recognizable Shooting Direction or Projection Direction in an Image Typically, when capturing a real space R with the camera 200, the camera 200 is pointed in a direction that the photographer considers noteworthy (important). Therefore, it is believed that there is a lot of important information (e.g., noteworthy objects) in the shooting direction XC (the visual axis direction of the camera 200). The shooting direction XC of the image captured of the real space R corresponds to the projection direction TD of the image projected into the virtual space V and corresponds to the direction of the approximate center of the image (projection range TR of the image) projected into the virtual space V. In other words, it is believed that there is a lot of important information in the direction of the center of the image projected into the virtual space V. As described above, in this embodiment, the projection direction TD of the image in the virtual space V changes based on the shooting direction data, and therefore the projection range TR and center of the image may change (see FIGS. 5 and 8). In particular, when the projection range TR of the image projected into the virtual space V is wide, it may be difficult for a user viewing the image with the HMD 10 to determine the center of the image. Therefore, an object for making it possible to recognize the shooting direction XC or the projection direction TD corresponding to the shooting direction XC may be displayed in the image projected into the virtual space V. This will be described below.

[0079] 13 and 14 are diagrams showing examples of screens displayed on the display unit 11 of the HMD 10. The screen G10 shown in Fig. 13 displays a field of view image of the field of view range, in which a frame object OB1 is superimposed on the image to make it possible to recognize the projection direction TD (= shooting direction XC = center of the projection range TR of the image).

[0080] Note that an object for making the projection direction TD recognizable may be displayed directly at the center of the projection range TR of the image, which is the projection direction TD. However, as mentioned above, since the projection direction TD is likely to contain a lot of important information, the object may be visually distracting. Therefore, as illustrated in Figures 13 and 14, a frame object OB1 is displayed around the projection direction TD (= shooting direction XC = center of the projection range TR of the image) so that the projection direction TD and the shooting direction XC can be inferred. The frame object OB1 is composed of four L-shaped frames representing the vertices of a rectangle centered on the projection direction TD of the image (= shooting direction XC). The user can recognize that the center of the rectangle of the frame object OB1 is the direction that is likely to require attention.

[0081] The size of the rectangle of the frame object OB1 (in other words, the distance of the four frames of the frame object OB1 from the center of the projection range TR of the image) can be set arbitrarily, but it is desirable that the size of the rectangle of the frame object OB1 be smaller than the size of the image in the field of view SR.

[0082] The color of the frame object OB1 can also be set arbitrarily. Note that if the color of the image on which the frame object OB1 is superimposed and the color of the frame object OB1 are similar in color, the frame object OB1 may be difficult to recognize. Therefore, the color of the image on which the frame object OB1 is superimposed may be determined using a known image recognition process, and the color of the frame object OB1 may be changed in accordance with the color of the image. For example, the color of the frame object OB1 is basically set to a predetermined default color. Then, if it is determined that the default color makes the frame object OB1 difficult to recognize in relation to the color of the image on which the frame object OB1 is superimposed, the color of the frame object OB1 may be changed to, for example, a color complementary to the default color.

[0083] In the example of Figure 13, the field of view direction SD (= center of the field of view image) is offset from the projection direction TD of the video, so the frame object OB1 is displayed at a position offset from the center of the field of view image displayed on the screen G10 of the HMD 10. When a user looking at the screen G10 of Figure 13 rotates their head (HMD 10) toward the center of the frame object OB1, the screen G11 displayed on the display unit 11 of the HMD 10 is the field of view image exemplified in Figure 14. In the display control of the HMD 10 of this embodiment, if the projection direction TD of the video in the virtual space V changes based on the shooting direction data, the position of the frame object OB1 also changes accordingly. By rotating their head (rotating the field of view direction SD) while following the center of the frame object OB1, the user can easily make the field of view direction SD follow the projection direction TD of the video that is considered to be of interest.

[0084] 15 , the display control device 20 can be configured to include an object display unit 33. The object display unit 33 has a function of displaying an object in the image corresponding to the image data projected into the virtual space V, for enabling recognition of the shooting direction XC or the projection direction TD of the shooting direction data corresponding to the image data.

[0085] Here, the "object for enabling the user to recognize the shooting direction XC or the projection direction TD" is an object displayed in an image projected into the virtual space V, and is an object for enabling the user to recognize the shooting direction XC at the time of shooting the image. Alternatively, it is an object for enabling the user to recognize the projection direction TD corresponding to the shooting direction XC. Here, the shooting direction XC of an image is the direction of approximately the center of the image. Furthermore, the projection direction TD corresponding to the shooting direction XC is the direction of approximately the center of the image projected in the virtual space V. Therefore, in other words, the object for enabling the user to recognize the shooting direction XC or the projection direction TD, which is displayed in the image projected into the virtual space V, is an object for enabling the user to recognize the direction of approximately the center of the image.

[0086] The object displayed in the image projected into the virtual space V may be displayed directly in the projection direction TD of the image (= shooting direction XC = center of the projection range TR of the image). Alternatively, the object displayed in the image may be displayed around the projection direction TD of the image, avoiding the projection direction TD of the image, so that the projection direction TD can be inferred. For example, a frame object OB1 (see FIGS. 13 and 14 ) that is an object displayed in the image and indicates each vertex of a rectangle centered on the projection direction TD of the image is an example of an "object for making the shooting direction XC or the projection direction TD recognizable." A frame object that is not limited to a rectangle and indicates each vertex of a polygon centered on the projection direction TD of the image is an example of an "object for making the shooting direction XC or the projection direction TD recognizable." Furthermore, for example, an object that indicates the circumference (or a part of the circumference) of a circle centered on the projection direction TD of the image is an example of an "object for making the shooting direction XC or the projection direction TD recognizable."

[0087] The object is placed in the virtual space V, for example, superimposed on an image projected into the virtual space V. As a result, if the image projected into the virtual space V is present within the field of view SR, the object is superimposed on the image and displayed on the display unit 11 of the HMD 10. The object may be placed in the virtual space V and displayed superimposed on the image only when the image projected into the virtual space V is displayed within the field of view SR. Alternatively, the object may be controlled to be placed in the virtual space V regardless of whether the image projected into the virtual space V is displayed within the field of view SR.

[0088] In this configuration, an object is displayed in the image to enable the user to recognize the shooting direction XC or the projection direction TD corresponding to the shooting direction XC, so that the user viewing the image with the HMD 10 can recognize the direction that is likely to require attention. In particular, when the projection range TR of the image projected into the virtual space V is wide, it becomes difficult to determine the center direction of the image, so applying this configuration is effective.

[0089] [6-2. Aspects of Displaying Information Outside the Image That Enables Recognition of the Projection Direction] In this embodiment, the viewing direction SD in the virtual space V and the viewing range SR determined by the viewing direction SD change depending on the orientation of the HMD 10 (i.e., the orientation of the head of the user wearing the HMD 10). Furthermore, the projection direction TD of the reproduced image in the virtual space V changes based on the shooting direction data included in the shooting data. For this reason, a large difference may sometimes occur between the viewing direction SD and the projection direction TD in the virtual space V, and no image may be displayed at all in the viewing range SR. In such cases, the user may lose track of the projection direction TD of the image. In particular, when an image with a narrow viewing range SR is displayed on the HMD 10, the area outside the viewing range SR inevitably becomes wide, making it difficult for the user to determine which direction is the projection direction TD of the image (in which direction the image is being displayed). Therefore, as described below, display control may be performed in the area outside the projection range TR of the image so that the user can recognize the projection direction TD of the image.

[0090] 16 is a diagram showing an example of a screen G12 displayed on the display unit 11 of the HMD 10 when an image projected into the virtual space V is not displayed within the field of view SR. An arrow object OB2 (an example of information suggesting the projection direction of the image) is displayed on the screen G12. This suggests that the projection direction TD of the image is in the direction indicated by the arrow object OB2 (leftward in FIG. 16).

[0091] The arrow object OB2 is displayed in the field of view SR when the image projected into the virtual space V is not displayed in the field of view SR (in other words, when the difference between the projection direction TD and the field of view SD is such that the image is not displayed in the field of view SR). For example, the horizontal field of view angle of the image projection range TR is 150 degrees, and the horizontal field of view angle of the field of view SR is 100 degrees. In this case, if the difference between the projection direction TD and the field of view SD (the angle between them) in the horizontal direction is 125 degrees or more, the image will not be displayed in the field of view SR. While an example of the horizontal direction is shown here, the same applies to the vertical direction. If the difference between the projection direction TD and the field of view SD (the angle between them) in the vertical direction is greater than a predetermined value, the image will not be displayed in the field of view SR. When the image projected into the virtual space V is not displayed in the field of view SR, the control unit 300 places the arrow object OB2 indicating the projection direction TD in the center of the field of view SR of the virtual space V. That is, the arrow object OB2 is placed in the viewing direction SD.

[0092] The position where the arrow object OB2 is placed (displayed) is not limited to the center of the view range SR, but may be any position within the view range SR. The color of the arrow object OB2 may also be set arbitrarily as long as it can be recognized in relation to the background color of the screen G12.

[0093] If the user changes the orientation of the HMD 10 (head direction) in the direction indicated by the arrow object OB2, the viewing direction SD approaches the image projection direction TD, and the image is displayed in the viewing range SR in the shortest possible time. Incidentally, since the virtual space V is a 360-degree space, left and right, up and down, even if the orientation of the HMD 10 is changed in the opposite direction to the direction indicated by the arrow object OB2, the image will ultimately be displayed in the viewing range SR. The direction indicated by the arrow object OB2 here is the direction that suggests the image projection direction TD, which will result in the image being displayed in the viewing range SR in the shortest possible time.

[0094] Furthermore, as illustrated in FIG. 16 , when an image projected into the virtual space V is not displayed in the field of view SR, the color (background color) of the background BC displayed in the field of view SR may be displayed in a gradation in addition to (or instead of) the arrow object OB2. FIG. 17 is a diagram illustrating the positional relationship in the virtual space V corresponding to the screen G12 shown in FIG. 16 . The gradation of this background BC, for example, has its darkest portion in the direction opposite to the projection direction TD and becomes brighter as it approaches the projection direction TD. This allows the user to visually determine the direction of the projection direction TD even when no image is projected in the field of view SR and the background BC is displayed, based on the color gradation of the background BC. In other words, by changing the orientation of the HMD 10 (head orientation) in the direction in which the background BC becomes brighter, the field of view direction SD approaches the projection direction TD of the image, and the image is displayed in the field of view SR in the shortest possible time.

[0095] The projection direction TD and viewing direction SD of the image in the virtual space V can change not only left and right (yaw direction, horizontal direction) but also up and down (pitch direction, vertical direction). Therefore, as shown in FIG. 18, an arrow object OB2 indicating the projection direction TD may indicate a diagonal direction. In this case, the gradation of the background BC becomes brighter as it approaches the projection direction TD indicated by the arrow object OB2. An arrow object OB2 indicating the upward or downward direction may also be displayed.

[0096] As illustrated in FIG. 15 , the display control device 20 may include a detection information acquisition unit 34, a field of view direction change unit 35, and a projection direction information display unit 36. The detection information acquisition unit 34 has a function of acquiring detection information related to the orientation of the HMD 10. Here, the "detection information related to the orientation of the HMD 10" refers to detection information related to the orientation of the HMD 10 that changes when the user changes the orientation of their head while wearing the HMD 10. The "detection information" can be acquired from a detection unit such as the sensor 12 that detects the orientation or changes in orientation of the HMD 10. For example, the detection results of the sensor 12 (such as an angular velocity sensor, acceleration sensor, or geomagnetic sensor) built into the HMD 10 are an example of the "detection information related to the orientation of the HMD 10." Furthermore, the measurement results of a tracking system that captures an image of the HMD 10 using an imaging unit provided outside the HMD 10 and measures the position and orientation of the HMD 10 are an example of the "detection information related to the orientation of the HMD 10." In addition, the measurement results of a tracking system that analyzes surrounding objects captured by an imaging unit installed in the HMD 10 itself and measures the position and direction of the HMD 10 correspond to an example of ``detection information regarding the orientation of the HMD 10.''

[0097] The viewing direction change unit 35 has a function of changing the viewing direction SD based on the detection information acquired by the detection information acquisition unit 34 .

[0098] The projection direction information display unit 36 ​​has the function of displaying information suggesting the projection direction TD of the image projected into the virtual space V in the field of view SR when the image is not displayed in the field of view SR.

[0099] Here, the "information suggesting the projection direction TD of the image" refers to information that is displayed in the field of view SR when the image projected into the virtual space V is not displayed in the field of view SR, and is information such as an object that allows the user to indirectly determine the projection direction TD of the image. Since the image is not displayed in the field of view, the projection direction TD does not exist in the field of view SR, and therefore the projection direction TD itself cannot be directly indicated, but any information that allows the user to indirectly determine the projection direction TD will suffice.

[0100] The field of view direction SD (and the field of view range SR determined by the field of view direction SD) changes depending on the orientation of the HMD 10, while the projection direction TD of the image changes based on the shooting direction data included in the shooting data. In other words, the field of view direction SD and the projection direction TD change independently. As mentioned above, if a difference of a predetermined amount or more occurs between the field of view direction SD and the projection direction TD, no image will be displayed in the field of view range SR. Therefore, "when the image projected into the virtual space V is not displayed in the field of view range SR" can be said to be a case where a difference of a predetermined amount or more occurs between the field of view direction SD and the projection direction TD.

[0101] For example, an arrow object OB2 (see FIGS. 16 to 18 ) displayed in the field of view SR where no image is displayed, which indicates the approach to the projection direction TD of the image, is an example of "information suggesting the projection direction TD of the image." In this case, if the user changes the orientation of the HMD 10 (head direction) in the direction indicated by the arrow object OB2, the field of view direction SD approaches the projection direction TD, and the image is displayed in the field of view SR in the shortest time. Furthermore, text information such as "left," "right," "up," "down," "upper right," and "lower left" may be displayed instead of or in addition to the arrow object OB2, and this text information is an example of "information suggesting the projection direction TD of the image."

[0102] Furthermore, for example, the gradation of the background BC, which is displayed in the field of view SR where no image is displayed and which becomes brighter as it approaches the projection direction TD, corresponds to an example of "information suggesting the projection direction TD of the image." In this case, if the user changes the orientation of the HMD 10 (head direction) in the direction where the background BC becomes brighter, the field of view direction SD will approach the projection direction TD, and the image will be displayed in the field of view SR in the shortest time. Note that instead of a gradation of brightness difference (light and dark), a gradation of hue difference or saturation difference may also be used. The aforementioned arrow object OB2 and the gradation of the background BC may be displayed together, or only one of them may be displayed.

[0103] In this configuration, when an image projected into the virtual space V is no longer displayed in the field of view SR, information suggesting the projection direction TD of the image is displayed in the field of view SR. This allows the user to easily know in which direction to point the HMD 10 (head) so that the image will be displayed in the field of view SR.

[0104] Next, an example of a process for displaying information that allows the above-mentioned projection direction TD to be recognized will be described. Fig. 19 is a flowchart showing an example of the process of the display control device 20. The flowchart of Fig. 19 is a modified example of the flowchart of Fig. 12, and the same processes as those in Fig. 12 are assigned the same step numbers and their description will be omitted. Also, in Fig. 19, the description of S100 to S104 is omitted.

[0105] 12, after executing S100 to S112, control unit 30 determines whether the image projected into virtual space V is displayed in view range SR (S200). For example, control unit 30 determines that the image will not be displayed in view range SR when the difference between projection direction TD and view direction SD (the angle between them) is equal to or greater than a predetermined value, and determines that the image will be displayed in view range SR when the difference is less than the predetermined value.

[0106] When the control unit 30 determines that the image projected into the virtual space V is displayed within the field of view SR (YES in S200), the control unit 30 displays (places) a frame object OB1 indicating the projection direction TD within the image (S202). That is, in S202, the control unit 30 places the rectangular frame object OB1, whose center is the projection direction TD of the image, in the virtual space V so as to be superimposed on the projected image.

[0107] On the other hand, if the control unit 30 determines that the image projected into the virtual space V is not displayed in the field of view SR (NO in S200), it displays (places) an arrow object OB2 indicating the projection direction TD in the center of the field of view SR (field of view direction SD) (S204). The control unit 30 also displays the background BC displayed in the field of view SR with a gradation that indicates the projection direction TD (S206). For example, the control unit 30 places, at least in the field of view SR, a background BC formed with a gradation that becomes brighter as it approaches the projection direction TD.

[0108] As shown in FIG. 17, a background BC having a gradation that becomes brighter as it approaches the projection direction TD may be placed not only in the field of view SR but also in the entire area outside the projection range TR of the image.

[0109] After executing S202 or S206, the control unit 30 generates an image of the field of view SR according to the field of view direction SD and outputs it to the HMD 10. The processing from S106 onwards is repeatedly executed for each frame of the video until the display ends (YES in S116).

[0110] As a result, when the image projected into the virtual space V is displayed in the field of view SR, i.e., when the image played back on the HMD 10 is displayed, a frame object OB1 is displayed as shown in Fig. 13 or 14. This allows the user to recognize the direction in the image that is considered to be of interest. On the other hand, when the image projected into the virtual space V is not displayed in the field of view SR, i.e., when the image played back on the HMD 10 is not displayed, an arrow object OB2 and a gradation of the background BC are displayed as shown in Fig. 16 or 18. This allows the user to recognize the projection direction TD of the image and easily know in which direction to point the HMD 10 (head) so that the image will be displayed.

[0111] 7. Aspects of Limiting Rotation (Change) of Projection Direction in At Least Some Rotational Directions The projection unit 32 may limit the rotation of the projection direction TD in at least some rotational directions among the yaw direction, pitch direction, and roll direction. Here, "limiting the rotation of the projection direction TD" means limiting changes in the projection direction TD of an image when the image is projected into the virtual space V. "Limiting the rotation of the projection direction TD" includes preventing the projection direction TD from rotating in the yaw direction, pitch direction, or roll direction. Furthermore, "limiting the rotation of the projection direction TD" includes limiting the range of rotation (range of rotation angle) of the projection direction TD in the yaw direction, pitch direction, or roll direction using a threshold value.

[0112] According to this configuration, the rotation of the projection direction TD is limited in at least some of the yaw direction, pitch direction, and roll direction, thereby reducing excessive strain on the user's neck and eyes when viewing with the HMD 10. Various examples of limiting the rotation of the projection direction TD are described below.

[0113] [7-1. Aspects of Limiting Part of the Rotational Direction Reflected in the Projection Direction] The projection unit 32 may not reflect part of the rotational direction components of the yaw direction, pitch direction, and roll direction components corresponding to the imaging direction XC in the projection direction TD. That is, when controlling the projection direction TD of the image in the virtual space V based on the imaging direction data indicating the imaging direction XC of the image data in the real space R, the projection unit 32 has a function of not reflecting part of the rotational direction components of the yaw direction, pitch direction, and roll direction components corresponding to the imaging direction XC in the control of the projection direction TD.

[0114] With this configuration, changes in the projection direction TD of the image in the virtual space V are limited to changes in only one or two of the rotational directions of yaw, pitch, and roll, thereby reducing strain on the user's neck and eyes.

[0115] [7-1-1. Aspects of Limiting Rotation of Projection Direction in the Pitch Direction] The projection unit 32 may not reflect a pitch direction component corresponding to the shooting direction XC in the projection direction TD. That is, the projection unit 32 has a function of not reflecting a pitch direction component corresponding to the shooting direction XC in the control of the projection direction TD when controlling the projection direction TD of the image in the virtual space V based on shooting direction data indicating the shooting direction XC of the image data in the real space R.

[0116] For example, when controlling the projection direction TD based on the shooting direction data shown in Figure 10 or 11, the projection unit 32 does not reflect (use) the shooting direction data in the pitch direction (data in the "pitch" field) in controlling the projection direction TD. In this case, the shooting direction data may omit recording of the unused "pitch" field data (i.e., the shooting direction data may not include information on the pitch direction component corresponding to the shooting direction XC). This reduces the storage capacity of the shooting direction data.

[0117] According to this configuration, the image displayed on the display unit 11 of the HMD 10 does not move in the pitch direction (up and down direction) relative to the HMD 10 (i.e., relative to the user's head and eyes), but is limited to movement in the horizontal direction (yaw direction). Generally, when a user moves their neck or line of sight, the pitch direction (up and down direction) places a greater burden on them than the yaw direction (left and right direction). Therefore, by limiting the movement of the image in the pitch direction with this configuration, it is possible to effectively reduce the burden on the user's neck and eyes.

[0118] Furthermore, the direction of the head of a user wearing the HMD 10 may change depending on the user's posture. For example, if the user is sitting in a reclining chair with an inclined backrest, the user's line of sight will be directed slightly upward. Even in such a case, by limiting the rotation of the projection direction TD in the pitch direction using this configuration, the projection direction TD of the image will remain horizontal with respect to the user's eyes, reducing strain on the user's neck and eyes.

[0119] 7-1-2. Aspects of Limiting Rotation of Projection Direction in the Roll Direction] The projection unit 32 may not reflect a component in the roll direction corresponding to the shooting direction XC in the projection direction TD. That is, the projection unit 32 has a function of not reflecting a component in the roll direction corresponding to the shooting direction XC in the control of the projection direction TD when controlling the projection direction TD of the image in the virtual space V based on shooting direction data indicating the shooting direction XC of the image data in the real space R.

[0120] For example, when controlling the projection direction TD based on the shooting direction data shown in Figure 10 or 11, the projection unit 32 does not reflect (use) the shooting direction data for the roll direction (data in the "ROLL" field) in controlling the projection direction TD. In this case, the shooting direction data may omit recording of the unused "ROLL" field data (i.e., the shooting direction data may not include information on the roll direction component corresponding to the shooting direction XC). This reduces the storage capacity of the shooting direction data.

[0121] With this configuration, the roll direction component corresponding to the shooting direction XC is not reflected in the projection direction TD of the image, which causes a change in the difference between the direction of gravity in the real world relative to the HMD 10 (i.e., relative to the user's head) and the direction of gravity in the image. In this case, it may be possible to display an image that is dynamic and impactful, which may be advantageous for image presentation. A specific example is shown below.

[0122] FIG. 20 is a diagram illustrating an example of an image projected into a virtual space V when a roll direction component corresponding to the shooting direction XC is reflected in the projection direction TD. The image in FIG. 20 is an image captured in a real space R by an onboard camera 200 of a motorcycle during a motorcycle race. In this example, the motorcycle leans to the left due to a left corner, and the shooting direction XC of the camera 200 also rolls to the left. When playing back video data that leaned to the left (rolled to the left) during shooting in the real space R, as in this example, if the roll direction component corresponding to the shooting direction XC is reflected in the projection direction TD, the image display shown in FIG. 20 results. In this case, the roll rotation of the shooting direction XC is also reproduced in the image, so that the gravity direction GD1 of the image and the gravity direction GD2 relative to the HMD 10 always coincide and remain unchanged. Images in which the gravity direction GD1 of the image and the gravity direction GD2 relative to the HMD 10 always coincide and remain unchanged have the advantage of being less likely to cause so-called VR sickness. On the other hand, the image tends to lack impact because there is little change in the image. Therefore, it is possible to intentionally limit the rotation in the roll direction of the projection direction TD of the image.

[0123] 21 is a diagram showing an example of an image projected into a virtual space V when the same image data as in FIG. 20 is reproduced, but the roll direction component corresponding to the shooting direction XC is not reflected in the projection direction TD. In this way, by intentionally restricting the roll direction rotation of the projection direction TD of the image, the gravity direction GD1 of the image can be changed with respect to the gravity direction GD2 relative to the HMD 10, thereby making it possible to display an image that is dynamic and impactful.

[0124] The above-mentioned process of "not reflecting in the projection direction TD some of the rotational direction components among the yaw, pitch, and roll direction components corresponding to the imaging direction XC" is performed, for example, by the following process in S106 of the flowchart in Fig. 12 or 19. That is, the control unit 30 specifies the projection direction TD of the image in the virtual space V using only the "data on the rotational direction components reflected in the projection direction TD" in the imaging direction data (see Fig. 10 or 11).

[0125] [7-1-3. Aspects of Restricting Rotation of Projection Direction Using a Threshold] The method of restricting the rotation of the projection direction TD of the image projected into the virtual space V is not limited to the above-described method of preventing a specific rotation direction component corresponding to the shooting direction XC from being reflected in the projection direction TD, and restriction may also be performed based on the degree of rotation (rotation angle) of the projection direction TD.

[0126] For example, if a user wearing the HMD 10 were to view a direction rotated 180 degrees from the reference direction in the yaw direction (direction directly opposite the user's back), using the direction facing forward as the reference direction, this would place a significant strain on the user's neck and eyes. Therefore, a threshold may be set for rotation directions in which the projection direction TD significantly changes from the reference direction (front direction), thereby restricting the projection direction TD from changing at rotation angles exceeding the threshold. For example, thresholds may be set for the rotation directions in which the projection direction TD rotates from the reference direction, such as 90 degrees left and right in the yaw direction and 60 degrees up and down in the pitch direction, and the rotation of the projection direction TD is restricted by the thresholds. Furthermore, since the roll direction may also place a strain on the neck, a threshold may be set for the rotation angle in the projection direction TD from the reference direction, such as 45 degrees left and right in the roll direction. Thresholds may be set for at least some of the rotation directions among the yaw direction, the pitch direction, and the roll direction, and the rotation of the projection direction TD may be restricted. Furthermore, the thresholds are not limited to the above example and can be set arbitrarily. Furthermore, when the shooting direction XC of the shooting direction data (or the projection direction TD corresponding to the shooting direction XC) exceeds the threshold, the image may be projected into the virtual space V with the threshold as the projection direction TD.

[0127] The projection unit 32 of this embodiment has a function of limiting the rotation of the projection direction TD in at least some of the rotation directions of the yaw direction, pitch direction, and roll direction so that the rotation angle of the projection direction TD from the reference direction does not exceed a predetermined threshold. Here, the "reference direction" refers to a reference direction in the virtual space V, and as in the example described above, it can be the X-axis direction (see Figures 2, 3, etc.) corresponding to the direction in which the user wearing the HMD 10 faces forward. Furthermore, for rotation in the roll direction, the Y-axis or Z-axis can be used as the reference direction (see Figures 2, 3, etc.). Note that the reference direction may be any direction that serves as a reference in the virtual space V, and may be set to a direction deviated from the X-axis, Y-axis, or Z-axis.

[0128] With this configuration, the rotation of the projection direction TD is limited by a threshold value for at least some of the yaw, pitch, and roll directions (for example, a threshold value of 90 degrees left or right in the yaw direction and 60 degrees up or down in the pitch direction from the reference direction), thereby reducing strain on the user's neck and eyes.

[0129] In addition, the projection unit 32 in this embodiment has the function of projecting an image in a projection direction TD corresponding to the threshold value when the rotation angle from the reference direction of at least some of the rotational direction components of the yaw direction, pitch direction, and roll direction components corresponding to the shooting direction XC exceeds the threshold value.

[0130] According to this configuration, when the rotation angle (change from the reference direction) of the imaging direction XC (or the projection direction TD corresponding to the imaging direction XC) from the reference direction exceeds a threshold value for at least some of the rotation directions among the yaw direction, pitch direction, and roll direction, the image is projected in the projection direction TD corresponding to the threshold value. As a result, if the change from the reference direction of the imaging direction XC is within the threshold value, the projection direction TD of the image changes according to the imaging direction XC, thereby enabling the reproduction of images with a higher sense of realism and presence than conventional images. Furthermore, when the rotation angle of the imaging direction XC from the reference direction exceeds the threshold value, the image is projected into the virtual space V while remaining in the projection direction TD corresponding to the threshold value. This eliminates the need for the user to move their neck or eyes to follow the movement of the image. This allows the reproduction of images with a higher sense of realism and presence than conventional images, while reducing the strain on the user's neck and eyes by limiting the range of movement (projection range) of the image in the virtual space V using the threshold value.

[0131] Next, an example of the process of limiting the rotation of the projection direction TD using a threshold value will be described. Fig. 22 is a flowchart showing an example of the process of the display control device 20, and is an example of the "limitation process using a threshold value" subroutine executed in place of S106 in Fig. 12 or 19.

[0132] In S300, the control unit 30 determines the projection direction TD of the image in the virtual space V based on the image capture direction data corresponding to the image data and indicating the image capture direction XC of the image data in the real space R. In S302, the control unit 30 determines whether the rotation angle of the projection direction TD in the yaw direction from the reference direction exceeds a threshold value (S304). If the result of S302 is YES, the control unit 30 corrects the yaw direction of the projection direction TD to a direction corresponding to the threshold value. On the other hand, if the result of S302 is NO, the process proceeds to S306. In S306, the control unit 30 determines whether the rotation angle of the projection direction TD in the pitch direction from the reference direction exceeds a threshold value. If the result of S306 is YES, the control unit 30 corrects the pitch direction of the projection direction TD to a direction corresponding to the threshold value (S308). On the other hand, if the result of S306 is NO, the process proceeds to S310. In S310, the control unit 30 determines whether the rotation angle of the projection direction TD in the roll direction from the reference direction exceeds a threshold value. If the answer is YES in S310, the roll direction of the projection direction TD is corrected to a direction corresponding to the threshold value (S312).On the other hand, if the answer is NO in S310, or after S312 is executed, the subroutine ends.

[0133] 22 shows the process when a threshold is set for each of the rotation directions, yaw, pitch, and roll, but if there is a rotation direction for which a threshold is not set, the process for that rotation direction can be omitted. For example, if no threshold is set for the roll direction, steps S310 and S312 can be omitted.

[0134] [7-1-4. Aspects in which the User Can Set the Limitation on Rotation of the Projection Direction] The user may be able to set the limit on rotation of the projection direction TD of the image described above. For example, the user may be able to set whether or not at least some of the rotation directions among the yaw direction, pitch direction, and roll direction are to be prevented from being reflected in the projection direction TD, or the user may be able to set the threshold value.

[0135] As illustrated in FIG. 15 , the display control device 20 may include a restriction setting unit 37. The restriction setting unit 37 has a function of setting a restriction on the rotation of the projection direction TD based on a user operation. Here, "setting a restriction on the rotation of the projection direction TD" includes setting whether or not to rotate the projection direction TD in all or part of the yaw, pitch, and roll directions. Furthermore, "setting a restriction on the rotation of the projection direction TD" includes setting a threshold for a range (range of rotation angles) in which the projection direction TD rotates in all or part of the yaw, pitch, and roll directions.

[0136] The restriction setting unit 37 displays a setting screen for setting the restriction on the rotation of the projection direction TD on, for example, a display unit provided in the display control device 20, a display unit externally connected to the display control device 20, or the display unit 11 of the HMD 10. The restriction setting unit 37 also accepts the setting of the restriction on the setting screen based on the user's operation on the operation unit 23.

[0137] This configuration allows the user to set the rotation restriction on the projection direction TD of the image, enabling a high degree of freedom in image playback. For example, the user can set a setting that prioritizes reducing strain on the neck and eyes (by increasing the rotation restriction on the projection direction TD), or a setting that prioritizes highly realistic and immersive image playback (by eliminating or reducing the rotation restriction on the projection direction TD).

[0138] The "processing for setting the restriction on the rotation of the projection direction TD based on the user's operation" according to this aspect can be executed, for example, before the start of playback of the captured image data. For example, in the flowcharts of Fig. 12 or 19, this process may be executed before S100, S102, or S104.

[0139] Furthermore, the processing according to this aspect may be executed even after the start of playback of the captured image data. For example, an interrupt process may be generated by a user performing a predetermined operation during playback of the captured image data, thereby interrupting (pausing) the playback and allowing the user to set the restriction on the rotation of the projection direction TD (or change the current setting).

[0140] In addition, the setting information of the restriction contents set by the user once may be saved, and the saved restriction contents may be applied to subsequent playback of captured image data until the user changes the setting.

[0141] 8. Aspects for Preventing VR Sickness It is important to take measures to prevent users of the HMD 10 from experiencing VR sickness. It has been found that a wide field of view SR makes it easier for users viewing images in the field of view SR displayed on the display unit 11 of the HMD 10 to experience VR sickness. Therefore, VR sickness may be suppressed by narrowing the field of view SR (or narrowing the display range of the image in the field of view SR).

[0142] 23 and 24 are diagrams showing an example of a screen displayed on the display unit 11 of the HMD 10, in which the display range of an image in the field of view SR is narrowed. Hereinafter, narrowing the field of view SR is considered to include narrowing the display range of an image in the field of view SR (or widening the non-display range of an image in the field of view SR).

[0143] As a specific example, the area near the center of the field of view SD (a predetermined range from the field of view SD in the field of view range SR) is displayed in a normal state, while the area outside the center is darkened as it moves away from the field of view SD, thereby narrowing the effective display range of the image in the field of view SR. To achieve this, for example, a mask process is used in which a mask (a mask for hiding part of the image) is placed in the virtual space V to narrow the display range of the image in the field of view SR. As described above, to darken the field of view image as it moves away from the field of view SD, a mask with variable opacity (or transparency) can be placed. When the display range of the image in the field of view SR is narrowed in this manner, the display unit 11 of the HMD 10 displays, for example, an image of screen G13 shown in FIG. 23 . As a result, the peripheral field of view of the HMD 10, other than the field of view near the center, becomes darker as it moves away from the center of the field of view, thereby narrowing the field of view.

[0144] In particular, it is desirable to narrow the view range SR in the left-right direction. The method of narrowing the view range SR is not limited to the above. For example, as shown in screen G14 in FIG. 24 , the image of specific left and right edge regions of the view range SR may simply be cut off. In this case, a mask process may be performed in which a mask is placed on the left and right edge regions of the view range SR, or the size (viewing angle) of the view range SR itself may be set narrower than the viewing angle (default viewing angle) that can be displayed by the HMD 10. As illustrated in FIG. 24 , only the horizontal viewing angle may be narrowed, or the vertical viewing angle may also be narrowed.

[0145] The narrowing of the field of view range SR may be manually set by the user. For example, the field of view angle that can be displayed by the HMD 10 may be set as the default field of view range (image display range), and the user may change the field of view range to a value narrower than the default field of view range. In this case, the size (narrowness) of the field of view range may be set in multiple stages.

[0146] Alternatively, the control unit 30 may automatically adjust the size (narrowness) of the field of view SR. In particular, as described above, in this embodiment, the projection direction TD of the image changes within the virtual space V independently of the user's head movement. Therefore, it is also a preferred aspect to control the width of the field of view SR in relation to the change in the projection direction TD of the image to prevent VR sickness. As exemplified below, the narrowness of the field of view SR may be automatically set based on the change in the projection direction TD (at least some of the rotation directions of the yaw direction, pitch direction, and roll direction).

[0147] For example, when the projection direction TD of the image in the virtual space V changes by more than a certain amount, the field of view range SR may be automatically set so that it is narrower than usual (compared to when it does not change by more than a certain amount).

[0148] For example, when the projection direction TD starts rotating from a stopped (not rotating) state, and the rotation angle of the projection direction TD exceeds a predetermined threshold, the field of view range SR may be automatically set to a narrower value.

[0149] Furthermore, it is believed that the faster the rate of change of the projection direction TD, the more likely it is that VR sickness will occur. Therefore, for example, when the rotation angle (i.e., angular velocity) per unit time in the projection direction TD exceeds a predetermined threshold, the field of view range SR may be automatically narrowed (the field of view range SR may be set narrower than before the threshold was exceeded).

[0150] Alternatively, control may be performed based on the angular velocity of the rotation of the projection direction TD, such that the larger the angular velocity, the narrower the field of view SR is set. In this case, the size of the field of view SR is automatically adjusted according to the angular velocity of the projection direction TD.

[0151] Furthermore, it is believed that the more rapidly the projection direction TD changes, the more likely VR sickness is to occur. Therefore, for example, when the angular acceleration of the rotation of the projection direction TD exceeds a predetermined threshold, the field of view SR may be automatically narrowed (the field of view SR may be set narrower than before the threshold was exceeded).

[0152] Alternatively, control may be performed based on the angular acceleration of the rotation of the projection direction TD, so that the larger the angular acceleration, the narrower the field of view SR is set. In this case, the size of the field of view SR is automatically adjusted according to the angular acceleration of the projection direction TD.

[0153] VR sickness can also occur due to changes in the field of view SD. Therefore, it is a preferred embodiment to prevent VR sickness by controlling the width of the field of view SR in relation to changes in the field of view SD. As exemplified below, the narrowness of the field of view SR may be automatically set based on changes in the field of view SR (at least some of the rotation directions of the yaw direction, pitch direction, and roll direction).

[0154] For example, the field of view range SR may be automatically set to be narrower than normal (compared to when the field of view direction SD in the virtual space V changes by more than a certain amount). For example, if the field of view direction SD starts to rotate from a stationary (not rotating) state and the rotation angle of the field of view direction SD exceeds a predetermined threshold, the field of view range SR may be automatically set to be narrower.

[0155] Furthermore, since it is believed that the faster the rate of change of the viewing direction SD, the more likely VR sickness is to occur, for example, when the angular velocity of the rotation of the viewing direction SD exceeds a predetermined threshold, the viewing range SR may be automatically narrowed (the viewing range SR may be set narrower than before the threshold was exceeded). Alternatively, based on the angular velocity of the rotation of the viewing direction SD, control may be performed such that the viewing range SR is set narrower the greater the angular velocity. In this case, the size of the viewing range SR is automatically adjusted according to the angular velocity of the viewing direction SD.

[0156] Furthermore, it is believed that the more rapidly the field of view direction SD changes, the more likely VR sickness is to occur. Therefore, for example, when the angular acceleration of the rotation of the field of view direction SD exceeds a predetermined threshold, the field of view range SR may be automatically narrowed (the field of view range SR may be set narrower than before the threshold was exceeded). Alternatively, based on the angular acceleration of the rotation of the field of view direction SD, control may be performed to set the field of view range SR narrower as the angular acceleration increases. In this case, the size of the field of view range SR is automatically adjusted according to the angular acceleration of the field of view direction SD.

[0157] Furthermore, the aforementioned automatic setting of the narrowness of the field of view SR based on a change in the field of view SR and the automatic setting of the narrowness of the field of view SR based on a change in the field of view direction SD may be used in combination.

[0158] [9. Modifications, etc.] While the embodiments of the present invention have been described above, the specific configurations are not limited to the above-described embodiments, and designs that do not deviate from the gist of the present invention are also included. Furthermore, the above-described configurations and aspects can be combined in any manner.

[0159] [9-1] The above mainly describes an example of playing back "shooting data" including "video data" and "shooting direction data." However, the present invention can also be applied to playing back "video data for which shooting direction data has not yet been associated." That is, while playing back video data, changes in the shooting direction may be identified in real time using known image analysis to generate shooting direction data. In this way, the projection direction TD of the image corresponding to the video data may be controlled based on the shooting direction data generated during playback of the video data.

[0160] [9-2] It is also possible to display on the HMD 10 live video captured in real time by a camera 200 capturing the real space R. For example, the camera 200 illustrated in FIG. 6 can be connected to the display control device 20 via a wired or wireless connection, and the live video captured by the camera 200 can be displayed on the HMD 10 as a VR image. The remote camera 200 and the display control device 20 may also be connected via a network such as the Internet for data communication. In this case, the live video of the remote location captured by the camera 200 can be displayed on the HMD 10 as a VR image. A server having a distribution service function may be set on the network, and the live video captured by the camera 200 may be distributed via the network via the server. In this case, the display control device 20 can receive the live video of the remote location distributed via the network, and the live video can be played back in approximately real time and displayed on the HMD 10.

[0161] That is, an image display system including a camera that captures real space R, a display control device of the above-mentioned configuration, and an HMD, or an image display system including the camera, a distribution server, a display control device of the above-mentioned configuration, and an HMD, can be constructed to control the projection direction TD of an image in virtual space V based on the capture direction data corresponding to the video data captured in real space R.

[0162] [9-3] As illustrated in Figure 1, when the HMD 10 and the display control device 20 both have the configuration and functions of an information processing device (computer) equipped with a processor and a storage device, some of the functions of the control unit 30 described above may be realized by the processor 13 of the HMD 10 that executes the program related to this embodiment, and the remaining functions may be realized by the processor 21 of the display control device 20 that executes the program related to this embodiment.

[0163] [9-4] Figure 26 is a schematic block diagram showing an example of the configuration of a standalone HMD 60 in which the functions of the display control device 20 are integrated with the HMD 10, or an information processing device 52 such as a smartphone used as the HMD 50 illustrated in Figure 25. The HMD 60 or the information processing device 52 includes a display unit 61, a sensor 62, a processor 63, a storage device 64, an operation unit 65, and a communication unit 66. The display unit 61, the sensor 62, the processor 63, the storage device 64, the operation unit 65, and the communication unit 66 have the same configurations as the display unit 11, the sensor 12, the processor 21, the storage device 22, the operation unit 23, and the communication unit 24 described above, respectively, and therefore a description thereof will be omitted. Note that the operation unit 65 or the communication unit 66 may be configured separately, such as externally, or may be omitted. In the case of this standalone HMD 60 or information processing device 52, it also has the configuration and functions of an information processing device (computer) equipped with a processor and a storage device, and each function of the control unit 30 described above is realized by a processor 63 that executes a program related to this embodiment.

[0164] [9-5] Some or all of the functions of the control unit 30 described above may be implemented by an integrated circuit such as an LSI (Large Scale Integration). Furthermore, each of the above functions may be implemented individually as a processor. Alternatively, some or all of the above functions may be integrated into a processor.

[0165] [9-6] The computer-readable program according to this embodiment is recorded on various computer-readable recording media, such as a hard disk, an optical disk (CD-ROM, DVD-ROM, etc.), a flexible disk, or a semiconductor memory, and is read from the recording media and executed by a computer constituting the image display system 1 or the display control device 20. The program can also be provided to a computer via a network, including a communication line such as the Internet, a WAN, a LAN, or a dedicated line. A computer can read a program stored on a file server (online storage). A computer can also receive a program distributed from a distribution server. The recording media also include internal or external recording media accessible from the distribution server for program distribution. The program code stored on the distribution server's recording media does not need to be in a format directly executable by the computer that receives it. In other words, the format of the program stored on the distribution server's recording media is arbitrary, as long as it can be installed in an executable manner on a computer after being downloaded from the distribution server. The program can also be divided into multiple parts, downloaded at different times, and then combined. Each of the divided programs can be distributed by a different distribution server. The computer-readable recording medium also includes a storage medium that stores the program for a certain period of time, such as a volatile memory such as RAM in a server that transmits the program via a network or in a computer that receives the program. The program may also be a differential program that can achieve the above-mentioned functions in combination with a program already stored in the computer.

[0166] [10. Supplementary Notes] From the above description, the present invention can be understood, for example, as follows: Note that, to facilitate understanding of the present invention, reference numerals in the accompanying drawings are conveniently added in parentheses, but this does not mean that the present invention is limited to the illustrated embodiments.

[0167] 1) A program according to one aspect of the present invention causes a computer that executes control to display an image of a field of view (SR) of a virtual space (V) viewed in a field of view (SD) from a virtual viewpoint (P) in the virtual space on a display unit (11) of a head-mounted display (10), to function as: a storage control unit (31) that stores in a storage device (22) image data of a real space (R) and image capture data including image capture direction data that corresponds to the image data and indicates the image capture direction (XC) of the image data in the real space; and a projection unit (32) that projects an image corresponding to the image data from the virtual viewpoint (P) into the virtual space (V) based on the image capture data, and the projection unit (32) controls a projection direction (TD) of the image corresponding to the image data in the virtual space (V) based on the image capture direction data that corresponds to the image data.

[0168] Here, the "computer" may include at least a processor and a storage device (memory). Here, the processor is, for example, a CPU. The processor may also include hardware such as a GPU, DSP, or FPGA in addition to or instead of the CPU. For example, personal computers, tablet computers, smartphones, stationary or portable game consoles, commercial game consoles, mobile phone terminals, PHS terminals, PDAs, multi-function televisions with information processing functions, servers, and other devices that include a processor and a storage device are all included in the "computer" category. Furthermore, a "computer" may be composed of multiple devices that can communicate with each other. For example, a system including a server and a terminal device is also included in the "computer" category. Furthermore, an HMD itself, which includes a processor and a storage device, is also included in the "computer" category. For example, in the case of an HMD that has an information processing device such as a smartphone attached to an attachment, the smartphone is also included in the "computer" category. Furthermore, for example, a standalone HMD is also included in the "computer" category.

[0169] According to the aspect described above in 1), even when reproducing images in a limited field of view in an HMD, it is possible to reproduce images with a higher sense of reality and presence than conventionally possible.

[0170] 2) In one aspect of the present invention, in the aspect described in 1) above, the computer is further made to function as an object display unit (33) that displays an object (OB1) in the image corresponding to the image data projected into the virtual space (V) to enable recognition of the shooting direction (XC) or the projection direction (TD) of the shooting direction data corresponding to the image data.

[0171] In the aspect described in 2) above, the shooting direction of the image captured in the real space corresponds to the projection direction of the image projected into the virtual space, which corresponds to the direction of approximately the center of the image. Typically, when capturing a real space with a camera, the camera is pointed in a direction that the photographer considers noteworthy (important), so it is believed that the shooting direction contains a lot of important information (such as noteworthy objects). In this aspect, an object is displayed in the image to enable recognition of the shooting direction or the projection direction corresponding to the shooting direction, allowing a user viewing the image through an HMD to recognize the direction that is believed to be noteworthy. In particular, when the projection range (full field of view) of the image projected into the virtual space is wide, it is difficult to determine the center direction of the image, so applying this configuration is effective.

[0172] 3) In one aspect of the present invention, in the aspect described in 1) or 2) above, the computer is further made to function as a detection information acquisition unit (34) that acquires detection information regarding the orientation of the head-mounted display (10), a field of view direction change unit (35) that changes the field of view direction (SD) based on the detection information acquired by the detection information acquisition unit (34), and a projection direction information display unit (36) that displays information suggesting the projection direction (TD) of the image in the field of view range (SR) when the image projected into the virtual space (V) is not displayed in the field of view range (SR).

[0173] According to the aspect described in 3) above, the field of view and the field of view area determined by the field of view change depending on the orientation of the HMD (i.e., the orientation of the head of the user wearing the HMD). Furthermore, the projection direction of the played-back image changes based on the shooting direction data included in the shooting data. This can result in a large difference between the field of view and the projection direction, and in some cases, the image is not displayed at all in the field of view. This can cause the user to lose track of the projection direction of the image. In this regard, in this aspect, when the image is no longer displayed in the field of view, information that enables the user to recognize the projection direction is displayed in the field of view, making it easy to see in which direction the HMD (head) should be pointed so that the image will be displayed in the field of view.

[0174] 4) In one aspect of the present invention, in any of the aspects described above in 1) to 3), the projection unit (32) limits the rotation of the projection direction (TD) with respect to at least some of the rotation directions among the yaw direction, pitch direction, and roll direction.

[0175] According to the aspect described in 4) above, by limiting the rotation of at least some of the projection directions among the yaw direction, pitch direction, and roll direction, excessive strain on the user's neck and eyes when viewing with an HMD can be reduced.

[0176] 5) In one aspect of the present invention, in the aspect described in 4) above, the projection unit (32) does not reflect some of the rotational direction components of the yaw direction, pitch direction, and roll direction components corresponding to the shooting direction (XC) in the projection direction (TD).

[0177] According to the aspect described in 5) above, the change in the projection direction is limited to changes in only one or two of the rotational directions of the yaw direction, pitch direction, and roll direction, thereby reducing the strain on the user's neck and eyes.

[0178] 6) In one aspect of the present invention, in the aspect described in 5) above, the projection unit (32) does not reflect the pitch direction component corresponding to the shooting direction (XC) in the projection direction (TD).

[0179] According to the aspect described in 6) above, the image displayed on the display unit of the HMD does not move in the pitch direction (up and down direction) relative to the HMD (i.e., relative to the user's head and eyes), but is limited to movement in the horizontal direction (yaw direction). Generally, when a user moves their neck or line of sight, the pitch direction (up and down direction) places a greater burden on them than the yaw direction (left and right direction). Therefore, by limiting the movement of the image in the pitch direction according to this aspect, the strain on the user's neck and eyes can be effectively reduced.

[0180] 7) In one aspect of the present invention, in the aspect described in 5) or 6) above, the projection unit (32) does not reflect the roll direction component corresponding to the shooting direction (XC) in the projection direction (TD).

[0181] According to the aspect described in 7) above, the roll direction component corresponding to the shooting direction is not reflected in the projection direction, so that a difference occurs between the direction of gravity in the real world relative to the HMD (i.e., relative to the user's head) and the direction of gravity in the image, which makes it possible to display images that are full of dynamism and impact.

[0182] 8) In one aspect of the present invention, in any of the aspects described above in 4 to 7), the projection unit (32) limits the rotation of the projection direction (TD) for at least some of the rotation directions among the yaw direction, the pitch direction, and the roll direction so that the rotation angle from a reference direction in the projection direction (TD) does not exceed a predetermined threshold value.

[0183] According to the aspect described in 8) above, the rotation of the projection direction is limited by a threshold value for at least some of the rotation directions of the yaw direction, pitch direction, and roll direction (for example, a threshold value of 90 degrees left and right in the yaw direction and 60 degrees up and down in the pitch direction from the reference direction). This reduces strain on the user's neck and eyes.

[0184] 9) In one aspect of the present invention, in the aspect described in 8) above, the projection unit (32) projects the image in the projection direction (TD) corresponding to the threshold value when the rotation angle from the reference direction of at least some of the rotational direction components of the yaw direction, pitch direction, and roll direction components corresponding to the shooting direction (XC) exceeds the threshold value.

[0185] According to the aspect described in 9) above, when the rotation angle (change from the reference direction) of the shooting direction (or the projection direction corresponding to the shooting direction) from the reference direction exceeds a threshold value for at least some of the rotation directions among the yaw direction, pitch direction, and roll direction, the image is projected in the projection direction corresponding to the threshold value. As a result, if the change from the reference direction of the shooting direction is within the threshold value, the projection direction of the image changes according to the shooting direction, allowing for the playback of images with a higher sense of realism and presence than conventional images. Furthermore, when the rotation angle of the shooting direction from the reference direction exceeds the threshold value, the image is projected into the virtual space while remaining in the projection direction corresponding to the threshold value, eliminating the need for the user to move their neck or eyes to follow the movement of the image. As a result, while playing images with a higher sense of realism and presence than conventional images, the strain on the user's neck and eyes can be reduced by setting a threshold value and limiting the movement range (projection range) of the image in the virtual space.

[0186] 10) In one aspect of the present invention, in the aspects described in 4) to 9) above, the computer further functions as a restriction content setting unit (37) that sets the restriction content for rotation of the projection direction (TD) based on user operation.

[0187] According to the aspect described in 10) above, the user can set the restrictions on the rotation of the projection direction by himself, enabling video playback with a high degree of freedom. For example, the user can set a setting that prioritizes reducing strain on the neck and eyes (by increasing the restrictions on the rotation of the projection direction) or a setting that prioritizes video playback with a high sense of realism and presence (by eliminating or reducing the restrictions on the rotation of the projection direction).

[0188] 11) A program according to one aspect of the present invention causes a computer that executes control to display an image of a field of view (SR) of a virtual space (V) viewed from a virtual viewpoint (P) in the field of view (SD) on a display unit (11) of a head-mounted display (10). The program causes the computer to function as a shooting direction identification unit that identifies the shooting direction (XC) in the real space (R) of video data captured in the real space (R), and a projection unit (32) that controls the projection direction (TD) of the video data in the virtual space (V) based on the shooting direction (XC) identified by the shooting direction identification unit. Here, the shooting direction identification unit can identify the shooting direction (or a change in the shooting direction) by reading "shooting direction data indicating the shooting direction of the video data in the real space" recorded in association with the video data. For example, the shooting direction identification unit can identify the shooting direction by reading "shooting direction data" included in the shooting data stored in the aforementioned storage control unit (31). Alternatively, the shooting direction identification unit can identify the shooting direction (or a change in the shooting direction) from the video data through image analysis. According to the aspect described in 11) above, the same effects as those of the aspect described in 1) above can be achieved.

[0189] 12) A display control device (20) according to one aspect of the present invention executes control to display on a display unit (11) of a head-mounted display (10) an image of a field of view (SR) of a virtual space (V) as seen in a field of view (SD) from a virtual viewpoint (P) in the virtual space. The display control device (20) includes: a storage control unit (31) that stores, in a storage device (22), image data obtained by capturing a real space (R) and image capture direction data corresponding to the image data and indicating a capture direction (XC) of the image data in the real space; and a projection unit (32) that projects an image corresponding to the image data from the virtual viewpoint (P) into the virtual space (V) based on the image capture data. The projection unit (32) controls a projection direction (TD) of the image corresponding to the image data in the virtual space (V) based on the image capture direction data corresponding to the image data. This achieves the same effect as the aspect described in 1) above.

[0190] 13) An image display system (1) according to one aspect of the present invention includes a head-mounted display (10) including a display unit (11), and a display control device (20) that controls the display unit (11) to display an image of a field of view (SR) of the virtual space (V) as seen in a field of view (SD) from a virtual viewpoint (P) in the virtual space. The display control device (20) includes a storage control unit (31) that stores, in a storage device (22), image data obtained by capturing a real space (R) and image capture direction data corresponding to the image data, indicating a capture direction (XC) of the image data in the real space. The display control device (20) also includes a storage control unit (31) that stores, in a storage device (22), image capture data including image data of the real space (R) and image capture direction data corresponding to the image data, and a projection unit (32) that projects an image corresponding to the image data from the virtual viewpoint (P) into the virtual space (V) based on the image capture data. The projection unit (32) controls a projection direction (TD) of the image corresponding to the image data in the virtual space (V) based on the image capture direction data corresponding to the image data. This achieves the same effect as the aspect described in 1) above.

[0191] 14) A control method according to one aspect of the present invention is a control method for controlling a computer that executes control to display an image of a field of view (SR) of a virtual space (V) as viewed in a field of view (SD) from a virtual viewpoint (P) within the virtual space (V) on a display unit (11) of a head-mounted display (10), the control method including: a storage control step (S102) for storing, in a storage device (22), image data obtained by capturing a real space (R) and image capture direction data corresponding to the image data, indicating an image capture direction (XC) of the image data in the real space; and a projection step (S108) for projecting an image corresponding to the image data from the virtual viewpoint (P) into the virtual space (V) based on the image capture data, wherein the projection step controls a projection direction (TD) of the image corresponding to the image data within the virtual space (V) based on the image capture direction data corresponding to the image data (S106, S108). This achieves the same effect as the aspect described in 1) above.

[0192] 15) An information storage medium according to one aspect of the present invention is a computer-readable information storage medium having recorded thereon the program according to any one of the aspects 1) to 11). This provides the same effects as those of the aspects 1) to 11).

[0193] It should be noted that the specific embodiments or examples given in the form for implementing the invention are merely intended to clarify the technical content of the present invention, and should not be interpreted narrowly as being limited to such specific examples, but can be implemented in various modified forms within the scope of the technical idea and claims of the present invention.

[0194] 1...image display system, 10...HMD, 11 / 61...display unit, 12 / 62...sensor, 13...processor, 14...storage device, 20...display control device, 21 / 63...processor, 22 / 64...storage device, 23 / 65...operation unit, 24 / 66...communication unit, 30...control unit, 31...storage control unit, 32...projection unit, 33...object display unit, 34...detection information acquisition unit, 35...view direction change unit, 36...projection direction information display unit, 37... Restriction content setting unit, 60...HMD (information processing device), 200...camera, OB1...frame object, OB2...arrow object, BC...background, R...real space, V...virtual space, P...virtual viewpoint, SD...view direction, SR...view range, XC...shooting direction, XC0...reference shooting direction, TD...projection direction, TD0...reference projection direction, TR...projection range, GD1...gravity direction of image, GD2...gravity direction relative to HMD, G10 to G14...screen

Claims

1. A program that causes a computer that executes control to display on the display unit of a head-mounted display an image of the field of view of a virtual space as seen from a virtual viewpoint within the virtual space; a storage control unit that stores in a storage device image data containing image data of a real space and image direction data corresponding to the image data that indicates the image shooting direction in the real space of the image data; and a projection unit that projects an image corresponding to the image data from the virtual viewpoint into the virtual space based on the image shooting data, wherein the projection unit controls the projection direction of the image corresponding to the image data within the virtual space based on the image shooting direction data corresponding to the image data.

2. The program according to claim 1, further causing the computer to function as an object display unit that displays an object within the image corresponding to the image data projected into the virtual space, to enable recognition of the shooting direction or projection direction of the shooting direction data corresponding to the image data.

3. The program according to claim 1 or 2, further causing the computer to function as: a detection information acquisition unit that acquires detection information regarding the orientation of the head-mounted display; a field of view direction change unit that changes the field of view direction based on the detection information acquired by the detection information acquisition unit; and a projection direction information display unit that displays information suggesting the projection direction of the image within the field of view when the image projected into the virtual space is not displayed within the field of view range.

4. The program according to any one of claims 1 to 3, wherein the projection unit limits the rotation of the projection direction with respect to at least some of the rotation directions of the yaw direction, pitch direction, and roll direction.

5. The program according to claim 4, wherein the projection unit does not reflect some of the rotational direction components among the yaw direction, pitch direction, and roll direction components corresponding to the shooting direction in the projection direction.

6. The program according to claim 5, wherein the projection unit does not reflect the pitch direction component corresponding to the shooting direction in the projection direction.

7. The program according to claim 5 or 6, wherein the projection unit does not reflect the component of the roll direction corresponding to the shooting direction in the projection direction.

8. A program described in any one of claims 4 to 7, wherein the projection unit limits the rotation of the projection direction so that the rotation angle from a reference direction in the projection direction does not exceed a predetermined threshold for at least some of the rotation directions among the yaw direction, the pitch direction, and the roll direction.

9. The program described in claim 8, wherein the projection unit projects the image in the projection direction corresponding to the threshold value when the rotation angle from the reference direction of at least some of the rotation direction components of the yaw direction, pitch direction, and roll direction components corresponding to the shooting direction exceeds the threshold value.

10. The program according to any one of claims 4 to 9, further causing the computer to function as a restriction setting unit that sets restrictions on the rotation of the projection direction based on a user operation.

11. A display control device that executes control to display an image of the field of view of a virtual space viewed in the field of view from a virtual viewpoint within the virtual space on the display unit of a head-mounted display, comprising: a storage control unit that stores, in a storage device, image data of a real space photographed and image capture direction data corresponding to the image data, which indicates the image capture direction in the real space of the image data; and a projection unit that projects an image corresponding to the image data from the virtual viewpoint into the virtual space based on the image capture data, wherein the projection unit controls the projection direction of the image corresponding to the image data within the virtual space based on the image capture direction data corresponding to the image data.

12. An image display system comprising a head-mounted display including a display unit, and a display control device that executes control to display on the display unit an image of the field of view of the virtual space as viewed in the field of view from a virtual viewpoint within the virtual space, wherein the display control device includes: a memory control unit that stores, in a memory device, image data of a real space photographed and image capture direction data corresponding to the image data that indicates the image capture direction in the real space of the image data; and a projection unit that projects an image corresponding to the image data from the virtual viewpoint into the virtual space based on the image capture data, wherein the projection unit controls the projection direction of the image corresponding to the image data within the virtual space based on the image capture direction data corresponding to the image data.

13. A control method for controlling a computer that executes control to display on a display unit of a head-mounted display an image of the field of view of a virtual space as viewed in the field of view from a virtual viewpoint within the virtual space, the control method comprising: a storage control step of storing, in a storage device, image data of a real space photographed and image capture direction data corresponding to the image data, which indicates the image capture direction in the real space of the image data; and a projection step of projecting, from the virtual viewpoint into the virtual space based on the image capture data, an image corresponding to the image data, wherein the projection step controls the projection direction of the image corresponding to the image data within the virtual space based on the image capture direction data corresponding to the image data.

14. A computer-readable recording medium having the program according to any one of claims 1 to 10 recorded thereon.

Images