Program, display control device, and image display system

Abstract

To realize display control of a virtual reality (VR) image that has a smaller sense of discomfort even if a close-range imaging object is included in a case of displaying the VR image on a head mounted display (HMD) using multiple images captured by multiple cameras.SOLUTION: A sight line direction specifying unit specifies a sight line direction which is a direction of a sight line of a user with respect to a virtual reality (VR) space. An image generation unit arranges a plurality of real images captured by multiple cameras having mutually-different imaging directions for imaging the real space within the VR space such that an overlapping region where portions of the visual field areas of the adjacent real images overlap each other is generated, and generates a VR image according to the sight line direction. The image generation unit sets a reference direction in a predetermined direction from a virtual viewpoint, and when the sight line direction crosses the reference direction, dynamically switches the real images displayed in the overlapping region from among the adjacent real images.SELECTED DRAWING: Figure 14

Description

Technical Field

[0001] The present invention relates to a program, a display control device, and an image display system.

Background Art

[0002] Conventionally, a head-mounted display (HMD) that can display a stereoscopic virtual reality (VR) image using binocular parallax has been known. With this HMD, it is possible to display a VR space generated by a computer, a real image (still image or moving image) obtained by imaging the real space, and the like. In the case of a real image, for example, even with a single stereo camera using a wide-angle lens, a wide-angle image with a certain field of view can be obtained. However, in a direction far from the optical axis direction of the camera, it becomes difficult to perform stereoscopic viewing. In order to reduce this problem of difficult stereoscopic viewing, there is also a method of obtaining a wide-angle image by performing compositing processing on a plurality of images captured by dividing regions using a plurality of cameras (for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Multiple cameras each have a volume, and it is physically impossible to arrange the multiple cameras so that their imaging centers converge at a single point. For this reason, when images are captured with multiple cameras, misalignment occurs at the region boundaries between adjacent images, resulting in seams (such as extreme differences in shading between pixels). Therefore, when creating a final single image data from images captured by multiple cameras that divide regions, it is necessary to perform stitching processing to join the images captured by each camera. This stitching processing can be automatically performed by dedicated software, but in order to obtain a more natural image with no visible seams, it needs to be done manually with effort. Here, the degree of misalignment between adjacent images depends on the distance between the camera and the subject being photographed. The closer the subject is to the camera, the greater the degree of misalignment and the more difficult the stitching processing becomes. In particular, when the distance between the camera and the subject is below a certain level (for example, 2 m or less), the degree of misalignment becomes quite large and stitching becomes difficult. Therefore, in the case of conventional technologies that synthesize multiple images captured by multiple cameras by stitching, it has been substantially impossible to create a VR image including a close-range photographed object such that the distance from the camera is below a certain level and display it on an HMD with little sense of discomfort.

[0005] Therefore, one of the objectives of the present invention is to realize display control of a VR image with little sense of discomfort even when including a close-range photographed object when displaying a VR image on an HMD using multiple images captured by multiple cameras.

Means for Solving the Problem

[0006] A program according to an aspect of the present invention functions as a computer that executes control for causing a head-mounted display to display, as a stereoscopic image using binocular disparity, a VR image representing a field of view from a virtual viewpoint in a virtual reality (VR) space on a display unit, and includes a line-of-sight direction specifying unit that specifies a line-of-sight direction, which is assumed to be a direction of a user's line of sight with respect to the VR space, and an image generation unit that arranges a plurality of real-shot images captured by a plurality of cameras having different imaging directions in the VR space so that a partial overlapping region where the field-of-view regions of adjacent real-shot images overlap with each other is generated, and generates the VR image according to the line-of-sight direction, and the image generation unit dynamically switches the real-shot image to be displayed in the overlapping region among the adjacent real-shot images based on the line-of-sight direction.

Brief Description of the Drawings

[0007]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Figure 31

Figure 32

Figure 33

Figure 34

Figure 35

Embodiments for Carrying Out the Invention

[0008] Hereinafter, an example of an embodiment of the present invention will be described with reference to the drawings.

[0009] [1. Configuration Example of Image Display System] FIG. 1 is a schematic block diagram showing a configuration example of the hardware of an image display system 1 according to an embodiment of the present invention. This image display system 1 includes an HMD 10 and a display control device 20 that executes display control for the HMD 10.

[0010] The HMD10 is worn on the user's head (see Figure 2), and can display a wide-angle stereoscopic image (still image or moving image) that utilizes binocular parallax for stereoscopic viewing. Further, the HMD10 is equipped with a sensor 12 that detects the movement and inclination of the HMD10, such as a gyro sensor, and detects changes in the movement and inclination of the head of the user wearing the HMD10, and displays a VR image representing the field of view in the VR space according to the changes. For example, the VR image displayed on the HMD10 changes to a VR image corresponding to the rightward field of view in the VR space when the user's head faces rightward, and changes to a VR image corresponding to the upward field of view in the VR space when the user looks upward, so that a sense of immersion as if the user were on the spot can be given to the user.

[0011] The HMD10 mainly includes a display unit 11, a sensor 12, a processor 13, a storage device 14, and the like. The display unit 11 is a display that displays various information such as VR images and text. The display unit 11 may be a display of a virtual image projection method that forms a virtual image by using, for example, a half mirror or the like. Further, the display unit 11 may be a display of a retinal projection method that directly forms a VR image on the retina by using, for example, the crystalline lens of the user's eye. The display unit 11 displays a stereoscopic image (right-eye image and left-eye image) that utilizes binocular parallax as a VR image representing the field of view from a virtual viewpoint in the VR space.

[0012] The sensor 12 detects attitude information such as the rotation angle and inclination of the HMD10. From the output of this sensor 12, detection information regarding the orientation of the HMD10 can be obtained. For example, the sensor 12 is an angular velocity sensor (gyro sensor) that detects the angular velocity of an object. Note that the sensor 12 may be a sensor that detects a change in direction, or a sensor that detects the direction itself. For example, the sensor 12 is not limited to a gyro sensor, and may be an acceleration sensor, an angular acceleration sensor, an inclination sensor, a geomagnetic sensor, or the like, and these can be realized by appropriately combining them.

[0013] The processor 13 functions as a control center that controls each part of the HMD 10. For example, the processor 13 is a CPU (Central Processing Unit). The processor 13 may be configured to include hardware such as a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), or an FPGA (Field Programmable Gate Array) in addition to or instead of the CPU. The processor 13 performs control to supply, for example, the VR image signal received from the display control device 20 to the display unit 11 via an interface. The storage device 14 stores the programs executed by the processor 13 and temporarily stores the data and parameters processed by the processor 13. The storage device 14 includes, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), a VRAM (Video Random Access Memory), an auxiliary storage device, etc. As the auxiliary storage device, a non-volatile semiconductor memory, a hard disk drive, a solid state drive, etc. can be used.

[0014] Also, the HMD 10 may be provided with a network adapter or the like so that a VR image can be input via a network. In addition, the HMD 10 may be provided with an audio output unit, a GPS (Global Positioning System) receiving unit, etc.

[0015] The display control device 20 is communicably connected to the HMD 10 wirelessly or by wire and executes display control for the HMD 10. The display control device 20 is, for example, a stationary or portable game console. The display control device 20 may be a business-use (commercial) game console installed in a game facility or the like. Alternatively, the display control device 20 may be a personal computer, a tablet computer, a smartphone, a mobile phone terminal, a PHS (Personal Handy-phone System) terminal, a personal digital assistant (PDA), a multifunctional television receiver with information processing functions (so-called smart TV), or the like.

[0016] The display control device 20 mainly includes a processor 21, a storage device 22, an operation unit 23, and a communication unit 24, which are interconnected via bus lines including an address bus, a data bus, a control bus, and the like. Note that interface circuits, image processing units, sound processing units, or the like may be interposed between the bus lines and each component as necessary, but are omitted here for illustration purposes.

[0017] The processor 21 interprets and executes program instructions to control the entire display control device 20. For example, the processor 21 is a CPU. The processor 21 may include hardware such as a GPU, a DSP, or an FPGA in addition to or instead of the CPU. The storage device 22 includes, for example, a ROM, a RAM, a VRAM, an auxiliary storage device, and the like. The ROM stores programs, data, and the like necessary for basic operation control of the display control device 20. The RAM or VRAM stores various programs and data and secures a working area for the processor 21. The auxiliary storage device stores programs, various data, and the like, and for example, a non-volatile semiconductor memory, a hard disk drive, a solid state drive, or the like can be used.

[0018] Further, the display control device 20 may include a recording medium drive. Examples of the recording medium drive include a DVD-ROM drive, a CD-ROM drive, a hard disk drive, an optical disk drive, a flexible disk drive, a silicon disk drive, a cassette medium reader, etc. In this case, examples of the recording medium include a DVD-ROM, a CD-ROM, a hard disk, an optical disk, a flexible disk, a semiconductor memory, etc. The recording medium drive reads out image data, audio data, and program data from the recording medium, and supplies the read data to the RAM or the like of the storage device 22 via a decoder.

[0019] For example, image data including a real image obtained by imaging the real space in advance for VR images, and programs such as games including the image data are stored in the storage device 22 (auxiliary storage device or the like) or read out from the recording medium drive.

[0020] The operation unit 23 is for the user to input various operation commands to the display control device 20. For example, the user performs operations for viewing VR images or operations for games including VR images. Examples of the operation unit 23 include a position input unit having a touch interface (components of a touch panel, etc.), physical buttons, a controller, an analog stick, a keyboard, a pointing device, etc. Further, it may be configured as an operation unit 23 capable of voice input by identifying voice input from a voice input unit such as a microphone. Further, the operation unit 23 may be for performing operations by the user's gestures.

[0021] The communication unit 24 is provided with a communication interface (not shown) and has a communication control function for data communication during the execution of games and the like. Here, the communication control function for data communication includes, for example, an Internet connection function, a wireless LAN (Local Area Network) connection function, a short-range wireless communication function using a predetermined frequency band (for example, a frequency band of 2.4 GHz), and the like. The communication unit 24 transmits a connection signal for connecting the display control device 20 to the network based on an instruction from the processor 21, and receives information transmitted from the communication partner side and supplies it to the processor 21.

[0022] As will be described later, when the image display system 1 or the display control device 20 of the present embodiment displays a VR image on the HMD 10, it is possible to reduce the discomfort of the seam (shift) between a plurality of real-shot images captured by a plurality of cameras without performing stitching processing. Therefore, it is also possible to receive a live video of a remote location distributed via the network by the communication unit 24 and display the live video as a VR image on the HMD 10 in substantially real time.

[0023] Further, a tracking unit for detecting the position of the HMD 10 (or the position of the head of the user wearing the HMD 10) may be connected to the display control device 20. For example, the HMD 10 is provided with a plurality of light sources (such as LEDs) for tracking, and the tracking unit is provided with a photographing unit for photographing the HMD 10. The tracking unit fixed at a predetermined position specifies the position of the HMD 10 based on the positions of the plurality of light source units reflected in the image photographed by the photographing unit. Further, the display control device 20 may be provided with a display unit such as a liquid crystal display or an organic EL (Electro-Luminescence) display.

[0024] [2. Regarding the direction in the VR space] FIG. 2 is a diagram showing the definition of the direction in the VR space according to the present embodiment. In the present embodiment, the vertical direction, which is the direction in which the user wearing the HMD 10 for displaying VR images in the VR space stands upright, is defined as the Z-axis. Further, among the horizontal axes orthogonal to the Z-axis, the direction of the line of sight of the user facing forward (the reference line-of-sight direction with respect to the VR space) is defined as the X-axis, and the horizontal axis orthogonal to the Z-axis and the X-axis is defined as the Y-axis.

[0025] For example, when the image display system 1 is activated, the line-of-sight direction of the HMD 10 (the direction orthogonal to the display surface of the display unit 11 of the HMD 10) is set as the reference line-of-sight direction with respect to the VR space. Further, for example, the reference line-of-sight direction with respect to the VR space may be adjusted by a user wearing the HMD 10 and facing forward performing a predetermined operation.

[0026] Here, a change in the rotational direction about the Z-axis is referred to as a change in the yaw direction (left-right direction), a change in the rotational direction about the Y-axis is referred to as a change in the pitch direction (up-down direction), and a change in the rotational direction about the X-axis is referred to as a change in the roll direction. For example, the above-described sensor 12 of the HMD 10 detects the angular velocity or angular acceleration in the rotational direction of each axis (yaw direction, pitch direction, and roll direction). Note that a change in the yaw direction may be referred to as a change in the left-right direction, and a change in the pitch direction may be referred to as a change in the up-down direction.

[0027] FIG. 3 is a diagram for explaining a VR image (field-of-view image) representing a field of view from a virtual viewpoint in the VR space according to the present embodiment. In this figure, the virtual viewpoint P (the virtual viewpoint of the user) in the VR space V is set as the intersection (origin) of the X-axis, Y-axis, and Z-axis. For example, assume that the user is facing forward and the direction of the user's line of sight with respect to the VR space V is in the X-axis (reference line-of-sight direction). In this case, the range of the VR image representing the field of view from the virtual viewpoint P in the VR space V is a range determined by the yaw angle α (the interior angle between the broken line a and the broken line b, and the interior angle between the broken line c and the broken line d) centered on the reference line-of-sight direction (X-axis direction) and the pitch angle β (the interior angle between the broken line a and the broken line d, and the interior angle between the broken line b and the broken line c). Here, the yaw angle α is the horizontal field-of-view angle, and the pitch angle β is the vertical field-of-view angle, which are preset as the field-of-view angles of the VR image to be displayed on the HMD 10 in the image display system 1.

[0028] For example, when the head of the user wearing the HMD 10 changes in the pitch direction or the yaw direction, the change in the orientation / posture of the HMD 10 (the change in the line-of-sight direction of the HMD 10) is detected by the sensor 12 or the like. Based on the detection information by the sensor 12 or the like, the processor 21 of the display control device 20 determines that the line-of-sight direction with respect to the VR space V has changed from the X-axis direction (reference line-of-sight direction) in the pitch direction or the yaw direction. In response to this change in the line-of-sight direction with respect to the VR space V, the range of the VR image displayed on the display unit 11 is changed. Similarly, when the head of the user wearing the HMD 10 changes in the roll direction, the change is detected by the sensor 12 or the like, and while the line-of-sight direction remains in the X-axis direction, the range of the VR image displayed on the display unit 11 rotates in the roll direction. Thus, the range of the VR image displayed on the display unit 11 is changed according to the orientation (posture) of the HMD 10.

[0029] Also, in the VR space V, display objects other than real images such as various objects, lines, symbols, characters, etc. can be arranged as needed. For example, the "boundary line" or the like described later may be superimposed on the real VR image and displayed on the display unit 11.

[0030] When displaying a stereoscopic image using binocular disparity, there are line-of-sight directions corresponding to the respective virtual viewpoints for the right eye and the left eye, and the right-eye VR image and the left-eye VR image in the respective line-of-sight directions are displayed on the display unit 11 of the HMD 10.

[0031] [Regarding the real image captured by the camera] In the image display system 1 of the present embodiment, a VR image capable of stereoscopic vision using binocular disparity is generated using a real image captured by a camera that captures the real space, and is displayed on the HMD 10. Before explaining the real image used in the image display system 1 of the present embodiment, problems with a real image captured by a single camera will be explained.

[0032] FIG. 4 is a diagram for explaining the problems of a real image captured by a single camera 100 (so-called stereo camera) including a left-eye camera 101 and a right-eye camera 102. The camera 100 is at different positions in the real space, and the camera 100 captures objects A and B that exist at the same distance from the camera 100. Object A is in the optical axis direction (≈ the direction of the lens center) XC of the camera 100, and object B is in the direction to the right with respect to the optical axis direction XC. Compared with the angle (convergence angle) θ1 formed by object A and the left-eye camera 101 and the right-eye camera 102, the angle θ2 formed by object B and the left-eye camera 101 and the right-eye camera 102 becomes smaller even though objects A and B are at the same distance from the camera 100. For this reason, there is a problem that, for a direction that is greatly deviated from the optical axis direction XC of the camera 100, it becomes difficult to recognize the sense of distance and the image is difficult to view stereoscopically.

[0033] In order to solve the above problems, as illustrated in FIG. 5, it is effective to divide the imaging area and perform imaging using a plurality of cameras 100 with different imaging directions for imaging the real space. In the example of FIG. 5, three cameras 100 divide the imaging area into three areas: left, front, and right, and perform imaging. By dividing the entire imaging area into a plurality of parts in this way, the optical axis direction XC of the camera exists within each imaging area, and the directions that are greatly separated from the optical axis direction XC can be reduced. Therefore, by using a plurality of images captured by a plurality of cameras 100 with different imaging directions, the problem of images being difficult to perform stereoscopic vision can be reduced.

[0034] Conventionally, a plurality of images captured by a plurality of cameras that divide the area were finally made into one image data. For this reason, it was necessary to perform stitching processing for connecting adjacent images captured by each camera. This stitching processing can be automatically performed by dedicated software, but in order to obtain a more natural image with an indistinguishable seam, it is necessary to perform it manually with effort. One of the factors that require effort in the stitching processing is the deviation in the positions of each camera used for imaging. That is, although a plurality of cameras are arranged at positions close to each other, as long as they image the real space, it is physically impossible to arrange them at exactly the same position at the same time. Therefore, a deviation occurs near the area boundary between adjacent images captured by adjacent cameras. This will be explained below.

[0035] FIG. 6 is a diagram for explaining an example in which a deviation occurs between images captured by a camera 100a that captures a front region and a camera 100b that captures a right region. Here, for convenience, the case of capturing an image with the left-eye camera 101 of each of the two cameras 100a and 100b will be illustrated and described, but the case of capturing an image with the right-eye camera 102 is the same. Objects C and D to be imaged are located on the right side of the camera 100a and on the left side of the camera 100b, and are present near the region boundary of the imaging regions of the cameras 100a and 100b. Also, the object C is located deeper (at a position farther from the cameras 100a and 100b) than the object D. In this case, when an image is captured with the camera 100a that captures the front region, an image is captured in which the object D is located slightly to the right of the object C that is deeper. On the other hand, when an image is captured with the camera 100b that captures the right region, an image is captured in which the object D is located slightly to the left of the object C. Thus, although the cameras 100a and 100b are capturing the same objects C and D simultaneously, a clear deviation occurs between the images captured by both of them.

[0036] In the conventional stitching process, in order to make the deviation between adjacent images as described above into a more natural image, it is necessary to manually process (disguise) the image, which is time-consuming. The degree of deviation near the region boundary between the adjacent images described above depends on the distance between the cameras 100a and 100b and the imaging target. That is, when the imaging target (the objects C, D, etc. described above) near the region boundary is at a position far enough from the cameras 100a and 100b, the degree of deviation between the adjacent images also becomes small, so the conventional stitching process is relatively easy. On the other hand, when the imaging target near the region boundary is at a position close to the cameras 100a and 100b, the degree of deviation between the adjacent images also becomes relatively large, and stitching becomes difficult. In particular, when the distance between the imaging target near the region boundary and the cameras 100a and 100b is below a certain level (for example, 2 m or less), the degree of deviation between the adjacent images also becomes quite large, and the stitching process becomes difficult. Therefore, in the conventional case where a plurality of images captured by a plurality of cameras are finally made into one VR image data by stitching, it has been substantially impossible to create VR image data including a close-range photographed object that is at a distance of less than a certain distance from the camera.

[0037] Therefore, in the image display system 1 or the display control device 20 of the present embodiment, instead of performing the conventional stitching process on a plurality of images captured by a plurality of cameras, the VR image display control described below is executed.

[0038] [4. VR Image Display Control] The outline of the VR image display control executed in the image display system 1 or the display control device 20 of the present embodiment is as follows. That is, the image display system 1 or the display control device 20 executes a characteristic process of arranging a plurality of real images captured by a plurality of cameras with different imaging directions for imaging the real space in the VR space without performing stitching processing. Here, when arranging the plurality of real images in the VR space, they are arranged in the VR space so that a partial overlapping area where the viewing areas of adjacent real images overlap each other is generated. Then, according to the line-of-sight direction with respect to the VR space (a direction that can be determined as the direction of the user's line of sight), the region boundary between adjacent real images in the overlapping area is dynamically changed. Thereby, the image deviation that easily occurs at the region boundary of adjacent real images is kept as far as possible from the line-of-sight direction, and a characteristic display control is realized in which a VR image with reduced unnaturalness due to the image deviation is displayed on the HMD 10 without performing stitching processing.

[0039] When explaining the VR image display control of this present embodiment, hereinafter, for the sake of simplifying the explanation, an example of dividing into two regions in the left and right directions to expand the horizontal viewing angle (that is, an example using two real images captured by two cameras that image in the left and right directions respectively) will be described. Note that the number of divided regions may be two or more. For example, it may be divided into three directions: left, front, and right as described above, or further divided into four directions including the rear. More specifically, it may be divided into eight regions. Also, in order to expand the vertical viewing angle, for example, it may be divided into two directions (or three or more directions) in the up and down directions. Also, hereinafter, for the sake of simplifying the explanation, regarding the inclination of the HMD 10, only the yaw direction (left and right direction) will be considered without considering the pitch direction (up and down direction) and the roll direction (visual axis rotation direction). Also, when a stereoscopic VR image using binocular parallax is displayed on the display unit 11 of the HMD 10, a VR image for the right eye and a VR image for the left eye are respectively displayed. Hereinafter, for the sake of simplifying the explanation, the explanation will be made without distinguishing between the right eye and the left eye.

[0040] FIG. 7 is a diagram showing an example of imaging a 180-degree horizontal angle region by dividing it into two with two stereo cameras having different imaging directions. The horizontal angle of view θP1 of the first camera 100L that images the leftward region is 120 degrees, and the horizontal angle of view θP2 of the second camera 100R that images the rightward region is also 120 degrees. The angle formed by the optical axis direction XC1 of the first camera 100L and the optical axis direction XC2 of the second camera 100R is 60 degrees, and the first camera 100L and the second camera 100R are arranged close to each other.

[0041] FIG. 8 is a diagram conceptually showing the viewing area A1 of the first image captured by the first camera 100L and the viewing area A2 of the second image captured by the second camera 100R. Actually, in each of the viewing areas A1 of the first image and A2 of the second image, there are for the right eye and for the left eye, but hereinafter, it is assumed to be either for the right eye or for the left eye. Here, the "viewing area" is an area that can be displayed on the display unit 11 of the HMD 10 in the real captured image captured by one camera, and is an area where the real captured image is arranged in the VR space. In the example of FIG. 8, the viewing angle θA1 of the viewing area A1 of the first image and the viewing angle θA2 of the viewing area A2 of the second image are both 120 degrees. And, as illustrated in FIG. 9, the adjacent viewing areas A1 and A2 constitute a 180-degree overall viewing area.

[0042] FIG. 9 conceptually shows an example of the arrangement in the VR space V such that an overlapping region AO where a part of the visual field region A1 of the first image on the left side and a part of the visual field region A2 of the second image on the right side overlap each other. In other words, FIG. 9 conceptually shows the XY cross-section of the VR space V in which the first image and the second image are arranged (the same applies to the figures showing the VR space V shown later, such as FIG. 10, etc.). In the example of FIG. 9, the overall visual field region of the VR space V is 180 degrees, the viewing angle θA1 of the visual field region A1 of the first image is 120 degrees, and the viewing angle θA2 of the visual field region A2 of the second image is 120 degrees. Therefore, the viewing angle θAO of the overlapping region AO where both overlap is 60 degrees. The overlapping region AO is a region sandwiched between the region boundaries BD1 and BD2 of the first image and the second image. In the example of FIG. 9, the direction passing through the center of the overlapping region AO from the virtual viewpoint P in the VR space V is the direction of the X-axis. And in the overlapping region AO, either the first image or the second image will be selectively displayed. Also, the visual field region only of the first image (the region sandwiched between the Y-axis and the region boundary BD2), and the visual field region only of the second image (the region sandwiched between the Y-axis and the region boundary BD1) both have a viewing angle of 60 degrees.

[0043] FIG. 10 shows an example of displaying the first image (the visual field region A1 of the first image) in the overlapping region AO in the VR space V. Also, FIG. 11 shows an example of displaying the second image (the visual field region A2 of the second image) in the overlapping region AO in the VR space V. When the first image of the left visual field region A1 is selectively displayed in the overlapping region AO, as illustrated in FIG. 10, the boundary between the first image and the second image is the region boundary BD1 at the right end of the overlapping region AO. On the other hand, when the second image of the right visual field region A2 is selectively displayed in the overlapping region AO, as illustrated in FIG. 11, the boundary between the first image and the second image is the region boundary BD2 at the left end of the overlapping region AO. Thus, by selectively switching the real-world images (the first image, the second image) to be displayed in the overlapping region AO, the region boundaries (BD1, BD2) between adjacent real-world images change.

[0044] Regarding which of the first image or the second image to display in the overlapping area AO, it is determined based on the line-of-sight direction S. The line-of-sight direction S can be specified by detection information from the sensor 12 etc. of the HMD 10 (that is, depending on the orientation of the HMD 10). Therefore, taking the line-of-sight direction of the HMD 10 as the line-of-sight direction S, an example of switching the real-world image to be displayed in the overlapping area AO based on the yaw direction orientation of the HMD 10 will be described. In addition, in FIGS. 10 and 11, in order to show the relationship between the orientation of the HMD 10 (the line-of-sight direction of the HMD 10) and the display switching of the overlapping area AO in the VR space V, an image of the orientation of the HMD 10 is displayed near the virtual viewpoint P (the same applies to the figures representing the VR space V shown later).

[0045] In FIGS. 10 and 11, an example is shown where a "reference direction RD", which is the criterion for switching the real-world image to be displayed in the overlapping area AO, is set in the direction passing through the center of the overlapping area AO from the virtual viewpoint P. In the example of FIG. 10, the line-of-sight direction S (the line-of-sight direction of the HMD 10) is on the left side (that is, the first image side) of the reference direction RD. In this case, the first image of the left visual field area A1 is displayed in the overlapping area AO. Also in this case, the boundary between the visual field area A1 of the first image and the visual field area A2 of the second image becomes the area boundary BD1, so the area boundary BD1 is away from the line-of-sight direction S. On the other hand, in the example of FIG. 11, the line-of-sight direction S (the line-of-sight direction of the HMD 10) is on the right side (that is, the second image side) of the reference direction RD. In this case, the second image of the right visual field area A2 is displayed in the overlapping area AO. Also in this case, the boundary between the visual field area A1 of the first image and the visual field area A2 of the second image becomes the area boundary BD2, so the area boundary BD2 is away from the line-of-sight direction S.

[0046] As in the above example, taking a specific direction (in the above case, the direction passing through the center of the overlapping region AO from the virtual viewpoint P) as the reference direction RD, the real-world image (the first image or the second image) to be displayed in the overlapping region AO is dynamically switched while regarding the orientation of the HMD 10 (the line-of-sight direction of the HMD 10) as the line-of-sight direction S. Thereby, it is possible to move the seam (shift) of the image that is likely to occur at the region boundary (BD1 or BD2) of adjacent real-world images away from the direction of the user's line of sight, and it is possible to reduce the discomfort of the seam. Therefore, even if a close-range photographed object exists at or near the region boundary, a VR image with less discomfort is displayed for the user. Also, since the stitching process becomes unnecessary, the labor of producing the VR image can be reduced.

[0047] (Regarding the setting of the reference direction RD) In addition, in the above, an example in which the reference direction RD is set to the direction passing through the center of the overlapping region AO from the virtual viewpoint P has been shown, but it is not limited to this. The reference direction RD may be set in any direction from the virtual viewpoint P as long as it is within the visual field region where the first image and the second image in the VR space V are arranged.

[0048] For example, as shown in FIG. 12, it is also possible to set the reference direction RD to the direction of a position outside the overlapping region AO from the virtual viewpoint P. FIG. 12 shows an example in which the reference direction RD is set in the visual field region A2 of only the second image that does not include the overlapping region AO. Also in this case, if the line-of-sight direction S exceeds the reference direction RD due to a change in the orientation of the HMD 10 in the yaw direction (left-right direction) in FIG. 12, the real-world image displayed in the overlapping region AO, as exemplified in (B) in FIG. 12, switches from the first image in the visual field region A1 to the second image in the visual field region A2. Thereby, the region boundary of adjacent real-world images switches from the region boundary BD1 to the region boundary BD2, and the region boundary moves away from the line-of-sight direction S. However, in this case, until the line-of-sight direction S exceeds the reference direction RD in the state of (A) in FIG. 12, the line-of-sight direction S will cross the region boundary BD1. Therefore, before the real-world image displayed in the overlapping region AO switches from the first image to the second image, there will be a timing when the region boundary BD1 overlaps with the line-of-sight direction S.

[0049] Therefore, it is a preferred aspect to set the reference direction RD in the direction from the virtual viewpoint P to a predetermined position (which may or may not be the center of the overlapping region AO) within the overlapping region AO. In this case, by setting the reference direction RD within the overlapping region AO (see FIG. 10 or FIG. 11), even if the line-of-sight direction S is directed toward the current region boundary (BD1 or BD2) of the overlapping region AO, before reaching the region boundary, since the line-of-sight direction S exceeds the reference direction RD, the real image to be displayed in the overlapping region AO is switched. As a result, before the line-of-sight direction S reaches the region boundary, the region boundary moves away from the line-of-sight direction S.

[0050] (An aspect without setting a reference direction) Next, an aspect is shown in which, without setting the reference direction RD, the real image to be displayed in the overlapping region AO is dynamically switched between adjacent real images (the first image and the second image) based on the line-of-sight direction S. In this aspect, as illustrated in FIG. 13, based on the current region boundary (BD1 or BD2) of the adjacent real images, the real image to be displayed in the overlapping region AO is dynamically switched based on the line-of-sight direction S.

[0051] For example, as shown in (A) in FIG. 13, assume that the first image of the left visual field region A1 is displayed in the overlapping region AO and the line-of-sight direction S is on the X-axis. In this case, the switching criterion for the real image is the current region boundary BD1. From here, if the HMD 10 rotates in the right direction and, as illustrated in (B) in FIG. 13, the line-of-sight direction S (the visual axis of the HMD 10) exceeds the region boundary BD1, the real image displayed in the overlapping region AO switches from the first image to the second image of the right visual field region A2. As a result, the switching criterion for the real image is changed to the current region boundary BD2. From here, if the HMD 10 rotates in the left direction and, as illustrated in (C) in FIG. 13, even if the line-of-sight direction S returns to the X-axis and does not exceed the current region boundary BD2, the real image displayed in the overlapping region AO does not switch. From here, if the HMD 10 rotates further in the left direction and, as illustrated in (D) in FIG. 13, the line-of-sight direction S exceeds the region boundary BD2, the real image displayed in the overlapping region AO switches from the second image to the first image.

[0052] In this way, based on the current region boundary (BD1 or BD2) of adjacent real-world images (the first image and the second image), if the line-of-sight direction S is on the first-image side (left side) of the current region boundary (BD1 or BD2), the first image is dynamically switched to be displayed in the overlapping region AO, and if it is on the second-image side (right side), the second image is displayed in the overlapping region AO.

[0053] Also, as a modification, the real-world image to be displayed in the overlapping region AO may be dynamically switched according to the following mode. That is, when the line-of-sight direction S approaches the current region boundary (BD1 or BD2) of adjacent real-world images by a predetermined amount or more (for example, when the angle formed by the line-of-sight direction S and the current region boundary becomes a predetermined value or less), the real-world image currently displayed in the overlapping region AO may be switched to the other real-world image. In the case of this mode, the real-world image to be displayed in the overlapping region AO is switched before the line-of-sight direction S reaches the region boundary, and the region boundary moves away from the line-of-sight direction S.

[0054] In the description using FIGS. 9 to 13, the display control for dividing the horizontal direction into a plurality of regions to expand the horizontal viewing angle has been described. However, in the case of display control for dividing the vertical direction into a plurality of regions to expand the vertical viewing angle, the same image switching process for the overlapping region as described above may be performed based on the orientation of the HMD 10 in the pitch direction (vertical direction).

[0055] [5. Functional Configuration of Display Control Device] FIG. 14 is a schematic functional block diagram showing an example of the functional configuration of the display control device 20. The display control device 20 executes control for displaying, on the display unit 11 of the HMD 10, a VR image representing a visual field from a virtual viewpoint in the VR space as a stereoscopic image using binocular parallax. As shown in FIG. 14, the display control device 20 includes a control unit 30. This control unit 30 is realized, for example, by the processor 21 executing a program stored in the storage device 22. The control unit 30 includes a line-of-sight direction specifying unit 31 and an image generation unit 32.

[0056] The line-of-sight direction specifying unit 31 has a function of specifying a line-of-sight direction that is assumed to be the direction of the user's line of sight with respect to the VR space. Here, the "line-of-sight direction" is a direction specified assuming that it is the direction of the user's line of sight with respect to the VR space of the user wearing the HMD 10. The "line-of-sight direction" only needs to be estimated to be substantially the direction of the user's line of sight based on detection information such as the detection information of the sensor 12 of the HMD 10, etc., and may coincide with the direction of the actual user's line of sight when wearing the HMD 10, or may deviate from the direction of the actual user's line of sight.

[0057] For example, the line-of-sight direction specifying unit 31 can acquire detection information regarding the orientation of the HMD 10 and specify the line-of-sight direction based on the detection information. Here, the "detection information regarding the orientation of the HMD 10" is detection information regarding the orientation of the HMD 10 that changes when the user changes the orientation of the head in a state where the HMD 10 is worn by the user. The above-mentioned "detection information" can be acquired from detection means such as sensors that detect the direction or change in direction of the HMD 10. For example, the detection result of the sensor 12 (angular velocity sensor, acceleration sensor, geomagnetic sensor, etc.) built into the HMD 10 corresponds to an example of the "detection information regarding the orientation of the HMD 10". Also, the measurement result of a tracking system that images the HMD 10 with an imaging unit provided outside the HMD 10 and measures the position and orientation of the HMD 10 corresponds to an example of the "detection information regarding the orientation of the HMD 10". Also, the measurement result of a tracking system that analyzes the surrounding objects imaged by an imaging unit mounted on the HMD 10 itself and measures the position and orientation of the HMD 10 corresponds to an example of the "detection information regarding the orientation of the HMD 10".

[0058] Also, the line-of-sight direction specifying unit 31 can acquire detection information regarding the user's line of sight and specify the line-of-sight direction based on the detection information. Here, the "detection information regarding the user's line of sight" is information that detects the direction, movement, etc. of the user's line of sight in a state where the HMD 10 is worn by the user. For example, the measurement result of an eye tracking system that tracks the direction and movement of the line of sight based on the position of the user's eyeballs corresponds to an example of the "detection information regarding the user's line of sight".

[0059] The image generation unit 32 has a function of arranging a plurality of real captured images captured by a plurality of cameras having different imaging directions for imaging the real space in a VR space so that a partial overlapping area where the visual field areas of adjacent real captured images overlap each other is generated, and generating a VR image corresponding to the line-of-sight direction.

[0060] Here, the “camera” refers to a real camera that images the real space. Note that the “camera” itself is not included in the components of the display control device 20. The “camera” includes a left-eye optical system (left-eye camera) that images an image for the left eye and a right-eye optical system (right-eye camera) that images an image for the right eye in order to obtain a stereoscopic image using binocular parallax. For example, like a stereo camera, one camera may be provided with a left-eye optical system and a right-eye optical system. Also, the left-eye camera and the right-eye camera may have different configurations. In order to obtain a VR image with a wide viewing angle that enables stereoscopic viewing from a plurality of real captured images, a plurality of cameras are used for each of the left eye and the right eye. For example, when obtaining a wide-viewing-angle image with a wide horizontal viewing angle, a plurality of cameras may be arranged horizontally in the real space so that the imaging directions are different from each other. Also, for example, when obtaining a wide-viewing-angle image with a wide vertical viewing angle, a plurality of cameras may be arranged vertically in the real space so that the imaging directions are different from each other. For each of the left eye and the right eye, the number of cameras can be any number as long as it is 2 or more. The angle of view of each camera is also arbitrary. The angles of view of the plurality of cameras may all be the same or may not be the same.

[0061] Also, the “imaging direction” corresponds to the direction in which the camera that images the real space images, and refers to the front direction (the direction indicating the subject side) on the optical axis of the imaging optical system of the camera.

[0062] In addition, the "adjacent real-world images" refer to two real-world images captured by two adjacent cameras with different imaging directions in the real space and partially overlapping imaging ranges. Here, the two adjacent cameras do not refer to the left-eye camera and the right-eye camera, but rather two cameras that both capture real-world images for the left eye, or two cameras that both capture real-world images for the right eye. The "real-world image" may be a video or a still image. Also, the "real-world image" may be a pre-captured image (video) or a live image (live video) captured in substantially real time.

[0063] In addition, the "field of view area" is an area that can be displayed on the display unit of the head-mounted display in the real-world image captured by one camera, and is an area where the real-world image is arranged in the VR space. For example, the field of view area of a real-world image captured by a camera with a horizontal viewing angle θH1 (e.g., 120 degrees) and a vertical viewing angle θV1 (e.g., 120 degrees) is basically an area having a horizontal field of view angle θH2 (=θH1) and a vertical field of view angle θV2 (=θV1). Note that when arranging the field of view area of the real-world image captured by a camera with a horizontal viewing angle θH1 and a vertical viewing angle θV1 in the VR space, it may be narrowed to a horizontal field of view angle θH2 (<θH1) and a vertical field of view angle θV2 (<θV1). For example, by not using the area near the image edge where distortion due to optical system aberration is likely to occur, as described above, the field of view area of the real-world image may be narrowed when arranged in the VR space.

[0064] In addition, the "overlap area" refers to an area in the VR space where two adjacent real-world images are partially overlapped and arranged. In the examples of FIGS. 9 to 13, an example where the viewing angle of the overlap area AO is 60 degrees is shown, but the overlap area AO may be narrower than 60 degrees or wider than 60 degrees. The size (viewing angle) of the "overlap area" can be arbitrarily set. As will be described later, when there are a plurality of overlap areas in the VR space, the sizes (viewing angles) of all the overlap areas may be the same, or the size of at least one overlap area may be different from the sizes of the other overlap areas.

[0065] The image generation unit 32 includes an image arrangement unit 321 that arranges a plurality of real-world images in the VR space such that a partial overlapping area where a part of the viewing areas of adjacent real-world images overlaps with each other is generated. The image arrangement unit 321 executes storage control for storing each of the plurality of real-world images in a predetermined storage area (a storage area for storing each of the plurality of real-world images arranged in the VR space) of a storage device 22 (such as VRAM) for forming the VR space. In the examples of FIGS. 9 to 13, the image arrangement unit 321 stores the data of the first image and the second image in a predetermined storage area for forming the VR space in the storage device 22.

[0066] In the present embodiment, together with the data of the plurality of real-world images to be displayed, arrangement information for arranging the plurality of real-world images in the VR space is stored in the storage device 22 in association with the plurality of real-world images. Alternatively, the arrangement information is also recorded on a recording medium that records a plurality of images to be displayed. When the processor 21 of the display control device 20 reads the data of the plurality of images to be displayed from the recording medium and stores it in the storage device 22, the arrangement information is also read out and stored in the storage device 22 for use in the image arrangement process in the VR space. Alternatively, as will be described later, when a live video is displayed on the HMD 10 as a VR image in substantially real time, the arrangement information is also input (received) together with the image of the live video to be displayed and used for the image arrangement process in the VR space.

[0067] Examples of the arrangement information include information such as the number of viewing areas (number of images) arranged in the VR space, the viewing angle of the entire viewing area, the viewing angle of each viewing area, the direction (position in the VR space) of each viewing area in the VR space, the number of overlapping areas, the viewing angle of each overlapping area, or the direction (position) of each overlapping area in the VR space. For example, the arrangement information may be recorded in the header portion of the real-world image file. Alternatively, the arrangement information may be recorded as information associated with the real-world image in a file separate from the real-world image file (data).

[0068] In addition, if it is premised that the display of the VR image (video) of the present embodiment is normalized (standardized) in advance and a real-shot image that conforms to the standard is used, it is not necessary to associate the above-described detailed arrangement information with a plurality of real-shot images to be displayed and save it together with the real-shot images. That is, the number of viewing regions (number of images) arranged in the VR space, the viewing angle of the entire viewing region, the viewing angle of each viewing region, the direction (position in the VR space) of each viewing region in the VR space, the number of overlapping regions, the viewing angle of each overlapping region, or the direction (position) of each overlapping region in the VR space, etc., are defined in advance as standards, and the image arrangement unit 321 can arrange each image in the VR space based on the information of the standards.

[0069] Also, for example, a plurality of standards may be defined, such as a first standard with a viewing angle of 180 degrees for the entire viewing region, a second standard with a viewing angle of 220 degrees, a third standard with a viewing angle of 270 degrees, and a fourth standard with a viewing angle of 360 degrees. In this case, the image to be displayed only needs to be associated with standard information indicating which standard the captured image conforms to. In this case, each standard information corresponds to an example of the above-described "arrangement information".

[0070] Also, a standard with a certain degree of freedom may be used. For example, only a part of the above-described arrangement information may be standardized, and the other items may be arbitrarily set or changed. For example, the viewing angle of the entire viewing region is defined in advance by standardization, but the number of viewing regions (number of images) arranged in the VR space, the viewing angle of each viewing region, etc. can be arbitrarily set. In this case, the content of the standard information and the arbitrarily set items may be used as "arrangement information" and associated with the image to be displayed.

[0071] In addition, the image generation unit 32 includes a switching unit 322. This switching unit 322 has a function of dynamically switching the real-shot image to be displayed in the overlapping region among adjacent real-shot images based on the viewing direction specified by the viewing direction specifying unit 31. The switching unit 322 dynamically switches the real-shot image to be displayed in the overlapping region among adjacent real-shot images so that the region boundary of the adjacent real-shot images moves away from the viewing direction based on the viewing direction.

[0072] Here, "dynamically switching the real-world image to be displayed in the overlapping region based on the line-of-sight direction" means that, among two adjacent real-world images (the first image and the second image) that can be displayed in the overlapping region, which real-world image to display in the overlapping region is specified each time based on the line-of-sight direction, and the real-world image displayed in the overlapping region is dynamically switched according to the change in the line-of-sight direction. For example, as shown in FIGS. 10 to 12, a reference direction RD serving as a criterion for switching between the adjacent first image and the second image is set. When the line-of-sight direction S is on the first-image side of the reference direction RD, the first image is displayed in the overlapping region AO; when it is on the second-image side, the second image is displayed. This corresponds to an example of "dynamically switching the real-world image to be displayed in the overlapping region based on the line-of-sight direction among adjacent real-world images".

[0073] Alternatively, without setting a reference direction, the real-world image to be displayed in the overlapping region may be dynamically switched based on the line-of-sight direction. For example, as illustrated in FIG. 13, taking the current region boundary (BD1 or BD2) between adjacent real-world images (the first image and the second image) as a reference, if the line-of-sight direction S is on the first-image side of the current region boundary, the first image is displayed in the overlapping region AO; if it is on the second-image side, the second image is displayed. This corresponds to an example of "dynamically switching the real-world image to be displayed in the overlapping region based on the line-of-sight direction among adjacent real-world images". Also, for example, when the line-of-sight direction approaches the current region boundary between adjacent real-world images by a predetermined amount or more (when the angle formed by the line-of-sight direction and the region boundary becomes a predetermined value or less), switching the real-world image currently displayed in the overlapping region to the other real-world image corresponds to an example of "dynamically switching the real-world image to be displayed in the overlapping region based on the line-of-sight direction among adjacent real-world images".

[0074] Also, for the process of "switching the real - life images to be displayed in the overlapping area", for example, in the overlapping area, two adjacent real - life images are arranged in different upper and lower layers. Then, without changing the transparency of the real - life image arranged in the lower layer (background) from 0%, the transparency of the real - life image arranged in the upper layer (foreground) is switched to either 100% or 0%. That is, when displaying the real - life image arranged in the lower layer in the overlapping area, the transparency of the upper layer is set to 100%, and when displaying the real - life image arranged in the upper layer, the transparency of the upper layer is set to 0%. In this way, in the overlapping area, arranging two adjacent real - life images in different upper and lower layers and switching the transparency of the real - life image arranged in the upper layer to either 100% or 0% corresponds to an example of "switching the real - life images to be displayed in the overlapping area". Also, two adjacent real - life images can be superimposed and arranged in the overlapping area, with the transparency of the real - life image to be displayed set to 0% and the transparency of the other real - life image set to 100%. That is, setting the transparency of one of the two superimposed real - life images to 0% and the transparency of the other real - life image to 100% and switching the transparency according to the real - life image to be displayed corresponds to an example of "switching the real - life images to be displayed in the overlapping area". Also, instead of superimposing and arranging two adjacent real - life images in the overlapping area, only the part of the real - life image to be displayed corresponding to the overlapping area is arranged in the overlapping area, and switching the arranged real - life image corresponds to an example of "switching the real - life images to be displayed in the overlapping area".

[0075] The switching period for switching the real - life images in the overlapping area can be set arbitrarily. It can be switched suddenly by making the switching period approximately zero, or a predetermined switching period (for example, 0.3 seconds, etc.) can be set. Also, when switching the real - life images in the overlapping area, there may be a period during which both of the two adjacent real - life images are displayed, for example, by semi - transparent synthesis or the like.

[0076] In addition, the image generation unit 32 (the switching unit 322 of the image generation unit 32) has a function of setting a reference direction in a predetermined direction from the virtual viewpoint and specifying a real image to be displayed in the overlapping area based on the line-of-sight direction with respect to the reference direction. Here, the "reference direction" is a direction set for determining whether to switch the real image to be displayed in the overlapping area in relation to the line-of-sight direction. The "reference direction" can be set in a predetermined direction from the virtual viewpoint in the VR space, and it can be set in any direction as long as it is within the visual field area where adjacent real images in the VR space are arranged. For example, as illustrated in FIGS. 10 and 11, the direction RD passing through the center of the overlapping area AO from the virtual viewpoint P in the VR space corresponds to an example of the "reference direction". The "reference direction" may be fixed, or may not be fixed and may change as described later (the predetermined direction changes).

[0077] For example, as shown in FIG. 10, when the line-of-sight direction S is on the left side of the reference direction RD (the visual field area A1 side of the first image), the image generation unit 32 specifies that the real image to be displayed in the overlapping area AO is the first image. Also, as illustrated in FIG. 11, when the line-of-sight direction S is on the right side of the reference direction RD (the visual field area A2 side of the second image), the image generation unit 32 specifies that the real image to be displayed in the overlapping area AO is the second image.

[0078] It is preferable that the image generation unit 32 (the switching unit 322 of the image generation unit 32) sets the reference direction in the direction of a predetermined position within the overlapping area from the virtual viewpoint. Here, "setting the reference direction in the direction of a predetermined position within the overlapping area from the virtual viewpoint" means setting the reference direction within the overlapping area (setting the reference direction so that the vector indicating the reference direction exists within the overlapping area). In this case, as illustrated in FIGS. 10 and 11, since the reference direction RD is set within the overlapping area AO, even if the line-of-sight direction S is directed toward the area boundary (BD1 or BD2) of the overlapping area AO, before reaching the area boundary, the real image to be displayed in the overlapping area AO is switched, and thus the area boundary also moves away from the line-of-sight direction S.

[0079] In addition, the image generation unit 32 has a function of acquiring detection information regarding the orientation of the HMD 10 and changing the range of the VR image to be displayed on the display unit 11 of the HMD 10 based on the detection information. For example, the image generation unit 32 changes the direction of the field of view from the virtual viewpoint P in the VR space based on the detection result of the sensor 12 such as an angular velocity sensor incorporated in the HMD 10, and changes the range of the VR image to be displayed on the display unit 11 of the HMD 10. For example, regarding the direction of the visual axis of the HMD 10 as the direction of the field of view, the range of the VR image to be displayed on the display unit 11 of the HMD 10 is changed according to the orientation of the HMD 10.

[0080] [6. Processing] Next, an example of the processing executed by the display control device 20 of the present embodiment will be described below. FIG. 15 is a flowchart showing an example of the processing of the display control device 20. The processing described below is realized by the control unit 30 (the processor 21 of the display control device 20) executing the program stored in the storage device 22 (the same applies to the processing with reference to the flowcharts of FIGS. 17, 23, 24, 27, 29, or 33). Here, an example of the VR image display control processing described with reference to FIGS. 9 to 12 will be described.

[0081] The control unit 30 sets a reference direction RD in the VR space V as illustrated in FIG. 10 or FIG. 12 etc. (S100). Data of the real-shot images (the first image and the second image) captured by two cameras with different imaging directions for imaging the real space are stored, for example, in the storage device 22 or a recording medium etc. The control unit 30 reads out the first image, the second image, and the arrangement information from the storage device 22 etc. Then, based on the arrangement information, the control unit 30 arranges the first image and the second image in the VR space V so that an overlapping region AO where a part of the visual field regions A1·A2 of the first image and the second image overlap each other is generated (S102). For example, in the overlapping region AO, the control unit 30 arranges the first image and the second image in different upper and lower layers. For example, the first image is arranged as the lower layer and the second image is arranged as the upper layer.

[0082] Further, the control unit 30 acquires detection information regarding the orientation of the HMD 10 from the sensor 12 (e.g., angular velocity sensor) of the HMD 10 (S104), and specifies the line-of-sight direction S (S106). Then, the control unit 30 determines whether the line-of-sight direction S is on the left side (first image side) of the reference direction RD (S108). If YES, the control unit 30 causes the first image to be displayed in the overlapping area AO as illustrated in FIG. 10 (S110). For example, in the overlapping area AO, the first image is caused to be displayed by fixing the transparency of the first image at 0% and setting the transparency of the second image arranged in a layer above the first image at 100%. On the other hand, if the line-of-sight direction S is on the right side (second image side) of the reference direction RD (NO in S108), the control unit 30 causes the second image to be displayed in the overlapping area AO as illustrated in FIG. 11 (S112). For example, in the overlapping area AO, the second image is caused to be displayed by setting the transparency of the second image arranged in a layer above the first image at 0%. In this way, the control unit 30 executes control to dynamically switch the real-world image (first image and second image) to be displayed in the overlapping area AO based on the line-of-sight direction S.

[0083] Then, the control unit 30 generates a VR image in a range corresponding to the orientation of the HMD 10 based on the detection information regarding the orientation of the HMD 10 in S104, and outputs the VR image to the HMD 10 (S114). For example, the control unit 30 outputs the VR image generated by performing rendering processing or the like to the HMD 10. As a result, the VR image is displayed on the HMD 10. The processes of S102 to S114 are repeated until the display ends (YES in S116) due to a playback stop operation being performed by the user or the like.

[0084] [7. Mode of Changing the Reference Direction] In the above description, an example of setting a fixed reference direction RD was shown. When the reference direction RD is fixed, if the line-of-sight direction (gaze direction S) of the HMD 10 is near the reference direction RD, the real-world images displayed in the overlapping region AO may frequently switch, which may result in an image that is difficult for the user to view. Therefore, at the timing when the switching of the real-world image to be displayed in the overlapping region AO is executed, the reference direction RD is changed so as to move away from the line-of-sight direction (gaze direction S) of the HMD 10. This will be described below.

[0085] As illustrated in (A) of FIG. 16, when the first image of the left visual field region A1 is displayed in the overlapping region AO, the reference direction RD1 is set in a direction shifted to the right (e.g., -20 degrees) from the direction passing through the center of the overlapping region AO from the virtual viewpoint P. Note that the angle is positive in the counterclockwise (leftward) direction. From this state, as the line-of-sight direction (gaze direction S) of the HMD 10 rotates to the right and the gaze direction S exceeds the reference direction RD1, as illustrated in (B) of FIG. 16, the real-world image to be displayed in the overlapping region AO switches from the first image to the second image of the visual field region A2. At this switching timing (simultaneously with the switching), the reference direction moves leftward so as to move away from the line-of-sight direction (gaze direction S) of the HMD 10, and is changed from the reference direction RD1 in (A) of FIG. 16 to the reference direction RD2 in (B) of FIG. 16. That is, as illustrated in (B) of FIG. 16, when the second image of the right visual field region A2 is displayed in the overlapping region AO, the reference direction RD is set in a direction shifted to the left (e.g., 20 degrees) from the direction passing through the center of the overlapping region AO from the virtual viewpoint P.

[0086] Also, from the state in (B) of FIG. 16, as the line-of-sight direction (gaze direction S) of the HMD 10 rotates to the left and the gaze direction S exceeds the reference direction RD2, as shown in (A) of FIG. 16, when the real-world image to be displayed in the overlapping region AO switches from the second image to the first image of the visual field region A1, the reference direction RD2 is changed to the reference direction RD1 at the same time. Note that "at the same time" includes being substantially simultaneous.

[0087] This is a mode of performing hysteresis control by changing the reference direction (RD1 or RD2) at the timing when the real image displayed in the overlapping region AO is switched. Also, when the real image displayed in the overlapping region AO is switched, there is also a mode of moving the reference direction RD toward the switched region boundary (BD1 or BD2) so that the reference direction moves away from the line-of-sight direction S. Also, there is a mode of changing the reference direction (RD1 or RD2) according to the real image displayed in the overlapping region AO (depending on whether it is the first image of the visual field region A1 or the second image of the visual field region A2).

[0088] The image generation unit 32 has a function of changing the reference direction (RD1 or RD2) according to the real image (depending on whether it is the first image of the visual field region A1 or the second image of the visual field region A2) to be displayed in the overlapping region AO at the timing when the real image to be displayed in the overlapping region AO is switched.

[0089] Next, with reference to FIG. 17, an example of the process of changing the above-described reference direction will be described. FIG. 17 is a flowchart showing an example of the process of changing the reference direction executed by the control unit 30 according to the present embodiment.

[0090] The control unit 30 sets the reference direction (RD1 or RD2) corresponding to the real image (the first image of the visual field region A1 or the second image of the visual field region A2) displayed in the overlapping region AO as illustrated in (A) or (B) in FIG. 16 (S200). Also, the control unit 30 acquires detection information regarding the orientation of the HMD 10 from the sensor 12 (for example, an angular velocity sensor) of the HMD 10 (S202), and specifies the line-of-sight direction S (S204). Further, the control unit 30 determines whether it is necessary to switch the real image to be displayed in the overlapping region AO based on the line-of-sight direction S with respect to the current reference direction (RD1 or RD2) (S206). If YES in this S206, the control unit 30 executes the switching of the real image to be displayed in the overlapping region AO (S208) and changes the reference direction to the direction corresponding to the real image after the switching (S210). On the other hand, if NO in S206, the process returns to S202. The processes of S202 to S210 are repeated until the display ends (YES in S212), for example, when a user performs a playback stop operation.

[0091] According to the above aspect, the reference direction (RD1 or RD2) is not fixed and is changed according to the live-action image displayed in the overlapping region AO at the timing when the live-action image displayed in the overlapping region AO is switched. As a result, after the live-action image is switched, the reference direction temporarily moves away from the line-of-sight direction S, so that it is possible to reduce the frequent switching of the live-action image displayed in the overlapping region AO.

[0092] [8. Aspect of visibly displaying the current region boundary] In the image display system 1 of the present embodiment, the live-action image (the first image or the second image) displayed in the overlapping region AO is switched to move the region boundary (BD1 or BD2) of the adjacent live-action images away from the line-of-sight direction S, but the current region boundary may be deliberately displayed visibly. This will be described below.

[0093] FIGS. 18 and 19 are diagrams showing an example of a screen displayed on the display unit 11 of the HMD 10. On the screen G10 illustrated in FIG. 18, a VR image of the range of the field of view corresponding to the orientation of the HMD 10 in the VR space V illustrated in FIG. 10 is displayed. That is, on the screen G10, a VR image representing the field of view in the viewing axis direction (line-of-sight direction S) of the HMD 10 from the virtual viewpoint P in the VR space V of FIG. 10 is displayed. Therefore, it can be said that the first image of the left field-of-view region A1 is displayed in the overlapping region AO. On this screen G10, a boundary line BL1 is visibly displayed at a portion corresponding to the current region boundary BD1 (the right end of the overlapping region AO) between adjacent live-action images (the first image and the second image). For example, the boundary line BL1 is a solid black line having a line width recognizable by the user.

[0094] When the user viewing the screen G10 in FIG. 18 rotates the head (HMD10) to the right, the screen displayed is the screen G11 illustrated in FIG. 19. In this screen G11, a VR image within the field of view corresponding to the orientation of the HMD10 in the VR space V illustrated in FIG. 11 is displayed. That is, since the viewing axis direction (line-of-sight direction S) of the HMD10 has rotated to the right and exceeded the reference direction RD, a VR image in which the real-world image displayed in the overlapping region AO has been switched from the VR image of the screen G10 in FIG. 18 is displayed on the screen G11 illustrated in FIG. 19. On this screen G11, a boundary line BL2 is visibly displayed at a position corresponding to the current region boundary BD2 (the left end of the overlapping region AO) between adjacent real-world images (the first image and the second image). For example, the boundary line BL2 is a solid black line having the same line width as the boundary line BL1.

[0095] As illustrated in FIG. 20, the display control device 20 can be configured to include a boundary line display unit 33. This boundary line display unit 33 has a function of displaying a boundary line visible to the user at the region boundary between adjacent real-world images. Here, the “region boundary between adjacent real-world images” is the boundary between two adjacent real-world images (the first image and the second image) arranged in the VR space. As illustrated in FIGS. 10 and 11, the position of the region boundary (BD1 or BD2) changes depending on the real-world image (the first image in the viewing region A1 or the second image in the viewing region A2) displayed in the overlapping region AO. Also, the “boundary line” is a line displayed at the region boundary so that the user can visually recognize the region boundary between the adjacent real-world images. The color, line width, transparency, etc. of the “boundary line” can be arbitrarily set. Also, the “boundary line” can have an arbitrarily set line type, such as a solid line, a dotted line, a dashed line, a wavy line, a double line, etc. Also, the “boundary line” may be a two-dimensional line or a three-dimensional (three-dimensional) raised line.

[0096] The boundary line display unit 33 arranges the boundary line (BD1 or BD2) at the portion of the current region boundary (BD1 or BD2) so that the boundary line is superimposed on the real-world VR image and displayed on the display unit 11.

[0097] The boundary (BD1 or BD2) of the area where an image seam (shift) occurs is changed by switching the real photographed image displayed in the overlapping area AO based on the line-of-sight direction S, as illustrated in FIGS. 10 and 11. That is, by the user changing the line-of-sight direction S of their own volition, the boundary (BD1 or BD2) of the area where an image seam (shift) occurs can be changed. That is, if there is an image seam in the direction the user wants to view, by the user changing the line-of-sight direction S of their own volition, the image seam in the desired direction can be eliminated to make the image easier to view. According to this aspect, by deliberately displaying the boundary lines (BL1 or BL2) visibly at the boundary (BD1 or BD2) between adjacent real photographed images, it becomes easier for the user to recognize the boundary of the area where an image seam (shift) occurs. The boundary lines (BL1 or BL2) at the seam of the image can be changed by the user changing the line-of-sight direction S of their own volition, as illustrated in FIGS. 18 and 19, and even after the change, it becomes easier for the user to recognize where the image seam has moved. That is, by visibly displaying the boundary lines (BL1 or BL2) at the boundary (BD1 or BD2), it is possible to facilitate the user's judgment as to which real photographed image should be displayed in the overlapping area AO of the adjacent real photographed images so that the boundary line is not displayed in the direction the user wants to view.

[0098] When the line-of-sight direction S is specified based on the detection information regarding the orientation of the HMD10 (for example, the detection information of the angular velocity sensor), the user may adjust the orientation of the head wearing the HMD10 and adjust the range of the VR image displayed on the display unit 11 of the HMD10 while changing the boundary line (BL1 or BL2).

[0099] Also, when the line-of-sight direction S is specified based on detection information regarding the user's line of sight (e.g., detection information by eye tracking), the user can change the boundary line (BL1 or BL2) displayed at the region boundary (BD1 or BD2) just by changing the line of sight without moving the head. For example, in a configuration where the range of the VR image to be displayed on the display unit 11 is changed based on detection information regarding the orientation of the HMD10 (e.g., detection information of an angular velocity sensor) and the line-of-sight direction S is specified based on detection information regarding the user's line of sight (e.g., detection information by eye tracking), the user can adjust an easy-to-view image as follows. That is, the user adjusts the orientation of the head to adjust the range of the VR image so that the image in the desired direction is displayed on the display unit 11 of the HMD10. Then, with the image in the desired direction being displayed on the display unit 11, if the user adjusts the line-of-sight direction and changes the boundary line (BL1 or BL2), even if there is a boundary line in the desired direction, it can be moved to display an easy-to-view image. In addition, in a configuration where the range of the VR image to be displayed on the display unit 11 is changed and the real-world image to be displayed in the overlapping region AO is switched based on detection information regarding the user's line of sight by eye tracking or the like, the user can adjust an easy-to-view image just by adjusting the line-of-sight direction.

[0100] (Mode of displaying additional information at or near the boundary line) Next, a mode that can further facilitate the user's determination of which real-world image should be displayed in the overlapping region AO among adjacent real-world images will be described below.

[0101] For example, on the screen G10 illustrated in FIG. 18, a boundary line BL1 is displayed. However, it is difficult for the user to immediately recognize whether the left side or the right side of the boundary line BL1 is the overlapping region AO. Also, among the adjacent real-shot images (the first image on the left or the second image on the right), it is difficult for the user to immediately recognize which real-shot image is currently displayed in the overlapping region AO. Further, in order to cause a switching of the real-shot image in the overlapping region AO (i.e., a change of the boundary line), it is difficult for the user to immediately recognize in which direction the line-of-sight direction S should be changed. Even if the above is difficult to immediately recognize, if the user changes the line-of-sight direction S to the left or right, the screen changes from the screen G10 illustrated in FIG. 18 to the screen G11 illustrated in FIG. 19, or from the screen G11 to the screen G10, and thus the user can recognize it. In this aspect, by displaying additional information at or near the boundary line, it becomes easier for the user to recognize in which direction the line-of-sight direction S should be changed.

[0102] FIGS. 21 and 22 are diagrams showing an example of a screen in which a gradation portion is displayed together with a boundary line. FIG. 21 is a screen G20 corresponding to the screen G10 illustrated in FIG. 18, and shows an example of a screen in which a gradation portion GD1 is displayed as additional information on the right side of the boundary line BL1.

[0103] The screen G20 in FIG. 21 corresponds to the VR space V in FIG. 10. Therefore, the left side of the boundary line BL1 corresponds to the overlapping region AO where the first image of the left visual field region A1 is displayed, and the region on the right side of the boundary line BL1 is the side that is not the overlapping region AO (the region of only the right visual field region A2). That is, the gradation portion GD1 of the screen G20 is displayed on the side opposite to the overlapping region AO across the boundary line BL1. This gradation portion GD1 is a translucent region that displays a black gradation in which the density or opacity continuously or stepwise decreases as it moves away from the boundary line BL1 (towards the right in FIG. 21). The width (distance in the direction orthogonal to the boundary line BL1) and color of the gradation portion GD1 can be arbitrarily set (the same applies to the gradation portion GD2 below).

[0104] Further, FIG. 22 is a view showing an example of a screen G21 corresponding to the screen G11 illustrated in FIG. 19, and shows an example of a screen in which a gradation portion GD2 is displayed as additional information on the left side of the boundary line BL2. The screen G21 of this FIG. 22 corresponds to the VR space V of FIG. 11. Therefore, the right side of the boundary line BL2 corresponds to "the overlapping region AO where the second image of the right visual field region A2 is displayed", and the region on the left side of the boundary line BL2 is the side that is not the overlapping region AO (the region of only the left visual field region A1). That is, the gradation portion GD2 of the screen G21 is displayed on the side opposite to the overlapping region AO with the boundary line BL2 interposed therebetween. This gradation portion GD2 is a translucent region that displays a gradation in which the density or opacity continuously or stepwise decreases as it moves away from the boundary line BL2 (toward the left in FIG. 22). The color of the gradation portion GD2 is, for example, a black gradation, but any color can be set.

[0105] In this way, the gradation portion (GD1 or GD2) is displayed on the side opposite to the overlapping region AO with the boundary line (BL1 or BL2) interposed therebetween. As a result, the user can recognize that the side opposite to the gradation portion (GD1 or GD2) of the boundary line (BL1 or BL2) (the left side of the boundary line BL1 in FIG. 21, the right side of the boundary line BL2 in FIG. 22) is the overlapping region AO. Also, the user can recognize that a real image (the first image on the left side in FIG. 21, the second image on the right side in FIG. 22) on the side opposite to the gradation portion (GD1 or GD2) with the boundary line (BL1 or BL2) interposed therebetween is currently being displayed in the overlapping region AO. Further, the user can intuitively recognize that when the line of sight direction S is moved in the direction in which the gradation portion (GD1 or GD2) becomes thinner (in the direction from the boundary line BL1 to the direction where the gradation portion is located), a switching (change of the boundary line) of the real image in the overlapping region AO can be caused.

[0106] That is, the gradation part (GD1 or GD2) is "information for indicating which side is the overlapping area across the boundary line", "information for indicating which of the adjacent real images is the real image displayed in the overlapping area", or "information for indicating in which direction the line of sight should be changed to cause switching of the real image in the overlapping area".

[0107] In addition, since the gradation part (GD1 or GD2) is displayed so as to overlap with the displaced part of the image that occurs at the boundary line (BL1 or BL2), it also has the effect of making the displacement of the image less noticeable. Also, the change in the density or opacity of the gradation part (GD1 or GD2) also serves to naturally represent the direction of the change in the line of sight S required for switching the real image in the overlapping area AO. Therefore, applying the gradation part (GD1 or GD2) as additional information to be displayed together with the boundary line is one of the preferred embodiments.

[0108] Note that as long as it can be recognized which side of the boundary line (BL1 or BL2) the overlapping area AO is on, the gradation part (GD1 or GD2) may be displayed on the overlapping area AO side across the boundary line (BL1 or BL2).

[0109] The boundary line display unit 33 in this aspect has a function of displaying additional information for indicating which side of the boundary line is the overlapping region across the boundary line near the boundary line. Here, the "additional information for indicating which side of the boundary line is the overlapping region" is additional information of the boundary line displayed near the boundary line or on it, and currently, it is information for enabling the user to recognize which side of the boundary line is the overlapping region. For example, as described above, the gradation portion (GD1 or GD2) displayed on the opposite side (or the overlapping region AO side) of the overlapping region AO across the boundary line (BL1 or BL2) is an example of the "additional information for indicating which side of the boundary line is the overlapping region". Note that the additional information is not limited to the gradation portion (GD1 or GD2). For example, an "arrow" or a "symbol" indicating which side of the boundary line is the overlapping region may be displayed over the boundary line (BL1 or BL2) or near the boundary line. In this case, the "arrow" or "symbol" etc. corresponds to an example of the additional information.

[0110] In addition, the additional information such as the "arrow" or "symbol" is also an example of the "information for indicating which of the adjacent real - world images is the one displayed in the overlapping region" or the "information for indicating in which direction the line of sight should be changed to cause the switching of the real - world images in the overlapping region".

[0111] The boundary line display unit 33 arranges the boundary line (BD1 or BD2) at the portion of the current region boundary (BD1 or BD2), and arranges additional information such as the above - mentioned gradation portion (GD1 or GD2), so that the boundary line and the additional information are superimposed on the real - world VR image and displayed on the display unit 11.

[0112] Next, with reference to FIG. 23, an example of the process when displaying the above - mentioned boundary line (BL1 or BL2) will be described. FIG. 23 is a flowchart showing an example of the process of the display control device 20 when displaying the boundary line. S300 to S312 in the flowchart of FIG. 23 are the same processes as S100 to S112 in FIG. 15, and the detailed description of each process will be omitted. Based on the line-of-sight direction S, the control unit 30 executes control to dynamically switch the live-action image to be displayed in the overlapping region AO among adjacent live-action images (the first image and the second image) (S304 to S312). Then, the control unit 30 superimposes a boundary line (BL1 or BL2) on a portion of the region boundary (BD1 or BD2) between adjacent live-action images (S314). Further, the control unit 30 superimposes a gradation unit (GD1 or GD2) on, for example, a region on the opposite side of the overlapping region AO across the boundary line (BL1 or BL2) (the region of the live-action image that is not displayed in the overlapping region AO) (S316). Then, the control unit 30 generates a VR image within a range corresponding to the orientation of the HMD 10 and outputs it to the HMD 10 (S318). As a result, a VR image including the boundary line (BL1 or BL2) and the gradation unit (GD1 or GD2) is displayed on the HMD 10. The processing from S302 to S318 is repeated until the display ends (YES in S320). Note that in the flowchart of FIG. 23, when the gradation unit (GD1 or GD2) is not displayed in the VR image, the process of S316 can be omitted.

[0113] (Aspect of making the display forms of the two boundary lines different) The boundary line display unit 33 may have different display forms for the boundary line BL1 displayed on the region boundary BD1 at one end (right end) of the overlapping region AO and the boundary line BL2 displayed on the region boundary BD2 at the other end (left end). Here, "having different display forms for the boundary lines" means making the colors, densities, transparencies, line widths, line types (solid lines, dotted lines, etc.), or whether they are planar (two-dimensional) or three-dimensional, etc. of the boundary lines different. Thereby, when using a plurality of images arranged in the horizontal direction to expand the horizontal viewing angle, the user can recognize whether the currently displayed boundary line (BL1 or BD2) is the misaligned part of the left-end image of the overlapping region AO or the misaligned part of the right-end image. Also, when using a plurality of images arranged in the vertical direction to expand the vertical viewing angle, the user can recognize whether the currently displayed boundary line (BL1 or BD2) is the misaligned part of the upper-end image of the overlapping region AO or the misaligned part of the lower-end image. In this aspect, the display of additional information such as the aforementioned gradation parts GD1 and GD2 may be omitted, or the additional information may be displayed together with the boundary lines.

[0114] (Aspect of setting the switching period) When switching between the two real-shot images of the overlapping region AO, instead of switching directly from one to the other, a switching period (for example, 0.3 seconds, etc.) may be provided, and the switching may be gradually performed according to the passage of time. For example, when switching the real-shot image displayed in the overlapping region AO from one to the other, the display of the overlapping region AO may be cross-faded during the switching period. Thereby, it becomes easier to visually recognize the boundary line (BL1 or BL2) more naturally before and after the switching of the real-shot image in the overlapping region AO. This will be explained below.

[0115] When switching the real-shot image to be displayed in the overlapping region AO, the image generation unit 32 in this aspect has a function of gradually switching from one real-shot image to the other according to the passage of time. Here, "gradually switching from one real - world image to another in accordance with the passage of time" means providing a switching period (for example, 0.3 seconds, etc.) from the start to the end of the switching, and gradually switching the real - world image (the first image or the second image) currently displayed in the overlapping area AO to the other real - world image during the passage of time within the switching period. The switching period can be set arbitrarily.

[0116] For example, within the switching period, synthesizing the two real - world images to be switched, which are displayed in the overlapping area, and gradually switching the real - world image displayed in the overlapping area from one to the other corresponds to an example of "gradually switching from one real - world image to another in accordance with the passage of time". An example of such image synthesis is alpha - blending (semi - transparent synthesis). As a specific example, in accordance with the passage of time within the switching period, the transparency of one real - world image is gradually increased from "0%" to "100%", and at the same time, the transparency of the other image is gradually decreased from "100%" to "0%", and the two real - world images are alpha - blended. Also, for example, in the overlapping area, two adjacent real - world images are arranged in different upper and lower layers. Then, the transparency of the real - world image arranged in the lower layer (background) remains unchanged at 0%, and the transparency of the real - world image arranged in the upper layer (foreground) may be gradually changed from "0%" to "100%" or from "100%" to "0%" in accordance with the passage of time within the switching period. For example, when switching from the real - world image in the upper layer to the real - world image in the lower layer, a process of gradually changing the transparency of the real - world image in the upper layer from "0%" to "100%" is performed in accordance with the passage of time within the switching period. Also, when switching from the real - world image in the lower layer to the real - world image in the upper layer, a process of gradually changing the transparency of the real - world image in the upper layer from "100%" to "0%" is performed in accordance with the passage of time within the switching period.

[0117] Also, during the switching period, executing a process of gradually replacing the pixels of the live-action image currently displayed in the overlapping region with the pixels of the other live-action image corresponds to an example of "gradually switching from one live-action image to the other according to the passage of time". In other words, a process of gradually decreasing the number (area) of pixels of the live-action image currently displayed in the overlapping region and gradually increasing the number (area) of pixels of the other live-action image according to the passage of time corresponds to an example of "gradually switching from one live-action image to the other according to the passage of time". As a specific example, during the switching period, gradually replacing the pixels from the current region boundary side to the region boundary side after the switching is completed, so that the region boundary slides (gradually moves), corresponds to an example of "gradually switching from one live-action image to the other according to the passage of time". The "replacing pixels" includes switching the transparency of the pixels in the upper layer of two adjacent live-action images arranged in different layers above and below the overlapping region from "0% to 100%" or from "100% to 0%". Also, the "replacing pixels" includes setting the transparency of the pixels of the live-action image to be replaced from 0% to 100% and setting the transparency of the corresponding pixels of the live-action image for replacement from 100% to 0%. In addition, the pixels may gradually switch from one to the other from the center to the periphery, from the periphery to the center, from the left end to the right end, from the right end to the left end, from the upper end to the lower end, or from the lower end to the upper end of the overlapping region.

[0118] Next, with reference to FIG. 24, an example of the process with the above-described switching period will be described. FIG. 24 is a flowchart showing an example of a process of gradually switching the live-action image displayed in the overlapping region in the display control device 20. The control unit 30 acquires detection information regarding the orientation of the HMD 10 from the sensor 12 (e.g., angular velocity sensor) of the HMD 10 (S400), and specifies the line-of-sight direction S (S402). Further, the control unit 30 determines whether it is necessary to switch the live-action image to be displayed in the overlapping area AO based on the line-of-sight direction S with respect to the current reference direction (RD1 or RD2) (S404). If the result in this S404 is YES, the control unit 30 initializes the variable α to the initial value "255" (S406). This variable α is the "α value" used for semi-transparent synthesis (alpha blending) of adjacent live-action images (the first image and the second image) arranged in the overlapping area AO in step S408, and can take a value from "0" to "255". Also, the "α value" is used for changing the opacity of the boundary line (BL1 or BL2) in steps S410 and S412.

[0119] The control unit 30 subtracts the α value by a predetermined amount Δα for each frame, and changes the α value so as to gradually decrease from "255" to "0" according to the passage of time during the switching period (S414). For example, assume the switching period is a 20-frame period. Note that one frame is, for example, 1 / 60 second. For example, if the predetermined amount Δα subtracted for each frame is "13", the α value will gradually change from "255" to "0" in a switching period of about 0.3 seconds.

[0120] In S408, the control unit 30 semi-transparently synthesizes two live-action images (the first image and the second image) in the overlapping area AO. In this case, the control unit 30 sets the alpha value of each pixel of the image that was displayed in the overlapping area AO before the switching (referred to as the "pre-switching image") to the value of the variable α. Also, in the overlapping area AO, the image to be displayed after the switching is referred to as the "post-switching image". Therefore, during the switching period in which the semi-transparent synthesis is executed, the pixel value of each pixel in the overlapping area AO is set to "the pixel value of the corresponding pixel in the pre-switching image * α + the pixel value of the corresponding pixel in the post-switching image * (255 - α)".

[0121] Also, in S410, the control unit 30 sets the opacity of the boundary line before switching to α. Also, in S412, the control unit 30 sets the opacity of the boundary line after switching to (255 - α) and superimposes it on the area boundary part after switching. Here, the boundary line becomes completely opaque (opacity 0%) at α = 255, and the opacity decreases as the α value decreases, and it becomes completely transparent (opacity 100%) at α = 0.

[0122] The above S408 to S414 are repeated until the α value becomes 0 (YES in S416). Thereby, for example, in a switching period of about 0.3 seconds, the α value gradually changes from "255" to "0". In the overlapping area AO, the transparency of the pre-switching image gradually increases, while the transparency of the post-switching image gradually decreases. Also, during the switching period, the boundary line before switching and the boundary line after switching are simultaneously displayed. As time passes during the switching period, the transparency of the boundary line before switching gradually increases, while the transparency of the boundary line after switching gradually decreases. And finally, when the α value becomes "0" (YES in S416), the switching of the image in the overlapping area AO is completed, the post-switching image is displayed in the overlapping area AO, and the boundary line before switching disappears and the boundary line after switching is displayed.

[0123] Here, an example is shown in which the boundary line before switching and the boundary line after switching are simultaneously displayed while changing the transparency during the switching period, but it is not limited to this. At the timing when the image switching process (switching period) in the overlapping area AO is started, the boundary line before switching may be erased, and at the timing when the switching process (switching period) is completed, the boundary line after switching may be displayed. In this case, S410 and S412 in FIG. 24 can be omitted. Alternatively, during the switching process (switching period), the boundary line before switching and the boundary line after switching may be displayed with the same transparency (both may be opaque), and at the timing when the switching process (switching period) is completed, the boundary line before switching may be erased and the boundary line after switching may be displayed.

[0124] (Mode of moving and displaying the boundary line during the switching period) In the aspect of providing the aforementioned switching period, during the switching period, the boundary line may be gradually moved toward the position to be displayed after the switching is completed. For example, when the first image of the left visual field region A1 is displayed in the overlapping region AO, as illustrated in FIG. 21, a boundary line BL1 is displayed on the screen G20. From this state, when the real image to be displayed in the overlapping region AO is switched to the second image of the right visual field region A2, as illustrated in FIG. 25, the boundary line BL1 at the position indicated by the dashed-dotted line is changed to the position of the boundary line BL2. At the time of this image switching, during the switching period (for example, 0.3 seconds, etc.), the boundary line is controlled to be displayed so as to gradually move horizontally from the position of BL1 indicated by the dashed-dotted line to the position of BL2. Even in this way, the position of the boundary line can be naturally recognized by the user.

[0125] When switching the real image to be displayed in the overlapping region AO, the image generation unit 32 in this aspect has a function of gradually switching from one real image to the other real image while gradually moving the boundary line toward the position to be displayed after the switching is completed according to the passage of time.

[0126] For example, the case where the real image in the overlapping region AO is switched from the first image on the left to the second image on the right will be described. As illustrated in FIG. 25, during the switching period, the boundary line BLmove is gradually moved horizontally from the region boundary BD1 before the switching toward the region boundary BD2 after the switching is completed. At this time, in the overlapping region AO, in the region on the right side of the moved boundary line BLmove (the region between the region boundary BD1 before the switching and the moved boundary line BLmove), each pixel of the first image is replaced with each pixel of the corresponding second image. Thereby, as the boundary line BLmove moves horizontally to the left, each pixel on the right side of the boundary line BLmove in the overlapping region AO is switched from the first image to the second image. This process is repeated until the boundary line BLmove reaches the region boundary BD2 after the switching is completed. When the boundary line BLmove reaches the region boundary BD2, it stops as the boundary line BL2. Even when the real image of the overlapping region AO is switched from the second image on the right to the first image on the left, the same processing as described above is executed.

[0127] As illustrated in FIG. 25, when displaying the gradation portion (GD1 or GD2) as additional information, the display of the gradation portion may be controlled as follows. For example, at the timing when the switching process (switching period) of the real image of the overlapping region AO starts, the gradation portion (GD1 or GD2) is erased, and at the timing when the switching process (switching period) ends, the gradation portion is displayed. Alternatively, in conjunction with the movement of the boundary line BLmove, the gradation portion may also be moved.

[0128] Note that in a configuration where the boundary line is moved and displayed according to the passage of time during the switching period, the switching process of the real image in the overlapping region AO is not limited to the above. For example, while executing the switching process of the real image of the overlapping region AO by the aforementioned semi-transparent composition (alpha blending), the boundary line may be gradually moved toward the position to be displayed after the switching is completed according to the passage of time during the switching process.

[0129] [An example of dividing the entire field of view region of the 9.VR space into three or more regions] Here, with reference to FIG. 26, an example of dividing the entire field of view region of the VR space V into three regions: left, front, and right will be described. Here, an example is shown in which a high field of view angle image with a horizontal field of view angle of 270 degrees can be reproduced using the left image, front image, and right image captured by three stereo cameras with a horizontal imaging angle of 120 degrees for imaging in each of the left, front, and right directions.

[0130] FIG. 26 conceptually shows an example in which the viewing area AL of the left image, the viewing area AF of the front image, and the viewing area AR of the right image are arranged in the VR space V. In the VR space V, the arrangement of the images is controlled such that a first overlapping area AO1 in which a part of the viewing area AL of the left image and a part of the viewing area AF of the front image overlap each other is generated. Also, the arrangement of the images is controlled such that a second overlapping area AO2 in which a part of the viewing area AF of the front image and a part of the viewing area AR of the right image overlap each other is generated. The viewing angle θall of the entire viewing area of the VR space V is 270 degrees, the viewing angle θAL of the left viewing area AL is 120 degrees, the viewing angle θAF of the front viewing area AF is 120 degrees, the viewing angle θAR of the right viewing area AR is 120 degrees, the viewing angle θAO1 of the first overlapping area AO1 is 45 degrees, and the viewing angle θAO2 of the second overlapping area AO2 is 45 degrees. Further, a reference direction RD1 serving as a reference for switching the actual image (left image or front image) of the overlapping area AO1 is set in the direction from the virtual viewpoint P to the center of the overlapping area AO1. Also, a reference direction RD2 serving as a reference for switching the actual image (front image or right image) of the overlapping area AO2 is set in the direction from the virtual viewpoint P to the center of the overlapping area AO2. This is an example, and the viewing angle of the entire viewing area, the viewing angles of the viewing areas of the respective images, the viewing angles of the respective overlapping areas, or the reference direction (presence / absence, its direction) can be arbitrarily set.

[0131] When the entire field of view region of the VR space is divided into three regions as in this example, two overlapping regions, i.e., the first overlapping region AO1 and the second overlapping region AO2, are generated. For each of the first and second overlapping regions AO1 and AO2, each process related to the overlapping region described above (and the processes shown hereinafter) may be executed. That is, in the relationship between the field of view region AL of the adjacent left image and the field of view region AF of the front image, since the left image is the "first image" described above, the front image is the "second image" described above, and the first overlapping region AO1 corresponds to the "overlapping region AO" described above, each process described above can be applied. Also, in the relationship between the field of view region AF of the adjacent front image and the field of view region AR of the right image, since the front image is the "first image" described above, the right image is the "second image" described above, and the second overlapping region AO2 corresponds to the "overlapping region AO" described above, each process described above can be applied.

[0132] The same applies when the entire field of view region of the VR space is divided into four or more regions. When n real-shot images (n is a natural number of 2 or more) captured by n cameras are arranged in the VR space, (n - 1) overlapping regions are generated. When there are a plurality of overlapping regions AO, one of the adjacent images arranged in each overlapping region may be regarded as the "first image" and the other as the "second image", and each process described above may be applied to each overlapping region AO.

[0133] Normally, the field of view image (VR image) in the range displayed on the display unit 11 of the HMD10 is a part of the entire field of view region of the VR space. Therefore, when at least one overlapping region AO is not included in the range of the VR image displayed on the display unit 11, the image switching process and the like for the overlapping region AO are unnecessary and can be prevented from being executed.

[0134] [Aspect of Providing a Time Lag until the Execution Start of the Switching Process] When performing the switching process of the overlapping region AO described above, a predetermined time lag (for example, 0.5 seconds, etc.) may be provided until the start of the execution of the process. By providing such a time lag, unnecessary switching processes can be omitted. This will be explained below.

[0135] Here, as shown in FIG. 26, a case where the regions of the VR space V are in three directions: left, front, and right is exemplified. Assume that a user wearing the HMD 10 is viewing a VR image in a range including the left visual field region AL and the first overlapping region AO1. And currently, assume that a left image is being displayed in the first overlapping region AO1. From this state, assume that the user greatly shakes their head to the right, and instantaneously, the viewing axis direction (line-of-sight direction S) of the HMD 10 is moved to the right visual field region AR beyond the front. Without time lag setting, the switching process in the first overlapping region AO1 and the switching process in the second overlapping region AO2 can occur within a short period. However, when the viewing axis direction of the HMD 10 reaches the right visual field region AR, the first overlapping region AO1 may already be outside the range of the VR image displayed on the display unit 11 of the HMD 10. Even in such a case, if the switching process is executed in a sensitive reaction, there is a possibility that the VR image displayed on the display unit 11 will shake and the visibility will deteriorate. On the other hand, in such a case as described above, by providing a time lag for the switching process, only the switching process in the second overlapping region AO2 can be executed as necessary, and the switching process in the first overlapping region AO1 can be omitted, improving the visibility of the image displayed on the display unit 11. Note that in the description without distinguishing between the first overlapping region AO1 and the second overlapping region AO2, it may be described as "overlapping region AO".

[0136] The image generation unit 32 in this aspect has a function of providing a predetermined waiting period from the determination that the switching process of the real image to be displayed in the overlapping region AO should be performed based on the line-of-sight direction S until the start of the switching process. Here, the waiting period corresponds to the aforementioned time lag. Also, after the elapse of the waiting period until the start of the switching process, the image generation unit 32 re-determines whether the switching process should be performed, and only when it is determined that it should be performed, the switching process is executed.

[0137] The standby period is, for example, 0.5 seconds, but is not limited thereto and can be arbitrarily set. When determining whether to perform the switching process of the overlapping area AO based on detection information regarding the orientation of the HMD 10 (detection information from an angular velocity sensor or the like of the HMD 10), and when determining whether to perform the switching process of the overlapping area AO based on detection information regarding the user's line of sight (detection information such as eye tracking), the length of the standby period may be made different. That is, in eye tracking, since the line of sight is tracked by analyzing the eye movement of a person, etc., the line of sight direction S can easily move freely. For this reason, it is better to set a longer standby period (time lag) for the determination based on eye tracking that tracks the user's line of sight than for the determination based on so-called head tracking that detects the orientation of the HMD 10. That is, when the switching process is executed by specifying the line of sight direction S based on eye tracking, in the case of a user who frequently moves their eyes, the switching process occurs frequently and the image is likely to shake. However, by increasing the standby period, unnecessary switching processes can be suppressed and the visibility of the image can be improved.

[0138] For example, in the case of the HMD 10 equipped with both a head tracking function using an angular velocity sensor or the like and an eye tracking function, the user may be able to select which of the two functions to use for determining whether to perform the switching process of the overlapping area AO by user operation. In this case, the standby period is changed based on the user's selection.

[0139] Next, with reference to FIG. 27, an example of the process with the above-described time lag will be described. FIG. 27 is a flowchart showing an example of the switching process of the live image to be displayed in the overlapping area in the display control device 20. The control unit 30 identifies the line-of-sight direction S based on the detection information from the sensor 12 (e.g., angular velocity sensor) of the HMD 10, and determines whether switching processing of the real-world image to be displayed in the overlapping area AO is necessary (S500). When the control unit 30 determines that the switching processing is necessary (YES in S500), it sets a waiting period (e.g., 0.5 seconds) and starts timing (S502). After that, the control unit 30 does not execute the switching processing for the overlapping area AO until the waiting period elapses. After the elapse of the waiting period (YES in S504), the control unit 30 identifies the line-of-sight direction S again based on the detection information from the sensor 12 of the HMD 10, and determines whether switching processing of the real-world image to be displayed in the overlapping area AO is necessary (S506). When the control unit 30 determines that the switching processing is necessary (YES in S506), it executes the switching processing (S508). On the other hand, when the control unit 30 determines that the switching processing is unnecessary (NO in S506), it ends the processing without executing the switching processing. As an example where the switching processing is unnecessary, as described above, the target overlapping area AO may be outside the range of the VR image already displayed on the display unit 11 of the HMD 10. Also, after it is determined that switching processing for the overlapping area AO is necessary because the line-of-sight direction S has once exceeded the reference direction RD, if the line-of-sight direction S returns to near the original position beyond the reference direction RD again before the waiting period elapses, the switching processing becomes unnecessary.

[0140] [11. Mode of changing the arrangement state of the image in the VR space] For example, in the examples of FIGS. 9 to 11, the number of visual field areas (number of images) arranged in the VR space is 2, and the entire visual field area with a horizontal field angle of 180 degrees is divided into two areas on the left and right. However, these are not fixed and may be changed according to the scene (scenario), content, etc. of the image to be displayed. For example, when the scene is switched in a game or other content using a real-world image, the number of visual field areas (number of images) arranged in the VR space, the field angle of the entire visual field area, or the direction (position in the VR space) of each visual field area in the VR space may be changed according to the content of the scene.

[0141] For example, as illustrated in FIG. 28, in a certain scene, as illustrated in (A) in the same figure, the 270-degree overall visual field region is divided into three regions, and in another scene, as illustrated in (B) in the same figure, the 220-degree overall visual field region is divided into two regions. Then, according to the scene, it is set to the arrangement state of (A) or the arrangement state of (B). For example, basically the arrangement of (A) is adopted, but in a scene where only the front region is focused on and the images of the rear region are unnecessary, the arrangement of (B) is adopted.

[0142] In yet another scene, as illustrated in FIG. 9, the 180-degree overall visual field region may be divided into two regions. Note that a scene with one region (for example, only one 140-degree overall visual field region) without dividing the overall visual field region may be partially included.

[0143] For example, in a certain scene, three cameras 200 shown in FIG. 26 are used to capture three images arranged in the VR space illustrated in (A) in FIG. 28. Also, in another scene, only two adjacent ones of the three cameras 200 are used to capture two images arranged in the VR space illustrated in (B) in FIG. 28. In this case, if necessary, the lens of at least one of the three cameras 200 may be replaced to expand (or contract) the angle of view, or the imaging direction (the direction of the optical axis) of each camera 200 may be adjusted. For example, when switching from an indoor scene to an outdoor scene, or when switching from a scene of one room to a scene of another room, etc., since the installation positions of the cameras 200 for capturing each scene also change, considering the arrangement state in the VR space suitable for each scene, real-world images may be captured.

[0144] As described above, together with the data of a plurality of real-shot images to be displayed, arrangement information for arranging the plurality of real-shot images in the VR space is stored in the storage device 22 in association with the plurality of real-shot images. For example, for each scene where the imaging location of the camera in the real space changes, the above-described arrangement information corresponding to the conditions imaged in each scene is generated and recorded together with the real-shot images in association with each scene. In this way, when the arrangement state of the plurality of real-shot images in the VR space is appropriately changed according to the scene, the arrangement information associated with each scene is stored in the storage device 22. Alternatively, the arrangement information for each scene is recorded in the header of the real-shot image file or the like, and the arrangement information is read out and used when the real-shot image is reproduced.

[0145] The image generation unit 32 in this aspect includes an image arrangement unit 321 that changes the arrangement state of a plurality of real-shot images in the VR space during the display control of the VR image to the display unit 11. Here, the objects of the change in the arrangement state in the VR space are, for example, the number of viewing regions (number of images) arranged in the VR space, the viewing angle of the entire viewing region, the viewing angle of each viewing region, the direction (position) of each viewing region in the VR space, the number of overlapping regions, the viewing angle of each overlapping region, or the direction (position) of each overlapping region in the VR space. These pieces of information are recorded as arrangement information in association with the plurality of real-shot images to be displayed (for example, in association with each scene of the plurality of real-shot images).

[0146] For example, in the arrangement state of the VR space in (A) of FIG. 28, the number of viewing areas (number of images) = 3, the viewing angle of the entire viewing area is "θall = 270 degrees", the direction of the entire viewing area (position in the VR space) is "-135 degrees to 135 degrees", and the viewing angles of the left, front, and right viewing areas AL, AF, and AR are "θAL = θAF = θAR = 120 degrees". The direction of the left viewing area AL is "15 degrees to 135 degrees", the direction of the front viewing area AF is "-60 degrees to 60 degrees", the direction of the right viewing area AR is "-135 degrees to -15 degrees", the viewing angle of the first overlapping area AO1 is "θAO1 = 45 degrees", the direction of the first overlapping area AO1 is "15 degrees to 60 degrees", the viewing angle of the second overlapping area AO2 is "θAO2 = 45 degrees", and the direction of the second overlapping area AO1 is "-60 to -15 degrees". All or part of this information is recorded as arrangement information. The angles of the directions shown here are horizontal angles with the X-axis direction (reference line-of-sight direction) being 0 degrees, and the vertical angles are omitted here. Also, in the arrangement state of the VR space in (B) of FIG. 28, the number of viewing areas (number of images) = 2, the viewing angle of the entire viewing area is "θall = 220 degrees", the direction of the entire viewing area is "-110 degrees to 110 degrees", and information such as the viewing angles of the left and right viewing areas AL, AF, and AR being "θAL = θAR = 135 degrees" (omitted below) is recorded as arrangement information.

[0147] The image arrangement unit 321 determines whether it is necessary to change the arrangement state of a plurality of real-shot images to be displayed in the VR space based on the arrangement information associated with the plurality of real-shot images, and appropriately changes the arrangement state based on the arrangement information.

[0148] In the image display system 1 of the present embodiment, as described above, one VR image data completed by stitching processing is not created in advance. Instead, while arranging a plurality of real-shot images in the VR space, VR image display control is executed. Therefore, during the reproduction of the VR image, it is also possible to appropriately change the arrangement state of the real-shot images in the VR space as in this aspect. For example, according to the scene, the number of viewing areas (the number of real-shot images) arranged in the VR space, the viewing angle of the entire viewing area, etc. can be changed. That is, the optimization of the VR space for each scene can be achieved. Therefore, it is possible to eliminate unnecessary viewing areas for each scene, and the amount of image data can be reduced.

[0149] In addition, by eliminating unnecessary viewing areas for each scene, not only can the data capacity be reduced, but also in the scene where unnecessary viewing areas are eliminated, the resolution of the VR image displayed on the display unit 11 of the HMD 10 can be increased. That is, an amount equivalent to the eliminated unnecessary viewing areas allows a VR image with a higher resolution to be expanded in the VRAM or the like. For example, a VR image with a higher resolution is displayed on the display unit 11 of the HMD 10 in the arrangement state of (B) rather than the arrangement state of (A) in Fig. 28. Therefore, in a scene where it is desired to increase the resolution of the image, the arrangement state of (B) in Fig. 28 may be adopted, and in other cases, the arrangement state of (A) may be adopted, etc., to optimize the VR space for each scene.

[0150] Next, with reference to Fig. 29, an example of the processing of this aspect will be described. Fig. 29 is a flowchart showing an example of the processing for changing the arrangement state of the image in the VR space in the display control device 20. The control unit 30 acquires the arrangement information of a plurality of actual captured images to be displayed (S600), and determines the arrangement state of the plurality of actual captured images in the VR space based on the arrangement information (S602). During VR image playback, the arrangement state in the VR space determined here is applied, and the VR space is arranged so that an overlapping area AO is generated where a part of the viewing areas of adjacent images overlap each other. Also, as described above, based on the line-of-sight direction S (for example, the optical axis direction of the HMD 10), the actual captured image to be displayed in the overlapping area AO is dynamically switched, and a VR image is generated. Further, when the scene is changed during the playback of the VR image (YES in S604), the control unit 30 acquires the arrangement information of the changed scene (S606), and determines whether to change the arrangement state of the images in the VR space (S608). Here, when the control unit 30 determines that a change in the arrangement state is necessary (YES in S608), it changes the arrangement state of the plurality of actual captured images in the VR space (S610). As a result, after the scene change, the arrangement state in the VR space suitable for the scene is applied, and after the images of the scene are arranged in the VR space, a VR image is generated. The processing of S604 to S610 is repeated until the display ends (YES in S612).

[0151] [12. Mode of Changing the Direction of the Entire Viewing Area of the VR Space] In the above description, regarding the camera that captures a plurality of actual captured images to be displayed, a change in the direction of the camera during imaging in the real space has not been considered, but the direction of the camera may be changed during imaging. In that case, the change in the imaging direction during the imaging may be reflected in the viewing area of the VR space.

[0152] For example, as illustrated in FIG. 26, assume that there are three cameras 200 for imaging the real space and there is a relationship with the viewing area of the VR space V where each image captured by the three cameras 200 is arranged. That is, the imaging direction of each camera 200 in FIG. 26 is the reference imaging direction of each camera, and the reference imaging direction of the front camera 200 corresponds to the X-axis direction (reference line-of-sight direction) of the VR space V. Here, FIG. 30 illustrates the case where the three cameras 200 in FIG. 26 are rotated to the right. FIG. 30 illustrates the relationship that the entire viewing area of the VR space V where each image captured by the three cameras 200 is arranged is also rotated according to the change in the direction (rotation of the imaging direction) when the three cameras 200 are imaging.

[0153] In the example of FIG. 26, in the case of the image captured by the camera 200 in the reference imaging direction, the direction of the entire viewing area in the VR space V is "-135 degrees to 135 degrees" with the X-axis direction (reference line-of-sight direction) being 0 degrees. On the other hand, in the example of FIG. 30, the imaging direction of the camera 200 is rotated 45 degrees to the right (i.e., -45 degrees) from the reference imaging direction in FIG. 26. In the case of the image captured by the camera 200 in FIG. 30, the direction of the entire viewing area in the VR space V is rotated 45 degrees to the right and changes to "-180 degrees to 90 degrees".

[0154] The information on the change in the imaging direction of the camera 200 may be detected from an angular velocity sensor (gyro sensor) or the like mounted on the camera 200 at the time of imaging and recorded together with the image data at the time of imaging. Alternatively, after imaging, the change in the imaging direction may be specified by known image analysis from the captured image, and the information on the change in the imaging direction may be recorded in association with the captured image. Alternatively, the display control device 20 may specify the change in the imaging direction in real time by image analysis from the captured image during the reproduction of the image.

[0155] For example, consider a scenario in real space where a car passes in front from the left and runs through to the right, and the camera captures images while changing its orientation from left to right in accordance with the movement of the car. FIG. 31 shows an example of the change in the viewing area of the VR space V where the images captured by the camera 200 are arranged when the information on the rotation of the imaging direction of the camera 200 is not used. When the information on the rotation of the imaging direction is not used, even if the camera 200 rotates, the direction of the overall viewing area does not change, and always the imaging direction of the front camera 200 corresponds to the X-axis direction of the VR space V. Therefore, when the captured images are displayed on the HMD10 without using the information on the rotation of the imaging direction, it is possible to display the video of the car without crossing the area boundaries of adjacent images. However, although the car is moving, since it is displayed in the same direction with respect to the visual axis (line-of-sight direction S) of the HMD10, the reality cannot be maintained as in the real world.

[0156] FIG. 32 shows an example of the change in the viewing area of the VR space V where each image captured by the camera 200 is arranged when the information on the rotation of the imaging direction of the camera 200 is used. When the information on the rotation of the imaging direction is used, the direction of the overall viewing area in the VR space V also rotates according to the rotation of the imaging direction. In this case, when the captured images are displayed on the HMD10, a video in which the car moves from left to right is also displayed with respect to the visual axis (line-of-sight direction S) of the HMD10. Therefore, when the car exists in the left direction during the reproduction of the images, the user cannot see the car unless the user turns the HMD10 in the left direction (or the car can be seen in the left direction when the user is facing forward). By turning the head in the direction where the car is located, the user can clearly see the car, and a realistic VR image can be displayed. Moreover, it becomes possible to display the video of the car on the display unit 11 of the HMD10 without crossing the area boundaries (boundary lines when the boundary lines are displayed) of adjacent images.

[0157] As illustrated in FIG. 20, the control unit 30 of the display control device 20 in this embodiment can be configured to include an imaging direction specifying unit 34. This imaging direction specifying unit 34 has a function of specifying changes in the imaging directions of a plurality of cameras that have captured a plurality of actual images. For example, the imaging direction specifying unit 34 can read out "information on changes in the imaging direction" recorded in association with the plurality of actual images and specify the changes in the imaging direction. Alternatively, the imaging direction specifying unit 34 may specify changes in the imaging direction by image analysis from the actual images. Then, the image generation unit 32 has a function of changing the direction of the viewing area (overall viewing area) of the entire plurality of actual images arranged in the VR space according to the change in the imaging direction specified by the imaging direction specifying unit 34.

[0158] Next, with reference to FIG. 33, an example of the processing of this embodiment will be described. FIG. 33 is a flowchart showing an example of the processing for changing the direction of the overall viewing area in the VR space in the display control device 20. At the start of reproduction of the VR image, the control unit 30 initializes the direction of the overall viewing area in the VR space to the direction corresponding to the reference imaging direction (S700). For example, as illustrated in FIG. 26, the horizontal direction of the overall viewing area is default-set to "-135 degrees to 135 degrees", and the direction from the virtual viewpoint P to the center of the front viewing area AF becomes the X-axis direction (reference line-of-sight direction). Further, the control unit 30 acquires information on changes in the imaging directions of the plurality of actual images to be displayed (S702) and specifies the changes in the imaging direction (S704). For example, the control unit 30 reads out and acquires the information on changes in the imaging direction recorded together with the data of the plurality of actual images to be displayed, for example, every frame, thereby specifying the changes in the imaging direction. Then, when the control unit 30 determines that the imaging direction has changed from the previous frame (YES in S706), it changes the direction of the overall viewing area in the VR space according to the change in the imaging direction (S708). On the other hand, when it is determined that the imaging direction has not changed (NO in S706), the process returns to step S702. The processing of S702 to S708 is repeated until the display ends (YES in S710).

[0159] [13. Summary] As described above, the program according to this embodiment is a program for causing a display control device 20 (an example of a computer) that executes control for causing a VR image representing a field of view from a virtual viewpoint P in a VR space V to be displayed on a display unit 11 of an HMD 10 as a stereoscopic image using binocular disparity to function as a line-of-sight direction specifying unit 31 and an image generation unit 32. The display control device 20 according to this embodiment includes a line-of-sight direction specifying unit 31 and an image generation unit 32. The line-of-sight direction specifying unit 31 specifies a line-of-sight direction S that is the direction of the user's line of sight with respect to the VR space. The image generation unit 32 arranges a plurality of real images captured by a plurality of cameras having different imaging directions for imaging the real space in the VR space V such that an overlapping region AO in which a part of the field of view regions of adjacent real images overlaps, and generates the VR image according to the line-of-sight direction S. Here, the image generation unit 32 dynamically switches the real image to be displayed in the overlapping region AO among the adjacent real images based on the line-of-sight direction S.

[0160] Here, the "VR image" in this configuration is a stereoscopic real image using binocular disparity that is displayed on the display unit 11 of the HMD 10, but display targets other than real images may also be included in the "VR image". For example, display targets other than real images such as CG (Computer Graphics), various objects, lines, symbols, characters, etc. may also be arranged in the VR space and superimposed on the real images to form a "VR image". The "VR image" includes an AR (Augmented Reality) image or an MR (Mixed Reality) image. The "VR image" may be a still image or a moving image. The "VR image" may be appropriately converted into a transmission format corresponding to the HMD 10, for example, when supplied (transmitted) to the display unit 11 of the HMD 10.

[0161] Further, as shown in FIG. 34 for example, the "HMD" may be an HMD 50 including an attachment (mounting device) 51 that can be worn on the user's head and an information processing device 52 such as a smartphone attached to the attachment 51. In this case, the display unit 61 of the information processing device 52 becomes the display unit of the HMD 50. Also, the "HMD" may be a so-called stand-alone type HMD in which the functions of the display control device 20 are integrated with the HMD 10. Also, the "HMD" may be a goggle type or glasses type that can be worn on the head. Also, the "HMD" is not limited to the narrow sense HMD, and for example, a headphone, a headset (headphone with a microphone), a glasses type camera, an ear-mounted camera, a hat with a camera, etc. may be provided with functions as an HMD.

[0162] Also, as long as the "display control device" according to the present embodiment is a device having information processing functions such as generation of VR images, various devices can be applied. For example, a stationary or portable dedicated game machine, a business (commercial) game machine, a personal computer, a tablet computer, a smartphone, a mobile phone terminal, a PHS terminal, a PDA, a multifunctional television receiver having information processing functions, etc. can be used as the "display control device". Also, when the HMD itself has information processing functions such as generation of VR images, the HMD can be the "display control device" of this configuration. For example, in the case of the HMD 50 including the attachment 51 and the information processing device 52 as illustrated in FIG. 34, the information processing device 52 such as a smartphone becomes an example of the "display control device" according to the present embodiment. Also, the stand-alone type HMD becomes an example of the "display control device" according to the present embodiment.

[0163] According to the above configuration, by dynamically switching the real image to be displayed in the overlapping region AO based on the line-of-sight direction S, the region boundaries (BD1 or BD2) between adjacent real images in the overlapping region AO can be moved away from the user's line-of-sight direction. As a result, the discomfort of the "seam (misalignment) between adjacent real images" generated at the region boundary can be reduced. As described above, in the conventional technology that performs stitching processing, if a close-up object whose distance from the camera is below a certain level (for example, 2 m or less) is at the image seam, stitching becomes difficult and VR image data cannot be created. In contrast, according to this configuration, even if a close-up object whose distance from the camera is below a certain level is included in the real image and the close-up object exists at or near the region boundary, a stereoscopic VR image with less discomfort for the user is displayed. That is, a VR image with less discomfort can be displayed regardless of the position of the close-up object within the VR space, and VR image display control capable of displaying close-up images can be realized. In addition, since the stitching process is unnecessary, the labor of creating VR images can be reduced. In addition, since the stitching process is unnecessary, a VR image with less discomfort at the seam between real images can be displayed on the display unit 11 in substantially real time using the real image (live video) that the camera is currently capturing.

[0164] In addition, the image generation unit 32 may set a reference direction RD in a predetermined direction from the virtual viewpoint P and specify the real image to be displayed in the overlapping region AO based on the line-of-sight direction S with respect to the reference direction RD. Thereby, the switching control of the real image to be displayed in the overlapping region AO can be easily realized.

[0165] In addition, the image generation unit 32 may set the reference direction RD in the direction of a predetermined position within the overlapping region AO from the virtual viewpoint P. As a result, since the reference direction RD is set within the overlapping region AO, even if the line-of-sight direction S is directed toward the region boundary (BD1 or BD2) of the overlapping region AO, the real image to be displayed in the overlapping region is switched before reaching the region boundary, and thus the region boundary also moves away from the user's line-of-sight direction.

[0166] Further, the image generation unit 32 may change the reference direction RD according to the real image to be displayed in the overlapping area AO at the timing when the real image to be displayed in the overlapping area AO is switched. As a result, after the real image is switched in the overlapping area AO, the reference direction RD temporarily moves away from the line-of-sight direction S, so that it is possible to reduce the frequent switching of the real image displayed in the overlapping area AO.

[0167] Further, the image generation unit 32 may specify the real image to be displayed in the overlapping area AO based on the line-of-sight direction S with respect to the current area boundary (BD1 or BD2) of adjacent real images. Thereby, even without setting a reference direction, it is possible to easily realize the switching control of the real image to be displayed in the overlapping area AO. Also, at the timing when the real image to be displayed in the overlapping area AO is switched, since the reference area boundary is changed, the same effect as the configuration of changing the reference direction RD described above is obtained. That is, it is possible to reduce the frequent switching of the real image displayed in the overlapping area AO.

[0168] Further, when the line-of-sight direction S approaches the current area boundary (BD1 or BD2) of adjacent real images by a predetermined amount or more (for example, when the angle formed by the line-of-sight direction S and the current area boundary becomes a predetermined value or less), the image generation unit 32 may switch the real image currently displayed in the overlapping area AO to the other real image. Also in this case, without setting a reference direction, it is possible to easily realize the switching control of the real image to be displayed in the overlapping area AO. Further, before the line-of-sight direction S reaches the area boundary, the real image to be displayed in the overlapping area is switched, and the area boundary can be moved away from the direction of the user's line of sight.

[0169] Further, the line-of-sight direction specifying unit 31 may acquire detection information regarding the orientation of the HMD 10 (for example, detection information of an angular velocity sensor) and specify the line-of-sight direction S based on the detection information. Thereby, the user can dynamically switch the real image to be displayed in the overlapping area AO by changing the orientation of the HMD 10.

[0170] Further, the line-of-sight direction specifying unit 31 may acquire detection information regarding the user's line of sight (for example, detection information of eye tracking), and specify the line-of-sight direction S based on the detection information. Thereby, the user can dynamically switch the live-action image displayed in the overlapping region AO by changing the movement of the eyes (direction of the line of sight).

[0171] Further, the image generation unit 32 may acquire detection information regarding the orientation of the HMD 10, and change the range of the VR image to be displayed on the display unit 11 based on the detection information. According to this configuration, the range of the VR image displayed on the display unit 11 is changed according to the orientation of the HMD 10. Further, in the VR image displayed on the display unit 11, the live-action image displayed in the overlapping region AO is dynamically switched based on the line-of-sight direction S. Here, when the line-of-sight direction S is specified based on the detection information regarding the orientation of the HMD 10, the user can dynamically switch the live-action image displayed in the overlapping region AO while changing the range of the VR image displayed on the display unit 11 by adjusting the orientation of the HMD 10. Further, when the line-of-sight direction S is specified based on the detection information regarding the user's line of sight (for example, detection information of eye tracking), the user can adjust the range of the VR image displayed on the display unit 11 by changing the orientation of the HMD 10, and can also dynamically switch the live-action image displayed in the overlapping region AO by changing the movement of the eyes (direction of the line of sight).

[0172] Further, the display control device 20 may be configured to include a boundary line display unit 33. The boundary line display unit 33 displays a boundary line (BL1 or BL1) visible to the user at the region boundary (BD1 or BD2) between adjacent live-action images. In this way, by deliberately displaying the boundary line visibly at the region boundary, it becomes easier for the user to recognize the region boundary where an image seam (misalignment) occurs. Thereby, it is possible to facilitate the user's determination as to which live-action image should be displayed in the overlapping region among adjacent live-action images so that the boundary line is not displayed in the direction the user wants to look. Further, the boundary line display unit 33 may vary the display form of the boundary line (BL1 or BL2) displayed on one end region boundary (BD1 or BD2) of the overlapping region AO and the boundary line displayed on the other end region boundary. As a result, the user can recognize whether the currently displayed boundary line is the misaligned portion of the image at one end (e.g., the left end) of the overlapping region or the misaligned portion of the image at the other end (e.g., the right end).

[0173] Further, the boundary line display unit 33 may display additional information (e.g., a gradation portion (GD1 or GD2)) for indicating which side of the boundary line is the overlapping region AO, on or near the boundary line (BL1 or BL1). In this way, by enabling the user to recognize which side of the boundary line is the overlapping region, it becomes easier for the user to determine in which direction the line of sight direction S should be changed for the real image displayed in the overlapping region AO to be switched.

[0174] Further, the additional information can be a semi-transparent gradation portion (GD1 or GD2) that is superimposed on the real image near the boundary line and whose density or opacity continuously or stepwise decreases as it moves away from the boundary line. This gradation portion has the effect of hiding and making less conspicuous the image misalignment occurring near the boundary line (BL1 or BL2). Also, the change in the density or opacity of the gradation portion (GD1 or GD2) (from dark to light or from light to dark) serves to naturally represent the direction of the change in the line of sight direction S required for switching the real image in the overlapping region AO.

[0175] Further, when switching the real image to be displayed in the overlapping region AO, the image generation unit 32 may gradually switch from one real image to the other in accordance with the passage of time. For example, during the switching period (e.g., 0.3 seconds), the two real images to be switched may be alpha-blended and cross-faded. When switching the real image to be displayed in the overlapping region AO from one to the other, by gradually switching during the passage of time in the switching period from the start to the end of the switching rather than switching abruptly, the boundary line can be made more easily visible in a more natural manner.

[0176] In addition, when the image generation unit 32 switches the real image to be displayed in the overlapping area AO, it may gradually switch from one real image to the other real image while gradually moving toward the position to be displayed after the boundary line (BL1 or BL1) is switched, in accordance with the passage of time. In this way, by moving the boundary line during the switching period of the real image to be displayed in the overlapping area AO, the position of the changed boundary line can be naturally recognized by the user.

[0177] In addition, the image generation unit 32 may provide a predetermined waiting period from the determination that the switching process of the real image to be displayed in the overlapping area AO should be performed based on the line-of-sight direction until the start of the switching process. Thereby, when the line-of-sight direction changes during the waiting period and the switching process becomes unnecessary (for example, when the overlapping area AO has already fallen outside the range of the VR image displayed on the display unit 11 of the HMD 10 after the expiration of the waiting period), the unnecessary switching process can be omitted.

[0178] In addition, in the case of the HMD 10 equipped with both a head tracking function using an angular velocity sensor or the like and an eye tracking function, the following may be done. That is, the image generation unit 32 may change which of the detection information regarding the orientation of the HMD 10 and the detection information regarding the user's line of sight (for example, the detection information of eye tracking) is used to specify the line-of-sight direction S, based on the user's selection operation. In this case, the waiting period may be changed based on the user's selection. For example, when it is selected to specify the line-of-sight direction S based on the detection information regarding the user's line of sight, the waiting period may be made longer. Also, since there are differences in the way the user moves their head and eyes, the waiting period may be set arbitrarily by the user. For example, the user may be able to set the waiting period within a predetermined range (for example, in the range of 0.1 seconds to 1.0 seconds). Also, the waiting period in the case of head tracking and the waiting period in the case of eye tracking may be set by the user respectively.

[0179] Also, during the display control of the VR image to the display unit 11 of the image generation unit 32, the arrangement state of a plurality of real-shot images in the VR space may be changed. For example, as shown in FIG. 28, in a certain scene, the overall viewing area of, for example, 270 degrees may be divided into three areas, or in another scene, the overall viewing area of, for example, 220 degrees may be divided into two areas. The number of viewing areas in the VR space, the direction of the viewing areas, etc. may be changed according to the content of the scene. In the display control of the VR image of the present embodiment, one VR image data completed by stitching processing as in the prior art is not created in advance. Instead, the display control of the VR image is executed while arranging a plurality of real-shot images in the VR space. Therefore, it is also possible to appropriately change the arrangement state of the real-shot images in the VR space during the reproduction of the VR image. For example, as described above, according to the scene, the number of viewing areas (the number of real-shot images) arranged in the VR space, the viewing angle of the overall viewing area, etc. can be changed. That is, the optimization of the VR space for each scene can be achieved. Therefore, it is possible to eliminate unnecessary viewing areas for each scene and reduce the amount of image data. Also, in a scene where unnecessary viewing areas are eliminated, the resolution of the VR image displayed on the display unit 11 of the HMD 10 can be increased.

[0180] Further, the display control device 20 may be configured to include an imaging direction specifying unit 34. The imaging direction specifying unit 34 specifies changes in the imaging directions of a plurality of cameras that have captured a plurality of actual images. Then, the image generation unit 32 changes the direction of the viewing area of the entire plurality of actual images arranged in the VR space according to the change in the imaging direction specified by the imaging direction specifying unit 34. According to this configuration, for example, in the case of an actual image in which the imaging direction of the camera changes in accordance with the movement of a moving subject (such as a vehicle), the direction of the viewing area of the entire actual image arranged in the VR space also changes according to the change in the imaging direction. Therefore, a realistic VR image in which the subject moves with respect to the line-of-sight direction S can be displayed on the display unit 11 of the HMD 10. Further, if an actual image in which the subject is imaged so as not to cross the region boundary of adjacent actual images is prepared at the time of imaging in accordance with the movement of the subject described above, it is possible to reproduce a VR image with excellent visibility in which the subject does not cross the region boundary.

[0181] [14. Modifications, etc.] As described above, the embodiments of the present invention have been described. However, the specific configuration is not limited to the above-described embodiments, and designs and the like within a range not departing from the gist of the present invention are also included. Further, the above-described respective configurations and aspects can be arbitrarily combined.

[0182] [14-1] As described above, the image display system or the display control device of the present embodiment can reduce the discomfort of the seam (misalignment) between a plurality of actual images captured by a plurality of cameras without performing stitching processing by display control for dynamically switching the actual images to be displayed in the overlapping region. Therefore, it is also possible to use the actual image (live video) captured in real time by the camera that images the real space as it is and display it on the HMD 10 as a VR image.

[0183] For example, it is also possible to connect a camera 200 illustrated in FIG. 26 to a display control device 20 by wire or wirelessly, and cause a live video captured by the camera 200 to be displayed as a VR image on the HMD 10. Further, the camera 200 and the display control device 20 may be connected so as to be capable of data communication via a network such as the Internet. In this case, it is also possible to cause a live video of a remote location captured by the camera 200 to be displayed as a VR image on the HMD 10. Further, a server having a distribution service function may be set up on the network, and the live video captured by the camera 200 may be distributed via the network through the server. In this case, it is also possible for the display control device 20 to receive the live video of the remote location distributed via the network and cause the live video to be displayed as a VR image on the HMD 10 in substantially real time.

[0184] That is, an image display system including a plurality of cameras each having a different imaging direction for imaging the real space, a display control device having the above-described configuration, and an HMD, or a system including the plurality of cameras, a distribution server, a display control device having the above-described configuration, and an HMD can be constructed to display a live video including a close-up object as a VR image on the HMD. [14-2] As described above, display control for causing a VR image representing a visual field from a virtual viewpoint in a VR space to be displayed on the display unit 11 of the HMD 10 as a stereoscopic image using binocular parallax has been described, but it can also be applied to display control of a non-stereoscopic image that does not use binocular parallax, that is, a two-dimensional (2D) image. That is, when binocular parallax is used, all or part of the above-described respective processes are executed for each of the right-eye image and the left-eye image, but for a 2D image, all or part of the above-described respective processes are executed for an image common to the right eye and the left eye. There is no difference in the processing itself between the stereoscopic image (right-eye image or left-eye image) and the non-stereoscopic image (left-and-right common image), and each of the above-described configurations can be directly applied to the display control of the non-stereoscopic image.

[0185] That is, it can be applied to a display control device that executes control for causing a VR image (3D image or 2D image) representing a field of view from a virtual viewpoint in a VR space to be displayed on a display unit of an HMD as a stereoscopic image or a non-stereoscopic image. Conventionally, when creating a wide-angle image (video or still image) by synthesizing a plurality of real-shot images obtained by imaging a real space with a plurality of cameras, regardless of whether it is a 3D image or a 2D image, one piece of image data completed by stitching processing is created. In contrast, in the present embodiment, regardless of whether there is stereoscopy, a plurality of real-shot images captured by a plurality of cameras having different imaging directions for imaging the real space are arranged in the VR space so that a partial overlapping region where the field of view regions of adjacent real-shot images overlap each other is generated, and based on the line-of-sight direction, among adjacent real-shot images, by dynamically switching the real-shot image to be displayed in the overlapping region, it is possible to realize display control that reduces the discomfort of the seam (misalignment) between real-shot images without performing stitching processing.

[0186] [14-3] As described above, the display control for causing an image to be displayed on the display unit of the HMD has been described, but it can also be applied to the display control for causing an image to be displayed on a display unit other than the HMD. For example, the application to the case of displaying a real-shot image with a wide field of view on a display unit such as a normal display that is not worn on the user's head or the like will be described below.

[0187] When displaying a wide-angle image (for example, an image with a 360-degree field of view) using a plurality of real-shot images obtained by imaging a real space with a plurality of cameras on a display unit, the user views the image while operating to change the range of the image displayed on the display unit using, for example, a mouse or other pointing device. Alternatively, when the display unit is a touch panel or the like equipped with a touch interface, the user views the wide-angle image while changing the range of the image displayed on the display unit by a touch operation with a finger or a stylus pen or the like. Alternatively, a hand tracking system that images the user with a camera and analyzes the movement of the hand may be used to view the image while operating to change the range of the image displayed on the display unit by the gesture of the user's hand. In this case, it is determined that the user's viewpoint or line of sight is at the center (center of the screen) of the current image range displayed on the display unit, and the line-of-sight direction is specified. Alternatively, for example, detection information regarding the user's line of sight may be acquired by an eye-tracking system or the like provided in the display unit or the like, and the line-of-sight direction may be specified based on the detection information.

[0188] In this way, even when a high field-of-view angle image using a plurality of real-world images captured by a plurality of cameras having different imaging directions for imaging the real space is displayed on a display unit other than the HMD, the display control may be performed as follows. That is, the plurality of real-world images are arranged in the VR space so that a partial overlapping region where the visual field regions of adjacent real-world images overlap each other is generated, and based on the line-of-sight direction, among the adjacent real-world images, the real-world image to be displayed in the overlapping region is dynamically switched. By this display control, even in the case of a display unit other than the HMD, it is possible to realize display control that reduces the discomfort of the seam (shift) between the real-world images without performing stitching processing. Note that the high field-of-view angle image displayed on the display unit other than the HMD may be a stereoscopic image using binocular parallax or a non-stereoscopic image. When displaying a stereoscopic image, an image for the right eye and an image for the left eye are displayed on a display unit capable of naked-eye stereoscopy. Alternatively, on the premise of using dedicated glasses (dedicated glasses such as the frame sequential method or the polarization method), the image for the right eye and the image for the left eye are alternately displayed on the display unit.

[0189] [14-4] As illustrated in FIG. 1, when both the HMD 10 and the display control device 20 have a configuration and function as an information processing device (computer) including a processor and a storage device, a part of the functions of the control unit 30 described above may be realized by the processor 13 of the HMD 10 that executes the program according to the present embodiment, and the remaining functions may be realized by the processor 21 of the display control device 20 that executes the program according to the present embodiment. FIG. 35 is a schematic block diagram showing an example of the configuration of a stand-alone type HMD 60 in which the functions of the display control device 20 are integrated with the HMD 10, or an information processing device 52 such as a smartphone used as the HMD 50 illustrated in FIG. 34. The HMD 60 or the information processing device 52 includes a display unit 61, a sensor 62, a processor 63, a storage device 64, an operation unit 65, a communication unit 66, and the like. These display unit 61, sensor 62, processor 63, storage device 64, operation unit 65, and communication unit 66 have the same configurations as the aforementioned display unit 11, sensor 12, processor 21, storage device 22, operation unit 23, and communication unit 24, respectively, and their descriptions are omitted. Note that the operation unit 65 or the communication unit 66 may be configured separately by external attachment or the like, or may be omitted. Also in the case of this stand-alone type HMD 60 or information processing device 52, it has a configuration and functions as an information processing device (computer) including a processor and a storage device, and each function of the control unit 30 described above is realized by a processor 63 that executes a program according to the present embodiment.

[0190] [14-5]Part or all of the functions of the control unit 30 described above may be realized by an integrated circuit such as an LSI (Large Scale Integration). Also, each of the above functions may be made into an individual processor. Alternatively, part or all of the above functions may be integrated and made into a processor.

[0191] [14-6] The computer-readable program according to this embodiment is recorded on various computer-readable recording media such as a hard disk, an optical disk (CD-ROM, DVD-ROM, etc.), a flexible disk, and a semiconductor memory, read from the recording media, and executed by a computer constituting the image display system 1 or the display control device 20. Further, the program can be provided to the computer via a network including a communication line such as the Internet, a WAN, a LAN, or a dedicated line. The computer may read a program stored in a file server (online storage). Also, the computer may receive a program distributed from a distribution server. The recording media include a recording media provided internally or externally accessible from the distribution server for distributing the program. The code of the program stored in the recording media of the distribution server does not have to be in a form directly executable on the computer that has received it. That is, the format of the program stored in the recording media of the distribution server is arbitrary as long as it can be installed to be executable on the computer after being downloaded from the distribution server. Also, the program may be divided into a plurality of parts, downloaded at different timings, and then combined. The distribution servers that distribute the respective divided programs may be different. Also, computer-readable recording media include those that hold a program for a certain period of time, such as volatile memory such as RAM in a server that transmits a program or a computer that receives it via a network. Also, the program may be a differential program that can realize the above-described functions in combination with a program already stored in the computer.

[0192] [15. Supplementary Note] The present invention can be understood as follows from the above description. For ease of understanding of the present invention, reference numerals in the accompanying drawings are appended in parentheses for convenience, but the present invention is not limited to the illustrated embodiments by this.

[0193] 1) A program according to one aspect of the present invention functions as a line-of-sight direction specifying unit (31) that specifies a line-of-sight direction (S) that is the direction of the user's line of sight with respect to the VR space, and an image generation unit (32) that arranges a plurality of real-shot images captured by a plurality of cameras with different imaging directions for imaging the real space in the VR space so that a overlapping region (AO) where a part of the visual field regions (A1, A2) of adjacent real-shot images overlap each other is generated, and generates the VR image according to the line-of-sight direction, to execute control for causing a head-mounted display (10) to display, as a stereoscopic image using binocular disparity, a VR image representing a visual field from a virtual viewpoint (P) in a virtual reality (VR) space (V) on a display unit (11). The image generation unit (32) dynamically switches the real-shot image to be displayed in the overlapping region among the adjacent real-shot images based on the line-of-sight direction. Here, the "computer" may be any device that includes at least a processor and a storage device (memory). For example, a stationary or portable dedicated game machine capable of performing display control of a VR image, a business (commercial) game machine, a personal computer, a tablet computer, a smartphone, a mobile phone terminal, a PHS terminal, a PDA, a multifunctional television receiver having an information processing function, and any other device including a processor and a storage device are all included in the "computer". Also, an HMD itself including a processor and a storage device is included in the "computer". For example, in the case of an HMD 50 including an attachment 51 and an information processing device 52 as illustrated in FIG. 34, the information processing device 52 such as a smartphone is included in the "computer". Also, a stand-alone type HMD is included in the "computer".

[0194] According to the aspect described in 1) above, by dynamically switching the real-world image to be displayed in the overlapping region based on the line-of-sight direction, the region boundary between adjacent real-world images in the overlapping region can be moved away from the direction of the user's line of sight. As a result, the discomfort of the "seam (misalignment) between adjacent real-world images" occurring at the region boundary can be reduced. Therefore, even if a close-up photographed object exists at or near the region boundary, a VR image with less discomfort is displayed for the user. That is, a VR image with less discomfort can be displayed regardless of the position of the close-up photographed object in the VR space, and VR image display control capable of displaying close-ups can be realized. In addition, since the stitching process is not required, the labor of VR image production can also be reduced. Further, since the stitching process is not required, a VR image with less discomfort at the seam between real-world images can be displayed on the display unit in substantially real time using the real-world image (live video) currently being captured by the camera.

[0195] 2) In one aspect of the present invention, in the aspect described in 1) above, the image generation unit (32) sets a reference direction (RD, RD1, RD2) in a predetermined direction from the virtual viewpoint (P), and based on the line-of-sight direction (S) with respect to the reference direction, identifies the real-world image to be displayed in the overlapping region (AO).

[0196] According to the aspect described in 2) above, switching control of the real-world image to be displayed in the overlapping region can be easily realized based on the line-of-sight direction with respect to the reference direction.

[0197] 3) In one aspect of the present invention, in the aspect described in 2) above, the image generation unit (32) sets the reference direction (RD, RD1, RD2) in the direction of a predetermined position within the overlapping region (AO) from the virtual viewpoint (P).

[0198] According to the aspect described in 3) above, since the reference direction is set within the overlapping region, even when the line-of-sight direction is directed towards the region boundary of the overlapping region, before reaching the region boundary, the real-world image to be displayed in the overlapping region is switched, and thus the region boundary also moves away from the direction of the user's line of sight. Thereby, VR image display control with excellent visibility can be realized.

[0199] 4) In one aspect of the present invention, in the aspect described in 2) or 3) above, the image generation unit (32) changes the reference direction (RD1, RD2) according to the real-world image to be displayed in the overlapping region at the timing when the real-world image to be displayed in the overlapping region (AO) is switched.

[0200] According to the aspect described in 4) above, the reference direction is not fixed and is changed according to the real-world image to be displayed in the overlapping region at the timing when the real-world image to be displayed in the overlapping region is switched. Thereby, after the real-world image is switched, the reference direction temporarily moves away from the line-of-sight direction, so that frequent switching of the real-world image to be displayed in the overlapping region can be reduced.

[0201] 5) In one aspect of the present invention, in the aspect described in any one of 1) to 4) above, the line-of-sight direction specifying unit (31) acquires detection information regarding the orientation of the head-mounted display (10) (for example, detection information of an angular velocity sensor), and specifies the line-of-sight direction (S) based on the detection information.

[0202] According to the aspect described in 5) above, the user can dynamically switch the real-world image to be displayed in the overlapping region by changing the orientation of the head-mounted display.

[0203] 6) In one aspect of the present invention, in the aspect described in any one of 1) to 4) above, the line-of-sight direction specifying unit (31) acquires detection information regarding the line of sight of the user (for example, detection information of eye tracking), and specifies the line-of-sight direction (S) based on the detection information.

[0204] According to the aspect described in 6) above, the user can dynamically switch the real image displayed in the overlapping area by changing the eye movement (line-of-sight direction).

[0205] 7) In one aspect of the present invention, in the aspect described in any of 1) to 6) above, the computer functions as a boundary line display unit (33) that displays boundary lines (BL1, BL1) visible to the user at the region boundaries (BD1, BD2) between adjacent real images.

[0206] According to the aspect described in 7) above, by deliberately displaying the boundary line visibly at the region boundary between adjacent real images, it becomes easier for the user to recognize the region boundary where an image seam (shift) occurs. This makes it easier for the user to determine which real image should be displayed in the overlapping area among adjacent real images so that the boundary line is not displayed in the direction the user desires.

[0207] 8) In one aspect of the present invention, in the aspect described in 7) above, the boundary line display unit (33) displays additional information (GD1, GD2) for indicating which side of the boundary line is the overlapping area (AO) on or near the boundary line (BL1, BL1).

[0208] According to the aspect described in 8) above, by enabling the user to recognize which side of the boundary line is the overlapping area, it becomes easier for the user to determine in which direction to change the line-of-sight direction for the real image displayed in the overlapping area to switch.

[0209] 9) In one aspect of the present invention, in the aspect described in any of 1) to 8) above, when switching the real image to be displayed in the overlapping area (AO), the image generation unit (32) gradually switches from one real image to the other real image according to the passage of time.

[0210] According to the aspect described in 9) above, when switching the real image to be displayed in the overlapping area from one to the other, rather than switching abruptly, by gradually switching during the passage of time in the switching period from the start to the end of the switching, the boundary line can be made more easily visible in a more natural manner.

[0211] 10) In one aspect of the present invention, in the aspect described in 9) above, when the image generation unit (32) switches the real image to be displayed in the overlapping area (AO), in accordance with the passage of time, while gradually moving the boundary line (BL1, BL1) toward the position to be displayed after the switching is completed, it gradually switches from one real image to the other real image.

[0212] According to the aspect described in 10) above, by moving the boundary line during the switching period of the real image to be displayed in the overlapping area, the position of the changed boundary line can be naturally recognized by the user.

[0213] 11) In one aspect of the present invention, in the aspect described in any one of 1) to 10) above, the image generation unit (32) provides a predetermined waiting period from when it determines that the switching process of the real image to be displayed in the overlapping area (AO) should be performed based on the line-of-sight direction (S) until it starts the switching process.

[0214] According to the aspect described in 11) above, by providing a predetermined waiting period until the switching process of the real image to be displayed in the overlapping area is started, when the line-of-sight direction changes during the waiting period and the switching process becomes unnecessary, the unnecessary switching process can be omitted.

[0215] 12) In one aspect of the present invention, in the aspect described in any one of 1) to 11) above, the image generation unit (32) changes the arrangement state of the plurality of real images in the VR space (V) during the display control of the VR image to the display unit.

[0216] According to the aspect described in the above (12), according to the scene, it is possible to change the number of viewing areas arranged in the VR space, the viewing angle of the entire viewing area, etc. That is, it is possible to optimize the VR space for each scene. Therefore, it is possible to eliminate unnecessary viewing areas for each scene and reduce the amount of image data. Also, in a scene where unnecessary viewing areas are eliminated, the resolution of the VR image displayed on the display unit of the HMD can be increased.

[0217] 13) In one aspect of the present invention, in the aspect described in any one of the above (1) to (11), the computer functions as an imaging direction specifying unit (34) that specifies a change in the imaging direction of a plurality of the cameras that have captured the plurality of the real-world images, and the image generation unit (32) changes the direction of the viewing area of the entire plurality of the real-world images arranged in the VR space (V) according to the change in the imaging direction specified by the imaging direction specifying unit.

[0218] According to the aspect described in the above (13), for example, in the case of a real-world image in which the imaging direction of the camera changes in accordance with the movement of a moving subject (such as a car), the direction of the viewing area of the entire real-world image arranged in the VR space also changes according to the change in the imaging direction. Therefore, a VR image with a sense of reality in which the subject moves with respect to the line-of-sight direction can be displayed on the display unit. Also, if a real-world image is prepared in which the subject is imaged so as not to cross the region boundary of adjacent real-world images during imaging in accordance with the movement of the subject described above, it is possible to reproduce a VR image with excellent visibility in which the subject does not cross the region boundary.

[0219] 14) In one aspect of the present invention, in the aspect described in any one of the above (1) to (13), the image generation unit (32) dynamically switches the real-world image to be displayed in the overlapping region (AO) among the adjacent real-world images so that the region boundaries (A1, A2) of the adjacent real-world images move away from the line-of-sight direction (S). Thereby, the same effect as the aspect described in the above (1) is obtained.

[0220] 15) In one aspect of the present invention, in the aspect described in any one of 1) to 14) above, the image generation unit (32) acquires detection information regarding the orientation of the head-mounted display (10), and based on the detection information, changes the range of the VR image to be displayed on the display unit (11).

[0221] According to the aspect described in 15) above, when the line-of-sight direction is specified based on the detection information regarding the orientation of the HMD, the user can dynamically switch the real-world image displayed in the overlapping region while changing the range of the VR image displayed on the display unit by adjusting the orientation of the HMD. Further, when the line-of-sight direction is specified based on the detection information regarding the user's line of sight such as eye tracking, the user can adjust the range of the VR image displayed on the display unit by changing the orientation of the HMD, and can also dynamically switch the real-world image displayed in the overlapping region by changing the movement of the eyes (the direction of the line of sight).

[0222] 16) In one aspect of the present invention, in the aspect described in any one of 1), 5) to 14) above, the image generation unit (32) specifies the real-world image to be displayed in the overlapping region AO based on the line-of-sight direction (S) with respect to the current region boundaries (BD1, BD2) of adjacent real-world images.

[0223] According to the aspect described in 16) above, the switching control of the real-world image to be displayed in the overlapping region can be easily realized without setting a reference direction. Further, since the reference region boundary is changed at the timing when the real-world image is switched, it is possible to reduce the frequent switching of the real-world image displayed in the overlapping region.

[0224] 17) In one aspect of the present invention, in the aspect described in 15) above, when the line-of-sight direction (S) approaches the current region boundaries (BD1, BD2) of adjacent real-world images by a predetermined amount or more (for example, when the angle formed by the line-of-sight direction and the current region boundary becomes a predetermined value or less), the image generation unit (32) switches the real-world image currently displayed in the overlapping region (AO) to the other real-world image.

[0225] According to the aspect described in 17) above, before the line-of-sight direction reaches the region boundary, the real image to be displayed in the overlapping region is switched, and the region boundary can be moved away from the direction of the user's line of sight.

[0226] 18) In one aspect of the present invention, in the aspect described in 8) above, the additional information is superimposed on the real image near the boundary line (BL1, BL1), and includes a semi-transparent gradation part (GD1, GD2) whose density or opacity gradually or stepwise decreases as it moves away from the boundary line.

[0227] According to the aspect described in 18) above, the image misalignment occurring near the boundary line can be hidden by the gradation so as not to be conspicuous. Also, the gradation can naturally represent the direction of the change in the line-of-sight direction required for switching the real image.

[0228] 19) In one aspect of the present invention, in the aspect described in any one of 7) to 10) above, the boundary line display unit (33) makes the display forms of the boundary line (BL1 or BL2) displayed on the region boundary (BD1 or BD2) at one end of the overlapping region (AO) and the boundary line displayed on the region boundary at the other end different.

[0229] According to the aspect described in 19) above, when using a plurality of images arranged in the horizontal direction to expand the horizontal viewing angle, the user can recognize whether the currently displayed boundary line is the misaligned part of the image at the left end of the overlapping region or the misaligned part of the image at the right end. Also, when using a plurality of images arranged in the vertical direction to expand the vertical viewing angle, the user can recognize whether the currently displayed boundary line is the misaligned part of the image at the upper end of the overlapping region or the misaligned part of the image at the lower end.

[0230] 20) The display control device (20) according to one aspect of the present invention executes control for displaying, on the display unit (11) of the head-mounted display (10), a VR image representing a field of view from a virtual viewpoint (P) in a virtual reality (VR) space (V) as a stereoscopic image using binocular disparity. The display control device includes a line-of-sight direction specifying unit (31) that specifies a line-of-sight direction (S) that is the direction of the user's line of sight with respect to the VR space, and an image generation unit (32) that arranges a plurality of real-shot images captured by a plurality of cameras each having a different imaging direction for imaging the real space in the VR space so that a partial overlapping region (AO) where the field-of-view regions (A1, A2) of adjacent real-shot images overlap each other is generated, and generates the VR image corresponding to the line-of-sight direction. The image generation unit (32) dynamically switches the real-shot image to be displayed in the overlapping region among the adjacent real-shot images based on the line-of-sight direction. Thereby, the same effect as the aspect described in 1) above is achieved.

[0231] 21) An image display system (1) according to one aspect of the present invention includes a head-mounted display (10) that displays, as a stereoscopic image using binocular disparity, a VR image representing a field of view from a virtual viewpoint (P) in a virtual reality (VR) space (V) on a display unit (11), and a display control device (20) that executes control for causing the display unit to display the VR image. The display control device (20) includes a line-of-sight direction specifying unit (31) that specifies a line-of-sight direction (S) that is the direction of the user's line of sight with respect to the VR space, and an image generation unit (32) that arranges a plurality of real-shot images captured by a plurality of cameras (200) each having a different imaging direction for imaging the real space in the VR space so that a partial overlapping region (AO) where the field-of-view regions (A1, A2) of adjacent real-shot images overlap each other is generated, and generates the VR image corresponding to the line-of-sight direction. The image generation unit (32) dynamically switches the real-shot image to be displayed in the overlapping region among the adjacent real-shot images based on the line-of-sight direction. Thereby, the same effect as the aspect described in 1) above is achieved.

[0232] 22) The information storage medium according to one aspect of the present invention is a computer-readable information storage medium on which the program according to any one of the aspects described in 1) to 19) is recorded. Thereby, the same effects as those of the aspects described in the above 1) to 19) are achieved.

[0233] 23) A control method for a display control device (20) according to one aspect of the present invention is a method for controlling a display control device (20) that executes control for causing a VR image representing a visual field from a virtual viewpoint (P) in a virtual reality (VR) space (V) to be displayed on a display unit (11) of a head-mounted display (10) as a stereoscopic image using binocular disparity. The method includes a line-of-sight direction specifying step (S106) for specifying a line-of-sight direction (S) that is the direction of the user's line of sight with respect to the VR space, and a plurality of real images captured by a plurality of cameras having different imaging directions for imaging the real space are arranged in the VR space (S102) such that an overlapping region (AO) in which a part of the visual field regions (A1, A2) of adjacent real images overlaps each other is generated, and an image generation step (S108 to S114) for generating the VR image according to the line-of-sight direction. The image generation step includes a step (S108 to S112) of dynamically switching the real image to be displayed in the overlapping region among the adjacent real images based on the line-of-sight direction. Thereby, the same effects as those of the aspect described in 1) above are achieved.

Description of Reference Numerals

[0234] 1…Image display system, 10…HMD, 11·61…Display unit, 12·62…Sensor, 13…Processor, 14…Storage device, 20…Display control device, 21·63…Processor, 22·64…Storage device, 23·65…Operation unit, 24·66…Communication unit, 30…Control unit, 31…Line-of-sight direction specifying unit, 32…Image generation unit, 321…Image arrangement unit, 322…Switching unit, 33…Boundary line display unit, 34…Imaging direction specifying unit, 60…HMD (information processing device), 200…Camera, A1·A2…Visual field region, AO…Overlapping region, RD…Reference direction, V…VR space, S…Line-of-sight direction, BL1·BL2…Boundary line, GD1·GD2…Gradation unit

Claims

1. A computer that executes control for displaying a virtual reality (VR) image, which represents a field of view from a virtual viewpoint in a VR space, on a display unit of a head mounted display as a stereoscopic image utilizing binocular parallax, A gaze direction specification unit that specifies a gaze direction as a direction of a user's gaze with respect to the VR space; an image generating unit that arranges a plurality of real-life images captured by a plurality of cameras each having a different imaging direction for capturing an image of a real space in the VR space such that an overlapping area is generated in which a part of the visual field area of ​​adjacent real-life images overlaps with another, and generates the VR image according to the visual line direction; and make it work. The image generating unit sets a reference direction in a predetermined direction from the virtual viewpoint, and when the line of sight direction exceeds the reference direction, dynamically switches the real-life image to be displayed in the overlapping area from among the adjacent real-life images. program.

2. The image generation unit sets the reference direction to a predetermined direction from the virtual viewpoint, and identifies the real-life image to be displayed in the overlapping area based on the line-of-sight direction with respect to the reference direction. The program according to claim 1.

3. The image generating unit sets the reference direction to a direction from the virtual viewpoint to a predetermined position within the overlapping area. The program according to claim 2.

4. The image generating unit changes the reference direction according to the real-life image to be displayed in the overlapping area at a timing when the real-life image to be displayed in the overlapping area is switched. The program according to claim 2.

5. The gaze direction specification unit acquires detection information regarding an orientation of the head mounted display, and specifies the gaze direction based on the detection information. The program according to claim 1.

6. The gaze direction identification unit acquires detection information regarding the user's gaze, and identifies the gaze direction based on the detection information. The program according to claim 1.

7. and causing the computer to function as a boundary display unit that displays a boundary line visible to the user at the area boundary between the adjacent real-life images. The program according to claim 1.

8. The boundary display unit displays additional information on or near the boundary to indicate which side of the boundary is the overlapping area. The program according to claim 7.

9. When switching the real-life images to be displayed in the overlapping area, the image generating unit gradually switches from one real-life image to the other real-life image as time passes. The program according to claim 7.

10. When switching the live-action image to be displayed in the overlapping area, the image generating unit gradually switches from one live-action image to the other live-action image while gradually moving the boundary line toward a position to be displayed after the switching is completed as time passes. The program according to claim 9.

11. The image generating unit provides a predetermined waiting period from when it is determined that a switching process of the real-life image to be displayed in the overlapping area should be performed based on the line of sight direction until when the switching process is started. The program according to claim 1.

12. The image generation unit changes the arrangement state of the plurality of real-life images in the VR space during display control of the VR image on the display unit.

12. The program according to claim 1.

13. A display control device that executes control for displaying a virtual reality (VR) image, which represents a field of view from a virtual viewpoint in a VR space, on a display unit of a head mounted display as a stereoscopic image utilizing binocular parallax, A gaze direction specification unit that specifies a gaze direction as a direction of a user's gaze with respect to the VR space; an image generating unit that arranges a plurality of real-life images captured by a plurality of cameras each having a different imaging direction for capturing an image of a real space in the VR space such that an overlapping area is generated in which a part of the visual field area of ​​adjacent real-life images overlaps with another, and generates the VR image according to the visual line direction; Including, The image generating unit sets a reference direction in a predetermined direction from the virtual viewpoint, and when the line of sight direction exceeds the reference direction, dynamically switches the real-life image to be displayed in the overlapping area from among the adjacent real-life images. Display control device.

14. An image display system including: a head mounted display that displays, on a display unit, a virtual reality (VR) image that represents a field of view from a virtual viewpoint in a VR space as a stereoscopic image utilizing binocular parallax; and a display control device that executes control for displaying the VR image on the display unit, The display control device includes: A gaze direction specification unit that specifies a gaze direction as a direction of a user's gaze with respect to the VR space; an image generating unit that arranges a plurality of real-life images captured by a plurality of cameras each having a different imaging direction for capturing an image of a real space in the VR space such that an overlapping area is generated in which a part of the visual field area of ​​adjacent real-life images overlaps with another, and generates the VR image according to the visual line direction; Including, The image generating unit sets a reference direction in a predetermined direction from the virtual viewpoint, and when the line of sight direction exceeds the reference direction, dynamically switches the real-life image to be displayed in the overlapping area from among the adjacent real-life images. Image display system.

Images