PROGRAM, DISPLAY CONTROL DEVICE AND IMAGE DISPLAY SYSTEM

Abstract

To achieve display control of VR images with less discomfort even if they include short-distance imaging objects when displaying the VR images on an HMD using a plurality of images imaged by a plurality of cameras.SOLUTION: A gaze direction identification unit identifies a gaze direction which is a direction of a user's line of sight to a VR space. An image generation unit generates VR images according to the gaze direction by arranging, in the VR space, a plurality of actual images imaged by a plurality of cameras with mutually different imaging directions for imaging a real space so that overlapping areas are generated where portions of viewing areas of adjacent actual images overlap each other. The image generation unit dynamically switches the actual image to be displayed in the overlapping area among the adjacent actual images based on the gaze direction.SELECTED DRAWING: Figure 14

Description

[Technical field]

[0001] The present invention relates to a program, a display control device, and an image display system. [Background technology]

[0002] Conventionally, a head mounted display (HMD) that displays a virtual reality (VR) image that can be viewed stereoscopically using binocular parallax has been known. This HMD can display real-life images (still images or videos) captured in a computer-generated VR space or real space. In the case of real-life images, for example, a single stereo camera using a wide-angle lens can obtain a wide-viewing-angle image with a certain degree of viewing angle, but stereoscopic viewing is difficult in a direction far away from the visual axis direction of the camera. In order to reduce the problem of difficulty in stereoscopic viewing, there is a method of obtaining a wide-viewing-angle image by synthesizing multiple images captured in separate regions using multiple cameras (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2011-227306 A Summary of the Invention [Problem to be solved by the invention]

[0004] Each of the cameras has a volume, and it is physically impossible to arrange the cameras so that the imaging centers are concentrated at one point. For this reason, images captured by the multiple cameras will have a misalignment at the boundary between adjacent images, resulting in a seam (such as an extreme difference in the shading between pixels). For this reason, when capturing images using multiple cameras with separate areas, stitching processing is required to join the images captured by each camera when creating a single final image data. This stitching processing can be performed automatically using dedicated software, but in order to create a more natural image with no visible seams, it must be performed manually and laboriously. Here, the degree of misalignment between adjacent images depends on the distance between the camera and the subject, and the closer the subject is to the camera, the greater the degree of misalignment, making stitching processing more difficult. In particular, when the distance between the camera and the subject is less than a certain level (for example, less than 2 m), the degree of misalignment becomes quite large, making stitching difficult. For this reason, with the conventional technology of stitching together multiple images taken by multiple cameras, it was virtually impossible to create VR images that included close-up objects shot within a certain distance from the camera and display them naturally on the HMD.

[0005] Therefore, one of the objects of the present invention is to realize display control of VR images that creates little discomfort even when close-up objects are included when a VR image is displayed on an HMD using multiple images captured by multiple cameras. [Means for solving the problem]

[0006] A program according to one aspect of the present invention causes a computer that executes control to display on a display unit of a head-mounted display a VR image representing the field of view from a virtual viewpoint in a virtual reality (VR) space as a stereoscopic image utilizing binocular parallax, to function as: a gaze direction identification unit that identifies a gaze direction that is the direction of the user's gaze toward the VR space; and an image generation unit that arranges a number of live-action images captured by a number of cameras each having different imaging directions for capturing real space in the VR space so as to create an overlapping area in which parts of the field of view of adjacent live-action images overlap with each other, and generates the VR image according to the gaze direction, and the image generation unit dynamically switches between the live-action images to be displayed in the overlapping area among the adjacent live-action images based on the gaze direction. [Brief description of the drawings]

[0007] [Figure 1] 1 is a schematic block diagram showing an example of a hardware configuration of an image display system according to an embodiment of the present invention. [Diagram 2] FIG. 1 is a diagram illustrating definitions of directions in a VR space. [Diagram 3] FIG. 1 is a diagram for explaining a VR image that represents a field of view from a virtual viewpoint in a VR space. [Figure 4] FIG. 1 is a diagram for explaining problems with real images captured by one stereo camera. [Diagram 5] FIG. 1 is a diagram showing an example of capturing an image using a plurality of stereo cameras. [Figure 6] FIG. 13 is a diagram for explaining an example in which a misalignment occurs between images captured by two adjacent cameras. [Figure 7] FIG. 13 is a diagram showing an example in which an area is divided into two and imaged by two cameras with different imaging directions. [Figure 8] 1 is a diagram conceptually illustrating an example of a field of view area of ​​actual images captured by two cameras each having a different imaging direction. [Figure 9] This is a conceptual diagram showing an example in which adjacent real-life images are arranged in a VR space so that an overlapping area occurs where the viewing areas of the images partially overlap each other. [Figure 10] 11 is a diagram illustrating an example in which a first image is displayed in an overlapping area in a VR space. [Figure 11] 13 is a diagram illustrating an example in which a second image is displayed in an overlapping area in the VR space. [Figure 12] FIG. 13 is a diagram showing an example in which a reference direction is set outside an overlapping region. [Figure 13] 13 is a diagram showing an example of dynamically switching between real-life images to be displayed in an overlapping area based on the current area boundaries of adjacent real-life images; FIG. [Figure 14] FIG. 2 is a schematic functional block diagram showing an example of a functional configuration of a display control device. [Figure 15] 10 is a flowchart illustrating an example of processing performed by a display control device. [Figure 16] FIG. 13 is a diagram illustrating an example of changing the reference direction. [Figure 17] 11 is a flowchart illustrating an example of a process for changing a reference direction; [Figure 18] FIG. 2 is a diagram showing an example of a screen displayed on a display unit of an HMD. [Figure 19] FIG. 2 is a diagram showing an example of a screen displayed on a display unit of an HMD. [Figure 20] FIG. 2 is a schematic functional block diagram showing an example of a functional configuration of a display control device. [Figure 21] FIG. 13 is a diagram showing an example of a screen on which a gradation portion is displayed together with a boundary line. [Figure 22] FIG. 13 is a diagram showing an example of a screen on which a gradation portion is displayed together with a boundary line. [Diagram 23] 13 is a flowchart illustrating an example of a process for displaying a boundary line. [Figure 24] 13 is a flowchart showing an example of a process for gradually switching between real-life images. [Diagram 25] FIG. 13 is a diagram showing an example of a screen on which a boundary line moves; [Figure 26] This is a conceptual diagram showing an example in which the viewing areas of a left image, a front image, and a right image captured by three cameras are arranged in a VR space. [Figure 27]13 is a flowchart showing an example of switching processing of real-life images in an overlapping area. [Figure 28] 11A and 11B are diagrams illustrating an example in which the arrangement state of an image in a VR space is changed depending on the scene. [Figure 29] 13 is a flowchart illustrating an example of a process for changing the arrangement state of an image in a VR space. [Diagram 30] 27 is a diagram conceptually illustrating an example of a viewing area of ​​the VR space when the camera in FIG. 26 is rotated to the right. FIG. [Diagram 31] 11 is a diagram showing an example of a change in the viewing area of ​​the VR space when information on the rotation of the imaging direction is not used. FIG. [Diagram 32] 11 is a diagram showing an example of a change in the viewing area of ​​the VR space when information on the rotation of the imaging direction is used. FIG. [Diagram 33] 13 is a flowchart illustrating an example of a process for changing the direction of the entire viewing area of ​​the VR space. [Diagram 34] FIG. 2 is a diagram illustrating an example of an HMD. [Diagram 35] 1 is a schematic block diagram showing an example of a hardware configuration of a stand-alone HMD or an information processing device used as an HMD. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, an example of an embodiment of the present invention will be described with reference to the drawings.

[0009] [1. Example of image display system configuration] 1 is a schematic block diagram showing an example of a hardware configuration of an image display system 1 according to an embodiment of the present invention. The image display system 1 includes an HMD 10 and a display control device 20 that executes display control for the HMD 10.

[0010] The HMD 10 is worn on the user's head (see FIG. 2) and can display a wide-angle image (still image or video) that can be viewed stereoscopically using binocular parallax. The HMD 10 also includes a sensor 12 such as a gyro sensor that detects the movement and tilt of the HMD 10, and detects changes in the movement and tilt of the head of the user wearing the HMD 10, and displays a VR image that represents the field of view in the VR space according to the changes. For example, the VR image displayed on the HMD 10 changes to a VR image corresponding to the field of view to the right in the VR space when the user's head is turned to the right, and changes to a VR image corresponding to the field of view to the upward direction in the VR space when the user's head is turned to the top, so that the user can feel immersed as if they were actually there.

[0011] The HMD 10 mainly includes a display unit 11, a sensor 12, a processor 13, a storage device 14, and the like. The display unit 11 is a display that displays various information such as VR images and text. The display unit 11 may be, for example, a display of a virtual image projection type that forms a virtual image by using a half mirror or the like. The display unit 11 may also be, for example, a display of a retinal projection type that forms a VR image directly on the retina by using the crystalline lens of the user's eye. The display unit 11 displays a stereoscopic image (an image for the right eye and an image for the left eye) using binocular parallax as a VR image that represents the field of view from a virtual viewpoint in the VR space.

[0012] The sensor 12 detects attitude information such as the rotation angle and tilt of the HMD 10. Detection information regarding the orientation of the HMD 10 can be obtained by the output of this sensor 12. For example, the sensor 12 is an angular velocity sensor (gyro sensor) that detects the angular velocity of an object. The sensor 12 may be a sensor that detects a change in direction, or a sensor that detects the direction itself. For example, the sensor 12 is not limited to a gyro sensor, and may be an acceleration sensor, an angular acceleration sensor, a tilt sensor, a geomagnetic sensor, or the like, and may be realized by appropriately combining these.

[0013] The processor 13 functions as a control center that controls each unit included in the HMD 10. For example, the processor 13 is a CPU (Central Processing Unit). The processor 13 may be configured to include hardware such as a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), or an FPGA (Field Programmable Gate Array) in addition to or instead of the CPU. The processor 13 performs control to supply a VR image signal received from the display control device 20 to the display unit 11 via an interface, for example. The storage device 14 stores programs executed by the processor 13 and temporarily stores data and parameters processed by the processor 13. The storage device 14 includes, for example, a read only memory (ROM), a random access memory (RAM), a video random access memory (VRAM), an auxiliary storage device, etc. As the auxiliary storage device, a non-volatile semiconductor memory, a hard disk drive, a solid state drive, etc. can be used.

[0014] The HMD 10 may also include a network adapter, etc., so that VR images can be input via a network. The HMD 10 may also include an audio output unit, a GPS (Global Positioning System) receiving unit, etc.

[0015] The display control device 20 is connected to the HMD 10 wirelessly or via a wired connection so as to be able to communicate with the HMD 10, and executes display control for the HMD 10. The display control device 20 is, for example, a stationary or portable game machine. The display control device 20 may be an arcade (commercial) game machine installed in an amusement facility or the like. Alternatively, the display control device 20 may be a personal computer, a tablet computer, a smartphone, a mobile phone terminal, a PHS (Personal Handy-phone System) terminal, a portable information terminal (PDA: Personal Digital Assistant), a multi-function television receiver equipped with an information processing function (a so-called smart TV), or the like.

[0016] The display control device 20 mainly comprises a processor 21, a storage device 22, an operation unit 23, and a communication unit 24, which are mutually connected via bus lines including an address bus, a data bus, a control bus, etc. Note that an interface circuit, an image processing unit, a sound processing unit, etc. are interposed between the bus lines and each component as necessary, but are not shown here.

[0017] The processor 21 interprets and executes program instructions, and controls the entire display control device 20. For example, the processor 21 is a CPU. The processor 21 may be configured to include hardware such as a GPU, a DSP, or an FPGA in addition to or instead of the CPU. The storage device 22 includes, for example, a ROM, a RAM, a VRAM, and an auxiliary storage device. The ROM stores programs and data necessary for basic operation control of the display control device 20. The RAM or VRAM stores various programs and data, and secures a working area for the processor 21. The auxiliary storage device stores programs and various data, and may be, for example, a non-volatile semiconductor memory, a hard disk drive, a solid state drive, or the like.

[0018] The display control device 20 may also include a recording medium drive. Examples of the recording medium drive include a DVD-ROM drive, a CD-ROM drive, a hard disk drive, an optical disk drive, a flexible disk drive, a silicon disk drive, and a cassette medium reader. In this case, examples of the recording medium include a DVD-ROM, a CD-ROM, a hard disk, an optical disk, a flexible disk, and a semiconductor memory. The recording medium drive reads out image data, audio data, and program data from the recording medium, and supplies the read data to the RAM of the storage device 22 or the like via a decoder.

[0019] For example, image data including real-life images captured in advance of a real space for VR images, and programs such as games that include the image data, are stored in a storage device 22 (such as an auxiliary storage device) or read from a recording medium drive.

[0020] The operation unit 23 is for the user to input various operation commands to the display control device 20. For example, the user performs an operation for viewing a VR image, or an operation for a game including a VR image. Examples of the operation unit 23 include a position input unit (a component of a touch panel, etc.) with a touch interface, a physical button, a controller, an analog stick, a keyboard, a pointing device, etc. Also, the operation unit 23 may be configured as a voice-inputtable operation unit 23 by identifying a voice input from a voice input unit such as a microphone. Also, the operation unit 23 may be for performing an operation by a gesture of the user.

[0021] The communication unit 24 includes a communication interface (not shown) and has a communication control function for data communication during execution of a game or the like. Here, the communication control function for data communication includes, for example, an Internet connection function, a wireless LAN (Local Area Network) connection function, and a short-distance wireless communication function using a predetermined frequency band (for example, a 2.4 GHz frequency band). The communication unit 24 transmits a connection signal for connecting the display control device 20 to a network based on a command from the processor 21, and also receives information transmitted from a communication partner and supplies it to the processor 21.

[0022] As described later, the image display system 1 or the display control device 20 of this embodiment can reduce the discomfort caused by the seams (misalignment) between multiple real-life images captured by multiple cameras without stitching processing when displaying a VR image on the HMD 10. Therefore, it is also possible to receive live video from a remote location distributed via a network by the communication unit 24 and display the live video as a VR image on the HMD 10 in approximately real time.

[0023] The display control device 20 may also be connected to a tracking unit that detects the position of the HMD 10 (or the position of the head of the user wearing the HMD 10). For example, the HMD 10 includes multiple light sources (LEDs, etc.) for tracking, and the tracking unit includes an image capture unit for capturing images of the HMD 10. The tracking unit, which is fixed to a predetermined position, identifies the position of the HMD 10 based on the positions of the multiple light source units that appear in an image captured by the image capture unit. The display control device 20 may also include a display unit such as a liquid crystal display or an organic EL (Electro-Luminescence) display.

[0024] [2. About the direction of VR space] 2 is a diagram showing the definition of the directions of the VR space according to this embodiment. In this embodiment, the vertical direction in which a user wearing an HMD 10 that displays a VR image in the VR space stands upright on his / her head is defined as the Z axis. The horizontal axis perpendicular to the Z axis and the direction of the user's line of sight facing forward (reference line of sight direction for the VR space) is defined as the X axis, and the horizontal axis perpendicular to the Z axis and the X axis is defined as the Y axis.

[0025] For example, when the image display system 1 is started, the visual axis direction of the HMD 10 (the direction perpendicular to the display surface of the display unit 11 of the HMD 10) is set as the reference line of sight direction for the VR space. Also, for example, the user wearing the HMD 10 on his / her head and facing forward may adjust the reference line of sight direction for the VR space by performing a predetermined operation.

[0026] Here, a change in the rotation direction about the Z axis is called a change in the yaw direction (left and right direction), a change in the rotation direction about the Y axis is called a change in the pitch direction (up and down direction), and a change in the rotation direction about the X axis is called a change in the roll direction. For example, the above-mentioned sensor 12 of the HMD 10 detects the angular velocity or angular acceleration in the rotation direction of each axis (yaw direction, pitch direction, and roll direction). Note that a change in the yaw direction is sometimes called a change in the left and right direction, and a change in the pitch direction is sometimes called a change in the up and down direction.

[0027] FIG. 3 is a diagram for explaining a VR image (field of view image) showing a field of view from a virtual viewpoint in a VR space according to this embodiment. In this figure, a virtual viewpoint P (user's virtual viewpoint) in a VR space V is set as an intersection (origin) of the X-axis, Y-axis, and Z-axis. For example, the user faces forward, and the direction of the user's line of sight with respect to the VR space V is on the X-axis (reference line of sight direction). In this case, the range of the VR image showing the field of view from the virtual viewpoint P in the VR space V is a range determined by a yaw angle α (the inner angle between the dashed lines a and b, and the inner angle between the dashed lines c and d) and a pitch angle β (the inner angle between the dashed lines a and d, and the inner angle between the dashed lines b and c) centered on the reference line of sight direction (X-axis direction). Here, the yaw angle α is a horizontal field of view angle, and the pitch angle β is a vertical field of view angle, which are preset as the field of view angles of the VR image to be displayed on the HMD 10 in the image display system 1.

[0028] For example, when the head of the user wearing the HMD 10 changes in the pitch direction or yaw direction, the change in the orientation and posture of the HMD 10 (change in the visual axis direction of the HMD 10) is detected by the sensor 12 or the like. Based on the detection information by the sensor 12 or the like, the processor 21 of the display control device 20 determines that the line of sight direction with respect to the VR space V has changed from the X-axis direction (reference line of sight direction) to the pitch direction or yaw direction. In response to this change in the line of sight direction with respect to the VR space V, the range of the VR image displayed on the display unit 11 is changed. Similarly, when the head of the user wearing the HMD 10 changes in the roll direction, the change is detected by the sensor 12 or the like, and the range of the VR image displayed on the display unit 11 rotates in the roll direction while the line of sight direction remains in the X-axis direction. In this way, the range of the VR image displayed on the display unit 11 is changed in response to the orientation (posture) of the HMD 10.

[0029] In addition, display objects other than the actual image, such as various objects, lines, symbols, and characters, can be arranged as necessary in the VR space V. For example, a "boundary line" (to be described later) may be superimposed on the actual VR image and displayed on the display unit 11.

[0030] When displaying a stereoscopic image using binocular parallax, there are viewing directions corresponding to the virtual viewpoints for the right eye and the left eye, and the VR image for the right eye and the VR image for the left eye for each viewing direction are displayed on the display unit 11 of the HMD 10.

[0031] [3. About actual images captured by the camera] In the image display system 1 of this embodiment, a real-life image captured by a camera that captures real space is used to generate a stereoscopic VR image utilizing binocular parallax, and the generated image is displayed on the HMD 10. Before explaining the real-life image used in the image display system 1 of this embodiment, problems with real-life images captured by a single camera will be explained.

[0032] FIG. 4 is a diagram for explaining problems with a real-life image captured by one camera 100 (a so-called stereo camera) equipped with a left-eye camera 101 and a right-eye camera 102. The camera 100 captures images of objects A and B that are at different positions in real space and at the same distance from the camera 100. The object A is in the visual axis direction (≒ direction toward the lens center) XC of the camera 100, and the object B is in the right direction with respect to the visual axis direction XC. Compared to the angle (convergence angle) θ1 between the object A and the left-eye camera 101 and the right-eye camera 102, the angle θ2 between the object B and the left-eye camera 101 and the right-eye camera 102 is smaller even though the objects A and B are at the same distance from the camera 100. For this reason, there is a problem that in a direction that is far from the visual axis direction XC of the camera 100, it is difficult to recognize the sense of distance, resulting in an image that is difficult to view stereoscopically.

[0033] In order to solve the above problem, it is effective to use a plurality of cameras 100 each having different imaging directions for imaging the real space, and capture the image by dividing the imaging area, as illustrated in FIG. 5. In the example of FIG. 5, the imaging area is divided into three areas, left, front, and right, and captured by three cameras 100. By dividing the entire imaging area into a plurality of areas in this way, the visual axis direction XC of the camera exists in each imaging area, and the number of directions that are significantly different from the visual axis direction XC can be reduced. Therefore, by using a plurality of images captured by a plurality of cameras 100 each having different imaging directions, the problem of images that are difficult to view stereoscopically can be reduced.

[0034] Conventionally, multiple images captured by multiple cameras with separate regions were finally combined into one image data. For this reason, it was necessary to perform a stitching process to join adjacent images captured by each camera. This stitching process can be performed automatically by dedicated software, but in order to create a more natural image with no visible seams, it is necessary to perform the stitching process manually, which requires a lot of work. One of the factors that make the stitching process time-consuming is the misalignment of the cameras used for capturing images. That is, although multiple cameras are placed in positions close to each other, it is physically impossible to place them in the exact same position at the same time since they capture real space. Therefore, a misalignment occurs near the boundary between adjacent images captured by adjacent cameras. This will be explained below.

[0035] FIG. 6 is a diagram for explaining an example in which a misalignment occurs between images captured by camera 100a capturing an area in front and camera 100b capturing an area to the right. Here, for convenience, an example in which images are captured by left-eye camera 101 of each of two cameras 100a and 100b is explained, but the same applies to the case in which images are captured by right-eye camera 102. Objects C and D to be captured are located to the right of camera 100a and to the left of camera 100b, and exist near the boundary between the imaging areas of cameras 100a and 100b. Object C exists behind object D (at a position farther away from cameras 100a and 100b). In this case, when the camera 100a capturing an area in front captures an image in which object D is located slightly to the right of object C located in the back, an image is captured. On the other hand, when the camera 100b capturing an area to the right captures an image in which object D is located slightly to the left of object C, an image is captured. Thus, even though the cameras 100a and 100b simultaneously capture the same objects C and D, respectively, there is a clear misalignment between the images captured by the two cameras.

[0036] In the conventional stitching process, in order to make the above-mentioned misalignment between adjacent images more natural, it is necessary to manually process (disguise) the images, which is time-consuming. The degree of misalignment near the region boundary between the above-mentioned adjacent images depends on the distance between the cameras 100a and 100b and the captured object. That is, when the captured object (such as the above-mentioned object C or D) near the region boundary is located sufficiently far from the cameras 100a and 100b, the degree of misalignment between the adjacent images is also small, and the conventional stitching process is relatively easy. On the other hand, when the captured object near the region boundary is located close to the cameras 100a and 100b, the degree of misalignment between the adjacent images is relatively large, and stitching becomes difficult. In particular, when the captured object near the region boundary and the cameras 100a and 100b are less than a certain distance (for example, less than 2 m), the degree of misalignment between the adjacent images becomes quite large, making stitching difficult. Therefore, in the conventional method in which multiple images taken with multiple cameras were stitched together to create a single VR image data set, it was virtually impossible to create VR image data that included close-up shots of objects that were within a certain distance from the camera.

[0037] Therefore, in the image display system 1 or the display control device 20 of the present embodiment, the conventional stitching process is not performed on a plurality of images captured by a plurality of cameras, and the display control of the VR image described below is executed.

[0038] [4. VR image display control] The outline of the VR image display control executed in the image display system 1 or the display control device 20 of the present embodiment is as follows. That is, the image display system 1 or the display control device 20 executes a characteristic process of arranging a plurality of real-life images captured by a plurality of cameras each having a different imaging direction for capturing a real space in a VR space without performing stitching processing. Here, when arranging the plurality of real-life images in the VR space, they are arranged in the VR space so that an overlapping area occurs in which a part of the visual field area of ​​adjacent real-life images overlaps with each other. Then, the area boundary between adjacent real-life images in the overlapping area is dynamically changed according to the line of sight direction (a direction that can be determined as the direction of the user's line of sight) with respect to the VR space. This realizes a characteristic display control in which image misalignment that is likely to occur at the area boundary between adjacent real-life images is moved as far away from the line of sight direction as possible, and a VR image in which unnaturalness due to the image misalignment is reduced is displayed on the HMD 10 without performing stitching processing.

[0039] In explaining the VR image display control of this embodiment, in the following, in order to simplify the explanation, an example will be explained in which the area is divided into two areas, left and right, in order to expand the horizontal viewing angle (i.e., an example using two real-life images captured by two cameras capturing the left and right directions, respectively). The number of divided areas may be two or more, so it may be divided into three directions, the left direction, the front direction, and the right direction, as described above, or may be further divided into four directions by adding the rear direction. It may also be divided more finely, for example, into areas in eight directions. In addition, in order to expand the vertical viewing angle, it may be divided into two directions, up and down (or three or more directions). Also, in the following, for the sake of simplicity, the tilt of the HMD 10 will be considered only in the yaw direction (left-right direction), without taking into account the pitch direction (up-down direction) and roll direction (direction of visual axis rotation). In addition, when a stereoscopic VR image utilizing binocular parallax is displayed on the display unit 11 of the HMD 10, a VR image for the right eye and a VR image for the left eye are displayed, but in the following, for the sake of simplicity, the explanation will be given without distinguishing between the VR image for the right eye and the VR image for the left eye.

[0040] 7 is a diagram showing an example in which an area with a horizontal angle of view of 180 degrees is divided into two areas and imaged by two stereo cameras with different imaging directions. The horizontal angle of view θP1 of the first camera 100L imaging the area to the left is 120 degrees, and the horizontal angle of view θP2 of the second camera 100R imaging the area to the right is also 120 degrees. The angle formed by the visual axis direction XC1 of the first camera 100L and the visual axis direction XC2 of the second camera 100R is 60 degrees, and the first camera 100L and the second camera 100R are disposed close to each other.

[0041] FIG. 8 is a diagram conceptually showing the field of view A1 of the first image captured by the first camera 100L and the field of view A2 of the second image captured by the second camera 100R. In reality, the field of view A1 of the first image and the field of view A2 of the second image are for the right eye and the left eye, respectively, but hereinafter, they are referred to as either the right eye or the left eye. Here, the "field of view" is an area that can be displayed on the display unit 11 of the HMD 10 in a real-life image captured by one camera, and is an area in which the real-life image is placed in the VR space. In the example of FIG. 8, the field of view angle θA1 of the field of view A1 of the first image and the field of view angle θA2 of the field of view A2 of the second image are both 120 degrees. Then, as illustrated in FIG. 9, the adjacent field of view areas A1 and A2 form a whole field of view of 180 degrees.

[0042] FIG. 9 is a conceptual diagram showing an example in which the first image on the left side and the second image on the right side are arranged in a VR space V so that an overlapping area AO occurs where a part of the field of view A1 and the field of view A2 of the second image overlap each other. In other words, FIG. 9 is a conceptual diagram showing an XY cross section of the VR space V in which the first image and the second image are arranged (for example, the same is true for the figures showing the VR space V shown later, such as FIG. 10). In the example of FIG. 9, the entire field of view of the VR space V is 180 degrees, the field of view θA1 of the field of view A1 of the first image is 120 degrees, and the field of view θA2 of the field of view A2 of the second image is 120 degrees, so that the field of view θAO of the overlapping area AO where the two overlap is 60 degrees. The overlapping area AO is an area sandwiched between the area boundaries BD1·BD2 of the first image and the second image. In the example of FIG. 9, the direction from the virtual viewpoint P in the VR space V through the center of the overlapping area AO is the direction of the X axis. In the overlapping area AO, either the first image or the second image is selectively displayed. In addition, the viewing area of ​​only the first image (area between the Y axis and the area boundary BD2) and the viewing area of ​​only the second image (area between the Y axis and the area boundary BD1) both have a viewing angle of 60 degrees.

[0043] FIG. 10 is a diagram showing an example in which a first image (field of view area A1 of the first image) is displayed in the overlapping area AO in the VR space V. FIG. 11 is a diagram showing an example in which a second image (field of view area A2 of the second image) is displayed in the overlapping area AO in the VR space V. When the first image in the field of view area A1 on the left side is selectively displayed in the overlapping area AO, as illustrated in FIG. 10, the boundary between the first image and the second image becomes the field boundary BD1 at the right end of the overlapping area AO. On the other hand, when the second image in the field of view area A2 on the right side is selectively displayed in the overlapping area AO, as illustrated in FIG. 11, the boundary between the first image and the second image becomes the field boundary BD2 at the left end of the overlapping area AO. In this way, the field boundaries (BD1, BD2) between adjacent real-life images change by selectively switching the real-life images (first image, second image) displayed in the overlapping area AO.

[0044] Whether the first image or the second image is to be displayed in the overlapping area AO is determined based on the line of sight direction S. The line of sight direction S can be specified based on detection information by the sensor 12 of the HMD 10 or the like (i.e., based on the orientation of the HMD 10). Here, an example will be described in which the visual axis direction of the HMD 10 is regarded as the line of sight direction S, and the real-life image to be displayed in the overlapping area AO is switched based on the yaw orientation of the HMD 10. In Figures 10 and 11, an image of the orientation of HMD10 is displayed near the virtual viewpoint P to show the relationship between the orientation of HMD10 (the visual axis direction of HMD10) and the display switching of the overlap area AO in VR space V (the same applies to the figures representing VR space V shown below).

[0045] 10 and 11 show an example in which a "reference direction RD" serving as a criterion for switching between real-life images displayed in the overlap area AO is set in a direction passing through the center of the overlap area AO from the virtual viewpoint P. In the example of FIG. 10, the line of sight direction S (the visual axis direction of the HMD 10) is to the left of the reference direction RD (i.e., the first image side). In this case, the first image of the left field of view area A1 is displayed in the overlap area AO. Also in this case, the boundary between the field of view area A1 of the first image and the field of view area A2 of the second image is the area boundary BD1, which is away from the line of sight direction S. 11, the line of sight direction S (the visual axis direction of the HMD 10) is to the right of the reference direction RD (i.e., toward the second image). In this case, the second image of the field of view A2 on the right side is displayed in the overlapping area AO. Also in this case, the boundary between the field of view A1 of the first image and the field of view A2 of the second image is the area boundary BD2, which is away from the line of sight direction S.

[0046] As in the above example, a specific direction (in the above case, the direction from the virtual viewpoint P through the center of the overlapping area AO) is set as the reference direction RD, and the real-life image (first image or second image) displayed in the overlapping area AO is dynamically switched by regarding the direction of the HMD 10 (the visual axis direction of the HMD 10) as the line of sight direction S. This makes it possible to move the image seam (misalignment) that is likely to occur at the area boundary (BD1 or BD2) of adjacent real-life images away from the direction of the user's line of sight, thereby reducing the sense of incongruity of the seam. Therefore, even if a close-up object is present at or near the area boundary, a VR image that is less uncomfortable for the user is displayed. In addition, since the stitching process is not required, the effort required for creating VR images can be reduced.

[0047] (Regarding setting of reference direction RD) In the above example, the reference direction RD is set to a direction passing through the center of the overlapping area AO from the virtual viewpoint P, but is not limited to this. The reference direction RD may be set to any direction from the virtual viewpoint P as long as it is within the viewing area in which the first image and the second image are arranged in the VR space V.

[0048] For example, as shown in FIG. 12, it is also possible to set the reference direction RD in the direction from the virtual viewpoint P to a position outside the overlapping area AO. FIG. 12 shows an example in which the reference direction RD is set in the field of view A2 of only the second image not including the overlapping area AO. In this case, too, if the line of sight S crosses the reference direction RD from the state of FIG. 12 (A) due to a change in the orientation of the HMD 10 in the yaw direction (left and right direction), the real-life image displayed in the overlapping area AO switches from the first image in the field of view A1 to the second image in the field of view A2 as illustrated in FIG. 12 (B). As a result, the area boundary of the adjacent real-life images switches from the area boundary BD1 to the area boundary BD2, and the area boundary moves away from the line of sight S. However, in this case, the line of sight S crosses the area boundary BD1 before crossing the reference direction RD in the state of FIG. 12 (A). Therefore, a timing occurs in which the area boundary BD1 overlaps with the line of sight S before the real-life image displayed in the overlapping area AO switches from the first image to the second image.

[0049] Therefore, it is a preferable embodiment to set the reference direction RD in the direction from the virtual viewpoint P to a predetermined position in the overlap area AO (which may or may not be the center of the overlap area AO). In this case, by setting the reference direction RD in the overlap area AO (see FIG. 10 or FIG. 11), even if the line of sight direction S heads toward the current area boundary (BD1 or BD2) of the overlap area AO, the line of sight direction S exceeds the reference direction RD before reaching the area boundary, and the real-life image displayed in the overlap area AO is switched. As a result, the area boundary moves away from the line of sight direction S before it reaches the area boundary.

[0050] (One mode in which the reference direction is not set) Next, a mode will be shown in which, without setting a reference direction RD, the real-life image to be displayed in the overlapping area AO is dynamically switched between adjacent real-life images (first image and second image) based on the line-of-sight direction S. In this mode, as illustrated in Fig. 13, the real-life image to be displayed in the overlapping area AO is dynamically switched based on the line-of-sight direction S with the current area boundary (BD1 or BD2) of the adjacent real-life images as a reference.

[0051] For example, as shown in (A) of FIG. 13, a first image of the left field of view A1 is displayed in the overlapping area AO, and the line of sight S is on the X-axis. In this case, the switching reference for the live-action image is the current area boundary BD1. If the HMD 10 rotates rightward from here, and the line of sight S (the visual axis of the HMD 10) crosses the area boundary BD1 as shown in (B) of FIG. 13, the live-action image displayed in the overlapping area AO is switched from the first image to the second image of the right field of view A2. This changes the switching reference for the live-action image to the current area boundary BD2. If the HMD 10 rotates leftward from here, and the line of sight S returns to the X-axis as shown in (C) of FIG. 13, the live-action image displayed in the overlapping area AO is not switched because it does not cross the current area boundary BD2. From here, when the HMD 10 rotates further left and the line of sight S crosses the area boundary BD2 as illustrated in (D) of FIG. 13, the real-life image displayed in the overlap area AO switches from the second image to the first image.

[0052] In this way, the current area boundary (BD1 or BD2) of adjacent real-life images (first image and second image) is used as a reference, and if the line of sight direction S is on the first image side (left side) of the current area boundary (BD1 or BD2), the first image is displayed in the overlap area AO, and if it is on the second image side (right side), the second image is displayed.

[0053] As a modified example, the real-life image displayed in the overlapping area AO may be dynamically switched in the following manner. That is, when the line of sight S approaches the current area boundary (BD1 or BD2) of the adjacent real-life image by a predetermined amount or more (for example, when the angle between the line of sight S and the current area boundary becomes a predetermined value or less), the real-life image currently displayed in the overlapping area AO may be switched to the other real-life image. In this manner, the real-life image displayed in the overlapping area AO is switched before the line of sight S reaches the area boundary, and the area boundary moves away from the line of sight S.

[0054] In the explanation using Figures 9 to 13, display control has been described in which the horizontal direction is divided into multiple regions to expand the horizontal viewing angle. In the case of display control in which the vertical direction is divided into multiple regions to expand the vertical viewing angle, however, image switching processing of the overlapping area similar to that described above can be performed based on the orientation of HMD10 in the pitch direction (up and down direction).

[0055] [5. Functional configuration of the display control device] 14 is a schematic functional block diagram showing an example of a functional configuration of the display control device 20. The display control device 20 executes control for displaying a VR image representing a field of view from a virtual viewpoint in a VR space on the display unit 11 of the HMD 10 as a stereoscopic image utilizing binocular parallax. As shown in FIG. 14, the display control device 20 includes a control unit 30. The control unit 30 is realized, for example, by the processor 21 executing a program stored in the storage device 22. The control unit 30 includes a line of sight direction identification unit 31 and an image generation unit 32.

[0056] The gaze direction specification unit 31 has a function of specifying the gaze direction that is the direction of the user's gaze with respect to the VR space. Here, the "gaze direction" is a direction that is specified by regarding it as the gaze direction of the user wearing the HMD 10 with respect to the VR space. The "gaze direction" can be estimated to be approximately the direction of the user's gaze based on, for example, detection information from the sensor 12 of the HMD 10, and may be consistent with the gaze direction of the actual user wearing the HMD 10, or may deviate from the gaze direction of the actual user.

[0057] For example, the gaze direction identification unit 31 can acquire detection information regarding the orientation of the HMD 10 and identify the gaze direction based on the detection information. Here, the "detection information regarding the orientation of the HMD 10" refers to detection information regarding the orientation of the HMD 10 that changes when the user changes the orientation of the head while wearing the HMD 10. The "detection information" can be acquired from a detection means such as a sensor that detects the direction or change in direction of the HMD 10. For example, the detection result of the sensor 12 (angular velocity sensor, acceleration sensor, geomagnetic sensor, etc.) built into the HMD 10 corresponds to an example of the "detection information regarding the orientation of the HMD 10". Also, the measurement result of a tracking system that captures the HMD 10 with an imaging unit provided outside the HMD 10 and measures the position and direction of the HMD 10 corresponds to an example of the "detection information regarding the orientation of the HMD 10". Also, the measurement result of a tracking system that analyzes surrounding objects captured by an imaging unit mounted on the HMD 10 itself and measures the position and direction of the HMD 10 corresponds to an example of the "detection information regarding the orientation of the HMD 10".

[0058] Furthermore, the gaze direction identification unit 31 can acquire detection information related to the user's gaze and identify the gaze direction based on the detection information. Here, "detection information related to the user's gaze" refers to information obtained by detecting the direction, movement, etc. of the user's gaze when the HMD 10 is worn by the user. For example, the measurement results of an eye tracking system that tracks the direction and movement of the gaze based on the position of the user's eyeballs corresponds to an example of "detection information related to the user's gaze."

[0059] The image generation unit 32 has the function of arranging multiple real-life images captured by multiple cameras each having a different imaging direction for capturing real space in a VR space so that overlapping areas are created where parts of the viewing areas of adjacent real-life images overlap with each other, and generating a VR image according to the line of sight direction.

[0060] Here, the "camera" refers to a real camera that captures real space. The "camera" itself is not included in the components of the display control device 20. The "camera" includes a left-eye optical system (left-eye camera) that captures an image for the left eye and a right-eye optical system (right-eye camera) that captures an image for the right eye in order to obtain a stereoscopic image using binocular parallax. For example, a single camera may be provided with an optical system for the left eye and an optical system for the right eye, as in a stereo camera. The left-eye camera and the right-eye camera may be configured separately. In order to obtain a wide-angle VR image that can be viewed stereoscopically from a plurality of real-life images, a plurality of cameras are used for each of the left eye and the right eye. For example, when obtaining a wide-angle image with a wide horizontal viewing angle, a plurality of cameras may be arranged in the horizontal direction so that the imaging directions are different in the real space. Also, for example, when obtaining a wide-angle image with a wide vertical viewing angle, a plurality of cameras may be arranged in the vertical direction so that the imaging directions are different in the real space. The number of cameras for the left eye and the right eye can be any number as long as it is two or more. The angle of view of each camera is also arbitrary. The angles of view of the multiple cameras may or may not all be the same.

[0061] The "imaging direction" refers to a direction corresponding to the orientation in which a camera captures an image of real space, and refers to the front direction (the direction pointing toward the subject) on the optical axis of the imaging optical system of the camera.

[0062] The "adjacent real-life images" refer to two real-life images captured by two adjacent cameras that have different directions for capturing real space and partially overlapping capture ranges. Here, the two adjacent cameras do not refer to a left-eye camera and a right-eye camera, but rather to two cameras that both capture real-life images for the left eye, or two cameras that both capture real-life images for the right eye. The "real-life image" may be a moving image or a still image. The "real-life image" may be an image (video) captured in advance, or a live image (live video) captured in approximately real time.

[0063] Moreover, the "viewing area" is an area that can be displayed on the display unit of the head mounted display in a real-life image captured by one camera, and is an area in which the real-life image is arranged in the VR space. For example, the viewing area of ​​a real-life image captured by a camera with a horizontal angle of view θH1 (e.g., 120 degrees) and a vertical angle of view θV1 (e.g., 120 degrees) is basically an area having a horizontal viewing angle θH2 (=θH1) and a vertical viewing angle θV2 (=θV1). Note that the viewing area of ​​a real-life image captured by a camera with a horizontal angle of view θH1 and a vertical angle of view θV1 may be narrowed when arranged in the VR space to a horizontal viewing angle θH2 (<θH1) and a vertical viewing angle θV2 (<θV1). For example, by not using an area near the edge of the image where distortion due to the distortion aberration of the optical system is likely to occur, the viewing area of ​​the real-life image may be narrowed when arranged in the VR space, as described above.

[0064] The "overlap area" refers to an area in the VR space where two adjacent real-life images are arranged to partially overlap each other. In the examples of Figs. 9 to 13, the viewing angle of the overlap area AO is 60 degrees, but the overlap area AO may be narrower or wider than 60 degrees. The width (viewing angle) of the "overlap area" can be set arbitrarily. As described later, when there are multiple overlap areas in the VR space, the width (viewing angle) of all the overlap areas may be the same, or the width of at least one overlap area may be different from the width of the other overlap areas.

[0065] The image generation unit 32 includes an image arrangement unit 321 that arranges a plurality of real-life images in the VR space so that an overlapping area is generated where adjacent real-life images partially overlap each other. The image arrangement unit 321 executes storage control to store the plurality of real-life images in a predetermined storage area (a storage area for storing the plurality of real-life images arranged in the VR space) of the storage device 22 (such as a VRAM) for forming the VR space. In the examples of Figs. 9 to 13, the image arrangement unit 321 stores data of the first image and the second image in a predetermined storage area for forming the VR space in the storage device 22.

[0066] In this embodiment, together with data of a plurality of real-life images to be displayed, arrangement information for arranging the plurality of real-life images in a VR space is stored in the storage device 22 in association with the plurality of real-life images. Alternatively, the arrangement information is also recorded in a recording medium on which the plurality of images to be displayed are recorded. Then, when the processor 21 of the display control device 20 reads out data of the plurality of images to be displayed from the recording medium and stores it in the storage device 22, the arrangement information is also read out and saved in the storage device 22 to be used for image arrangement processing in the VR space. Alternatively, when live video is displayed on the HMD 10 as a VR image in approximately real time as described later, the arrangement information is also input (received) together with the images of the live video to be displayed, and used for image arrangement processing in the VR space.

[0067] The placement information includes, for example, information such as the number of viewing areas (number of images) placed in the VR space, the viewing angle of the overall viewing area, the viewing angle of each viewing area, the direction of each viewing area in the VR space (position within the VR space), the number of overlapping areas, the viewing angle of each overlapping area, or the direction (position) of each overlapping area in the VR space. For example, the location information may be recorded in a header portion of a live-action image file, or the location information may be recorded as information associated with the live-action image in a file separate from the live-action image file (data).

[0068] In addition, if it is assumed that the display of the VR image (video) of this embodiment is standardized in advance and that live-action images conforming to the standard are used, it is not necessary to associate the detailed arrangement information described above with the multiple live-action images to be displayed and store them together with the live-action images. In other words, if the number of viewing areas (number of images) arranged in the VR space, the viewing angle of the entire viewing area, the viewing angle of each viewing area, the direction of each viewing area in the VR space (position in the VR space), the number of overlapping areas, the viewing angle of each overlapping area, or the direction (position) of each overlapping area in the VR space, etc. are determined in advance as a standard, the image arrangement unit 321 can arrange each image in the VR space based on the information of the standard.

[0069] Also, multiple standards may be defined, such as a first standard for the viewing angle of the entire viewing area of ​​180 degrees, a second standard for 220 degrees, a third standard for 270 degrees, and a fourth standard for 360 degrees. In this case, standard information indicating which standard the image was captured in may be associated with the image to be displayed. In this case, each piece of standard information corresponds to an example of the above-mentioned "arrangement information."

[0070] Also, the standard may have a certain degree of freedom. For example, only a part of the above-mentioned arrangement information may be standardized, and the other items may be set or changed arbitrarily. For example, the viewing angle of the entire viewing area is standardized and determined in advance, but the number of viewing areas (the number of images) arranged in the VR space and the viewing angle of each viewing area may be set arbitrarily. In this case, the contents of the standard information and the arbitrary setting items may be associated with the image to be displayed as "arrangement information."

[0071] Moreover, the image generating unit 32 includes a switching unit 322. This switching unit 322 has a function of dynamically switching between the real-life images to be displayed in the overlapping area, among the adjacent real-life images, based on the line of sight direction identified by the line of sight direction identifying unit 31. The switching unit 322 dynamically switches between the real-life images to be displayed in the overlapping area, among the adjacent real-life images, based on the line of sight direction, so that the area boundary between the adjacent real-life images moves away from the line of sight direction.

[0072] Here, "dynamically switching between adjacent live-action images to be displayed in the overlapping area based on the line of sight" means determining each time which of two adjacent live-action images (a first image and a second image) that can be displayed in the overlapping area is to be displayed in the overlapping area based on the line of sight, and dynamically switching between the live-action images to be displayed in the overlapping area in response to changes in the line of sight. For example, as shown in Figures 10 to 12, setting a reference direction RD that serves as a basis for switching between adjacent first and second images, and displaying the first image in the overlap area AO when the line of sight direction S is on the first image side of the reference direction RD, and displaying the second image in the overlap area AO when the line of sight direction S is on the second image side, is an example of "dynamically switching between adjacent real-life images to be displayed in the overlap area based on the line of sight direction."

[0073] Also, without setting a reference direction, the live-action images to be displayed in the overlapping area may be dynamically switched based on the line of sight direction. For example, as illustrated in FIG. 13, using the current area boundary (BD1 or BD2) of adjacent live-action images (first image and second image) as a reference, if the line of sight direction S is closer to the first image side than the current area boundary, the first image is displayed in the overlapping area AO, and if the line of sight direction S is closer to the second image side than the current area boundary, the second image is displayed in the overlapping area AO. This corresponds to an example of "dynamically switching the live-action images to be displayed in the overlapping area from among the adjacent live-action images based on the line of sight direction." Also, for example, when the line of sight direction approaches the current area boundary between the adjacent live-action images by a predetermined amount or more (when the angle between the line of sight direction and the area boundary becomes a predetermined value or less), switching the live-action image currently displayed in the overlapping area to the other live-action image corresponds to an example of "dynamically switching the live-action images to be displayed in the overlapping area from among the adjacent live-action images based on the line of sight direction."

[0074] As for the process of "switching live-action images to be displayed in the overlapping area", for example, two adjacent live-action images are arranged on different layers (image layers) above and below in the overlapping area. Then, the transparency of the live-action image arranged on the lower layer (background) is left unchanged at 0%, and the transparency of the live-action image arranged on the upper layer (foreground) is switched to either 100% or 0%. That is, when the live-action image arranged on the lower layer in the overlapping area is to be displayed, the transparency of the upper layer is set to 100%, and when the live-action image arranged on the upper layer is to be displayed, the transparency of the upper layer is set to 0%. In this way, arranging two adjacent live-action images on different layers above and below in the overlapping area and switching the transparency of the live-action image arranged on the upper layer to either 100% or 0% corresponds to an example of "switching live-action images to be displayed in the overlapping area". Alternatively, two adjacent live-action images may be arranged in an overlapping area, with the transparency of one live-action image to be displayed being 0% and the transparency of the other live-action image being 100%. In other words, setting the transparency of one of the two live-action images arranged in an overlapping area to 0% and the transparency of the other live-action image to 100%, and switching the transparency depending on which live-action image is to be displayed, is an example of "switching the live-action images to be displayed in the overlapping area." In addition, rather than superimposing two adjacent live-action images in the overlapping area, placing only the portion of the live-action image to be displayed that corresponds to the overlapping area in the overlapping area and switching the live-action image to be placed is an example of "switching the live-action image to be displayed in the overlapping area."

[0075] The switching period for switching the real-life images in the overlapping area can be set arbitrarily. The switching period may be set to approximately zero to switch suddenly, or a predetermined switching period (e.g., 0.3 seconds) may be set. When switching the real-life images in the overlapping area, there may be a period during which both of the two adjacent real-life images are displayed, for example, by semi-transparent synthesis.

[0076] Moreover, the image generating unit 32 (the switching unit 322 of the image generating unit 32) has a function of setting a reference direction in a predetermined direction from the virtual viewpoint, and specifying a live-action image to be displayed in the overlapping area based on the line-of-sight direction relative to the reference direction. Here, the "reference direction" is a direction that is set to determine whether or not to switch the live-action image to be displayed in the overlapping area in relation to the line-of-sight direction. The "reference direction" can be set in a predetermined direction from the virtual viewpoint in the VR space, and may be set in any direction within the viewing area in which adjacent live-action images are arranged in the VR space. For example, as illustrated in FIG. 10 and FIG. 11, a direction RD passing through the center of the overlapping area AO from the virtual viewpoint P in the VR space corresponds to an example of the "reference direction". The "reference direction" may be fixed, or may be changed (the predetermined direction changes) instead of fixed as described later.

[0077] For example, as shown in Fig. 10, when the line of sight S is to the left of the reference direction RD (toward the field of view A1 of the first image), the image generating unit 32 determines that the real-life image to be displayed in the overlapping area AO is the first image. Also, as shown in Fig. 11, when the line of sight S is to the right of the reference direction RD (toward the field of view A2 of the second image), the image generating unit 32 determines that the real-life image to be displayed in the overlapping area AO is the second image.

[0078] It is preferable that the image generating unit 32 (the switching unit 322 of the image generating unit 32) sets the reference direction in the direction from the virtual viewpoint to a predetermined position in the overlapping area. Here, "setting the reference direction in the direction from the virtual viewpoint to a predetermined position in the overlapping area" means setting the reference direction within the overlapping area (setting the reference direction so that a vector indicating the reference direction exists within the overlapping area). In this case, as illustrated in Fig. 10 and Fig. 11, since the reference direction RD is set within the overlapping area AO, even if the line of sight direction S heads toward the area boundary (BD1 or BD2) of the overlapping area AO, the real-life image displayed in the overlapping area AO is switched before reaching the area boundary, and therefore the area boundary also moves away from the line of sight direction S.

[0079] The image generation unit 32 also has a function of acquiring detection information regarding the orientation of the HMD 10, and changing the range of the VR image to be displayed on the display unit 11 of the HMD 10 based on the detection information. For example, the image generation unit 32 changes the direction of the field of view from the virtual viewpoint P in the VR space based on the detection result of a sensor 12 such as an angular velocity sensor built into the HMD 10, and changes the range of the VR image to be displayed on the display unit 11 of the HMD 10. For example, the direction of the visual axis of the HMD 10 is set as the direction of the field of view, and the range of the VR image to be displayed on the display unit 11 of the HMD 10 is changed according to the orientation of the HMD 10.

[0080] [6. Processing] Next, an example of the processing executed by the display control device 20 of this embodiment will be described below. Fig. 15 is a flowchart showing an example of the processing of the display control device 20. The processing described below is realized by the control unit 30 (processor 21 of the display control device 20) executing a program stored in the storage device 22 (the same applies to the processing referring to the flowcharts of Figs. 17, 23, 24, 27, 29, or 33). Here, an example of the display control process of the VR image described with reference to FIG. 9 to FIG. 12 will be described.

[0081] The control unit 30 sets a reference direction RD in the VR space V, as exemplified in FIG. 10 or FIG. 12 (S100). Data of real-life images (first and second images) captured by two cameras with different imaging directions for capturing real space is stored in, for example, the storage device 22 or a recording medium. The control unit 30 reads out the first image, the second image, and the arrangement information from the storage device 22 or the like. Then, based on the arrangement information, the control unit 30 arranges the first image and the second image in the VR space V so that an overlapping area AO is generated in which the viewing areas A1 and A2 of the first image and the second image partially overlap each other (S102). For example, the control unit 30 arranges the first image and the second image in different layers, one above the other, in the overlapping area AO. For example, the control unit 30 arranges the first image and the second image in the overlapping area AO. For example, the control unit 30 arranges the first image as the lower layer and the second image as the upper layer.

[0082] The control unit 30 also acquires detection information on the orientation of the HMD 10 from the sensor 12 (e.g., an angular velocity sensor) of the HMD 10 (S104) and identifies the line of sight direction S (S106). The control unit 30 then determines whether the line of sight direction S is to the left (first image side) of the reference direction RD (S108), and if YES, displays the first image in the overlapping area AO as illustrated in FIG. 10 (S110). For example, the first image is displayed in the overlapping area AO by fixing the transparency of the first image to 0% in the overlapping area AO and setting the transparency of the second image arranged in a layer above the first image to 100%. On the other hand, if the line of sight direction S is to the right (second image side) of the reference direction RD (NO in S108), the control unit 30 displays the second image in the overlapping area AO as illustrated in FIG. 11 (S112). For example, the transparency of the second image arranged in a layer above the first image in the overlapping area AO is set to 0%, thereby causing the second image to be displayed in the overlapping area AO. In this manner, the control unit 30 executes control to dynamically switch the real-life image to be displayed in the overlapping area AO from among the adjacent real-life images (the first image and the second image) based on the line-of-sight direction S.

[0083] Then, the control unit 30 generates a VR image of a range according to the orientation of the HMD 10 based on the detection information related to the orientation of the HMD 10 in S104, and outputs the VR image to the HMD 10 (S114). For example, the control unit 30 outputs the VR image generated by performing a rendering process or the like to the HMD 10. As a result, the VR image is displayed on the HMD 10. The processes of S102 to S114 are repeated until the display is terminated (YES in S116) due to a playback stop operation or the like performed by the user.

[0084] [7. How to change the reference direction] In the above description, an example in which a fixed reference direction RD is set has been shown. If the reference direction RD is fixed, when the visual axis direction (line of sight direction S) of the HMD 10 is near the reference direction RD, the real-life image displayed in the overlapping area AO may be frequently switched, which may result in an image that is difficult for the user to see. Therefore, when the real-life image displayed in the overlapping area AO is switched, the reference direction RD is changed so that it moves away from the visual axis direction (line of sight direction S) of the HMD 10. This is described below.

[0085] As shown in FIG. 16A, when the first image of the left field of view A1 is displayed in the overlapping area AO, the reference direction RD1 is set to a direction shifted to the right (e.g., -20 degrees) from the direction passing through the center of the overlapping area AO from the virtual viewpoint P. Note that the angle is positive in the counterclockwise direction (left direction). When the visual axis direction (line of sight direction S) of the HMD 10 rotates to the right from this state and the line of sight direction S exceeds the reference direction RD1, the real-life image displayed in the overlapping area AO is switched from the first image to the second image of the field of view A2, as shown in FIG. 16B. At the timing of this switch (simultaneous with the switch), the reference direction moves to the left so as to move away from the visual axis direction (line of sight direction S) of the HMD 10, and is changed from the reference direction RD1 in FIG. 16A to the reference direction RD2 in FIG. 16B. That is, as illustrated in (B) of Figure 16, when a second image of the right-side viewing area A2 is displayed in the overlap area AO, the reference direction RD is set to a direction shifted to the left (e.g., 20 degrees) from the direction passing through the center of the overlap area AO from the virtual viewpoint P.

[0086] Furthermore, when the visual axis direction (line of sight direction S) of the HMD 10 rotates leftward from the state of (B) in Fig. 16 and the line of sight direction S crosses the reference direction RD2, the real-life image displayed in the overlapping area AO is switched from the second image to the first image in the field of view A1, and at the same time is changed from the reference direction RD2 to the reference direction RD1, as shown in (A) in Fig. 16. Note that "simultaneously" includes "approximately simultaneously."

[0087] This is a mode in which hysteresis control is performed by changing the reference direction (RD1 or RD2) at the timing when the real-life image displayed in the overlap area AO is switched. In addition, when the actual image to be displayed in the overlap area AO is switched, the reference direction RD is moved toward the switched area boundary (BD1 or BD2) so that the reference direction moves away from the line of sight direction S. Also, this is an aspect in which the reference direction (RD1 or RD2) is changed depending on the actual image displayed in the overlapping area AO (whether it is the first image in the viewing area A1 or the second image in the viewing area A2).

[0088] The image generation unit 32 has a function of changing the reference direction (RD1 or RD2) depending on the real-life image to be displayed in the overlap area AO (whether it is the first image in the field of view A1 or the second image in the field of view A2) at the timing when the real-life image to be displayed in the overlap area AO is switched.

[0089] Next, an example of the above-mentioned process of changing the reference direction will be described with reference to Fig. 17. Fig. 17 is a flowchart showing an example of the process of changing the reference direction executed by the control unit 30 according to this embodiment.

[0090] The control unit 30 sets a reference direction (RD1 or RD2) according to the real-life image (the first image in the field of view A1 or the second image in the field of view A2) displayed in the overlapping area AO (S200), as exemplified in (A) or (B) in FIG. 16. The control unit 30 also acquires detection information on the orientation of the HMD 10 from the sensor 12 (e.g., an angular velocity sensor) of the HMD 10 (S202) and identifies the line-of-sight direction S (S204). The control unit 30 also determines whether it is necessary to switch the real-life image displayed in the overlapping area AO based on the line-of-sight direction S relative to the current reference direction (RD1 or RD2) (S206). If the answer is YES in S206, the control unit 30 switches the real-life image displayed in the overlapping area AO (S208) and changes the reference direction to a direction according to the real-life image after the switching (S210). On the other hand, if the answer is NO in S206, the process returns to S202. The processes of S202 to S210 are repeated until the display is terminated (YES in S212) due to a playback stop operation or the like performed by the user.

[0091] According to the above aspect, the reference direction (RD1 or RD2) is not fixed, but is changed according to the real-life image displayed in the overlapping area AO at the timing when the real-life image displayed in the overlapping area AO is switched. As a result, after the real-life image is switched, the reference direction temporarily moves away from the line of sight direction S, so that frequent switching of the real-life image displayed in the overlapping area AO can be reduced.

[0092] [8. How to visibly display the current area boundary] In the image display system 1 of the present embodiment, the real-life image (first image or second image) displayed in the overlapping area AO is switched to move the area boundary (BD1 or BD2) of the adjacent real-life images away from the line of sight S, but the current area boundary may be displayed so as to be visible. This will be described below.

[0093] 18 and 19 are diagrams showing an example of a screen displayed on the display unit 11 of the HMD 10. A VR image of a range of a field of view according to the orientation of the HMD 10 in the VR space V shown in FIG. 10 is displayed on the screen G10 shown in FIG. 18. That is, a VR image showing a field of view in the visual axis direction (line of sight direction S) of the HMD 10 from the virtual viewpoint P in the VR space V shown in FIG. 10 is displayed on the screen G10. Therefore, the first image of the left field of view area A1 is displayed in the overlapping area AO. On this screen G10, a boundary line BL1 is visibly displayed at a portion corresponding to the current area boundary BD1 (right end of the overlapping area AO) between the adjacent real-life images (first image and second image). For example, the boundary line BL1 is a black solid line having a line width recognizable by the user.

[0094] A screen G11 shown in FIG. 19 is displayed when a user viewing a screen G10 in FIG. 18 rotates his / her head (HMD10) to the right. This screen G11 displays a VR image in a range of view according to the orientation of the HMD10 in the VR space V shown in FIG. 11. That is, since the visual axis direction (line of sight direction S) of the HMD10 rotates to the right and exceeds the reference direction RD, a VR image in which the real-life image displayed in the overlapping area AO has been switched from the VR image on the screen G10 in FIG. 18 is displayed on the screen G11 shown in FIG. 19. On this screen G11, a boundary line BL2 is visibly displayed at a portion corresponding to the current area boundary BD2 (left end of the overlapping area AO) between adjacent real-life images (first image and second image). For example, the boundary line BL2 is a black solid line having the same line width as the boundary line BL1.

[0095] As illustrated in FIG. 20, the display control device 20 may include a boundary display unit 33. The boundary display unit 33 has a function of displaying a boundary line visible to the user at the boundary between adjacent live-action images. Here, the "boundary line between adjacent live-action images" refers to the boundary between two adjacent live-action images (a first image and a second image) arranged in the VR space. As illustrated in FIG. 10 and FIG. 11, the position of the boundary line (BD1 or BD2) changes depending on the live-action image (the first image in the field of view A1 or the second image in the field of view A2) displayed in the overlapping area AO. The "boundary line" refers to a line displayed at the boundary between adjacent live-action images so that the user can visually recognize the boundary between the live-action images. The color, line width, transparency, etc. of the "boundary line" can be set arbitrarily. The type of the line can also be set arbitrarily, such as a solid line, a dotted line, a dashed line, a wavy line, a double line, etc. The "boundary line" may be a two-dimensional line or a three-dimensional (three-dimensional) raised line.

[0096] The boundary display unit 33 places a boundary line (BD1 or BD2) on the current area boundary (BD1 or BD2) and displays the boundary line on the display unit 11 in a manner superimposed on the actual VR image.

[0097] The area boundary (BD1 or BD2) where the image seam (displacement) occurs is changed by switching the actual image displayed in the overlap area AO based on the line of sight direction S, as exemplified in Figures 10 and 11. That is, the user can change the area boundary (BD1 or BD2) where the image seam (displacement) occurs by changing the line of sight direction S at his / her own will. In other words, if there is an image seam in the direction the user wants to view, the user can change the line of sight direction S at his / her own will to eliminate the image seam in the direction the user wants to view, making the image easier to view. According to this embodiment, a boundary line (BL1 or BL2) is displayed visibly at the area boundary (BD1 or BD2) between adjacent real-life images, making it easier for the user to recognize the area boundary where the image seam (shift) occurs. The boundary line (BL1 or BL2) at the image seam can be changed by the user by changing the line of sight S at his / her own will, as illustrated in Figs. 18 and 19, and even after the change, the user can easily recognize where the image seam has moved to. In other words, by displaying the boundary line (BL1 or BL2) visibly at the area boundary (BD1 or BD2), it is possible to easily determine which of the adjacent real-life images should be displayed in the overlapping area AO and view it, so that the boundary line is not displayed in the direction the user wants to view.

[0098] When the gaze direction S is determined based on detection information regarding the orientation of the HMD10 (e.g., detection information from an angular velocity sensor), the user can adjust the orientation of the head wearing the HMD10 to adjust the range of the VR image displayed on the display unit 11 of the HMD10 while changing the boundary line (BL1 or BL2).

[0099] Furthermore, when the line of sight direction S is specified based on detection information (e.g., detection information by eye tracking) regarding the user's line of sight, the user can change the boundary line (BL1 or BL2) displayed on the area boundary (BD1 or BD2) simply by changing the line of sight without moving the head. For example, in a configuration in which the range of the VR image displayed on the display unit 11 is changed based on detection information (e.g., detection information by an angular velocity sensor) regarding the orientation of the HMD 10 and the line of sight direction S is specified based on detection information (e.g., detection information by eye tracking) regarding the user's line of sight, the user can adjust the image to be easily viewed by himself / herself as follows. That is, the user adjusts the direction of the head to adjust the range of the VR image so that the image in the direction he / she wants to view is displayed on the display unit 11 of the HMD 10. Then, by adjusting the line of sight direction and changing the boundary line (BL1 or BL2) while the image in the direction he / she wants to view is displayed on the display unit 11, even if there is a boundary line in the direction he / she wants to view, the boundary line can be moved to display an image that is easy to view. In addition, in a configuration in which the range of the VR image displayed on the display unit 11 is changed based on detection information regarding the user's line of sight using eye tracking or the like, and the live-action image displayed in the overlap area AO is switched, the user can adjust the image to be easy to see by simply adjusting the direction of their line of sight.

[0100] (Displaying additional information on or near the border) Next, a mode that can make it easier for the user to determine which of the adjacent real-life images should be displayed and viewed in the overlap area AO will be described below.

[0101] For example, in the screen G10 illustrated in Fig. 18, a boundary line BL1 is displayed, but it is difficult for a user to immediately recognize whether the overlapping area AO is on the left side or the overlapping area AO on the right side of the boundary line BL1. In addition, it is difficult for a user to immediately recognize which of the adjacent real-life images (the first image on the left side or the second image on the right side) is currently displayed in the overlapping area AO. In addition, it is difficult for a user to immediately recognize in which direction the line of sight S should be changed in order to switch the real-life images in the overlapping area AO (i.e., change the boundary line). Even if the above is difficult to immediately recognize, if the user changes the line of sight direction S to the left or right, the screen G10 illustrated in Fig. 18 changes to screen G11 illustrated in Fig. 19, or the screen G11 changes to screen G10, so the user can recognize it. In this embodiment, additional information is displayed on or near the boundary line, making it easier for the user to recognize which direction the line of sight direction S should be changed to.

[0102] Fig. 21 and Fig. 22 are diagrams showing an example of a screen on which a gradation portion is displayed together with a boundary line. Fig. 21 is a diagram showing an example of a screen G20 corresponding to the screen G10 shown in Fig. 18, and showing a gradation portion GD1 as additional information to the right of the boundary line BL1.

[0103] The screen G20 in FIG. 21 corresponds to the VR space V in FIG. 10. Therefore, the left side of the boundary line BL1 corresponds to the "overlapped area AO where the first image of the left viewing area A1 is displayed," and the area on the right side of the boundary line BL1 is the side that is not the overlapped area AO (the area of ​​only the right viewing area A2). In other words, the gradation part GD1 of the screen G20 is displayed on the opposite side of the boundary line BL1 from the overlapped area AO. This gradation part GD1 is a semi-transparent area that displays a black gradation whose density or opacity decreases continuously or stepwise as it moves away from the boundary line BL1 (toward the right in FIG. 21). The width (distance in the direction perpendicular to the boundary line BL1) and color of the gradation part GD1 can be set arbitrarily (the same applies to the gradation part GD2 described below).

[0104] FIG. 22 is a diagram showing an example of a screen G21 corresponding to the screen G11 illustrated in FIG. 19, and displaying a gradation portion GD2 as additional information to the left of the boundary line BL2. The screen G21 in FIG. 22 corresponds to the VR space V in FIG. 11. Therefore, the right side of the boundary line BL2 corresponds to the "overlapped area AO in which the second image of the right-side viewing area A2 is displayed," and the area to the left of the boundary line BL2 is the side that is not the overlapped area AO (the area of ​​only the left-side viewing area A1). In other words, the gradation portion GD2 of the screen G21 is displayed on the opposite side of the overlapped area AO across the boundary line BL2. The gradation portion GD2 is a semi-transparent area that displays a gradation in which the density or opacity decreases continuously or stepwise as it moves away from the boundary line BL2 (toward the left in FIG. 22). The color of the gradation portion GD2 is, for example, a black gradation, but any color can be set.

[0105] In this way, the gradation portion (GD1 or GD2) is displayed on the opposite side of the overlapping area AO with the boundary line (BL1 or BL2) in between. This allows the user to recognize that the opposite side of the boundary line (BL1 or BL2) from the gradation portion (GD1 or GD2) (the left side of the boundary line BL1 in FIG. 21, and the right side of the boundary line BL2 in FIG. 22) of the boundary line (BL1 or BL2) is the overlapping area AO. The user can also recognize that the real-life image (the first image on the left side in FIG. 21, and the second image on the right side in FIG. 22) on the opposite side of the boundary line (BL1 or BL2) from the gradation portion (GD1 or GD2) is currently displayed in the overlapping area AO. The user can also intuitively recognize that moving the line of sight S in the direction in which the gradation portion (GD1 or GD2) becomes thinner (from the boundary line BL1 to the direction in which the gradation portion is present) can cause a change in the real-life image in the overlapping area AO (a change in the boundary line).

[0106] In other words, the gradation portion (GD1 or GD2) is "information indicating which side of the boundary line is the overlapping area," "information indicating which of the adjacent live-action images is the live-action image that is displayed in the overlapping area," or "information indicating in which direction the line of sight should be changed to cause a switch in the live-action images in the overlapping area."

[0107] In addition, the gradation portion (GD1 or GD2) is displayed superimposed on the image misalignment that occurs at the boundary (BL1 or BL2), which also has the effect of making the image misalignment less noticeable. Also, the change in density or opacity of the gradation portion (GD1 or GD2) naturally indicates the direction of change in the line of sight S required to switch the real-life images in the overlapping area AO. Therefore, applying the gradation portion (GD1 or GD2) as additional information to be displayed together with the boundary is one of the preferred aspects.

[0108] In addition, since it is sufficient to be able to recognize which side of the boundary line (BL1 or BL2) is the overlapping area AO, a gradation portion (GD1 or GD2) may be displayed on the overlapping area AO side of the boundary line (BL1 or BL2).

[0109] The boundary display unit 33 in this embodiment has a function of displaying additional information on the boundary line or in its vicinity to indicate which side of the boundary line is the overlapping area. Here, the "additional information on the boundary line to indicate which side of the boundary line is the overlapping area" is additional information on the boundary line displayed on the boundary line or in its vicinity, and is information for indicating which side of the boundary line is the overlapping area so that the user can recognize which side of the boundary line is currently the overlapping area. For example, as described above, the gradation part (GD1 or GD2) displayed on the opposite side (or the overlapping area AO side) of the boundary line (BL1 or BL2) is an example of "additional information on the boundary line to indicate which side of the boundary line is the overlapping area". Note that the additional information is not limited to the gradation part (GD1 or GD2). For example, an "arrow" or a "symbol" or the like may be displayed on top of or in the vicinity of the boundary line to indicate which side of the boundary line is the overlapping area. In this case, the "arrow" or the "symbol" or the like corresponds to an example of additional information.

[0110] In addition, additional information such as the "arrow" or "symbol" is also an example of "information indicating which of the adjacent live-action images is the live-action image displayed in the overlapping area" or "information indicating in which direction the line of sight should be changed to cause a switch in the live-action images in the overlapping area."

[0111] The boundary display unit 33 places a boundary line (BD1 or BD2) at the current area boundary (BD1 or BD2) and also places additional information such as the aforementioned gradation portion (GD1 or GD2) so that the boundary line and additional information are superimposed on the real-life VR image and displayed on the display unit 11.

[0112] Next, an example of the process for displaying the above-mentioned boundary line (BL1 or BL2) will be described with reference to Fig. 23. Fig. 23 is a flowchart showing an example of the process of the display control device 20 for displaying the boundary line. Steps S300 to S312 in the flowchart of Fig. 23 are the same as steps S100 to S112 in Fig. 15, and detailed description of each step will be omitted. The control unit 30 executes control to dynamically switch the real-life image to be displayed in the overlapping area AO from among the adjacent real-life images (the first image and the second image) based on the line of sight S (S304 to S312). Then, the control unit 30 superimposes a boundary line (BL1 or BL2) on the area boundary (BD1 or BD2) between the adjacent real-life images (S314). Furthermore, the control unit 30 superimposes a gradation portion (GD1 or GD2) on the area (the area of ​​the real-life image not displayed in the overlapping area AO) on the opposite side of the boundary line (BL1 or BL2) from the overlapping area AO (S316). Then, the control unit 30 generates a VR image of a range according to the orientation of the HMD 10 and outputs it to the HMD 10 (S318). As a result, the VR image including the boundary line (BL1 or BL2) and the gradation portion (GD1 or GD2) is displayed on the HMD 10. The processes of S302 to S318 are repeated until the display ends (YES in S320). In addition, in the flowchart of FIG. 23, if the gradation portion (GD1 or GD2) is not displayed in the VR image, the process of S316 can be omitted.

[0113] (Aspect of making the display form of two boundary lines different) The boundary display unit 33 may change the display form of the boundary line BL1 displayed on the area boundary BD1 at one end (right end) of the overlapping area AO from the boundary line BL2 displayed on the area boundary BD2 at the other end (left end). Here, "changing the display form of the boundary line" refers to changing the color, density, transparency, line width, line type (solid line, dotted line, etc.), or whether the boundary line is flat (two-dimensional) or solid (three-dimensional), etc. In this way, when multiple images arranged in the horizontal direction are used to expand the horizontal viewing angle, the user can recognize whether the currently displayed boundary line (BL1 or BD2) is the misaligned part of the image at the left end of the overlapping area AO or the misaligned part of the image at the right end. In addition, when multiple images arranged in the vertical direction are used to expand the vertical viewing angle, the user can recognize whether the currently displayed boundary line (BL1 or BD2) is the misaligned part of the image at the top end of the overlapping area AO or the misaligned part of the image at the bottom end. In this embodiment, the display of additional information such as the gradation parts GD1 and GD2 described above may be omitted, or the additional information may be displayed together with the boundary line.

[0114] (How to set the transition period) When switching between the two real-life images in the overlapping area AO, a switching period (e.g., 0.3 seconds) may be provided so that the switching is performed gradually over time, rather than suddenly switching from one to the other. For example, when switching from one real-life image displayed in the overlapping area AO to the other, the display of the overlapping area AO may be cross-faded during the switching period. This makes it easier to view the boundary line (BL1 or BL2) more naturally before and after switching the real-life images in the overlapping area AO. This is explained below.

[0115] The image generating section 32 of this embodiment has a function of gradually switching from one real-life image to the other real-life image over time when switching between real-life images to be displayed in the overlapping area AO. Here, "gradually switching from one live-action image to the other live-action image over time" means that a switching period (e.g., 0.3 seconds) is provided from the start to the end of the switching, and the live-action image (the first image or the second image) currently displayed in the overlap area AO is gradually switched to the other live-action image over the course of the switching period. The switching period can be set arbitrarily.

[0116] For example, performing image synthesis of two live-action images to be switched and displayed in the overlapping area during the switching period and gradually switching the live-action image displayed in the overlapping area from one to the other corresponds to an example of "gradually switching from one live-action image to the other live-action image as time passes." One example of the image synthesis is alpha blending (semi-transparent synthesis). As a specific example, the transparency of one live-action image is gradually increased from "0% to 100%" and the transparency of the other is gradually decreased from "100% to 0%" as time passes during the switching period, thereby alpha blending both live-action images. Also, for example, in an overlapping area, two adjacent live-action images are arranged on different layers, one above the other. The transparency of the live-action image arranged on the lower layer (background) may be left unchanged at 0%, while the transparency of the live-action image arranged on the upper layer (foreground) may be gradually changed from "0% to 100%" or "100% to 0%" according to the passage of time in the switching period. For example, when switching from a live-action image on the upper layer to a live-action image on the lower layer, a process is performed in which the transparency of the live-action image on the upper layer is gradually changed from "0% to 100%" according to the passage of time in the switching period. Also, when switching from a live-action image on the lower layer to a live-action image on the upper layer, a process is performed in which the transparency of the live-action image on the upper layer is gradually changed from "100% to 0%" according to the passage of time in the switching period.

[0117] Also, performing a process of gradually replacing pixels of the live-action image currently displayed in the overlapping area with pixels of the other live-action image during the switching period corresponds to an example of "gradually switching from one live-action image to the other live-action image as time passes." In other words, a process of gradually decreasing the number of pixels (area) of the live-action image currently displayed in the overlapping area and gradually increasing the number of pixels (area) of the other live-action image as time passes corresponds to an example of "gradually switching from one live-action image to the other live-action image as time passes." As a specific example, gradually replacing pixels from the current area boundary side to the area boundary side after switching is completed during the switching period, so that the area boundary slides (gradually moves), corresponds to an example of "gradually switching from one live-action image to the other live-action image as time passes." The "replace pixels" includes "switching the transparency of pixels in the upper layer from 0% to 100%" or "switching from 100% to 0%" in two adjacent live-action images arranged on different layers above and below the overlapping area. The "replace pixels" also includes changing the transparency of a pixel in the live-action image to be replaced from 0% to 100% and changing the transparency of the corresponding pixel in the live-action image to be replaced from 100% to 0%. In addition, the pixels may be gradually switched from one to the other from the center to the periphery, from the periphery to the center, from the left edge to the right edge, from the right edge to the left edge, from the top edge to the bottom edge, or from the bottom edge to the top edge of the overlapping area.

[0118] Next, an example of a process in which the above-mentioned switching period is provided will be described with reference to Fig. 24. Fig. 24 is a flow chart showing an example of a process in which the real-life images to be displayed in the overlapping area in the display control device 20 are gradually switched. The control unit 30 acquires detection information on the orientation of the HMD 10 from the sensor 12 (e.g., an angular velocity sensor) of the HMD 10 (S400), and identifies the line of sight direction S (S402). The control unit 30 also determines whether it is necessary to switch the real-life image displayed in the overlapping area AO based on the line of sight direction S relative to the current reference direction (RD1 or RD2) (S404). If the answer is YES in S404, the control unit 30 initializes the variable α to the initial value "255" (S406). This variable α is an "α value" used for semi-transparent blending (alpha blending) of adjacent real-life images (first image and second image) arranged in the overlapping area AO in step S408, and can take a value of "0 to 255". The "α value" is also used to change the opacity of the boundary line (BL1 or BL2) in steps S410 and S412.

[0119] The control unit 30 subtracts a predetermined amount Δα from the α value for each frame, and changes the α value so that it gradually decreases from "255" to "0" as the switching period elapses (S414). For example, the switching period is a 20-frame period. One frame is, for example, 1 / 60 seconds. For example, if the predetermined amount Δα to be subtracted for each frame is "13", the α value will gradually change from "255" to "0" over a switching period of approximately 0.3 seconds.

[0120] In S408, the control unit 30 semi-transparently composites two real-life images (a first image and a second image) in the overlapping area AO. In this case, the control unit 30 sets the alpha value of each pixel of the image displayed in the overlapping area AO before switching (this is referred to as the "pre-switching image") to the value of the variable α. Also, the image displayed in the overlapping area AO after switching is referred to as the "post-switching image." Therefore, during the switching period in which the semi-transparent composite is performed, the pixel value of each pixel in the overlapping area AO is set to "pixel value of the corresponding pixel in the pre-switching image*α+pixel value of the corresponding pixel in the post-switching image*(255-α)."

[0121] Furthermore, in S410, the control unit 30 sets the opacity of the boundary line before the switch to α. Furthermore, in S412, the control unit 30 sets the opacity of the boundary line after the switch to (255-α) and superimposes it on the area boundary after the switch. Here, the boundary line is completely opaque (0% transparency) when α=255, and the smaller the α value, the lower the opacity becomes, and it is completely transparent (100% transparency) when α=0.

[0122] The above steps S408 to S414 are repeated until the α value becomes 0 (YES in S416). As a result, for example, the α value gradually changes from "255" to "0" over a switching period of about 0.3 seconds, and in the overlapping area AO, the transparency of the pre-switched image gradually increases while the transparency of the post-switched image gradually decreases. Furthermore, during the switching period, the pre-switched boundary line and the post-switched boundary line are displayed simultaneously, and as time passes during the switching period, the transparency of the pre-switched boundary line gradually increases while the transparency of the post-switched boundary line gradually decreases. Finally, when the α value becomes 0 (YES in S416), the switching of the images in the overlapping area AO ends, and the post-switched image is displayed in the overlapping area AO, while the pre-switched boundary line disappears and the post-switched boundary line is displayed.

[0123] Here, an example is shown in which the pre-switching border and the post-switching border are simultaneously displayed while changing the transparency during the switching period, but the present invention is not limited to this. The pre-switching border may be erased at the timing when the switching process (switching period) of the image in the overlapping area AO starts, and the post-switching border may be displayed at the timing when the switching process (switching period) ends. In this case, S410 and S412 in FIG. 24 can be omitted. Alternatively, during the switching process (switching period), the pre-switching border and the post-switching border may be displayed with the same transparency (both may be opaque), and at the timing when the switching process (switching period) ends, the pre-switching border may be erased and the post-switching border may be displayed.

[0124] (Aspect of moving the boundary line during the transition period) In the above-mentioned embodiment in which the switching period is provided, the boundary line may be gradually moved toward the position to be displayed after the switching is completed during the switching period. For example, when the first image of the left viewing area A1 is displayed in the overlapping area AO, a boundary line BL1 is displayed on the screen G20 as illustrated in FIG. 21. When the real-life image displayed in the overlapping area AO is switched from this state to the second image of the right viewing area A2, the boundary line BL1 at the position indicated by the dashed line is changed to the position of the boundary line BL2 as illustrated in FIG. 25. During this image switching, the boundary line is displayed and controlled so as to gradually move horizontally from the position of BL1 indicated by the dashed line to the position of BL2 during the switching period (for example, 0.3 seconds). This also allows the user to naturally recognize the position of the boundary line.

[0125] When switching between live-action images to be displayed in the overlapping area AO, the image generating unit 32 of this embodiment has the function of gradually switching from one live-action image to the other live-action image by gradually moving the boundary line over time toward the position that will be displayed after the switching is completed.

[0126] For example, a case will be described where the actual image in the overlapping area AO is switched from a first image on the left to a second image on the right. As illustrated in FIG. 25, during the switching period, the boundary line BLmove is gradually moved horizontally from the area boundary BD1 before the switching to the area boundary BD2 after the switching is completed. At this time, in the overlapping area AO, in the area to the right of the boundary line BLmove after the movement (the area between the area boundary BD1 before the switching and the boundary line BLmove after the movement), each pixel of the first image is replaced with each pixel of the corresponding second image. As a result, with the horizontal movement of the boundary line BLmove to the left, each pixel to the right of the boundary line BLmove in the overlapping area AO is switched from the first image to the second image. This process is repeated until the boundary line BLmove reaches the area boundary BD2 after the switching is completed. When the boundary line BLmove reaches the area boundary BD2, it stops as the boundary line BL2. The same process as above is executed when the actual image in the overlap area AO is switched from the second image on the right to the first image on the left.

[0127] As shown in the example of Fig. 25, when the gradation portion (GD1 or GD2) is displayed as additional information, the display of the gradation portion may be controlled as follows. For example, the gradation portion (GD1 or GD2) is erased when the switching process (switching period) of the real-life image in the overlapping area AO starts, and is displayed when the switching process (switching period) ends. Alternatively, the gradation portion may be moved in conjunction with the movement of the boundary line BLmove.

[0128] In addition, in a configuration in which the boundary line is moved and displayed according to the passage of time during the switching period, the switching process of the real-life image in the overlapping area AO is not limited to the above. For example, while the switching process of the real-life image in the overlapping area AO is performed by the above-mentioned semi-transparent composition (alpha blend), the boundary line may be gradually moved toward the position to be displayed after the switching is completed according to the passage of time during the switching process.

[0129] [9. An example of dividing the entire viewing area of ​​the VR space into three or more areas] Here, an example of dividing the entire viewing area of ​​the VR space V into three areas, left, front, and right, will be described with reference to Fig. 26. Here, an example is shown in which a wide viewing angle image with a horizontal viewing angle of 270 degrees can be reproduced using a left image, a front image, and a right image captured by three stereo cameras with a horizontal viewing angle of 120 degrees capturing images in the left, front, and right directions.

[0130] FIG. 26 is a conceptual diagram showing an example in which the left image viewing area AL, the front image viewing area AF, and the right image viewing area AR are arranged in a VR space V. In the VR space V, the arrangement of the images is controlled so that a first overlapping area AO1 is generated in which the left image viewing area AL and the front image viewing area AF partially overlap each other. The arrangement of the images is controlled so that a second overlapping area AO2 is generated in which the front image viewing area AF and the right image viewing area AR partially overlap each other. The viewing angle θall of the entire viewing area of ​​the VR space V is 270 degrees, the viewing angle θAL of the left viewing area AL is 120 degrees, the viewing angle θAF of the front viewing area AF is 120 degrees, the viewing angle θAR of the right viewing area AR is 120 degrees, the viewing angle θAO1 of the first overlapping area AO1 is 45 degrees, and the viewing angle θAO2 of the second overlapping area AO2 is 45 degrees. Moreover, a reference direction RD1 serving as a reference for switching between the real-life images (left image or front image) in the overlapping area AO1 is set in the direction from the virtual viewpoint P to the center of the overlapping area AO1. Moreover, a reference direction RD2 serving as a reference for switching between the real-life images (front image or right image) in the overlapping area AO2 is set in the direction from the virtual viewpoint P to the center of the overlapping area AO2. This is just one example, and the viewing angle of the entire viewing area, the viewing angle of the viewing area of ​​each image, the viewing angle of each overlapping area, or the reference direction (presence or absence, and its direction) can be set arbitrarily.

[0131] As in this example, when the entire viewing area of ​​the VR space is divided into three areas, two overlapping areas, a first overlapping area AO1 and a second overlapping area AO2, are generated. The above-mentioned processes related to the overlapping areas (and the processes described below) may be executed for each of the first and second overlapping areas AO1 and AO2. That is, in the relationship between the viewing area AL of the adjacent left image and the viewing area AF of the front image, the left image corresponds to the above-mentioned "first image", the front image corresponds to the above-mentioned "second image", and the first overlapping area AO1 corresponds to the above-mentioned "overlapping area AO", so that the above-mentioned processes and the like can be applied. In addition, in the relationship between the viewing area AF of the adjacent front image and the viewing area AR of the right image, the front image corresponds to the above-mentioned "first image", the right image corresponds to the above-mentioned "second image", and the second overlapping area AO2 corresponds to the above-mentioned "overlapping area AO", so that the above-mentioned processes and the like can be applied.

[0132] The same applies when dividing the entire field of view of the VR space into four or more regions. When placing n real-life images captured by n cameras (n is a natural number equal to or greater than 2) in the VR space, (n-1) overlapping regions will result. When there are multiple overlapping regions AO, one of the adjacent images placed in each overlapping region can be considered as the "first image" and the other as the "second image," and the above-mentioned processes can be applied to each overlapping region AO.

[0133] Usually, the field of view image (VR image) of the range displayed on the display unit 11 of the HMD 10 is a part of the entire field of view area of ​​the VR space. Therefore, if at least one overlapping area AO is not included in the range of the VR image displayed on the display unit 11, image switching processing for the overlapping area AO is unnecessary and can be prevented from being executed.

[0134] [10. How to provide a time lag before starting the switching process] When switching the overlapping area AO, a predetermined time lag (for example, 0.5 seconds) may be provided before the execution of the process starts. By providing such a time lag, unnecessary switching processes can be omitted. This will be described below.

[0135] Here, as shown in FIG. 26, a case where the area of ​​the VR space V is in three directions, left, front, and right, is illustrated. It is assumed that a user wearing the HMD10 is viewing a VR image in a range including the left viewing area AL and the first overlapping area AO1. It is also assumed that the left image is currently displayed in the first overlapping area AO1. It is assumed that from this state, the user swings his head largely to the right, and the visual axis direction (line of sight direction S) of the HMD10 moves instantaneously past the front to the right viewing area AR. If no time lag is set, the switching process in the first overlapping area AO1 and the switching process in the second overlapping area AO2 may occur in a short time. However, when the visual axis direction of the HMD10 reaches the right viewing area AR, the first overlapping area AO1 may already be outside the range of the VR image displayed on the display unit 11 of the HMD10. Even in such a case, if the switching process is executed in a sensitive reaction, the VR image displayed on the display unit 11 may shake, and visibility may be deteriorated. On the other hand, in the above-mentioned case, by providing a time lag in the switching process, it is possible to execute only the switching process of the second overlap area AO2 as necessary and omit the switching process of the first overlap area AO1, thereby improving the visibility of the image displayed on the display unit 11. In addition, in the description where no distinction is made between the first overlap region AO1 and the second overlap region AO2, they may be referred to as "overlap region AO."

[0136] The image generating unit 32 in this embodiment has a function of providing a predetermined waiting period from when it is determined that the real-life image to be displayed in the overlapping area AO should be switched based on the line of sight S until the switching process is started. Here, the waiting period corresponds to the above-mentioned time lag. Furthermore, after the waiting period until the start of the switching process has elapsed, the image generating unit 32 re-determines whether or not the switching process should be performed, and executes the switching process only when it determines that the switching process should be performed.

[0137] The waiting period is, for example, 0.5 seconds, but is not limited thereto and can be set arbitrarily. The length of the waiting period may be different between a case where the gaze direction S is specified based on detection information (detection information from an angular velocity sensor of the HMD 10, etc.) regarding the orientation of the HMD 10 and whether or not the overlapping area AO needs to be switched, and a case where the gaze direction S is specified based on detection information (detection information from eye tracking, etc.) regarding the user's gaze and whether or not the overlapping area AO needs to be switched. That is, in eye tracking, the gaze is tracked by analyzing the movement of a person's eyeball, etc., so the gaze direction S is likely to move freely. For this reason, it is better to set the waiting period (time lag) longer for a determination based on eye tracking that tracks the gaze of the user than for a determination based on so-called head tracking that detects the orientation of the HMD 10. That is, when the gaze direction S is specified based on eye tracking and the switching process is performed, in the case of a user who moves his / her eyes a lot, the switching process occurs frequently and the image is likely to shake, but by lengthening the waiting period, unnecessary switching processes can be suppressed and the visibility of the image can be improved.

[0138] For example, in the case of the HMD 10 equipped with both a head tracking function using an angular velocity sensor or the like and an eye tracking function, the user may be able to select which of the two functions to use to determine whether or not the overlap area AO needs to be switched. In this case, the standby period is changed based on the user's selection.

[0139] Next, an example of the above-mentioned process with a time lag will be described with reference to Fig. 27. Fig. 27 is a flow chart showing an example of the process of switching between real-life images to be displayed in the overlapping area in the display control device 20. The control unit 30 specifies the line of sight direction S based on detection information from the sensor 12 (e.g., an angular velocity sensor) of the HMD 10, and determines whether or not switching processing of the real-life image displayed in the overlapping area AO is required (S500). When the control unit 30 determines that switching processing is required (YES in S500), it sets a waiting period (e.g., 0.5 seconds) and starts timing (S502). Thereafter, the control unit 30 does not execute switching processing for the overlapping area AO until the waiting period has elapsed. After the waiting period has elapsed (YES in S504), the control unit 30 again specifies the line of sight direction S based on the detection information from the sensor 12 of the HMD 10, and determines whether or not switching processing of the real-life image displayed in the overlapping area AO is required (S506). When the control unit 30 determines that switching processing is required (YES in S506), it executes switching processing (S508). On the other hand, when the control unit 30 determines that switching processing is not required (NO in S506), it ends the processing without executing switching processing. As an example of a case where the switching process is unnecessary, as described above, the target overlap area AO is already out of the range of the VR image displayed on the display unit 11 of the HMD 10. In addition, after it is determined that the switching process for the overlap area AO is necessary because the line of sight direction S has once crossed the reference direction RD, the switching process is also unnecessary if the line of sight direction S crosses the reference direction RD again and returns to the vicinity of the original position before the waiting period has elapsed.

[0140] [11. How to change the placement of images in VR space] 9 to 11, the number of viewing areas (number of images) arranged in the VR space is two, and the overall viewing area with a horizontal viewing angle of 180 degrees is divided into two areas on the left and right, but these are not fixed and may be changed depending on the scene (scene) and content of the image to be displayed. For example, when a scene changes in a game or other content using live-action images, the number of viewing areas (number of images) arranged in the VR space, the viewing angle of the overall viewing area, or the direction of each viewing area in the VR space (position in the VR space), etc. may be changed according to the content of the scene.

[0141] For example, as shown in Fig. 28, in one scene, the 270-degree overall field of view is divided into three regions as shown in (A) in the figure, and in another scene, the 220-degree overall field of view is divided into two regions as shown in (B) in the figure. Depending on the scene, the arrangement state may be (A) or (B). For example, the arrangement (A) is basically adopted, but in a scene where only the front area is of interest and an image of the rear area is not necessary, the arrangement (B) is adopted.

[0142] In yet another scene, the 180-degree overall viewing area may be divided into two areas, as shown in Fig. 9. Note that the overall viewing area may not be divided, and a scene may include only one area (for example, only a single 140-degree overall viewing area) as part of the scene.

[0143] For example, in a certain scene, three images arranged in a VR space as illustrated in (A) of FIG. 28 are captured using three cameras 200 shown in FIG. 26. In another scene, two adjacent cameras 200 out of the three cameras 200 are used to capture two images arranged in a VR space as illustrated in (B) of FIG. 28. In this case, the lens of at least one of the three cameras 200 may be replaced as necessary to enlarge (or reduce) the angle of view, or the imaging direction (direction of the visual axis) of each camera 200 may be adjusted. For example, when switching from an indoor scene to an outdoor scene, or from a scene in one room to a scene in another room, the installation position of the camera 200 capturing each scene also changes, so that a real-life image may be captured while considering the placement state in the VR space suitable for each scene.

[0144] As described above, together with the data of the plurality of live-action images to be displayed, the arrangement information for arranging the plurality of live-action images in the VR space is stored in the storage device 22 in association with the plurality of live-action images. For example, for each scene in which the camera's shooting location in the real space changes, the arrangement information described above according to the shooting conditions in each scene is generated and associated with each scene and recorded together with the live-action images. In this way, when the arrangement state of the plurality of live-action images in the VR space is appropriately changed according to the scene, the arrangement information associated with each scene is stored in the storage device 22. Alternatively, the arrangement information for each scene is recorded in the header of the live-action image file, etc., and the arrangement information is read out and used when the live-action image is played back.

[0145] The image generating unit 32 in this embodiment includes an image arranging unit 321 that changes the arrangement state of a plurality of real-life images in the VR space during display control of the VR image on the display unit 11. Here, the objects to be changed in the arrangement state in the VR space include, for example, the number of viewing areas (number of images) arranged in the VR space, the viewing angle of the entire viewing area, the viewing angle of each viewing area, the direction (position) of each viewing area in the VR space, the number of overlapping areas, the viewing angle of each overlapping area, or the direction (position) of each overlapping area in the VR space. These pieces of information are recorded as arrangement information in association with the plurality of real-life images to be displayed (for example, in association with each scene of the plurality of real-life images).

[0146] For example, in the VR space arrangement of (A) in Figure 28, the number of viewing areas (number of images) = 3, the viewing angle of the overall viewing area is "θall = 270 degrees", the direction of the overall viewing area (position in VR space) is "-135 degrees to 135 degrees", the viewing angles of the left, front, and right viewing areas AL, AF, and AR are "θAL = θAF = θAR = 120 degrees", the direction of the left viewing area AL is "15 degrees to 135 degrees", the direction of the front viewing area AF is "-60 degrees to 60 degrees", the direction of the right viewing area AR is "-135 degrees to -15 degrees", the viewing angle of the first overlap area AO1 is "θAO1 = 45 degrees", the direction of the first overlap area AO1 is "15 degrees to 60 degrees", the viewing angle of the second overlap area AO2 is "θAO2 = 45 degrees", and the direction of the second overlap area AO1 is "-60 degrees to -15 degrees". All or part of this information is recorded as placement information. The directional angles shown here are horizontal angles with the X-axis direction (reference line of sight direction) set to 0 degrees, and the vertical angles are omitted here. In addition, in the VR space arrangement state of (B) in Figure 28, the number of viewing areas (number of images) = 2, the viewing angle of the entire viewing area is "θall = 220 degrees", the direction of the entire viewing area is "-110 degrees to 110 degrees", the viewing angles of the left and right viewing areas AL, AF, AR are "θAL = θAR = 135 degrees", etc. (omitted below) are recorded as arrangement information.

[0147] The image arrangement unit 321 determines whether or not it is necessary to change the arrangement state of multiple live-action images to be displayed in the VR space based on arrangement information associated with the multiple live-action images, and changes the arrangement state appropriately based on the arrangement information.

[0148] In the image display system 1 of this embodiment, as described above, one VR image data completed by stitching processing is not created in advance, but a plurality of real-life images are arranged in the VR space while performing VR image display control. Therefore, while playing back a VR image, it is also possible to appropriately change the arrangement state of the real-life images in the VR space as in this embodiment. For example, the number of viewing areas (the number of real-life images) arranged in the VR space and the viewing angle of the entire viewing area can be changed depending on the scene. That is, it is possible to optimize the VR space for each scene. Therefore, it is possible to eliminate unnecessary viewing areas for each scene, and the amount of image data can be reduced.

[0149] In addition, by eliminating unnecessary viewing areas for each scene, not only can the data volume be reduced, but also the resolution of the VR image displayed on the display unit 11 of the HMD 10 can be increased in the scene where the unnecessary viewing areas have been eliminated. In other words, by eliminating unnecessary viewing areas, a VR image with a higher resolution can be deployed in a VRAM or the like. For example, a VR image with a higher resolution is displayed on the display unit 11 of the HMD 10 in the arrangement state (B) of the VR space in FIG. 28 than in the arrangement state (A). Therefore, the VR space for each scene may be optimized by adopting the arrangement state (B) in FIG. 28 for scenes where it is desired to increase the image resolution, and adopting the arrangement state (A) for other scenes.

[0150] Next, an example of the processing of this embodiment will be described with reference to Fig. 29. Fig. 29 is a flowchart showing an example of the processing of changing the arrangement state of images in the VR space in the display control device 20. The control unit 30 acquires arrangement information of the plurality of real-life images to be displayed (S600), and determines the arrangement state of the plurality of real-life images in the VR space based on the arrangement information (S602). When the VR image is played back, the arrangement state in the VR space determined here is applied, and the images are arranged in the VR space so that an overlapping area AO occurs in which parts of the viewing areas of adjacent images overlap each other. Also, as described above, the real-life images to be displayed in the overlapping area AO are dynamically switched based on the line of sight S (for example, the visual axis direction of the HMD 10), and the VR image is generated. Also, when the scene is changed during the playback of the VR image (YES in S604), the control unit 30 acquires arrangement information of the changed scene (S606), and determines whether or not the arrangement state of the images in the VR space should be changed (S608). Here, when the control unit 30 determines that the arrangement state needs to be changed (YES in S608), it changes the arrangement state of the plurality of real-life images in the VR space (S610). As a result, after the scene is changed, the arrangement state in the VR space suitable for the scene is applied, and the image of the scene is arranged in the VR space, and then the VR image is generated. The processes of S604 to S610 are repeated until the display is ended (YES in S612).

[0151] [12. How to change the direction of the entire viewing area in VR space] In the above description, regarding the camera capturing multiple real-life images of the display target, changing the direction of the camera during image capture in the real space was not considered, but the camera direction may be changed during image capture. In this case, the change in the image capture direction during image capture may be reflected in the field of view of the VR space.

[0152] For example, as illustrated in FIG. 26, there is a relationship between three cameras 200 capturing images of a real space and a viewing area of ​​a VR space V in which the images captured by the three cameras 200 are arranged. That is, the imaging direction of each camera 200 in FIG. 26 is the reference imaging direction of each camera, and the reference imaging direction of the front camera 200 corresponds to the X-axis direction (reference line of sight direction) of the VR space V. Here, FIG. 30 illustrates a case in which the three cameras 200 in FIG. 26 are rotated to the right. FIG. 30 illustrates a relationship in which the entire viewing area of ​​the VR space V in which the images captured by the three cameras 200 are arranged is also rotated according to a change in the direction (rotation of the imaging direction) during imaging of the three cameras 200.

[0153] In the example of Fig. 26, in the case of an image captured by camera 200 in the reference imaging direction, the direction of the entire viewing area in VR space V is "-135 degrees to 135 degrees" with the X-axis direction (reference line of sight direction) being 0 degrees. On the other hand, in the example of Fig. 30, the imaging direction of camera 200 is rotated 45 degrees (i.e. -45 degrees) to the right from the reference imaging direction in Fig. 26. In the case of an image captured by camera 200 in Fig. 30, the direction of the entire viewing area in VR space V rotates 45 degrees to the right and changes to "-180 degrees to 90 degrees".

[0154] Information on the change in the imaging direction of the camera 200 may be detected from an angular velocity sensor (gyro sensor) or the like mounted on the camera 200 at the time of imaging, and may be recorded together with the image data at the time of imaging. Alternatively, after imaging, the change in the imaging direction may be identified from the captured image by known image analysis, and the information on the change in the imaging direction may be recorded in association with the captured image. Alternatively, the display control device 20 may identify the change in the imaging direction in real time from the captured image by image analysis during image playback.

[0155] For example, in a real space, a scene is considered in which a car passes in front of the viewer from the left and drives past to the right, and the camera is rotated from left to right to capture the image. FIG. 31 shows an example of a change in the field of view of the VR space V in which the image captured by the camera 200 is placed, when the information on the rotation of the imaging direction of the camera 200 is not used. When the information on the rotation of the imaging direction is not used, the direction of the entire field of view does not change even if the camera 200 rotates, and the imaging direction of the front camera 200 always corresponds to the X-axis direction of the VR space V. Therefore, when the captured image is displayed on the HMD 10 without using the information on the rotation of the imaging direction, it is possible to display the image of the car without crossing the boundary between the areas of the adjacent images, but the car is displayed in the same direction with respect to the visual axis (line of sight direction S) of the HMD 10, even though the car is moving, and the reality is not maintained, unlike reality.

[0156] FIG. 32 shows an example of a change in the field of view of the VR space V in which each image captured by the camera 200 is arranged when information on the rotation of the imaging direction of the camera 200 is used. When information on the rotation of the imaging direction is used, the direction of the entire field of view in the VR space V also rotates according to the rotation of the imaging direction. In this case, when the captured image is displayed on the HMD 10, an image of a car moving from left to right is also displayed with respect to the visual axis (line of sight direction S) of the HMD 10. Therefore, if a car is present on the left during image playback, the user must turn the HMD 10 to the left in order to see the car (or the car will be seen to the left when the user faces forward). By turning the head in the direction of the car, the user can see the car clearly, and a realistic VR image can be displayed. In addition, it becomes possible to display the image of the car on the display unit 11 of the HMD 10 without crossing the area boundary (boundary if a boundary is displayed) of adjacent images.

[0157] The control unit 30 of the display control device 20 of this embodiment can be configured to include an imaging direction identification unit 34, as exemplified in Fig. 20. This imaging direction identification unit 34 has a function of identifying changes in the imaging directions of the multiple cameras that captured the multiple real-life images. For example, the imaging direction identification unit 34 can identify the change in the imaging direction by reading out "information on change in imaging direction" that is recorded in association with the multiple real-life images. Alternatively, the imaging direction identification unit 34 may identify the change in the imaging direction from the real-life images by image analysis. The image generating unit 32 has a function of changing the direction of the field of view (total field of view) of the entire plurality of real-life images arranged in the VR space in response to the change in the imaging direction identified by the imaging direction identifying unit 34.

[0158] Next, an example of the processing of this embodiment will be described with reference to Fig. 33. Fig. 33 is a flowchart showing an example of the processing of changing the direction of the entire visual field area of ​​the VR space in the display control device 20. When the reproduction of the VR image starts, the control unit 30 initializes the direction of the entire viewing area in the VR space to the direction corresponding to the reference imaging direction (S700). For example, as illustrated in FIG. 26, the horizontal direction of the entire viewing area is set to "-135 degrees to 135 degrees" as a default, and the direction of the center of the viewing area AF in front of the virtual viewpoint P is set to the X-axis direction (reference line of sight direction). The control unit 30 also acquires information on the change in the imaging direction of the plurality of real-life images to be displayed (S702) and identifies the change in the imaging direction (S704). For example, the control unit 30 identifies the change in the imaging direction by reading and acquiring information on the change in the imaging direction recorded together with the data of the plurality of real-life images to be displayed, for example, for each frame. Then, when the control unit 30 determines that the imaging direction has changed from the previous frame (YES in S706), it changes the direction of the entire viewing area in the VR space according to the change in the imaging direction (S708). On the other hand, when it determines that the imaging direction has not changed (NO in S706), it returns to step S702. The processes of S702 to S708 are repeated until the display ends (YES in S710).

[0159] [13. Summary] As described above, the program of this embodiment is a program for causing a display control device 20 (an example of a computer) that executes control to display a VR image representing the field of view from a virtual viewpoint P in a VR space V as a stereoscopic image utilizing binocular parallax on the display unit 11 of the HMD 10 to function as a gaze direction identification unit 31 and an image generation unit 32. The display control device 20 according to the present embodiment includes a line of sight direction specification unit 31 and an image generation unit 32. The line of sight direction specification unit 31 specifies a line of sight direction S, which is the direction of the user's line of sight with respect to the VR space. The image generation unit 32 arranges a plurality of real-life images captured by a plurality of cameras each having a different imaging direction for capturing an image of a real space in the VR space V so that an overlapping area AO is generated in which adjacent real-life images partially overlap each other, and generates the VR image according to the line of sight direction S. Here, the image generation unit 32 dynamically switches the real-life image to be displayed in the overlapping area AO among the adjacent real-life images based on the line of sight direction S.

[0160] Here, the "VR image" in this configuration is a real-life image that can be viewed stereoscopically using binocular parallax and is displayed on the display unit 11 of the HMD 10, but a display object other than a real-life image may be included in the "VR image". For example, a display object other than a real-life image, such as CG (Computer Graphics), various objects, lines, symbols, and characters, may be arranged in the VR space and superimposed on a real-life image to form a "VR image". The "VR image" includes an AR (Augmented Reality) image or an MR (Mixed Reality) image. The "VR image" may be a still image or a video. When the "VR image" is supplied (transmitted) to the display unit 11 of the HMD 10, for example, it may be appropriately converted into a transmission format corresponding to the HMD 10.

[0161] Also, the "HMD" may be, for example, an HMD 50 including an attachment (wearing tool) 51 that can be worn on the user's head and an information processing device 52, such as a smartphone, that is attached to the attachment 51, as shown in FIG. 34. In this case, a display unit 61 of the information processing device 52 becomes the display unit of the HMD 50. Also, the "HMD" may be a so-called standalone type HMD in which the function of the display control device 20 is integrated with the HMD 10. Also, the "HMD" may be a goggle type or eyeglass type that can be worn on the head. Also, the "HMD" is not limited to an HMD in the narrow sense, and may be, for example, a headphone, a headset (a headphone with a microphone), an eyeglass type camera, an ear-mounted camera, a hat with a camera, or the like that has the function of an HMD.

[0162] In addition, the "display control device" according to this embodiment can be applied to various devices that have an information processing function such as generating a VR image. For example, a stationary or portable game machine, an arcade (commercial) game machine, a personal computer, a tablet computer, a smartphone, a mobile phone terminal, a PHS terminal, a PDA, a multi-function television receiver with an information processing function, and the like can be the "display control device". In addition, if the HMD itself has an information processing function such as generating a VR image, the HMD can be the "display control device" of this configuration. For example, as illustrated in FIG. 34, in the case of an HMD 50 including an attachment 51 and an information processing device 52, the information processing device 52 such as a smartphone is an example of the "display control device" according to this embodiment. In addition, the standalone HMD is an example of the "display control device" according to this embodiment.

[0163] According to the above configuration, by dynamically switching the live-action images to be displayed in the overlapping area AO based on the line of sight S, the area boundary (BD1 or BD2) between the adjacent live-action images in the overlapping area AO can be moved away from the direction of the user's line of sight. This can reduce the sense of incongruity caused by the "seam (shift) between adjacent live-action images" occurring at the area boundary. As described above, in the conventional technology for performing stitching processing, if a close-up object whose distance from the camera is less than a certain value (for example, less than 2 m) is present at the image seam, stitching becomes difficult and VR image data cannot be created. In contrast, according to the present configuration, even if a close-up object whose distance from the camera is less than a certain value is included in the live-action image and the close-up object is present at or near the area boundary, a stereoscopic VR image that is less uncomfortable for the user is displayed. In other words, a VR image that is less uncomfortable can be displayed no matter where the close-up object is located in the VR space, and a VR image display control that allows close-up display can be realized. In addition, since the stitching process is no longer necessary, the effort required for creating VR images is reduced. In addition, since the stitching process is no longer necessary, it is possible to use the actual images (live footage) currently being captured by the camera to display VR images on the display unit 11 in approximately real time, with minimal awkwardness at the seams between the actual images.

[0164] The image generating unit 32 may set a reference direction RD in a predetermined direction from the virtual viewpoint P, and specify an actual image to be displayed in the overlapping area AO based on the line of sight direction S relative to the reference direction RD. This makes it possible to easily realize switching control of the actual image to be displayed in the overlapping area AO.

[0165] Furthermore, the image generating unit 32 may set a reference direction RD in the direction from the virtual viewpoint P to a predetermined position in the overlapping area AO. As a result, the reference direction RD is set within the overlapping area AO, so that even if the line of sight direction S heads toward the area boundary (BD1 or BD2) of the overlapping area AO, the real-life image displayed in the overlapping area is switched before reaching the area boundary, and therefore the area boundary also moves away from the direction of the user's line of sight.

[0166] Furthermore, the image generating unit 32 may change the reference direction RD according to the real-life image to be displayed in the overlapping area AO at the timing when the real-life image to be displayed in the overlapping area AO is switched. This causes the reference direction RD to temporarily move away from the line of sight direction S after the real-life image is switched in the overlapping area AO, thereby reducing frequent switching of the real-life image displayed in the overlapping area AO.

[0167] The image generating unit 32 may specify the real-life image to be displayed in the overlapping area AO based on the line-of-sight direction S with respect to the current area boundary (BD1 or BD2) of the adjacent real-life images. This makes it possible to easily realize switching control of the real-life images to be displayed in the overlapping area AO without setting a reference direction. Furthermore, since the reference area boundary is changed at the timing when the real-life images to be displayed in the overlapping area AO are switched, the same effect as the configuration for changing the reference direction RD described above is achieved. In other words, it is possible to reduce frequent switching of the real-life images to be displayed in the overlapping area AO.

[0168] Furthermore, the image generating unit 32 may switch the live-action image currently displayed in the overlapping area AO to the other live-action image when the line of sight direction S approaches the current area boundary (BD1 or BD2) of the adjacent live-action image by a predetermined amount or more (for example, when the angle between the line of sight direction S and the current area boundary becomes a predetermined value or less). In this case, too, it is possible to easily realize control of switching the live-action images displayed in the overlapping area AO without setting a reference direction. Moreover, before the line of sight direction S reaches the area boundary, the live-action image displayed in the overlapping area is switched, and the area boundary can be moved away from the direction of the user's line of sight.

[0169] Furthermore, the gaze direction identification unit 31 may obtain detection information (e.g., detection information from an angular velocity sensor) regarding the orientation of the HMD 10, and identify the gaze direction S based on the detection information. This allows the user to dynamically switch the real-life image displayed in the overlap area AO by changing the orientation of the HMD 10.

[0170] Furthermore, the gaze direction specifying unit 31 may obtain detection information (e.g., eye tracking detection information) regarding the user's gaze, and specify the gaze direction S based on the detection information. This allows the user to dynamically switch between real-life images to be displayed in the overlapping area AO by changing the movement of the user's eyes (the direction of the gaze).

[0171] The image generating unit 32 may also obtain detection information regarding the orientation of the HMD 10 and change the range of the VR image to be displayed on the display unit 11 based on the detection information. According to this configuration, the range of the VR image to be displayed on the display unit 11 is changed according to the orientation of the HMD 10, and further, in the VR image displayed on the display unit 11, the real-life image to be displayed in the overlapping area AO is dynamically switched based on the line of sight direction S. Here, when the line of sight direction S is specified based on the detection information regarding the orientation of the HMD 10, the user can dynamically switch the real-life image to be displayed in the overlapping area AO while changing the range of the VR image to be displayed on the display unit 11 by adjusting the orientation of the HMD 10. Also, when the line of sight direction S is specified based on detection information regarding the user's line of sight (for example, detection information of eye tracking), the user can adjust the range of the VR image to be displayed on the display unit 11 by changing the orientation of the HMD 10, and can dynamically switch the real-life image to be displayed in the overlapping area AO by changing the movement of the eyes (direction of line of sight).

[0172] The display control device 20 may also be configured to include a boundary display unit 33. The boundary display unit 33 displays a boundary (BL1 or BL2) visible to the user at the area boundary (BD1 or BD2) between adjacent real-life images. In this way, intentionally displaying the boundary at the area boundary in a visible manner makes it easier for the user to recognize the area boundary where a seam (shift) of images occurs. This makes it easier for the user to determine which of the adjacent real-life images should be displayed in the overlapping area and viewed, so that the boundary is not displayed in the direction the user wants to view. Furthermore, the boundary display unit 33 may display the boundary line (BL1 or BL2) displayed at the area boundary (BD1 or BD2) at one end of the overlap area AO in a different manner from the boundary line displayed at the area boundary at the other end. This allows the user to recognize whether the currently displayed boundary line is the misaligned part of the images at one end (e.g., the left end) of the overlap area or the misaligned part of the images at the other end (e.g., the right end).

[0173] Furthermore, the boundary display unit 33 may display additional information (for example, a gradation portion (GD1 or GD2)) on or near the boundary line (BL1 or BL1) to indicate which side of the boundary line is the overlapping area AO. In this way, by enabling the user to recognize which side of the boundary line is the overlapping area, it becomes easier for the user to determine in which direction the line of sight S should be changed to switch the live-action image displayed in the overlapping area AO.

[0174] The additional information can be a semi-transparent gradation section (GD1 or GD2) that is superimposed on the real image near the boundary line and whose density or opacity decreases continuously or stepwise as it moves away from the boundary line. This gradation section has the effect of hiding the image misalignment occurring near the boundary line (BL1 or BL2) and making it less noticeable. Also, the change in density or opacity of the gradation section (GD1 or GD2) (from dark to light, or from light to dark) naturally represents the direction of the change in the line of sight S required to switch the real image in the overlapping area AO.

[0175] Furthermore, when switching between live-action images displayed in the overlapping area AO, the image generating unit 32 may gradually switch from one live-action image to the other over time. For example, during a switching period (e.g., 0.3 seconds), the two live-action images to be switched may be alpha-blended and cross-faded. When switching between live-action images displayed in the overlapping area AO, the boundary line can be more naturally and easily recognized by gradually switching over the time that passes during the switching period from the start to the end of the switching, rather than switching suddenly.

[0176] Furthermore, when switching between live-action images displayed in the overlapping area AO, the image generating unit 32 may gradually switch from one live-action image to the other live-action image while gradually moving the boundary line (BL1 or BL1) toward the position to be displayed after the switching is completed as time passes. In this way, by moving the boundary line during the switching period between live-action images displayed in the overlapping area AO, the user can naturally recognize the position of the boundary line to be changed.

[0177] Furthermore, the image generating unit 32 may provide a predetermined waiting period from when it is determined based on the line of sight that the real-life image to be displayed in the overlapping area AO should be switched to another image to be displayed in the overlapping area AO, until the switching process is started. This makes it possible to omit unnecessary switching process when the line of sight changes during the waiting period, resulting in a state in which the switching process is no longer necessary (for example, when the waiting period has elapsed, the overlapping area AO has already fallen outside the range of the VR image displayed on the display unit 11 of the HMD 10, etc.).

[0178] In addition, in the case of the HMD 10 equipped with both a head tracking function using an angular velocity sensor or the like and an eye tracking function, the following may be done. That is, the image generating unit 32 may change whether to specify the gaze direction S based on the detection information on the orientation of the HMD 10 or on the detection information on the user's gaze (for example, the detection information on eye tracking) based on a selection operation by the user. In this case, the waiting period may be changed based on the user's selection. For example, when specifying the gaze direction S based on the detection information on the user's gaze is selected, the waiting period may be made longer. In addition, since there are differences in the way users move their heads and eyes, the waiting period may be set arbitrarily by the user. For example, the waiting period may be set by the user within a predetermined range (for example, a range of 0.1 to 1.0 seconds). In addition, the waiting period for head tracking and the waiting period for eye tracking may be set by the user.

[0179] Furthermore, the image generating unit 32 may change the arrangement state of a plurality of real-life images in the VR space during display control of the VR image on the display unit 11. For example, as shown in Fig. 28, in one scene, for example, a total viewing area of ​​270 degrees may be divided into three areas, and in another scene, for example, a total viewing area of ​​220 degrees may be divided into two areas, and so on, so that the number of viewing areas and the direction of the viewing areas in the VR space may be changed according to the content of the scene. In the display control of the VR image of this embodiment, one VR image data completed by stitching processing is not created in advance as in the past, but the display control of the VR image is executed while arranging a plurality of real-life images in the VR space. Therefore, it is also possible to appropriately change the arrangement state of the real-life images in the VR space during the playback of the VR image. For example, as described above, the number of viewing areas (the number of real-life images) arranged in the VR space and the viewing angle of the entire viewing area can be changed depending on the scene. That is, it is possible to optimize the VR space for each scene. Therefore, it is possible to eliminate unnecessary viewing areas for each scene, and the amount of image data can be reduced. In addition, in a scene where the necessary viewing areas are eliminated, the resolution of the VR image displayed on the display unit 11 of the HMD 10 can be increased.

[0180] The display control device 20 may also be configured to include an imaging direction specification unit 34. The imaging direction specification unit 34 specifies the change in the imaging direction of the multiple cameras that captured the multiple real-life images. The image generation unit 32 then changes the direction of the field of view of the multiple real-life images arranged in the VR space in accordance with the change in the imaging direction specified by the imaging direction specification unit 34. With this configuration, for example, in the case of a real-life image in which the imaging direction of the camera changes in accordance with the movement of a moving subject (such as a car), the direction of the field of view of the entire real-life images arranged in the VR space also changes in accordance with the change in the imaging direction. Therefore, the display unit 11 of the HMD 10 can display a realistic VR image in which the subject moves relative to the line of sight S. Also, if a real-life image in which the subject is captured so as not to cross the boundary between adjacent real-life images is prepared during imaging in accordance with the above-mentioned movement of the subject, it becomes possible to reproduce a VR image with excellent visibility in which the subject does not cross the boundary between the areas.

[0181] [14. Modifications, etc.] Although the embodiments of the present invention have been described above, the specific configurations are not limited to the above-mentioned embodiments, and designs that do not deviate from the gist of the present invention are also included. In addition, the above-mentioned configurations and aspects can be combined in any combination.

[0182] [14-1] As described above, the image display system or display control device of the present embodiment can reduce the discomfort of the seams (misalignment) between multiple real-life images captured by multiple cameras by display control that dynamically switches between real-life images displayed in overlapping areas, without stitching processing. Therefore, it is also possible to use real-life images (live video) captured in real time by a camera that captures real space as they are and display them on the HMD 10 as VR images.

[0183] For example, the camera 200 illustrated in FIG. 26 may be connected to the display control device 20 by wire or wirelessly, and the live video captured by the camera 200 may be displayed on the HMD 10 as a VR image. The camera 200 and the display control device 20 may be connected to each other via a network such as the Internet so as to enable data communication. In this case, the live video of a remote location captured by the camera 200 may be displayed on the HMD 10 as a VR image. A server having a distribution service function may be set on the network, and the live video captured by the camera 200 may be distributed via the network through the server. In this case, the display control device 20 may receive the live video of a remote location distributed via the network, and the live video may be displayed on the HMD 10 as a VR image in approximately real time.

[0184] In other words, an image display system including multiple cameras each having different imaging directions for imaging real space, a display control device of the above-mentioned configuration, and an HMD, or an image display system including the multiple cameras, a distribution server, a display control device of the above-mentioned configuration, and an HMD, can be constructed to display live footage including close-up shots as VR images on the HMD. [14-2] In the above, the display control for displaying a VR image representing a field of view from a virtual viewpoint in a VR space on the display unit 11 of the HMD 10 as a stereoscopic image using binocular parallax has been described, but it can also be applied to the display control of a non-stereoscopic image that does not use binocular parallax, that is, a two-dimensional (2D) image. That is, when binocular parallax is used, all or part of the above-mentioned processes are executed for each of the right-eye image and the left-eye image, but in the case of a 2D image, all or part of the above-mentioned processes are executed for an image common to the right and left eyes. There is no difference in the processing itself between a stereoscopic image (image for the right eye or image for the left eye) and a non-stereoscopic image (image common to both the left and right), and each of the above-mentioned configurations can be applied as it is to the display control of a non-stereoscopic image.

[0185] That is, the present invention can be applied to a display control device that executes control to display a VR image (3D image or 2D image) representing a field of view from a virtual viewpoint in a VR space as a stereoscopic image or a non-stereoscopic image on the display unit of an HMD. Conventionally, when a wide-viewing-angle image (moving image or still image) is created by synthesizing a plurality of real-life images captured by a plurality of cameras of a real space, whether a 3D image or a 2D image, a single piece of image data completed by a stitching process is created. In contrast, in the present embodiment, regardless of the presence or absence of stereoscopic vision, a plurality of real-life images captured by a plurality of cameras with different imaging directions capturing a real space are arranged in a VR space so that an overlapping region occurs in which a part of the field of view region of the adjacent real-life images overlaps with each other, and a control is performed to dynamically switch the real-life images to be displayed in the overlapping region among the adjacent real-life images based on the line of sight, thereby realizing a display control that reduces the sense of incongruity of the seams (shifts) between the real-life images without stitching.

[0186] [14-3] The above describes display control for displaying an image on the display unit of an HMD, but the present invention can also be applied to display control for displaying an image on a display unit other than an HMD. For example, the following describes an application to displaying a wide-angle real-life image on a display unit such as a normal display that is not worn on the user's head.

[0187] When a wide-angle image (e.g., an image with a 360-degree viewing angle) using multiple real-life images captured by multiple cameras of a real space is displayed on a display unit, the user views the image while changing the range of the image displayed on the display unit using, for example, a mouse or other pointing device. Alternatively, when the display unit is a touch panel equipped with a touch interface, the wide-angle image is viewed while changing the range of the image displayed on the display unit by touching with a finger or a stylus pen. Alternatively, a hand tracking system that captures an image of the user with a camera and performs image analysis of the hand movements may be used, allowing the user to view the image while changing the range of the image displayed on the display unit with the user's hand gestures. In this case, it is determined that the user's viewpoint or line of sight is at the center of the current image range displayed on the display unit (the center of the screen), and the line of sight direction is identified. Alternatively, for example, detection information regarding the user's line of sight may be obtained by an eye tracking system or the like provided on the display unit, and the line of sight direction may be identified based on the detection information.

[0188] In this way, when displaying a wide viewing angle image using multiple live-action images captured by multiple cameras with different imaging directions for capturing real space on a display unit other than an HMD, the display can be controlled as follows. That is, the multiple live-action images are arranged in the VR space so that an overlapping area occurs where adjacent live-action images partially overlap each other, and the live-action image to be displayed in the overlapping area among the adjacent live-action images is dynamically switched based on the line of sight. This display control can realize display control that reduces the discomfort of the seams (misalignment) between live-action images even on a display unit other than an HMD, without stitching processing. The wide viewing angle image displayed on a display unit other than the HMD may be a stereoscopic image using binocular parallax or a non-stereoscopic image. When displaying a stereoscopic image, images for the right eye and the left eye are displayed on a display unit that allows stereoscopic viewing with the naked eye. Alternatively, images for the right eye and the left eye are displayed alternately on the display unit, assuming the use of dedicated glasses (dedicated glasses using a frame sequential method or a polarized method, etc.).

[0189] [14-4] As illustrated in FIG. 1, when the HMD 10 and the display control device 20 both have the configuration and functions of an information processing device (computer) equipped with a processor and a storage device, some of the functions of the control unit 30 described above may be realized by a processor 13 of the HMD 10 that executes a program related to this embodiment, and the remaining functions may be realized by a processor 21 of the display control device 20 that executes the program related to this embodiment. FIG. 35 is a schematic block diagram showing an example of the configuration of a stand-alone type HMD 60 in which the function of the display control device 20 is integrated with the HMD 10, or an information processing device 52 such as a smartphone used as the HMD 50 illustrated in FIG. 34. The HMD 60 or the information processing device 52 includes a display unit 61, a sensor 62, a processor 63, a storage device 64, an operation unit 65, and a communication unit 66. The display unit 61, the sensor 62, the processor 63, the storage device 64, the operation unit 65, and the communication unit 66 are respectively configured similarly to the display unit 11, the sensor 12, the processor 21, the storage device 22, the operation unit 23, and the communication unit 24 described above, and their description will be omitted. Note that the operation unit 65 or the communication unit 66 may be configured separately, for example, by external attachment, or may be omitted. The stand-alone type HMD 60 or the information processing device 52 also has the configuration and function of an information processing device (computer) equipped with a processor and a storage device, and each function of the control unit 30 described above is realized by the processor 63 that executes the program according to this embodiment.

[0190] [14-5] Some or all of the functions of the control unit 30 described above may be realized by an integrated circuit such as an LSI (Large Scale Integration). Each of the above functions may be individually implemented as a processor. Alternatively, some or all of the above functions may be integrated into a processor.

[0191] [14-6] The computer-readable program according to the present embodiment is recorded on various computer-readable recording media such as a hard disk, an optical disk (CD-ROM, DVD-ROM, etc.), a flexible disk, and a semiconductor memory, and is read from the recording media and executed by a computer constituting the image display system 1 or the display control device 20. The program may be provided to the computer via a network including a communication line such as the Internet, a WAN, a LAN, or a dedicated line. The computer may read out a program stored in a file server (online storage). The computer may receive a program distributed from a distribution server. The recording media include internal or external recording media accessible from the distribution server to distribute the program. The code of the program stored in the recording media of the distribution server does not have to be in a format directly executable by the computer that receives it. In other words, the format of the program stored in the recording media of the distribution server is arbitrary as long as it can be installed in a computer executable manner after being downloaded from the distribution server. The program may be divided into a plurality of programs, downloaded at different times, and then combined. The divided programs may be distributed by different distribution servers. The computer-readable recording medium also includes a storage medium that stores the program for a certain period of time, such as a volatile memory such as a RAM in a server that transmits the program via a network or in a computer that receives the program. The program may also be a differential program that can realize the above-mentioned functions in combination with a program already stored in the computer.

[0192] [15. Notes] From the above description, the present invention can be understood, for example, as follows: In order to facilitate understanding of the present invention, reference numerals in the accompanying drawings are conveniently written in parentheses, but the present invention is not limited to the illustrated embodiments.

[0193] 1) A program according to one embodiment of the present invention causes a computer that executes control for displaying, on a display unit (11) of a head-mounted display (10), a VR image representing a field of view from a virtual viewpoint (P) in a virtual reality (VR) space (V) as a stereoscopic image utilizing binocular parallax, to function as: a gaze direction identification unit (31) that identifies a gaze direction (S) that is the direction of a user's gaze into the VR space; and an image generation unit (32) that arranges a plurality of real-life images captured by a plurality of cameras each having different imaging directions for capturing an image of a real space in the VR space so as to generate an overlapping area (AO) in which portions of the field-of-view areas (A1, A2) of adjacent real-life images overlap with each other, and generates the VR image according to the gaze direction, and the image generation unit (32) dynamically switches between the real-life images to be displayed in the overlapping area among the adjacent real-life images based on the gaze direction. Here, the "computer" may include at least a processor and a storage device (memory). For example, a stationary or portable game machine capable of controlling the display of a VR image, an arcade (commercial) game machine, a personal computer, a tablet computer, a smartphone, a mobile phone terminal, a PHS terminal, a PDA, a multifunctional television receiver with an information processing function, and other devices including a processor and a storage device are all included in the "computer". In addition, the HMD itself including the processor and the storage device is also included in the "computer". For example, as illustrated in FIG. 34, in the case of an HMD 50 including an attachment 51 and an information processing device 52, the information processing device 52 such as a smartphone is included in the "computer". In addition, a stand-alone HMD is also included in the "computer".

[0194] According to the aspect described in 1) above, by dynamically switching the live-action image to be displayed in the overlapping area based on the line of sight, the area boundary between adjacent live-action images in the overlapping area can be moved away from the direction of the user's line of sight. This can reduce the sense of incongruity of the "seam (shift) between adjacent live-action images" that occurs at the area boundary. Therefore, even if a close-up photographed object is present at or near the area boundary, a VR image that is less uncomfortable for the user is displayed. In other words, a VR image that is less uncomfortable can be displayed no matter where the close-up photographed object is located in the VR space, and VR image display control that allows close-up display can be realized. In addition, since the stitching process is unnecessary, the effort required for VR image production can be reduced. In addition, since the stitching process is unnecessary, a VR image that is less uncomfortable at the seam between live-action images can be displayed on the display unit in approximately real time using the live-action image (live video) currently captured by the camera.

[0195] 2) In one aspect of the present invention, in the aspect described in 1) above, the image generation unit (32) sets a reference direction (RD, RD1, RD2) in a predetermined direction from the virtual viewpoint (P), and identifies the actual image to be displayed in the overlap area (AO) based on the line of sight direction (S) relative to the reference direction.

[0196] According to the aspect described in 2) above, it is possible to easily realize control for switching between real-life images to be displayed in the overlapping area based on the line-of-sight direction with respect to the reference direction.

[0197] 3) In one aspect of the present invention, in the aspect described in 2) above, the image generation unit (32) sets the reference direction (RD, RD1, RD2) in the direction from the virtual viewpoint (P) to a predetermined position within the overlap area (AO).

[0198] According to the aspect described in 3) above, since the reference direction is set within the overlapping area, even if the line of sight is directed toward the boundary of the overlapping area, the real-life image displayed in the overlapping area is switched before the line of sight reaches the boundary, and therefore the boundary also moves away from the direction of the user's line of sight. This allows for VR image display control with excellent visibility.

[0199] 4) In one aspect of the present invention, in the aspect described in 2) or 3) above, the image generation unit (32) changes the reference directions (RD1, RD2) in accordance with the actual image to be displayed in the overlapping area (AO) at the timing when the actual image to be displayed in the overlapping area (AO) is switched.

[0200] According to the aspect described in 4) above, the reference direction is not fixed, but is changed according to the real-life image displayed in the overlapping area at the timing when the real-life image displayed in the overlapping area is switched. As a result, the reference direction temporarily moves away from the line of sight after the real-life image is switched, so that frequent switching of the real-life image displayed in the overlapping area can be reduced.

[0201] 5) In one aspect of the present invention, in any of the aspects described in 1) to 4) above, the gaze direction identification unit (31) acquires detection information (e.g., detection information from an angular velocity sensor) regarding the orientation of the head mounted display (10), and identifies the gaze direction (S) based on the detection information.

[0202] According to the aspect described in 5) above, the user can dynamically switch the real-life image displayed in the overlapping area by changing the orientation of the head mounted display.

[0203] 6) In one aspect of the present invention, in any of the aspects described in 1) to 4) above, the gaze direction identification unit (31) acquires detection information regarding the user's gaze (e.g., eye tracking detection information) and identifies the gaze direction (S) based on the detection information.

[0204] According to the aspect described in 6) above, the user can dynamically switch the real-life image to be displayed in the overlapping area by changing the movement of his / her eyes (the direction of the line of sight).

[0205] 7) In one aspect of the present invention, in any of the aspects described in 1) to 6) above, the computer is made to function as a boundary display unit (33) that displays a boundary line (BL1, BL1) visible to the user at the area boundary (BD1, BD2) between adjacent real-life images.

[0206] According to the aspect described in 7) above, by intentionally displaying a boundary line visibly at the boundary between adjacent real-life images, the boundary line where the image seam (shift) occurs becomes easier for the user to recognize. This makes it easier for the user to determine which real-life image should be displayed in the overlapping area of ​​the adjacent real-life images so that the boundary line is not displayed in the direction the user wants to see.

[0207] 8) In one aspect of the present invention, in the aspect described in 7) above, the boundary display unit (33) displays additional information (GD1, GD2) on or near the boundary line (BL1, BL1) to indicate which side of the boundary line is the overlapping area (AO).

[0208] According to the aspect described in 8) above, by enabling the user to recognize which side of the boundary line is the overlapping area, it becomes easier for the user to determine in which direction the line of sight needs to be changed to switch the live-action image displayed in the overlapping area.

[0209] 9) In one aspect of the present invention, in any of the aspects described in 1) to 8) above, when switching the live-action image to be displayed in the overlapping area (AO), the image generation unit (32) gradually switches from one live-action image to the other live-action image over time.

[0210] According to the aspect described in 9) above, when switching the live-action image displayed in the overlapping area from one to the other, by gradually switching over time during the switching period from the start to the end of the switching, rather than switching suddenly, the boundary line can be made more natural and easier to see.

[0211] 10) In one aspect of the present invention, in the aspect described in 9) above, when switching the live-action image to be displayed in the overlapping area (AO), the image generation unit (32) gradually switches from one live-action image to the other live-action image by gradually moving the boundary line (BL1, BL1) toward the position that will be displayed after the switching is completed over time.

[0212] According to the aspect described in 10) above, by moving the boundary line during the switching period of the real-life image displayed in the overlapping area, the user can naturally recognize the position of the boundary line that is being changed.

[0213] 11) In one aspect of the present invention, in any of the aspects described in 1) to 10) above, the image generation unit (32) sets a predetermined waiting period from when it determines, based on the line of sight (S), that a switching process should be performed for the actual image to be displayed in the overlap area (AO) until it starts the switching process.

[0214] According to the aspect described in 11) above, by providing a predetermined waiting period before starting the switching process of the real-life image to be displayed in the overlapping area, if the line of sight direction changes during the waiting period and the switching process becomes unnecessary, it is possible to omit unnecessary switching process.

[0215] 12) In one aspect of the present invention, in any of the aspects described above in 1) to 11), the image generation unit (32) changes the arrangement state of the multiple real-life images in the VR space (V) while controlling the display of the VR image on the display unit.

[0216] According to the aspect described in 12) above, the number of viewing areas arranged in the VR space and the viewing angle of the entire viewing area can be changed depending on the scene. In other words, the VR space can be optimized for each scene. Therefore, it is possible to eliminate unnecessary viewing areas for each scene, and the amount of image data can be reduced. In addition, in scenes where unnecessary viewing areas are eliminated, the resolution of the VR image displayed on the display unit of the HMD can be increased.

[0217] 13) In one aspect of the present invention, in any of the aspects described in 1) to 11) above, the computer is made to function as an imaging direction identification unit (34) that identifies a change in the imaging direction of the multiple cameras that captured the multiple live-action images, and the image generation unit (32) changes the direction of the viewing area of ​​the entire multiple live-action images to be placed in the VR space (V) in accordance with the change in the imaging direction identified by the imaging direction identification unit.

[0218] According to the aspect described in 13) above, for example, in the case of a real-life image in which the imaging direction of the camera changes in accordance with the movement of a moving subject (such as a car), the direction of the viewing area of ​​the entire real-life image arranged in the VR space also changes in accordance with the change in the imaging direction. Therefore, the display unit can display a realistic VR image in which the subject moves relative to the line of sight. In addition, if a real-life image in which the subject is imaged so as not to cross the boundary between adjacent real-life images is prepared during imaging in accordance with the above-mentioned movement of the subject, a VR image with excellent visibility in which the subject does not cross the boundary between the adjacent real-life images can be reproduced.

[0219] 14) In one aspect of the present invention, in any of the aspects described in 1) to 13) above, the image generating unit (32) dynamically switches the real-life images to be displayed in the overlapping area (AO) among the adjacent real-life images based on the line of sight (S) so that the area boundaries (A1, A2) of the adjacent real-life images move away from the line of sight (S). This provides the same effect as the aspect described in 1) above.

[0220] 15) In one aspect of the present invention, in any of the aspects described above in 1) to 14), the image generation unit (32) acquires detection information regarding the orientation of the head mounted display (10) and changes the range of the VR image to be displayed on the display unit (11) based on the detection information.

[0221] According to the aspect described in 15) above, when the gaze direction is specified based on the detection information regarding the orientation of the HMD, the user can dynamically switch the live-action image displayed in the overlapping area while changing the range of the VR image displayed on the display unit by adjusting the orientation of the HMD. Also, when the gaze direction is specified based on the detection information regarding the user's gaze, such as eye tracking, the user can dynamically switch the live-action image displayed in the overlapping area by changing the orientation of the HMD to adjust the range of the VR image displayed on the display unit and by changing the movement of the eyes (the gaze direction).

[0222] 16) In one aspect of the present invention, in any of the aspects described above in 1), 5) to 14), the image generation unit (32) identifies the real-life image to be displayed in the overlap area AO based on the line of sight direction (S) relative to the current area boundary (BD1, BD2) of the adjacent real-life images.

[0223] According to the aspect described in 16) above, it is possible to easily realize switching control of real-life images displayed in the overlapping area without setting a reference direction. In addition, since the reference area boundary is changed at the timing when the real-life images are switched, it is possible to reduce frequent switching of the real-life images displayed in the overlapping area.

[0224] 17) In one aspect of the present invention, in the aspect described in 15) above, the image generation unit (32) switches the live-action image currently displayed in the overlap area (AO) to the other live-action image when the line of sight (S) approaches the current area boundary (BD1, BD2) of the adjacent live-action image by a predetermined amount or more (for example, when the angle between the line of sight and the current area boundary becomes a predetermined value or less).

[0225] According to the aspect described in 17) above, the real-life image displayed in the overlapping area is switched before the line of sight reaches the area boundary, so that the area boundary can be moved away from the direction of the user's line of sight.

[0226] 18) In one aspect of the present invention, in the aspect described in 8) above, the additional information includes a translucent gradation portion (GD1, GD2) that is superimposed on the actual image near the boundary line (BL1, BL1) and whose density or opacity decreases continuously or in stages the further away from the boundary line.

[0227] According to the aspect described in 18) above, the image misalignment occurring near the boundary line can be hidden by the gradation to make it less noticeable. In addition, the gradation can naturally represent the direction of the change in line of sight required for switching between real images.

[0228] 19) In one aspect of the present invention, in any of the aspects described in 7) to 10) above, the boundary display unit (33) makes the display form of the boundary line (BL1 or BL2) displayed at the area boundary (BD1 or BD2) at one end of the overlap area (AO) different from the display form of the boundary line displayed at the area boundary at the other end.

[0229] According to the aspect described in 19) above, when multiple images arranged in the horizontal direction are used to expand the horizontal viewing angle, the user can recognize whether the currently displayed boundary line is the misaligned part of the image at the left edge of the overlapping area or the misaligned part of the image at the right edge. Also, when multiple images arranged in the vertical direction are used to expand the vertical viewing angle, the user can recognize whether the currently displayed boundary line is the misaligned part of the image at the top edge of the overlapping area or the misaligned part of the image at the bottom edge.

[0230] 20) A display control device (20) according to one aspect of the present invention executes control for displaying, on a display unit (11) of a head mounted display (10), a VR image representing a field of view from a virtual viewpoint (P) in a virtual reality (VR) space (V) as a stereoscopic image utilizing binocular parallax, and includes a gaze direction specification unit (31) that specifies a gaze direction (S) that is the direction of a user's gaze with respect to the VR space, and an image generation unit (32) that arranges a plurality of real-life images captured by a plurality of cameras each having a different imaging direction for capturing a real space in the VR space so as to generate an overlapping area (AO) in which a portion of the visual field areas (A1, A2) of adjacent real-life images overlap each other, and generates the VR image according to the gaze direction, and the image generation unit (32) dynamically switches the real-life image to be displayed in the overlapping area among the adjacent real-life images based on the gaze direction. This achieves the same effect as the aspect described in 1) above.

[0231] 21) An image display system (1) according to one aspect of the present invention includes a head-mounted display (10) that displays, on a display unit (11), a VR image representing a field of view from a virtual viewpoint (P) in a virtual reality (VR) space (V) as a stereoscopic image using binocular parallax, and a display control device (20) that executes control to display the VR image on the display unit, the display control device (20) includes a gaze direction specification unit (31) that specifies a gaze direction (S) that is the direction of a user's gaze with respect to the VR space, and an image generation unit (32) that arranges a plurality of real-life images captured by a plurality of cameras (200) each having a different imaging direction for capturing an image of a real space in the VR space so that an overlapping area (AO) is generated in which adjacent real-life images have their field of view areas (A1, A2) overlap each other, and generates the VR image according to the gaze direction, and the image generation unit (32) dynamically switches the real-life image to be displayed in the overlapping area among the adjacent real-life images based on the gaze direction. This achieves the same effect as the aspect described in 1) above.

[0232] 22) An information storage medium according to one aspect of the present invention is a computer-readable information storage medium having the program according to any one of the aspects 1) to 19) recorded thereon, thereby achieving the same effects as those of the aspects 1) to 19) above.

[0233] 23) A control method for a display control device (20) according to one aspect of the present invention is a method for controlling a display control device (20) that executes control for displaying a VR image representing a field of view from a virtual viewpoint (P) in a virtual reality (VR) space (V) on a display unit (11) of a head-mounted display (10) as a stereoscopic image utilizing binocular parallax, the method including: a line-of-sight direction specifying step (S106) for specifying a line-of-sight direction (S) that is assumed to be the direction of a user's line of sight with respect to the VR space; and an image generating step (S108 to S114) for arranging a plurality of real-life images captured by a plurality of cameras each having a different imaging direction for capturing a real space in the VR space so as to generate an overlapping area (AO) in which part of the field-of-view areas (A1, A2) of adjacent real-life images overlap each other (S102), and generating the VR image according to the line-of-sight direction, the image generating step including a step (S108 to S112) for dynamically switching the real-life image to be displayed in the overlapping area among the adjacent real-life images based on the line-of-sight direction. This achieves the same effect as the aspect described in 1) above. [Explanation of symbols]

[0234] 1...image display system, 10...HMD, 11...61...display unit, 12...62...sensor, 13...processor, 14...storage device, 20...display control device, 21...63...processor, 22...64...storage device, 23...65...operation unit, 24...66...communication unit, 30...control unit, 31...gaze direction determination unit, 32...image generation unit, 321...image arrangement unit, 322...switching unit, 33...boundary line display unit, 34...imaging direction determination unit, 60...HMD (information processing device), 200...camera, A1...A2...view area, AO...overlap area, RD...reference direction, V...VR space, S...gaze direction, BL1...BL2...boundary line, GD1...GD2...gradation unit

Claims

1. A computer that executes control for displaying a virtual reality (VR) image, which represents a field of view from a virtual viewpoint in a VR space, on a display unit of a head mounted display as a stereoscopic image utilizing binocular parallax, A gaze direction specification unit that specifies a gaze direction as a direction of a user's gaze with respect to the VR space; an image generating unit that arranges a plurality of real-life images captured by a plurality of cameras each having a different imaging direction for capturing an image of a real space in the VR space such that an overlapping area is generated in which a part of the visual field area of ​​adjacent real-life images overlaps with another, and generates the VR image according to the visual line direction; an imaging direction specification unit that specifies a change in imaging direction of the plurality of cameras that captured the plurality of real-life images; and make it work. the image generation unit dynamically switches between the adjacent real-life images to be displayed in the overlapping area based on the line-of-sight direction; The image generating unit changes a direction of a visual field of the entirety of the plurality of real-life images arranged in the VR space in response to a change in the imaging direction specified by the imaging direction specifying unit. program.

2. The image generation unit sets a reference direction in a predetermined direction from the virtual viewpoint, and identifies the real-life image to be displayed in the overlapping area based on the line-of-sight direction with respect to the reference direction. The program according to claim 1.

3. The image generating unit sets the reference direction to a direction from the virtual viewpoint to a predetermined position within the overlapping area. The program according to claim 2.

4. The image generating unit changes the reference direction according to the real-life image to be displayed in the overlapping area at a timing when the real-life image to be displayed in the overlapping area is switched. The program according to claim 2.

5. The gaze direction specification unit acquires detection information regarding an orientation of the head mounted display, and specifies the gaze direction based on the detection information. The program according to claim 1.

6. The gaze direction identification unit acquires detection information regarding the user's gaze, and identifies the gaze direction based on the detection information. The program according to claim 1.

7. and causing the computer to function as a boundary display unit that displays a boundary line visible to the user at the area boundary between the adjacent real-life images. The program according to claim 1.

8. The boundary display unit displays additional information on or near the boundary to indicate which side of the boundary is the overlapping area. The program according to claim 7.

9. When switching the real-life images to be displayed in the overlapping area, the image generating unit gradually switches from one real-life image to the other real-life image as time passes. The program according to claim 7.

10. When switching the real-life image to be displayed in the overlapping area, the image generating unit gradually switches from one real-life image to the other real-life image while gradually moving the boundary line toward a position to be displayed after the switching is completed as time passes. The program according to claim 9.

11. The image generating unit provides a predetermined waiting period from when it is determined that a switching process of the real-life image to be displayed in the overlapping area should be performed based on the line of sight direction until when the switching process is started. The program according to claim 1.

12. The image generation unit changes the arrangement state of the plurality of real-life images in the VR space during display control of the VR image on the display unit.

12. The program according to claim 1.

13. A display control device that executes control for displaying a virtual reality (VR) image, which represents a field of view from a virtual viewpoint in a VR space, on a display unit of a head mounted display as a stereoscopic image utilizing binocular parallax, comprising: A gaze direction specification unit that specifies a gaze direction as a direction of a user's gaze with respect to the VR space; an image generating unit that arranges a plurality of real-life images captured by a plurality of cameras each having a different imaging direction for capturing an image of a real space in the VR space such that an overlapping area is generated in which a part of the visual field area of ​​adjacent real-life images overlaps with another, and generates the VR image according to the visual line direction; an imaging direction specification unit that specifies a change in imaging direction of the plurality of cameras that captured the plurality of real-life images; Including, the image generation unit dynamically switches between the adjacent real-life images to be displayed in the overlapping area based on the line-of-sight direction; The image generating unit changes a direction of a visual field of the entirety of the plurality of real-life images arranged in the VR space in response to a change in the imaging direction specified by the imaging direction specifying unit. Display control device.

14. An image display system including: a head mounted display that displays, on a display unit, a virtual reality (VR) image that represents a field of view from a virtual viewpoint in a VR space as a stereoscopic image utilizing binocular parallax; and a display control device that executes control for displaying the VR image on the display unit, The display control device includes: A gaze direction specification unit that specifies a gaze direction as a direction of a user's gaze with respect to the VR space; an image generating unit that arranges a plurality of real-life images captured by a plurality of cameras each having a different imaging direction for capturing an image of a real space in the VR space such that an overlapping area is generated in which a part of the visual field area of ​​adjacent real-life images overlaps with another, and generates the VR image according to the visual line direction; an imaging direction specification unit that specifies a change in imaging direction of the plurality of cameras that captured the plurality of real-life images; Including, the image generation unit dynamically switches between the adjacent real-life images to be displayed in the overlapping area based on the line-of-sight direction; The image generating unit changes a direction of a visual field of the entirety of the plurality of real-life images arranged in the VR space in response to a change in the imaging direction specified by the imaging direction specifying unit. Image display system.

Images