Program, display control device, image display system, and recording medium

Abstract

The line-of-sight direction determination unit determines a line-of-sight direction, which is the direction of the line of sight of the user with respect to the VR space. The image generation unit generates a VR image corresponding to the line-of-sight direction by disposing, in a VR space, a plurality of live-shot images captured by a plurality of cameras having different imaging directions for imaging an actual space so as to generate an overlapping region in which parts of the field-of-view regions of the adjacent live-shot images overlap each other, the plurality of live-shot images being captured by the plurality of cameras, and the VR image generating unit generates a VR image corresponding to the line-of-sight direction. Furthermore, the image generation unit dynamically switches, in accordance with the line-of-sight direction, the live image displayed in the overlapping region among the adjacent live images.

Description

Technical Field

[0001] This application claims priority based on Japanese Patent Application No. 2022-188764 filed with the Japan Patent Office on November 25, 2022, and the entire content thereof is incorporated herein by reference.

[0002] The present invention relates to a program, a display control device, an image display system, and a recording medium. Background Art

[0003] Conventionally, a head-mounted display (HMD) that displays a stereoscopically viewable virtual reality (VR) image using binocular parallax has been known. In this HMD, it is possible to display a live-action image (still image or moving image) obtained by photographing a VR space or an actual space generated by a computer. In the case of a live-action image, for example, even with a single stereoscopic camera using a wide-angle lens, a wide-angle field-of-view image with a certain degree of field-of-view angle can be obtained, but it is difficult to perform stereoscopic viewing in a direction significantly away from the optical axis direction of the camera. In order to reduce this problem of difficult stereoscopic viewing, there is also a method of performing composite processing on a plurality of images captured by dividing regions using a plurality of cameras to obtain a wide-angle field-of-view image (for example, Patent Document 1).

[0004] Prior Art Documents

[0005] Patent Documents

[0006] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2011-227306 Summary of the Invention

[0007] Problems to be Solved by the Invention

[0008] Multiple cameras each have a volume, and it is physically impossible to arrange multiple cameras in such a way that their imaging centers converge at a single point. Therefore, the images captured by multiple cameras are misaligned at the regional boundaries between adjacent images, resulting in seams (such as extreme differences in shading between pixels). Thus, in the case of shooting with multiple cameras divided into regions, when finally forming a single image data, it is necessary to perform stitching processing to join the images captured by each camera. This stitching processing can also be automatically performed by dedicated software, but it requires laborious effort to form a more natural image without visible seams. Here, the degree of misalignment between adjacent images depends on the distance between the camera and the subject. The closer the subject is to the camera, the greater the degree of misalignment and the more difficult the stitching processing becomes. In particular, when the distance between the camera and the subject is below a certain level (such as below 2 m, etc.), the above-mentioned degree of misalignment also becomes quite large, making stitching difficult. Therefore, in the case of the prior art of stitching and synthesizing multiple images captured by multiple cameras, it is substantially impossible to generate a VR image including close-range objects at a distance below a certain level from the camera and display it on an HMD with less discomfort.

[0009] Therefore, one of the objectives of the present invention is to achieve display control of a VR image with less discomfort even when including close-range objects in the case of displaying a VR image on an HMD using multiple images captured by multiple cameras.

[0010] Means for Solving the Problem

[0011] A program according to one aspect of the present invention causes a computer that executes control to function as the following components. The control is for displaying a VR image representing a field of view from a virtual viewpoint in a virtual reality (VR) space as a stereoscopic image utilizing binocular parallax on a display unit of a head-mounted display. The functions include: a line-of-sight direction determination unit that determines a line-of-sight direction, which is set as the direction of the user's line of sight relative to the VR space; and an image generation unit that arranges multiple real-shot images captured by multiple cameras with different shooting directions of the actual space in the VR space in such a way that a part of the field-of-view regions of adjacent real-shot images overlaps to generate an overlapping region, and generates the VR image corresponding to the line-of-sight direction. The image generation unit dynamically switches the real-shot image displayed in the overlapping region among adjacent real-shot images according to the line-of-sight direction.

[0012] Another mode of the display control device of the present invention performs the following control for displaying, as a stereoscopic image utilizing binocular parallax, a VR image representing a field of view from a virtual viewpoint in a virtual reality (VR) space on a display unit of a head-mounted display. The display control device includes: a line-of-sight direction determination unit that determines a line-of-sight direction, which is set as the direction of the user's line of sight relative to the VR space; and an image generation unit that arranges a plurality of actual captured images captured by a plurality of cameras with different shooting directions in the actual space in the VR space in such a way that a part of the field-of-view regions of adjacent actual captured images overlap each other to generate an overlapping region, and generates the VR image corresponding to the line-of-sight direction. The image generation unit dynamically switches the actual captured images displayed in the overlapping region among the adjacent actual captured images according to the line-of-sight direction.

[0013] Another mode of the image display system of the present invention includes: a head-mounted display that displays, as a stereoscopic image utilizing binocular parallax, a VR image representing a field of view from a virtual viewpoint in a virtual reality (VR) space on a display unit; and a display control device that performs control for displaying the VR image on the display unit. The display control device includes: a line-of-sight direction determination unit that determines a line-of-sight direction, which is set as the direction of the user's line of sight relative to the VR space; and an image generation unit that arranges a plurality of actual captured images captured by a plurality of cameras with different shooting directions in the actual space in the VR space in such a way that a part of the field-of-view regions of adjacent actual captured images overlap each other to generate an overlapping region, and generates the VR image corresponding to the line-of-sight direction. The image generation unit dynamically switches the actual captured images displayed in the overlapping region among the adjacent actual captured images according to the line-of-sight direction. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a schematic block diagram showing an example of the hardware configuration of an image display system according to an embodiment of the present invention.

[0015] Figure 2 is a diagram showing the definition of directions in the VR space.

[0016] Figure 3 is a diagram for explaining a VR image representing a field of view from a virtual viewpoint in the VR space.

[0017] Figure 4 is a diagram for explaining the problems of actual captured images captured by a single stereoscopic camera.

[0018] Figure 5 is a diagram showing an example of shooting using a plurality of stereoscopic cameras.

[0019] Figure 6 This is a diagram for explaining an example of misalignment occurring between images captured by two adjacent cameras respectively.

[0020] Figure 7 This is a diagram showing an example of dividing an area into two parts for shooting by two cameras with different shooting directions.

[0021] Figure 8 This is a diagram schematically showing an example of the field-of-view areas of actual shooting images captured by two cameras with different shooting directions respectively.

[0022] Figure 9 This is a diagram schematically showing an example of being arranged in a VR space in such a way as to generate an overlapping area where a part of the field-of-view areas of adjacent actual shooting images overlaps each other.

[0023] Figure 10 This is a diagram showing an example of displaying a first image in the overlapping area in the VR space.

[0024] Figure 11 This is a diagram showing an example of displaying a second image in the overlapping area in the VR space.

[0025] Figure 12 This is a diagram showing an example of setting a reference direction outside the overlapping area.

[0026] Figure 13 This is a diagram showing an example of dynamically switching the actual shooting image displayed in the overlapping area based on the current area boundary of adjacent actual shooting images.

[0027] Figure 14 This is a schematic functional block diagram showing an example of the functional structure of a display control device.

[0028] Figure 15 This is a flowchart showing an example of the processing of a display control device.

[0029] Figure 16 This is a diagram showing an example of changing the reference direction.

[0030] Figure 17 This is a flowchart showing an example of the processing of changing the reference direction

[0031] Figure 18 This is a diagram showing an example of a screen displayed on the display unit of an HMD.

[0032] Figure 19 This is a diagram showing an example of a screen displayed on the display unit of an HMD.

[0033] Figure 20 This is a schematic functional block diagram showing an example of the functional structure of a display control device.

[0034] Figure 21 This is a diagram showing an example of a screen in which a gradient section is displayed together with a boundary line.

[0035] Figure 22 This is a diagram showing an example of a screen in which a gradient section is displayed together with a boundary line.

[0036] Figure 23 This is a flowchart showing an example of a process for displaying a boundary line.

[0037] Figure 24 This is a flowchart showing an example of a process for sequentially switching live-action images.

[0038] Figure 25 This is a diagram showing an example of a screen in which a boundary line moves.

[0039] Figure 26 This is a diagram schematically showing an example of arranging the field-of-view regions of the left image, the front image, and the right image captured by three cameras in a VR space.

[0040] Figure 27 This is a flowchart showing an example of a process for switching live-action images in an overlapping region.

[0041] Figure 28 This is a diagram explaining an example of changing the configuration state of an image in a VR space according to a scene.

[0042] Figure 29 This is a flowchart showing an example of a process for changing the configuration state of an image in a VR space.

[0043] Figure 30 This is a schematic illustration showing Figure 26 an example of the field-of-view region of a VR space when the camera rotates to the right.

[0044] Figure 31 This is a diagram showing an example of the change in the field-of-view region of a VR space without using rotation information of the shooting direction.

[0045] Figure 32 This is a diagram showing an example of the change in the field-of-view region of a VR space when using rotation information of the shooting direction.

[0046] Figure 33 This is a flowchart showing an example of a process for changing the direction of the entire field-of-view region of a VR space.

[0047] Figure 34 This is a diagram showing an example of an HMD.

[0048] Figure 35This is a schematic block diagram showing an example of the hardware configuration of a standalone HMD or an information processing device used as an HMD. Detailed implementation mode

[0049] Hereinafter, an example of an embodiment of the present invention will be described with reference to the accompanying drawings.

[0050] [1. Example of the structure of the image display system]

[0051] Figure 1 This is a schematic block diagram showing an example of the hardware configuration of the image display system 1 according to the embodiment of the present invention. The image display system 1 includes: an HMD 10; and a display control device 20 that performs display control for the HMD 10.

[0052] The HMD 10 is worn on the user's head (refer to Figure 2 ), and is capable of displaying a stereoscopically viewable wide field-of-view image (static image or dynamic image) utilizing binocular parallax. In addition, the HMD 10 is equipped with sensors 12 such as a gyro sensor to detect the movement and inclination of the HMD 10, detect changes in the movement and inclination of the head of the user wearing the HMD 10, and display a VR image representing the field of view in the VR space according to the change. For example, if the user's head turns to the right, the VR image displayed on the HMD 10 changes to a VR image corresponding to the field of view in the right direction in the VR space, and if it turns upward, it changes to a VR image corresponding to the field of view in the upward direction in the VR space, so that the user can experience a sense of immersion as if on the spot.

[0053] The HMD 10 mainly includes a display unit 11, sensors 12, a processor 13, a storage device 14, etc.

[0054] The display unit 11 is a display for displaying various information such as VR images and texts. The display unit 11 can be, for example, a virtual image projection type display that forms a virtual image by using a semi-reflective mirror or the like. In addition, the display unit 11 can also be, for example, a retinal projection type display that directly forms a VR image on the retina by using the lens of the user's eye. The display unit 11 displays a stereoscopic image (right-eye image and left-eye image) utilizing binocular parallax as a VR image representing the field of view from the virtual viewpoint in the VR space.

[0055] The sensor 12 detects posture information such as the rotation angle and tilt of the HMD 10. Based on the output of the sensor 12, detection information related to the orientation of the HMD 10 can be obtained. For example, the sensor 12 is an angular velocity sensor (gyro sensor) that detects the angular velocity of an object. Additionally, the sensor 12 can be a sensor that detects a change in direction or a sensor that detects the direction itself. For example, the sensor 12 is not limited to a gyro sensor and can also be an acceleration sensor, an angular acceleration sensor, an inclination sensor, a geomagnetic sensor, etc., and they can be combined to achieve this.

[0056] The processor 13 functions as a control center for controlling each part of the HMD 10. For example, the processor 13 is a CPU (Central Processing Unit). The processor 13 can also be configured to include hardware such as a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), or an FPGA (Field Programmable Gate Array) in addition to or instead of the CPU. The processor 13, for example, performs control to provide the VR image signal received from the display control device 20 to the display unit 11 via the interface.

[0057] The storage device 14 stores programs executed by the processor 13 or temporarily stores data and parameters processed by the processor 13. The storage device 14 includes, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), a VRAM (Video Random Access Memory), an auxiliary storage device, etc. As the auxiliary storage device, a non-volatile semiconductor memory, a hard disk drive, a solid state drive, etc. can be used.

[0058] In addition, the HMD 10 can have a network adapter or the like and can input VR images via the network. In addition to this, the HMD 10 can also have an audio output unit, a GPS (Global Positioning System) receiver unit, etc.

[0059] The display control device 20 is communicably connected to the HMD 10 in a wireless or wired manner and performs display control for the HMD 10. The display control device 20 is, for example, a stationary or portable game console. The display control device 20 may also be a commercial (business-use) game console installed in a game facility or the like. Alternatively, the display control device 20 may be a personal computer, a tablet computer, a smart phone, a mobile phone terminal, a PHS (Personal Handy-phone System) terminal, a portable information terminal (PDA (Personal Digital Assistant)), a multifunctional television receiver with an information processing function (so-called smart TV), or the like.

[0060] The display control device 20 mainly includes a processor 21, a storage device 22, an operation unit 23, and a communication unit 24, which are interconnected via buses including an address bus, a data bus, and a control bus, etc. Additionally, as needed, an interface circuit, an image processing unit, an audio processing unit, or the like is interposed between the bus and each structural element, and the illustration thereof is omitted here.

[0061] The processor 21 interprets and executes the commands of the program and controls the entire display control device 20. For example, the processor 21 is a CPU. The processor 21 may also be configured to include hardware such as a GPU, a DSP, or an FPGA in addition to or instead of the CPU. The storage device 22 includes, for example, a ROM, a RAM, a VRAM, an auxiliary storage device, etc. The ROM stores programs, data, etc. required for basic operation control of the display control device 20. The RAM or VRAM stores various programs and data to ensure a working area for the processor 21. The auxiliary storage device stores programs and various data, etc., and for example, a non-volatile semiconductor memory, a hard disk drive, a solid state drive, etc. may be used.

[0062] In addition, the display control device 20 may also have a recording medium drive. As the recording medium drive, for example, a DVD-ROM drive, a CD-ROM drive, a hard disk drive, an optical disc drive, a floppy disk drive, a silicon disk drive, a cartridge medium reader, etc. can be used. In this case, as the recording medium, a DVD-ROM, a CD-ROM, a hard disk, an optical disc, a floppy disk, a semiconductor memory, etc. can be used. The recording medium drive reads out image data, voice data, and program data from the recording medium and provides the read data to the RAM, etc. of the storage device 22 via a decoder.

[0063] For example, image data including actual images pre-shot for VR images, programs such as games including the image data are stored in the storage device 22 (auxiliary storage device, etc.) or read out from the recording medium drive.

[0064] The operation unit 23 is used for a user to input various operation commands into the display control device 20. For example, the user performs an operation for viewing a VR image or an operation for a game including a VR image. As an example of the operation unit 23, a position input unit having a touch interface (structural elements of a touch panel, etc.), physical buttons, a controller, an analog joystick, a keyboard, a pointing device, etc. can be cited. In addition, the operation unit 23 can also be configured to perform voice input operations by recognizing voices input from a voice input unit such as a microphone. In addition, the operation unit 23 can also be used for operations by the user's gestures.

[0065] The communication unit 24 has a communication interface (not shown) and has a communication control function for data communication when executing a game or the like. Here, the communication control function for data communication includes, for example, an Internet connection function, a wireless LAN (Local Area Network) connection function, a short-range wireless communication function using a specified frequency band (for example, a 2.4 GHz band). The communication unit 24 sends a connection signal for connecting the display control device 20 to a network according to a command from the processor 21, and receives information sent from the communication partner side and provides it to the processor 21.

[0066] As described later, in the case where the image display system 1 or the display control device 20 of the present embodiment displays a VR image on the HMD 10, even without performing a stitching process, it is possible to reduce the discomfort of the seams (misalignment) between a plurality of captured real images captured by a plurality of cameras. Therefore, it is also possible to receive a remote live video published via a network through the communication unit 24 and display the live video on the HMD 10 as a VR image almost in real time.

[0067] In addition, a tracking unit for detecting the position of the HMD 10 (or the position of the head of the user wearing the HMD 10) may be connected to the display control device 20. For example, the HMD 10 has a plurality of light sources (LEDs, etc.) for tracking, and the tracking unit has a photographing unit for photographing the HMD 10. The tracking unit fixed at a specified position determines the position of the HMD 10 according to the positions of the plurality of light source units captured in the image captured by the photographing unit. In addition, the display control device 20 may also have a display unit such as a liquid crystal display or an organic EL (Electro-Luminescence) display.

[0068] [2. Regarding the direction of the VR space]

[0069] Figure 2This is a diagram showing the definition of the directions in the VR space of the present embodiment. In the present embodiment, the vertical direction, which is the direction in which a user wearing the HMD 10 that displays VR images in the VR space stands upright, is set as the Z-axis. In addition, the direction of the user's line of sight facing forward in the horizontal direction orthogonal to the Z-axis (the reference line-of-sight direction with respect to the VR space) is set as the X-axis, and the axis in the horizontal direction orthogonal to the Z-axis and the X-axis is set as the Y-axis.

[0070] For example, when the image display system 1 is started, the line-of-sight direction of the HMD 10 (the direction orthogonal to the display surface of the display unit 11 of the HMD 10) is set as the reference line-of-sight direction with respect to the VR space. In addition, for example, by performing a specified operation by a user wearing the HMD 10 and facing forward, the line-of-sight direction with respect to the reference of the VR space can also be adjusted.

[0071] Here, a change in the rotational direction about the Z-axis is referred to as a change in the yaw direction (left-right direction), a change in the rotational direction about the Y-axis is referred to as a change in the pitch direction (up-down direction), and a change in the rotational direction about the X-axis is referred to as a change in the roll direction. For example, the above-described sensor 12 of the HMD 10 detects the angular velocity or angular acceleration of the rotational direction of each axis (yaw direction, pitch direction, and roll direction). Additionally, a change in the yaw direction is sometimes referred to as a change in the left-right direction, and a change in the pitch direction is referred to as a change in the up-down direction.

[0072] Figure 3 This is a diagram for explaining a VR image (field-of-view image) representing the field of view from the virtual viewpoint in the VR space of the present embodiment. In this diagram, the virtual viewpoint P (the user's virtual viewpoint) in the VR space V is set as the intersection point (origin) of the X-axis, Y-axis, and Z-axis. For example, assume that the user is facing forward and the direction of the user's line of sight with respect to the VR space V is on the X-axis (the reference line-of-sight direction). In this case, the range of the VR image representing the field of view from the virtual viewpoint P in the VR space V is determined by the yaw angle α (the interior angle between the dashed line a and the dashed line b, and the interior angle between the dashed line c and the dashed line d) centered on the reference line-of-sight direction (X-axis direction) and the pitch angle β (the interior angle between the dashed line a and the dashed line d, and the interior angle between the dashed line b and the dashed line c). Here, the yaw angle α is the horizontal field-of-view angle, and the pitch angle β is the vertical field-of-view angle, which are preset as the field-of-view angles of the VR images displayed on the HMD 10 in the image display system 1.

[0073] For example, when the head of a user wearing the HMD 10 changes in the pitch direction or the yaw direction, the change in the orientation / pose of the HMD 10 (the change in the line-of-sight direction of the HMD 10) is detected by the sensor 12 or the like. The processor 21 of the display control device 20 determines, based on the detection information of the sensor 12 or the like, that the line-of-sight direction with respect to the VR space V has changed from the X-axis direction (the reference line-of-sight direction) to the pitch direction or the yaw direction. The range of the VR image displayed on the display unit 11 is changed according to the change in the line-of-sight direction with respect to the VR space V. Similarly, when the head of a user wearing the HMD 10 changes in the roll direction, the change is detected by the sensor 12 or the like, the line-of-sight direction remains in the X-axis direction, and the range of the VR image displayed on the display unit 11 rotates in the roll direction. In this way, the range of the VR image displayed on the display unit 11 is changed according to the orientation (pose) of the HMD 10.

[0074] In addition, various display objects other than real shots, such as various objects, lines, symbols, characters, etc., can be arranged in the VR space V as needed. For example, the "boundary line" described later can also be displayed on the display unit 11 overlapping the real-shot VR image.

[0075] In addition, in the case of displaying a stereoscopic image using binocular parallax, there are line-of-sight directions corresponding to the respective virtual viewpoints for the right eye and the left eye, and the right-eye VR image and the left-eye VR image in each line-of-sight direction are displayed on the display unit 11 of the HMD 10.

[0076] [3. Real-shot image captured by a camera]

[0077] In the image display system 1 of the present embodiment, a real-shot image captured by a camera that captures an actual space is used to generate a stereoscopically viewable VR image using binocular parallax and is displayed on the HMD 10. Before explaining the real-shot image used in the image display system 1 of the present embodiment, the problems of a real-shot image captured by one camera and the like are explained.

[0078] Figure 4This is a diagram for explaining the problems of a captured image taken by one camera 100 (so-called stereo camera) having a left-eye camera 101 and a right-eye camera 102. The cameras 100 are located at different positions in the actual space and capture objects A and B that exist at the same distance from the cameras 100. Object A is located in the optical axis direction (≈ the direction of the lens center) XC of the camera 100, and object B is located in the right direction relative to the optical axis direction XC. Compared with the angle (convergence angle) θ1 formed by object A with the left-eye camera 101 and the right-eye camera 102, the angle θ2 formed by object B with the left-eye camera 101 and the right-eye camera 102 becomes smaller regardless of whether objects A and B are located at equal distances from the camera 100. Therefore, there is a problem that in the direction where the angle deviates significantly from the optical axis direction XC of the camera 100, it becomes difficult to recognize the sense of distance and difficult to perform stereo observation of the image.

[0079] To solve the above problems, effectively, as Figure 5 illustrated, it is possible to divide the shooting area and shoot by using a plurality of cameras 100 having different shooting directions for the actual space. In Figure 5 the example, three cameras 100 divide the shooting area into three areas: left, front, and right, and shoot. By dividing the entire shooting area into a plurality of areas in this way, there is an optical axis direction XC of the camera in each shooting area, and it is possible to reduce the direction in which the angle deviates significantly from the optical axis direction XC. Therefore, if a plurality of images taken by a plurality of cameras 100 having different shooting directions are used, it is possible to reduce the problem of an image that is difficult to perform stereo observation.

[0080] In addition, conventionally, a plurality of images taken by a plurality of cameras that divide the area are finally made into one image data. Therefore, it is necessary to perform a stitching process for joining adjacent images taken by each camera. This stitching process can also be automatically performed by dedicated software, but in order to form a more natural image with unknown seams, it requires laborious work. One of the factors that require effort in the stitching process is the misalignment of the positions of each camera used during shooting. That is, although a plurality of cameras are arranged in positions close to each other, as long as they shoot the actual space, they cannot be physically arranged at exactly the same position at the same time. Therefore, a misalignment occurs near the area boundary between adjacent images taken by adjacent cameras. The following explains this.

[0081] Figure 6This is a diagram for explaining an example of misalignment occurring between images captured by camera 100a that captures the front area and camera 100b that captures the right area. Here, for the sake of convenience in explanation, the case where the left-eye cameras 101 of two cameras 100a and 100b respectively capture images is illustrated, but the same applies to the case where the right-eye cameras 102 capture images. The objects C and D of the subject are located on the right side of camera 100a and on the left side of camera 100b, and are near the area boundary of the capture areas of cameras 100a and 100b. In addition, object C is located at a position deeper (farther from cameras 100a and 100b) than object D. In this case, when captured by camera 100a that captures the front area, an image is captured in which object D is located slightly to the right of the inner object C. On the other hand, when captured by camera 100b that captures the right area, an image is captured in which object D is located slightly to the left of object C. Thus, although cameras 100a and 100b capture the same objects C and D simultaneously, there is an obvious misalignment between the images captured by the two.

[0082] In the conventional stitching process, in order to make the misalignment between adjacent images as described above into a more natural image, it is necessary to manually process (roughen) the images, which takes effort. The degree of misalignment near the area boundary between the above-mentioned adjacent images depends on the distance between cameras 100a and 100b and the subject. That is, when the subject (such as the above objects C and D) located near the area boundary is at a position far enough from cameras 100a and 100b, the degree of misalignment between adjacent images also becomes smaller, so the conventional stitching process is relatively easy. On the other hand, when the subject located near the area boundary is at a position close to cameras 100a and 100b, the degree of misalignment between adjacent images also relatively increases, and stitching becomes difficult. In particular, when the subject located near the area boundary is at a distance of a certain value or less (for example, 2 m or less) from cameras 100a and 100b, the degree of misalignment between adjacent images also becomes quite large, and the stitching process becomes difficult. Therefore, in the conventional case where multiple images captured by multiple cameras are finally made into one VR image data through stitching, it is substantially impossible to generate VR image data including close-range captured objects at a distance of a certain value or less from the cameras.

[0083] Therefore, in the image display system 1 or the display control device 20 of the present embodiment, for multiple images captured by multiple cameras, instead of performing the conventional stitching process, the display control of the VR image described below is executed.

[0084] [4. Display Control of VR Image]

[0085] The outline of the VR image display control executed in the image display system 1 or the display control device 20 of the present embodiment is as follows. That is, the image display system 1 or the display control device 20 executes the following characteristic processing: A plurality of actual shooting images captured by a plurality of cameras with different shooting directions of the actual space are arranged in the VR space without performing a splicing process. Here, when arranging the plurality of actual shooting images in the VR space, they are arranged in the VR space in such a way that a part of the field-of-view regions of adjacent actual shooting images overlaps with each other to generate an overlapping region. Then, according to the line-of-sight direction with respect to the VR space (a direction that can be determined as the user's line-of-sight direction), the region boundary between adjacent actual shooting images in the overlapping region is dynamically changed. Thereby, the following characteristic display control is achieved: making the image misalignment that easily occurs at the region boundary of adjacent actual shooting images as far away from the line-of-sight direction as possible, and displaying a VR image with reduced unnaturalness caused by the above-mentioned image misalignment on the HMD 10 without performing a splicing process.

[0086] When explaining the VR image display control of the present embodiment, hereinafter, for the sake of simplicity of explanation, as an example of dividing the region into two directions of left and right to expand the horizontal field-of-view angle (that is, an example using two actual shooting images captured by two cameras shooting in the left direction and the right direction respectively) will be described. In addition, regarding the number of divisions of the region, as long as it is two or more, for example, it can be divided into three directions of left, front, and right as described above, or it can be further divided into four directions including the rear. For example, it can also be more finely divided into regions of eight directions. In addition, for example, it can also be divided into two directions of up and down (or three or more directions) to expand the vertical field-of-view angle.

[0087] In addition, hereinafter, for the sake of simplicity of explanation, it is assumed that for the inclination of the HMD 10, the pitch direction (up and down direction) and the roll direction (visual axis rotation direction) are not considered, and only the yaw direction (left and right direction) is considered.

[0088] In addition, when a VR image that can be stereoscopically observed using binocular parallax is displayed on the display unit 11 of the HMD 10, the VR image for the right eye and the VR image for the left eye are respectively displayed, but hereinafter, for the sake of simplicity of explanation, the distinction between the right-eye use and the left-eye use is not made.

[0089] Figure 7FIG. is an example showing that a region with a horizontal field of view angle of 180 degrees is divided into two and photographed by two stereo cameras with different photographing directions. The horizontal field of view angle θP1 of the first camera 100L that photographs the leftward region is 120 degrees, and the horizontal field of view angle θP2 of the second camera 100R that photographs the rightward region is also 120 degrees. The angle formed by the optical axis direction XC1 of the first camera 100L and the optical axis direction XC2 of the second camera 100R is set to 60 degrees, and the first camera 100L and the second camera 100R are arranged close to each other.

[0090] Figure 8 FIG. schematically shows the field-of-view region A1 of the first image photographed by the first camera 100L and the field-of-view region A2 of the second image photographed by the second camera 100R. Actually, in the field-of-view region A1 of the first image and the field-of-view region A2 of the second image, there are respectively for the right eye and the left eye. Hereinafter, it is assumed to be either for the right eye or the left eye. Here, the "field-of-view region" is the region that can be displayed on the display unit 11 of the HMD 10 in the captured image photographed by one camera, and is the region where the captured image is arranged in the VR space. In Figure 8 the example, the field-of-view angle θA1 of the field-of-view region A1 of the first image and the field-of-view angle θA2 of the field-of-view region A2 of the second image are both 120 degrees. Then, as Figure 9 illustrated, the entire 180-degree field-of-view region is composed of adjacent field-of-view regions A1 and A2.

[0091] Figure 9 FIG. schematically shows an example of being arranged in the VR space V in such a way that an overlapping region AO where a part of the field-of-view region A1 of the first image on the left side and a part of the field-of-view region A2 of the second image on the right side overlap is generated. In other words, Figure 9 FIG. schematically shows an XY cross-section of the VR space V in which the first image and the second image are arranged (for example Figure 10 and so on. The same applies to the diagrams of the VR space V shown below). In Figure 9 the example, the entire field-of-view region of the VR space V is 180 degrees, the field-of-view angle θA1 of the field-of-view region A1 of the first image is 120 degrees, and the field-of-view angle θA2 of the field-of-view region A2 of the second image is 120 degrees. Therefore, the field-of-view angle θAO of the overlapping region AO where the two overlap is 60 degrees. The overlapping region AO is the region sandwiched between the region boundaries BD1 and BD2 of the first image and the second image. In Figure 9In the example, the direction passing through the center of the overlapping region AO from the virtual viewpoint P in the VR space V is the direction of the X-axis. Then, either the first image or the second image is selectively displayed in the overlapping region AO. In addition, the field-of-view regions of only the first image (the region between the Y-axis and the region boundary BD2) and only the second image (the region between the Y-axis and the region boundary BD1) both have a field-of-view angle of 60 degrees.

[0092] Figure 10 FIG. is an example showing the first image (the field-of-view region A1 of the first image) displayed in the overlapping region AO in the VR space V. In addition, Figure 11 FIG. is an example showing the second image (the field-of-view region A2 of the second image) displayed in the overlapping region AO in the VR space V. When the first image with the left field-of-view region A1 is selectively displayed in the overlapping region AO, as Figure 10 illustrated, the boundary between the first image and the second image is the region boundary BD1 at the right end of the overlapping region AO. On the other hand, when the second image with the right field-of-view region A2 is selectively displayed in the overlapping region AO, as Figure 11 illustrated, the boundary between the first image and the second image is the region boundary BD2 at the left end of the overlapping region AO. Thus, by selectively switching the real-shot images (the first image, the second image) displayed in the overlapping region AO, the region boundaries (BD1, BD2) between adjacent real-shot images change.

[0093] Based on the line-of-sight direction S, it is determined which of the first image or the second image is to be displayed in the overlapping region AO. The line-of-sight direction S can be determined by detection information such as that from the sensor 12 of the HMD 10 (i.e., based on the orientation of the HMD 10). Therefore, the following example is described: regarding the optical axis direction of the HMD 10 as the line-of-sight direction S, and switching the real-shot image displayed in the overlapping region AO according to the yaw direction orientation of the HMD 10.

[0094] In addition, in Figure 10 and Figure 11 , in order to show the relationship between the orientation of the HMD 10 (the optical axis direction of the HMD 10) and the display switching of the overlapping region AO in the VR space V, an image of the orientation of the HMD 10 is displayed near the virtual viewpoint P (the same applies to the diagrams showing the VR space V below).

[0095] In Figure 10 and Figure 11 , an example is shown where the "reference direction RD", which is the criterion for switching the real-shot image displayed in the overlapping region AO, is set as the direction passing through the center of the overlapping region AO from the virtual viewpoint P. In Figure 10In the example, the line-of-sight direction S (the line-of-sight direction of the HMD 10) is located on the left side (i.e., the first-image side) with respect to the reference direction RD. In this case, the first image of the left field-of-view region A1 is displayed in the overlapping region AO. Further, in this case, the boundary between the field-of-view region A1 of the first image and the field-of-view region A2 of the second image is the region boundary BD1, and thus the region boundary BD1 is away from the line-of-sight direction S.

[0096] On the other hand, in Figure 11 In the example, the line-of-sight direction S (the line-of-sight direction of the HMD 10) is located on the right side (i.e., the second-image side) with respect to the reference direction RD. In this case, the second image of the right field-of-view region A2 is displayed in the overlapping region AO. Further, in this case, the boundary between the field-of-view region A1 of the first image and the field-of-view region A2 of the second image is the region boundary BD2, and thus the region boundary BD2 is away from the line-of-sight direction S.

[0097] As in the above examples, a specific direction (in the above case, the direction passing through the center of the overlapping region AO from the virtual viewpoint P) is taken as the reference direction RD, and the orientation of the HMD 10 (the line-of-sight direction of the HMD 10) is regarded as the line-of-sight direction S, and the live-action images (the first image or the second image) displayed in the overlapping region AO are dynamically switched. Thereby, it is possible to move the seam (misalignment) of the image that is likely to occur at the region boundary (BD1 or BD2) of the adjacent live-action images away from the direction of the user's line of sight, and the discomfort caused by the seam can be reduced. Therefore, even if a close-up object exists at or near the region boundary, a VR image with less discomfort for the user can be displayed. Further, since no stitching process is required, the effort for producing the VR image can also be reduced.

[0098] (Regarding the setting of the reference direction RD)

[0099] In addition, in the above, an example in which the reference direction RD is set to the direction passing through the center of the overlapping region AO from the virtual viewpoint P is shown, but it is not limited thereto. The reference direction RD may be set to any direction from the virtual viewpoint P as long as it is in the field-of-view regions of the first image and the second image configured in the VR space V.

[0100] For example, as Figure 12 shown, it is also possible to set the reference direction RD to the direction of a position outside the overlapping region AO from the virtual viewpoint P. In Figure 12 , an example in which the reference direction RD is set to the field-of-view region A2 of the second image that does not include the overlapping region AO is shown. In this case, if the line-of-sight direction S exceeds the reference direction RD from the state of (A) in Figure 12 due to the change in the orientation of the HMD 10 in the yaw direction (left-right direction), then as Figure 12As illustrated in (B) thereof, the live-action image displayed in the overlapping region AO also switches from the first image of the field-of-view region A1 to the second image of the field-of-view region A2. As a result, the region boundary of adjacent live-action images switches from the region boundary BD1 to the region boundary BD2, and the region boundary moves away from the line-of-sight direction S. However, in this case, the line-of-sight direction S crosses the region boundary BD1 until the line-of-sight direction S exceeds the reference direction RD in the state of (A) until Figure 12 Therefore, before the live-action image displayed in the overlapping region AO switches from the first image to the second image, there is a timing when the region boundary BD1 overlaps with the line-of-sight direction S.

[0101] Therefore, a preferred method is to set the reference direction RD as the direction of a specified position (which can be the center of the overlapping region AO or not the center) within the overlapping region AO starting from the virtual viewpoint P. In this case, by setting the reference direction RD in the overlapping region AO (refer to Figure 10 or Figure 11 ), even if the line-of-sight direction S is directed towards the current region boundary (BD1 or BD2) of the overlapping region AO, since the line-of-sight direction S exceeds the reference direction RD before reaching the region boundary, the live-action image displayed in the overlapping region AO is also switched. As a result, before the line-of-sight direction S reaches the region boundary, the region boundary moves away from the line-of-sight direction S.

[0102] (A method without setting a reference direction)

[0103] Next, a method of dynamically switching the live-action image displayed in the overlapping region AO in adjacent live-action images (the first image and the second image) according to the line-of-sight direction S without setting the reference direction RD is shown. In this method, as Figure 13 is illustrated, based on the current region boundary (BD1 or BD2) of adjacent live-action images, the live-action image displayed in the overlapping region AO is dynamically switched according to the line-of-sight direction S.

[0104] For example, as shown in (A) of Figure 13 , it is assumed that the first image of the left field-of-view region A1 is displayed in the overlapping region AO and the line-of-sight direction S is on the X-axis. In this case, the switching reference of the live-action image is the current region boundary BD1. From here, when the HMD 10 rotates to the right direction, as Figure 13 is illustrated in (B), when the line-of-sight direction S (the optical axis of the HMD 10) exceeds the region boundary BD1, the live-action image displayed in the overlapping region AO switches from the first image to the second image of the right field-of-view region A2. As a result, the switching reference of the live-action image changes to the current region boundary BD2. From here, when the HMD 10 rotates to the left direction, as Figure 13As exemplified by (C) therein, even when the line-of-sight direction S returns to the X-axis, since it does not exceed the current region boundary BD2, the real-shot image displayed in the overlapping region AO is not switched. From here on, the HMD 10 further rotates in the left direction. As Figure 13 exemplified by (D) therein, when the line-of-sight direction S exceeds the region boundary BD2, the real-shot image displayed in the overlapping region AO is switched from the second image to the first image.

[0105] In this way, if the line-of-sight direction S is located on the side (left side) of the first image with respect to the current region boundary (BD1 or BD2) of the adjacent real-shot images (the first image and the second image), the first image is dynamically switched to be displayed in the overlapping region AO. If the line-of-sight direction S is located on the side (right side) of the second image, the second image is dynamically switched to be displayed in the overlapping region AO.

[0106] In addition, as a modification example, the real-shot image displayed in the overlapping region AO can also be dynamically switched in the following manner. That is, when the line-of-sight direction S approaches more than the specified current region boundary (BD1 or BD2) of the adjacent real-shot images (for example, when the angle formed by the line-of-sight direction S and the current region boundary is below the specified value), the real-shot image currently displayed in the overlapping region AO can also be switched to another real-shot image. In this case, before the line-of-sight direction S reaches the region boundary, the real-shot image displayed in the overlapping region AO is switched, and the region boundary moves away from the line-of-sight direction S.

[0107] In addition, in the description using Figures 9 - 13 , the display control for expanding the horizontal field of view by dividing the horizontal direction into multiple regions has been described. However, in the case of display control for expanding the vertical field of view by dividing the vertical direction into multiple regions, according to the orientation of the HMD 10 in the pitch direction (up and down direction), the same switching process of the image in the overlapping region as described above can be performed.

[0108] [5. Functional Structure of Display Control Device]

[0109] Figure 14 is a schematic functional block diagram showing an example of the functional structure of the display control device 20. The display control device 20 executes control to display a VR image representing the field of view from the virtual viewpoint in the VR space as a stereoscopic image using binocular parallax on the display unit 11 of the HMD 10. As Figure 14 shown, the display control device 20 includes a control unit 30. This control unit 30 is realized, for example, by the processor 21 executing a program stored in the storage device 22. The control unit 30 includes a line-of-sight direction determination unit 31 and an image generation unit 32.

[0110] The line-of-sight direction determination unit 31 has a function of determining the line-of-sight direction, which is set as the direction of the user's line of sight relative to the VR space. Here, the "line-of-sight direction" is a direction determined as the direction of the line of sight of the user wearing the HMD 10 relative to the VR space. The "line-of-sight direction" can be estimated to be approximately the direction of the user's line of sight based on the detection information of the sensor 12 etc. of the HMD 10, for example, and it may be consistent with the actual line-of-sight direction of the user wearing the HMD 10, or may be misaligned with the actual line-of-sight direction of the user.

[0111] For example, the line-of-sight direction determination unit 31 can obtain detection information related to the orientation of the HMD 10, and determine the line-of-sight direction based on this detection information. Here, the "detection information related to the orientation of the HMD 10" refers to the detection information related to the orientation of the HMD 10 that changes due to the user changing the orientation of the head while the HMD 10 is worn on the user. The above-mentioned "detection information" can be obtained by a detection unit such as a sensor that detects the direction or direction change of the HMD 10. For example, the detection result of the sensor 12 (angular velocity sensor, acceleration sensor, geomagnetic sensor, etc.) built in the HMD 10 is an example of the "detection information related to the orientation of the HMD 10". In addition, the measurement result of a tracking system that photographs the HMD 10 by a photographing unit provided outside the HMD 10 and measures the position and direction of the HMD 10 is an example of the "detection information related to the orientation of the HMD 10". In addition, the measurement result of a tracking system that analyzes the surrounding objects photographed by the photographing unit mounted on the HMD 10 itself and measures the position and direction of the HMD 10 is an example of the "detection information related to the orientation of the HMD 10".

[0112] In addition, the line-of-sight direction determination unit 31 can obtain detection information related to the user's line of sight, and determine the above-mentioned line-of-sight direction based on this detection information. Here, the "detection information related to the user's line of sight" refers to the information obtained by detecting the direction, movement, etc. of the user's line of sight while the HMD 10 is worn on the user. For example, the measurement result of an eye movement tracking system that tracks the direction and movement of the line of sight based on the position of the user's eyeballs is an example of the "detection information related to the user's line of sight".

[0113] The image generation unit 32 has the following function: arranging a plurality of actual captured images captured by a plurality of cameras with different shooting directions of the actual space in the VR space in such a way that a part of the field-of-view regions of adjacent actual captured images overlap each other, and generating a VR image corresponding to the above-mentioned line-of-sight direction.

[0114] Here, the above "camera" refers to a real-time camera that captures the actual space. Additionally, the "camera" itself is not included in the structural elements of the display control device 20. The "camera" includes a left-eye optical system (left-eye camera) that captures an image for the left eye and a right-eye optical system (right-eye camera) that captures an image for the right eye, in order to obtain a stereoscopic image utilizing the binocular parallax of the left and right eyes. For example, like a stereo camera, a single camera can have a left-eye optical system and a right-eye optical system. Additionally, it can also be the case that the left-eye camera and the right-eye camera have different structures. In order to obtain a VR image with a wide viewing angle that can be stereoscopically observed from multiple captured images, multiple cameras are used respectively for the left eye and the right eye. For example, in the case of obtaining a wide-viewing-angle image with an expanded viewing angle in the horizontal direction, multiple cameras can also be arranged in the horizontal direction such that in the actual space, the shooting directions are different from each other. Additionally, for example, in the case of obtaining a wide-viewing-angle image with an expanded viewing angle in the vertical direction, multiple cameras can also be arranged in the vertical direction such that in the actual space, the shooting directions are different from each other. As long as the number of cameras is 2 or more respectively for the left eye and the right eye, any number can be set. The viewing angle of each camera is also arbitrary. The viewing angles of multiple cameras can all be the same or can be different.

[0115] Furthermore, the above "shooting direction" refers to the direction corresponding to the orientation of the camera that captures the actual space, and refers to the front direction (the direction indicating the subject side) of the optical axis of the shooting optical system that the camera has.

[0116] Furthermore, the above "adjacent captured images" refer to two captured images captured by two adjacent cameras that have different shooting directions for the actual space and whose shooting ranges partially overlap. Here, the two adjacent cameras are not the left-eye camera and the right-eye camera, but two cameras that both capture left-eye captured images or two cameras that both capture right-eye captured images.

[0117] The "captured image" can be a moving image or a still image. Additionally, the "captured image" can be an image (video) captured in advance, or a live image (live video) captured almost in real time.

[0118] In addition, the above-mentioned "field-of-view region" is the region that can be displayed on the display unit of the head-mounted display in the real-shot image captured by one camera, and is the region where the real-shot image is arranged in the VR space. For example, the field-of-view region of the real-shot image captured by a camera with a horizontal field-of-view angle θH1 (e.g., 120 degrees) and a vertical field-of-view angle θV1 (e.g., 120 degrees) is basically a region with a horizontal field-of-view angle θH2 (=θH1) and a vertical field-of-view angle θV2 (=θV1). Additionally, the field-of-view region of the real-shot image captured by a camera with a horizontal field-of-view angle θH1 and a vertical field-of-view angle θV1 can be narrowed when arranged in the VR space, and set to a horizontal field-of-view angle θH2 (<θH1) and a vertical field-of-view angle θV2 (<θV1). For example, by not using the region near the end of the image that is prone to distortion caused by the aberrations of the optical system, the field-of-view region of the real-shot image can be narrowed when arranged in the VR space as described above.

[0119] In addition, the above-mentioned "overlapping region" refers to the region in the VR space where two adjacent real-shot images are arranged with partial overlap. In Figures 9 - 13 the example, an example is shown where the field-of-view angle of the overlapping region AO is set to 60 degrees, but the overlapping region AO can be narrower than 60 degrees or wider than 60 degrees. The width (field-of-view angle) of the "overlapping region" can be set arbitrarily. As will be described later, in the case where there are multiple overlapping regions in the VR space, the widths (field-of-view angles) of all overlapping regions can be the same, or the width of at least one overlapping region can be different from the widths of other overlapping regions.

[0120] The image generation unit 32 includes an image arrangement unit 321 that arranges a plurality of real-shot images in the VR space in such a way that no overlapping region is formed where a part of the field-of-view regions of adjacent real-shot images overlap. The image arrangement unit 321 performs storage control in which the plurality of real-shot images are respectively stored in a specified storage region (a storage region for storing the plurality of real-shot images arranged in the VR space) of the storage device 22 (VRAM, etc.) used to form the VR space. In Figures 9 - 13 the example, the image arrangement unit 321 stores the data of the first image and the second image in the specified storage region of the storage device 22 used to form the VR space.

[0121] In the present embodiment, configuration information for arranging the plurality of actual shot images together with data of the plurality of actual shot images of the display object in the VR space is stored in the storage device 22 in association with the plurality of actual shot images. Alternatively, the above-described configuration information is also recorded together in a recording medium that records the plurality of images of the display object. Then, when the processor 21 of the display control device 20 reads out the data of the plurality of images of the display object from the recording medium and stores it in the storage device 22, the above-described configuration information is also read out and saved in the storage device 22 for image arrangement processing in the VR space. Alternatively, as will be described later, when a live video is displayed as a VR image on the HMD 10 in substantially real time, the above-described configuration information is also input (received) together with the image of the live video of the display object for image arrangement processing in the VR space.

[0122] As the above-described configuration information, for example, it includes information such as the number of field-of-view regions (number of images) arranged in the VR space, the field-of-view angle of the entire field-of-view region, the field-of-view angle of each field-of-view region, the direction (position in the VR space) of each field-of-view region in the VR space, the number of overlapping regions, the field-of-view angle of each overlapping region, or the direction (position) of each overlapping region in the VR space.

[0123] For example, the above-described configuration information can be recorded in the title part of the actual shot image file. Alternatively, the above-described configuration information can also be recorded as information associated with the actual shot image, as a file different from the file (data) of the actual shot image.

[0124] In addition, if normalization (standardization) is performed in advance for the display of the VR image (moving image) of the present embodiment and it is premised on using actual shot images that conform to the standard, it is not necessary to save the above-described detailed configuration information in association with the plurality of actual shot images of the display object together with the actual shot images. That is, when the number of field-of-view regions (number of images) arranged in the VR space, the field-of-view angle of the entire field-of-view region, the field-of-view angle of each field-of-view region, the direction (position in the VR space) of each field-of-view region in the VR space, the number of overlapping regions, the field-of-view angle of each overlapping region, or the direction (position) of each overlapping region in the VR space, etc. are predetermined as standards, the image arrangement unit 321 can arrange each image in the VR space based on the information of the standard.

[0125] Furthermore, for example, a plurality of standards can be defined such as a first standard with a field-of-view angle of 180 degrees for the entire field-of-view region, a second standard with 220 degrees, a third standard with 270 degrees, a fourth standard with 360 degrees, etc. In this case, standard information indicating which standard the image of the display object is shot according to can be associated with the image of the display object. In this case, each standard information is an example of the above-described "configuration information".

[0126] In addition, it can also be set as a standard with a certain degree of freedom. For example, only a part of the above configuration information can be standardized, and other items can be set or changed arbitrarily. For example, the field of view angle of the entire field of view area is determined by prior standardization, but the number of field of view areas (number of images) configured in the VR space, the field of view angle of each field of view area, etc. can be set arbitrarily. In this case, the content of the standard information and the arbitrarily set items can be associated with the image of the display object as "configuration information".

[0127] In addition, the image generation unit 32 includes a switching unit 322. The switching unit 322 has the following function: according to the viewing direction determined by the above-mentioned viewing direction determination unit 31, dynamically switch the live-action images displayed in the overlapping area among adjacent live-action images. The switching unit 322 dynamically switches the live-action images displayed in the overlapping area among adjacent live-action images in such a way that the area boundary of the adjacent live-action images moves away from the viewing direction according to the viewing direction.

[0128] Here, "dynamically switch the live-action images displayed in the overlapping area among adjacent live-action images according to the viewing direction" means that according to the viewing direction, each time it is determined which of the two adjacent live-action images (the first image and the second image) that can be displayed in the overlapping area is to be displayed in the overlapping area, and dynamically switch the live-action image displayed in the overlapping area according to the change of the viewing direction.

[0129] For example, as Figures 10 - 12 shown, a reference direction RD for switching between the adjacent first image and the second image is set. When the viewing direction S is located closer to the first image side than the reference direction RD, the first image is displayed in the overlapping area AO. When the viewing direction S is located closer to the second image side than the reference direction RD, the second image is displayed in the overlapping area AO. This is an example of "dynamically switch the live-action images displayed in the overlapping area among adjacent live-action images according to the viewing direction".

[0130] In addition, it is also possible to dynamically switch the live-action images displayed in the overlapping area according to the viewing direction without setting a reference direction. For example, as Figure 13As illustrated, if the line of sight direction S is located on the side of the first image with respect to the current region boundary (BD1 or BD2) of adjacent captured images (the first image and the second image), the first image is displayed in the overlapping region AO. If it is located on the side of the second image, the second image is displayed in the overlapping region AO. This corresponds to an example of "dynamically switching the captured image to be displayed in the overlapping region among adjacent captured images according to the line of sight direction". In addition, for example, when the line of sight direction is close to the current region boundary between adjacent captured images by a specified amount or more (when the angle formed by the line of sight direction and the region boundary is equal to or less than a specified value), the captured image currently displayed in the overlapping region is switched to the other captured image. This corresponds to an example of "dynamically switching the captured image to be displayed in the overlapping region among adjacent captured images according to the line of sight direction".

[0131] In addition, regarding the process of "switching the captured image to be displayed in the overlapping region", for example, in the overlapping region, two adjacent captured images are arranged on different layers (image layers) above and below. Then, the transparency of the captured image arranged on the lower layer (background) is kept at 0% unchanged, and the transparency of the captured image arranged on the upper layer (foreground) is switched to either 100% or 0%. That is, when displaying the captured image on the lower layer in the overlapping region, the transparency of the upper layer is set to 100%. When displaying the captured image on the upper layer, the transparency of the upper layer is set to 0%. In this way, in the overlapping region, two adjacent captured images are arranged on different layers above and below, and the transparency of the captured image arranged on the upper layer is switched to either 100% or 0%. This corresponds to an example of "switching the captured image to be displayed in the overlapping region".

[0132] In addition, two adjacent captured images can be arranged overlapping in the overlapping region. The transparency of one captured image to be displayed is set to 0%, and the transparency of the other captured image is set to 100%. That is, the transparency of one of the two overlapping captured images is set to 0%, and the transparency of the other captured image is set to 100%. The transparency is switched according to the captured image to be displayed. This corresponds to an example of "switching the captured image to be displayed in the overlapping region".

[0133] In addition, instead of arranging two adjacent captured images overlapping in the overlapping region, only the part corresponding to the overlapping region of one of the captured images to be displayed is arranged in the overlapping region, and the captured image to be arranged is switched. This corresponds to an example of "switching the captured image to be displayed in the overlapping region".

[0134] The switching period for switching the live-action images in the overlapping area can be set arbitrarily. The switching can be made to occur suddenly with the switching period being approximately zero, or a specified switching period (e.g., 0.3 seconds, etc.) can be set. In addition, when switching the live-action images in the overlapping area, there can also be a period of displaying both of the adjacent two live-action images, for example, through semi-transparent synthesis, etc.

[0135] In addition, the image generation unit 32 (the switching unit 322 of the image generation unit 32) has the following function: setting the reference direction to a specified direction starting from the virtual viewpoint, and determining the live-action image to be displayed in the overlapping area according to the line-of-sight direction relative to the reference direction. Here, the "reference direction" refers to the direction set for judging whether to switch the live-action image displayed in the overlapping area according to the relationship with the line-of-sight direction. The "reference direction" can be set to a specified direction starting from the virtual viewpoint in the VR space, and as long as it is in the field-of-view area where adjacent live-action images are arranged in the VR space, it can be set to any direction. For example, as Figure 10 and Figure 11 illustrate, the direction RD passing through the center of the overlapping area AO starting from the virtual viewpoint P in the VR space is an example of the "reference direction". The "reference direction" can be fixed, or it can change (the above-mentioned specified direction changes) and is not fixed as described later.

[0136] For example, as Figure 10 shows, when the line-of-sight direction S is located on the left side (the field-of-view area A1 side of the first image) relative to the reference direction RD, the image generation unit 32 determines the live-action image to be displayed in the overlapping area AO as the first image. In addition, as Figure 11 illustrates, when the line-of-sight direction S is located on the right side (the field-of-view area A2 side of the second image) relative to the reference direction RD, the image generation unit 32 determines the live-action image to be displayed in the overlapping area AO as the second image.

[0137] The image generation unit 32 (the switching unit 322 of the image generation unit 32) preferably sets the reference direction to the direction of a specified position in the overlapping area starting from the virtual viewpoint. Here, "setting the reference direction to the direction of a specified position in the overlapping area starting from the virtual viewpoint" means setting the reference direction in the overlapping area (setting the reference direction so that the vector representing the reference direction exists in the overlapping area). In this case, as Figure 10 and Figure 11 illustrate, since the reference direction RD is set in the overlapping area AO, even if the line-of-sight direction S faces the area boundary (BD1 or BD2) of the overlapping area AO, the live-action image displayed in the overlapping area AO is switched before reaching the area boundary, so the area boundary is also far from the line-of-sight direction S.

[0138] In addition, the image generation unit 32 has the following function: obtaining detection information related to the orientation of the HMD 10, and changing the range of the VR image displayed on the display unit 11 of the HMD 10 according to the detection information. For example, the image generation unit 32 changes the direction of the field of view from the virtual viewpoint P in the VR space according to the detection result of the sensor 12 such as the angular velocity sensor built in the HMD 10, and changes the range of the VR image displayed on the display unit 11 of the HMD 10. For example, taking the direction of the visual axis of the HMD 10 as the direction of the field of view, the range of the VR image displayed on the display unit 11 of the HMD 10 is changed according to the orientation of the HMD 10.

[0139] [6. Processing]

[0140] Next, an example of the processing executed by the display control device 20 of the present embodiment will be described below. Figure 15 It is a flowchart showing an example of the processing of the display control device 20. The processing described below is realized by the control unit 30 (the processor 21 of the display control device 20) executing the program stored in the storage device 22 (the same applies to the processing of each flowchart referred to Figure 17 , Figure 23 , Figure 24 , Figure 27 , Figure 29 or Figure 33 ).

[0141] Here, an example of the display control process of the VR image described with reference to Figures 9 - 12 will be described.

[0142] As Figure 10 or Figure 12 illustrated, the control unit 30 sets the reference direction RD in the VR space V (S100).

[0143] Data of the real-shot images (the first image and the second image) taken by two cameras with different shooting directions of the actual space are stored in, for example, the storage device 22 or a recording medium. The control unit 30 reads out the first image, the second image, and the configuration information from the storage device 22 or the like. Then, the control unit 30 arranges the first image and the second image in the VR space V so that a part of the field-of-view regions A1 and A2 of the first image and the second image overlap with each other to generate an overlapping region AO (S102). For example, the control unit 30 arranges the first image and the second image in different upper and lower layers in the overlapping region AO. For example, the first image is arranged in the lower layer and the second image is arranged in the upper layer.

[0144] In addition, the control unit 30 obtains detection information related to the orientation of the HMD 10 from the sensor 12 (e.g., angular velocity sensor) of the HMD 10 (S104), and determines the line-of-sight direction S (S106). Then, the control unit 30 determines whether the line-of-sight direction S is located on the left side (first image side) of the reference direction RD (S108). If it is "yes", as Figure 10 illustrated, the first image is displayed in the overlapping area AO (S110). For example, in the overlapping area AO, the transparency of the first image is fixed at 0%, and the transparency of the second image arranged at a position above the first image is set to 100%, and the first image is displayed in the overlapping area AO. On the other hand, if the line-of-sight direction S is located on the right side (second image side) of the reference direction RD ( "no" in S108), then as Figure 11 illustrated, the control unit 30 displays the second image in the overlapping area AO (S112). For example, in the overlapping area AO, the transparency of the second image arranged at a position above the first image is set to 0%, and the second image is displayed in the overlapping area AO. In this way, the control unit 30 performs the following control: dynamically switches the real-shot image (the first image and the second image) adjacent to the real-shot image displayed in the overlapping area AO according to the line-of-sight direction S.

[0145] Then, the control unit 30 generates a VR image in a range corresponding to the orientation of the HMD 10 based on the detection information related to the orientation of the HMD 10 in S104 above and outputs it to the HMD 10 (S114). For example, the control unit 30 outputs the VR image generated by performing rendering processing and the like to the HMD 10. Thereby, the VR image is displayed on the HMD 10.

[0146] The processing of S102 to S114 above is repeated until the display ends by a reproduction stop operation or the like performed by the user ( "yes" in S116).

[0147] [7. Method of Changing the Reference Direction]

[0148] In the above description, an example of setting a fixed reference direction RD is shown. When fixing the reference direction RD, when the line-of-sight direction (line-of-sight direction S) of the HMD 10 is near the reference direction RD, the real-shot image displayed in the overlapping area AO may be frequently switched, so it may become an image that is difficult for the user to see. Therefore, when the real-shot image displayed in the overlapping area AO is switched, the reference direction RD is changed in such a way that the reference direction RD is away from the line-of-sight direction (line-of-sight direction S) of the HMD 10. The following describes this content.

[0149] As Figure 16As exemplified by (A) therein, when the first image of the left field-of-view region A1 is displayed in the overlapping region AO, the reference direction RD1 is set to a direction that is shifted to the right (e.g., -20 degrees) from the direction passing through the center of the overlapping region AO starting from the virtual viewpoint P. Also, the angle is positive in the counterclockwise (left-handed) direction. From this state, when the visual axis direction (line-of-sight direction S) of the HMD 10 rotates to the right direction, and when the line-of-sight direction S exceeds the reference direction RD1, as Figure 16 exemplified by (B) therein, the real-shot image displayed in the overlapping region AO is switched from the first image to the second image of the field-of-view region A2. At this switching timing (simultaneously with the switching), it moves to the left in such a way that the reference direction moves away from the visual axis direction (line-of-sight direction S) of the HMD 10, and from Figure 16 the reference direction RD1 in (A) therein is changed to Figure 16 the reference direction RD2 in (B) therein. That is, as Figure 16 exemplified by (B) therein, when the second image of the right field-of-view region A2 is displayed in the overlapping region AO, the reference direction RD is set to a direction that is shifted to the left (e.g., 20 degrees) from the direction passing through the center of the overlapping region AO starting from the virtual viewpoint P.

[0150] In addition, from Figure 16 the state of (B) therein, when the visual axis direction (line-of-sight direction S) of the HMD 10 rotates to the left direction, and when the line-of-sight direction S exceeds the reference direction RD2, as Figure 16 exemplified by (A) therein, while the real-shot image displayed in the overlapping region AO is switched from the second image to the first image of the field-of-view region A1, it is changed from the reference direction RD2 to the reference direction RD1. Also, "simultaneously" includes approximately simultaneously.

[0151] This is a method of performing hysteresis control by changing the reference direction (RD1 or RD2) at the timing of switching the real-shot image displayed in the overlapping region AO.

[0152] In addition, it is also a method of moving the reference direction RD to the switched region boundary (BD1 or BD2) in such a way that the reference direction moves away from the line-of-sight direction S when the real-shot image displayed in the overlapping region AO is switched.

[0153] In addition, it is still a method of changing the reference direction (RD1 or RD2) according to the real-shot image displayed in the overlapping region AO (depending on whether it is the first image of the field-of-view region A1 or the second image of the field-of-view region A2).

[0154] The image generation unit 32 has the following function: when switching the live-action image displayed in the overlapping area AO, the reference direction (RD1 or RD2) is changed according to the live-action image displayed in the overlapping area AO (depending on whether it is the first image of the field-of-view area A1 or the second image of the field-of-view area A2).

[0155] Next, with reference to Figure 17 , an example of the process of changing the above reference direction will be described. Figure 17 is a flowchart showing an example of the process of changing the reference direction executed by the control unit 30 of the present embodiment.

[0156] As Figure 16 illustrated in (A) or (B) of, the control unit 30 sets a reference direction (RD1 or RD2) corresponding to the live-action image (the first image of the field-of-view area A1 or the second image of the field-of-view area A2) displayed in the overlapping area AO (S200). In addition, the control unit 30 obtains detection information related to the orientation of the HMD 10 from the sensor 12 (e.g., angular velocity sensor) of the HMD 10 (S202), and determines the line-of-sight direction S (S204). In addition, the control unit 30 determines whether it is necessary to switch the live-action image displayed in the overlapping area AO according to the line-of-sight direction S with respect to the current reference direction (RD1 or RD2) (S206). If the result in S206 is "yes", the control unit 30 executes the switching of the live-action image displayed in the overlapping area AO (S208), and changes the reference direction to the direction corresponding to the switched live-action image (S210). On the other hand, if the result in S206 is "no", the process returns to S202.

[0157] The above processes of S202 to S210 are repeatedly performed until the display ends (when the result in S212 is "yes") due to a reproduction stop operation or the like performed by the user.

[0158] According to the above method, the reference direction (RD1 or RD2) is not fixed, but is changed according to the live-action image displayed in the overlapping area AO when the live-action image displayed in the overlapping area AO is switched. Thus, after switching the live-action image, the reference direction temporarily moves away from the line-of-sight direction S, so that it is possible to reduce the frequent switching of the live-action image displayed in the overlapping area AO.

[0159] [8. Display the current area boundary in a visually recognizable manner]

[0160] In the image display system 1 of the present embodiment, when switching the live-action image (the first image or the second image) displayed in the overlapping area AO, the area boundary (BD1 or BD2) between adjacent live-action images is moved away from the line-of-sight direction S, but the current area boundary may also be deliberately displayed in a visually recognizable manner. The following describes this content.

[0161] Figure 18 and Figure 19 1 is a diagram showing an example of a screen displayed on the display unit 11 of the HMD 10. Figure 18 The example screen G10 shows Figure 10 The VR image of the range of the field of view corresponding to the orientation of the HMD 10 in the VR space V is shown in the example. Figure 10 The VR image of the field of view in the visual axis direction (line of sight direction S) of the HMD 10 from the virtual viewpoint P in the VR space V is displayed. Therefore, the first image of the field of view area A1 on the left is displayed in the overlapping area AO. In this screen G10, a boundary line BL1 is displayed in a visually recognizable manner at a position corresponding to the current area boundary BD1 (the right end of the overlapping area AO) between the adjacent real-shot images (the first image and the second image). For example, the boundary line BL1 is a black solid line with a line width that can be visually recognized by the user.

[0162] In observation Figure 18 When the user of the screen G10 rotates the head (HMD 10) to the right, the screen displayed is Figure 19 The screen G11 shown in the example. Figure 11 The VR image of the range of the field of view corresponding to the orientation of the HMD 10 in the VR space V shown in the example. That is, since the visual axis direction (sight direction S) of the HMD 10 is rotated to the right and exceeds the reference direction RD, Figure 18 The VR image of the screen G10 is switched to display the real image of the overlapping area AO, and the VR image obtained is displayed on Figure 19 The illustrated screen G11. In the screen G11, the boundary line BL2 is displayed in a visually recognizable manner at a position corresponding to the current area boundary BD2 (the left end of the overlapping area AO) between adjacent real-shot images (the first image and the second image). For example, the boundary line BL2 is a black solid line having the same line width as the boundary line BL1.

[0163] like Figure 20 As shown in the example, the display control device 20 can be a structure including a boundary line display unit 33. The boundary line display unit 33 has a function of displaying a boundary line that can be visually recognized by the user on the area boundary between adjacent real-shot images. Here, the above-mentioned "area boundary between adjacent real-shot images" refers to the boundary between two adjacent real-shot images (the first image and the second image) arranged in the VR space. Figure 10 and Figure 11As illustrated, the position of the region boundary (BD1 or BD2) changes according to the real-shot image (the first image of the field-of-view region A1 or the second image of the field-of-view region A2) displayed in the overlapping region AO. In addition, the above-mentioned "boundary line" refers to a line displayed on the region boundary in such a way that the user can visually recognize the region boundary between adjacent real-shot images. The color, line width, transparency, etc. of the "boundary line" can be set arbitrarily. In addition, the "boundary line" is, for example, a solid line, a dotted line, a dash-dot line, a wavy line, a double line, etc., and the type of the line can also be set arbitrarily. In addition, the "boundary line" can be a two-dimensional line or a line that bulges three-dimensionally (stereoscopically).

[0164] The boundary line display unit 33 disposes the boundary line (BD1 or BD2) at the current region boundary (BD1 or BD2) portion and displays it on the display unit 11 so that the boundary line overlaps with the real-shot VR image.

[0165] As Figure 10 and Figure 11 As illustrated, etc., the region boundary (BD1 or BD2) that generates an image seam (misalignment) is changed by switching the real-shot image displayed in the overlapping region AO according to the line-of-sight direction S. That is, by the user changing the line-of-sight direction S according to his or her own will, the region boundary (BD1 or BD2) that generates an image seam (misalignment) can be changed. That is, when there is an image seam in the direction that the user wants to observe, if the user changes the line-of-sight direction S according to his or her own will, the image seam located in the direction that the user wants to observe can be eliminated and an easily observable image can be obtained.

[0166] Then, according to this method, the boundary line (BL1 or BL2) is deliberately displayed in a visually recognizable manner on the region boundary (BD1 or BD2) between adjacent real-shot images, whereby the user can easily identify the region boundary that generates an image seam (misalignment). As Figure 18 and Figure 19 As illustrated, the boundary line (BL1 or BL2) of the image seam portion can be changed by the user changing the line-of-sight direction S according to his or her own will. Even after the change, the user can easily identify where the image seam has moved. That is, by displaying the boundary line (BL1 or BL2) in a visually recognizable manner on the region boundary (BD1 or BD2), the user can easily judge which of the adjacent real-shot images should be displayed in the overlapping region AO for observation, so that the boundary line is not displayed in the direction that the user wants to observe.

[0167] In the case of determining the line of sight direction S based on detection information related to the orientation of the HMD 10 (e.g., detection information of an angular velocity sensor), the user can adjust the range of the VR image displayed on the display unit 11 of the HMD 10 by adjusting the orientation of the head wearing the HMD 10, and change the boundary line (BL1 or BL2).

[0168] In addition, in the case of determining the line of sight direction S based on detection information related to the user's line of sight (e.g., detection information based on eye tracking), even if the user does not move the head, the user can change the boundary line (BL1 or BL2) displayed at the area boundary (BD1 or BD2) only by changing the line of sight. For example, in the structure where the range of the VR image displayed on the display unit 11 is changed based on detection information related to the orientation of the HMD 10 (e.g., detection information of an angular velocity sensor) and the line of sight direction S is determined based on detection information related to the user's line of sight (e.g., detection information based on eye tracking), the user can adjust the image that is easy to observe by performing the following processing. That is, the user adjusts the orientation of the head to adjust the range of the VR image so that the image in the direction to be observed is displayed on the display unit 11 of the HMD 10. Then, if the line of sight direction is adjusted in the state where the image in the direction to be observed is displayed on the display unit 11 and the boundary line (BL1 or BL2) is changed, even if there is a boundary line in the direction to be observed, it can be moved to display an image that is easy to observe.

[0169] In addition, in the structure where the range of the VR image displayed on the display unit 11 is changed based on detection information related to the user's line of sight such as eye tracking and the real image displayed in the overlapping area AO is switched, the user can adjust the image that is easy to observe only by adjusting the line of sight direction.

[0170] (The method of displaying additional information at or near the boundary line)

[0171] Next, hereinafter, a method will be described in which the user can more easily determine which of the adjacent real images should be displayed in the overlapping area AO for observation.

[0172] For example, in Figure 18 the illustrated screen G10 displays the boundary line BL1, but it is difficult for the user to immediately recognize whether the overlapping area AO is on the left or the right across the boundary line BL1. In addition, it is difficult for the user to immediately recognize which of the adjacent real images (the first image on the left or the second image on the right) is currently displayed in the overlapping area AO. In addition, it is difficult for the user to immediately recognize in which direction the line of sight direction S should be changed to cause the switching of the real image in the overlapping area AO (i.e., the change of the boundary line).

[0173] In addition, even in a situation where it is difficult to immediately recognize the above situation, if the user changes the line-of-sight direction S to the left or right for observation, then from Figure 18 the illustrated screen G10 changes to Figure 19 the illustrated screen G11, or from screen G11 to screen G10, so it can also be recognized. In this method, additional information is displayed at or near the boundary line, so that it is easy for the user to recognize in which direction the line-of-sight direction S needs to change.

[0174] Figure 21 and Figure 22 are diagrams showing an example of a screen in which the gradient portion is displayed together with the boundary line. Figure 21 is Figure 18 a diagram showing an example of a screen G20 corresponding to the illustrated screen G10 and displaying the gradient portion GD1 as additional information on the right side of the boundary line BL1.

[0175] Figure 21 The screen G20 of Figure 10 corresponds to the VR space V of Figure 10 . Therefore, the left side of the boundary line BL1 corresponds to "the overlapping region AO where the first image of the left visual field region A1 is displayed", and the region on the right side of the boundary line BL1 is on the side of the non-overlapping region AO (only the region of the right visual field region A2). That is, the gradient portion GD1 of the screen G20 is displayed on the side opposite to the overlapping region AO across the boundary line BL1. This gradient portion GD1 is a semi-transparent region showing a black gradient, and the concentration or opacity of this black gradient continuously or stepwise decreases as it gets farther from the boundary line BL1 (towards the right in Figure 21 ). The width (distance in the direction orthogonal to the boundary line BL1), color, etc. of the gradient portion GD1 can be arbitrarily set (the same applies to the gradient portion GD2 described below).

[0176] In addition, Figure 22 is a diagram showing an example of a screen G21 corresponding to the illustrated screen G11 and displaying the gradient portion GD2 as additional information on the left side of the boundary line BL2. This Figure 19 screen G21 of Figure 22 corresponds to the VR space V of Figure 11 . Therefore, the right side of the boundary line BL2 corresponds to "the overlapping region AO where the second image of the right visual field region A2 is displayed", and the region on the left side of the boundary line BL2 is on the side of the non-overlapping region AO (only the region of the left visual field region A1). That is, the gradient portion GD2 of the screen G21 is displayed on the side opposite to the overlapping region AO across the boundary line BL2. This gradient portion GD2 is a semi-transparent region showing a gradient, and the gradient continuously or stepwise decreases as it gets farther from the boundary line BL2 (in Figure 22In the case of China-Vietnam facing left, the concentration or opacity decreases continuously or stepwise. The color of the gradient portion GD2 is, for example, a black gradient, but any color can be set.

[0177] In this way, the gradient portion (GD1 or GD2) is displayed on the side opposite to the overlapping area AO across the boundary line (BL1 or BL2). Thereby, the user can recognize the side opposite to the gradient portion (GD1 or GD2) of the boundary line (BL1 or BL2) (the left side of the boundary line BL1 in Figure 21 is the left side of the boundary line BL1, and in Figure 22 is the right side of the boundary line BL2) is the overlapping area AO. In addition, the user can recognize that a real-shot image of the side opposite to the gradient portion (GD1 or GD2) across the boundary line (BL1 or BL2) is currently displayed in the overlapping area AO (the first image on the left side in Figure 21 is the first image on the left side, and in Figure 22 is the second image on the right side). In addition, when the user moves the line of sight direction S in the direction in which the gradient portion (GD1 or GD2) thins (from the boundary line BL1 toward the direction having the gradient portion), the user can intuitively recognize that a switch (change of the boundary line) of the real-shot image in the overlapping area AO may occur.

[0178] That is, the gradient portion (GD1 or GD2) is "information for indicating which side across the boundary line is the overlapping area", "information for indicating which of the adjacent real-shot images is the real-shot image displayed in the overlapping area", or "information indicating in which direction the line of sight direction should be changed to cause a switch of the real-shot image in the overlapping area".

[0179] In addition, the gradient portion (GD1 or GD2) is displayed overlappingly on the misaligned portion of the image generated in the portion of the boundary line (BL1 or BL2), so that the effect of not significantly misaligning the image is also produced. In addition, the change in the concentration or opacity of the gradient portion (GD1 or GD2) also serves to naturally indicate the change direction of the line of sight direction S required for the switch of the real-shot image in the overlapping area AO. Therefore, a preferred mode is to apply the gradient portion (GD1 or GD2) as additional information displayed together with the boundary line.

[0180] In addition, as long as it is possible to recognize which side across the boundary line (BL1 or BL2) is the overlapping area AO, the gradient portion (GD1 or GD2) may also be displayed on the overlapping area AO side across the boundary line (BL1 or BL2).

[0181] The boundary line display unit 33 of this method has a function of displaying, at or near the above-mentioned boundary line, additional information for indicating which side of the above-mentioned boundary line is the above-mentioned overlapping area. Here, the "additional information for indicating which side of the boundary line is the overlapping area" refers to the additional information of the boundary line displayed at or near the boundary line, and is information that enables the user to recognize which side of the boundary line is the overlapping area at present. For example, as described above, the gradient part (GD1 or GD2) displayed on the side opposite to the overlapping area AO (or the overlapping area AO side) across the boundary line (BL1 or BL2) is an example of the "additional information for indicating which side of the boundary line is the overlapping area". In addition, the additional information is not limited to the gradient part (GD1 or GD2). For example, an "arrow" or "symbol" indicating which side of the boundary line is the above-mentioned overlapping area may be displayed overlapping with the boundary line (BL1 or BL2) or near the boundary line. In this case, the above-mentioned "arrow" or "symbol", etc. are equivalent to an example of the additional information.

[0182] In addition, the above-mentioned additional information such as "arrow" or "symbol" also becomes an example of the "information for indicating which of the actual shooting images displayed in the overlapping area in the adjacent actual shooting images" or "the information indicating in which direction the line of sight direction should be changed to generate the switching of the actual shooting images in the overlapping area".

[0183] The boundary line display unit 33 arranges the boundary line (BD1 or BD2) at a part of the current area boundary (BD1 or BD2), and arranges additional information such as the above-mentioned gradient part (GD1 or GD2), so that the boundary line and the additional information are displayed on the display unit 11 overlapping with the actual shooting VR image.

[0184] Next, with reference to Figure 23 , an example of the process in the case of displaying the above-mentioned boundary line (BL1 or BL2) will be described. Figure 23 It is a flowchart showing an example of the process of the display control device 20 in the case of displaying the boundary line. Figure 23 In the flowchart of S300 to S312 of Figure 15 , the processes are the same as those of S100 to S112 of

[0185] The control unit 30 executes control (S304 to S312) to dynamically switch the live-action image (the first image and the second image) to be displayed in the overlapping region AO among adjacent live-action images according to the line-of-sight direction S. Then, the control unit 30 overlaps a boundary line (BL1 or BL2) on a part of the region boundary (BD1 or BD2) between adjacent live-action images (S314). Further, the control unit 30 overlaps, for example, a gradient part (GD1 or GD2) on a region on the side opposite to the overlapping region AO across the boundary line (BL1 or BL2) (a region of the live-action image not displayed in the overlapping region AO) (S316). Then, the control unit 30 generates a VR image in a range corresponding to the orientation of the HMD 10 and outputs it to the HMD 10 (S318). Thereby, a VR image including the boundary line (BL1 or BL2) and the gradient part (GD1 or GD2) is displayed on the HMD 10. The processing of S302 to S318 is repeated until the display ends (Yes in S320).

[0186] In addition, in Figure 23 the flowchart of, when not displaying the gradient part (GD1 or GD2) in the VR image, the process of S316 can be omitted.

[0187] (Method of making the display modes of two boundary lines different)

[0188] The boundary line display unit 33 can also make the display modes of the boundary line BL1 of the region boundary BD1 displayed at one end (the right end) of the overlapping region AO and the boundary line BL2 of the region boundary BD2 displayed at the other end (the left end) different. Here, "making the display modes of the boundary lines different" means making the color, density, transparency, line width, type of line (solid line, dotted line, etc.), or whether it is planar (two-dimensional) or three-dimensional, etc., of the boundary lines different. Thereby, when using a plurality of images arranged in the horizontal direction to expand the horizontal field of view angle, the user can recognize whether the currently displayed boundary line (BL1 or BD2) is a misaligned part of the image at the left end of the overlapping region AO or the right end. In addition, when using a plurality of images arranged in the vertical direction to expand the vertical field of view angle, the user can recognize whether the currently displayed boundary line (BL1 or BD2) is a misaligned part of the image at the upper end of the overlapping region AO or the lower end. In this method, the display of additional information such as the above-mentioned gradient parts GD1 and GD2 can be omitted, or the additional information can be displayed together with the boundary lines.

[0189] (Method of setting the switching period)

[0190] When switching between two real-shot images in the overlapping area AO, instead of suddenly switching from one to the other, a switching period (such as 0.3 seconds, etc.) can be set, and the switching can be gradually performed over time. For example, when switching the real-shot image displayed in the overlapping area AO from one to another, the display of the overlapping area AO can be cross-faded during the switching period. Thus, it is easier to visually recognize the boundary line (BL1 or BL2) more naturally before and after the switching of the real-shot images in the overlapping area AO. The following describes this content.

[0191] The image generation unit 32 of this method has the following function: when switching the real-shot image displayed in the overlapping area AO, it gradually switches from one real-shot image to another over time.

[0192] Here, "gradually switching from one real-shot image to another over time" means setting a switching period from the start to the end of the switching (such as 0.3 seconds, etc.), and during the time process of passing through the switching period, the real-shot image currently displayed in the overlapping area AO (the first image or the second image) can be gradually switched to another real-shot image. The switching period can be set arbitrarily.

[0193] For example, during the above-mentioned switching period, the two real-shot images to be switched in the overlapping area are image-combined, and the real-shot image displayed in the overlapping area is gradually switched from one to another, which is an example of "gradually switching from one real-shot image to another over time". As an example of the above image combination, there is alpha blending (semi-transparent combination). As a specific example, over the time in the switching period, the transparency of one real-shot image is gradually increased "from 0% to 100%", and the transparency of the other is gradually decreased from "100% to 0%", and the two real-shot images are alpha-blended.

[0194] In addition, for example, in the overlapping area, two adjacent real-shot images are arranged on different upper and lower layers. Moreover, the transparency of the real-shot image arranged on the lower layer (background) can be kept at 0% unchanged, while the transparency of the real-shot image arranged on the upper layer (foreground) gradually changes "from 0% to 100%" or "from 100% to 0%" over the time in the switching period. For example, when switching from the real-shot image on the upper layer to the real-shot image on the lower layer, a process is performed in which the transparency of the real-shot image on the upper layer gradually changes "from 0% to 100%" over the time in the switching period. In addition, when switching from the real-shot image on the lower layer to the real-shot image on the upper layer, a process is performed in which the transparency of the real-shot image on the upper layer gradually changes "from 100% to 0%" over the time in the switching period.

[0195] In addition, during the above-described switching, a process is performed in which the pixels of the live-action image currently displayed in the overlapping area are sequentially replaced with the pixels of another live-action image, which is an example of "gradually switching from one live-action image to another over time". In other words, a process in which the number (area) of pixels of the live-action image currently displayed in the overlapping area gradually decreases and the number (area) of pixels of another live-action image gradually increases over time is an example of "gradually switching from one live-action image to another over time". As a specific example, during the switching period, the pixels are gradually replaced from the current region boundary side to the region boundary side after the switching is completed, and the region boundary slides (gradually moves), which is an example of "gradually switching from one live-action image to another over time". The above "switching pixels" includes switching the transparency of the pixels in the upper layer "from 0% to 100%" or "from 100% to 0%" in two adjacent live-action images arranged in different layers above and below the overlapping area. In addition, the above "switching pixels" includes changing the transparency of the pixels of the live-action image to be replaced from 0% to 100% and changing the transparency of the corresponding pixels of the live-action image to be replaced from 100% to 0%. In addition, it may be that the pixels are gradually switched from one side to the other side from the center of the overlapping area to the periphery, from the peripheral part to the center, from the left end to the right end, from the right end to the left end, from the upper end to the lower end, or from the lower end to the upper end.

[0196] Next, with reference to Figure 24 , an example of the process in which the above-described switching period is set will be described. Figure 24 is a flowchart showing an example of a process of sequentially switching the live-action images displayed in the overlapping area of the display control device 20.

[0197] The control unit 30 acquires detection information related to the orientation of the HMD 10 from the sensor 12 (e.g., angular velocity sensor) of the HMD 10 (S400), and determines the line-of-sight direction S (S402). In addition, the control unit 30 determines whether it is necessary to switch the live-action image displayed in the overlapping area AO based on the line-of-sight direction S with respect to the current reference direction (RD1 or RD2) (S404). If the result in S404 is "Yes", the control unit 30 initializes the variable α to the initial value "255" (S406). This variable α is the "α value" used for the semi-transparent synthesis (α blending) of adjacent live-action images (the first image and the second image) in the overlapping area AO in step S408, and can take values from "0 to 255". In addition, the "α value" is used to change the opacity of the boundary line (BL1 or BL2) in steps S410 and S412.

[0198] The control unit 30 subtracts a prescribed amount Δα from the α value every 1 frame, and gradually decreases the α value from "255" to "0" as time elapses during the switching period (S414). For example, the switching period is set to 20 frames. In addition, 1 frame is, for example, 1 / 60 second. For example, when the prescribed amount Δα subtracted every 1 frame is set to "13", the α value gradually changes from "255" to "0" during a switching period of approximately 0.3 seconds.

[0199] In S408, the control unit 30 semi-transparently synthesizes two captured images (the first image and the second image) in the overlapping area AO. In this case, the control unit 30 sets the alpha value of each pixel of the image (referred to as the "pre-switching image") that was displayed in the overlapping area AO before the switch to the value of the variable α. In addition, the image displayed after the switch in the overlapping area AO is referred to as the "post-switching image". Therefore, during the switching period when semi-transparent synthesis is performed, the pixel value of each pixel in the overlapping area AO is set to "the pixel value of the corresponding pixel of the pre-switching image * α + the pixel value of the corresponding pixel of the post-switching image * (255 - α)".

[0200] In addition, in S410, the control unit 30 sets the opacity of the boundary line before the switch to α. In addition, in S412, the control unit 30 sets the opacity of the boundary line after the switch to (255 - α), overlapping with the partial area boundary after the switch. Here, when α = 255, the boundary line becomes a completely opaque state (transparency 0%), and the smaller the α value, the lower the opacity. When α = 0, it becomes a completely transparent state (transparency 100%).

[0201] The above S408 to S414 are repeated until the α value = 0 ("Yes" in S416). Thus, for example, during a switching period of approximately 0.3 seconds, the α value gradually changes from "255" to "0". In the overlapping area AO, the transparency of the pre-switching image gradually increases, while the transparency of the post-switching image gradually decreases. In addition, during the switching period, the boundary line before the switch and the boundary line after the switch are displayed simultaneously, and as time elapses during the switching period, the transparency of the boundary line before the switch gradually increases, while the transparency of the boundary line after the switch gradually decreases. Then, finally, at the moment when the α value becomes "0" ("Yes" in S416), the switching of the image in the overlapping area AO ends, the post-switching image is displayed in the overlapping area AO, and the boundary line before the switch disappears, and the boundary line after the switch is displayed.

[0202] Here, an example is shown in which the boundary lines before and after the switch are displayed while changing the transparency during the switching period, but it is not limited to this. It is also possible to delete the boundary line before the switch at the timing when the switching process (during the switching period) of the image in the overlapping area AO starts, and display the boundary line after the switch at the timing when the switching process (during the switching period) ends. In this case, S410 and S412 of Figure 24 can be omitted. Alternatively, it is also possible to display the boundary line before the switch and the boundary line after the switch with the same transparency (either one can be opaque) during the switching process (during the switching period), delete the boundary line before the switch at the timing when the switching process (during the switching period) ends, and display the boundary line after the switch.

[0203] (The method of moving and displaying the boundary line during the switching period)

[0204] In the method of setting the above switching period, it is also possible to move the boundary line to the position where it will be displayed after the switch is completed sequentially during the switching period. For example, when the first image of the left field of view area A1 is displayed in the overlapping area AO, as Figure 21 illustrated, the boundary line BL1 is displayed on the screen G20. When switching the live-action image displayed in the overlapping area AO from this state to the second image of the right field of view area A2, as Figure 25 illustrated, the boundary line BL1 at the position indicated by the dashed line changes to the position of the boundary line BL2. When switching this image, during the switching period (for example, 0.3 seconds, etc.), the boundary line is displayed and controlled to gradually move horizontally from the position of BL1 indicated by the dashed line to the position of BL2. Even so, the user can naturally recognize the position of the boundary line.

[0205] The image generation unit 32 of this method has the following function: when switching the live-action image displayed in the overlapping area AO, it gradually moves the boundary line to the position where it will be displayed after the switch is completed over time, and at the same time gradually switches from one live-action image to another live-action image.

[0206] For example, the case of switching the live-action image in the overlapping area AO from the first image on the left to the second image on the right will be described. As Figure 25As shown in the example, during the switching period, the boundary line BLmove is moved horizontally from the area boundary BD1 before the switching to the area boundary BD2 after the switching. At this time, in the overlapping area AO, in the area to the right of the moved boundary line BLmove (the area between the area boundary BD1 before the switching and the moved boundary line BLmove), each pixel of the first image is replaced with each corresponding pixel of the second image. Thus, as the boundary line BLmove moves horizontally to the left, each pixel to the right of the boundary line BLmove in the overlapping area AO is switched from the first image to the second image. This process is repeated until the boundary line BLmove reaches the area boundary BD2 after the switching. When the boundary line BLmove reaches the area boundary BD2, it stops as the boundary line BL2.

[0207] The same processing as described above is also performed when the real-shot image of the overlapping area AO is switched from the second image on the right to the first image on the left.

[0208] like Figure 25 As shown in the example, when the gradient part (GD1 or GD2) is displayed as additional information, the display of the gradient part can be controlled as follows. For example, when the switching process of the real-shot image of the overlapping area AO is started (during the switching period), the gradient part (GD1 or GD2) is deleted, and when the switching process (during the switching period) is completed, the gradient part is displayed. Alternatively, the gradient part can also move in conjunction with the movement of the boundary line BLmove.

[0209] In addition, in the structure where the boundary line is moved and displayed as time passes during the switching period, the switching process of the real-shot image in the overlapping area AO is not limited to the above content. For example, the switching process of the real-shot image in the overlapping area AO can also be performed by the above-mentioned semi-transparent synthesis (α blending), and the boundary line is gradually moved to the position displayed after the switching is completed as time passes during the switching process.

[0210] [9. An example of dividing the entire field of view of VR space into three or more areas]

[0211] Here, refer to Figure 26 , an example of dividing the entire field of view area of ​​the VR space V into three areas of left, front, and right is described. Here, an example of being able to reproduce a high field of view image with a horizontal field of view angle of 270 degrees using the left image, front image, and right image captured by three stereo cameras with a horizontal field of view angle of 120 degrees in each direction of the left, front, and right is shown.

[0212] Figure 26FIG. is a diagram schematically showing an example in which the field-of-view region AL of the left image, the field-of-view region AF of the front image, and the field-of-view region AR of the right image are arranged in the VR space V. In the VR space V, the arrangement of the images is controlled in such a way that a first overlapping region AO1 is generated where a part of the field-of-view region AL of the left image and a part of the field-of-view region AF of the front image overlap each other. In addition, the arrangement of the images is controlled in such a way that a second overlapping region AO2 is generated where a part of the field-of-view region AF of the front image and a part of the field-of-view region AR of the right image overlap each other. The field-of-view angle θall of the entire field-of-view region of the VR space V is 270 degrees, the field-of-view angle θAL of the left field-of-view region AL is 120 degrees, the field-of-view angle θAF of the front field-of-view region AF is 120 degrees, the field-of-view angle θAR of the right field-of-view region AR is 120 degrees, the field-of-view angle θAO1 of the first overlapping region AO1 is 45 degrees, and the field-of-view angle θAO2 of the second overlapping region AO2 is 45 degrees.

[0213] In addition, a reference direction RD1, which is a reference for switching the real-shot image (left image or front image) of the overlapping region AO1, is set to the central direction of the overlapping region AO1 starting from the virtual viewpoint P. In addition, a reference direction RD2, which is a reference for switching the real-shot image (front image or right image) of the overlapping region AO2, is set to the central direction of the overlapping region AO2 starting from the virtual viewpoint P.

[0214] This is only an example, and the field-of-view angle of the entire field-of-view region, the field-of-view angles of the field-of-view regions of the respective images, the field-of-view angles of the respective overlapping regions, or the reference direction (presence / absence, its direction) can be arbitrarily set.

[0215] In the case where the entire field-of-view region of the VR space is divided into three regions as in this example, two overlapping regions such as the first overlapping region AO1 and the second overlapping region AO2 are generated. It is sufficient to perform each process related to the above overlapping region (and the processes shown below) for the first overlapping region AO1 and the second overlapping region AO2, respectively. That is, in the relationship between the adjacent left-image field-of-view region AL and the front-image field-of-view region AF, the left image corresponds to the above-mentioned "first image", the front image corresponds to the above-mentioned "second image", and the first overlapping region AO1 corresponds to the above-mentioned "overlapping region AO", so that each of the above processes can be applied. In addition, in the relationship between the adjacent front-image field-of-view region AF and the right-image field-of-view region AR, the front image corresponds to the above-mentioned "first image", the right image corresponds to the above-mentioned "second image", and the second overlapping region AO2 corresponds to the above-mentioned "overlapping region AO", so that each of the above processes can be applied.

[0216] The same applies when the entire field-of-view region of the VR space is divided into four or more regions. When n real-shot images captured by n cameras (n is a natural number of 2 or more) are arranged in the VR space, (n - 1) overlapping regions are generated. In the case of having a plurality of overlapping regions AO, considering one of the adjacent images arranged in each overlapping region as the "first image" and the other as the "second image", the above-described respective processes can be applied to each overlapping region AO.

[0217] Generally, the field-of-view image (VR image) within the range displayed on the display unit 11 of the HMD 10 is a part of the entire field-of-view region of the VR space. Therefore, when at least one overlapping region AO is not included in the range of the VR image displayed on the display unit 11, there is no need to perform image switching processing or the like for that overlapping region AO, and it can be not executed.

[0218] [10. Method of setting a time lag before starting the execution of the switching process]

[0219] In the switching process of the above-described overlapping region AO, a predetermined time lag (for example, 0.5 seconds, etc.) can also be set before starting the execution of the process. By setting such a time lag, unnecessary switching processes can be omitted. The following explains this content.

[0220] Here, as Figure 26 shown, an example is illustrated where the regions of the VR space V are in three directions: left, front, and right. Assume that a user wearing the HMD 10 sees a VR image within a range including the left field-of-view region AL and the first overlapping region AO1. Then, assume that a left image is currently being displayed in the first overlapping region AO1. Consider the following situation: From this state, the user swings their head significantly to the right, and the viewing axis direction (line-of-sight direction S) of the HMD 10 instantaneously moves across the front and reaches the right field-of-view region AR. In the case where there is no time lag set, the switching process in the first overlapping region AO1 and the switching process in the second overlapping region AO2 can occur within a short period of time. However, at the moment when the viewing axis direction of the HMD 10 reaches the right field-of-view region AR, the first overlapping region AO1 may sometimes have moved out of the range of the VR image displayed on the display unit 11 of the HMD 10. In such a case, when the switching process is executed sensitively, the VR image displayed on the display unit 11 may shake and the visual recognition performance may deteriorate. On the other hand, in the above-described situation, by setting a time lag in the switching process, it is possible to, as needed, only execute the switching process for the second overlapping region AO2 and omit the switching process for the first overlapping region AO1, and the visual recognition performance of the image displayed on the display unit 11 can be improved.

[0221] In addition, in the description where the first overlapping region AO1 and the second overlapping region AO2 are not distinguished, it is sometimes described as "overlapping region AO".

[0222] The image generation unit 32 of this method has the following function: according to the line-of-sight direction S, a prescribed standby period is set from when it is determined that the switching process of the live-action image to be displayed in the overlapping region AO should be performed until the start of this switching process. Here, the standby period corresponds to the above-mentioned time lag.

[0223] In addition, after the standby period until the start of the above-mentioned switching process has elapsed, the image generation unit 32 re-determines whether the switching process should be performed, and only when it is determined that it should be performed, executes the switching process.

[0224] The standby period is, for example, 0.5 seconds, but is not limited thereto and can be set arbitrarily. In the case of determining the line-of-sight direction S based on detection information related to the orientation of the HMD 10 (detection information of an angular velocity sensor of the HMD 10, etc.) and determining whether a switching process of the overlapping region AO is required, and in the case of determining the line-of-sight direction S based on detection information related to the user's line of sight (detection information such as eye tracking) and determining whether a switching process of the overlapping region AO is required, the length of the standby period can also be different. That is, in eye tracking, since the line of sight is traced by analyzing the movement of a person's eyeballs, etc., the line-of-sight direction S can easily move freely. Therefore, compared with the determination based on so-called head tracking that detects the orientation of the HMD 10, it is better to set the above-mentioned standby period (time lag) longer in the determination based on eye tracking that traces the user's line of sight. That is, in the case of performing a switching process by determining the line-of-sight direction S based on eye tracking, in the case of a user whose eyeballs often move, the switching process occurs frequently and the image is likely to shake, but by extending the standby period, unnecessary switching processes can be suppressed and the visual recognition of the image can be improved.

[0225] For example, in the case of the HMD 10 equipped with both a head tracking function and an eye tracking function such as an angular velocity sensor, it is possible to select by the user's operation which of the above two functions to use in the determination of whether a switching process of the overlapping region AO is required. In this case, the standby period is changed according to the user's selection.

[0226] Next, with reference to Figure 27 , an example of the process in which the above-mentioned time lag is set will be described. Figure 27 is a flowchart showing an example of the switching process of the live-action image displayed in the overlapping region of the display control device 20.

[0227] The control unit 30 determines the line-of-sight direction S based on the detection information from the sensor 12 (such as an angular velocity sensor) of the HMD 10, and determines whether it is necessary to perform a switching process on the captured image displayed in the overlapping area AO (S500). When the control unit 30 determines that the switching process is necessary (Yes in S500), it sets a standby period (such as 0.5 seconds) and starts timing (S502). Then, the control unit 30 does not perform the switching process for the overlapping area AO until the standby period has elapsed. After the standby period has elapsed (Yes in S504), the control unit 30 determines the line-of-sight direction S again based on the detection information from the sensor 12 of the HMD 10, and determines whether it is necessary to perform a switching process on the captured image displayed in the overlapping area AO (S506). When the control unit 30 determines that the switching process is necessary (Yes in S506), it performs the switching process (S508). On the other hand, when the control unit 30 determines that the switching process is not necessary (No in S506), it ends the process without performing the switching process. As an example of when the switching process is necessary, as described above, the overlapping area AO of the object may sometimes have fallen out of the range of the VR image displayed on the display unit 11 of the HMD 10. In addition, after it is determined that the switching process for the overlapping area AO is necessary because the line-of-sight direction S temporarily exceeds the reference direction RD, if the line-of-sight direction S exceeds the reference direction RD again and returns to the vicinity of the original position before the standby period has elapsed, the switching process is not necessary either.

[0228] [11. Method of changing the configuration state of an image in a VR space]

[0229] For example, in Figures 9 - 11 's example, the number of field-of-view areas (number of images) arranged in the VR space is set to 2, and the entire field-of-view area with a horizontal field-of-view angle of 180 degrees is divided into left and right 2 areas, but they are not fixed and can also change according to the scene (scene), content, etc. of the image of the display object. For example, when switching scenes in a game or other content using a captured image, the number of field-of-view areas (number of images) arranged in the VR space, the field-of-view angle of the entire field-of-view area, or the direction (position in the VR space) of each field-of-view area in the VR space can also be changed in combination with the content of the scene.

[0230] For example, as Figure 28As illustrated, in a certain scenario, as illustrated in (A) in the figure, the entire 270-degree field of view region is divided into three regions. In another scenario, as illustrated in (B) in the figure, the entire 220-degree field of view region is divided into two regions. Then, according to the scenario, it is set to the configuration state of (A) or the configuration state of (B). For example, basically adopt the configuration of (A), but in a scenario where only the front region needs to be focused on and the image of the rear region does not need to be focused on, adopt the configuration of (B).

[0231] In another scenario, as Figure 9 illustrated, the entire 180-degree field of view region can also be divided into two regions. In addition, there can also be a scenario where the entire field of view region is not divided and one part includes one region (for example, only one entire 140-degree field of view region).

[0232] For example, in a certain scenario, use Figure 26 the three cameras 200 shown to capture three images of the VR space illustrated in (A) in Figure 28 . In addition, in another scenario, only use two adjacent ones of the above three cameras 200 to capture two images of the VR space illustrated in (B) in Figure 28 . In this case, at least one camera lens among the three cameras 200 can also be replaced as needed to enlarge (or reduce) the field of view angle, or adjust the shooting direction (the direction of the optical axis) of each camera 200. For example, when switching from an indoor scenario to an outdoor scenario, or from the scenario of one room to the scenario of another room, etc., the installation positions of the cameras 200 for capturing each scenario also change. Therefore, it is only necessary to capture the actual shooting images in consideration of the configuration state in the VR space suitable for each scenario.

[0233] As described above, the configuration information for arranging the data of the plurality of actual shooting images and the plurality of actual shooting images of the display object in the VR space is stored in the storage device 22 in association with the plurality of actual shooting images. For example, for each scenario where the shooting location of the camera in the actual space changes, the above configuration information corresponding to the shooting conditions in each scenario is generated and recorded together with the actual shooting images in association with each scenario. In this way, when appropriately changing the configuration state of the plurality of actual shooting images in the VR space according to the scenario, the configuration information associated with each scenario is stored in the storage device 22. Or, the configuration information for each scenario is recorded in the title or the like of the actual shooting image file, and the configuration information is read out and used when reproducing the actual shooting image.

[0234] The image generation unit 32 of this method includes an image configuration unit 321, which changes the configuration state of a plurality of real-shot images in the VR space during the display control of the VR image on the display unit 11. Here, the content that becomes the object of change in the configuration state in the VR space is, for example, the number of field-of-view regions (number of images) configured in the VR space, the field-of-view angle of the entire field-of-view region, the field-of-view angle of each field-of-view region, the direction (position) of each field-of-view region in the VR space, the number of overlapping regions, the field-of-view angle of each overlapping region, or the direction (position) of each overlapping region in the VR space, etc. These pieces of information are recorded as configuration information in association with the plurality of real-shot images to be displayed (for example, in association with each scene of the plurality of real-shot images).

[0235] For example, in Figure 28 the configuration state of the VR space in (A) below, the number of field-of-view regions (number of images) = 3, the field-of-view angle of the entire field-of-view region is "θall = 270 degrees", the direction (position in the VR space) of the entire field-of-view region is "-135 degrees to 135 degrees", the field-of-view angles of the left, front, and right field-of-view regions AL, AF, and AR are "θAL = θAF = θAR = 120 degrees", the direction of the left field-of-view region AL is "15 degrees to 135 degrees", the direction of the front field-of-view region AF is "-60 degrees to 60 degrees", the direction of the right field-of-view region AR is "-135 degrees to -15 degrees", the field-of-view angle of the first overlapping region AO1 is "θAO1 = 45 degrees", the direction of the first overlapping region AO1 is "15 degrees to 60 degrees", the field-of-view angle of the second overlapping region AO2 is "θAO2 = 45 degrees", and the direction of the second overlapping region AO1 is "-60 to -15 degrees". All or part of this information is recorded as configuration information. The angles of the directions shown here are the angles in the horizontal direction with the X-axis direction (reference line-of-sight direction) set to 0 degrees, and the angles in the vertical direction are omitted here.

[0236] In addition, in Figure 28 the configuration state of the VR space in (B) below, the number of field-of-view regions (number of images) = 2, the field-of-view angle of the entire field-of-view region is "θall = 220 degrees", the direction of the entire field-of-view region is "-110 degrees to 110 degrees", the field-of-view angles of the left / right field-of-view regions AL and AR are "θAL = θAR = 135 degrees", etc. (the following is omitted) information is recorded as configuration information.

[0237] The image configuration unit 321 determines whether it is necessary to change the configuration state of the plurality of real-shot images to be displayed in the VR space according to the configuration information associated with the plurality of real-shot images, and appropriately changes the configuration state according to this configuration information.

[0238] In the image display system 1 of the present embodiment, as described above, instead of pre-generating one VR image data completed by stitching processing, a plurality of actual captured images are arranged in the VR space, and VR image display control is performed simultaneously. Therefore, during the reproduction process of the VR image, as shown in this embodiment, the arrangement state of the actual captured images in the VR space can also be appropriately changed. For example, it is possible to change the number of field-of-view regions (the number of actual captured images) arranged in the VR space, the field-of-view angle of the entire field-of-view region, etc. according to the scene. That is, it is possible to optimize the VR space for each scene. Therefore, it is possible to exclude unnecessary field-of-view regions for each scene, and the amount of image data can be reduced.

[0239] In addition, by excluding unnecessary field-of-view regions for each scene, not only can the data capacity be reduced, but also in the scene where unnecessary field-of-view regions are excluded, the resolution of the VR image displayed on the display unit 11 of the HMD 10 can be improved. That is, by excluding unnecessary field-of-view regions, correspondingly, a VR image with a high resolution can be expanded to VRAM or the like. For example, compared with Figure 28 the arrangement state of the VR space in (A), in the arrangement state of (B), a VR image with a high resolution is displayed on the display unit 11 of the HMD 10. Therefore, it is also possible to optimize the VR space for each scene by adopting the arrangement state of (B) in Figure 28 in the scene where the resolution of the image is desired to be improved, and adopting the arrangement state of (A) in other scenes.

[0240] Next, with reference to Figure 29 , an example of the processing of this embodiment will be described. Figure 29 is a flowchart showing an example of the processing of changing the arrangement state of an image in the VR space in the display control device 20.

[0241] The control unit 30 acquires the configuration information of a plurality of actual captured images of the display object (S600), and determines the configuration state of the plurality of actual captured images in the VR space based on this configuration information (S602). When reproducing the VR image, the configuration state in the VR space determined here is applied, and the images are configured in the VR space in such a way that a part of the field-of-view regions of adjacent images overlap with each other to generate an overlapping region AO. In addition, as described above, according to the line-of-sight direction S (for example, the optical axis direction of the HMD 10), the actual captured image displayed in the overlapping region AO is dynamically switched to generate a VR image. In addition, when the scene is changed during the reproduction of the VR image (Yes in S604), the control unit 30 acquires the configuration information of the changed scene (S606), and determines whether the configuration state of the images in the VR space should be changed (S608). Here, when the control unit 30 determines that the configuration state needs to be changed (Yes in S608), it changes the configuration state of the plurality of actual captured images in the VR space (S610). Thereby, after the scene is changed, the configuration state in the VR space suitable for the scene is applied, and after the images of the scene are configured in the VR space, a VR image is generated. The processing of S604 to S610 is repeated until the display ends (Yes in S612).

[0242] [12. Way of changing the direction of the entire field-of-view region of the VR space]

[0243] In the above description, regarding the cameras that capture a plurality of actual captured images of the display object, the change in the direction of the cameras during actual space shooting is not considered, but the orientation of the cameras can also be changed during shooting. In this case, the change in the shooting direction during the shooting process can also be reflected in the field-of-view region of the VR space.

[0244] For example, as Figure 26 illustrated, it is assumed that there is a relationship between the three cameras 200 that shoot the actual space and the field-of-view regions of the VR space V where the images captured by the three cameras 200 are configured. That is, Figure 26 the shooting direction of each camera 200 is the reference shooting direction of each camera, and the reference shooting direction of the front camera 200 corresponds to the X-axis direction (reference line-of-sight direction) of the VR space V. Here, in Figure 30 it is illustrated that the three cameras 200 in Figure 26 are rotated in the right direction. Figure 30 It is illustrated that the entire field-of-view region of the VR space V where the images captured by the three cameras 200 are configured also rotates according to the change in the shooting direction (rotation of the shooting direction) of the three cameras 200 during shooting.

[0245] In Figure 26In the example, in the case of an image captured by the camera 200 in the reference shooting direction, the X-axis direction (the reference line-of-sight direction) is set to 0 degrees, and the direction of the entire field-of-view area in the VR space V is “-135 degrees to 135 degrees”. On the other hand, in Figure 30 the example, the shooting direction of the camera 200 rotates 45 degrees (i.e., -45 degrees) to the right from the Figure 26 reference shooting direction. In the case of an image captured by this Figure 30 camera 200, the direction of the entire field-of-view area in the VR space V rotates 45 degrees to the right and changes to “-180 degrees to 90 degrees”.

[0246] The change information of the shooting direction of the camera 200 can also be detected from an angular velocity sensor (gyro sensor) etc. mounted on the camera 200 during shooting and recorded together with the image data during shooting. Alternatively, after shooting, the change in the shooting direction can be determined by known image analysis based on the captured image, and the change information of the shooting direction can be recorded in association with the captured image. Alternatively, the display control device 20 can also determine the change in the shooting direction in real time by image analysis based on the captured image during the reproduction process of the image.

[0247] For example, consider the following situation: In the real space, in a scene where a vehicle passes in front of the eyes from left to right, the camera changes its orientation from left to right as the vehicle moves and shoots at the same time. Figure 31 An example of the change in the field-of-view area of the VR space V in which the image captured by the camera 200 is arranged without using the rotation information of the shooting direction of the camera 200 is shown. Without using the rotation information of the shooting direction, even if the camera 200 rotates, the direction of the entire field-of-view area does not change, and the shooting direction of the front camera 200 always corresponds to the X-axis direction of the VR space V. Therefore, when the captured image is displayed on the HMD 10 without using the rotation information of the shooting direction, the image of the vehicle can be displayed in such a way that it does not cross the area boundary of adjacent images. However, although the vehicle is moving, since it is displayed in the same direction with respect to the visual axis (line-of-sight direction S) of the HMD 10, it is different from reality and the sense of reality cannot be maintained.

[0248] Figure 32An example of the change in the field-of-view area of the VR space V in which the images captured by the camera 200 are arranged is shown when using the rotation information of the shooting direction of the camera 200. When using the rotation information of the shooting direction, the direction of the entire field-of-view area in the VR space V also rotates according to the rotation of the shooting direction. In this case, when the captured image is displayed on the HMD 10, an image of the vehicle moving from left to right is also displayed with respect to the visual axis (line-of-sight direction S) of the HMD 10. Therefore, during the reproduction process of the image, when the vehicle is present in the left direction and the user does not turn the HMD 10 towards the left direction, the vehicle cannot be seen (or, when the user is facing forward, the vehicle is seen in the left direction). The user can fully see the vehicle by turning the head towards the direction where the vehicle is located, and a VR image with a sense of reality can be displayed. Also, the image of the vehicle can be displayed on the display unit 11 of the HMD 10 in such a way that it does not cross the regional boundary of adjacent images (the boundary line in the case where the boundary line is displayed).

[0249] As Figure 20 illustrated by way of example, the control unit 30 of the display control device 20 of this method can be configured to include a shooting direction determination unit 34. The shooting direction determination unit 34 has a function of determining the change in the shooting direction of a plurality of cameras that have captured a plurality of actual shooting images. For example, the shooting direction determination unit 34 can read the "shooting direction change information" recorded in association with the plurality of actual shooting images and determine the change in the shooting direction. Alternatively, the shooting direction determination unit 34 can also determine the change in the shooting direction by image analysis based on the above-mentioned actual shooting images.

[0250] Then, the image generation unit 32 has the following function: according to the change in the shooting direction determined by the shooting direction determination unit 34, change the direction of the entire field-of-view area (the entire field-of-view area) of the plurality of actual shooting images arranged in the VR space.

[0251] Next, with reference to Figure 33 , an example of the processing of this method will be described. Figure 33 is a flowchart showing an example of the process of changing the direction of the entire field-of-view area of the VR space in the display control device 20.

[0252] When starting the reproduction of the VR image, the control unit 30 initializes the direction of the entire field-of-view area in the VR space to the direction corresponding to the reference shooting direction (S700). For example, as Figure 26As illustrated, the horizontal direction of the entire field of view region is default set to "-135 degrees to 135 degrees", and the direction of the center of the front field of view region AF from the virtual viewpoint P becomes the X-axis direction (reference line-of-sight direction). In addition, the control unit 30 acquires the change information of the shooting directions of a plurality of actual shooting images of the display object (S702), and determines the change in the shooting direction (S704). For example, the control unit 30 determines the change in the shooting direction by, for example, reading and acquiring the change information of the shooting direction recorded together with the data of a plurality of actual shooting images of the display object for each frame. Then, when it is determined that the shooting direction has changed compared to the previous frame (Yes in S706), the control unit 30 changes the direction of the entire field of view region in the VR space according to the change in the shooting direction (S708). On the other hand, when it is determined that the shooting direction has not changed (No in S706), the process returns to step S702. The processing of S702 to S708 is repeated until the display ends (Yes in S710).

[0253] [13. Summary]

[0254] As described above, the program of the present embodiment is a program for causing the display control device 20 (an example of a computer) to function as the line-of-sight direction determination unit 31 and the image generation unit 32, and the display control device 20 executes control to display a VR image representing the field of view from the virtual viewpoint P in the VR space V as a stereoscopic image utilizing binocular parallax on the display unit 11 of the HMD 10.

[0255] The display control device 20 of the present embodiment includes a line-of-sight direction determination unit 31 and an image generation unit 32. The line-of-sight direction determination unit 31 determines the line-of-sight direction S, which is set as the direction of the user's line of sight with respect to the above VR space. The image generation unit 32 arranges a plurality of actual shooting images captured by a plurality of cameras with different shooting directions of the actual space in the VR space V in such a way that a part of the field of view regions of adjacent actual shooting images overlaps to form an overlapping region AO, and generates the above VR image corresponding to the above line-of-sight direction S. Here, the image generation unit 32 dynamically switches the actual shooting images displayed in the above overlapping region AO among the adjacent actual shooting images according to the above line-of-sight direction S.

[0256] Here, the "VR image" of this structure is a stereoscopically viewable live-action image utilizing binocular parallax that is displayed on the display unit 11 of the HMD 10. However, display objects other than live-action can also be included in the "VR image". For example, display objects other than live-action such as CG (Computer Graphics), various objects, lines, symbols, characters, etc. can also be arranged in the VR space, and they can be overlapped with the live-action image to form the "VR image". The "VR image" includes AR (Augmented Reality) images or MR (Mixed Reality) images. The "VR image" can be a static image or a dynamic image. For example, when the "VR image" is provided (transmitted) to the display unit 11 of the HMD 10, it can also be appropriately converted into a transmission format corresponding to the HMD 10.

[0257] In addition, for example, as Figure 34 shown, the "HMD" can be the HMD 50, which includes: a fitting (wearing device) 51 that can be worn on the user's head; and an information processing device 52 such as a smartphone that is installed on the fitting 51. In this case, the display unit 61 of the information processing device 52 becomes the display unit of the HMD 50. In addition, the "HMD" can also be a so-called standalone HMD obtained by integrating the functions of the display control device 20 and the HMD 10. In addition, the "HMD" can also be a goggle-type or glasses-type that can be worn on the head. In addition, the "HMD" is not limited to the narrow sense of the HMD. For example, a head-mounted earphone, an earphone (head-mounted earphone with a microphone), a glasses-type camera, an ear-hung camera, a hat with a camera, etc. can also have the function of an HMD.

[0258] In addition, for the "display control device" of this embodiment, as long as it is a device with information processing functions such as the generation of VR images, various devices can be applied. For example, a desktop or portable game dedicated device, a business-use (commercial) game console, a personal computer, a tablet computer, a smartphone, a mobile phone terminal, a PHS terminal, a PDA, a multifunctional television receiver with information processing functions, etc. can be set as the "display control device". In addition, when the HMD itself has information processing functions such as the generation of VR images, the HMD can become the "display control device" of this structure. For example, as Figure 34 illustrated, in the case of the HMD 50 including the accessory 51 and the information processing device 52, the information processing device 52 such as a smartphone is an example of the "display control device" of this embodiment. In addition, the above-mentioned standalone HMD is an example of the "display control device" of this embodiment.

[0259] According to the above structure, by dynamically switching the live-action images displayed in the overlapping area AO according to the line-of-sight direction S, it is possible to move the area boundaries (BD1 or BD2) of adjacent live-action images in the overlapping area AO away from the direction of the user's line of sight. As a result, it is possible to reduce the discomfort of the "seam (misalignment) between adjacent live-action images" generated at the area boundary. As described above, in the prior art that performs splicing processing, if a close-up object at a certain distance or less from the camera (for example, 2 m or less) is located at the seam of the image, splicing becomes difficult and VR image data cannot be generated. In contrast, according to this structure, a close-up object at a certain distance or less from the camera is included in the live-action image, and even if the close-up object exists at or near the area boundary, a stereoscopic VR image with less discomfort for the user is displayed. That is, regardless of the position of the close-up object in the VR space, a VR image with less discomfort can be displayed, and VR image display control that can be displayed at a close distance can be realized.

[0260] In addition, since there is no need for a splicing process, the effort required for VR image production can also be reduced.

[0261] In addition, since there is no need for a splicing process, the live-action images currently being captured by the camera (live video) can be used, and a VR image with less discomfort at the seam between the live-action images can be displayed on the display unit 11 approximately in real time.

[0262] In addition, the above image generation unit 32 may also set the reference direction RD as a specified direction starting from the virtual viewpoint P, and determine the live-action image to be displayed in the overlapping area AO according to the line-of-sight direction S with respect to the reference direction RD. As a result, it is possible to easily realize the switching control of the live-action image displayed in the overlapping area AO.

[0263] In addition, the above image generation unit 32 may also set the reference direction RD as the direction of a specified position within the overlapping area AO starting from the virtual viewpoint P. As a result, since the reference direction RD is set within the overlapping area AO, even if the line-of-sight direction S faces the area boundary (BD1 or BD2) of the overlapping area AO, since the live-action image displayed in the overlapping area is switched before reaching the area boundary, the area boundary is also moved away from the direction of the user's line of sight.

[0264] In addition, the above image generation unit 32 may also change the reference direction RD according to the live-action image displayed in the overlapping area AO when switching the live-action image displayed in the overlapping area AO. As a result, after switching the live-action image in the overlapping area AO, the reference direction RD temporarily moves away from the line-of-sight direction S, so it is possible to reduce the frequent switching of the live-action image displayed in the overlapping area AO.

[0265] In addition, the above-mentioned image generation unit 32 may also determine the live-action image to be displayed in the overlapping area AO based on the line-of-sight direction S with respect to the current area boundary (BD1 or BD2) of the adjacent live-action image. Thereby, even without setting a reference direction, it is possible to easily implement the switching control of the live-action image displayed in the overlapping area AO. In addition, since the area boundary serving as a reference is changed when the live-action image displayed in the overlapping area AO is switched, the same effect as the above-described structure for changing the reference direction RD is achieved. That is, it is possible to reduce the frequent switching of the live-action image displayed in the overlapping area AO.

[0266] In addition, the above-mentioned image generation unit 32 may also switch the live-action image currently displayed in the overlapping area AO to another live-action image when the line-of-sight direction S approaches the current area boundary (BD1 or BD2) of the adjacent live-action image by a specified amount or more (for example, when the angle formed by the line-of-sight direction S and the current area boundary is equal to or less than a specified value). In this case, even without setting a reference direction, it is possible to easily implement the switching control of the live-action image displayed in the overlapping area AO. In addition, by switching the live-action image displayed in the overlapping area before the line-of-sight direction S reaches the area boundary, it is possible to move the area boundary away from the direction of the user's line of sight.

[0267] In addition, the above-mentioned line-of-sight direction determination unit 31 may also obtain detection information related to the orientation of the HMD 10 (for example, detection information of an angular velocity sensor), and determine the line-of-sight direction S based on this detection information. Thereby, the user can dynamically switch the live-action image displayed in the overlapping area AO by changing the orientation of the HMD 10.

[0268] In addition, the above-mentioned line-of-sight direction determination unit 31 may also obtain detection information related to the user's line of sight (for example, detection information of eye movement tracking), and determine the line-of-sight direction S based on this detection information. Thereby, the user can dynamically switch the live-action image displayed in the overlapping area AO by changing the movement of the eyes (the orientation of the line of sight).

[0269] In addition, the above-described image generation unit 32 may also acquire detection information related to the orientation of the HMD 10, and change the range of the VR image displayed on the display unit 11 according to the detection information. With this configuration, the range of the VR image displayed on the display unit 11 is changed according to the orientation of the HMD 10, and further, in the VR image displayed on the display unit 11, the live-action image displayed in the overlapping area AO is dynamically switched according to the line-of-sight direction S. Here, when the line-of-sight direction S is determined based on the detection information related to the orientation of the HMD 10, the user can change the range of the VR image displayed on the display unit 11 by adjusting the orientation of the HMD 10, and at the same time, dynamically switch the live-action image displayed in the overlapping area AO. In addition, when the line-of-sight direction S is determined based on the detection information related to the user's line of sight (for example, the detection information of eye tracking), the user can adjust the range of the VR image displayed on the display unit 11 by changing the orientation of the HMD 10, and in addition, by changing the movement of the eyes (the orientation of the line of sight), dynamically switch the live-action image displayed in the overlapping area AO.

[0270] In addition, the display control device 20 may also have a configuration including a boundary line display unit 33. The boundary line display unit 33 displays a boundary line (BL1 or BL1) that can be visually recognized by the user at the region boundary (BD1 or BD2) between adjacent live-action images. In this way, by deliberately displaying the boundary line in a visually recognizable manner at the region boundary, it is easy for the user to identify the region boundary where the image seam (misalignment) occurs. As a result, the user can easily determine which live-action image among adjacent live-action images should be displayed in the overlapping area for viewing, so that the boundary line is not displayed in the direction the user wants to observe.

[0271] In addition, the boundary line display unit 33 may also make the display mode of the boundary line (BL1 or BL2) at one end of the overlapping area AO different from the display mode of the boundary line at the other end. Thereby, the user can identify whether the currently displayed boundary line is the misaligned part of the image at one end (for example, the left end) of the overlapping area or the misaligned part of the image at the other end (for example, the right end).

[0272] In addition, the boundary line display unit 33 may also display additional information (for example, a gradient part (GD1 or GD2)) indicating which side of the boundary line is the overlapping area AO on or near the boundary line (BL1 or BL1). In this way, the user can identify which side of the boundary line is the overlapping area, and thus it is easy for the user to judge which direction the line-of-sight direction S should be changed to in order to switch the live-action image displayed in the overlapping area AO.

[0273] In addition, the above additional information can be set as a semi-transparent gradient part (GD1 or GD2), which overlaps with the real-shot image near the boundary line. The farther away from the boundary line, the concentration or opacity decreases continuously or stepwise. This gradient part has the effect of hiding the misalignment of the image generated near the boundary line (BL1 or BL2) and making it less noticeable. In addition, the change in the concentration or opacity of the gradient part (from thick to thin or from thin to thick) serves to naturally indicate the direction of change in the line of sight S required for the switching of the real-shot image in the overlapping area AO.

[0274] In addition, when the image generation unit 32 switches the real-shot image displayed in the overlapping area AO, it can also gradually switch from one real-shot image to another over time. For example, during the switching period (e.g., 0.3 seconds), the two real-shot images to be switched can be alpha-blended to fade them out crosswise. When switching the real-shot image in the overlapping area AO from one to another, compared with a sudden switch, by gradually switching during the time course from the start to the end of the switching period, it is easier to visually recognize the boundary line more naturally.

[0275] In addition, when the image generation unit 32 switches the real-shot image displayed in the overlapping area AO, it can also move the boundary line (BL1 or BL1) to the position to be displayed after the switching is completed over time, while gradually switching from one real-shot image to another. In this way, during the switching period of the real-shot image displayed in the overlapping area AO, by moving the boundary line, the user can naturally recognize the position of the changed boundary line.

[0276] In addition, the image generation unit 32 can also set a prescribed standby period according to the line of sight direction, from when it is determined that the switching process of the real-shot image to be displayed in the overlapping area AO should be performed to the start of this switching process. Thus, in a situation where the line of sight direction changes during the standby period and there is no need to perform the switching process (for example, when the overlapping area AO has already moved out of the range of the VR image displayed on the display unit 11 of the HMD 10 when the standby period has elapsed), unnecessary switching processes can be omitted.

[0277] In addition, in the case of the HMD 10 equipped with both a head tracking function and an eye tracking function including an angular velocity sensor or the like, the following processing can also be performed. That is, the image generation unit 32 can also change, according to the user's selection operation, which one of the detection information related to the orientation of the HMD 10 and the detection information related to the user's line of sight (for example, the detection information of eye tracking) is used to determine the line of sight direction S. In this case, the standby period can also be changed according to the user's selection. For example, when it is selected to determine the line of sight direction S based on the detection information related to the user's line of sight, the standby period can be further extended. In addition, since the movement patterns of the head and eyes vary from user to user, the user can arbitrarily set the above-mentioned standby period. For example, it can be that the user can set the standby period within a specified range (for example, in the range of 0.1 second to 1.0 second). In addition, it can also be that the user can separately set the standby period in the case of head tracking and the standby period in the case of eye tracking.

[0278] In addition, the image generation unit 32 can also change the arrangement state of a plurality of real-shot images in the VR space during the display control of the VR image on the display unit 11. For example, it can be, as Figure 28 shown, in a certain scene, for example, set to an arrangement state in which the entire 270-degree field of view area is divided into three areas, or in another scene, for example, set to an arrangement state in which the entire 220-degree field of view area is divided into two areas, etc., and change the number of field of view areas, the direction of the field of view area, etc. of the VR space according to the content of the scene.

[0279] In the display control of the VR image of the present embodiment, instead of pre-generating one VR image data completed by stitching processing as in the past, a plurality of real-shot images are arranged in the VR space, and the display control of the VR image is performed at the same time. Therefore, during the reproduction process of the VR image, the arrangement state of the real-shot images in the VR space can also be appropriately changed. For example, as described above, the number of field of view areas (the number of real-shot images) arranged in the VR space, the field of view angle of the entire field of view area, etc. can be changed according to the scene. That is, optimization of the VR space for each scene can be achieved. Therefore, unnecessary field of view areas can be excluded for each scene, and the amount of image data can be reduced. In addition, in a scene where necessary field of view areas are excluded, the resolution of the VR image displayed on the display unit 11 of the HMD 10 can be improved.

[0280] In addition, the display control device 20 may also have a structure including a photographing direction determination unit 34. The photographing direction determination unit 34 determines changes in the photographing directions of a plurality of cameras that have photographed a plurality of actual photographed images. Then, the image generation unit 32 changes the direction of the entire field-of-view area of the plurality of actual photographed images arranged in the VR space according to the change in the photographing direction determined by the photographing direction determination unit 34. According to this structure, for example, in the case of actual photographed images where the photographing direction of the camera changes in accordance with the movement of a moving subject (such as a vehicle), the direction of the entire field-of-view area of the actual photographed images arranged in the VR space also changes according to the change in the photographing direction. Therefore, a VR image with a sense of reality that the subject moves relative to the line-of-sight direction S can be displayed on the display unit 11 of the HMD 10. In addition, if actual photographed images obtained by photographing the subject in such a way that the area boundary between adjacent actual photographed images is not crossed are prepared during photographing in accordance with the movement of the above-described subject, a VR image with excellent visual recognition in which the subject does not cross the area boundary can be reproduced.

[0281] [14. Modifications, etc.]

[0282] As described above, the embodiments of the present invention have been described, but the specific structure is not limited to the above-described embodiments, and also includes designs and the like within the scope not departing from the gist of the present invention. In addition, the above-described respective structures and modes can be arbitrarily combined.

[0283] [14-1] As described above, the image display system or the display control device of the present embodiment can reduce the discomfort of seams (misalignment) between a plurality of actual photographed images captured by a plurality of cameras even without performing a stitching process by dynamically switching the display control of the actual photographed images displayed in the overlapping area. Therefore, it is also possible to directly use the actual photographed images (live images) captured in real time by the cameras photographing the actual space and display them on the HMD 10 as VR images.

[0284] For example, it is also possible to connect the Figure 26 illustrated camera 200 to the display control device 20 in a wired or wireless manner and display the live image captured by the camera 200 on the HMD 10 as a VR image. In addition, the camera 200 and the display control device 20 may be connected via a network such as the Internet in a manner capable of data communication. In this case, it is also possible to display the remote live image captured by the camera 200 on the HMD 10 as a VR image. In addition, a server having a distribution service function may be set up on the network, and the real-time image captured by the camera 200 may be distributed via the network by the server. In this case, it is also possible to receive the remote real-time image distributed via the network by the display control device 20 and display the real-time image on the HMD 10 as a VR image substantially in real time.

[0285] That is, a system for displaying a live video including a close-up object as a VR image on an HMD can be constructed by an image display system including a plurality of cameras with different shooting directions for shooting an actual space, a display control device having the above structure, and an HMD, or an image display system including the above plurality of cameras, a distribution server, a display control device having the above structure, and an HMD.

[0286] [14-2] As described above, the display control for displaying a VR image representing a field of view from a virtual viewpoint in a VR space as a stereoscopic image using binocular parallax on the display unit 11 of the HMD 10 has been described, but it can also be applied to the display control of a non-stereoscopic image that does not use binocular parallax, that is, a two-dimensional (2D) image. That is, in the case of using binocular parallax, all or part of the above-described respective processes are executed for the right-eye image and the left-eye image, respectively. However, in a 2D image, all or part of the above-described respective processes are executed for an image shared by the right eye and the left eye. There is no difference in the processing itself between the stereoscopic image (right-eye image or left-eye image) and the non-stereoscopic image (image shared by left and right), and the above-described respective structures can also be directly applied to the display control of the non-stereoscopic image.

[0287] That is, it can be applied to a display control device that executes control for displaying a VR image (3D image or 2D image) representing a field of view from a virtual viewpoint in a VR space as a stereoscopic image or a non-stereoscopic image on the display unit of the HMD. Conventionally, in any of 3D images or 2D images, when generating a wide-angle-of-view image (moving image or still image) by synthesizing a plurality of actual captured images obtained by capturing an actual space with a plurality of cameras, one image data completed by a stitching process is generated. In contrast, in the present embodiment, by arranging a plurality of actual captured images captured by a plurality of cameras with different shooting directions for shooting an actual space in a VR space in such a way that a part of the field-of-view regions of adjacent actual captured images overlap each other to generate an overlapping region, and dynamically switching the actual captured image to be displayed in the overlapping region among adjacent actual captured images according to the line-of-sight direction, display control with reduced discomfort due to seams (misalignment) between actual captured images can be achieved without performing a stitching process.

[0288] [14-3] As described above, the display control for displaying an image on the display unit of the HMD has been described, but it can also be applied to the display control for displaying an image on a display unit other than the HMD. For example, hereinafter, the application in the case of displaying an actual captured image with a high field of view on the display unit of a normal display or the like that is not worn on the user's head will be described.

[0289] When displaying an image with a high viewing angle (e.g., an image with a 360-degree viewing angle) that uses multiple actual shooting images captured by multiple cameras of an actual space on a display unit, a user, for example, uses a mouse, other pointing devices, etc. to perform an operation of changing the range of the image displayed on the display unit while observing the image. Or, when the display unit is a touch panel or the like having a touch interface, the range of the image displayed on the display unit is changed by a touch operation using a finger or a stylus while observing the image with a large viewing angle. Or, a hand tracking system that captures the user and analyzes the movement of the hand by a camera may be used to perform an operation of changing the range of the image displayed on the display unit by the gesture of the user's hand while observing the image.

[0290] In this case, it is determined that the center (center of the screen) of the current image range displayed on the display unit has the user's viewing point or line of sight, and the line-of-sight direction is determined. Or, for example, detection information related to the user's line of sight may be obtained by an eye movement tracking system or the like provided on the display unit or the like, and the line-of-sight direction may be determined based on the detection information.

[0291] Thus, when displaying an image with a high viewing angle that uses multiple actual shooting images captured by multiple cameras with different shooting directions of an actual space on a display unit other than an HMD, the following display control may be performed. That is, the above-mentioned multiple actual shooting images are arranged in a VR space in such a way that a part of the viewing field regions of adjacent actual shooting images overlaps with each other, and the actual shooting image displayed in the overlapping region among the adjacent actual shooting images is dynamically switched according to the line-of-sight direction. Through this display control, even on a display unit other than an HMD, even without performing a stitching process, it is possible to achieve display control that reduces the discomfort of seams (misalignment) between actual shooting images.

[0292] In addition, the image with a high viewing angle displayed on a display unit other than an HMD may be a stereoscopic image using binocular parallax or a non-stereoscopic image. In the case of displaying a stereoscopic image, the right-eye image and the left-eye image are displayed on a display unit that can be observed in a naked-eye stereoscopic manner. Or, on the premise of using special glasses (such as special glasses in a frame sequential method or a polarization method), the right-eye image and the left-eye image are alternately displayed on the display unit.

[0293] [14-4]

[0294] As Figure 1As exemplified, when both the HMD 10 and the display control device 20 have the structure / function of an information processing device (computer) having a processor and a storage device, a part of the functions of the above-described control unit 30 can also be implemented by the processor 13 of the HMD 10 that executes the program of the present embodiment, and the remaining functions can be implemented by the processor 21 of the display control device 20 that executes the program of the present embodiment.

[0295] Figure 35 It shows an example of a schematic block diagram of the structure of a stand-alone HMD 60 obtained by integrating the functions of the display control device 20 with the HMD 10 or an information processing device 52 such as a smart phone used as the Figure 34 exemplified HMD 50. The HMD 60 or the information processing device 52 includes a display unit 61, a sensor 62, a processor 63, a storage device 64, an operation unit 65, a communication unit 66, and the like. These display unit 61, sensor 62, processor 63, storage device 64, operation unit 65, and communication unit 66 have the same structure as the above-described display unit 11, sensor 12, processor 21, storage device 22, operation unit 23, and communication unit 24, and the description thereof is omitted. In addition, the operation unit 65 or the communication unit 66 may be configured based on other peripherals or the like, or may be omitted. In the case of this stand-alone HMD 60 or information processing device 52, it also has the structure / function of an information processing device (computer) having a processor and a storage device, and each function of the above-described control unit 30 is implemented by the processor 63 that executes the program of the present embodiment.

[0296] [14-5] A part or all of the functions of the above-described control unit 30 can also be implemented by an integrated circuit such as an LSI (Large Scale Integration). In addition, the above-described respective functions can be individually modularized. Alternatively, a part or all of the above functions can be integrated and then modularized.

[0297] [14-6] The computer-readable program of this embodiment is recorded on various computer-readable recording media such as hard disks, optical discs (CD-ROM, DVD-ROM, etc.), floppy disks, and semiconductor memories, and read out from the recording media, and executed by the computer constituting the image display system 1 or the display control device 20. In addition, the program can also be provided to the computer via a network including communication lines such as the Internet, WAN, LAN, or dedicated lines. The computer can read the program stored in the file server (online memory). In addition, the computer can receive the program published from the publishing server. The above-mentioned recording media also includes an internal or external recording media accessible from the publishing server for publishing the program. The program code stored in the recording media of the publishing server may not be in a form that can be directly executed in the computer that receives the program. That is, as long as it can be installed in a form that can be executed in the computer after being downloaded from the publishing server, the form of the program stored in the recording media of the publishing server is arbitrary. In addition, the program can also be divided into multiple parts and combined after being downloaded at different times. The publishing servers for publishing each divided program can be different. In addition, the computer-readable recording media also includes a medium that holds the program for a certain period of time, such as a volatile memory such as RAM in the server that sends the program or the computer that receives the program via the network. In addition, the program can also be a differential program that can achieve the above functions through combination with the program already stored in the computer.

[0298] [15. Remarks]

[0299] According to the above description, for example, the present invention can be understood as follows. In addition, for the convenience of understanding the present invention, reference numerals are simply attached in parentheses for explanation, but it does not mean that the present invention is limited to the forms shown in the drawings.

[0300] 1) A program according to one embodiment of the present invention causes a computer that performs the following control to function as the following components. The control is for displaying a VR image representing a field of view from a virtual viewpoint (P) in a virtual reality (VR) space (V) as a stereoscopic image utilizing binocular parallax on a display unit (11) of a head-mounted display (10). The components include: a line-of-sight direction determination unit (31) that determines a line-of-sight direction (S), which is set as the direction of the user's line of sight with respect to the above VR space; and an image generation unit (32) that arranges a plurality of actual captured images captured by a plurality of cameras with different shooting directions of the actual space in the above VR space in such a manner that a part of the field-of-view regions (A1, A2) of the adjacent actual captured images overlaps to generate an overlapping region (AO), and generates the above VR image corresponding to the above line-of-sight direction. The image generation unit (32) dynamically switches the actual captured images displayed in the above overlapping region among the adjacent actual captured images according to the above line-of-sight direction.

[0301] Here, the above "computer" only needs to include at least a processor and a storage device (memory). For example, devices including a desktop or portable game dedicated device, a business-use game console, a personal computer, a tablet computer, a smartphone, a mobile phone terminal, a PHS terminal, a PDA, a multi-functional television receiver having an information processing function, and other processors and storage devices that can perform display control of VR images are all included in the "computer". In addition, the HMD itself including a processor and a storage device is also included in the "computer". For example, as Figure 34 illustrated, in the case of the HMD 50 including an accessory 51 and an information processing device 52, the information processing device 52 such as a smartphone is included in the "computer". In addition, a stand-alone HMD is also included in the "computer".

[0302] According to the above-described 1), by dynamically switching the actual captured images displayed in the overlapping region according to the line-of-sight direction, it is possible to move the region boundary between the adjacent actual captured images in the overlapping region away from the direction of the user's line of sight. As a result, it is possible to reduce the discomfort of the "seam (misalignment) between adjacent actual captured images" generated at the region boundary. Therefore, even if a close-range object exists at or near the region boundary, a VR image with less discomfort for the user is displayed. That is, regardless of the position of the close-range object in the VR space, a VR image with less discomfort can be displayed, and a VR image display control that can display at a close range can be achieved. In addition, since no splicing process is required, the effort for producing VR images can also be reduced. In addition, since no splicing process is required, it is possible to use the actual captured images (live images) currently being captured by the cameras and display a VR image with less discomfort at the seam between the actual captured images on the display unit almost in real time.

[0303] 2) In one aspect of the present invention, in the aspect described in 1) above, the image generation unit (32) sets the reference direction (RD, RD1, RD2) as a specified direction starting from the virtual viewpoint (P), and determines the captured image displayed in the overlapping region (AO) based on the line-of-sight direction (S) relative to the reference direction.

[0304] According to the aspect described in 2) above, it is possible to easily achieve switching control of the captured image displayed in the overlapping region based on the line-of-sight direction relative to the reference direction.

[0305] 3) In one aspect of the present invention, in the aspect described in 2) above, the image generation unit (32) sets the reference direction (RD, RD1, RD2) as the direction of a specified position within the overlapping region (AO) starting from the virtual viewpoint (P).

[0306] According to the aspect described in 3) above, since the reference direction is set in the overlapping region, even if the line-of-sight direction faces the region boundary of the overlapping region, before reaching the region boundary, the captured image displayed in the overlapping region is switched, so that the region boundary is also away from the direction of the user's line of sight. Thus, it is possible to achieve VR image display control with excellent visual recognition.

[0307] 4) In one aspect of the present invention, in the aspect described in 2) or 3) above, when switching the captured image displayed in the overlapping region (AO), the image generation unit (32) changes the reference direction (RD1, RD2) according to the captured image displayed in the overlapping region.

[0308] According to the aspect described in 4) above, the reference direction is not fixed, but is changed according to the captured image displayed in the overlapping region when switching the captured image displayed in the overlapping region. Thus, after switching the captured image, the reference direction temporarily moves away from the line-of-sight direction, so that it is possible to reduce the frequent switching of the captured image displayed in the overlapping region.

[0309] 5) In one aspect of the present invention, in any one of the aspects described in 1) to 4) above, the line-of-sight direction determination unit (31) obtains detection information related to the orientation of the head-mounted display (10) (for example, detection information of an angular velocity sensor), and determines the

[0310] above line-of-sight direction (S).

[0311] According to the aspect described in 5) above, the user can dynamically switch the captured image displayed in the overlapping region by changing the orientation of the head-mounted display.

[0312] 6) In one aspect of the present invention, in any of the aspects described in 1) to 4) above, the line-of-sight direction determination unit (31) acquires detection information related to the line of sight of the user (for example, detection information of eye movement tracking), and determines the line-of-sight direction (S) based on this detection information.

[0313] According to the aspect described in 6) above, by changing the movement of the eyes (the orientation of the line of sight), the user can dynamically switch the real-shot image displayed in the overlapping area.

[0314] 7) In one aspect of the present invention, in any of the aspects described in 1) to 6) above, the computer functions as a boundary line display unit (33): the boundary line display unit (33) displays boundary lines (BL1, BL1) that can be visually recognized by the user at the regional boundaries (BD1, BD2) between adjacent real-shot images.

[0315] According to the aspect described in 7) above, by specifically displaying the boundary line in a visually recognizable manner at the regional boundaries between adjacent real-shot images, it is easy for the user to identify the regional boundaries where image seams (misalignments) occur. Thus, the user can easily determine which of the adjacent real-shot images should be displayed in the overlapping area for viewing, so that the boundary line is not displayed in the direction the user wants to observe.

[0316] 8) In one aspect of the present invention, in the aspect described in 7) above, the boundary line display unit (33) displays additional information (GD1, GD2) at or near the boundary line (BL1, BL1), and this additional information (GD1, GD2) is used to indicate which side of the boundary line is the overlapping area (AO).

[0317] According to the aspect described in 8) above, the user can identify which side of the boundary line is the overlapping area. Thus, it is easy for the user to judge which direction the line of sight should be changed to in order to switch the real-shot image displayed in the overlapping area.

[0318] 9) In one aspect of the present invention, in any of the aspects described in 1) to 8) above, when switching the real-shot image displayed in the overlapping area (AO), the image generation unit (32) gradually switches from one real-shot image to another over time.

[0319] According to the aspect described in 9) above, when switching the real-shot image displayed in the overlapping area from one to another, compared with a sudden switch, by gradually switching during the time period from the start to the end of the switch, it is easier to visually recognize the boundary line more naturally.

[0320] 10) In one aspect of the present invention, in the aspect described in 9) above, when the image generation unit (32) switches the live-action image displayed in the overlapping area (AO), the boundary lines (BL1, BL1) are gradually moved to the position to be displayed after the switching is completed over time, and at the same time, the live-action image is gradually switched from one live-action image to the other live-action image.

[0321] According to the aspect described in 10) above, by moving the boundary line during the switching of the live-action image displayed in the overlapping area, the user can naturally recognize the position of the changed boundary line.

[0322] 11) In one aspect of the present invention, in any one of the aspects described in 1) to 10) above, the image generation unit (32) sets a predetermined standby period from when it is determined that the switching process of the live-action image to be displayed in the overlapping area (AO) should be performed until the start of the switching process, according to the line-of-sight direction (S).

[0323] According to the aspect described in 11) above, by setting a predetermined standby period until the start of the switching process of the live-action image displayed in the overlapping area, in a state where the line-of-sight direction changes during the standby period and the switching process is not required, unnecessary switching processes can be omitted.

[0324] 12) In one aspect of the present invention, in any one of the aspects described in 1) to 11) above, the image generation unit (32) changes the arrangement state of a plurality of the live-action images in the VR space (V) in the display control of the VR image on the display unit.

[0325] According to the aspect described in 12) above, the number of field-of-view areas arranged in the VR space, the field-of-view angle of the entire field-of-view area, etc. can be changed according to the scene. That is, optimization of the VR space for each scene can be achieved. Therefore, unnecessary field-of-view areas can be excluded for each scene, and the amount of image data can be reduced. In addition, in a scene where necessary field-of-view areas are excluded, the resolution of the VR image displayed on the display unit of the HMD can be improved.

[0326] 13) In one aspect of the present invention, in any one of the aspects described in 1) to 11) above, the computer functions as a shooting direction determination unit (34) that determines changes in the shooting directions of a plurality of cameras that have captured a plurality of the live-action images, and the image generation unit (32) changes the direction of the entire field-of-view area of the plurality of live-action images arranged in the VR space (V) according to the changes in the shooting directions determined by the shooting direction determination unit.

[0327] According to the method described in the above 13), for example, in the case of a live-action image where the shooting direction of a camera changes as a moving subject (such as a vehicle) moves, the direction of the field-of-view area of the entire live-action image arranged in the VR space also changes according to the change in the shooting direction. Therefore, a VR image with a sense of reality in which the subject moves relative to the line-of-sight direction can be displayed on the display unit. In addition, if a live-action image obtained by shooting the subject in such a way that it does not cross the area boundary between adjacent live-action images is prepared during shooting in accordance with the movement of the above-described subject, a VR image with excellent visual recognition in which the subject does not cross the area boundary can be reproduced.

[0328] 14) In one aspect of the present invention, in the method described in any one of the above 1) to 13), the image generation unit (32) dynamically switches the live-action image displayed in the overlapping area (AO) among the adjacent live-action images so that the area boundaries (A1, A2) of the adjacent live-action images are away from the line-of-sight direction (S). Thereby, the same effect as the method described in the above 1) is achieved.

[0329] 15) In one aspect of the present invention, in the method described in any one of the above 1) to 14), the image generation unit (32) obtains detection information related to the orientation of the head-mounted display (10), and changes the range of the VR image displayed on the display unit (11) according to this detection information.

[0330] According to the method described in the above 15), in the case where the line-of-sight direction is determined based on the detection information related to the orientation of the HMD, the user can change the range of the VR image displayed on the display unit by adjusting the orientation of the HMD, and at the same time dynamically switch the live-action image displayed in the overlapping area. In addition, in the case where the line-of-sight direction is determined based on detection information related to the user's line of sight such as eye tracking, the user can adjust the range of the VR image displayed on the display unit by changing the orientation of the HMD, and in addition, dynamically switch the live-action image displayed in the overlapping area by changing the movement of the eyes (the orientation of the line of sight).

[0331] 16) In one aspect of the present invention, in the method described in any one of the above 1), 5) to 14), the image generation unit (32) determines the live-action image to be displayed in the overlapping area AO based on the line-of-sight direction (S) with respect to the current area boundaries (BD1, BD2) of the adjacent live-action images.

[0332] According to the method described in 16) above, even without setting a reference direction, it is possible to easily achieve the switching control of the live-action image displayed in the overlapping area. In addition, since the area boundary serving as the reference is changed at the timing of switching the live-action image, it is possible to reduce the frequent switching of the live-action image displayed in the overlapping area.

[0333] 17) In one aspect of the present invention, in the method described in 15) above, when the line-of-sight direction (S) approaches a predetermined value or more of the current area boundaries (BD1, BD2) of adjacent live-action images (for example, when the angle formed by the line-of-sight direction and the current area boundary is a predetermined value or less), the image generation unit (32) switches the live-action image currently displayed in the overlapping area (AO) to another live-action image.

[0334] According to the method described in 17) above, it is possible to switch the live-action image displayed in the overlapping area before the line of sight reaches the area boundary, and move the area boundary away from the direction of the user's line of sight.

[0335] 18) In one aspect of the present invention, in the method described in 8) above, the additional information includes semi-transparent gradient portions (GD1, GD2) that overlap the live-action image near the boundary lines (BL1, BL1), and the density or opacity continuously or stepwise decreases as the distance from the boundary line increases.

[0336] According to the method described in 18) above, it is possible to hide the image misalignment generated near the boundary line by the gradient and make it less noticeable. In addition, it is possible to naturally indicate the direction of the change in the line-of-sight direction required for switching the live-action image by the gradient.

[0337] 19) In one aspect of the present invention, in any of the methods described in 7) to 10) above, the boundary line display unit (33) makes the display modes of the boundary lines (BL1 or BL2) of the area boundary (BD1 or BD2) displayed at one end of the overlapping area (AO) and the boundary lines displayed at the other end different.

[0338] According to the method described in 19) above, when using a plurality of images arranged in the horizontal direction to expand the horizontal field of view angle, the user can recognize whether the currently displayed boundary line is the misaligned portion of the image at the left end or the right end of the overlapping area. In addition, when using a plurality of images arranged in the vertical direction to expand the vertical field of view angle, the user can recognize whether the currently displayed boundary line is the misaligned portion of the image at the upper end or the lower end of the overlapping area.

[0339] 20) A display control device (20) according to one embodiment of the present invention performs control for displaying, on a display unit (11) of a head-mounted display (10), a VR image representing a field of view from a virtual viewpoint (P) in a virtual reality (VR) space (V) as a stereoscopic image utilizing binocular parallax. The display control device (20) includes: a line-of-sight direction determination unit (31) that determines a line-of-sight direction (S) which is the direction of the user's line of sight with respect to the VR space; and an image generation unit (32) that arranges a plurality of captured images captured by a plurality of cameras with different shooting directions of the actual space in the VR space such that a part of the field-of-view regions (A1, A2) of adjacent captured images overlaps to form an overlapping region (AO), and generates the VR image corresponding to the line-of-sight direction. The image generation unit (32) dynamically switches the captured images displayed in the overlapping region among the adjacent captured images according to the line-of-sight direction. Thereby, the same effect as the embodiment described in 1) above is achieved.

[0340] 21) An image display system (1) according to one embodiment of the present invention includes: a head-mounted display (10) that displays, on a display unit (11), a VR image representing a field of view from a virtual viewpoint (P) in a virtual reality (VR) space (V) as a stereoscopic image utilizing binocular parallax; and a display control device (20) that performs control for displaying the VR image on the display unit. The display control device (20) includes: a line-of-sight direction determination unit (31) that determines a line-of-sight direction (S) which is the direction of the user's line of sight with respect to the VR space; and an image generation unit (32) that arranges a plurality of captured images captured by a plurality of cameras (200) with different shooting directions of the actual space in the VR space such that a part of the field-of-view regions (A1, A2) of adjacent captured images overlaps to form an overlapping region (AO), and generates the VR image corresponding to the line-of-sight direction. The image generation unit (32) dynamically switches the captured images displayed in the overlapping region among the adjacent captured images according to the line-of-sight direction. Thereby, the same effect as the embodiment described in 1) above is achieved.

[0341] 22) An information storage medium according to one embodiment of the present invention is an information storage medium readable by a computer recording a program according to any one of 1) to 19) above. Thereby, the same effect as the embodiments described in 1) to 19) above is achieved.

[0342] 23) A control method for a display control device (20) according to an aspect of the present invention controls the display control device (20), and the display control device (20) performs control for displaying a VR image representing a field of view from a virtual viewpoint (P) in a virtual reality (VR) space (V) as a stereoscopic image utilizing binocular parallax on a display unit (11) of a head-mounted display (10). The control method includes the following steps: a line-of-sight direction specifying step (S106) for determining a line-of-sight direction (S), which is set as the direction of the user's line of sight with respect to the VR space; and an image generation step (S108 to S114) for arranging a plurality of actual shot images captured by a plurality of cameras having different shooting directions of the actual space in the VR space (S102) so that a part of the field-of-view regions (A1, A2) of the adjacent actual shot images overlaps to form an overlapping region (AO), and generating the VR image corresponding to the line-of-sight direction. The image generation step includes a step (S108 to S112) of dynamically switching the actual shot image displayed in the overlapping region among the adjacent actual shot images according to the line-of-sight direction. Thereby, the same effect as the aspect described in 1) above is achieved.

[0343] In addition, the specific embodiments or examples proposed in the embodiments for implementing the invention are only for making the technical content of the present invention clear, and should not be interpreted narrowly only limited to such specific examples, and various changes can be made and implemented within the scope of the technical concept and claims of the present invention.

[0344] Reference Numeral Explanation

[0345] 1: Image display system; 10: HMD; 11, 61: Display unit; 12, 62: Sensor; 13: Processor; 14: Storage device; 20: Display control device; 21, 63: Processor; 22, 64: Storage device; 23, 65: Operation unit; 24, 66: Communication unit; 30: Control unit; 31: Line-of-sight direction determination unit; 32: Image generation unit; 321: Image arrangement unit; 322: Switching unit; 33: Boundary line display unit; 34: Shooting direction determination unit; 60: HMD (information processing device); 200: Camera; A1, A2: Field-of-view region; AO: Overlapping region; RD: Reference direction; V: VR space; S: Line-of-sight direction; BL1, BL2: Boundary line; GD1, GD2: Gradient part.

Claims

1. A program that causes a computer performing the following control to function as the following components. The control is for displaying, on a display unit of a head-mounted display, a VR image representing a field of view from a virtual viewpoint in a virtual reality (VR) space as a stereoscopic image utilizing binocular parallax. The components include: A line-of-sight direction determination unit that determines a line-of-sight direction, which is set as the direction of the user's line of sight relative to the VR space; And An image generation unit that arranges a plurality of real-shot images captured by a plurality of cameras with different shooting directions of the real space in the VR space in such a way that a part of the field-of-view regions of adjacent real-shot images overlap each other, and generates the VR image corresponding to the line-of-sight direction. The image generation unit dynamically switches the real-shot images displayed in the overlapping region among the adjacent real-shot images according to the line-of-sight direction.

2. The program according to claim 1, wherein The image generation unit sets a reference direction as a specified direction from the virtual viewpoint, and determines the real-shot image displayed in the overlapping region according to the line-of-sight direction relative to the reference direction.

3. The program according to claim 2, wherein The image generation unit sets the reference direction as the direction of a specified position in the overlapping region from the virtual viewpoint.

4. The program according to claim 2 or 3, wherein When switching the real-shot images displayed in the overlapping region, the image generation unit changes the reference direction according to the real-shot images displayed in the overlapping region.

5. The program according to any one of claims 1 to 4, wherein The line-of-sight direction determination unit obtains detection information related to the orientation of the head-mounted display, and determines the line-of-sight direction according to the detection information.

6. The program according to any one of claims 1 to 4, wherein The line-of-sight direction determination unit obtains detection information related to the user's line of sight, and determines the line-of-sight direction according to the detection information.

7. The program according to any one of claims 1 to 6, wherein The program causes the computer to function as a boundary line display unit that displays a boundary line recognizable by the user at the region boundary between adjacent real-shot images.

8. The program according to claim 7, wherein The boundary line display unit displays additional information indicating which side of the boundary line is the overlapping region on the boundary line or near it.

9. The program according to claim 7, wherein When switching the real-shot images displayed in the overlapping region, the image generation unit gradually switches from one real-shot image to another real-shot image over time.

10. The program according to claim 9, wherein When switching the real-shot images displayed in the overlapping region, the image generation unit gradually moves the boundary line to the position displayed after the switching is completed over time, and at the same time gradually switches from one real-shot image to another real-shot image.

11. The program according to any one of claims 1 to 10, wherein the image generation unit sets a prescribed standby period from when it is determined that switching processing of the captured image to be displayed in the overlapping region should be performed until the start of the switching processing, based on the line-of-sight direction.

12. The program according to any one of claims 1 to 11, wherein the image generation unit changes the arrangement state of the plurality of captured images in the VR space during the display control of the VR image on the display unit.

13. The program according to any one of claims 1 to 11, wherein the program causes the computer to function as a shooting direction determination unit that determines changes in the shooting directions of a plurality of cameras that have captured a plurality of the captured images, and the image generation unit changes the direction of the entire field-of-view region of the plurality of captured images arranged in the VR space according to the changes in the shooting directions determined by the shooting direction determination unit.

14. A display control device performs control for displaying, on a display unit of a head-mounted display, a VR image representing a field of view from a virtual viewpoint in a virtual reality (VR) space as a stereoscopic image using binocular parallax, where The display control device includes: a line-of-sight direction determination unit that determines a line-of-sight direction, which is set as the direction of the user's line of sight with respect to the VR space; and an image generation unit that arranges a plurality of captured images captured by a plurality of cameras having different shooting directions of the actual space in the VR space such that a part of the field-of-view regions of the adjacent captured images overlap to generate the VR image corresponding to the line-of-sight direction, wherein the image generation unit dynamically switches the captured image to be displayed in the overlapping region among the adjacent captured images according to the line-of-sight direction.

15. An image display system having: a head-mounted display that displays a VR image representing a field of view from a virtual viewpoint in a virtual reality, i.e., a VR space, as a stereoscopic image using binocular parallax on a display unit; and a display control device that executes control for displaying the VR image on the display unit, wherein the display control device includes: a line-of-sight direction determination unit that determines a line-of-sight direction, which is set as the direction of the user's line of sight with respect to the VR space; and an image generation unit that arranges a plurality of captured images captured by a plurality of cameras having different shooting directions of the actual space in the VR space such that a part of the field-of-view regions of the adjacent captured images overlap to generate the VR image corresponding to the line-of-sight direction, wherein the image generation unit dynamically switches the captured image to be displayed in the overlapping region among the adjacent captured images according to the line-of-sight direction.

16. A computer-readable recording medium that records the program according to any one of claims 1 to 13.

Images