VR image providing system, VR image transmitting device, VR image receiving device, and information processing method

Abstract

To provide a system for transmitting a live-action VR image and a CG command from a VR image transmitter to a VR image receiver and synthesizing a CGVR image generated in the VR image receiver with the live-action VR image for display, the system being configured to prevent time lag in synthesizing the CGVR image with the live-action VR image.SOLUTION: A VR image transmitter 10 generates a CG command including synthesis timing information of a CGVR image to be added to a live-action VR image, and transmits the live-action VR image with the CG command. A VR image receiver 20 synthesizes a CGVR image generated based on the command added to the received live-action VR image, with the live-action VR image at a timing based on the synthesis timing information. Accordingly, the timing of synthesizing a CGVR image with a live-action VR image based on synthesis timing information is uniformly specified, thereby preventing time lag for synthesis.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a VR image providing system, a VR image transmitting device, a VR image receiving device, and an information processing method, and is particularly suitable for use in a system in which a live-action VR image and a CG command are transmitted from a VR image transmitting device to a VR image receiving device, and a CGVR image generated by the VR image receiving device in accordance with the CG command is synthesized with the live-action VR image and displayed. [Background technology]

[0002] A technology has been known in the past that combines VR images (hereinafter referred to as live-action VR images) consisting of 360-degree images taken with a VR (Virtual Reality) camera (also called a 360-degree camera) with computer graphics (hereinafter referred to as CGVR images) to provide specific dramatic effects. By using this technology, it is possible to add dramatic effects such as making it rain with a CGVR image in a specific scene of a live-action VR image, or launching fireworks with a CGVR image in a specific scene of a live-action VR image.

[0003] When transmitting such a composite image of a live-action VR image and a CGVR image (hereinafter referred to as a composite VR image) from an information processing device to a head-mounted display (HMD) for display, the composite VR image is generally compressed to reduce the amount of data transmitted. However, in this case, the quality of the CGVR image expanded and played back on the HMD is significantly reduced, resulting in a problem that fine details cannot be reproduced.

[0004] One way to avoid this problem is to send a compressed live-action VR image and a command to draw CG from an information processing device to an HMD, have the HMD generate a CGVR image in response to the command, and then composite the CGVR image with the expanded live-action VR image to display the composite VR image. However, in this case, the time required for the expansion process of the live-action VR image and the rendering process of the CGVR image executed by the HMD will differ depending on the HMD receiving the distribution, which may result in the CGVR image not being composited at the appropriate timing for a specific scene in the live-action VR image.

[0005] There is known a technique for displaying an MR image by transmitting a CG command from an information processing device to an HMD, rendering the CG in the HMD, and superimposing the CG on a live-action image captured by a camera in the HMD (see, for example, Patent Document 1). Patent Document 1 discloses that timing information (reference video frame information) for preventing a time lag in combining the CG with the live-action image is added to the CG command, and the CG combining timing is adjusted based on this timing information. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2018-55196 Summary of the Invention [Problem to be solved by the invention]

[0007] In the system described in Patent Document 1, live-action images are taken by a camera mounted on an HMD. Therefore, it is necessary to first transmit the live-action images taken by the camera to an information processing device, and then transmit CG commands generated by the information processing device based on the images to the HMD. Therefore, the technology described in Patent Document 1 cannot be directly applied to a system that transmits live-action VR images from an information processing device to an HMD.

[0008] The present invention has been made to solve such problems, and aims to prevent a time lag in the synthesis of a CGVR image with a live-action VR image in a system in which a live-action VR image and CG commands are transmitted from a VR image transmitting device to a VR image receiving device, and a CGVR image generated by the VR image receiving device in accordance with the CG commands is synthesized with the live-action VR image and displayed. [Means for solving the problem]

[0009] In the VR image providing system of the present invention, a live-action VR image, which is a VR image composed of live action, and a CG command instructing the generation of a CGVR image, which is a VR image composed of CG, are transmitted from a VR image transmitting device to a VR image receiving device, and the CGVR image generated by the VR image receiving device in accordance with the CG command is composited with the live-action VR image and displayed. To solve the above-mentioned problem, in the VR image providing system of the present invention, the VR image transmitting device generates a CG command including CG generation information instructing the generation content of the CGVR image and synthesis timing information instructing the synthesis timing of the CGVR image, and attaches the CG command to the live-action VR image, and transmits the live-action VR image with the CG command attached to the VR image receiving device. Furthermore, the VR image receiving device generates a CGVR image based on the CG generation information included in the CG command attached to the live-action VR image received from the VR image transmitting device, and composites the generated CGVR image with the live-action VR image at a timing based on the synthesis timing information included in the CG command to generate a composite VR image. [Effects of the Invention]

[0010] According to the present invention configured as described above, in a system in which a live-action VR image and a CG command are transmitted from a VR image transmitting device to a VR image receiving device, and a CGVR image generated by the VR image receiving device in accordance with the CG command is synthesized with the live-action VR image and displayed, the timing of synthesizing the CGVR image with the live-action VR image can be uniformly specified in accordance with the synthesis timing information attached to the live-action VR image, thereby preventing a time lag in synthesizing the CGVR image with the live-action VR image from occurring for each VR image receiving device. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a diagram illustrating an example of the overall configuration of a VR image providing system according to this embodiment. [Figure 2] FIG. 2 is a block diagram illustrating an example of the functional configuration of a VR image transmitting device according to the present embodiment. [Figure 3] FIG. 2 is a block diagram showing an example of the functional configuration of a VR image receiving device according to the present embodiment. [Figure 4] FIG. 1 is a diagram showing the zenith and nadir regions of a 360-degree video. [Figure 5] 10 is a flowchart showing an example of the operation of the VR image providing system according to this embodiment. [Figure 6] FIG. 10 is a block diagram showing an example of the functional configuration of a VR image receiving device according to a first modified example. [Figure 7] FIG. 10 is a block diagram showing an example of the functional configuration of a VR image receiving device according to a second modified example. DETAILED DESCRIPTION OF THE INVENTION

[0012] An embodiment of the present invention will be described below with reference to the drawings. Fig. 1 is a diagram showing an example of the overall configuration of a VR image providing system according to this embodiment. As shown in Fig. 1, the VR image providing system 1 of this embodiment includes a VR image transmitting device 10 and a VR image receiving device 20. A 360-degree camera 100 is connected to the VR image transmitting device 10. A head-mounted display 200 (HMD: equivalent to a display in the claims) is connected to the VR image receiving device 20.

[0013] The 360-degree camera 100 is a camera that can capture a 360-degree view around the location where the camera is installed. In this embodiment, a spherical 360-degree camera 100 that can capture 360 ​​degrees in both the vertical and horizontal directions is used. Images captured by the 360-degree camera 100 are called "360-degree videos" or "360-degree images."

[0014] The VR image transmitting device 10 is configured by, for example, a personal computer. The VR image transmitting device 10 generates a VR image based on a 360-degree image captured by a 360-degree camera 100 in accordance with instructions based on user operations on input devices such as a keyboard or mouse, compresses the generated VR image, and transmits it to the VR image receiving device 20. The VR image generated by the VR image transmitting device 10 is a VR image made up of live-action images (hereinafter referred to as live-action VR images). The generation of a VR image based on a 360-degree image includes, for example, a stitching process that joins together videos captured from various angles, and image adjustment processes such as brightness adjustment and blur correction.

[0015] The VR image transmitting device 10 further generates a CG command that instructs the generation of a VR image made of CG (hereinafter referred to as a CGVR image) in accordance with an instruction based on a user's operation on an input device. The CG command generated by the VR image transmitting device 10 includes CG generation information that instructs the content of the CGVR image to be generated, and synthesis timing information that instructs the synthesis timing of the CGVR image with respect to the live-action VR image (the temporal position within the live-action VR image). The VR image transmitting device 10 adds the CG command to the live-action VR image and transmits it to the VR image receiving device 20.

[0016] The VR image receiving device 20 is configured by, for example, a personal computer. The VR image receiving device 20 receives a live-action VR image with a CG command transmitted from the VR image transmitting device 10, expands the compressed live-action VR image, and displays it on the HMD 200. The VR image receiving device 20 also generates a CGVR image according to CG generation information included in the CG command, synthesizes the CGVR image with the live-action VR image, and displays it on the HMD 200. The synthesis timing of the CGVR image with the live-action VR image is controlled by synthesis timing information included in the CG command.

[0017] 1 shows an example in which the VR image receiving device 20 and the HMD 200 are configured as separate entities, but this is not limiting. For example, the HMD 200 may be configured as an integrated device that incorporates the functions of the VR image receiving device 20. Furthermore, the VR image receiving device 20 may be a smartphone, and the HMD 200 may be so-called VR goggles (VR glasses) configured to be worn and usable with the smartphone. In this case, a VR image is displayed on a display provided on the smartphone, and the user views it through the VR goggles.

[0018] Fig. 2 is a block diagram showing an example of the functional configuration of the VR image transmitting device 10. As shown in Fig. 2, the VR image transmitting device 10 of this embodiment includes, as its functional configuration, a 360-degree image input unit 11, a VR image generating unit 12, a command generating unit 13, a command adding unit 14, a compression processing unit 15, and a VR image transmitting unit 16.

[0019] The functional blocks 11 to 16 can be configured by any of hardware, a DSP (Digital Signal Processor), and software. For example, when configured by software, the functional blocks 11 to 16 are actually configured with a CPU, RAM, ROM, etc. of a computer, and are realized by the operation of a program stored in a storage medium such as the RAM, ROM, hard disk, or semiconductor memory.

[0020] The 360-degree image input unit 11 inputs a 360-degree image captured by the 360-degree camera 100. Note that, although Fig. 1 shows an example in which the 360-degree image input unit 11 directly inputs the 360-degree image from the 360-degree camera 100, this is not limiting. For example, the 360-degree image captured by the 360-degree camera 100 may be stored in a storage medium, and the 360-degree image input unit 11 may input the 360-degree image from this storage medium.

[0021] The VR image generation unit 12 generates a live-action VR image based on the 360-degree image input by the 360-degree image input unit 11, in accordance with instructions based on user operations on the input device. As described above, the VR image generation unit 12 generates a live-action VR image by performing a stitching process that joins together 360-degree images taken from various angles, and image adjustment processes such as brightness adjustment and blur correction. The generated live-action VR image is stored and saved in a storage medium.

[0022] The command generation unit 13 generates CG commands including CG generation information that indicates the content of the CGVR image to be generated and synthesis timing information that indicates the timing of synthesis of the CGVR image, in accordance with instructions based on user operations on an input device. The CG generation information includes, for example, designation information for material data (image objects related to various dramatic effects such as rain, fireworks, and light) to be used as CG, processing instruction information indicating how to process the material data, and position designation information indicating the display position of the CGVR image obtained by processing the material data within the live-action VR image. The synthesis timing information includes, for example, frame information indicating the temporal position of the live-action VR image with which the CGVR image should be synthesized. The frame information may be a relative value indicating the number of frames from a predetermined position in the live-action VR image (for example, the start position or the nearest chapter setting position), or an absolute value such as a frame number.

[0023] For example, while playing back and displaying a live-action VR image (a live-action VR image stored in a storage medium) generated by the VR image generation unit 12 on an editing screen, a frame indicating the display start position of the CGVR image can be specified to generate synthesis timing information, and necessary information, such as material data specification information, processing instruction information, and position specification information, can be entered into a predetermined property screen to generate CG generation information that indicates the content of the CGVR image to be displayed from the specified frame. In addition to a frame indicating the display start position of the CGVR image, a frame indicating the display end position may also be specified. The display position of the CGVR image may also be specified by operating a mouse or the like on the live-action VR image displayed on the editing screen. Note that the CG command generation method described here is merely an example and is not limited thereto. Publicly known techniques for generating CG commands may also be used.

[0024] The command generating unit 13 may encode the CG command generated as described above. For example, the command generating unit 13 encodes the CG command into a two-dimensional code.

[0025] The command adding unit 14 adds the CG command (which may be an encoded two-dimensional code; the same applies below) generated by the command generating unit 13 to the live-action VR image. The time position of the live-action VR image to which the CG command is added is a position before the frame indicating the display start position (combination start position) of the CGVR image. As will be described later, the VR image receiving device 20 extracts the added CG command while playing the live-action VR image, generates a CGVR image according to the CG command, and combines it with the live-action VR image. Therefore, it is necessary that the generation of the CGVR image has already been completed before the frame indicating the display start position of the CGVR image. For example, the command adding unit 14 adds the CG command to a position shortly after the start of the live-action VR image.

[0026] In this embodiment, the live-action VR image is a VR image of a spherical 360-degree video. As shown in FIG. 4, the command adding unit 14 adds the CG command generated by the command generating unit 14 to a portion of the 360-degree video that will become the zenith region 41 or the nadir region 42. Here, the zenith region 41 refers to an area of ​​a predetermined size that includes the zenith. Similarly, the nadir region 42 refers to an area of ​​a predetermined size that includes the nadir. In the following description, the zenith-nadir regions 41 and 42 may be referred to as the zenith-nadir regions, but this means at least one of the zenith region 41 and the nadir region 42.

[0027] Like other areas, the zenith-nadir regions 41 and 42 of the live-action VR image are displayed on the display when the user wearing the HMD 200 performs an action or operation to change the viewpoint. However, when playback of the live-action VR image begins, the user's viewpoint is often initially set to directly in front, halfway between the zenith and nadir. Therefore, at the viewpoint position for a frontal view, the zenith-nadir regions 41 and 42 are not displayed, or even if they are displayed, they are barely within the user's field of view (the range visible without moving the eyes from the front), so even if CG command information is displayed there, it would not be a significant eyesore.

[0028] The compression processing unit 15 compresses the live-action VR image to which the CG commands have been added by the command adding unit 14. Note that the zenith and nadir regions 41 and 42 to which the CG commands have been added may be excluded from the target of the compression processing.

[0029] The VR image transmitting unit 16 transmits the live-action VR image with CG commands compressed by the compression processing unit 15 to the VR image receiving device 20. In the following description, the live-action VR image with CG commands transmitted from the VR image transmitting device 10 to the VR image receiving device 20 may be simply referred to as a "live-action VR image."

[0030] Fig. 3 is a block diagram showing an example of the functional configuration of the VR image receiving device 20. As shown in Fig. 3, the VR image receiving device 20 of this embodiment includes, as its functional configuration, a VR image receiving unit 21, an expansion processing unit 22, a CGVR image generating unit 23, a composite VR image generating unit 24, and a composite VR image display unit 25.

[0031] The functional blocks 21 to 25 can be configured by any of hardware, DSP, and software. For example, when configured by software, the functional blocks 21 to 25 are actually configured with a CPU, RAM, ROM, etc. of a computer, and are realized by the operation of a program stored in a storage medium such as RAM, ROM, a hard disk, or a semiconductor memory.

[0032] The VR image receiving unit 21 receives live-action VR images transmitted from the VR image transmitting unit 16 of the VR image transmitting device 10. The decompression processing unit 22 decompresses the compressed live-action VR images and supplies the decompressed live-action VR images to the composite VR image generating unit 24. The decompression processing unit 22 also extracts CG commands for the zenith and nadir regions 41 and 42 from the decompressed live-action VR images, and supplies CG generation information included in the extracted CG commands to the CG VR image generating unit 23, as well as synthesis timing information to the composite VR image generating unit 24. Here, if the CG commands are encoded in a two-dimensional code, the decompression processing unit 22 obtains the CG generation information and synthesis timing information by decode the two-dimensional code.

[0033] The CGVR image generation unit 23 generates a CGVR image based on the CG generation information supplied from the decompression processing unit 22. That is, the CGVR image generation unit 23 acquires material data (image objects) to be used as CG in accordance with the specification information included in the CG generation information. The specification information includes address information of the website or database where the material data is stored and identification information of the material data, and the CGVR image generation unit 23 acquires the material data based on this information. The CGVR image generation unit 23 then processes the image objects in accordance with the processing instruction information to generate a CGVR image. Processing of the image objects includes at least one of deformation, rotation, resizing, color correction, various image quality adjustments, various filter processes, etc.

[0034] The composite VR image generation unit 24 generates a composite VR image by combining the CGVR image generated by the CGVR image generation unit 23 with the live-action VR image received by the VR image receiving unit 21 and expanded by the expansion processing unit 22 at a timing based on the synthesis timing information supplied from the expansion processing unit 22. During this synthesis, the composite VR image generation unit 24 composites the CGVR image at a display position within the live-action VR image indicated by the position designation information included in the CG generation information. The composite VR image generation unit 24 then outputs the generated composite VR image to the composite VR image display unit 25.

[0035] Note that the CGVR image is synthesized with the live-action VR image only in the time range starting from the frame of the display start position specified by the synthesis timing information included in the CG command. In other time ranges, the CGVR image is not synthesized with the live-action VR image, and only the live-action VR image expanded by the expansion processing unit 22 is output to the synthesized VR image display unit 25. In this specification, a live-action VR image in which a CGVR image is synthesized in a part of the time range in this way is referred to as a synthesized VR image.

[0036] The composite VR image display unit 25 displays the composite VR image generated by the composite VR image generation unit 24 on the HMD 200.

[0037] Figure 5 is a flowchart showing an example of operation of the VR image providing system 1 according to this embodiment configured as described above, where Figure 5(a) shows an example of operation of the VR image transmitting device 10, and Figure 5(b) shows an example of operation of the VR image receiving device 20.

[0038] 5(a), the 360-degree image input unit 11 of the VR image transmission device 10 inputs a 360-degree image captured by the 360-degree camera 100 (step S11). Next, the VR image generation unit 12 generates a live-action VR image based on the 360-degree image input by the 360-degree image input unit 11, in accordance with instructions based on a user's operation on the input device (step S12). Note that the input process of the 360-degree image in step S11 and the generation process of the live-action VR image in step S12 do not necessarily have to be performed consecutively. In other words, the generation process of the live-action VR image can be performed at any timing after the input process of the 360-degree image.

[0039] Next, the command generation unit 13 generates a CG command including CG generation information and synthesis timing information according to an instruction based on the user's operation on the input device, and encodes the CG command (step S13). Note that the process of generating a live-action VR image in step S12 and the process of generating a CG command in step S13 do not necessarily have to be performed consecutively. In other words, the process of generating a CG command can be performed at any timing after the process of generating a live-action VR image.

[0040] Next, the command adding unit 14 adds the CG commands generated by the command generating unit 13 to the live-action VR image (step S14). Here, the command adding unit 14 adds the CG commands to the portions that will become the zenith-nadir regions 41, 42 of the 360-degree video. Next, the compression processing unit 15 compresses the live-action VR image to which the CG commands have been added by the command adding unit 14 (step S15). Then, the VR image transmitting unit 16 transmits the live-action VR image with the CG commands compressed by the compression processing unit 15 to the VR image receiving device 20 (step S16). This completes the operation of the VR image transmitting device 10 shown in FIG. 5(a).

[0041] 5(b), the VR image receiving unit 21 of the VR image receiving device 20 receives the live-action VR image transmitted from the VR image transmitting device 10 (step S21). Next, the decompression processing unit 22 decompresses the live-action VR image received by the VR image receiving unit 21 and extracts, for example, a CG command added at a position shortly from the start of the live-action VR image (step S22). Then, the CGVR image generating unit 23 generates a CGVR image based on the CG generation information included in the CG command extracted by the decompression processing unit 22 (step S23). The generated CGVR image is stored in memory until the synthesis timing.

[0042] Next, the composite VR image generation unit 24 generates a composite VR image by combining the CGVR image generated by the CGVR image generation unit 23 with the live-action VR image expanded by the expansion processing unit 22 at a timing based on the synthesis timing information included in the CG command extracted by the expansion processing unit 22 (step S24). Then, the composite VR image display unit 25 displays the composite VR image generated by the composite VR image generation unit 24 on the HMD 200 (step S25). This completes the operation of the VR image receiving device 20 shown in FIG. 5(b).

[0043] It is not necessary that the process of receiving the live-action VR image in step S21 and the process from step S22 onwards are performed consecutively. That is, the live-action VR image received by the VR image receiving unit 21 may be stored and saved in a storage medium, and the process from step S22 onwards may be performed when the user issues a playback instruction at any timing thereafter.

[0044] As explained in detail above, in this embodiment, the VR image transmitting device 10 generates CG commands including CG generation information and synthesis timing information, adds them to live-action VR images, and transmits the live-action VR images with the CG commands added to them to the VR image receiving device 20. Furthermore, the VR image receiving device 20 generates a CGVR image based on the CG generation information included in the CG commands added to the live-action VR images received from the VR image transmitting device 10, and synthesizes the generated CGVR image with the live-action VR image at a timing based on the synthesis timing information included in the CG commands, thereby generating a synthesized VR image.

[0045] According to this embodiment configured as described above, in a system in which a live-action VR image and a CG command are transmitted from a VR image transmitting device 10 to a VR image receiving device 20, and a CGVR image generated by the VR image receiving device 20 in accordance with the CG command is synthesized with the live-action VR image and displayed, the timing of synthesizing the CGVR image with the live-action VR image can be uniformly specified according to the synthesis timing information attached to the live-action VR image, thereby preventing a time lag in synthesizing the CGVR image with the live-action VR image from occurring for each VR image receiving device 20.

[0046] Furthermore, in this embodiment, CG commands are added to the zenith and nadir regions 41 and 42 of the live-action VR image, so that even if the CG command information is displayed on the HMD 200, it does not become a major eyesore.

[0047] The VR image receiving device 20' may be configured as shown in FIG. 6. FIG. 6 is a block diagram showing an example of the functional configuration of a VR image receiving device 20' according to a first modified example of this embodiment. In FIG. 6, components having the same reference numerals as those shown in FIG. 3 have the same functions, and therefore redundant explanations will be omitted here. As shown in FIG. 6, the VR image receiving device 20' according to the first modified example further includes a second CGVR image generation unit 26. Moreover, the VR image receiving device 20' according to the first modified example includes a composite VR image generation unit 24' instead of the composite VR image generation unit 24.

[0048] The second CGVR image generation unit 26 generates a second CGVR image to be displayed in the zenith / nadir regions 41 and 42 of the 360-degree video. This second CGVR image is an image that covers the CG commands added to the zenith / nadir regions 41 and 42 so that they cannot be seen by the user. For example, the second CGVR image generation unit 26 generates a second CGVR image that has the same shape as the zenith / nadir regions 41 and 42 and is made of a single color.

[0049] The composite VR image generation unit 24' generates a composite VR image by combining the CGVR image generated by the CGVR image generation unit 23 with the live-action VR image at a timing based on the synthesis timing information, and by combining the second CGVR image generated by the second CGVR image generation unit 26 with the parts that become the zenith-nadir regions 41, 42.

[0050] Here, the composite VR image generation unit 24' may composite the second CGVR image into the portions of the zenith-nadir regions 41, 42 throughout the entire time range of the live-action VR image, or may composite the second CGVR image only in the time range in which CG commands are added to the portions of the zenith-nadir regions 41, 42. For example, the time range in which CG commands are added may be determined in advance to be near the beginning of the live-action VR image, and the second CGVR image may be composited only in this predetermined time range. In this case, the second CGVR image generation unit 26 may generate a second CGVR image that imitates the image of the portions of the live-action VR image surrounding the zenith-nadir regions 41, 42 in the time range in which CG commands are added.

[0051] In this way, it is possible to completely hide the CG command information added to the zenith-nadir regions 41 and 42. As a result, even if the user changes the viewpoint toward the zenith-nadir regions 41 and 42, the CG command information is not displayed, and it is possible to avoid giving the user a sense of incongruity.

[0052] As a second modified example, the VR image receiving device 20" may be configured as shown in Figure 7, and the zenith-nadir areas 41, 42 of the live-action VR image may be hidden. Figure 7 is a block diagram showing an example of the functional configuration of the VR image receiving device 20" relating to the second modified example. In this Figure 7, components with the same reference numerals as those shown in Figure 3 have the same functions, so redundant explanations will be omitted here. As shown in Figure 7, the VR image receiving device 20" relating to the second modified example has a composite VR image generation unit 24" instead of the composite VR image generation unit 24.

[0053] The composite VR image generation unit 24" generates a composite VR image using a partial live-action VR image of the area of ​​the live-action VR image received by the VR image receiving unit 21 and expanded by the expansion processing unit 22, excluding the parts that become the zenith-nadir areas 41, 42. Here, the composite VR image generation unit 24" may generate a composite VR image using a partial live-action VR image over the entire time range of the live-action VR image, or may generate a composite VR image using a partial live-action VR image only in the time range over the entire time range of the live-action VR image where CG commands are added to the parts that become the zenith-nadir areas 41, 42. Even when configured in this way, the CG commands added to the zenith-nadir areas 41, 42 can be made completely invisible.

[0054] In the above embodiment, an example of generating a live-action VR image of a type that allows for viewpoint changes was described, but it is also possible to generate a live-action VR image of a type that does not allow for viewpoint changes (this type is sometimes called a 360-degree image to distinguish it from VR images, but in this specification this type is also defined as a live-action VR image).

[0055] Furthermore, the above-described embodiments are merely examples of specific embodiments for carrying out the present invention, and the technical scope of the present invention should not be construed as being limited thereby. In other words, the present invention can be carried out in various forms without departing from the gist or main characteristics thereof. [Explanation of symbols]

[0056] 1. VR image provision system 10 VR image transmission device 11 360-degree image input unit 12 VR image generation unit 13 Command Generation Unit 14 Command Addition Section 15 Compression processing section 16 VR image transmission unit 20,20',20" VR image receiving device 21 VR image receiving unit 22 Decompression processing unit 23 CGVR Image Generation Department 24,24',24" Synthetic VR image generation unit 25. Synthetic VR image display unit 26 Second CGVR image generation unit

Claims

1. A VR image providing system that transmits a live-action VR image, which is a VR image made from live action, and a CG command instructing the generation of a CGVR image, which is a VR image made from CG, from a VR image transmitting device to a VR image receiving device, and composites the CGVR image generated by the VR image receiving device in response to the CG command with the live-action VR image and displays the composite image, The VR image transmission device a command generating unit that generates the CG command, the CG command including CG generation information that indicates the content of the CGVR image to be generated and synthesis timing information that indicates the synthesis timing that indicates the temporal position within the live-action VR image of the CGVR image to be synthesized with the live-action VR image; a command adding unit that adds the CG command generated by the command generating unit to the live-action VR image; a VR image transmitting unit that transmits the live-action VR image to which the CG command has been added by the command adding unit to the VR image receiving device, The VR image receiving device is a VR image receiving unit that receives the live-action VR image transmitted from the VR image transmitting device; a CGVR image generating unit that generates the CGVR image based on the CG generation information included in the CG command added to the live-action VR image received by the VR image receiving unit; a composite VR image generation unit that generates a composite VR image by combining the CGVR image generated by the CGVR image generation unit with the live-action VR image received by the VR image receiving unit at a timing based on the synthesis timing information included in the CG command; a composite VR image display unit that displays the composite VR image generated by the composite VR image generation unit on a display. A VR image providing system characterized by:

2. The live-action VR image is a 360-degree video VR image, The VR image providing system described in claim 1, characterized in that the command addition unit adds the CG command generated by the command generation unit to the zenith or nadir area of ​​the 360-degree video.

3. The VR image receiving device further includes a second CGVR image generating unit that generates a second CGVR image to be displayed in the zenith region or the nadir region, The composite VR image generation unit generates the composite VR image by combining the CG VR image with the live-action VR image at a timing based on the combination timing information and combining the second CG VR image with the portion that will become the zenith region or the nadir region.

3. The VR image providing system according to claim 2.

4. The VR image providing system described in claim 3, characterized in that the synthetic VR image generation unit synthesizes the second CG VR image with the part that becomes the zenith region or the nadir region only in the time range in which the CG command is added to the part that becomes the zenith region or the nadir region of the entire time range of the live-action VR image.

5. The VR image providing system described in claim 2, characterized in that the synthetic VR image generation unit generates the synthetic VR image using a partial live-action VR image of the area of ​​the live-action VR image received by the VR image receiving unit, excluding the part that becomes the zenith area or the nadir area.

6. The VR image providing system described in claim 5, characterized in that the synthetic VR image generation unit generates the synthetic VR image using the partial live-action VR image only in the time range in which the CG command is added to the part that becomes the zenith or nadir region of the entire time range of the live-action VR image, and generates the synthetic VR image using the live-action VR image including the part that becomes the zenith or nadir region in other time ranges.

7. a command generating unit that generates a CG command including CG generation information that indicates the content of a CGVR image, which is a VR image made of CG, and synthesis timing information that indicates the synthesis timing that indicates the time position within the live-action VR image of the CGVR image to be synthesized with the live-action VR image, which is a VR image made of live action; a command adding unit that adds the CG command generated by the command generating unit to the live-action VR image; a VR image transmitting unit that transmits the live-action VR image to which the CG command has been added by the command adding unit to a VR image receiving device that generates the CG VR image in response to the CG command, combines the CG VR image with the live-action VR image, and displays the combined image on a display. A VR image transmission device characterized by:

8. The live-action VR image is a 360-degree video VR image, The VR image transmission device described in claim 7, characterized in that the command addition unit adds the CG command generated by the command generation unit to a part that becomes the zenith area or nadir area of ​​the 360-degree video.

9. a VR image receiving unit that receives a live-action VR image to which a CG command is added, the CG command including CG generation information that indicates the content of the CGVR image, which is a VR image made of CG, and synthesis timing information that indicates the time position within the VR image as the synthesis timing of the CGVR image with the live-action VR image; a CGVR image generating unit that generates the CGVR image based on the CG generation information included in the CG command added to the live-action VR image received by the VR image receiving unit; a composite VR image generation unit that generates a composite VR image by combining the CGVR image generated by the CGVR image generation unit with the live-action VR image received by the VR image receiving unit at a timing based on the synthesis timing information included in the CG command; a composite VR image display unit that displays the composite VR image generated by the composite VR image generation unit on a display. A VR image receiving device characterized by:

10. The VR image receiving device described in claim 9, characterized in that the live-action VR image is a VR image of a 360-degree video, and the CG command is added to the zenith or nadir region of the 360-degree video.

11. a second CGVR image generating unit configured to generate a second CGVR image to be displayed in the zenith region or the nadir region; The composite VR image generation unit generates the composite VR image by combining the CG VR image with the live-action VR image at a timing based on the combination timing information and combining the second CG VR image with the portion that will become the zenith region or the nadir region.

11. The VR image receiving device according to claim 10.

12. The VR image receiving device described in claim 11, characterized in that the synthetic VR image generation unit synthesizes the second CG VR image with the part that becomes the zenith region or the nadir region only in the time range in which the CG command is added to the part that becomes the zenith region or the nadir region of the entire time range of the live-action VR image.

13. The VR image receiving device described in claim 10, characterized in that the synthetic VR image generation unit generates the synthetic VR image using a partial live-action VR image of the area of ​​the live-action VR image received by the VR image receiving unit, excluding the part that becomes the zenith area or the nadir area.

14. The VR image receiving device described in claim 13, characterized in that the synthetic VR image generation unit generates the synthetic VR image using the partial live-action VR image only in the time range in which the CG command is added to the part that becomes the zenith or nadir region of the entire time range of the live-action VR image, and in other time ranges, generates the synthetic VR image using the live-action VR image including the part that becomes the zenith or nadir region.

15. a first step in which a command generation unit of the VR image transmission device generates a CG command including CG generation information indicating the content of the generation of a CGVR image, which is a VR image made of CG, and synthesis timing information indicating the synthesis timing indicating the temporal position within the live-action VR image of the CGVR image to be synthesized with the live-action VR image, which is a VR image made of live action; a second step in which a command addition unit of the VR image transmission device adds the CG command generated by the command generation unit to the live-action VR image; a third step in which the VR image transmitting unit of the VR image transmitting device transmits the live-action VR image to which the CG command has been added by the command adding unit to a VR image receiving device configured to generate the CG VR image in response to the CG command, combine the CG VR image with the live-action VR image, and display the combined image on a display.

1. An information processing method comprising:

16. A first step in which a VR image receiving unit of a VR image receiving device receives a live-action VR image to which a CG command is added, the CG command including CG generation information indicating the content of generation of a CGVR image, which is a VR image made of CG, and synthesis timing information indicating a time position within the VR image as the synthesis timing of the CGVR image with a live-action VR image; a second step in which a CGVR image generating unit of the VR image receiving device generates the CGVR image based on the CG generation information included in the CG command added to the live-action VR image received by the VR image receiving unit; a third step in which a composite VR image generating unit of the VR image receiving device generates a composite VR image by synthesizing the CGVR image generated by the CGVR image generating unit with the live-action VR image received by the VR image receiving unit at a timing based on the synthesis timing information included in the CG command; a fourth step of causing a composite VR image display unit of the VR image receiving device to display the composite VR image generated by the composite VR image generation unit on a display.

1. An information processing method comprising:

Images