Video display system, information processing method, and program

Abstract

A video display system for displaying a display video using a display device, the video display system comprising: a VR device including a display device for displaying a display video; an observation device mounted on a mobile body and moved, the observation device including a capturing unit for capturing a video, a data acquiring unit for acquiring data concerning the moving speed of the capturing unit, and a transmission unit for transmitting the captured video together with the data; a reception unit for receiving the video and data; a video generating unit for generating a display video including the received video; and a presenting unit which generates and outputs, from the received data, graphics corresponding to the moving speed of the capturing unit, and which causes the video generating unit to present the moving speed by superimposing the output graphics.

Description

[Technical Field]

[0001] The present disclosure relates to a video display system, an observation device, an information processing method, and a program. [Background technology]

[0002] In recent years, there has been active development of so-called head-mounted displays, which are head-mounted display devices. For example, Patent Document 1 discloses a head-mounted display that can present (i.e., display) content images and external world images. The head-mounted display disclosed in Patent Document 1 adjusts the brightness of at least one of the content images and the external world images, thereby reducing the sense of discomfort felt by the user when switching between the content image and the external world image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-090773 Summary of the Invention [Problem to be solved by the invention]

[0004] One application of display devices such as head-mounted displays, which take advantage of their high immersiveness, is to simulate an experience at a certain location by viewing video from a remote location. In this case, the display device must be able to provide appropriate video.

[0005] The present disclosure has been made in view of the above, and aims to provide a video display system and the like that is capable of displaying appropriate video. [Means for solving the problem]

[0006] In order to achieve the above-mentioned object, one aspect of the video display system according to the present disclosure is a video display system for displaying a display image on a display device, comprising: a VR device having the display device that displays the display image; an observation device that is mounted on a moving body and moves, the observation device having a shooting unit that shoots image, a data acquisition unit that acquires data related to the moving speed of the shooting unit, and a transmission unit that transmits the shot image together with the data; a receiving unit that receives the image and the data; an image generation unit that generates the display image including the received image; and a presentation unit that generates and outputs graphics according to the moving speed of the shooting unit from the received data, and superimposes the output graphics to cause the video generation unit to present the moving speed.

[0007] Furthermore, one aspect of the observation device according to the present disclosure is an observation device used in a video display system for displaying a display image on a display device, and includes a shooting unit that captures video as a wide-viewing-angle video, a data acquisition unit that acquires data relating to the movement speed of the shooting unit, a metadata acquisition unit that acquires metadata based on the acquired data, the metadata including the movement state of the shooting unit, and a transmission unit that transmits the captured wide-viewing-angle video together with the metadata.

[0008] Furthermore, one aspect of the information processing method according to the present disclosure is an information processing method for displaying a display image on a display device, which receives data relating to the movement speed of a camera unit that moves together with a moving object and captures the image, and generates and outputs graphics that indicate the movement state of the camera unit based on the movement speed of the camera unit in the received data, and which are superimposed on the captured image to present the movement state of the camera unit to a user of the display device.

[0009] These comprehensive or specific aspects may be realized as a system, device, integrated circuit, computer program, or computer-readable recording medium such as a CD-ROM, or may be realized as any combination of a system, device, integrated circuit, computer program, and recording medium. [Effects of the Invention]

[0010] According to the present disclosure, an image display system and the like capable of displaying appropriate images is provided. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a first diagram for explaining a conventional example. [Figure 2] FIG. 2 is a second diagram for explaining the conventional example. [Figure 3] FIG. 3 is a third diagram for explaining the conventional example. [Figure 4] FIG. 4 is a fourth diagram for explaining the conventional example. [Figure 5] FIG. 5 is a fifth diagram for explaining a conventional example. [Figure 6] FIG. 6 is a sixth diagram for explaining a conventional example. [Figure 7] FIG. 7 is a seventh diagram for explaining a conventional example. [Figure 8] FIG. 8 is an eighth diagram for explaining a conventional example. [Figure 9] FIG. 9 is a ninth diagram for explaining a conventional example. [Figure 10] FIG. 10 is a diagram illustrating a conventional example. [Figure 11] FIG. 11 is an eleventh diagram for explaining a conventional example. [Figure 12] FIG. 12 is a diagram illustrating a conventional example. [Figure 13] FIG. 13 is a diagram illustrating a conventional example. [Figure 14] FIG. 14 is a diagram showing a schematic configuration of a video display system according to an embodiment. [Figure 15] FIG. 15 is a diagram showing an example of an image displayed in the image display system according to the embodiment. [Figure 16]FIG. 16 is a block diagram illustrating a functional configuration of a video display system according to an embodiment. [Figure 17] FIG. 17 is a flowchart showing the operation of the video display system according to the embodiment. [Figure 18] FIG. 18 is a more detailed block diagram showing the functional configuration of the display device according to the embodiment. [Figure 19] FIG. 19 is a flowchart showing the operation of the video display system according to the embodiment. [Figure 20] FIG. 20 is a schematic diagram of a configuration example of a video display system according to an embodiment. [Figure 21] FIG. 21 is a diagram showing the correspondence between the moving state (flight state) of a moving object and the graphics that are generated and superimposed. [Figure 22] FIG. 22 is a diagram showing the correspondence between the moving state (flight state) of a moving object and the graphics that are generated and superimposed. [Figure 23] FIG. 23 is a diagram illustrating an example of an operation flow of the video display system according to the embodiment. [Figure 24] FIG. 24 is a schematic diagram of a configuration example of a video display system according to another example of the embodiment. [Figure 25] FIG. 25 is a diagram showing an example of a display image generated by the configuration of FIG. [Figure 26] FIG. 26 is a diagram showing an example of a display image generated by the configuration of FIG. [Figure 27] FIG. 27 is a diagram showing an example of a display image generated by the configuration of FIG. [Figure 28] FIG. 28 is a diagram showing an example of a display image generated by the configuration of FIG. [Figure 29] FIG. 29 is a diagram showing an example of a display image generated by the configuration of FIG. [Figure 30] FIG. 30 is a diagram showing an example of a display image generated by the configuration of FIG. [Figure 31] FIG. 31 is a schematic diagram of a configuration example of a video display system according to still another example of the embodiment. [Figure 32] FIG. 32 is a diagram showing an example of a display image generated by the configuration of FIG. [Figure 33] FIG. 33 is a diagram illustrating an example of a functional configuration of a video display system according to an embodiment. [Figure 34] FIG. 34 is a diagram illustrating an example of an operation flow for compositing graphics according to the embodiment. [Figure 35] FIG. 35 is a diagram illustrating another example of the functional configuration of the video display system according to the embodiment. [Figure 36] FIG. 36 is a diagram illustrating an example of an operation flow for compositing graphics according to the embodiment. [Figure 37] FIG. 37 is a diagram illustrating an example of the configuration of metadata according to the embodiment. [Figure 38] FIG. 38 is a diagram illustrating an example of the configuration of metadata according to the embodiment. [Figure 39] FIG. 39 is a diagram illustrating a configuration example in which the video display system according to the embodiment is realized by using the cloud. [Figure 40] FIG. 40 is a diagram illustrating a configuration example in which the video display system according to the embodiment is realized by using the cloud. [Figure 41] FIG. 41 is a diagram illustrating a configuration example in which the video display system according to the embodiment is realized by using the cloud. DETAILED DESCRIPTION OF THE INVENTION

[0012] (Knowledge that served as the basis for disclosure) In recent years, display devices have been developed that allow users to wear them on their heads, placing a display unit in front of their eyes and viewing images as if they were on a large screen. These display devices, called head-mounted displays (HMDs), have the characteristic of allowing users to view images as if they were on an optically large screen. Some HMDs also display images with parallax shifts corresponding to the user's right and left eyes, allowing users to perceive the images as three-dimensional. Furthermore, recent improvements in communication quality have made it possible to view images captured by remotely located observation devices in near real time with a delay of a few milliseconds to tens of milliseconds, allowing users to experience the sensation of being present at a location without actually visiting it. This technology has also enabled virtual tourism experiences (hereinafter referred to as "pseudo-tourism" or "virtual reality (VR) tourism"), such as sightseeing trips, exhibition tours, inspection tours, factory tours, and tours of art galleries, museums, zoos, and aquariums.

[0013] In this type of VR tourism, cameras capable of capturing 360-degree (all longitude) images (so-called spherical cameras) are used as observation devices. Images captured by the observation devices have a wide viewing angle, and users of the display device can crop and display any direction from the image. For example, if a display device is equipped with a function that can detect the direction the user is facing, it can crop and display a portion of the 3D image space that corresponds to the user's orientation, making it possible to provide viewing experiences that meet the needs of many users from a single camera image.

[0014] When a user is viewing a portion of a video in a given direction and the observation device moves, the video may move in a way that the user did not intend. For example, if the observation device moves in a way that causes the 3D video space to move in the 12 o'clock direction, and the user is facing the 2 o'clock direction in the 3D video space, the video will suddenly move in the 10 o'clock direction from the user's perspective. This type of movement creates a discrepancy between the virtual and real worlds, causing the user to feel uncomfortable. This phenomenon is known as VR sickness, and it makes it difficult to watch videos for long periods of time.

[0015] In recent years, by mounting a camera on a small flying object called a drone, it has become possible to combine the above-mentioned VR experience with the experience of flying. In such applications, the effects of VR sickness become more pronounced, and countermeasures are required.

[0016] Therefore, the present disclosure aims to provide a video display system that can present the user with the direction in which an image moves when the image moves, in order to suppress the occurrence of such VR sickness. While the present disclosure describes a 360-degree wide-viewing-angle image captured by an observation device, the wide-viewing-angle image may be an image captured over any angle range, such as 270 degrees or more, 180 degrees or more, or 150 degrees or more. Such wide-viewing-angle image may have a viewing angle at least wider than the viewing angle of the image displayed on the display device by the user. Furthermore, the present disclosure describes a video display system that assumes both image movement occurring within a horizontal plane and at a plane intersecting the horizontal plane, including a vertical component. However, the present disclosure is also applicable to image movement occurring only in a horizontal plane or at a plane intersecting the horizontal plane.

[0017] Conventional video display systems and the like will be described in more detail below with reference to the accompanying drawings. Figure 1 is the first diagram for explaining a conventional example. As shown in Figure 1, a service called VR tourism (first-person experience) has been provided. With VR tourism, if the local VR space is properly reproduced, it is possible to have a sightseeing experience that makes you feel as if you are actually in that place. Examples of services using 360° camera photography include FirstAirlines (https: / / firstairlines.jp / index.html) and Tabisuke (https: / / www.tokyotravelpartners.jp / kaigotabisuke-2 / ). Examples of services using 3D CG (computer graphics) include Google Earth VR and Boulevard (https: / / www.blvrd.com / ).

[0018] Fig. 2 is the second diagram for explaining a conventional example. As shown in Fig. 2, in addition to VR tourism, a service (also called a third-person experience) is also provided in which footage taken on-site is displayed on a display device such as a television and the footage is viewed from a third-person perspective. The third-person experience offers services tailored to the user, such as expert guides, and has the advantage of being able to generate revenue if it suits the individual's interests.

[0019] FIG. 3 is a third diagram illustrating a conventional example. As shown in FIG. 3, the basic components of VR tourism include a VR system main body 311, a controller 312, a computer or smartphone 313, a network, a cloud 314, and an observation system 315. While the VR system main body 311 was previously only available as a heavy, HMD type that covered a large portion of the face, compact, eyeglass-like VR glasses have become more widely used due to their ease of use over extended periods of time. The VR system main body 311 can be classified into an all-in-one type that includes all the necessary functions, and a tethered type that delegates some functions to a computer or smartphone 313. The controller 312 is used to select menus and navigate the VR space. The computer or smartphone 313 may have only communication functions or may constitute part of the VR system.

[0020] A network, cloud 314, connects the observation system 315 and the VR system 311, and some functions of the observation system 315 or the VR system 311 may be performed by a computer system on the cloud. The observation system 315 uses a wireless 360-degree camera, or a 360-degree camera, 180-degree camera, or wide-angle camera connected wirelessly or by cable to a computer or smartphone 313. Through these devices, the user can view the guide and the buildings and scenery of the sightseeing spots in the VR space.

[0021] While the explanation uses an example of VR tourism using a 360° camera, it is acceptable to use a 180° camera or other device that allows participants to change their viewpoint using VR glasses. Also, while an example of filming and guiding actual scenery is sometimes described, sightseeing can also be realized by using a virtual camera in a virtual space created using computer graphics instead of actual scenery, with the guide also entering the virtual space using VR glasses or the like and playing the video within the virtual space. Therefore, the present invention can also be applied to such uses. A typical example of the above is VR travel to areas and spaces that are difficult for ordinary travelers to reach, such as a trip to the moon.

[0022] FIG. 4 is a fourth diagram illustrating a conventional example. FIG. 4 shows the schematic configuration of a VR tourism service using a 360-degree camera (unguided: upper panel (hereinafter referred to as Conventional Example 1); guided: middle panel (hereinafter referred to as Conventional Example 2)) and a conventional example of a Zoom (registered trademark) tourism service, an example of a third-person experience (lower panel (hereinafter referred to as Conventional Example 3)). Hereinafter, in this invention, audio, audio data, and audio information are defined to include not only conversation but also music and, in some cases, audio signals containing ultrasonic waves outside the audible range. In a VR tourism service, the observation system (tourist destination) transmits pre-recorded video, or the VR system operates a 360-degree camera, robot, or drone, allowing the VR video to be viewed on the VR system. As shown in the middle panel, a guide or camera operator is present on the observation system, and VR video from a 360-degree camera or the like can be enjoyed as VR on the VR system. Additionally, as shown in the bottom row, in the third-person experience, 2D images are sent from the observation system in 2D via a remote conversation service with multiple people using audio and video, such as Zoom, allowing users to enjoy viewing images of tourist spots in remote locations.

[0023] FIG. 5 is the fifth diagram for explaining a conventional example. The overall system configuration of conventional example 2 will be explained. Conventional example 1 differs from conventional example 2 in that it uses pre-recorded VR video or the operation is performed from the VR system side, and this difference will also be explained. The observation system of conventional example 2 consists of a camera for VR shooting, for example a 360° camera, and a communication device for sending the captured information to a remote location.

[0024] A 360° camera for VR filming stitches together images from multiple cameras shooting in different directions to create a single video, which is then mapped onto a plane using, for example, equirectangular projection (ERP), appropriately compressed as an ERP image, and transmitted to a remote VR system from a communication device along with audio data captured by a microphone. 360° cameras may also be mounted on robots or drones. While 360° cameras and the robots or drones they are mounted on are operated by a photographer or guide, in Conventional Example 1, they may also be operated by the VR system, or the VR system may receive pre-recorded video.

[0025] The VR system, in contrast to the observation system, converts the received planar image (ERP image) into a spherical image, cuts out a portion according to the observer's orientation and position, and displays it on a VR display device. In Conventional Example 3, the received image is 2D, so it is displayed in 2D, and in most cases a 2D display device, such as a tablet, smartphone, or TV, is used. The above is also true when receiving pre-recorded image in Conventional Example 1.

[0026] When operating from the VR system side, the observation system may operate in conjunction with the orientation and position of the VR system, or the observation system may operate by operating keys on a mouse, tablet, joystick, keyboard, etc., or by selecting menus or icons on the screen. In this case, appropriate control data must be sent from the VR system to the observation system, and the observation system's status, i.e., orientation and position, must be sent from the VR system to the VR system.

[0027] Figure 6 is the sixth diagram for explaining a conventional example. Using the comparison between the 360-degree video shown in Figure 6 and a normal video, we will explain the resolution when viewing 360-degree video on a VR system. When viewing 4K 360-degree video on a VR device with a field of view (FOV) of 100 degrees, the resolution of the video cropped for VR display is only 1067 x 600 (about twice that of SD video). A VR system using a panel with a resolution of 2K x 2K for each eye displays on a square panel, so the vertical direction is further stretched by two times, resulting in an image with extremely low resolution.

[0028] The resolution of 8K video for VR display is 2133 x 1200, which is 1.23 times the area of ​​Full HD (1920 x 1080) in terms of data volume, but since it is stretched twice vertically, the image is about the same as Full HD. When shooting 11K (10560 x 5940), the VR resolution is 2933 x 1650, which is equivalent to a VR system.

[0029] To provide a high-resolution, highly immersive VR tourism experience, it is necessary to shoot in at least 8K, and ideally 11K. Shooting in 8K or 11K requires large equipment, high video transfer rates, and large storage capacity, which makes filming and distribution expensive.

[0030] For this reason, it is essential to avoid VR sickness, make it easy to understand and use, and encourage more users to use it, thereby lowering the cost per user. In addition, the effective use of VR recorded content will also be important for business viability.

[0031] FIG. 7 is a seventh diagram for explaining a conventional example. An example of the configuration of the main functions of conventional examples 1 and 2 will be explained function by function. An observation system 751 of conventional examples 1 and 2 includes a VR shooting means 762 (VR shooting camera) for performing VR shooting, a VR image processing means 758 for processing images shot by the VR shooting means 762 to make them suitable for transmission, a VR image compression means 756 for compressing the VR images processed by the VR image processing means 758 to make them suitable for transmission at a data rate and in a video signal format, an audio input means 763 consisting of a microphone for inputting guide and surrounding audio, an audio compression means 760 for making the audio signal input by the audio input means 763 suitable for transmission at a data rate and in a video signal format, a graphics generation means 759 for generating auxiliary information as graphics, a VR image compression means 760 for compressing the VR images processed by the VR image processing means 758 to make them suitable for transmission at a data rate and in a video signal format, a graphics generation means 759 for generating auxiliary information as graphics, a VR image compression means 760 for compressing the VR images processed by the VR image processing means 758 to make them suitable for transmission at a data rate and in a video signal format, an audio input means 763 consisting of a microphone for inputting guide and surrounding audio, an audio compression means 760 for compressing the audio signal input by the audio input means 763 to make them suitable for transmission at a data rate and in a video signal format, a graphics generation means 759 for generating auxiliary information as graphics, a VR image compression means 760 for compressing the VR images processed by the VR image processing means 758 ... The system is composed of a compression means 756, a multiplexing means 757 that converts the video signal, audio signal, and graphics information compressed by the graphics generation means 759 and audio compression means 760 into signals suitable for transmission, a communication means 754 that sends the communication observation signals multiplexed by the multiplexing means 757 to multiple VR systems 701 and receives communication audio signals from the multiple VR systems 701, a separation means 755 that extracts compressed audio signals from the communication audio signals received by the communication means 754, an audio decoding means 761 that extracts audio signals from the compressed audio signals from the separation means 755, and an audio output means 764 that outputs the audio signals decoded by the audio decoding means 761 as sound.

[0032] In this example, the VR video processing means 758, VR video compression means 756, and graphics generation means 759 are realized within the GPU, and the audio compression means 760, multiplexing means 757, separation means 755, and audio decoding means 761 are realized within the CPU, but this is not necessarily limited to this, and in a simpler configuration, the CPU and GPU may be realized as a single processor, but the functional configuration and operation are the same.

[0033] The VR shooting means 762 is, for example, a 360° camera, but is composed of multiple cameras that shoot in different directions, and the VR image processing means synthesizes (stitches) the outputs of the multiple cameras to create a single video, which is then mapped onto a plane using, for example, raw cylindrical projection (ERP) and output as an ERP image.

[0034] The VR system 701 of the conventional examples 1 and 2 is the opposite of the observation system 751, and includes a communication means 716 that receives a communication observation signal sent from the observation system 751 or sends the audio input by the VR system 701 to the observation system 751 as communication audio information, a separation means 715 that separates and outputs a compressed VR video (ERP image), graphics information, and compressed audio information from the communication observation signal from the communication means 716, a VR video decoding means 710 that decodes the compressed VR video (ERP image) from the separation means 715, a VR video decoding means 710 that converts the ERP image from the VR video decoding means 710 into a spherical video, cuts out a part of it according to control information from the VR control means 707, and makes it into a video that can be displayed by the VR display means 704, and the separation means 715 outputs The VR display control means 708 outputs VR video to be displayed on the VR display means 704 together with graphics information of a graphics generation means 712 that converts the received graphics information into graphics to be displayed, the VR display means 704 outputs the VR video from the VR display control means 708 so that it can be viewed with both eyes, and the outputs of a rotation detection means 703 that detects the tilt of the VR display means 704 in the front, rear, left, and right directions or the direction of the whites of the eyes and a position detection means 702 that detects the position of the VR display means 704 in the front, rear, left, right, and height directions are sent to the VR control means 707, and the video displayed on the VR display means 704 and the audio output from the audio playback control means 709 are appropriately controlled by the output of the VR control means 707. The compressed audio information separated by the separation means 715 is decoded by an audio decoding means 713 and sent as audio information to the audio playback control means 709, where the audio playback control means 709 performs balance in the front, rear, left, right, and height directions, and in some cases, frequency characteristics and delay processing, or synthesis of an alarm as the VR system 701, according to the control information from the VR control means 707. The graphics generation means 712 also generates graphics for displaying the system menu and warnings of the VR system 701, and these are displayed by the VR display means 704 while being superimposed on the VR image.The VR system 701 is equipped with an audio input means 706 for inputting the voice of the user of the VR system 701, and the audio information from the audio input means 706 is compressed by an audio compression means 714 and sent as compressed audio information to a multiplexing means 717, where it is sent as audio information for communication from a communication means 716 to an observation system 751.

[0035] 8 is an eighth diagram for explaining a conventional example. As a typical example of realizing an observation system of conventional example 2, an example of realizing a 360° camera 801 will be explained.

[0036] A typical example of a 360° camera 801 combines two imaging systems, namely an ultra-wide-angle lens 854, a shutter 853, and an imaging element 852, to capture 360° video in all directions. In some cases, two or more imaging systems are combined to capture higher quality images, and in this example, the VR camera 804 is illustrated as having two or more imaging systems. The imaging system may be configured by combining independent cameras, in which case there is generally a high-speed digital video I / F after the video ADC 851, which is connected to a high-speed digital video input connected to a video system bus connected to a GPU (Graphics Processing Unit) 803 or CPU (Central Processing Unit) 802; however, the following description will be given assuming that the systems are integrated.

[0037] The main components of the 360° camera 801 are the VR camera 804 from the multiple imaging systems mentioned above, the GPU 803 that mainly processes video data and graphics, the CPU 802 that performs general data processing, input / output processing, and overall control of the 360° camera 801, and an EEPROM (Electrically Erasable Programmable Read Only Memory) for storing programs for operating the CPU 802 and GPU 803. ROM) 813, RAM 814 used to store data for the operation of CPU 802 and GPU 803, SD card (registered trademark) 821 which is a removable memory for saving video, audio and programs, wireless communication element 820 which performs wireless communication via WiFi (registered trademark) or Bluetooth (registered trademark) for exchanging data with the outside and receiving operations from the outside, buttons and display element 808 for operation and display, battery 807 and power control element 812, an audio input unit consisting of multiple microphones (microphone group 819) or microphone terminal 825 for inputting audio, microphone amplifier 818, ADC 817, speaker 826 or headphone terminal 824, amplifier 823, audio output unit consisting of DAC 822, The system is comprised of a video bus that connects the VR shooting camera 804 and CPU 802 and is used to read digital video data; a memory bus that connects the aforementioned EEPROM 813, RAM 814, SD card 821, GPU 803, and CPU 802 and exchanges data with the memory; a system bus that connects the aforementioned CPU 802, GPU 803, wireless communication element 820, audio input unit, and audio output unit and exchanges control and data; an I / O bus that connects the aforementioned buttons and display element 808, power control element 812, and the audio input unit, audio output unit, and VR shooting camera 804 (not shown) and exchanges low-speed data; and several bus conversion units 815 and 816 that connect the respective buses. A motion / position detection unit 860 is further connected to the I / O bus. Whether some processing is performed by the GPU 803 or the CPU 802 may differ from this example, and the bus configuration may also differ from this example, but the functional configuration and operation described below remain the same.

[0038] The VR shooting camera 804 is composed of a lens 854 for capturing wide-angle images, an image sensor 852 that converts the light collected by the lens 854 into an electrical signal, a shutter 853 that is located between the lens 854 and the image sensor 852 and blocks light, an aperture (not shown here) that is located in the same position as the shutter 853 and controls the intensity of light from the lens 854, and an ADC 851 that converts the analog electrical signal from the image sensor 852 into a digital video signal, and although not shown, each of these components is controlled by the CPU 802 via an I / O bus, and their status is notified to the CPU 802.

[0039] The buttons include a power switch 806 for turning the power on and off, a shooting start / end button 811 for starting and stopping shooting, a shooting mode selection button 809 for changing the shooting mode (although this may not be provided), and a zoom button 810 for moving the lens 854 and digitally controlling the angle of view to zoom in and out.

[0040] The power supply control element 812, which may be integrated with the battery 807, stabilizes the voltage, manages the battery capacity, and supplies power to all components (not shown). It also supplies power to the HMD / VR glasses via USB or AV output.

[0041] Each function implemented by the GPU 803 is implemented by dedicated hardware and programs for image processing, etc., whereas functions implemented by the CPU 802 are generally implemented by general-purpose hardware and programs. As an example, the GPU 803 is used to implement a VR video processing unit 842, a VR video compression unit 841, and a graphics generation unit 843. Also, as an example, the CPU 802 is used to implement a memory control unit 835, a multiplexing unit 832, an audio compression unit 833, an audio decoding unit 834, and a separation unit 831.

[0042] Fig. 9 is a ninth diagram for explaining a conventional example. Based on Fig. 9, an example of realizing a VR system 901 will be described as a typical example of realizing the observation system of conventional example 2. In this example, the VR system 901 is configured from a computer or smartphone 951 and an HMD or VR glasses 902 connected to it. There are also examples where the VR system is realized using only the HMD or VR glasses 902, but in that case, the functions of the CPU and GPU of both are combined into one, and the peripheral functions can also be considered to be integrated.

[0043] The main components of the computer / smartphone 951 in the VR system 901 are a high-speed communication element 970 such as WiFi or Ethernet (registered trademark) for connecting to the observation system, a GPU 954 that mainly processes video data and graphics, a CPU 965 that performs general data processing and controls the entire computer / smartphone 951, a non-volatile memory 962 such as a hard disk or flash memory for storing programs for operating the CPU 965 and GPU 954, RAM 961 used to store data for operating the CPU 965 and GPU 954, a power switch 963, and a memory for supplying power to each part. the power supply control element 964, an AV output 952 for outputting video and audio signals to the HMD / VR glasses 902, an I / F such as a USB 953 for controlling the HMD / VR glasses 902 and acquiring data therefrom, a memory bus for connecting the RAM 961 and nonvolatile memory 962 and for access by the CPU 965 and GPU 954, a system bus for accessing the CPU 965 and GPU 954 to the AV output 952, USB 953, and communication element 970, a bus connection (bus conversion unit 960) for connecting the system bus and memory bus, and a display device, an input device for operation, and other general-purpose I / Fs, all of which are not shown here.

[0044] Whether some processing is performed by the GPU 954 or the CPU 965 may differ from this example, and the bus configuration may also differ from this example, but the functional configuration and operation described below remain the same. As an example, the GPU 954 is used to realize a motion / position detection processing unit 955, a VR control unit 956, a VR display control unit 957, a VR video decoding unit 958, and a graphics generation unit 959. Also, as an example, the CPU 965 is used to realize an audio decoding unit 966, an audio playback control unit 967, a multiplexing unit 968, and a separation unit 969.

[0045] The AV output 952 and USB 953 can also be replaced with a high-speed bidirectional I / F, such as USB Type-C (registered trademark). In that case, the HMD / VR glasses 902 are also connected via the same I / F, or via a converter that converts the I / F. Generally, when sending video via USB 953, the CPU 965 or GPU 954 performs appropriate video compression to reduce the data volume, and the video is then sent to the HMD / VR glasses 902 via USB 953.

[0046] The main components of the HMD / VR glasses 902 in the VR system 901 are a microphone 906 for inputting voice, a microphone amplifier 917, an audio input unit consisting of an ADC 918, a speaker 907 or a headphone terminal 908, an amplifier 919, and an audio output unit consisting of a DAC 920, a VR display unit consisting of two pairs of lenses 904 and a display element 905 for the user to view VR images, a motion / position sensor 903 consisting of a motion / position detection unit and an orientation detection unit consisting of a gyro sensor, a camera or an ultrasonic microphone, etc., and a Bluetooth for communicating with a controller (not shown). a wireless communication element 927 such as a Bluetooth, a volume button 909 for controlling the output volume from the audio output unit, a power switch 921 for turning the power of the HMD / VR glasses on and off, a power control element 924 for power control, the aforementioned EEPROM 913, RAM 914, a memory bus connecting the SD card and GPU 910, and CPU 915 and exchanging data with the memory, the aforementioned CPU 915, GPU 910, wireless communication element 927, an AV input 925 for receiving video signals and audio signals from a computer / smartphone 951, an I / F such as a USB 926 for receiving control signals from a smartphone 951 and sending video and audio signals and motion and position data; a CPU 915 that mainly performs audio compression (realized by an audio compression unit 916), controls switches and power, and controls the HMD / VR glasses 902 as a whole; and a GPU 910 that mainly performs video display processing (realized by an video display processing unit 912) for adjusting the video on the VR display unit and motion and position detection (realized by a motion and position detection unit 911) for correcting and shaping the motion and position information sent to the computer / smartphone 951 from information from a motion and position sensor 903. EEPROM 913 for storing programs and data for operating the CPU 915 and GPU 910, RAM 914 for storing data when the CPU 915 and GPU 910 are operating, a memory bus for connecting the CPU 915, GPU 910, RAM 914 and EEPROM 913, a system bus to which the CPU 915, GPU 910, USB 926, audio input unit, audio output unit and wireless communication element 927 are connected and which performs control and data exchange, the above-mentioned buttons and power control element 924, motion / position sensor 903 and also an audio input unit (not shown),It is composed of an I / O bus that controls and exchanges low-speed data, including an audio output unit and a VR shooting camera, and several bus conversion units 922 that connect each bus.Whether some processing is performed by the GPU 910 or the CPU 910 may differ from this example, and the bus configuration may also differ from this example, but there is no difference in the functional configuration and operation described below.

[0047] Since the video data from the AV input 925 is large in volume and high in speed, if the system bus does not have a sufficient speed, it is illustrated as being taken directly into the GPU 910 .

[0048] The image information captured by the camera of the motion / position sensor 903 may be sent to a display element as information for the user to check the area around the HMD / VR glasses 902, or may be sent to a computer / smartphone 951 via USB 926 to monitor whether the user is in a dangerous situation.

[0049] The power supply control element 924 receives power from a USB 926 or an AV input 925, stabilizes the voltage, manages the battery capacity, etc., and supplies power to all components (not shown). In some cases, a battery 923 may be provided internally or externally and connected to the power supply control element 924.

[0050] The states of the buttons and cursor of the controller (not shown) are acquired by the CPU 915 via the wireless communication element 927 and are used for button operation, movement, and application operation in the VR space. The position and orientation of the controller are detected by a camera or ultrasonic sensor in the motion / position detection unit, and after appropriate processing is performed by the motion / position sensor, the data is used for control by the CPU 915 and is also sent to the computer / smartphone 951 via USB 926 and used for programs executed by the CPU 915 or for graphics drawing and image processing executed by the GPU 910. The basic operations are not directly related to the present invention and will not be described here.

[0051] 10 is a tenth diagram for explaining a conventional example. An example of realizing an integrated VR system 1001 in which HMD / VR glasses are provided with a function for VR in a computer / smartphone will be explained.

[0052] As can be seen in Figure 10, the functions of the computer / smartphone and HMD / VR glasses are integrated, and the functions of the CPU and GPU are realized by a single CPU and GPU.

[0053] The communication element 1033 is typically a WiFi device that performs wireless communication, and since it does not have a power cable, it has a battery 1026. It has an interface with a general-purpose computer such as a USB 1034 for charging the battery 1026 and for initial setup.

[0054] The integrated VR system 1001 does not require an AV output, AV input, or USB to connect a computer / smartphone and an HMD / VR glasses, and therefore enables high-quality, delay-free transmission of AV information and efficient control. However, as an integrated system, there are size limitations, and it may not be possible to use a high-performance CPU 1027 or GPU 1006 due to limitations on power, heat, and space, which may result in limited VR functionality.

[0055] However, not being connected by a cable increases the degree of freedom and broadens the range of applications.

[0056] Furthermore, by implementing some of the functions on a computer in the cloud, it is possible to compensate for the lack of performance and realize a highly functional application.

[0057] 8 and 9, the integrated VR system 1001 also includes a lens 1002, a display element 1011, a microphone 1003, a microphone amplifier 1007, an ADC 1009, a speaker 1004, a headphone terminal 1005, an amplifier 1008, a DAC 1010, a RAM 1019, an EEPROM 1020, a bus converter 1021, a motion position sensor 1022, a power switch 1023, a volume button 1024, and a power control element 1025. In addition, a video display process 1012, a motion and position detection process 1013, VR control 1014, VR display control 1015, motion and position detection 1016, VR video decoding 1017, and graphics generation 1018 are implemented using a GPU 1006. Furthermore, the CPU 1027 is used to implement audio compression 1028 , audio decoding 1029 , audio playback control 1030 , multiplexing 1031 , and demultiplexing 1032 .

[0058] Fig. 11 is an eleventh diagram for explaining a conventional example. Based on Fig. 11, a more detailed configuration of a VR image processing unit 1103 that processes images captured by a VR shooting camera 1151 in the observation systems of conventional examples 1 and 2 will be described.

[0059] As mentioned above, the VR shooting camera has multiple cameras cm, typically cameras cm with ultra-wide-angle lenses, for capturing 360° images from above, below, and all around. Individual rectangular images with the same pixels captured by each camera cm are input to the VR image processing unit 1103, which is implemented by a program within the GPU 1101 or a dedicated circuit.

[0060] The VR video processing unit 1103 first evaluates the shooting direction of each camera cm and the images obtained by shooting the multiple input images, and inputs the images shot by each camera cm to a stitching processing unit 1105 which performs processing to synthesize and stitch together the input images to create a continuous spherical image. The spherical image data output from the stitching processing unit 1105 is mapped onto a plane by a VR video mapping unit 1104 using, for example, equirectangular projection (ERP), and is output from the VR video processing unit 1103 as an ERP image and passed to the subsequent VR video compression unit 1102.

[0061] Note that the connection between the video bus and the cameras is illustrated as if each camera is connected to the bus, but it is also possible to combine them into one signal within the VR shooting camera 1151, and send the images shot by each camera in a time-division manner to the video bus and input them to the VR video processing unit 1103. In a simple configuration, since there are two cameras cm, it is also possible to configure the GPU 1101 to receive the outputs of the two cameras respectively, rather than using a bus, and have the VR video processing unit 1103 receive and process the images shot in parallel.

[0062] 12 is a twelfth diagram for explaining a conventional example. A more detailed configuration of the VR display control unit 1204 of the VR systems of conventional examples 1 and 2 will be explained based on FIG.

[0063] As described above, the VR display control unit 1204 is realized by a program or a dedicated circuit in the GPU 1201 of the computer / smartphone, and is composed of a mapping unit 1206 and a display VR video conversion unit 1205.

[0064] It operates as follows: The communication element 1261 receives communication data sent from the observation system, the compressed video is separated by the separation unit 1232 of the CPU 1231, the video is received by the GPU 1201 via the memory bus, and the video is decoded by the VR video decoding unit 1207 to become a flat video (ERP image). The flat video is converted into a 360° spherical video by the mapping unit 1206 of the VR display control unit 1204, and the next display VR video conversion 1205 cuts out the portion to be displayed on the VR display means 1202 based on the control information output by the VR control unit 1203.

[0065] Specifically, the center of the ERP image is the front view, and is the origin of the 360° spherical image. The initial image of the VR image displayed on the VR display means 1202 is centered on the origin, and depending on the capabilities of the VR display means 1202, the image for the right eye is slightly shifted to the right, and the image for the left eye is slightly shifted to the left, and the image is cut out using the initial setting value in the height direction, and displayed on the display elements for the right eye and left eye. From here, the position of the cutout changes depending on whether the VR system is rotated left or right, or whether it is viewed up or down.

[0066] Generally, images from a 360° camera do not change when the VR system moves, but if they are generated using CG, their position will change when the VR system moves or when the controller is operated.

[0067] The initial value for cutting out from a 360° spherical video may be the previous cutting position, but generally there is a function to return it to the initial position.

[0068] 13 is a thirteenth diagram for explaining the conventional example. An example of the operation of the conventional example 2 will be explained based on FIG.

[0069] In the observation system, audio is input through the audio input unit (microphone group, microphone terminal, microphone amplifier, ADC) (S1325), and the audio is compressed by the audio compression unit (S1326).

[0070] At the same time, video is shot using multiple cameras (lenses, shutters, image sensors, ADC) of the VR shooting camera (S1321), and the stitching processing unit of the VR video processing unit stitches the images into a spherical image with camera 1 at the center (S1322).The VR video mapping unit generates an ERP image using equirectangular projection or the like (S1323), and the VR video compression unit appropriately compresses the images (S1324).

[0071] The compressed ERP image and audio information are multiplexed by the multiplexing unit (S1327) into a transmittable format, and are sent (transmitted) to the VR system by the wireless communication element (S1328).

[0072] As time passes, the device may move to a new direction or position (S1329), and audio input and transmission from multiple VR cameras are repeated.

[0073] Here, graphics information is omitted, although it may be superimposed on the video before compression or may be multiplexed as graphics information together with video and audio.

[0074] In the VR system, the computer / smartphone receives information from the observation system using a communication element (S1301) and sends it to the separation unit. The separation unit separates the compressed video information from the compressed audio information (S1302). The compressed audio information separated by the separation unit is sent to the audio decoding unit and decoded (S1303) to produce uncompressed audio information. The audio information is sent from the audio decoding unit to the audio playback control unit, where it is processed based on the position and orientation information of the VR observation system sent from the GPU's VR control unit via the system bus (S1304). The processed audio information is sent via the system bus, AV output, or USB to the audio output unit (DAC, amplifier, speaker, and headphone jack) of the HMD / VR glasses, where it is output as audio (S1305). Audio processing includes controlling the volume balance between left and right and within the space, changing frequency characteristics, delay, movement within the space, similar processing for only specific sound sources, and adding sound effects.

[0075] The compressed video signal is sent from the separation unit of the computer / smartphone CPU to the VR video decoding unit of the GPU via the memory bus, where it is decoded (S1307) and input as an ERP image to the VR display control unit. In the VR display control unit, the mapping unit maps the ERP image onto a 360° spherical image (S1308), and the display VR image conversion unit extracts the appropriate portion of the image from the 360° spherical image based on the position and orientation information of the VR system from the VR control unit (S1309), and the VR display unit (display element, lens) displays it as a VR image (S1310).

[0076] The image display and audio output are performed repeatedly after receiving from the observation system.

[0077] Regarding graphics, there are cases where the graphics are separated at the same time as the video and audio are separated and then superimposed on the VR video by the VR display control unit, or where the graphics are generated within the VR system and then superimposed on the VR video, and so a description of this will be omitted here.

[0078] (Disclosure Summary) The outline of the present disclosure is as follows.

[0079] An image display system according to one embodiment of the present disclosure is an image display system for displaying a display image on a display device, and includes: a VR device having a display device that displays the display image; an observation device that is mounted on a moving body and moves, the observation device having a shooting unit that shoots image, a data acquisition unit that acquires data related to the moving speed of the shooting unit, and a transmission unit that transmits the shot image together with the data; a receiving unit that receives the image and data; an image generation unit that generates a display image including the received image; and a presentation unit that generates and outputs graphics according to the moving speed of the shooting unit from the received data and superimposes the output graphics to cause the image generation unit to present the moving speed.

[0080] Such a video display system uses data to generate graphics corresponding to the moving speed of the camera unit. Then, by overlaying the graphics, the moving speed can be presented to the user. As a result, problems caused by the camera unit moving unexpectedly by the user, such as a sudden movement of the camera unit, can be reduced. Therefore, the video display system can display appropriate images from the perspective of reducing problems caused by the camera unit moving unexpectedly by the user.

[0081] Furthermore, for example, the data may include information relating to the moving state of the moving object, and the graphics may be an arrow indicating the moving state on the displayed image, displayed in a manner corresponding to the moving state.

[0082] This allows the movement state of the moving object to be presented by the arrow in a manner that corresponds to the movement state of the moving object.

[0083] Furthermore, for example, the graphics may display a mask, which is an image for covering at least a part of the displayed image other than the side in the direction of movement.

[0084] This allows the user to be informed of the movement state by graphics that display a mask, which is an image that covers at least a portion of the displayed image other than the movement direction side.

[0085] Furthermore, for example, the moving body may be an air vehicle.

[0086] This makes it possible to display appropriate images of observation equipment that is mounted on an aircraft and moves.

[0087] Furthermore, for example, the graphics may display a mask to obscure the area vertically below as seen from the aircraft.

[0088] This allows the user to be informed of the state of movement by graphics that display a mask that obscures the area vertically below as seen from the aircraft.

[0089] Furthermore, for example, the data acquisition unit may acquire data relating to the moving speed of the image capture unit from an operation history of operating a control device for controlling the movement of a moving object.

[0090] This makes it possible to automatically obtain data relating to the moving speed of the imaging unit from the operation history of the control device for controlling the movement of the moving body.

[0091] Furthermore, for example, the moving state may be information indicating changes in the moving speed and the direction of movement of the observation device, and the continuing movement, and the moving speed may be information including the status of changes in the speed of the observation device.

[0092] According to this, information indicating changes in the moving speed and change and continuation of the moving direction of the observation device can be used as the moving state, and information including the status of changes in the speed of the observation device can be used as the moving speed.

[0093] Furthermore, for example, the image capturing unit may capture a virtual image by capturing an image in a virtual image space created by computer graphics.

[0094] This makes it possible to display appropriate images of the image capturing unit moving within a virtual image space created by computer graphics, from the perspective of reducing problems that may arise when the image capturing unit moves in a way that is unexpected by the user.

[0095] Furthermore, for example, the system may further include an information processing device that includes at least one of a receiving unit, an image generating unit, and a presentation unit, and is connected to the observation device and the VR device via a network.

[0096] According to this, a video display system can be realized by the observation device, the VR device, and the information processing device.

[0097] Furthermore, for example, the information processing device may have a receiving unit, an image generating unit, a presentation unit, as well as a movement state calculation unit and a processing device transmitting unit, wherein the receiving unit receives wide-viewing-angle image and data from the observation device, the movement state calculation unit calculates the movement state of the observation device based on the wide-viewing-angle image and data, the presentation unit generates graphics indicating the calculated movement state, the image generating unit superimposes the graphics on a portion of the wide-viewing-angle image based on the data and the movement state, and the processing device transmitting unit transmits the wide-viewing-angle image together with other information.

[0098] According to this, a video display system can be realized by the observation device, VR device, and information processing device configured as described above.

[0099] Furthermore, for example, the information processing device may have a receiving unit and a presentation unit, as well as a movement state calculation unit, a metadata composition unit, and a processing device transmission unit, where the receiving unit receives wide-field-of-view image and data from the observation device, the movement state calculation unit calculates the movement state of the observation device based on the wide-field-of-view image and data, the presentation unit generates graphics based on the movement state calculated by the movement state calculation unit, the metadata composition unit generates metadata including the graphics, data, and flight state, and the processing device transmission unit transmits information including the wide-field-of-view image and metadata.

[0100] According to this, a video display system can be realized by the observation device, VR device, and information processing device configured as described above.

[0101] Furthermore, for example, the information processing device may have a first receiving unit that receives wide-field-of-view image and data from the observation system, a second receiving unit that receives data regarding the orientation of the VR device, a movement state calculation unit that calculates the movement state of the observation device from the wide-field-of-view image and data, a difference calculation unit that calculates the relative movement direction, which is the movement direction of the image capture unit relative to the orientation of the VR device, based on the difference between the orientation of the VR device and movement information regarding the movement of the image capture unit, a presentation unit that generates and outputs graphics showing the calculated difference and the movement state of the observation device, an image synthesis unit that synthesizes graphics onto the wide-field-of-view image based on the difference and the flight state, and a transmission unit that transmits the image and other information synthesized by the image synthesis unit.

[0102] According to this, a video display system can be realized by the observation device, VR device, and information processing device configured as described above.

[0103] Also, for example, the information processing device may be provided on a cloud connected to a wide area network, and connected to the observation device and the VR device via the wide area network.

[0104] According to this, a video display system can be realized by an observation device, a VR device, and an information processing device that is connected to the observation device and the VR device via a wide area network and is provided on the cloud.

[0105] In addition, an observation device according to one aspect of the present disclosure is an observation device used in an image display system for displaying an image on a display device, and includes a shooting unit that captures an image as a wide-viewing-angle image, a data acquisition unit that acquires data related to the movement speed of the shooting unit, a metadata acquisition unit that acquires metadata based on the acquired data, the metadata including the movement state of the shooting unit, and a transmission unit that transmits the captured wide-viewing-angle image together with the metadata.

[0106] When used in the above-described video display system, such an observation device can achieve the same effects as the above-described video display system.

[0107] Furthermore, an information processing method according to one aspect of the present disclosure is an information processing method for displaying a display image on a display device, which receives data relating to the movement speed of a camera unit that moves together with a moving object and captures the image, and generates and outputs graphics that indicate the movement state of the camera unit based on the movement speed of the camera unit in the received data, and which are superimposed on the captured image to present the movement state of the camera unit to a user of the display device.

[0108] Such an information processing method can achieve the same effects as the video display system described above.

[0109] Furthermore, a program according to one aspect of the present disclosure is a program for causing a computer to execute the information processing method described above.

[0110] Such a program can be used on a computer to achieve the same effects as the video display system described above.

[0111] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0112] The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, component placement positions, connection forms, steps, and step sequences shown in the following embodiments are merely examples and are not intended to limit the scope of the claims. Furthermore, among the components in the following embodiments, components that are not described in the independent claims are described as optional components.

[0113] It should be noted that the drawings are not necessarily strict illustrations, and the same reference numerals are used to designate substantially the same components in the drawings, and redundant explanations will be omitted or simplified.

[0114] Furthermore, in this specification, terms indicating the relationship between elements, such as parallelism, terms indicating the shape of elements, such as rectangle, as well as numerical values ​​and numerical ranges, are not expressions that only express a strict meaning, but are expressions that also include a substantially equivalent range, for example, differences such as an error of a few percent.

[0115] (Embodiment) [composition] First, an overview of the video display system according to the embodiment will be described with reference to Fig. 14 and Fig. 15. Fig. 14 is a diagram showing a schematic configuration of the video display system according to the embodiment. Fig. 15 is a diagram showing an example of a video displayed in the video display system according to the embodiment.

[0116] As shown in FIG. 14, the video display system 500 of this embodiment is realized by an observation device 300, a server device 200 connected via a network 150, and a display device 100 connected via the network 150.

[0117] Observation device 300 is a so-called omnidirectional camera capable of capturing 360-degree video. Observation device 300 may be, for example, handheld camera device 300a, or observation device 300b, mounted on a tripod or similar. Handheld camera device 300a allows for easy shooting while moving around. However, because observation device 300 is intended to be mounted on a moving object, it is desirable for the observation device to be configured appropriately for the moving object. Examples of such moving objects include aircraft, ships, and vehicles. In particular, moving objects with speeds that are difficult for humans to predict are likely to induce VR sickness. Therefore, moving objects with speeds greater than a human's walking speed or running speed—or, more precisely, objects with accelerations that indicate the degree of instantaneous speed increase—are intended. Hereinafter, observation device 300a and observation device 300b will be referred to as observation device 300 without any distinction between them. The observation device 300 has optical elements such as a fisheye lens and can capture a wide field of view, e.g., 180 degrees, with a single sensor array. Using multiple combinations of optical elements and sensor arrays arranged to complement each other's different wide-field-of-view areas, it can capture a 360-degree wide-field-of-view image. Images captured by the multiple sensor arrays are stitched together by identifying corresponding elements. This results in a single image that can be converted between a plane, such as a regular cylindrical image, and a sphere. By continuously generating such images in the time domain, a video (moving image) that changes over time is generated. The interior of a spherical image is also referred to as a 3D video space.

[0118] In addition, in this embodiment, two 3D video spaces are generated with a difference corresponding to human parallax. These two 3D video spaces may be generated from one 3D video space by simulation or the like, or may be generated by two cameras with a difference in parallax. In this embodiment, VR video can be displayed from within this 3D video space, allowing the user to view any direction in the 3D video space.

[0119] Network 150 is a communication network for connecting observation device 300, server device 200, and display device 100 so that they can communicate with one another. Here, a communication network such as the Internet is used as network 150, but is not limited to this. Furthermore, the connection between observation device 300 and network 150, the connection between server device 200 and network 150, and the connection between display device 100 and network 150 may each be made by wireless communication or by wired communication.

[0120] The server device 200 is a device for performing information processing and the like, and is realized using, for example, a processor and a memory. The server device 200 may be realized by an edge computer or a cloud computer. Furthermore, one server device 200 may be provided for one video display system 500, or one server device 200 may be provided for multiple video display systems 500. In other words, the server device 200 may perform various processes in multiple video display systems 500 in parallel. The server device 200 is not an essential component of the video display system 500.

[0121] For example, by allocating each functional unit of the server device 200 (described later) to the observation device 300 and the display device 100, respectively, a video display system including only the observation device 300 and the display device 100 can be realized. In particular, if the display device 100 is implemented as an information processing terminal such as a smartphone that also has a display panel, the functional units of the server device 200 can be easily implemented using the processor of the information processing terminal. Alternatively, by having the functions of the observation device 300 and the display device 100 in the server device 200, it is possible to reduce some of the functions of the observation device 300 or the display device 100 and repurpose an existing observation device or display device. In other words, by consolidating various functions in the server device 200, a video display system can be easily realized. The functional units of the server device 200 will be described later using FIG. 16 and other figures.

[0122] The display device 100 is a glasses-type HMD that supports two separate lens barrels by engaging temples extending from the left and right earlobes with the earlobes, thereby holding the two lens barrels at positions corresponding to the user's right and left eyes. Each lens barrel of the display device 100 has a built-in display panel, which projects images with parallax misalignment toward the user's left and right eyes, as shown in FIG. 15, for example. In FIG. 15, (L) shows one frame of an image for the left eye, and (R) shows the same frame of an image for the right eye. Note that the display device 100 does not have to be a device dedicated to displaying images. The display device of the present disclosure can also be realized using a display panel included in a smartphone, tablet device, PC, or the like.

[0123] Hereinafter, a more detailed configuration of the video display system 500 according to the present embodiment will be described with reference to Fig. 16. Fig. 16 is a block diagram showing the functional configuration of the video display system according to the embodiment. As shown in Fig. 16 and as described in Fig. 14, the video display system 500 includes a display device 100, a server device 200, and an observation device 300.

[0124] The display device 100 includes a display unit 101 and an orientation estimation unit 102. The display unit 101 is a functional unit that outputs an optical signal corresponding to image information using a backlight, a liquid crystal panel, an organic EL display, a micro LED, or the like. The display unit 101 controls the output optical signal so that an image is formed on the retina of the user's eye via optical elements such as a lens and an optical panel. As a result, the user can visually recognize the image from the image formed on the retina. The display unit 101 outputs the above-mentioned images continuously in the time domain, thereby allowing the user to visually recognize continuous images, i.e., a video. In this way, the display unit 101 displays a video to the user of the display device 100.

[0125] The orientation estimation unit 102 is a functional unit for estimating the orientation of the display device 100. The orientation estimation unit 102 is realized by various sensors, such as an acceleration sensor and a gyro sensor, built into appropriate positions of the display device 100. The orientation estimation unit 102 estimates the orientation of the display device 100 by estimating in what direction and by what angle the attitude of the display device 100 has changed with respect to a reference direction that is preset in the display device 100. As described above, the display device 100 is supported by the user's head (the auricle and the bridge of the nose) and therefore moves together with the user's head.

[0126] Then, by estimating the orientation of the display device 100, a field of view portion corresponding to the orientation can be cut out from the wide-viewing-angle image and displayed. In other words, based on the orientation of the display device 100 estimated by the orientation estimation unit 102, the direction in which the user's head is facing can be regarded as the field of view area that the user wants to see in the 3D image space, and the field of view area can be displayed. Note that the estimated orientation of the display device 100 here is the direction along the normal direction of the display panel of the display device 100. Since the display panel is disposed so as to face the user's eyes, the user's eyes are usually located in the normal direction of the display panel. Therefore, the orientation of the display device 100 coincides with the direction connecting the user's eyes and the display panel.

[0127] However, there are cases where the direction of the display device 100 and the direction of the user's gaze are misaligned due to the user's eye movement. In this case, if the display device 100 is equipped with a sensor (eye tracker) that detects the user's gaze, the detected user's gaze may be taken as the direction of the display device 100. In other words, the eye tracker is another example of a direction estimation unit.

[0128] In addition to the above, the display device 100 is equipped with a power supply, various input switches, a circuit for driving the display panel, wired and wireless communication modules for input and output, audio signal processing circuits such as signal converters and amplifiers, and a microphone and speaker for audio input and output.

[0129] The server device 200 has a receiving unit 201, a presenting unit 203, and an image generating unit 204. The receiving unit 201 is a processing unit that receives (acquires) various signals from an observation device 300 (described later). The receiving unit 201 receives wide-viewing-angle images captured by the observation device 300. The receiving unit 201 also receives metadata acquired by the observation device 300. Furthermore, the receiving unit 201 receives information related to the orientation of the display device 100 estimated by the display device 100.

[0130] The presentation unit 203 is a processing unit that presents the movement speed of the image capturing unit 300 to the user of the display device 100. Here, an example will be described in which the presentation unit 203 causes the image generating unit 204 to make the above presentation by including content indicating the movement speed in the display image generated by the image generating unit 204, but the presentation of the movement speed is not limited to the example in which it is included in the display image. For example, the movement speed may be presented as sound corresponding to the movement speed in a 3D sound field, or may be presented by vibrating a device such as a vibration device held by the user in both hands in a manner corresponding to the movement speed. Detailed operations of the presentation unit 203 will be described later.

[0131] The image generation unit 204 cuts out a portion of the image corresponding to the field of view portion according to the orientation of the display device 100 estimated by the orientation estimation unit 102 from the received wide-viewing-angle image, and further generates a display image including content indicating the moving direction in addition to the moving speed of the image capture unit 300, if necessary. The detailed operation of the image generation unit 204 will be described later together with the detailed operation of the presentation unit 203. The server device 200 also has a communication module for transmitting the generated display image to the display device 100.

[0132] The observation device 300 comprises an image capturing unit 301, an input interface 302, a position detection unit 303, a data acquisition unit 304, a metadata acquisition unit 305, and a transmission unit 306. The image capturing unit 301 is a functional unit related to capturing images and is configured integrally with other functional components of the observation device 300. Therefore, moving the image capturing unit 301 means moving the image capturing unit 301 along with the observation device 300. However, the observation device 300 is mounted on a moving object. Therefore, the image capturing unit 301 captures images while moving along with the movement of the moving object. The image capturing unit 301 may be separated from the other functional components of the observation device 300 by wired or wireless communication. In this case, the image capturing unit 301 can also be moved independently. In this example, it is sufficient that at least the image capturing unit 301 is mounted on the moving object. The image capturing unit 301 includes optical elements, a sensor array, an image processing circuit, etc. The imaging unit 301 outputs the luminance value of light received by each pixel on a sensor array via an optical element as 2D luminance value data. The image processing circuit performs post-processing such as noise removal from the luminance value data, as well as processing such as stitching to generate a 3D image space from the 2D image data.

[0133] The input interface 302 is a functional unit used when an operator operating the observation device 300 makes an input. For example, the input interface 302 may be connected to a control device for controlling the movement of a moving object equipped with the image capturing unit 301, and configured to acquire an operation history of the control device. In this way, it becomes possible to estimate the moving speed of the moving object from the operation history. By simply moving the image capturing unit 301, the operator can input the moving speed of the image capturing unit 301, which moves in accordance with the movement, into the system. Note that the input interface 302 is not an essential component. This embodiment can be realized if only either the input interface 302 or the position detection unit 303 described below is provided.

[0134] The position detection unit 303 is a sensor that detects the moving direction and moving speed of the imaging unit 301, i.e., the observation device 300. The position detection unit 303 is realized by a distance measurement sensor such as LiDAR, an odometer, etc., and can detect the current position of the observation device 300. The position detection unit 303 can detect the moving direction and moving speed of the observation device 300 by detecting this current position over time.

[0135] The data acquisition unit 304 is a functional unit that acquires data including information about the moving direction and moving speed of the imaging unit 301, i.e., the moving speed of the observation device 300, from the input interface 302, the position detection unit 303, etc. The data acquisition unit 304 is connected to at least one of the input interface 302 and the position detection unit 303, and acquires the moving direction and moving speed of the observation device 300 as physical quantities from these functional units.

[0136] The metadata acquisition unit 305 is a functional unit that acquires metadata by converting data related to the direction and speed of movement of the observation device 300 acquired by the data acquisition unit 304 into metadata to be added to the captured video data. The acquired metadata may include various data used within the video display system 500, in addition to data related to the direction and speed of movement of the observation device 300. In other words, the metadata acquisition unit 305 is an example of a metadata configuration unit that configures metadata that allows multiple pieces of data to be read from a single piece of information by combining multiple pieces of data into one.

[0137] The transmitting unit 306 is a communication module that transmits the captured video (wide-viewing-angle video) and the acquired metadata. The transmitting unit 306 communicates with the receiving unit 201 of the server device 200 to transmit the captured video and the acquired metadata, which are then received by the receiving unit 201. As a result, information about the moving direction and moving speed of the capturing unit 301, which is included in the metadata, is acquired.

[0138] Fig. 17 is a more detailed block diagram showing the functional configuration of an observation device according to an embodiment. Fig. 18 is a more detailed block diagram showing the functional configuration of a display device according to an embodiment. Figs. 17 and 18 show in more detail the peripheral functional configurations of the observation device 300 and the display device 100. Some of the functions shown in these figures may be realized by the configuration of the server device 200.

[0139] The data input means 51 corresponds to the input interface 302 and the position input unit 303, and inputs the position and direction of the object of gaze using a switch, tablet, smartphone, etc. that is physically operated by the operator or guide of the observation device 300.

[0140] The data input means 51 may obtain cue information from video obtained from the VR video processing means 67 or audio information obtained from the audio input means 71. The audio input means 71 is another example of an input interface. The VR video processing means 67 is connected to a VR shooting means 69 that corresponds to a shooting unit.

[0141] The data obtained from the data input means 51 is sent to the position / orientation / flight status detection means 53, where it is processed together with the position, orientation, and flight status of the observation device 300, and in some cases the status is stored and converted into suitable data which is sent as metadata to the multiplexing means 61, where it is multiplexed with video, audio, and graphics and then transmitted to the display device 100 by the communication means 55 via the server device 200. In addition to the above, the observation device 300 also includes a separation means 57, a VR video compression means 59, an audio compression means 63, an audio decoding means 65, and an audio output means 73.

[0142] In the display device 100, the communication means 39 receives communication information from the observation device 300, and the separation means 37 separates the metadata and sends it to the position / orientation / flight state determination means 31. In the position / orientation / flight state determination means 31, data is extracted from the metadata, subjected to predetermined processing, sent to the graphics generation means 33 to display the moving speed as a graphic, and then superimposed on the VR video by the VR display means 15 for display, or sent to the VR control means 21, where the VR video is appropriately processed by the VR display control means 23 together with the position and orientation state of the display device 100 and displayed by the VR display means 15, or the audio playback control means 25 generates a guide voice for guidance or appropriately processes the played voice.

[0143] As a specific example, when an arrow indicating the direction of a flying object is displayed and the image is controlled so that the arrow changes shape according to the flying object's speed, processing is performed to change the shape of the arrow to make it easier to imagine acceleration, such as making the arrow thicker at the tip. Furthermore, for example, as the speed of the moving object increases, control is performed such that the image of areas other than the direction of movement is masked, so that only the direction of movement is clear. In this way, graphics displayed on the display device 100 are generated according to the magnitude of the moving speed contained in the data. In addition to the above, the display device 100 includes position detection means 11, rotation detection means 13, audio playback means 17, audio input means 19, VR control means 21, audio compression means 27, audio decoding means 35, and multiplexing means 41. The components shown in FIG. 16 are realized by including one or more combinations of the components shown in FIGS. 17 and 18.

[0144] [Operation] Next, the operation of the video display system 500 configured as above will be described with reference to Fig. 19. Fig. 19 is a flowchart showing the operation of the video display system according to the embodiment.

[0145] When the operation of the video display system 500 starts, the image capturing unit 301 captures an image, and the input interface 302, the position detection unit 303, the data acquisition unit 304, and the metadata acquisition unit 305 operate to acquire metadata including data on the moving direction and moving speed of the image capturing unit 301. The metadata is received by the server device 200 together with the captured image via the transmission unit 306 and the reception unit 201 (S101).

[0146] Furthermore, the orientation estimation unit 102 of the display device 100 continuously estimates the orientation of the display device 100. The display device 100 transmits the orientation of the display device 100 estimated by the orientation estimation unit to the server device 200. As a result, the server device 200 receives the estimated orientation of the display device 100 (S102). Note that the order of steps S101 and S102 may be reversed. The server device 200 generates graphics corresponding to the movement direction and movement speed from data regarding the movement direction and movement speed of the image capture unit 301 (S103). From step S103, the server device 200 begins an operation for presenting the movement speed to the user of the display device 100. The presentation unit 203 generates graphics corresponding to the movement speed, and the image generation unit 204 cuts out a field of view portion corresponding to the orientation of the display device 100 from the wide-viewing-angle image (S104), and generates a display image by superimposing the graphics generated by the presentation unit 203 on the cut-out portion of the image (S105). Detailed examples of graphics will be described later, but the graphics are appropriately selected or combined and used, such as the arrows and masks mentioned above. When a mask is used as a graphic, a portion of the image cut out by the mask is obscured. As a result, the portion not obscured by the mask is emphasized, creating a visual effect. For example, the direction of movement can be expressed as if moving in a direction not masked, and the size of the masked area can be used to express the speed of movement.

[0147] Generating and overlaying a mask as graphics can significantly alter the displayed image, providing a variety of visual effects to the user. In particular, when a user views an image from an aircraft, a space extending vertically below the aircraft corresponds to the aircraft's flight altitude. Some users may feel fear when viewing information across such a high space. Therefore, by obscuring the vertically downward side with a mask resembling the aircraft's cockpit, this sense of fear can be alleviated. In this example, the remaining image from the area obscured by the mask can be a view from a window in the cockpit, which is also effective in enhancing the realism of the VR experience. Note that "obscuring" here also includes covering the obscured area with a semi-transparent image, allowing a portion of the obscured area to be seen through.

[0148] In this way, the user can understand the speed of movement and, in some cases, the direction of movement, so that even if the image subsequently moves, the user is less likely to become confused about the movement. In this way, the image display system 500 can display appropriate images on the display device 100 from the perspective of suppressing VR sickness and the like.

[0149] [Example] The following provides a more detailed explanation based on examples of the embodiments. In this example, the problem of the likelihood of VR sickness is addressed, and a more specific example for solving this problem is described. VR sickness is likely to occur when there is a discrepancy between information obtained through vision and hearing and acceleration information obtained through the vestibular sense (stimuli cannot be sensed in the case of uniform motion). Normal VR devices (such as HMDs) can reproduce information for the eyes and ears and can provide stimulation to the eyes and ears, but it is difficult to provide stimulation that causes acceleration to the vestibular organs without installing a large-scale motion platform, and this is limited to facility-based VR devices such as theme parks and VR game centers.

[0150] If VR sickness occurs during a VR experience, such as during a VR tour, the experience itself becomes painful and users stop using it. For this reason, it is important to avoid VR sickness.

[0151] Countermeasures for VR sickness from aerial footage require visual guidance similar to that described above.

[0152] The vestibular system only responds to acceleration, so when hovering and flying straight at a constant speed, the system experiences the same unstimulated state, but it detects acceleration when the body tilts due to acceleration, deceleration, or turning.

[0153] Therefore, VR sickness can be reduced by displaying guides that allow users to understand hovering, constant speed flight, acceleration / deceleration, turning, ascent, and descent. This measure is effective not only for aircraft, but also for ships, trains, buses, and other moving objects equipped with observation systems. It is also effective for images captured by virtual cameras installed on virtual aircraft, including virtual aircraft, in 3D spaces constructed using computer graphics.

[0154] FIG. 20 is a schematic diagram of an example of the configuration of a video display system according to an embodiment. As shown in FIG. 20, in this example, a guide display is changed according to the moving speed of a moving object contained in metadata. Metadata including the moving speed and the like can be automatically generated, for example, from the operation history of a drone (an example of a moving object) during operation. As a result, as shown in FIG. 20, when the moving speed is equal to 0 due to hovering or the like, a dot indicating this can be generated as a graphic and superimposed on the displayed image. FIGS. 21 and 22 are diagrams illustrating the correspondence between the moving state (flight state) of a moving object and the generated and superimposed graphics. As shown in FIG. 21, an example of an arrow display during accelerating flight is an arrow hatched to the right, an example of an arrow display during decelerating flight is an arrow hatched to the left, and an example of an arrow display during constant speed flight is an arrow that follows the direction of travel but is bent in the direction of rotation, for example, during a turn. Similarly, as shown in FIG. 22, etc., arrows can be displayed to indicate hovering, constant speed flight, acceleration / deceleration, ascent / descent, and turning of a moving object. It is also possible to distinguish by color. In this case, for example, red can be assigned to deceleration, green to acceleration, and blue to constant speed. Also, distinction can be made by adding a pattern such as hatching. For example, deceleration can be assigned to hatching that slopes upward to the left, acceleration to hatching that slopes upward to the right, and constant speed without hatching.

[0155] Furthermore, depending on the direction of the arrow, it is possible to assign ascent and descent, such as upwards for ascent and downwards for descent. This can also be done using 3D display. In other words, it is possible to assign a wide range of ascent to the top and a wide range of descent to the bottom. Furthermore, the shape of the arrow can represent acceleration, deceleration, stopping, hovering, etc. The form of such expression should be determined by the system so that it is as intuitive as possible, and by performing simple learning in advance, the effectiveness of preventing VR sickness can be improved.

[0156] 23 is a diagram showing an example of an operation flow of the video display system according to the embodiment. The operation of the embodiment of the present invention will be described.

[0157] In the position and orientation information detection step of the observation system according to the embodiment of the present invention, the position and orientation of the observation system are detected by a position and orientation information detection unit implemented by a program in the CPU from signals from the movement and position detection unit (S2330). If there is no change in position or orientation at this time, it is determined to be in a stopped state (before flight, after landing, hovering, etc.). If the position changes constantly, that is, if the system is moving at a constant speed, it is determined to be in a constant-speed movement state (low-speed horizontal flight, low-speed ascent, low-speed descent, etc.). If the position change increases over time, it is determined to be in an acceleration state, and if the position change decreases over time, it is determined to be in a deceleration state. If the position changes without a change in orientation, it is determined to be in a straight-line state (horizontal, ascent, descent, etc.), and if the orientation changes, it is determined to be in a turning state (up, down, left, right).

[0158] In the next metadata detection step, the detected position, direction, and flight status (stop, constant speed movement, acceleration, deceleration, turning) are converted into metadata (S2331), and in the next video / audio / metadata multiplexing step, the metadata is multiplexed with video, audio, and graphics by a multiplexing unit (S2327).In the next transmission step, the multiplexed information is sent (transmitted) to the VR system by a wireless communication element (S2328).

[0159] Here, position and orientation detection may be input from operational information obtained during drone operation. In the case of already recorded video, or when drone operational information is unavailable, metadata must be extracted by analyzing the video. In this case, the video is analyzed by a position and orientation analysis unit implemented as a GPU program or dedicated circuit, and the analysis results are sent to the CPU's position and orientation detection unit via the system bus.

[0160] In addition to sending position and orientation information to the VR system as metadata, the position and orientation detection unit's graphics generation unit can generate graphics data, such as graphics data of an arrow indicating the direction of movement, from the information and send it to the VR system as graphics data.

[0161] In some cases, the flight status is not determined in the position and direction information detection step. In this case, the metadata converted in the metadata detection step does not contain parameters indicating the flight status, and the flight status of the observation system is determined by the position and direction determination unit of the VR system, which will be described later.

[0162] In the VR system, in a receiving step, a computer / smartphone receives metadata sent from the observation system via a communication element (S2301). In a video / audio / metadata separation step, the CPU's separation unit separates the metadata (S2302) and sends it to the CPU's position / orientation determination unit and to the GPU's position / orientation determination unit via the memory bus. In a metadata analysis step, the CPU's position / orientation determination unit analyzes the metadata (S2306) and sends the obtained position / orientation information and flight status to the CPU's audio playback control unit and to the GPU's VR control unit or graphics generation unit via the memory bus. Alternatively, the GPU's position / orientation determination unit analyzes the metadata and extracts the flight status, which is then sent to the GPU's VR control unit or graphics generation unit.

[0163] In the video extraction step based on the position and orientation of the VR system and metadata, audio, video, and graphics processing (e.g., steps S2304 and S2309) is performed by the audio playback control unit, VR control unit, and graphics generation unit, respectively, based on the position and orientation information sent from the observation system.

[0164] When position and orientation information is sent as graphics information, the graphics generation unit generates graphics, and the VR display control unit appropriately changes the direction and superimposes it on the VR image. When graphics are generated on the VR system side, it may mask the image other than the front of the movement direction with semi-transparent graphics, or display an arrow indicating the movement direction, or display a map showing the movement.

[0165] The above processing may be performed by the observation system or a computer system such as a cloud system located between the observation system and the VR system.

[0166] Also, for steps not described above, the description of similar steps in Fig. 13 will be referred to and will not be repeated here. Specifically, step S2303 corresponds to step S1303, step S2305 corresponds to step S1305, step S2307 corresponds to step S1307, step S2308 corresponds to step S1308, step S2310 corresponds to step S1310, step S2321 corresponds to step S1321, step S2322 corresponds to step S1322, step S2323 corresponds to step S1323, step S2324 corresponds to step S1324, step S2325 corresponds to step S1325, and step S2326 corresponds to step S1326.

[0167] Fig. 24 is a schematic diagram of a configuration example of a video display system according to another example of the embodiment. Figs. 25 to 30 are diagrams showing examples of display images generated by the configuration of Fig. 24. Here, a configuration is shown that includes measures to address symptoms such as acrophobia in addition to measures to counter VR sickness caused by aerial images. In this case, the sense of fear is reduced by obscuring the image vertically below using graphic synthesis, etc., to make it appear as if the user is sitting in the passenger seat of a helicopter.

[0168] There are also methods that display guides to show flight conditions such as hovering, constant speed flight, acceleration / deceleration, and turning, and methods that provide guidance by tilting the cockpit itself or the outside scenery (i.e., the 3D image space itself) only when turning.

[0169] Fig. 31 is a schematic diagram of a configuration example of a video display system according to yet another example of the embodiment, and Fig. 32 is a diagram showing an example of a display video generated by the configuration of Fig. 31.

[0170] In this example, in addition to tilting the cockpit itself and tilting the outside scenery only when graphics are displayed or the aircraft is turning, the outside is visible from all windows during normal flight, and when accelerating, decelerating, or turning, all windows except the front are made semi-transparent, guiding the viewer's gaze to focus on the front, thereby reducing VR sickness. In addition to semi-transparency, image processing such as lowering the resolution and frame rate for windows other than the front may also be used. Also, by adding a driver and adding movements when changing direction, the direction can be detected in advance, leading to a reduction in VR sickness.

[0171] 33 is a diagram illustrating an example of the functional configuration of a video display system according to an embodiment of the present invention. The configuration of an embodiment of the present invention will be described.

[0172] Graphics such as arrows indicating the flight status (movement status) of the observation system (observation device) 3351 are sent to the VR system 3301 in advance when the entire system is initialized and stored in the graphics storage means (graphics storage unit) of the VR system 3301, or are stored in the graphics storage means of the VR system 3301 in the initial state.

[0173] In some cases, graphics data may be sent from the observation system 3351 as metadata or sent as VR video and converted into graphics and stored by the VR system 3301, but this will not be explained here.

[0174] The position, direction or flight state of the observation system 3351 is detected by a position, direction and flight state detection means (metadata conversion unit) 3352 and sent as metadata to the VR system via a communication means (transmission unit) 3354.

[0175] In the VR system (display device) 3301, metadata from the observation system 3351 is sent from the separation means 3315 via the communication means (receiving unit) 3316 to the position, orientation, and flight state determination means (judgment unit) 3311, where the position, orientation, and movement or flight state of the observation system 3351 are extracted and sent to the graphics generation means (graphics generation means) 3312.

[0176] Next, the graphics generation means 3312 retrieves one of the graphics from the graphics storage means (graphics storage unit) according to the position, orientation, movement and flight state of the observation system 3351 from the position / orientation / flight state determination means 3311, and sends it to the VR display control means (display control unit) 3308.

[0177] The VR display control means 3308 displays the spherical image sent from the observation system 3351 and the graphics from the graphics generation means 3312 on the VR display means (display unit) 3304 as a VR image that has been appropriately cut out based on the orientation of the VR system 3301 and the position and orientation of the observation system 3351 sent from the observation system 3351 as metadata.

[0178] The VR display means 3304 displays graphics that are appropriately synthesized according to the position, orientation, and movement of the observation system 3351.

[0179] In this example, the position of the graphics and the position of the VR image can also be changed by the position and orientation of the VR system 3301, the position and orientation of the observation system 3351, the speed of movement, etc., which increases the degree of freedom, for example, to make the VR image more realistic by tilting the graphics relative to the image when turning, to reduce VR sickness by hiding or blurring part of the VR image with graphics, to display different graphics on multiple VR systems, and to change the graphics display for users who are resistant to VR sickness and users who are not.

[0180] If the observation system 3351 generates graphics and sends them to all VR systems as metadata, there is no need for the VR system 3301 to store the graphics, which has the advantage of making it easier to implement the VR system 3301, but it cannot be optimized for each VR system 3301.

[0181] Furthermore, if the observation system 3351 superimposes graphics onto the VR image and sends it, the VR system 3301 does not need a graphics generation means 3312, making it even easier to implement, but it is not possible to change the graphics according to the state of the VR system 3301, and the graphics only reflect the state of the observation system.

[0182] Note that for the configuration not described above, the description of the similar configuration in FIG. 7 will be referred to and the description here will be omitted. Specifically, the position detection means 3302 corresponds to the position detection means 702, the rotation detection means 3303 corresponds to the rotation detection means 703, the VR display means 3304 corresponds to the VR display means 704, the audio reproduction means 3305 corresponds to the audio reproduction means 705, the audio input means 3306 corresponds to the audio input means 706, the VR control means 3307 corresponds to the VR control means 707, the VR display control means 3308 corresponds to the VR display control means 708, the audio reproduction control means 3309 corresponds to the audio reproduction control means 709, the VR video decoding means 3310 corresponds to the VR video decoding means 710, the graphics generation means 3312 corresponds to the graphics generation means 712, the audio decoding means 3313 corresponds to the audio decoding means 713, the audio compression means 3314 corresponds to the audio compression means 714, and the separation means 3315 corresponds to the separation means 715. 715, communication means 3316 corresponds to communication means 716, multiplexing means 3317 corresponds to multiplexing means 717, communication means 3354 corresponds to communication means 754, separation means 3355 corresponds to separation means 755, VR video compression means 3356 corresponds to VR video compression means 756, multiplexing means 3357 corresponds to multiplexing means 757, VR video processing means 3358 corresponds to VR video processing means 758, graphics generation means 3359 corresponds to graphics generation means 759, audio compression means 3360 corresponds to audio compression means 760, audio decoding means 3361 corresponds to audio decoding means 761, VR shooting means 3362 corresponds to VR shooting means 762, audio input means 3363 corresponds to audio input means 763, and audio output means 3364 corresponds to audio output means 764.

[0183] 34 is a diagram showing an example of an operation flow for compositing graphics according to an embodiment. The configuration and operation for solving the problems of the present invention will be described.

[0184] Graphics corresponding to moving objects to be simulated by the observation system, such as helicopters, airplanes, ships, and buses, are prepared and sent to the VR system in advance, and stored in the graphics storage means of the VR system (S3401).

[0185] Next, the graphics data is sent as metadata, or sent as VR video and converted into graphics in the VR system and stored (S3402).

[0186] Next, the position and direction of the observation system are detected by the position and direction detection means (S3403) and sent to the VR system as metadata (S3404).

[0187] In the VR system, the metadata from the observation system is sent from the separation means via the communication means to the position and orientation determination means, where the position, orientation and movement of the observation system are extracted and sent to the graphics generation means.

[0188] Next, the graphics generation means retrieves the graphics sent from the observation system from the graphics storage means, and in some cases, although not shown here, corrects the graphics according to the orientation of the VR system and sends them to the VR display control means.

[0189] The VR display control means displays the spherical image sent from the observation system and the graphics from the graphics generation means, and displays the VR image appropriately cut out based on the orientation of the VR system and the position and orientation of the observation system sent as metadata from the observation system on the VR display means (S3405).

[0190] The VR display means displays graphics that are appropriately synthesized according to the position, orientation, and movement of the observation system.

[0191] In this example, the position of the graphics and the position of the VR image can also be changed by the position and orientation of the VR system, the position and orientation of the observation system, the speed of movement, etc., which increases the degree of freedom, for example, to make the VR image more realistic by tilting the graphics relative to the image when turning, to reduce VR sickness by hiding or blurring part of the VR image with graphics, to display different graphics on multiple VR systems, and to change the graphics display for users who are resistant to VR sickness and users who are not.

[0192] Fig. 35 is a diagram showing another example of the functional configuration of a video display system according to an embodiment. The configuration of an embodiment of the present invention will be described. Fig. 36 is a diagram showing an example of an operation flow for compositing graphics according to an embodiment. The configuration and operation for solving the problems of the present invention will be described.

[0193] In this example, there is no need to send metadata from the observation system to the VR system, so the data input means is omitted.

[0194] Graphics corresponding to moving objects to be simulated by the observation system, such as helicopters, airplanes, ships, buses, etc., are prepared in advance (S3401), stored in the graphics storage means 3366 (S3402), and sent to the VR processing means.

[0195] Next, the position and direction of the observation system are detected by the position and direction detection means 3365 (S3403) and sent to the VR image processing means (S3404).

[0196] The VR processing means changes the received graphics data according to the received position and direction, and synthesizes it with the spherical image from the VR shooting means.

[0197] Next, the spherical image synthesized with the graphics data is sent to the VR system via a VR image compression means, multiplexing means, and communication means.

[0198] In the VR system, the spherical image from the observation system is transmitted from the communication means to the separation means, VR image decoding means, and VR display control means, and then displayed on the VR display means, as usual (S3405).

[0199] The VR display means displays graphics that are appropriately synthesized according to the position, orientation, and movement of the observation system.

[0200] In this example, the positions of the VR video and graphics are fixed, and the position of the clip from the spherical video changes depending on the position and orientation of the VR system.

[0201] Note that for the configuration not described above, the description of the similar configuration in FIG. 7 will be referred to and the description here will be omitted. Specifically, the position detection means 3302 corresponds to the position detection means 702, the rotation detection means 3303 corresponds to the rotation detection means 703, the VR display means 3304 corresponds to the VR display means 704, the audio reproduction means 3305 corresponds to the audio reproduction means 705, the audio input means 3306 corresponds to the audio input means 706, the VR control means 3307 corresponds to the VR control means 707, the VR display control means 3308 corresponds to the VR display control means 708, the audio reproduction control means 3309 corresponds to the audio reproduction control means 709, the VR video decoding means 3310 corresponds to the VR video decoding means 710, the graphics generation means 3312 corresponds to the graphics generation means 712, the audio decoding means 3313 corresponds to the audio decoding means 713, the audio compression means 3314 corresponds to the audio compression means 714, and the separation means 3315 corresponds to the separation means 715. 715, communication means 3316 corresponds to communication means 716, multiplexing means 3317 corresponds to multiplexing means 717, communication means 3354 corresponds to communication means 754, separation means 3355 corresponds to separation means 755, VR video compression means 3356 corresponds to VR video compression means 756, multiplexing means 3357 corresponds to multiplexing means 757, VR video processing means 3358 corresponds to VR video processing means 758, graphics generation means 3359 corresponds to graphics generation means 759, audio compression means 3360 corresponds to audio compression means 760, audio decoding means 3361 corresponds to audio decoding means 761, VR shooting means 3362 corresponds to VR shooting means 762, audio input means 3363 corresponds to audio input means 763, and audio output means 3364 corresponds to audio output means 764. 37 and 38 are diagrams showing an example of the configuration of metadata according to this embodiment. An example of the configuration of metadata according to this embodiment will be described.

[0202] The metadata type is a predetermined code or character string that indicates that it is metadata of the present invention. The version number is a number used when the metadata structure is changed, and is used like a major version and a minor version, such as 0.81 (0081) during the evaluation stage, 0.92 (0092) during demonstration experiments, and 1.0 (0100) at release, with the idea being that compatibility is guaranteed between versions of the same major version.

[0203] If the function code is 0, it indicates that the metadata information is invalid, and if it is anything else, it indicates the type of information in the metadata. For example, 0001 indicates a format that describes the reference position, camera, guide, and target positions, as well as the direction and speed of movement. 0002 indicates graphics data, 0003 indicates VR system information, 0011 indicates 0001 with cue data sent from the observation system, and 0021 indicates a moving target with cue data.

[0204] The reference position is the position data that serves as the reference for position data, and is determined in advance, including the units, for example, X (east-west distance), Y (north-south distance), Z (altitude distance) or longitude and altitude for the entire system. A reference position of 0 indicates that the position at the time of resetting the entire system is used as the reference. The camera position and guide position should also be determined in advance as absolute coordinates or relative coordinates from the reference position.

[0205] The direction and speed of movement indicate the movement status of the observation system or guide, and if there is cue data, it indicates how it will move from now on.

[0206] In the case of VR tourism, the number of targets indicates the destinations to be visited. If the number of targets is 0, it means there are no targets.

[0207] The verification code is a code for verifying whether the metadata data is correct during transmission, and may be, for example, a CRC.

[0208] Fig. 39 is a diagram for explaining an example of a configuration in which a video display system according to an embodiment is realized using the cloud. In the configuration shown in Fig. 39, the cloud has a function for controlling graphics, VR video, audio, and controller vibration according to the position, orientation, and flight state of the observation system and the position and orientation of the VR system, and providing appropriate information to the user of the VR system, so that the effects of the present invention can be achieved even with a simple VR system.

[0209] In the configuration shown in Figure 39, by having a position, direction, and flight state detection means 3940 in the cloud (computer system 3931), the position, direction, and flight state of an observation system 3961 on the cloud are read from metadata separated by a separation means 3942 from data sent from the observation system 3961, and accordingly, an image and arrow of a helicopter or the like are generated by a graphics generation means 3936. The position and direction of the observation system 3961 and the position and direction of the VR system 3901 are determined by a VR control means 3907, and the VR display control means 3937 appropriately combines the VR image with graphics or processes the VR image, and the audio playback control means 3939 changes the position of the audio or changes the content of the audio, thereby enabling display and audio output appropriate for the position and direction of the VR system 3901. In addition, although not shown here, it is possible to appropriately control the controller of the VR system 3901 and notify the user of the direction and position of the VR system by vibration or the like.

[0210] Note that the description of the components not described above will be omitted here by referring to the description of the components with similar names in Fig. 33. The VR system 3901 includes a position detection means 3902, a rotation detection means 3903, a VR display means 3904, an audio reproduction means 3905, an audio input means 3906, a VR control means 3907, a VR display control means 3908, an audio decoding means 3909, an audio compression means 3910, a VR video decoding means 3911, a separation means 3912, a multiplexing means 3913, and a communication means 3914, and the computer system 3931 includes a separation means 3932, a VR video compression means 3914, and a compression means 3933, multiplexing means 3934, communication means 3935, graphics generation means 3936, VR display control means 3937, VR image expansion means 3938, audio playback control means 3939, position, direction, flight state storage means 3940, communication means 3941, and separation means 3942, as well as data input means 3962, multiplexing means 3963, communication means 3964, separation means 3965, VR image compression means 3966, audio compression The VR image processing means 3967, the audio decoding means 3968, the VR image processing means 3969, the VR shooting means 3970, the audio input means 3971, and the audio output means 3972 are respectively configured as a position detection means 3302, a rotation detection means 3303, a VR display means 3304, an audio playback means 3305, an audio input means 3306, a VR control means 3307, a VR display control means 3308, an audio playback control means 3309, a VR image decoding means 3310, a graphics generation means 3312, an audio decoding means 3313, and a 313, audio compression means 3314, separation means 3315, communication means 3316, multiplexing means 3317, communication means 3354, separation means 3355, VR video compression means 3356, multiplexing means 3357, VR video processing means 3358, graphics generation means 3359, audio compression means 3360, audio decoding means 3361, VR shooting means 3362, audio input means 3363, and audio output means 3364 correspond to each other in a one-to-one, many-to-one, one-to-many, or many-to-many manner.

[0211] Note that the functions provided on the cloud are not limited to the configuration shown in Figure 39, and functions to be provided on the cloud can be selected so that the overall functions and operations are roughly similar depending on the configuration and functions of the connected observation system and VR system. For example, if the observation system does not detect the position and orientation of the observation system, but instead detects the position and orientation of the observation system on the cloud and sends it to the VR system as graphics superimposed on the image, there are limitations on changing the graphics depending on the position and orientation of the VR system, but no special functions are required for the VR system. Also, if the VR system is configured to have position and orientation control means and graphics generation means that modify graphics depending on the position and orientation of the VR system, it will also be possible to change the graphics depending on the position and orientation of the VR system.

[0212] FIG. 40 is a diagram illustrating an example of a configuration in which a video display system according to an embodiment is realized using a cloud. As shown in FIG. 40, the position, orientation, and flight state detection means of the observation system may be realized by a computer system, such as a cloud, located between the observation system and the VR system. In this case, metadata indicating direction is not sent from the observation system, or data entered by the operator is sent as metadata. For example, the position, orientation, and flight state detection means in the cloud detects the position, orientation, or movement of the observation system, guide, or target from the video, audio, or metadata sent from the observation system, and sends the detected information to the VR system as metadata. This allows the effects of this embodiment to be achieved even with existing 360-degree cameras.

[0213] Furthermore, the position and orientation determination means on the VR system side and the resulting control of VR images and audio may also be realized in a computer system located between the VR system and the observation system, such as a cloud. In this case, the same processing can be performed in one place, making it easy to simultaneously apply the same effect to multiple VR systems, and it is expected that the effects of the present invention can be applied to existing systems. However, in order to reflect the direction and position of the VR system, it is necessary for the VR system to send the position and direction of the VR system to the cloud side, and a processing unit corresponding to each VR system must be installed on the cloud side.

[0214] The configuration in Figure 40 is an example of a case where the position and direction of the VR system are not sent to the cloud side. In this case, it becomes difficult to display arrows or change audio according to the position and direction of the VR system, but it is possible for the VR display control means to perform processing such as changing the resolution of the VR image, masking, and changing the positioning of audio according to the output of the position / direction / flight state detection means.

[0215] Note that the description of the components not described above will be omitted here by referring to the description of the components with similar names in Fig. 33. The VR system 4001 includes a position detection means 4002, a rotation detection means 4003, a VR display means 4004, an audio reproduction means 4005, an audio input means 4006, a VR control means 4007, a VR display control means 4008, an audio decoding means 4009, an audio compression means 4010, a VR video decoding means 4011, a separation means 4012, a multiplexing means 4013, an audio reproduction control means 2017, and a communication means 4014, and the computer system 4031 includes The observation system 4061 includes a VR image compression means 4033, a multiplexing means 4034, a communication means 4035, a graphics generation means 4036, a VR display control means 4037, a VR image decompression means 4038, a position, direction, and flight state storage means 4040, a communication means 4041, and a separation means 4042, and a data input means 4062, a multiplexing means 4063, a communication means 4064, a separation means 4065, a VR image compression means 4066, an audio compression means 4067, and a data input means 4068. 7, the audio decoding means 4068, the VR video processing means 4069, the VR shooting means 4070, the audio input means 4071, and the audio output means 4072 are respectively a position detection means 3302, a rotation detection means 3303, a VR display means 3304, an audio playback means 3305, an audio input means 3306, a VR control means 3307, a VR display control means 3308, an audio playback control means 3309, a VR video decoding means 3310, a graphics generation means 3312, and an audio decoding means 331 3. The audio compression means 3314, the separation means 3315, the communication means 3316, the multiplexing means 3317, the communication means 3354, the separation means 3355, the VR video compression means 3356, the multiplexing means 3357, the VR video processing means 3358, the graphics generation means 3359, the audio compression means 3360, the audio decoding means 3361, the VR shooting means 3362, the audio input means 3363, and the audio output means 3364 correspond to each other in a one-to-one, many-to-one, one-to-many, or many-to-many manner.

[0216] Fig. 41 is a diagram for explaining an example configuration in which a video display system according to an embodiment is realized using a cloud. In the configuration of Fig. 40, it was difficult to display an arrow or change audio according to the position and direction of the VR system, but in the configuration shown in Fig. 41, by providing a position / direction determination means in the VR system, the position and direction of the observation system on the cloud are read from metadata separated by a separation means from data sent from the observation system, and accordingly, an image of a helicopter or the like and an arrow are generated by a graphics generation means, which are converted into metadata together with the position and direction information sent from the observation system by a metadata conversion means, multiplexed by a multiplexing means, and sent to the VR system.

[0217] In the VR system, graphics are generated from the metadata separated by the separation means, and the VR control means determines the position and orientation of the observation system and the position and direction of the VR system obtained from the position detection means and rotation detection means, and the VR display control means appropriately combines the VR video with the graphics, or processes the VR video, and the audio playback control means changes the position of the audio, changes the content of the audio, etc., making it possible to output display and audio appropriate for the position and direction of the VR system.In addition, although not shown here, it is possible to appropriately control the VR system controller and notify the user of the VR system of the direction and position by vibration, etc.

[0218] Note that the description of the components not described above will be omitted here, and reference will be made to the description of the components with similar names in Fig. 33. The VR system 4101 includes a position detection means 4102, a rotation detection means 4103, a VR display means 4104, an audio reproduction means 4105, an audio input means 4106, a VR control means 4107, a VR display control means 4108, an audio decoding means 4109, an audio compression means 4110, a VR video decoding means 4111, a separation means 4112, a multiplexing means 4113, a graphics generation means 4116, an audio reproduction control means 4117, and a communication means 4114. The computer system 4131 includes a multiplexing means 4134, a communication means 4135, a graphics generating means 4136, a VR display control means 4137, a position, direction, and flight state storage means 4140, a communication means 4141, and a separation means 4142, and the observation system 4161 includes a data input means 4162, a multiplexing means 4163, a communication means 4164, a separation means 4165, a VR image compression means 4166, an audio compression means 4167, an audio decoding means 4168, and a The VR image decoding means 3310, the VR image processing means 4168, the VR image processing means 4169, the VR shooting means 4170, the audio input means 4171, and the audio output means 4172 are respectively connected to a position detection means 3302, a rotation detection means 3303, a VR display means 3304, an audio playback means 3305, an audio input means 3306, a VR control means 3307, a VR display control means 3308, an audio playback control means 3309, a VR image decoding means 3310, a graphics generation means 3312, an audio decoding means 3313, an audio The audio compression means 3314, separation means 3315, communication means 3316, multiplexing means 3317, communication means 3354, separation means 3355, VR video compression means 3356, multiplexing means 3357, VR video processing means 3358, graphics generation means 3359, audio compression means 3360, audio decoding means 3361, VR shooting means 3362, audio input means 3363, and audio output means 3364 correspond to each other in a one-to-one, many-to-one, one-to-many, or many-to-many relationship.

[0219] The position and orientation information of the VR system detected by the position detection means and rotation detection means of the VR system is multiplexed with other information as metadata by the multiplexing means, and sent to a computer system on the cloud by the communication means. This function is almost always provided in general VR systems.

[0220] (Other embodiments) Although the embodiments and the like have been described above, the present disclosure is not limited to the above-described embodiments and the like.

[0221] Furthermore, although the components constituting the video display system have been exemplified in the above embodiments, the functions of the components provided in the video display system may be distributed in any manner among the multiple parts constituting the video display system.

[0222] In the above-described embodiments, each component may be realized by executing a software program suitable for that component, or by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0223] Furthermore, each component may be realized by hardware. For example, each component may be a circuit (or integrated circuit). These circuits may form a single circuit as a whole, or each may be a separate circuit. Furthermore, each of these circuits may be a general-purpose circuit or a dedicated circuit.

[0224] Furthermore, the general or specific aspects of the present disclosure may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0225] In addition, this disclosure also includes forms obtained by applying various modifications to the embodiments, etc. that a person skilled in the art would think of, or forms realized by arbitrarily combining the components and functions of the embodiments, etc. within the scope that does not deviate from the intent of this disclosure. [Industrial Applicability]

[0226] The present disclosure is useful in applications where an appropriate image is displayed on a display device. [Explanation of symbols]

[0227] 100 display device 101 Display section 102 Orientation detection unit 150 Network 200 Server device 201 Receiving unit 202 Difference calculation part 203 Presentation section 204 Image Generation Unit 300 Observation Equipment 300a, 300b Imaging device 301 Photography Department 302 input interface 303 Position detection unit 304 Data Acquisition Department 305 Metadata Acquisition Unit 306 Transmission Unit 500 Video Display System

Claims

1. A video display system for displaying a display video on a display device, comprising: a VR device having the display device that displays the display image; an observation device mounted on a moving body and moving, the observation device having: a photographing unit that photographs video; a data acquisition unit that acquires data relating to the moving speed of the photographing unit and the moving state of the moving body, as well as cue data that indicates how the moving body will move from now on; and a transmission unit that transmits the photographed video together with the data and the cue data; a receiving unit for receiving the video, the data, and the cue data; an image generating unit that generates the display image including the received image; a presentation unit that generates and outputs graphics that display a guide display using figures on the display image in a manner assigned to each of the movement states in the received data, the graphics being in accordance with the movement speed of the image capturing unit and how the moving object will move from now, from the received data and the queue data, and that causes the image generation unit to present the movement speed and how the moving object will move from now by superimposing the output graphics before the moving object starts moving. Video display system.

2. The graphics display an arrow indicating the movement state on the display image in a manner corresponding to the movement state. The video display system according to claim 1 .

3. The graphics display a mask, which is an image for covering at least a part of the displayed image other than the moving direction side of the moving object. The video display system according to claim 1 .

4. The moving body is an aircraft. The video display system according to claim 1 .

5. The graphics display a mask for obscuring the vertically downward area as seen from the aircraft.

5. The video display system according to claim 4.

6. The data acquisition unit acquires data regarding the moving speed of the image capture unit and the cue data from an operation history of an operation of a control device for controlling the movement of the moving body. The video display system according to claim 1 .

7. the movement state is information indicating a change in the movement speed and a change and continuation of the movement direction of the observation device; The moving speed is information including the status of changes in the speed of the observation device.

3. The video display system according to claim 2.

8. The photographing unit photographs a virtual image by photographing in a virtual image space constructed by computer graphics. The video display system according to claim 1 .

9. an information processing device that includes at least one of the receiving unit, the image generating unit, and the presentation unit, and is connected to the observation device and the VR device via a network; The video display system according to claim 1 .

10. the information processing device includes the receiving unit, the image generating unit, the presenting unit, a moving state calculating unit, and a processing device transmitting unit; the receiving unit receives the wide-viewing-angle image as the image, the data, and the cue data from the observation device; the movement state calculation unit calculates a movement state of the observation device based on the wide-viewing-angle image, the data, and the cue data; the presentation unit generates graphics showing the calculated movement state; the image generation unit superimposes the graphics on a part of the wide-viewing-angle image based on the data and the movement state; The processing device transmitting unit transmits the wide viewing angle image together with other information before the moving object starts moving.

10. The video display system according to claim 9.

11. the information processing device includes the receiving unit, the presenting unit, a moving state calculating unit, a metadata constructing unit, and a processing device transmitting unit; the receiving unit receives the wide-viewing-angle image as the image and the data from the observation device; the movement state calculation unit calculates a movement state of the observation device based on the wide-viewing-angle image, the data, and the cue data; the presentation unit generates graphics based on the movement state calculated by the movement state calculation unit; the metadata construction unit generates metadata including the graphics, the data, and the movement state; The processing device transmitting unit transmits information including the wide-viewing-angle image and the metadata before the moving object starts moving.

10. The video display system according to claim 9.

12. The information processing device includes: a first receiving unit that receives the wide-viewing-angle image as the image, the data, and the queue data from the observation device; a second receiving unit that receives data related to the orientation of the VR device; a movement state calculating unit that calculates the movement state of the observation device from the wide-viewing-angle image, the data, and the queue data; a difference calculating unit that calculates the relative movement direction, which is the movement direction of the image capturing unit relative to the orientation of the VR device, based on the difference between the orientation of the VR device and movement information related to the movement of the image capturing unit; a presentation unit that generates and outputs graphics that show the calculated difference and the movement state of the observation device; an image synthesizing unit that synthesizes the graphics with the wide-viewing-angle image based on the difference and the movement state; and a transmission unit that transmits the image synthesized by the image synthesizing unit and other information before the moving body starts moving.

10. The video display system according to claim 9.

13. The information processing device It is located on a cloud connected to a wide area network, connected to the observation device and the VR device via the wide area network; The video display system according to any one of claims 9 to 12.

14. An information processing method for displaying a display image on a display device of a VR device, comprising: receiving data relating to the moving speed of a camera unit that captures video and that moves with the moving object, data relating to the moving state of the moving object, and cue data indicating how the moving object will move from now on; The graphics are displayed on the display image as a guide using figures in a manner assigned to each moving state, based on the received data and the moving speed of the shooting unit in the queue data and how the moving object will move from now on. The graphics are generated and output before the moving object starts moving, by being superimposed on the captured image to present the moving state of the shooting unit to the user of the display device of the VR device. Information processing methods.

15. A method for causing a computer to execute the information processing method according to claim 14. program.

Images