Image display system and information processing method

Abstract

This video display system is provided with: an observation device including an imaging unit for imaging a video as a wide viewing angle video, a data acquiring unit for acquiring data relating to a movement direction of the imaging unit, a metadata configuring unit for acquiring metadata based on the acquired data, and a transmitting unit for transmitting the imaged wide viewing angle video together with the metadata; and a VR device including a receiving unit for receiving the wide viewing angle video and the metadata, an orientation estimating unit for estimating the orientation of a display device, a difference calculating unit for calculating a relative movement direction, which is a relative movement direction of the imaging unit, on the basis of a difference between the orientation of the display device and the movement direction of the imaging unit in the metadata, a presenting unit for presenting the calculated relative movement direction to a user of the display device, a video generating unit for generating a display video including a portion of the received wide viewing angle video, and the display device for displaying the display video.

Description

Technical Field

[0001] The present disclosure relates to a video display system, an information processing method, and a program.

Background Art

[0002] In recent years, development of a so-called head-mounted display, which is a head-mounted display device, has been actively carried out. For example, Patent Document 1 discloses a head-mounted display capable of presenting (that is, displaying) a video of content and a video of the outside world. In the head-mounted display disclosed in Patent Document 1, by adjusting the brightness of at least one of the video of content and the video of the outside world, the discomfort given to the user when switching between the video of content and the video of the outside world is reduced.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, as an application that makes use of the high level of immersion of a display device such as a head-mounted display, there is an application such as pseudo-experiencing an experience at a certain point by viewing a video from a remote location. At this time, it is required that an appropriate video be provided to the display device.

[0005] The present disclosure has been made in view of the above, and an object thereof is to provide a video display system and the like capable of displaying an appropriate video.

Means for Solving the Problems

[0006] In order to achieve the above object, one aspect of the video display system according to the present disclosure is a video display system for displaying a display video by a display device, including a photographing unit that photographs a video as a wide-angle video, a data acquisition unit that acquires data regarding the moving direction of the photographing unit, a metadata configuration unit that acquires metadata based on the acquired data, and a transmission unit that transmits the photographed wide-angle video together with the metadata. The video display system further includes a VR device including a reception unit that receives the wide-angle video and the metadata, an orientation estimation unit that estimates the orientation of the display device, a difference calculation unit that calculates a relative movement direction, which is the moving direction of the photographing unit relative to the orientation of the display device, based on the difference between the estimated orientation of the display device and the moving direction of the photographing unit on the metadata, a presentation unit that presents the calculated relative movement direction to a user of the display device, a video generation unit that generates the display video including a part of the video corresponding to a visual field portion according to the orientation of the display device estimated by the orientation estimation unit from the received wide-angle video, and the display device that displays the display video.

[0007] Also, one aspect of the information processing method according to the present disclosure is an information processing method for causing a display device to display a display video, including receiving metadata based on data regarding the moving direction of a photographing unit that photographs a video as a wide-angle video, and calculating and outputting a relative movement direction, which is the moving direction of the photographing unit relative to the orientation of the display device, based on the difference between the estimated orientation of the display device and the moving direction of the photographing unit on the metadata.

[0008] These general or specific aspects may be implemented by a system, a device, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM, or may be implemented by any combination of a system, a device, an integrated circuit, a computer program, and a recording medium.

Advantages of the Invention

[0009] According to the present disclosure, there is provided a video display system or the like capable of displaying an appropriate video.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Figure 31

Figure 32

Figure 33

Figure 34

Figure 35

Figure 36

Figure 37

Figure 38

Figure 39

Figure 40

Figure 41

Figure 42

Figure 43

Figure 44

Figure 45

Figure 46

Figure 47

Figure 48

DETAILED DESCRIPTION OF THE INVENTION

[0011] (Knowledge underlying the disclosure) In recent years, display devices have been developed that can be worn on the user's head to place a display unit in front of the eyes, enabling the user to visually recognize the seemingly displayed image on a large screen. Such a display device is called a head-mounted display (HMD) and has the characteristic that the image can be optically recognized as a large screen. In addition, in some HMDs, by displaying a video that generates a deviation in visual parallax corresponding to each of the user's right and left eyes, the user can feel the viewed video stereoscopically. And due to the improvement of communication quality in recent years, it is possible to view the video captured by an observation device placed at a remote location with a delay of about several milliseconds to several tens of milliseconds in almost real time, enabling an experience as if being on the spot without visiting the location. Utilizing this technology, virtual tourism experiences such as sightseeing tours, exhibition visits, inspections, factory tours, and visits to art museums, museums, zoos, and aquariums (hereinafter also referred to as pseudo-tourism or VR (Virtual Reality) tourism) have also come to be realized.

[0012] In such VR tourism, cameras that can capture 360-degree (full longitude) videos (so-called omnidirectional cameras) are used as observation devices. The video captured by the observation device is a wide-angle-of-view video, and the user side of the display device can display it by cutting out an arbitrary direction of the video. For example, if a function is installed that can detect the direction the user of the display device is facing, a part of the video corresponding to the user's orientation in the 3D video space can be cut out and displayed, so it is possible to provide a viewing experience that meets the needs of many users from a single camera video.

[0013] Here, when the user is watching a part of the video in an arbitrary direction and there is movement on the side of the observation device, there may be movement of the video that the user does not intend. For example, when there is movement of the observation device such that the 3D video space moves in the 12 o'clock direction, if the user is facing the 2 o'clock direction within this 3D video space, the video will suddenly move in the 10 o'clock direction as seen from the user. Such movement causes a sense of deviation between the virtual space and the real space, which becomes a factor for the user to feel discomfort. This phenomenon is also called so-called VR sickness and makes it difficult to watch videos for a long time.

[0014] Therefore, in the present disclosure, in order to suppress the occurrence of such VR sickness, an object is to provide a video display system capable of presenting to the user the direction in which the video moves when the video moves. In the present disclosure, it will be described assuming that a 360-degree wide viewing angle video is captured by the observation device. However, the wide viewing angle video may be a video captured in an arbitrary angle range such as, for example, 270 degrees or more, 180 degrees or more, or 150 degrees or more. Such a wide viewing angle video only needs to have a viewing angle wider than at least the viewing angle of the video displayed by the user on the display device side. Also, in the present disclosure, a video display system assuming the occurrence of video movement in the horizontal plane will be described, but it is also applicable to video movement occurring on an intersection plane intersecting the horizontal plane including a vertical component.

[0015] Hereinafter, a conventional video display system and the like will be described in more detail with reference to the drawings. FIG. 1 is a diagram for explaining a conventional example. As shown in FIG. 1, conventionally, a service called VR tourism (first-person experience) has been provided. In VR tourism, when the local VR space is appropriately reproduced, a tourism experience as if one were at that place is possible. Examples of services using 360° camera shooting include FirstAirlines (https: / / firstairlines.jp / index.html) and Travel Assistant (https: / / www.tokyotravelpartners.jp / kaigotabisuke-2 / ). Examples of services using 3D CG (computer graphics) include Google Earth VR and Boulevard (https: / / www.blvrd.com / ).

[0016] FIG. 2 is a diagram for explaining a conventional example. As shown in FIG. 2, in addition to VR tourism, there is also a service (also called a third-person experience) in which a video shot on-site is displayed on a display device such as a television and viewed from a third-person perspective. In the third-person experience, there is a service specialized for users guided by experts, and it has features such as being able to be monetized according to an individual's hobbies.

[0017] FIG. 3 is a diagram for explaining a conventional example. As shown in FIG. 3, when developing VR tourism, as a basic configuration, a VR system main body, a controller, a computer or a smartphone, a network, a cloud, an observation system, etc. are required. The main body of the VR system was conventionally only of the HMD type that was heavy and covered the face considerably, but with the small glasses-type VR glasses, it has become easier to use for a long time and is becoming more widely used. The main body of the VR system includes an All-in-One type that includes the functions necessary for the VR system main body and a tethered type that entrusts some functions to a computer or a smartphone. The controller is used to select a menu or move in the VR space. The computer or smartphone may be only for communication functions or may constitute a part of the VR system.

[0018] The network and the cloud may connect the observation system 315 and the VR system 311 and implement some functions of the observation system or the VR system on the computer system 314 on the cloud. For the observation system, a 360° camera with a wireless function or a 360° camera wirelessly or wiredly connected to a smartphone or a computer, a 180° camera, or a wide-angle camera is used. Through these devices and the like, the user 312 can visually recognize the guide and the buildings or scenery of the tourist destination within the VR space.

[0019] In the description, VR tourism using a 360° camera is taken as an example, but any VR glasses that allow participants to change their viewpoints, such as those using a 180° camera, may be used. Also, although an example of photographing and guiding the actual scenery may be described, instead of the actual scenery, a virtual camera may be used in a virtual space composed of computer graphics, and the guide may also enter the virtual space using VR glasses or the like and play the video within the virtual space to realize tourism. Therefore, the present invention can also be applied to such applications. As typical examples of the above, it is represented by areas or spaces where ordinary travelers cannot easily go, such as lunar travel.

[0020] FIG. 4 is a diagram for explaining a conventional example. In FIG. 4, for the case of 360° camera shooting, the schematic configurations of VR tourism services (without a guide: upper part (hereinafter referred to as Conventional Example 1), with a guide: middle part (hereinafter referred to as Conventional Example 2)) and a conventional example of a Zoom (registered trademark) tourism service which is an example of a three-person experience (lower part (hereinafter referred to as Conventional Example 3)) are shown. Hereinafter, in the present invention, voice or voice data, and voice information include not only conversations but also audio signals including music and, in some cases, ultrasonic waves outside the audible band. In the VR tourism service, on the observation system (tourist destination) side, pre-recorded video is sent, or the VR system side operates a 360° camera, a robot, or a drone so that the VR system side can view the VR video. As shown in the middle part, it is also possible that a guide or a camera operator exists on the observation system side and the VR video such as a 360° camera is enjoyed as VR on the VR system. Also, as shown in the lower part, in the three-person experience, a 2D video is sent from the observation system side in 2D by a multi-person remote conversation service with voice and video such as Zoom, and the video of the tourist destination can be viewed and enjoyed remotely.

[0021] FIG. 5 is a diagram for explaining a conventional example. The overall system configuration of Conventional Example 2 will be described. Conventional Example 1 is different from Conventional Example 2 in that pre-recorded VR video is used or the operation is performed from the VR system side, and the difference will also be described. The observation system of Conventional Example 2 is composed of a camera for VR shooting, for example, a 360° camera, and a communication device for sending the captured information to a remote location.

[0022] A 360° camera for VR shooting synthesizes (stitches) the images of multiple cameras shooting in different directions into one video, maps it onto a plane by, for example, the equirectangular projection (ERP) method, appropriately compresses it as an ERP image, and sends it from a communication device to a remote VR system together with audio data captured by a microphone. The 360° camera may be mounted on a robot, a drone, etc. The 360° camera or a robot, a drone, etc. on which it is mounted is operated by a photographer or a guide. In Conventional Example 1, there are also cases where it is operated on the VR system side or where pre-recorded images, etc. are received on the VR system side.

[0023] On the VR system side, contrary to the observation system, the received planar image (ERP image) is converted into a spherical image, and a part is cut out according to the orientation and position of the observer, etc., and displayed on a VR display device. In Conventional Example 3, since the received image is 2D, it is displayed as 2D, and mostly a 2D display device, such as a tablet, a smartphone, or a TV, is used. The above is the same for the case of receiving pre-recorded images in Conventional Example 1.

[0024] When operating on the VR system side, when the observation system operates in conjunction with the orientation and position on the VR system side, there are cases where the observation system operates by key operations such as a mouse, a tablet, a joystick, a keyboard, etc., or by selecting a menu or an icon on the screen. In such cases, appropriate control data is sent from the VR system side to the observation system, and it is necessary for the state of the observation system, that is, the orientation, position, etc., to be sent to the VR system.

[0025] FIG. 6 is a diagram for explaining a conventional example. Using the comparison between the 360° video and the normal video shown in FIG. 6, the resolution when viewing the 360° video in a VR system will be explained. When viewing a 4K video of 360° with a VR device having a 100-degree field of view (FOV), the resolution of the video cut out for VR display is only 1067×600 (about twice that of SD video). Since a VR system using a panel with a resolution of 2K×2K for one eye is displayed on a square panel and stretched further by a factor of 2 in the vertical direction, the resulting video has a very low resolution.

[0026] In the case of an 8K video, the VR display resolution is 2133×1200, and in terms of data volume, it is 1.23 times the area of Full HD (1920×1080). However, since it is stretched by a factor of 2 in the vertical direction, the resulting video is of about the Full HD level. For 11K shooting (10560×5940), the VR resolution is 2933×1650, which is comparable to a VR system.

[0027] In order to provide a VR tourism experience with high resolution and a high sense of presence, shooting at a minimum of 8K and preferably 11K is required. Shooting at 8K and 11K requires large equipment, a high video transfer rate, and a large capacity. Therefore, both shooting and distribution become expensive.

[0028] Therefore, it is essential to avoid VR sickness, make it easy to understand and use, and thereby enable many users to use it and reduce the unit usage cost per user. Also, the effective use of VR recording content becomes important for establishing a business.

[0029] FIG. 7 is a diagram for explaining a conventional example. The main functional configuration examples of Conventional Examples 1 and 2 will be described by function. The observation systems 751 of Conventional Examples 1 and 2 include VR imaging means 762 (VR imaging camera) for performing VR imaging, VR video processing means 758 for processing the video captured by the VR imaging means 762 into an image suitable for transmission, VR video compression means 756 for compressing the VR video processed by the VR video processing means 758 into a data rate and video signal format suitable for transmission, voice input means 763 consisting of a microphone for inputting guides and surrounding voices, voice compression means 760 for converting the voice signal input by the voice input means 763 into a data rate and voice signal format suitable for transmission, graphics generation means 759 for generating auxiliary information as graphics, multiplexing means 757 for converting the video signal, voice signal, and graphics information compressed by the VR video compression means 756, the graphics generation means 759, and the voice compression means 760 into a signal suitable for transmission, communication means 754 for sending the multiplexed communication observation signal to a plurality of VR systems 701 and receiving the communication voice signal from the plurality of VR systems 701, separation means 755 for extracting the compressed voice signal from the communication voice signal received by the communication means 754, voice decoding means 761 for extracting the voice signal from the compressed voice signal from the separation means 755, and voice output means 764 for outputting the voice signal decoded by the voice decoding means 761 as sound.

[0030] In this example, it is assumed that the VR video processing means 758, the VR video compression means 756, and the graphics generation means 759 are realized within the GPU, and the voice compression means 760, the multiplexing means 757, the separation means 755, and the voice decoding means 761 are realized within the CPU. However, it is not necessarily limited to this. In a simpler configuration, the CPU and the GPU may be realized as one processor, but their functional configurations and operations are the same.

[0031] The VR shooting means 762 is, for example, a 360° camera, but is composed of a plurality of cameras that shoot in different directions. In the VR video processing means, the outputs of the plurality of cameras are synthesized (stitched) into one video, and this is mapped onto a plane by, for example, the equirectangular projection (ERP) method and output as an ERP image.

[0032] Conversely, the VR systems 701 of the conventional examples 1 and 2 include a communication means 716 that receives the communication observation signal sent from the observation system 751 or sends the voice input in the VR system 701 to the observation system 751 as communication voice information, a separation means 715 that separates and outputs the compressed VR video (ERP image), graphics information, and compressed voice information from the communication observation signal from the communication means 716, a VR video decoding means 710 that decodes the compressed VR video (ERP image) from the separation means 715, converts the ERP image from the VR video decoding means 710 into a spherical video, cuts out a part according to the control information from the VR control means 707, and makes it a video that can be displayed on the VR display means 704, and a VR display control means 708 that outputs a VR video to be displayed on the VR display means 704 together with the graphics information of the graphics generation means 712 that converts the graphics information to be displayed from the graphics information output from the separation means 715. The VR display means 704 outputs the VR video from the VR display control means 708 for viewing with both eyes. The outputs of the rotation detection means 703 that detects the inclination of the VR display means 704 in the front-back, left-right directions or the direction of the white of the eyes and the position detection means 702 that detects the position of the VR display means 704 in the left-right, front-back, and height directions are sent to the VR control means 707, and the video displayed on the VR display means 704 by the output of the VR control means 707 and the voice output by the voice reproduction means 709 by the voice reproduction control means are appropriately controlled. The compressed voice information separated by the separation means 715 is decoded by the voice decoding means 713 and sent to the voice reproduction control means 709 as voice information. In the voice reproduction control means 709, according to the control information from the VR control means 707, the balance in the left-right, front-back, and height directions, and in some cases, frequency characteristics, delay processing, or synthesis of an alarm as the VR system 701, etc. are performed. Also in the graphics generation means 712, graphics for displaying the system menu, warnings, etc. of the VR system 701 are generated and overlaid on the VR image and displayed on the VR display means 704.The VR system 701 is provided with voice input means 706 for inputting the voice of the user of the VR system 701. The voice information from the voice input means 706 is compressed by the voice compression means 714 and sent as compressed voice information to the multiplexing means 717, where it is sent as voice information for communication from the communication means 716 to the observation system 751.

[0033] FIG. 8 is a diagram for explaining a conventional example. As a typical realization example of the observation system of the conventional example 2, a realization example of a 360° camera 801 will be described.

[0034] A typical example of the 360° camera 801 combines two imaging systems, namely an ultra-wide-angle lens 854, a shutter 853, and an image sensor 852, to capture an image of 360° in the front, back, up, and down directions. Since it may be necessary to combine two or more imaging systems to capture higher-quality images, in this example, the VR imaging camera 804 is shown with two or more imaging systems. The imaging system may be configured by combining independent cameras. In that case, generally, there is a high-speed digital video I / F after the ADC 851 of the video, and thus it is connected to a high-speed digital video input connected to a video system bus connected to the GPU (Graphics Processing Unit) 803 or the CPU (Central Processing Unit) 802. Here, it will be described as an integrated unit.

[0035] The main components of the 360° camera 801 include the VR shooting camera 804 composed of the plurality of imaging systems described above, the GPU 803 mainly for processing video data and graphics, the CPU 802 for general data processing, processing related to input / output, and controlling the entire 360° camera 801, the EEPROM (Electrical Erasable Programmable ROM) 813 for storing programs for operating the CPU 802 and GPU 803, the RAM 814 used for storing data for the operation of the CPU 802 and GPU 803, the SD card (registered trademark) 821 which is a removable memory for storing videos, voices, or programs, the wireless communication element 820 for performing wireless communication via WiFi (registered trademark) or Bluetooth (registered trademark) for data exchange with the outside and receiving operations from the outside, the buttons and display elements 808 for operations and displays, the battery 807 and the power control element 812, the voice input unit composed of a plurality of microphones (microphone group 819) or microphone terminals 825 for inputting voices, the microphone amplifier 818, and the ADC 817, the voice output unit composed of the speaker 826 or the headphone terminal 824, the amplifier 823, and the DAC 822, the video system bus mainly connecting the VR shooting camera 804 and the CPU 802 and used for reading digital video data, the memory bus connecting the aforementioned EEPROM 813, RAM 814, SD card 821 and the GPU 803, CPU 802 for data exchange with the memory, the system bus to which the aforementioned CPU 802, GPU 803, wireless communication element 820, voice input unit, and voice output unit are connected for control and data exchange, the I / O bus to which the aforementioned buttons and display elements 808, power control element 812, and although not shown, the voice input unit, voice output unit, VR shooting camera 804, etc. are included for control and low-speed data exchange, and several bus conversion units 815 and 816 connecting each bus. The motion and position detection unit 860 is further connected to the I / O bus. For some processes, whether they are performed by the GPU 803 or the CPU 802 may be different in this example, and the bus configuration may also be different from this example, but there is no difference in the functional configuration and operation described later.

[0036] Each VR shooting camera 804 includes a lens 854 for shooting a wide-angle video, an image sensor 852 that converts the light collected by the lens 854 into an electrical signal, a shutter 853 that is located between the lens 854 and the image sensor 852 and blocks the light, and a diaphragm (not shown here) that is located at the same position as the shutter 853 and controls the intensity of the light from the lens 854. It is composed of an ADC 851 that converts the analog electrical signal from the image sensor 852 into a digital video signal. Although not shown, each is controlled by the CPU 802 through the I / O bus, and the state is notified to the CPU 802.

[0037] Buttons include a power switch 806 for turning the power on / off, a shooting start / end button 811 for the operation of starting / ending shooting, a shooting mode selection button 809 (which may not be provided) for changing the shooting mode, and a zoom button 810 for moving the lens 854 and digitally controlling the angle of view for zooming in and out.

[0038] The power control element 812 may be integrated with the battery 807. It stabilizes the voltage, manages the battery capacity, etc., and supplies power to all components (not shown). Further, it supplies power to the HMD / VR glasses through USB or AV output.

[0039] Each function realized by the GPU 803 is realized by dedicated hardware and programs such as image processing. Generally, the functions realized by the CPU 802 are realized by general-purpose hardware and programs. As an example, the GPU 803 is used to realize a VR video processing unit 842, a VR video compression unit 841, a graphics generation unit 843, etc. Also, as an example, the CPU 802 is used to realize a memory control unit 835, a multiplexing unit 832, an audio compression unit 833, an audio decoding unit 834, and a separation unit 831.

[0040] FIG. 9 is a diagram for explaining a conventional example. Based on FIG. 9, an example of the realization of VR system 901 will be described as a typical realization example of the observation system of Conventional Example 2. In this realization example, it is assumed that VR system 901 is composed of a computer or smartphone 951 and an HMD or VR glasses 902 connected thereto. There are also examples where it is realized by the HMD or VR glasses 902 alone. In that case, it can be considered that the functions of both CPUs and GPUs are integrated, and the peripheral functions are also combined.

[0041] The main components of the computer / smartphone 951 in the VR system 901 include a high-speed communication element 970 such as WiFi or Ethernet (registered trademark) for connecting to the observation system, a GPU 954 mainly for processing video data and graphics, a CPU 965 for general data processing and overall control of the computer / smartphone 951, a non-volatile memory 962 such as a hard disk or flash memory for storing programs for operating the CPU 965 and GPU 954, a RAM 961 used for storing data for the operation of the CPU 965 and GPU 954, a power switch 963 and a power control element 964 for supplying power to each part, an AV output 952 for outputting video and audio signals to the HMD / VR glasses 902, an I / F such as a USB 953 for controlling the HMD / VR glasses 902 and acquiring data therefrom, a memory bus for connecting the RAM 961 and non-volatile memory 962 and for the CPU 965 and GPU 954 to access, a system bus for the CPU 965 and GPU 954 to access the AV output 952, USB 953, and communication element 970, a bus connection (bus conversion unit 960) for connecting the system bus and the memory bus, and although not shown here, it is composed of a display device, an input device for operation, and other general-purpose I / Fs.

[0042] For some processes, whether they are performed by the GPU 954 or the CPU 965 may be different in this example, and the bus configuration may also be different from this example, but there is no difference in the functional configuration and operations described later. As an example, the GPU 954 is used to implement a motion / position detection processing unit 955, a VR control unit 956, a VR display control unit 957, a VR video decoding unit 958, a graphics generation unit 959, and the like. Also, as an example, the CPU 965 is used to implement an audio decoding unit 966, an audio playback control unit 967, a multiplexing unit 968, and a demultiplexing unit 969.

[0043] Also, the AV output 952 and the USB 953 can be replaced with a high-speed bidirectional I / F, for example, an I / F such as USB Type-C (registered trademark). In that case, the HMD / VR glasses 902 side will also be connected with the same I / F or connected with a converter that converts the I / F. Generally, when sending video via the USB 953, appropriate video compression is performed by the CPU 965 or the GPU 954 to compress the data volume, and the video is sent to the HMD / VR glasses 902 via the USB 953.

[0044] The main components of the HMD / VR glasses 902 in the VR system 901 include a voice input unit consisting of a microphone 906, a microphone amplifier 917, and an ADC 918, a voice output unit consisting of a speaker 907 or a headphone terminal 908, an amplifier 919, and a DAC 920, a VR display unit consisting of two sets of lenses 904 and a display element 905 for the user to view VR images, a motion / position sensor 903 consisting of a motion / position detection unit and an orientation detection unit composed of a gyro sensor, a camera, or an ultrasonic microphone, etc., a wireless communication element 927 such as Bluetooth for communicating with a controller (not shown), a volume button 909 for controlling the output volume from the voice output unit, a power switch 921 for turning on / off the power of the HMD / VR glasses, a power control element 924 for power control, the aforementioned EEPROM 913, RAM 914, an SD card, a GPU 910, a CPU 915, a memory bus for connecting them and performing data exchange with the memory, the aforementioned CPU 915, GPU 910, wireless communication element 927, an AV input 925 for receiving video and audio signals from a computer / smartphone 951, an I / F such as a USB 926 for receiving control signals from the computer / smartphone 951 and sending video, audio signals, and motion / position data, a CPU 915 for mainly controlling voice compression (implemented by the voice compression unit 916), switches, power, etc., and overall control of the HMD / VR glasses 902, a GPU 910 for mainly performing video display processing (implemented by the video display processing unit 912) for adjusting the video to the VR display unit and correcting / shaping the motion / position information sent to the computer / smartphone 951 from the information of the motion / position sensor 903 (implemented by the motion / position detection unit 911), an EEPROM 913 for storing programs and data for operating the CPU 915 and GPU 910, a RAM 914 for storing data during the operation of the CPU 915 and GPU 910, a memory bus for connecting the CPU 915, GPU 910, RAM 914, and EEPROM 913, a system bus to which the CPU 915, GPU 910, USB 926, voice input unit, voice output unit, and wireless communication element 927 are connected for control and data exchange, the aforementioned buttons, power control element 924, motion / position sensor 903, and although not shown, the voice input unit,It is composed of an I / O bus that performs control and low-speed data exchange, including an audio output unit and a VR shooting camera, and several bus conversion units 922 that connect the respective buses. For some processes, whether they are performed by the GPU 910 or the CPU 910 may be different from this example, and the bus configuration may also be different from this example, but there is no difference in the functional configuration and operation described later.

[0045] Since the video data from the AV input 925 has a large data volume and is high-speed, it is illustrated as being directly taken in by the GPU 910 when the system bus does not have sufficient speed.

[0046] Note that the video information captured by the camera of the motion / position sensor 903 may be sent to the display element as information for the user to check the periphery of the HMD / VR glasses 902, or may be sent to the computer / smartphone 951 through the USB 926 for the user to monitor whether there is a dangerous situation.

[0047] The power control element 924 receives power supply from the USB 926 or the AV input 925, performs voltage stabilization, battery capacity management, etc., and supplies power to all components (not shown). In some cases, the battery 923 may be provided internally or externally and connected to the power control element 924.

[0048] The states of the buttons and cursor of the controller (not shown) are acquired by the CPU 915 through the wireless communication element 927 and are used for button operations, movement, and application operations in the VR space. The position and orientation of the controller are detected by a camera or an ultrasonic sensor in the motion / position detection unit. After appropriate processing is performed by the motion / position sensor, it is used for control by the CPU 915 and is also sent to the computer / smartphone 951 via the USB 926 and used for the drawing of graphics and image processing executed by the program executed by the CPU 915 or the GPU 910. Since the basic operations are not directly related to the present invention, they are omitted.

[0049] FIG. 10 is a diagram for explaining a conventional example. An implementation example of an integrated VR system 1001 having functions for VR in a computer / smartphone and HMD / VR glasses will be described.

[0050] As can be seen in FIG. 10, the functions of the computer / smartphone and the HMD / VR glasses are integrated, and the respective functions of the CPU and GPU are realized by one CPU and GPU.

[0051] The communication element 1033 is typically WiFi for performing wireless communication, and has a battery 1026 because it does not have a power cable. It has an interface with a general-purpose computer such as USB1034 for charging the battery 1026 and for initial settings.

[0052] Since the integrated VR system 1001 does not require an AV output, AV input, or USB to connect the computer / smartphone and the HMD / VR glasses, high-quality and delay-free transmission of AV information and efficient control are possible. However, by integrating them, due to limitations in size, it may not be possible to use a high-performance CPU 1027 or GPU 1006 due to power, heat, and space limitations, and the VR function may be limited.

[0053] However, not being connected by a cable increases the degree of freedom and can expand the range of applications.

[0054] Also, by realizing part of the functions on a computer in the cloud, etc., it is possible to compensate for the lack of performance and realize high-functional applications.

[0055] The integrated VR system 1001, similar to the configuration described in FIGS. 8 and 9, further includes a lens 1002, a display element 1011, a microphone 1003, a microphone amplifier 1007, an ADC 1009, a speaker 1004, a headphone terminal 1005, an amplifier 1008, a DAC 1010, a RAM 1019, an EEPROM 1020, a bus conversion 1021, a motion position sensor 1022, a power switch 1023, a volume button 1024, and a power control element 1025. Also, video display processing 1012, motion / position detection processing 1013, VR control 1014, VR display control 1015, motion / position detection 1016, VR video decoding 1017, and graphics generation 1018 are realized using the GPU 1006. Further, audio compression 1028, audio decoding 1029, audio playback control 1030, multiplexing 1031, and demultiplexing 1032 are realized using the CPU 1027.

[0056] FIG. 11 is a diagram for explaining a conventional example. Based on FIG. 11, a more detailed configuration of a VR video processing unit 1103 that processes video captured by a VR shooting camera 1151 of the observation systems of Conventional Examples 1 and 2 will be described.

[0057] As described above, the VR shooting camera has a plurality of cameras cm for shooting 360° omnidirectional video, typically cameras cm with ultra-wide-angle lenses, and rectangular individual videos with the same pixels captured by each camera cm are input to a VR video processing unit 1103 realized by a program or a dedicated circuit in the GPU 1101.

[0058] In the VR video processing unit 1103, first, the input plurality of videos are respectively evaluated for the shooting directions of the respective cameras cm and the videos obtained by shooting, and the videos captured by the respective cameras cm are input to a stitching processing unit 1105 that performs a process of synthesizing and connecting them so as to form a continuous spherical video. The spherical video data output from the stitching processing unit 1105 is mapped onto a plane by, for example, the equirectangular projection (ERP) method by a VR video mapping unit 1104, output from the VR video processing unit 1103 as an ERP image, and passed to the next VR video compression unit 1102.

[0059] Although the connection between the video bus and the cameras is shown such that each camera is connected to the bus, within the VR shooting camera 1151, the videos shot by each camera may be combined into one signal and sent to the video bus in a time-division manner, and then input to the VR video processing unit 1103. In a simple configuration, since there are two cameras cm, instead of using a bus, the GPU 1101 receives the outputs of the two cameras respectively, and the VR video processing unit 1103 receives and processes the videos shot in parallel.

[0060] FIG. 12 is a diagram for explaining a conventional example. Based on FIG. 12, a more detailed configuration of the VR display control unit 1204 of the VR systems of Conventional Examples 1 and 2 will be described.

[0061] As described above, the VR display control unit 1204 is implemented by a program or a dedicated circuit in the GPU 1201 of a computer / smartphone, and is composed of a mapping unit 1206 and a display VR video conversion unit 1205.

[0062] The operation is as follows. The communication element 1261 receives the communication data sent from the observation system, the compression video is separated by the separation unit 1232 of the CPU 1231, the GPU 1201 receives the video via the memory bus, and is decoded by the VR video decoding unit 1207 to become a planar video (ERP image). The planar video is converted into a 360° spherical video by the mapping unit 1206 of the VR display control unit 1204, and then in the following display VR video conversion 1205, the part to be displayed by the VR display means 1202 is cut out based on the control information output by the VR control unit 1203.

[0063] Specifically, the center of the ERP image is taken as the entire surface and the origin of the 360° spherical video. The initial video of the VR video displayed on the VR display means 1202 is, with the origin as the center and according to the capabilities of the VR display means 1202, the video for the right eye is slightly shifted to the right, the video for the left eye is slightly shifted to the left, and in the height direction, the videos are cut out using the initial set values and are displayed on the display elements for the right eye and the left eye. From here, the cut-out position changes according to the rotation of the VR system to the left and right, looking up and down, etc.

[0064] Generally, the video from a 360° camera does not change with the movement of the VR system. However, in the case of video generated by CG, the position changes due to the movement of the VR system or operations with a controller.

[0065] The initial value of the cutout from a 360° spherical video may be from the previous cutout position. Generally, however, a function to return to the initial position is provided.

[0066] Figure 13 is a diagram for explaining a conventional example. An operation example of Conventional Example 2 will be described based on Figure 13.

[0067] In the observation system, the audio input unit (microphone array, microphone terminal, microphone amplifier, ADC) inputs audio (S1325), and the audio compression unit compresses the audio (S1326).

[0068] At the same time, a plurality of cameras (lenses, shutters, image sensors, ADCs) of the VR shooting camera shoot a moving image (S1321). The stitching processing unit of the VR video processing unit stitches it into a spherical video with camera 1 at the center as the center (S1322). This is used by the VR video mapping unit to generate an ERP image by an orthographic cylindrical projection method or the like (S1323), and the VR video compression unit appropriately compresses it (S1324).

[0069] The compressed ERP image and audio information are multiplexed by the multiplexing unit (S1327) into a transmissible format, and are sent (transmitted) to the VR system by the wireless communication element (S1328).

[0070] Over time, in some cases, it moves to a new direction and position (S1329), and the sending from the audio input and shooting with a plurality of VR shooting cameras is repeated.

[0071] Here, the graphics information may be superimposed on the video before video compression or may be multiplexed together with the video and audio as the graphics information, but this is omitted.

[0072] In the VR system, in a computer / smartphone, information sent from the observation system is received by a communication element (S1301) and sent to a separation unit. In the separation unit, the sent compressed video information and compressed audio information are separated (S1302). The compressed audio information separated by the separation unit is sent to an audio decoding unit and decoded (S1303) to become uncompressed audio information. The audio information from the audio decoding unit is sent to an audio playback control unit, and audio processing is performed by the audio playback control unit based on the position / orientation information of the VR observation system sent via the system bus from the VR control unit of the GPU (S1304). The audio information on which the audio processing has been performed is sent via the system bus, either through AV output or via USB, to the audio output unit (DAC, amplifier, speaker, and headphone terminal) of the HMD / VR glasses and output as audio (S1305). As audio processing, balance control of volume in the left and right or in space, change of frequency characteristics, delay, movement in space, similar processing for only a specific sound source, addition of sound effects, etc. are performed.

[0073] The compressed video signal is sent from the video data from the separation unit of the CPU of the computer / smartphone to the VR video decoding unit of the GPU via the memory bus and decoded in the VR video decoding unit (S1307) and input as an ERP image to the VR display control unit. In the VR display control unit, the ERP video is mapped to a 360° spherical video by the mapping unit (S1308), and an appropriate part is cut out from the 360° spherical video based on the position / orientation information of the VR system from the VR control unit in the display VR video conversion unit (S1309), and is displayed as a VR video by the VR display unit (display element, lens) (S1310).

[0074] Receiving from the observation system, video display, and audio output are repeated.

[0075] Note that regarding graphics here, there are cases where graphics are separated simultaneously with video / audio separation and superimposed on the VR video by the VR display control unit, or cases where they are generated within the VR system and superimposed on the VR video, etc., and the explanation is omitted here.

[0076] (Summary of the Disclosure) The summary of the present disclosure is as follows.

[0077] A video display system according to an aspect of the present disclosure is a video display system for displaying a display video by a display device, and includes a photographing unit that photographs a video as a wide-angle-of-view video, a data acquisition unit that acquires data regarding a moving direction of the photographing unit, a metadata configuration unit that acquires metadata based on the acquired data, and a transmission unit that transmits the photographed wide-angle-of-view video together with the metadata. An observation device, a VR device, a receiving unit that receives the wide-angle-of-view video and the metadata, an orientation estimation unit that estimates the orientation of the display device, and a difference calculation based on the difference between the estimated orientation of the display device and the moving direction of the photographing unit on the metadata. A difference calculation unit that calculates a relative movement direction that is the moving direction of the photographing unit relative to the orientation of the display device, a presentation unit that presents the calculated relative movement direction to the user of the display device, and an orientation estimated by the orientation estimation unit from the received wide-angle-of-view video. A VR device including a video generation unit that generates a display video including a part of the video corresponding to the visual field portion according to the orientation of the display device, and a display device that displays the display video.

[0078] In such a video display system, a relative movement direction indicating the movement direction of the photographing unit is calculated by using metadata. And since the relative movement direction is presented to the user, it is possible to suppress problems associated with the photographing unit moving unexpectedly, such as the photographing unit suddenly moving. Therefore, according to the video display system, it is possible to display an appropriate video from the viewpoint of suppressing problems associated with the photographing unit moving unexpectedly.

[0079] Further, for example, the presentation unit may generate and output graphics indicating the calculated relative movement direction, and superimpose the output graphics on a part of the video to present the relative movement direction to the video generation unit.

[0080] According to this, the relative movement direction can be presented to the user by the graphics.

[0081] Further, for example, the graphics may display an arrow indicating the relative movement direction on the display video.

[0082] According to this, the relative movement direction can be presented to the user by the graphics that display an arrow indicating the relative movement direction on the display video.

[0083] Further, for example, the graphics may display a mask which is an image for covering at least a part other than the relative movement direction side on the display video.

[0084] According to this, the relative movement direction can be presented to the user by the graphics that display a mask which is an image for covering at least a part other than the relative movement direction side on the display video.

[0085] Further, for example, the estimated orientation of the display device is a discrete display direction that changes by a first discrete value based on the actual orientation of the display device, and the difference calculation unit may calculate the difference between the discrete display direction and the movement direction of the imaging unit in the metadata.

[0086] According to this, the relative movement direction that changes discretely can be presented to the user.

[0087] Further, for example, the first discrete value may change such that it becomes minimum when the actual orientation of the display device and the inclination in the horizontal plane of the movement direction of the imaging unit in the metadata coincide, and becomes maximum when the difference between the actual orientation of the display device and the inclination in the horizontal plane of the movement direction of the imaging unit in the metadata becomes maximum.

[0088] According to this, the relative movement direction that changes discretely such that it becomes minimum when the actual orientation of the display device and the inclination in the horizontal plane of the movement direction of the imaging unit in the metadata coincide, and becomes maximum when the difference between the actual orientation of the display device and the inclination in the horizontal plane of the movement direction of the imaging unit in the metadata becomes maximum can be presented to the user.

[0089] Further, for example, the moving direction of the imaging unit on the metadata is a discrete moving direction that changes by a second discrete value based on the actual orientation of the display device, and the difference calculation unit may calculate the difference between the estimated orientation of the display device and the discrete moving direction.

[0090] According to this, the relative moving direction that changes discretely can be presented to the user.

[0091] Further, for example, the observation device may have an input interface for an operator to input for moving the video unit, and the data acquisition unit may acquire data input by the operator via the input interface.

[0092] According to this, data input by the operator can be acquired via the input interface to configure the metadata.

[0093] Further, for example, the observation device may have a position detection unit for detecting the position of the imaging unit, and the data acquisition unit may acquire data based on the position of the imaging unit detected over time by the position detection unit.

[0094] According to this, data can be acquired based on the position of the imaging unit detected over time by the position detection unit to configure the metadata.

[0095] Further, for example, the imaging unit may capture a virtual wide-angle video by capturing within a virtual image space constituted by computer graphics.

[0096] According to this, in the virtual wide-angle video, it becomes possible to display an appropriate video from the viewpoint of suppressing problems associated with the imaging unit making an unexpected movement for the user.

[0097] Further, for example, it may include at least a part of the functions provided in the observation device and the VR device, the observation device and the VR device may be connected by a network, and an information processing device that undertakes a part of the processing of the observation device or the VR device may be provided.

[0098] According to this, a video display system can be realized by the observation device, the VR device, and the information processing device.

[0099] Further, for example, the information processing device may include a receiving unit that receives a wide-angle video and data from the observation device, a metadata configuring unit that generates metadata based on the wide-angle video and the data, a movement information calculating unit that calculates movement information regarding the movement of the imaging unit on the metadata, and a presenting unit that generates and outputs graphics indicating the calculated movement information, and the graphics is superimposed on a part of the video corresponding to the visual field portion according to the estimated orientation of the display device among the wide-angle videos, so as to present the movement direction of the imaging unit to the user of the display device, and a transmitting unit that transmits the wide-angle video, the graphics, and the metadata.

[0100] According to this, a video display system can be realized by the observation device, the VR device, and the information processing device having the above-described configuration.

[0101] Further, for example, the information processing device may include a receiving unit that receives a wide-angle video and data from the observation device, a metadata configuring unit that generates metadata based on the wide-angle video and the data, a movement information calculating unit that calculates movement information regarding the movement of the imaging unit on the metadata, and a transmitting unit that transmits the wide-angle video, the movement information, and the metadata.

[0102] According to this, a video display system can be realized by the observation device, the VR device, and the information processing device having the above-described configuration.

[0103] Further, for example, the information processing apparatus may include a receiving unit that receives a wide-angle video, data, and data related to the orientation of the display device, a metadata configuring unit that generates metadata based on the wide-angle video, the data, and the data related to the orientation of the display device, a difference calculating unit that calculates a relative movement direction, which is the movement direction of the imaging unit relative to the orientation of the display device, based on the difference between the orientation of the display device and movement information related to the movement of the imaging unit, a presenting unit that generates and outputs graphics indicating the calculated relative movement direction, the graphics being superimposed on a part of the video corresponding to the visual field portion according to the estimated orientation of the display device among the wide-angle video to present the relative movement direction to the user of the display device, and a transmitting unit that transmits the wide-angle video, the graphics, and the metadata.

[0104] According to this, a video display system can be realized by the observation device, the VR device, and the information processing apparatus having the above-described configuration.

[0105] Further, for example, the information processing apparatus may be provided on a cloud connected to a wide area network and connected to the observation device and the VR device via the wide area network.

[0106] According to this, a video display system can be realized by the observation device, the VR device, and the information processing apparatus provided on the cloud and connected to the observation device and the VR device via the wide area network.

[0107] Further, an information processing method according to an aspect of the present disclosure is an information processing method for causing a display device to display a display video, the method including receiving metadata based on data related to the movement direction of an imaging unit that captures a video as a wide-angle video, and calculating and outputting a relative movement direction, which is the movement direction of the imaging unit relative to the orientation of the display device, based on the difference between the estimated orientation of the display device and the movement direction of the imaging unit in the metadata.

[0108] Such an information processing method can achieve the same effects as the video display system described above.

[0109] Also, a program according to one aspect of the present disclosure is a program for causing a computer to execute the information processing method described above.

[0110] Such a program can achieve the same effects as the video display system described above by using a computer.

[0111] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0112] Note that each of the embodiments described below shows comprehensive or specific examples. The numerical values, shapes, materials, components, arrangement positions of the components, connection forms, steps, order of steps, etc. shown in the following embodiments are merely examples and are not intended to limit the scope of the claims. In addition, among the components in the following embodiments, the components not described in the independent claims are described as optional components.

[0113] Note that each figure is not necessarily drawn precisely. In each figure, substantially the same configurations are denoted by the same reference numerals, and overlapping descriptions are omitted or simplified.

[0114] Also, in this specification, terms indicating the relationship between elements such as parallel, terms indicating the shape of elements such as rectangular, as well as numerical values and numerical ranges are not expressions representing only strict meanings, but are expressions meaning that they include substantially equivalent ranges, for example, differences such as an error of about several percent.

[0115] (Embodiment) [Configuration] First, an overview of the video display system in the embodiment will be described with reference to FIGS. 14 and 15. FIG. 14 is a diagram showing a schematic configuration of the video display system according to the embodiment. FIG. 15 is a diagram showing an example of a video displayed in the video display system according to the embodiment.

[0116] As shown in FIG. 14, the video display system 500 of the present embodiment is realized by an observation device 300, a server device 200 connected via a network 150, and a display device 100 connected via the network 150.

[0117] The observation device 300 is a so-called omnidirectional camera that can capture images of the entire 360-degree surrounding area. The observation device 300 may be, for example, a photographing device 300a that is held by hand for photographing, or an observation device 300b that is fixed with a tripod or the like. In the case of the photographing device 300a held by hand, it is easy to take pictures while moving around. Hereinafter, these types will be collectively referred to as the observation device 300 without particularly distinguishing them. The observation device 300 has an optical element such as a fish-eye lens, and can capture a wide viewing angle area, for example, 180 degrees, with a single sensor array. Then, using a plurality of combinations of optical elements and sensor arrays arranged to complement different wide viewing angle areas, a 360-degree wide viewing angle video can be captured. Note that, for the images captured by each of the plurality of sensor arrays, a process (stitching) of identifying and overlapping corresponding elements with each other is performed. As a result, for example, one image that can mutually convert a plane such as an orthographic cylindrical view and a spherical surface is generated. By continuously generating such images in the time domain, a video (moving image) that changes in the time domain is generated. Note that the inside of the spherical video is also referred to as a 3D video space.

[0118] Also, in the present embodiment, two 3D video spaces with a shift corresponding to the human parallax are generated. Such two 3D video spaces may be generated by simulation or the like from one 3D video space, or may be generated by two cameras with a parallax shift. In the present embodiment, a VR video in which the user can view an arbitrary direction of the 3D video space can be displayed from within the 3D video space.

[0119] The network 150 is a communication network for communicably connecting the observation device 300, the server device 200, and the display device 100 to each other. Here, a communication network such as the Internet is used as the network 150, but it is not limited to this. Further, the connection between the observation device 300 and the network 150, the connection between the server device 200 and the network 150, and the connection between the display device 100 and the network 150 may be performed by wireless communication or by wired communication, respectively.

[0120] The server device 200 is a device for performing information processing and the like, and is realized using, for example, a processor and a memory. The server device 200 may be realized by an edge computer or may be realized by a cloud computer. Further, one server device 200 may be provided for one video display system 500, or one server device 200 may be provided for a plurality of video display systems 500. That is, the server device 200 may perform various processes in a plurality of video display systems 500 in parallel. Note that the server device 200 is not an essential component in the video display system 500.

[0121] For example, by distributing and arranging each functional unit of the server device 200 described later to each of the observation device 300 and the display device 100, it is also possible to realize a video display system including only the observation device 300 and the display device 100. In particular, if the display device 100 is realized by an information processing terminal such as a smartphone having a display panel, the functional unit of the server device 200 can be easily realized using a processor or the like of the information processing terminal. Alternatively, by giving the functions of the observation device 300 and the display device 100 to the server device 200, a part of the functions of the observation device 300 or the display device 100 can be reduced, and an existing observation device or display device can be diverted. That is, by integrating various functions in the server device 200, it becomes possible to easily realize a video display system. Each functional unit of the server device 200 will be described later with reference to FIG. 16 and the like.

[0122] The display device 100 is a glass-type HMD that supports two separated lens barrels by locking the temple parts extending from the left and right sides to the auricles, thereby holding the two lens barrels at positions corresponding to the user's right and left eyes respectively. A display panel is built into each lens barrel of the display device 100. For example, as shown in FIG. 15, an image with binocular disparity is projected toward each of the user's left and right eyes. In FIG. 15, (L) shows an image of one frame in the left-eye video, and (R) shows an image of the same one frame in the right-eye video. Note that the display device 100 does not have to be a terminal dedicated to such video display. It is also possible to implement the display device of the present disclosure using a display panel provided in a smartphone, a tablet terminal, a PC, or the like.

[0123] Hereinafter, with reference to FIG. 16, a more detailed configuration of the video display system 500 of the present embodiment will be described. FIG. 16 is a block diagram showing a functional configuration of the video display system according to the embodiment. As shown in FIG. 16 and as described in FIG. 14, the video display system 500 includes a display device 100, a server device 200, and an observation device 300.

[0124] The display device 100 includes a display unit 101 and an orientation estimation unit 102. The display unit 101 is a functional unit that outputs an optical signal according to image information using a backlight, a liquid crystal panel, an organic EL, a micro LED, or the like. The display unit 101 controls the output optical signal so that an image is formed on the retina of the user's eye via optical elements such as a lens and an optical panel. As a result, the user can visually recognize the image formed on the retina. The display unit 101 can cause the user to visually recognize continuous images, that is, video, by continuously outputting the above-described images in the time domain. In this way, the display unit 101 displays video for the user of the display device 100.

[0125] The orientation estimation unit 102 is a functional unit for estimating the orientation of the display device 100. The orientation estimation unit 102 is realized by various sensors such as an acceleration sensor and a gyro sensor built in an appropriate position of the display device 100. The orientation estimation unit 102 estimates the orientation of the display device 100 by estimating how much the posture has changed in which direction with respect to a reference direction preset in the display device 100. As described above, since the display device 100 is supported by the user's head (pinna and nasal root), it moves together with the user's head.

[0126] Then, by estimating the orientation of the display device 100, a visual field portion corresponding to the orientation can be cut out from the wide-angle video and displayed. That is, depending on the orientation of the display device 100 estimated by the orientation estimation unit 102, the visual field area that is the visual field area that the user wants to view in the 3D video space when looking in the direction in which the user's head is facing can be displayed. Here, the orientation of the display device 100 estimated here is the direction along the normal direction of the display panel of the display device 100. Since the display panel is arranged to face the user's eyes, the user's eyes are usually located in the normal direction of the display panel. For this reason, the direction of the display device 100 coincides with the direction connecting the user's eyes and the display panel.

[0127] However, there may be a case where the direction of the display device 100 and the user's line of sight direction deviate due to the user's eye movement. In this case, if the display device 100 is equipped with a sensor (eye tracker) for detecting the user's line of sight, the detected user's line of sight may be used as the direction of the display device 100. That is, the eye tracker is another example of the orientation estimation unit.

[0128] In addition to the above, the display device 100 is equipped with a power supply, various input switches, a circuit for driving the display panel, wired and wireless communication modules for input and output, an audio signal processing circuit such as a signal converter and an amplifier, and a microphone and a speaker for audio input and output.

[0129] The server device 200 includes a receiving unit 201, a difference calculation unit 202, a presentation unit 203, and a video generation unit 204. The receiving unit 201 is a processing unit that receives (acquires) various signals from an observation device 300 described later. The receiving unit 201 receives a wide-angle video captured by the observation device 300. The receiving unit 201 also receives metadata acquired by the observation device 300. Furthermore, the receiving unit 201 receives information regarding the orientation of the display device 100 estimated by the display device 100.

[0130] The difference calculation unit 202 is a processing unit that calculates a relative movement direction, which is the movement direction of the imaging unit 301 relative to the orientation of the display device 100, based on the difference between the orientation of the display device 100 and the movement direction of the imaging unit 301 included in the metadata. The detailed operation of the difference calculation unit 202 will be described later.

[0131] The presentation unit 203 is a processing unit that presents the relative movement direction calculated by the difference calculation unit 202 to the user of the display device 100. Here, an example will be described in which the presentation unit 203 causes the video generation unit 204 to perform the above presentation by including content indicating the relative movement direction in the display video generated by the video generation unit 204. However, the presentation of the relative movement direction is not limited to the example of including it in the above display video. For example, it may be presented as sound from a predetermined arrival direction corresponding to the relative movement direction within a 3D sound field, or it may be presented by vibrating a device such as a vibration device held by the user's both hands on the side corresponding to the relative movement direction. The detailed operation of the presentation unit 203 will be described later together with the detailed operation of the difference calculation unit 202.

[0132] The video generation unit 204 cuts out a part of the video corresponding to the visual field portion according to the orientation of the display device 100 estimated by the orientation estimation unit 102 from the received wide-angle video, and further generates a display video including content indicating the calculated relative movement direction if necessary. The detailed operation of the video generation unit 204 will be described later together with the detailed operations of the difference calculation unit 202 and the presentation unit 203. The server device 200 also has a communication module for transmitting the generated display video to the display device 100.

[0133] The observation device 300 includes a photographing unit 301, an input interface 302, a position detection unit 303, a data acquisition unit 304, a metadata acquisition unit 305, and a transmission unit 306. The photographing unit 301 is a functional part related to image photographing and is integrally configured with other functional components of the observation device 300. Therefore, moving the photographing unit 301 means moving the entire observation device 300 including the photographing unit 301. Note that the photographing unit 301 may be separated from other functional components of the photographing device 300 by wired or wireless communication. In this case, the photographing unit 301 can also be moved independently. The photographing unit 301 includes an optical element, a sensor array, an image processing circuit, etc. The photographing unit 301 outputs, for example, the luminance value of the light of each pixel received on the sensor array via the optical element as 2D luminance value data. The image processing circuit performs post-processing such as noise removal of the luminance value data, and also performs processing for generating a 3D image space from 2D image data such as stitching.

[0134] The input interface 302 is a functional unit used when an input is made by an operator who operates the observation device 300. For example, the input interface 302 includes a stick that can be tilted in each of 360 degrees corresponding to the moving direction of the photographing unit 301, and a physical sensor that detects the tilting direction. The operator can input the moving direction of the photographing unit 301 to the system by moving the photographing unit 301 and tilting the stick in the moving direction. Also, the photographing unit 301 may be a self-propelled device, and may be configured such that the self-propelled photographing unit 301 moves in that direction according to an input to the input interface 302. Note that the input interface 302 is not an essential configuration. If either the position detection unit 303 described later or only one of them is provided, the present embodiment can be realized.

[0135] The position detection unit 303 is a sensor that detects the moving direction of the imaging unit 301, that is, the observation device 300. The position detection unit 303 is realized by a distance measurement sensor such as LiDAR and an odometer, etc., and can detect the current position of the observation device 300. By detecting this current position over time, the position detection unit 303 can detect the moving direction of the observation device 300.

[0136] The data acquisition unit 304 is a functional unit that acquires data regarding the moving direction of the imaging unit 301, that is, the moving direction of the observation device 300, from the input interface 302, the position detection unit 303, etc. The data acquisition unit 304 is connected to at least one of the input interface 302 and the position detection unit 303, and acquires the moving direction of the observation device 300 as a physical quantity from these functional units.

[0137] The metadata acquisition unit 305 is a functional unit that acquires this metadata by converting the data regarding the moving direction of the observation device 300 acquired by the data acquisition unit 304 into metadata for adding to the captured video data. The acquired metadata may include various data used within the video display system 500 in addition to the data regarding the moving direction of the observation device 300. That is, the metadata acquisition unit 305 is an example of a metadata configuration unit that constitutes metadata capable of reading a plurality of data from one piece of information by combining a plurality of data into one.

[0138] The transmission unit 306 is a communication module that transmits the captured video (wide-angle video) and the acquired metadata. The transmission unit 306 communicates with the reception unit 201 of the server device 200 to transmit the captured video and the acquired metadata and have them received by the reception unit.

[0139] [Operation] Next, the operation of the video display system 500 configured as described above will be described with reference to FIGS. 17 to 21. FIG. 17 is a flowchart showing the operation of the video display system according to the embodiment.

[0140] When the operation of the video display system 500 is started, the imaging unit 301 captures video, and the input interface 302, the position detection unit 303, the data acquisition unit 304, and the metadata acquisition unit 305 operate, so that metadata including data regarding the moving direction of the imaging unit 301 is acquired. The metadata is received by the server device 200 together with the video captured via the transmission unit 306 and the reception unit 201 (S101).

[0141] Also, the orientation estimation unit 102 of the display device 100 continuously estimates the orientation of the display device 100. The display device 100 transmits the orientation of the display device 100 estimated by the orientation estimation unit to the server device 200. As a result, the server device 200 receives the estimated orientation of the display device 100 (S102). Note that the order of step S101 and step S102 may be interchanged. The server device 200 determines whether there is movement of the imaging unit 301 from the data regarding the moving direction of the imaging unit 301 (S103). If it is determined that there is movement of the imaging unit (Yes in S103), the server device 200 enters an operation for presenting the relative moving direction to the user of the display device 100. Specifically, the difference calculation unit 202 calculates the relative moving direction of the video with respect to the direction in which the user is looking (that is, corresponding to the orientation of the display device 100) as the relative moving direction based on the orientation of the display device 100 and the data regarding the moving direction of the imaging unit 301 in the metadata (S104).

[0142] Here, the calculation of the relative moving direction will be described with reference to FIGS. 18 and 19. FIG. 18 is a first graph for explaining the calculation of the relative moving direction according to the embodiment. FIG. 19 is a second graph for explaining the calculation of the body moving direction according to the embodiment.

[0143] The relative moving direction is calculated using, for example, the following formula (1).

[0144]

Equation

[0145] However, in the above formula (1), Do indicates the actual moving direction of the imaging unit 301, Dv indicates the actual orientation of the display device 100, and Da indicates the angular difference between them.

[0146] Normally, when a person identifies an angle on an image, there is a range that is regarded as the same angle. Furthermore, when the relative moving direction of a video is shown on an image, there may be cases where a subtle angular difference does not need to be considered. In the present embodiment, by discretely handling such an angular region that does not need to be continuous, it is possible to clearly convey in which direction the video is moving. Specifically, in the present embodiment, as the orientation of the display device 100, a discrete display direction that changes by a first discrete value based on the actual orientation is used, and as the moving direction of the imaging unit 301, a discrete moving direction that changes by a second discrete value based on the actual orientation is used. Therefore, here, it becomes as follows in formula (2) below.

[0147]

Number

[0148] However, in the above formula (2), int(x) indicates a function that truncates the decimal part of x, and DD indicates the first discrete value and the second discrete value (the same value). Note that the first discrete value and the second discrete value do not have to be the same value. Also, in the above formula (2), with respect to the point where the orientation of the display device 100 coincides with the reference direction (Dv = 0) when the imaging unit 301 moves in the reference direction (Do = 0), a parallel translation of the graph is applied so that it becomes a point-symmetric graph.

[0149] According to the above formula (2), for example, when DD = 90 degrees and Do = 0, as shown in FIG. 18, Da that discretely increases and decreases by 90 degrees with respect to Dv can be obtained. In FIG. 18, when the reference direction is 0 degrees, in the range where the user's orientation is -180 degrees or more and less than -135 degrees (-180 ≦ Dv < -135), the relative movement direction is 180 degrees, indicating that the video is moving to the rear side of the user. Also, when the reference direction is 0 degrees, in the range where the user's orientation is -135 degrees or more and less than -45 degrees (-135 ≦ Dv < -45), the relative movement direction is 90 degrees, indicating that the video is moving to the side of the user.

[0150] Furthermore, when the reference direction is 0 degrees, in the range where the user's orientation is -45 degrees or more and 45 degrees or less (-45 ≦ Dv ≦ 45), the relative movement direction is 0 degrees, indicating that the video is moving to the front side of the user. Also, when the reference direction is 0 degrees, in the range where the user's orientation is greater than 45 degrees and 135 degrees or less (45 < Dv ≦ 135), the relative movement direction is -90 degrees, indicating that the video is moving to the side of the user. Also, when the reference direction is 0 degrees, in the range where the user's orientation is greater than 135 degrees and 180 degrees or less (135 < Dv ≦ 180), the relative movement direction is -180 degrees, indicating that the video is moving to the rear side of the user.

[0151] Also, there may be problems when discrete values are set to constant values in this way. For example, regarding the change in the relative movement direction on the front side of the user with respect to the change in the movement direction on the rear side of the user, the user's discriminability may be high. In this case, the first discrete value may be changed according to the user's orientation (the orientation of the display device 100). Specifically, when the user's orientation is in the range of -45 degrees or more and 45 degrees or less (-45 ≦ Dv ≦ 45), DD = 5 degrees; when the user's orientation is in the range of -135 degrees or more and less than -45 degrees, and in the range of more than 45 degrees and 135 degrees or less (-135 ≦ Dv < -45, 45 < Dv ≦ 135), DD = 20 degrees; when the user's orientation is in the range of -180 degrees or more and less than -135 degrees, and in the range of more than 135 degrees and 180 degrees or less (-180 ≦ Dv < -135, 135 < Dv ≦ 180), DD = 45 degrees may be set. By doing so, as shown in FIG. 19, when the user's orientation is close to the reference direction of the movement of the imaging unit 301, the relative movement direction can be changed sensitively, and when the user's orientation is away from the reference direction of the movement of the imaging unit 301, the change in the relative movement direction can also be made less sensitive.

[0152] In this way, it is also effective to change the first discrete value so that the value when the actual orientation of the display device 100 is 180 degrees or -180 degrees is larger than the value when the actual orientation of the display device 100 is 0 degrees.

[0153] Returning to the flowchart of FIG. 17, after calculating the relative movement direction, the presentation unit 203 generates graphics corresponding to this relative movement direction (S105). Then, the video generation unit 204 cuts out a field-of-view portion corresponding to the orientation of the display device 100 from the wide-angle video (S106), and generates a display video by superimposing the graphics generated by the presentation unit 203 on the cut-out partial video (S107). FIG. 20 is a conceptual diagram for explaining the generation of the display video according to the embodiment. In FIG. 20, (a) shows a partial video cut out from the wide-angle video, (b) shows the graphics 99 generated by the presentation unit 203, and (c) shows the display video generated by superimposition. By superimposing the graphics 99 on the partial video, an arrow 99a indicating the relative movement direction is displayed in the display video.

[0154] Also, FIG. 21 is another conceptual diagram for explaining the generation of the display video according to the embodiment. As shown in FIG. 20, the arrow 99a is visually easy to understand in terms of its direction, but the change on the video may be difficult to understand. Therefore, as shown in FIG. 21, it may be configured to give a large change on the display video by generating and superimposing a mask 99b, which is an image for covering a part on the side opposite to the relative movement direction side, as the graphics 99. In this example, the user makes a line-of-sight movement to visually recognize the remaining video from the area covered by the mask. Since the direction of this line-of-sight movement corresponds to the relative movement direction, there is an advantage that the change on the video is easy to understand and the relative movement direction is naturally easy to recognize. Here, "covering" also includes being covered by a semi-transparent image in which the area to be covered can be seen partially through.

[0155] Returning again to the flowchart of FIG. 17, if it is determined that there is no movement of the photographing unit (No in S103), the server device 200 does not perform an operation for presenting the relative movement direction to the user of the display device 100, and proceeds to step S108. Then, the video generation unit 204 cuts out a field-of-view portion corresponding to the orientation of the display device 100 from the wide-angle video (S108), and generates the cut-out partial video as the display video as it is.

[0156] In this way, the user can be made to grasp the relative movement direction. Therefore, even if the video moves later, it is less likely to cause the user to have a sense of discomfort with respect to the movement. Thus, in the video display system 500, it is possible to cause the display device 100 to display an appropriate video from the viewpoint of suppressing VR sickness and the like.

[0157] [Embodiment] Hereinafter, a more detailed description will be given based on the embodiments of the embodiment. In this embodiment, taking the problem that VR sickness is likely to occur as an example, a more specific example for solving this problem will be described. VR sickness is likely to occur when there is a discrepancy between the information obtained from vision and hearing and the acceleration information (in the case of uniform motion, the stimulus cannot be perceived) obtained from the vestibular sense. Usually, in a VR device (such as an HMD), information regarding vision and hearing can be reproduced, and stimuli can be given to vision and hearing. However, it is difficult to give a stimulus that makes the vestibular organ feel acceleration without installing a large-scale motion platform or the like, and it is limited to facility-type VR devices such as theme parks and VR game centers.

[0158] When "VR sickness" occurs when experiencing the VR space in VR sightseeing or the like, the VR sightseeing itself becomes a painful experience, and the user will stop using it. Therefore, it is important to avoid VR sickness.

[0159] When VR games, large equipment, buildings, etc. are confirmed in VR, one can move according to one's own will. In this case, generally, by performing jump movement or "warp movement", "VR sickness" is avoided. In the case of VR sightseeing that constitutes a VR space with 360° camera video, one cannot move the camera according to one's own will. The on-site photographer will move the camera. When this camera movement is performed independently of one's own will, it is easy to get VR sickness, just like getting drunk while sitting in the passenger seat of a car.

[0160] In the case of a VR experience like a roller coaster, although the direction of one's own movement cannot be determined, the movement direction of the vehicle can be anticipated, so it is less likely to get dizzy like when driving a car oneself.

[0161] When a 360° camera moves or pans (changes the default direction of progress) when the VR viewer does not intend it (cannot anticipate it), there is a risk of causing VR dizziness.

[0162] Figure 22 is a schematic diagram of a configuration example of a video display system according to an embodiment. Further, Figure 23 is a schematic diagram of another configuration example of the video display system according to the embodiment.

[0163] In the present invention, in order to avoid VR dizziness associated with the movement of a 360° camera used in VR tourism or the like, in addition to the 360° video, metadata (such as the start time and movement direction of the camera movement or pan for shooting the 360° video) is sent from the observation system to the VR system, and on the VR system side, a visual effect (such as display on the screen) or non-visual effect (such as sound, vibration of the controller, etc.) that appropriately notifies the viewer of the movement just before or during the camera movement is generated, so as to prompt the viewer to prepare or prevent dizziness. By displaying a visual effect video different from the normal one, VR dizziness is avoided (reduced).

[0164] Therefore, there is a photographer with a 360° camera in a remote location who sends the 360° video + metadata, and the viewer can generate a CG that guides the movement direction based on information such as α with a VR device and synthesize it with the sent 360° video, so that one can prepare for the movement, reduce VR dizziness, and then view it. Therefore, according to the present invention, VR dizziness can be reduced when providing a VR tourism service, and a more comfortable VR tourism experience can be provided.

[0165] FIG. 24 is a diagram showing an example of the overall system configuration of the video display system according to the embodiment. The detailed configuration of the embodiment of the present invention will be described. The present invention, compared with Conventional Examples 1 and 2, further generates metadata indicating the orientation, position, etc. of the observation system in the observation system, sends it to the VR system side, and performs appropriate processing on the VR system side to reduce VR sickness and obtain orientation and position in the VR space to receive appropriate guidance in the VR space and realize not getting lost. Centering on the differences compared with Conventional Example 2, it will be explained here.

[0166] Regarding the closed communication means between multiple VR systems, it can be further added to the configuration described here. Also, when operating from the VR system side and when information from the observation system is recorded in advance, since the configuration described here is the basis, the differences will be explained separately.

[0167] In the configuration of the present invention, regarding the shooting, stitching, ERP image conversion, sending in the observation system, receiving in the VR system, conversion from the ERP image to the spherical video, and cropping of the display image to the VR system according to the orientation, position, etc. of the VR system, it is the same as Conventional Example 2.

[0168] In the present invention, the observation system has a metadata generation unit for generating metadata for notifying the VR system side of the orientation, position, or moving direction, etc. The metadata generation unit generates metadata including the movement, orientation, and position of the VR shooting device by the guide on the observation system side or the photographer inputting data from the data input device, or the data acquisition unit acquiring data related to the shooting position and moving direction of the observation system. The generated metadata is sent from the communication device to the VR system together with video (wide-angle video), audio, etc.

[0169] The VR system extracts metadata from the communication device and sends the metadata to the metadata analysis unit. Here, as an example, based on the content of the metadata and the orientation and position of the VR system (the estimation of the direction of the user's face (head) by rotation detection means described later and head tracking using a motion / position sensor, and the detection of the position and movement of the VR system), an appropriate message is sent to the user of the VR system. For example, an arrow as an operation instruction is generated as graphics from the GFX generation unit, superimposed on the VR video (field of view video), and displayed on the VR display device.

[0170] Not only the display of graphics but also instructions in voice and appropriate processing of VR videos may be performed.

[0171] Also, although not shown, it is also possible to appropriately vibrate the operation input means (controller) of the VR system according to the metadata and the orientation / position of the VR system.

[0172] FIG. 25 is a diagram showing another example of the overall system configuration of the video display system according to the embodiment. In another example of the embodiment of the present invention, instead of the guide or the photographer on the observation system side inputting data from the data input device, metadata is generated by detecting appropriate markers from the orientation, position, movement of the observation system, or the video being photographed.

[0173] The metadata generation unit generates metadata based on the photographed video or the information on the orientation, position, and movement of the observation system, and sends it to the VR system from the communication device. The configuration on the VR system side is the same as that in the above embodiment. That is, the data input device is not an essential configuration in the present invention, and the invention can be implemented without providing the data input device.

[0174] As important elements in the present invention, there are two: (i) a method for video switching to avoid VR dizziness, and (ii) a method for detecting, recording, and transmitting direction and orientation. (i) further includes four sub-elements: a video switching method when changing direction, a video switching method when changing location, a video transmission method when moving on foot, and a video transmission method when moving by vehicle. Also, (ii) further includes two sub-elements: detection of direction and orientation, and transmission of direction and orientation.

[0175] FIG. 26 is a diagram showing an example of the functional configuration of a video display system according to an embodiment. FIG. 27 is a diagram showing an example of the functional configuration of an observation system according to an embodiment. FIG. 28 is a diagram showing an example of the functional configuration of a VR system according to an embodiment.

[0176] An example of the realization of a 360° camera among the realization examples of the observation system (observation device) 2651 of the embodiment of the present invention will be described with reference to FIGS. 26 and 27.

[0177] The 360° camera 2701 of the embodiment of the present invention is substantially the same as the realization example of the 360° camera of Comparative Example 2, and the differences will be described. In the 360° camera 2701 of the embodiment of the present invention, a position and orientation detection unit 2702 is added as a program of the CPU 802, and metadata based on the position and orientation detected by the position and orientation detection unit 2702 is multiplexed and sent to the VR system (display device) 2601 via the wireless communication element (transmission unit) 820.

[0178] The detection of position and orientation is to detect the moving direction, position, and start timing of movement. However, the operator of the camera inputs the position, direction, and movement timing of the moving destination by operating a movement input button (data input means (input interface)), and it is processed by the position and orientation detection unit 2702 (data acquisition unit, metadata conversion unit) and sent to the VR system 2601 as metadata.

[0179] In some cases, the position and orientation analysis units 2703 and 2704 indicated by dashed lines analyze the captured video to detect the timing of starting movement, detect the destination by comparing with a pre-specified image, detect the position and orientation of the moving destination, analyze the gestures and body movements of the guide to detect the direction of movement, provide light-emitting elements such as buttons and LEDs on an object like an indicator rod held by the guide, detect the direction and position of the moving destination by detecting the pulsed light emission pattern of the light-emitting element due to the operation of the guide, or select one from a plurality of pre-determined destinations. Alternatively, analyze the voice from the microphone, identify the destination from the words spoken by the guide, detect the moving direction, and convert it into appropriate metadata by the position and orientation detection unit 2702 executed by the CPU 802 and send it to the VR system 2601.

[0180] An implementation example of the VR system (display device) 2601 according to an embodiment of the present invention will be described with reference to FIGS. 26 and 28.

[0181] The difference between the VR system of Conventional Example 2 of the VR system 2601 according to an embodiment of the present invention and the VR system will be mainly described.

[0182] In the embodiment, the position and orientation determination units 2802 and 2803 are added as programs to the CPU 965 and GPU 954 of the computer / smartphone 2801 of the VR system of Conventional Example 2.

[0183] In the position and orientation determination unit 2803 of the CPU 965, the position and orientation of the observation system 2651 or the guide and the target object are received as metadata from the observation system 2651 via the communication element (reception unit) 970. When changing the voice, the guide voice is generated by the voice playback control unit 967 implemented by the program of the CPU 965, and the playback voice is appropriately processed. When changing the VR video or graphics, the position and orientation determination unit 2803 of the CPU 965 sends the metadata to the GPU 954 via the system bus.

[0184] In the GPU 954, the metadata received by the position / orientation determination unit 2802 is processed, and the information is processed by the graphics generation unit 959 for display as a graphic, and is superimposed and displayed on the VR video by the VR display control unit 957, or is sent to the VR control unit 956. The VR display control unit (display control unit) 957 appropriately processes the VR video together with the position / orientation state of the VR system 2801 detected by the motion / position detection processing unit (detection unit) 955 from the motion / position sensor 903, sends the video data from the AV output 952 to the AV input 925, and the video display processing unit 912 displays it as a VR video (field of view video) on the display element (display unit).

[0185] The above processing of audio, graphics, and VR video is realized independently. Even when no other processing is performed, multiple processes may be realized, and there may be cases where a process is selected during the operation of the VR system or the observation system.

[0186] Also, the position / orientation detection processing of the observation system 2651 may be realized by a computer system between the observation system 2651 such as the cloud and the VR system 2801. In this case, either no metadata is sent from the observation system 2651, or data input by the operator is sent as metadata. For example, with the position / orientation detection means in the cloud, the position / orientation or movement of the observation system, the guide, or the target object is detected from the video, audio, or metadata sent from the observation system and sent as metadata to the VR system. Thereby, the effects of this embodiment can also be exhibited with an existing 360° camera.

[0187] Regarding some processes, whether they are performed by the GPU 954 or the CPU 965 may be different from this example, and the bus configuration may also be different from this example, but there is no difference in the functional configuration and operation described later.

[0188] Regarding the integrated VR system, this example is almost the same as the conventional example. By realizing the functions with one CPU 965 and one GPU 954 respectively, a small integrated VR system can be realized.

[0189] FIG. 29 is a diagram showing an example of the operation flow of the video display system according to the embodiment. The operation of the embodiment of the present invention will be described.

[0190] The audio input in the observation system, the input of VR video from the VR shooting camera, the basic operation of the graphics, and the basic operations of the audio information, video information, and graphics in the VR system are the same as those in the conventional example 2, so they will be omitted.

[0191] In the position and orientation information detection step of the observation system according to the embodiment of the present invention, the position and orientation of the observation system are detected by a position and orientation information detection unit (data acquisition unit) realized by a program in the CPU from the signal from the motion and position detection unit (S2930). In the next metadata detection step, the detected position and orientation are converted into metadata (S2931), and in the next video, audio, and metadata multiplexing step, the metadata is multiplexed together with the video, audio, or graphics by the multiplexing unit (S2927). In the next transmission step, the multiplexed information is sent to the VR system by the wireless communication element (S2928).

[0192] Here, when the detection of the position and orientation is performed by analyzing the video captured by the VR shooting camera or the audio input by the audio input unit, the operator or guide of the observation system may input by the movement input button. When detecting from the audio, the audio is analyzed by a position and orientation analysis unit realized by a program in the CPU, and the analysis result is sent to the position and orientation detection unit. When analyzing the video, the video is analyzed by a position and orientation analysis unit realized by a program in the GPU or a dedicated circuit, and the analysis result is sent to the position and orientation detection unit of the CPU via the system bus.

[0193] Also, the movement input button may be a switch like a remote control or a GUI like a smartphone. The position and orientation analysis can be based on voice recognition. Specifically, it can recognize words such as "right side" and "left side", add inaudible voice outside the audible band to the voice emitted from the tools the guide has, or modulate it within the audible band, and analyze the instructions regarding position and orientation from the input voice. It can recognize the guide, specific buildings, or scenery from the captured images and compare them with the position and orientation of the observation system. It can photograph and recognize the movement of tools the guide has, such as a small flag. It can use visible light or non-visible light such as infrared light for the tools the guide has, make them emit light with appropriate pulses, and analyze the captured video, etc.

[0194] In addition to sending the position and orientation information to the VR system as metadata, graphics data, such as graphics data of an arrow indicating the movement direction, can be generated by the graphics generation unit of the GPU from the information of the position and orientation detection unit and sent to the VR system as graphics data.

[0195] In the VR system, in the receiving step, the metadata sent from the observation system by the computer / smartphone is received via the communication element (S2901). In the video / audio / metadata separation step, the separation unit of the CPU separates the metadata (S2902) and sends it to the position and orientation determination unit of the CPU and, via the memory bus, to the position and orientation determination unit of the GPU. In the metadata analysis step, the position and orientation determination unit of the CPU analyzes the metadata (S2906) and sends the obtained position and orientation information to the voice playback control unit of the CPU, and also to the VR control unit or the graphics generation unit of the GPU via the memory bus. Alternatively, the position and orientation determination unit of the GPU analyzes the metadata and sends it to the VR control unit or the graphics generation unit of the GPU.

[0196] In the video cut-out step based on the position, orientation, and metadata of the VR system, processing of audio, video, and graphics (for example, steps S2304 and S2309) is performed by the audio playback control unit, the VR control unit, and the graphics generation unit respectively based on the position and orientation information sent from the observation system.

[0197] Specifically, in the audio processing unit, there are operations such as playing a guide voice indicating the moving direction, playing noise before the start of movement, for example, the rattling sound at the start of vehicle movement, changing the level of the audio from the observation system during movement, or playing appropriate sound effects, changing the spatial spread to clear only the moving direction, etc.

[0198] In the VR control unit, operations include reducing the frame rate to perform time-lapse frame-by-frame transmission, displaying still images during movement and returning to video when movement is completed, reducing the resolution other than the display of the moving direction, tilting the VR video according to the moving direction and speed, applying a mask other than the display of the moving direction to make it less visible, etc.

[0199] In the graphics generation unit, when the position and orientation information is sent as graphics information, the graphics generation unit generates graphics, and the VR display control unit appropriately changes the direction and superimposes it on the VR video. When generating graphics on the VR system side, there are operations such as masking the video other than the front of the moving direction with semi-transparent graphics, or displaying an arrow indicating the moving direction, or displaying a map indicating the movement.

[0200] The above processing may also be processed by a computer system such as an observation system or on a cloud between the observation system and the VR system.

[0201] In addition, for steps not described above, the explanations here are omitted by referring to the explanations in the same steps in FIG. 13. Specifically, step S2303 corresponds to step S1303, step S2305 corresponds to step S1305, step S2307 corresponds to step S1307, step S2308 corresponds to step S1308, step S2310 corresponds to step S1310, step S2321 corresponds to step S1321, step S2322 corresponds to step S1322, step S2323 corresponds to step S1323, step S2324 corresponds to step S1324, step S2325 corresponds to step S1325, and step S2326 corresponds to step S1326.

[0202] FIG. 30 is a diagram for explaining the result of the operation of the video display system in the embodiment. The example of FIG. 30 shows the case where arrow display graphics are superimposed. Specifically, the position and orientation determination means receives the position and orientation of the observation system or the guide and the target as metadata, sends the information to the graphics generation means to display it as a graphic, and superimposes and displays it on the VR video by the VR display means.

[0203] For example, in the example at the left end of the figure, when it is stopped, the state where the front image is stopped is shown. Next, in the example at the upper left, just before starting to move, the state where an arrow in the direction in which it should start to move is displayed is shown. Next, in the example at the upper right, when the user of the VR system is facing right, an arrow pointing left, when facing left, an arrow pointing right, and when facing backward, an arrow in the backward direction are shown (in the figure, as an example, an arrow pointing left when facing right is shown). When the user changes the direction, it becomes a straight-ahead display like the example at the upper left. Since it is basic that the user changes the direction in the traveling direction, it becomes an arrow to the right or left (not a straight-ahead display) until the traveling direction and the user's direction almost match. This is the same when changing the direction to the left or right. Next, in the example at the right end, the case of MAP display is shown. Also in the case of MAP display, the state where the arrow and the MAP are appropriately rotated and displayed according to the direction is shown.

[0204] FIG. 31 is a diagram for explaining the result of the operation of the video display system in the embodiment. The example of FIG. 31 shows the case where the voice guide is reproduced. Specifically, in the position / orientation determination means, the position / orientation of the observation system or the guide and the target are received as metadata, and based on that information, the voice reproduction control means generates a guide voice and appropriately processes the reproduced voice.

[0205] For example, in the leftmost example in the figure, when stopped, it shows that the voice is played so that the voice of the guide can be heard from the left side where the guide stands. Next, in the example on the upper left, when starting to walk straight, it shows that the voice of the guide can be heard from the front. In the case of a moving object such as a vehicle, a vibration sound can be heard before starting to move. Next, in the example on the upper right, when about to turn right, it shows that the voice of the guide saying "Turn right" can be heard from the right side. Next, in the rightmost example, when the user of the VR system turns to the right or the back and the original right hand side changes to correspond to the left hand side, it shows that the voice of the guide can be heard from the left hand or the front. In case of confusion, replace "right" in the voice with "left", or combine with graphics to show the direction with an arrow or the like.

[0206] Figure 32 is a diagram for explaining the result of the operation of the video display system in the embodiment. The example of Figure 32 shows the case where the controller vibrates. Specifically, it can convey moving by vibrating a control means (controller) held by the operator of the VR system, etc., and when there are multiple controllers, by selecting one and vibrating it, give meaning of the direction of movement to the vibration, and give meaning of the speed of movement by the intensity of the vibration. Regarding the moving speed, it may be combined with conveying by graphics such as presenting by the color and size of the arrow, or conveying by the volume of the voice.

[0207] For example, in the leftmost example in the figure, when stopped, it shows that the front image stops and the controller is not vibrating. Next, in the example on the upper left, immediately before starting to move, when going straight, it shows that the left and right controllers vibrate. Also, a pattern such as vibrating gently while moving and vibrating strongly immediately before stopping may be adopted. The vibration pattern is not limited to this example. Next, in the example on the upper right, when turning right, it shows that only the right controller vibrates. Next, in the rightmost example, when the user of the VR system is facing the right side, it shows that the left controller vibrates and continues to vibrate until facing the front.

[0208] FIG. 33 is a diagram for explaining the result of the operation of the video display system in the embodiment. The example of FIG. 33 shows the case of converting a VR image. Specifically, in the position / orientation determination means, the position / orientation of the observation system or a guide or a target is received as metadata, and this information is sent to the VR control means. Together with the position / orientation state of the VR system, the VR display control means appropriately processes the VR video and displays it on the VR display means.

[0209] For example, in the example at the left end of the figure, when stopped, the state where the front image is stopped is shown. Next, in the example at the upper left of the center, it is shown that, immediately before starting to move, except for the direction of starting to move, a mask is applied to narrow the field of view and clarify the moving direction. Next, in the example at the upper right of the center, an arrow is shown before starting to move, and until the movement stops, the resolution of the video is reduced, or it becomes a still image while moving. Next, in the example at the right end, it is shown that the mask situation changes according to the orientation of the user of the VR system.

[0210] Next, an embodiment of a method for detecting directions and orientations will be described. As a method for detecting, recording, and transmitting an absolute (static) orientation, the orientation can be detected by a compass function or GPS, and a marker indicating the orientation can be embedded in the video during shooting. Together with embedding a marker indicating the position of the front of the 360° camera or a predetermined position (for example, taking the center of the ERP video to be transmitted as the front), (a) embedding a mark indicating the orientation in the video itself, (b) embedding a mark indicating the orientation in the header of the video, (c) transmitting the orientation before transmitting the video, (d) in the case of a moving image, periodically transmitting a mark indicating the orientation as metadata, and the like. Also, it is possible to store the orientation at which the photographer starts shooting as the home orientation, and thereafter record or transmit the orientation at an angle from that orientation. Also, it is possible to use a building or a scenery as a marker.

[0211] On the one hand, as a method for detecting, recording, and transmitting relative directions, a main guide or emcee can be determined, and control can always be performed based on the relative direction from that person. In that case, there are (a) detection by sound, (b) detection by the shape of a marker (for example, something like a tour guide flag), (c) detection by light, etc. Also in this case, the relative directions and positions from markers such as buildings and scenery may be utilized.

[0212] It is also possible to configure to convey more detailed information to the user of the VR system by changing the size of the graphics, the length of the arrow, the color usage, the color density, the volume of the sound, the frequency of the sound, the length of the vibration of the controller, the vibration pattern, and the strength of the vibration according to the distance and speed to the destination. For example, when the distance is short, the arrow can be made short, and when it is far, it can be made long. Also, when the distance is short, the arrow can be made thin, and when it is far, it can be made thick. Also, when the distance is short, the sound can be made small, and when it is far, it can be made large. Also, when the distance is short, the vibration of the controller can be made weak, and when it is far, it can be made strong.

[0213] FIG. 34 and FIG. 35 are diagrams showing an example of the operation flow of a method for detecting position and orientation information according to an embodiment. As a specific example of detecting orientation and position, for example, there is an example of detecting orientation and position from a GPS device provided in an observation system. Based on the coordinates that change moment by moment with respect to the coordinates of the starting point of movement, the moving speed and direction (orientation) are detected and recorded and transmitted as metadata. When processing on the cloud side, there are cases where the GPS data is sent as it is, or cases where it is appropriately thinned out and sent. The moving direction (orientation) is processed so as to change smoothly, such as by appropriately sampling or taking the average over a predetermined time.

[0214] Also, for example, there is an example of detecting the moving speed and direction from the movement of the scenery being photographed by an observation system, and recording and transmitting them as metadata. As an example, the direction of change is calculated from the photographed data to detect the moving speed and direction. Also, as another example, the moving speed and direction (orientation) are detected from a predetermined target building or scenery.

[0215] In addition, for example, there is a case where a photographer sets the direction of movement for an observation system. As an example, a guide or the photographer moves using a switch or remote control provided in the observation system, sets the direction for the observation system, and sends this as metadata. For this purpose, the switch may be in the guide's hand and send information to the observation system by appropriately modulating radio waves, sound, or light. In the case of light or sound, the observation system may not perform any special processing, but instead record it directly in the video or audio sent to the VR system and send it to the VR system, allowing the VR system to determine the start of movement and the direction. Similarly, in some cases, a cloud computer system between the observation system and the VR system may detect the start and direction of movement and send them to the VR system as metadata.

[0216] As another example, the guide's hand gestures, voice, flag direction, etc. are judged, and when the conditions predefined in the observation system are met, the movement is started and the direction is sent as metadata. For this purpose, similar to the above, in some cases, the start and direction of movement may be detected by a computer system on the cloud or the VR system. Also, it is possible for the guide or the photographer to perform actions such as setting or gesturing well before a change occurs, and this can be notified to the users of the VR system in advance.

[0217] As specific examples of recording and sending direction, orientation, and movement speed, for example, recording and sending as metadata separately from the video and audio sent to the VR system, or recording and sending by synthesizing markers such as arrows in the video sent to the VR system. However, this is effective in cases like this when it is assumed that all users of the VR system are in a substantially the same state, and when not only markers but also cockpit or window graphics, etc. are sent, as it is independent of the state of the users on the VR system side.

[0218] In addition, for example, it may be possible to add explanations about direction, orientation, position, speed, etc. to the audio sent to the VR system and then send it. In this case as well, it is assumed that all users of the VR system are in a substantially the same state.

[0219] Also, for example, processing such as dropping frames, reducing resolution, or hiding a part of the video and audio to be sent to the VR system, and indicating the direction and orientation can be mentioned. These processes may be processed by a computer system on the cloud between the observation system and the VR system and sent to the VR system, or may be processed on the VR system side.

[0220] FIG. 36 is a diagram showing a configuration example of metadata according to an embodiment. A configuration example of the metadata of this embodiment will be described.

[0221] The type of metadata contains a predefined code or character string indicating that it is the metadata of the present invention. The version number is a number for when the metadata structure is changed. For example, in the evaluation stage, it is used like 0.81 (0081), during the proof-of-concept experiment it is 0.92 (0092), and at the time of release it is 1.0 (0100), etc., using major and minor versions, and is used with the idea of guaranteeing compatibility between the same major versions.

[0222] When the function code is 0, it indicates that the metadata information is invalid, and in other cases, it indicates the type of information in the metadata. For example, 0001 indicates that it is a format for describing the reference position, camera, positions and moving directions and speeds of the guide and the target. 0002 indicates graphics data, 0003 indicates information of the VR system, 0011 is one with queue data sent from the observation system attached to 0001, 0021 is one with queue data and defines a moving target, etc.

[0223] The reference position is the data of the position that serves as the reference for the position data, and it is defined in advance as a whole system including units such as X (east-west distance), Y (north-south distance), Z (height-direction distance), or expressed in terms of longitude, latitude, and altitude. When the reference position is 0, it indicates that the position at the time of resetting the entire system is used as the reference. Also, for the position of the camera and the position of the guide, it is determined in advance whether they are absolute coordinates or relative coordinates from the reference position.

[0224] The moving direction and speed indicate the observation system or the moving situation of the guide. When there is queue data, it indicates how it will move in the future.

[0225] The number of targets indicates the destination for tourism in the case of VR tourism. When the number of targets is 0, it indicates that there are no targets.

[0226] The verification code is a code for verifying whether the metadata data is incorrect during transmission, and for example, CRC or the like is used.

[0227] Regarding the order, content, and values of each item of the metadata, even if they are different from this configuration example, they may have the same function.

[0228] Figure 37 is a diagram showing the operation flow related to the metadata according to the embodiment. Figure 38 is a diagram showing the operation flow related to the metadata according to the embodiment. Figure 39 is a diagram showing the operation flow related to the metadata according to the embodiment. Figure 40 is a diagram showing the operation flow related to the metadata according to the embodiment. Figure 41 is a diagram showing the operation flow related to the metadata according to the embodiment. Figure 42 is a diagram showing the operation flow related to the metadata according to the embodiment. Figure 43 is a diagram showing the operation flow related to the metadata according to the embodiment.

[0229] Each of FIGS. 38 to 43 shows a sub - flow in the operation flow shown in FIG. 37. As shown in FIG. 37, first, the entire system is reset (S3701). Next, initial values are set in the observation system (S3702). Next, VR sightseeing is started (S3703). Next, an explanation of the tourist destination is given (S3704). Then, it is determined whether there is movement (S3705). If there is no movement (No in S3705), it is further determined whether VR sightseeing has ended (S3706). If VR sightseeing has not ended (No in S3706), the process returns to the explanation of the tourist destination (S3704). When VR sightseeing has ended (Yes in S3706), the end process of the entire system is performed and the flow ends.

[0230] On the other hand, if there is movement, an input of the movement direction is made in the observation system (S3708). Then, the metadata is sent to the VR system (S3709), and an arrow according to the metadata is generated in the VR system (S3710). Next, the arrow is corrected according to the position and orientation of the VR system (S3711). The corrected arrow is synthesized into the VR video of the VR system (S3712), and it is determined whether the movement has ended (S3713). If the movement has not ended (No in S3713), the process returns to the correction of the arrow according to the position and orientation of the VR system (S3711).

[0231] If the movement has ended (Yes in S3713), it is further determined whether the movement of the observation system has ended (S3714). When the movement of the observation system has ended (Yes in S3714), the process returns to the determination of whether VR sightseeing has ended (S3706). Also, if the movement of the observation system has not ended (No in S3714), the process returns to the input of the movement direction in the observation system (S3708).

[0232] As shown in FIG. 38, in the reset (S3701) of the entire system, the metadata is initialized in the position / azimuth detection unit of the observation system (S3801). At this time, predetermined values are set for the type and version number of the metadata, the function code, the reference position, the position of the camera, the position of the guide, the moving direction, the speed, and the number of targets are set to 0, the verification code is calculated and set, and the current position of the observation system is set at the reference position. Then, the metadata multiplexed by the multiplexing unit from the observation system is sent to the VR system through the wireless communication element (S3802). After that, the VR system receives the metadata through the communication element, the verification code is confirmed by the position / azimuth determination unit, and it is confirmed that the metadata is normal (if it is not a precision instrument, the observation system is requested to retransmit the metadata) (S3803). Since the function code of the metadata is 0, the internal data related to the position / azimuth such as the position / azimuth determination unit of the VR system is reset (S3804) and the process ends.

[0233] As shown in FIG. 39, in the setting of the initial value in the observation system (S3702), a predetermined code is set for the function code of the metadata in the position / azimuth detection unit of the observation system (here, it is described as 0001) (S3901). After that, according to the settings of the entire system, the reference position, the position of the camera, the position of the guide, and the position of the target are set in the metadata by the position / azimuth detection unit of the observation system (here, assuming that the east-west distance (east is positive), the north-south distance (north is positive), and the height (sky is positive) are represented in meters as the relative position from the reference position), and the process ends (S3902).

[0234] As shown in FIG. 40, in the input of the moving direction in the observation system (S3708), the data of the destination is set from the moving direction and speed to the metadata by the position and orientation detection unit in any of the following pre-selected methods, and the process is terminated (S4001). The pre-selected methods are: (a) selecting a previously specified target (destination) with a movement input button and sending it to the position and orientation detection unit; (b) inputting the start timing and moving direction from the movement input button and sending it to the position and orientation detection unit (the input moving direction and speed use pre-determined values and are updated periodically with the movement direction and speed); (c) recognizing the movement of the guide, the movement of the flag, etc. from the video captured by the VR camera by the position and orientation analysis unit of the GPU and sending it to the position and orientation detection unit of the CPU (the moving direction and speed are the same as above); (d) operating the light-emitting device or audio output device of the guide, selecting a previously set target, and outputting the corresponding code by light or sound. This is captured by the camera or microphone of the observation system, the code is recognized by the position and orientation analysis unit, and sent to the CPU position and orientation detection unit.

[0235] As shown in Fig. 41, in the generation of arrows according to metadata in the VR system (S3710), graphics preparation is performed (S4101). At this time, the metadata for sending graphics data from the observation system in advance at the time of system startup or at the start of VR sightseeing is sent, or it is prepared at the time of installation of the VR sightseeing application and stored in the RAM or non-volatile memory. Then, the graphics generation unit reads the above-mentioned graphics data from the RAM or non-volatile memory, generates an arrow, and ends the process (S4102). At this time, the front of the observation system is centered on the ERP image or marked on the ERP image by other methods and recognized by the VR display control unit, but the graphics generation unit always takes the center of the image as the front. The direction of the arrow is assumed to be the same as the front of the observation system in the VR system, and the direction of the arrow is corrected according to the moving direction or the direction of the target in the metadata sent from the position and orientation determination unit. The display position of the arrow is calculated as the display position according to the mode of the display position of the graphics in the metadata and the display position of the graphics data.

[0236] As shown in Fig. 42, in the correction of the arrow according to the position and orientation of the VR system (S3711), the VR control unit acquires the orientation of the VR system from the position detection unit and the orientation detection unit (S4201). Then, the VR display control unit compares the orientation of the VR system with the moving direction of the observation system, generates correction information for the direction of the arrow, corrects it, and ends the process (S4202). As a result, when the moving direction and the orientation of the VR system are the same within a predetermined range, the direction in the previous step is maintained. When the orientation of the VR system is rightward within a predetermined range with respect to the moving direction, a correction is made to direct the arrow to the left. When the orientation of the VR system is leftward within a predetermined range with respect to the moving direction, a correction is made to direct the arrow to the right. When the orientation of the VR system is opposite within a predetermined range with respect to the moving direction, a correction is made to direct the arrow to the opposite side.

[0237] As shown in FIG. 43, in the synthesis of the corrected arrow into the VR video of the VR system (S3712), the VR display control unit acquires the correction information on the direction of the arrow from the VR control unit (S4301). Then, the VR display control unit acquires the graphics data of the arrow from the graphics generation unit (S4302). And then, the VR display control unit rotates the graphics of the arrow according to the correction information on the direction of the arrow, and synthesizes it into the VR video according to the display position acquired from the graphics generation means to end the process (S4303).

[0238] FIG. 44 is a diagram showing another configuration example of the metadata according to the embodiment. For example, when sending graphics data and still images, the metadata as shown in the figure is configured. FIG. 45 is a diagram showing still another configuration example of the metadata according to the embodiment. For example, when a part of the functions are realized in the cloud and are sent from the VR system to the cloud computer, the metadata as shown in the figure is configured.

[0239] Here, returning to FIG. 26, another configuration of the embodiment of the present invention will be described. In FIG. 26, the embodiment describes the observation system of FIG. 27 and the VR system of FIG. 28 by integrating the two systems not as actual connections but as data and control flows in terms of functional blocks.

[0240] In the observation system, the VR shooting camera in Fig. 27 corresponds to the VR shooting means in Fig. 26. Similarly, the VR video processing unit is the VR video processing means, the VR video compression unit is the VR compression means, the microphone array, microphone terminal, microphone amplifier and ADC are the audio input means, the audio compression unit is the audio compression means, the movement input button is the data input means, the movement / position detection unit, the position / orientation detection unit and the two position / orientation analysis units of the GPU and CPU are the position / orientation detection means, the multiplexing unit is the multiplexing means, the wireless communication element is the communication means, the separation unit is the separation means, the audio decoding unit is the audio decoding means, the DAC, amplifier, headphone element and speaker correspond to the audio output means. Since the video bus, memory bus, system bus, I / O bus, bus conversion, RAM, EEPROM, SD card, power switch, power control element, battery, display element, shooting mode selection button, zoom button, shooting start / end button are not directly related to the operation of the present invention, their illustration is omitted.

[0241] In the VR system, the communication element in Fig. 28 corresponds to the communication means in Fig. 26. Similarly, the separation unit is the separation means, the audio decoding unit is the audio decoding means, the audio playback control unit is the audio playback control means, the DAC, amplifier, speaker, headphone terminal are the audio output means, the VR video decoding unit is the VR video decoding means, the graphics generation unit is the graphics generation means, the position / orientation determination units in the CPU and GPU respectively are the position / orientation determination means, the movement / position sensor and the movement / position detection unit are the position detection means and rotation detection means, the movement / position detection processing unit and the VR control unit are the VR control means, the VR display control unit is the VR display control means, the video display processing unit, display element, lens are the VR video display means, the microphone, microphone amplifier, ADC are the audio input means, the audio compression unit is the audio compression means, the multiplexing unit is the multiplexing means. Since the video bus, memory bus, system bus, I / O bus, bus conversion, RAM, EEPROM, non-volatile memory, power switch, power control element, battery, volume button, AV output, AV input, USB are not directly related to the operation of the present invention or have been described as one system, their illustration is omitted. The wireless communication element is necessary for communication with the controller, but since the controller is omitted in Fig. 26, its illustration is omitted.

[0242] In the second conventional example, data input means and position / azimuth detection means of the observation system are added. Metadata based on the position / azimuth detected by the position / azimuth detection means is multiplexed by the multiplexing means and sent to the VR system via the communication means.

[0243] In the position / azimuth detection means, the movement of the observation system itself, a guide, or a target may be detected from the temporal change of the position / azimuth. Alternatively, the operator of the observation system may input the movement direction or timing by the data input means. Alternatively, movement, movement direction, or timing may be detected from video information or audio information from the VR video processing means or audio input means, or guidance or input from the operator may be detected (broken line).

[0244] In the VR system, position / azimuth judgment means is added, and functions are added to the VR control means, VR display control means, and graphics generation means. The communication means receives communication information from the observation system, separates the metadata by the separation means, and sends it to the position / azimuth judgment means.

[0245] In the position / azimuth judgment means, the position / azimuth of the observation system or a guide or target is received as metadata, sent to the graphics generation means to be displayed as a graphic, and superimposed on the VR video by the VR display means; or sent to the VR control means, and the VR video is appropriately processed by the VR display control means together with the position / azimuth state of the VR system and displayed by the VR display means; or a guide voice is generated by the voice playback control means or the playback voice is appropriately processed, etc.

[0246] Each means of the observation system and the VR system may be realized by hardware, but may also be realized by software and added and implemented as an application to an existing observation system or VR system.

[0247] Furthermore, part or all of the position and orientation determination means on the VR system side and the control of VR video and audio may also be realized by a computer system between the VR system such as the cloud and the observation system. In this case, effects such as performing the same processing at one location and easily giving the same effect to multiple VR systems, and easily giving the effect of the present invention to an existing system can be expected.

[0248] Although not shown here, it is possible to convey movement by vibrating a control means (controller) held by the operator of the VR system, etc. Also, when there are multiple controllers, by selecting one and vibrating it, the direction of movement can be given, and the speed of movement can be given by the intensity of the vibration. Furthermore, when the controller is held in the hand and rotated around the body, when it is almost in line with the moving direction, by increasing the intensity of the vibration, a finer direction can be notified to the user of the VR system.

[0249] More specifically, two controllers vibrating simultaneously once indicates the direction. Then, depending on the direction, one of the left and right controllers vibrates to indicate right or left, two vibrating strongly simultaneously indicates forward, and two vibrating weakly simultaneously indicates backward. However, the vibration pattern only needs to have the same effect and is not limited to this.

[0250] Regarding the moving speed, it is possible to convey it graphically, such as presenting it by the color and size of an arrow, or convey it by the volume of the sound.

[0251] FIG. 46 is a diagram for explaining a configuration example of realizing the video display system according to the embodiment using a cloud. As shown in FIG. 46, the position / azimuth detection means 4640 of the observation system may be realized by a computer system 4631 between the observation system 4661 such as a cloud and the VR system 4601. In this case, either metadata indicating a direction is not sent from the observation system 4661, or data input by an operator is sent as metadata. For example, in the position / azimuth detection means 4640 in the cloud, the position, azimuth, or movement of the observation system 4661, the guide, or the target is detected from the video, audio, or metadata sent from the observation system 4661 and sent to the VR system 4601 as metadata. Thereby, the effects of this embodiment can be exhibited even with an existing 360° camera.

[0252] Furthermore, the position / azimuth determination means on the VR system side and the control of VR video and audio thereby may also be realized by a computer system 4631 between the VR system 4601 such as a cloud and the observation system 4661. In this case, the same processing is performed at one location, and effects such as easily giving the same effect to a plurality of VR systems 4601 simultaneously and being able to give the effects of the present invention to an existing system can be expected. However, in order to reflect the direction and position of the VR system 4601, it is necessary to send the position and direction of the VR system 4601 from the VR system 4601 to the cloud side, and it is necessary to provide a processing unit corresponding to each VR system 4601 on the cloud side.

[0253] The configuration of FIG. 46 is an example in the case where the position and azimuth of the VR system 4601 are not sent to the cloud side. In this case, it becomes difficult to display an arrow, change the audio, etc. according to the position and azimuth of the VR system 4601. However, with the VR display control means 4608, it is possible to perform processing such as changing the resolution of the VR video, masking, and changing the sound localization according to the output of the position / azimuth detection means 4640.

[0254] Note that for configurations not described above, the descriptions here are omitted by referring to the descriptions of the configurations with the same names in FIG. 26. Each of the position detection means 4602, rotation detection means 4603, VR display means 4604, audio reproduction means 4605, audio input means 4606, VR control means 4607, VR display control means 4608, audio decoding means 4609, audio compression means 4610, VR video decoding means 4611, separation means 4612, multiplexing means 4613, and communication means 4614 provided in the VR system 4601, and the VR video compression means 4633, multiplexing means 4634, communication means 4635, graphics generation means 4636, VR display control means 4637, VR video decompression means 4638, position / orientation detection means 4640, communication means 4641, and separation means 4642 provided in the computer system 4631, and the data input means 4662, multiplexing means 4663, communication means 4664, separation means 4665, VR video compression means 4666, audio compression means 4667, audio decoding means 4668, VR video processing means 4669, VR shooting means 4670, audio input means 4671, and audio output means 4672 provided in the observation system 4661 respectively correspond to each of the position detection means 702, rotation detection means 703, VR display means 704, audio reproduction means 705, audio input means 706, VR control means 707, VR display control means 708, audio reproduction control means 709, VR video decoding means 710, position / orientation determination means 2602, graphics generation means 712, audio decoding means 713, audio compression means 714, separation means 715, communication means 716, multiplexing means 717, communication means 754, separation means 755, VR video compression means 756, multiplexing means 757, VR video processing means 758, graphics generation means 759, audio compression means 760, audio decoding means 761, VR shooting means 762, audio input means 763, audio output means 764, data input means 2652, and position / orientation determination means 2653 in a one-to-one, many-to-one, one-to-many, or many-to-many manner.

[0255] FIG. 47 is a diagram for explaining a configuration example of realizing the video display system according to the embodiment using a cloud. In the configuration of FIG. 46, it was difficult to display an arrow or change the voice according to the position and orientation of the VR system. However, in the configuration shown in FIG. 47, by providing the VR system with position / orientation determination means, the position and orientation of the observation system on the cloud are read from the metadata separated by the separation means from the data sent from the observation system. Accordingly, graphics such as an image of a helicopter and an arrow are generated by the graphics generation means, and this is converted into metadata together with the position / orientation information etc. sent from the observation system by the metadata conversion means, multiplexed by the multiplexing means, and sent to the VR system.

[0256] In the VR system, graphics are generated from the metadata separated by the separation means, and the position and orientation of the observation system and the position and direction of the VR system obtained from the position detection means and the rotation detection means are determined by the VR control means, and appropriately synthesized with the VR video and the graphics by the VR display control means, or the VR video is processed. Also, by changing the sound localization or changing the content of the voice by the voice playback control means, etc., it is possible to output a display and voice suitable for the position and orientation of the VR system. Although not shown here, it is possible to appropriately control the controller of the VR system and notify the user of the VR system of the direction and position by vibration etc.

[0257] Note that for configurations not described above, the descriptions here are omitted by referring to the descriptions of configurations with the same names in FIG. 26. Each of the position detection means 4702, rotation detection means 4703, VR display means 4704, audio playback means 4705, audio input means 4706, VR control means 4707, VR display control means 4708, audio decoding means 4709, audio compression means 4710, VR video decoding means 4711, separation means 4712, multiplexing means 4713, and communication means 4714, position / orientation determination means 4715, graphics generation means 4716, audio playback control means 4717 included in the VR system 4701, the multiplexing means 4734, communication means 4735, graphics generation means 4736, VR display control means 4737, position / orientation detection means 4740, communication means 4741, and separation means 4742 included in the computer system 4731, and the data input means 4762, multiplexing means 4763, communication means 4764, separation means 4765, VR video compression means 4766, audio compression means 4767, audio decoding means 4768, VR video processing means 4769, VR shooting means 4770, audio input means 4771, and audio output means 4772 included in the observation system 4761 respectively corresponds to each of the position detection means 702, rotation detection means 703, VR display means 704, audio playback means 705, audio input means 706, VR control means 707, VR display control means 708, audio playback control means 709, VR video decoding means 710, position / orientation determination means 2602, graphics generation means 712, audio decoding means 713, audio compression means 714, separation means 715, communication means 716, multiplexing means 717, communication means 754, separation means 755, VR video compression means 756, multiplexing means 757, VR video processing means 758, graphics generation means 759, audio compression means 760, audio decoding means 761, VR shooting means 762, audio input means 763, audio output means 764, data input means 2652, and position / orientation determination means 2653 in a one-to-one, many-to-one, one-to-many, or many-to-many manner.

[0258] FIG. 48 is a diagram for explaining a configuration example of realizing the video display system according to the embodiment using a cloud. In the configuration of FIG. 47, the position and orientation of the observation system and the position and orientation of the VR system are used to control graphics, VR videos, audio, and the vibration of the controller in the VR system to provide appropriate information to the user of the VR system. However, additional functions are required for the VR system. In the configuration shown in FIG. 48, by having these functions on the cloud, the same effect can be achieved even in a simple VR system.

[0259] The position and orientation information of the VR system detected by the position detection means and rotation detection means of the VR system is used as metadata, multiplexed with other information in the multiplexing means, and sent to the computer system on the cloud by the communication means. This function is generally provided in a general VR system.

[0260] In the cloud, processing substantially the same as the configuration in FIG. 47 is performed, and the changed VR video and audio are sent to the VR system. Although the additional means for the configuration in FIG. 47 need to be provided in each of the connected VR systems, since these are realized by software on the cloud, the realization is easy. It is also common to vibrate the controller of the VR system, and the difficulty is small.

[0261] Note that for the configurations not described above, the explanations here are omitted by referring to the explanations in the configurations with the same names in FIG. 26. Each of the position detection means 4802, rotation detection means 4803, VR display means 4804, audio reproduction means 4805, audio input means 4806, VR control means 4807, VR display control means 4808, audio decoding means 4809, audio compression means 4810, VR video decoding means 4811, separation means 4812, multiplexing means 4813, and communication means 4814 provided in the VR system 4801, and the separation means 4832, VR video compression means 4833, multiplexing means 4834, communication means 4835, graphics generation means 4836, VR display control means 4837, VR video decompression means 4838, audio reproduction control means 4839, position / azimuth detection means 4840, communication means 4841, and separation means 4842 provided in the computer system 4831, and the data input means 4862, multiplexing means 4863, communication means 4864, separation means 4865, VR video compression means 4866, audio compression means 4867, audio decoding means 4868, VR video processing means 4869, VR shooting means 4870, audio input means 4871, and audio output means 4872 provided in the observation system 4861 respectively correspond to the position detection means 702, rotation detection means 703, VR display means 704, audio reproduction means 705, audio input means 706, VR control means 707, VR display control means 708, audio reproduction control means 709, VR video decoding means 710, position / azimuth determination means 2602, graphics generation means 712, audio decoding means 713, audio compression means 714, separation means 715, communication means 716, multiplexing means 717, communication means 754, separation means 755, VR video compression means 756, multiplexing means 757, VR video processing means 758, graphics generation means 759, audio compression means 760, audio decoding means 761, VR shooting means 762, audio input means 763, audio output means 764, data input means 2652, and position / azimuth determination means 2653 in a one-to-one, many-to-one, one-to-many, or many-to-many manner.

[0262] (Other Embodiments) As described above with respect to the embodiments and the like, the present disclosure is not limited to the above-described embodiments and the like.

[0263] Also, although the components constituting the video display system have been exemplified in the above embodiments and the like, the functions of the components included in the video display system may be distributed in any manner among a plurality of parts constituting the video display system.

[0264] Further, in the above embodiment, each component may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory.

[0265] Also, each component may be realized by hardware. For example, each component may be a circuit (or an integrated circuit). These circuits may constitute one circuit as a whole, or may be separate circuits respectively. Further, these circuits may be general-purpose circuits or dedicated circuits respectively.

[0266] Also, the general or specific aspects of the present disclosure may be realized by a system, an apparatus, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM. Further, they may be realized by any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0267] In addition, forms obtained by applying various modifications conceivable by those skilled in the art to the embodiments and the like, or forms realized by arbitrarily combining the components and functions in the embodiments and the like without departing from the spirit of the present disclosure are also included in the present disclosure.

Industrial Applicability

[0268] The present disclosure is useful in applications for displaying appropriate video on a display device.

Explanation of Signs

[0269] 99 Graphic 99a Arrow 99b mask 100 display device 101 display unit 102 orientation detection unit 150 network 200 server device 201 receiving unit 202 difference calculation unit 203 presentation unit 204 video generation unit 300 observation device 300a, 300b imaging device 301 imaging unit 302 input interface 303 position detection unit 304 data acquisition unit 305 metadata acquisition unit 306 transmission unit 500 video display system

Claims

1. A video display system for displaying a display video by a display device, comprising: a photographing unit that photographs a video as a wide-angle video; a data acquisition unit that acquires data regarding the moving direction of the photographing unit; a metadata configuration unit that acquires metadata based on the acquired data; and a transmission unit that transmits the photographed wide-angle video together with the metadata, an observation device; a VR device, a receiving unit that receives the wide-angle video and the metadata; an orientation estimation unit that estimates the orientation of the display device; a difference calculation unit that calculates a relative movement direction, which is the movement direction of the photographing unit relative to the orientation of the display device, based on a difference between the estimated orientation of the display device and the movement direction of the photographing unit in the metadata; a presentation unit that presents the calculated relative movement direction to a user of the display device; a video generation unit that generates the display video including a part of the video corresponding to a visual field portion corresponding to the orientation of the display device estimated by the orientation estimation unit from the received wide-angle video; a VR device including the display device that displays the display video, and a video display system.

2. The presentation unit generates and outputs graphics indicating the calculated relative movement direction, and superimposes the output graphics on the part of the video to cause the video generation unit to present the relative movement direction. The video display system according to claim 1.

3. The graphics display an arrow indicating the relative movement direction on the display video. The video display system according to claim 2.

4. The graphics display a mask, which is an image for covering at least a part other than the relative movement direction side on the display video. The video display system according to claim 2.

5. The estimated orientation of the display device is a discrete display direction that changes by a first discrete value based on the actual orientation of the display device, and the difference calculation unit calculates a difference between the discrete display direction and the movement direction of the photographing unit in the metadata. The video display system according to claim 1.

6. The first discrete value becomes minimum when the inclination in the horizontal plane of the actual orientation of the display device and the movement direction of the photographing unit in the metadata coincides, and changes so as to become maximum when the difference between the inclination in the horizontal plane of the actual orientation of the display device and the movement direction of the photographing unit in the metadata becomes maximum. The video display system according to claim 5.

7. The moving direction of the imaging unit on the metadata is a discrete moving direction that changes by a second discrete value based on the actual orientation of the display device, The difference calculation unit calculates the difference between the estimated orientation of the display device and the discrete moving direction. The video display system according to claim 1.

8. The observation device has an input interface for an operator to input for moving the imaging unit, The data acquisition unit acquires the data input by the operator via the input interface. The video display system according to claim 1.

9. The observation device has a position detection unit for detecting the position of the imaging unit, The data acquisition unit acquires the data based on the position of the imaging unit detected over time by the position detection unit. The video display system according to claim 1.

10. The imaging unit captures the virtual wide-angle video by capturing in a virtual image space constituted by computer graphics. The video display system according to claim 1.

11. A video display system for displaying a display video by a display device, An observation device, An information processing device connected to the observation device via a network, A VR device having a display device for displaying the display video connected to the information processing device via a network and an orientation estimation unit for estimating the orientation of the display device, The observation device has an imaging unit for capturing a video as a wide-angle video and a data acquisition unit for acquiring data regarding the moving direction of the imaging unit, The information processing device, A receiving unit for receiving the wide-angle video and the data from the observation device, A metadata configuration unit for generating metadata based on the wide-angle video and the data, A movement information calculation unit for calculating movement information regarding the movement of the imaging unit on the metadata, A graphics indicating the calculated movement information, which is superimposed on a part of the video corresponding to the visual field portion corresponding to the estimated orientation of the display device among the wide-angle videos, so as to present the moving direction of the imaging unit to the user of the display device, and a presentation unit for generating and outputting the graphics, A transmission unit for transmitting the wide-angle video, the graphics, and the metadata, The VR device, A receiving unit that receives the wide-angle video, the graphics, and the metadata; A video generation unit that generates the display video including a part of the video corresponding to a visual field portion according to the estimated orientation of the display device from the received wide-angle video, and the graphics superimposed on the part of the video. A video display system.

12. A video display system for displaying a display video by a display device, An observation device; An information processing device connected to the observation device via a network; A VR device having a display device that displays the display video, which is connected to the information processing device via a network, and an orientation estimation unit that estimates the orientation of the display device. The observation device includes a photographing unit that photographs a video as a wide-angle video, and a data acquisition unit that acquires data related to a moving direction of the photographing unit. The information processing device A receiving unit that receives the wide-angle video and the data from the observation device; A metadata configuration unit that generates metadata based on the wide-angle video and the data; A movement information calculation unit that calculates movement information related to the movement of the photographing unit on the metadata; A transmission unit that transmits the wide-angle video, the movement information, and the metadata. The VR device A receiving unit that receives the wide-angle video, the movement information, and the metadata; A difference calculation unit that calculates a relative movement direction, which is a movement direction of the photographing unit relative to the orientation of the display device, based on a difference between the estimated orientation of the display device and the movement direction of the photographing unit in the movement information; A presentation unit that presents the calculated relative movement direction to a user of the display device; A video generation unit that further includes a display video including a part of the video corresponding to a visual field portion according to the orientation of the display device estimated by the orientation estimation unit from the received wide-angle video. A video display system.

13. A video display system for displaying a display video by a display device, An observation device; An information processing device connected to the observation device via a network; A VR device having a display device that displays the display video, which is connected to the information processing device via a network, and an orientation estimation unit that estimates the orientation of the display device. The observation device includes a photographing unit that photographs an image as a wide-angle-of-view image, and a data acquisition unit that acquires data regarding the moving direction of the photographing unit. The information processing device includes a receiving unit that receives the wide-angle-of-view image, the data, and data regarding the orientation of the display device, a metadata configuring unit that generates metadata based on the wide-angle-of-view image, the data, and the data regarding the orientation of the display device, a difference calculation unit that calculates a relative movement direction, which is the moving direction of the photographing unit relative to the orientation of the display device, based on the difference between the orientation of the display device and movement information regarding the movement of the photographing unit, a presentation unit that generates and outputs graphics indicating the calculated relative movement direction, and superimposes the graphics on a part of the image corresponding to the visual field portion according to the estimated orientation of the display device among the wide-angle-of-view image, so as to present the relative movement direction to the user of the display device, and a transmitting unit that transmits the wide-angle-of-view image, the graphics, and the metadata. The VR device includes a receiving unit that receives the wide-angle-of-view image, the graphics, and the metadata, and an image generation unit that generates a display image including a part of the image corresponding to the visual field portion according to the orientation of the display device estimated by the orientation estimation unit from the received wide-angle-of-view image, and the graphics superimposed on the part of the image. An image display system.

14. The information processing device is provided on a cloud connected to a wide area network, and is connected to the observation device and the VR device via the wide area network. The image display system according to any one of claims 11 to 13.

15. An information processing method for causing a display device to display a display image, the method comprising: receiving metadata based on data regarding the moving direction of a photographing unit that photographs an image as a wide-angle-of-view image; and calculating, based on the difference between the estimated orientation of the display device and the moving direction of the photographing unit in the metadata, a relative movement direction, which is the moving direction of the photographing unit relative to the orientation of the display device, and outputting the relative movement direction to the display device. An information processing method.

Images