Live video distribution method using unmanned mobile body, video distribution device used in the live video distribution method, and video archive device for storing video data file generated by the video distribution device

Abstract

To provide a live video distribution method for allowing even a user having no skills and qualifications for operating an unmanned mobile body including an unmanned aircraft to enjoy a video of a live video signal transmitted from the unmanned mobile body moving near an object to be photographed.SOLUTION: In the live video distribution method for simultaneously transmitting the live video signal obtained by photographing the object to be photographed to a user device used by each of a plurality of users from a video distribution device and displaying the video of the live video signal on display means included in the user device, the live video signal is generated on the basis of a result of photographing the object to be photographed by imaging means included in the unmanned mobile body moving in a space near the object to be photographed and the unmanned mobile body is configured so as to move on the basis of a movement control signal generated on the basis of a result of operating an operator device by an operator other than the plurality of users.SELECTED DRAWING: Figure 8

Description

Technical Field

[0001] The present invention relates to a live video distribution method, and more particularly to a live video distribution method for simultaneously transmitting a live video signal generated based on the result of shooting a shooting target by an imaging means provided in an unmanned moving body to a plurality of user devices. The present invention also relates to a video distribution device used in the live video distribution method. Further, the present invention relates to a video archive device for storing a video data file generated by the video distribution device.

Background Art

[0002] The tourism industry currently accounts for about 10% of the world's GDP and is predicted to become "the largest industry in the 21st century" in the future. In such a trend, there is an increasing need for tourism in frontier spaces such as uncharted areas, and even outer space and the deep sea, as tourists are tired of traditional tourism destinations.

[0003] However, when a large number of tourists flock to uncharted areas, there is a problem that rare natural environments and precious relics are damaged. In addition, tourism in frontier spaces such as outer space and the deep sea not only requires a huge cost to travel to the site, but also poses an excessive burden on the body due to the influence of the surrounding environment such as zero gravity and high atmospheric pressure, and there is a risk that life may be endangered if the means of transportation to the site (such as a spaceship or a deep-sea research ship) is damaged.

[0004] In addition, not only in uncharted areas and frontier spaces, but also in general tourism, it takes time and cost to travel to tourist destinations. In particular, for people with physical handicaps, it is difficult to travel to tourist destinations themselves.

[0005] As a method for solving the above problems, an aerial unmanned aircraft is flown over an area such as a tourist destination, and a live video signal generated based on the result of imaging means provided in the aerial unmanned aircraft imaging a shooting target in the area is transmitted to a user device used by a user located at a location remote from the tourist destination, so that the user can experience a pseudo-tour without visiting the tourist destination. A live video distribution method has been proposed.

[0006] However, in the conventional live video distribution method, since a user in a remote location remotely operates the aerial unmanned aircraft by operating the user device at hand relying only on the video of the live video signal transmitted from the aerial unmanned aircraft without directly visually recognizing the aerial unmanned aircraft, a certain level of skill or more for remotely operating the unmanned aircraft is required of the user. In particular, when the area where the aerial unmanned aircraft flies is an area where the flight of the unmanned aircraft is restricted, an even higher skill and a certain level of qualification or more for remotely operating the unmanned aircraft are required of the user. For this reason, there has been a problem that users who do not have the skills and qualifications for remotely operating such an unmanned aircraft cannot enjoy the pseudo-tour as described above.

[0007] Also, in the conventional live video distribution method, when each user is located at a location remote from each other, they can only enjoy the local video and audio alone, and cannot receive explanations from a guide with rich knowledge about the tourist destination like in a group sightseeing tour, or exchange impressions with each other. Therefore, there has been a problem that the enjoyment of the "tourism experience" is not sufficiently enhanced.

[0008] Also, in the conventional live video distribution method, there has been a problem that the video taken by the aerial unmanned aircraft is only subjected to a one-time live distribution, and the video taken with great effort cannot be enjoyed afterwards.

Prior Art Documents

Patent Documents

[0009]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0010] The present invention has been made in view of such circumstances, and even a user who does not have the skills and qualifications to operate unmanned moving bodies such as unmanned aerial vehicles can enjoy the video of the live video signal transmitted from the unmanned moving body moving in the space near the shooting target. An object of the present invention is to provide a live video distribution method.

[0011] Another object of the present invention is to provide a live video distribution method in which a plurality of users located at remote locations can simultaneously observe a target existing at a tourist destination or the like in real time, receive explanations from a guide, and have a sense of presence as if they were together at a tourist destination or the like. It is an object of the present invention to provide a live video distribution method that enables conversation and dialogue.

[0012] Another object of the present invention is to provide a video distribution device and a video archive device that enable users who could not receive the live distribution to enjoy the video captured by the imaging means of the unmanned moving body even after the fact.

Means for Solving the Problems

[0013] To achieve the above object, a first aspect of the present invention relating to a live video distribution method is A live video distribution method in which a live video signal obtained by shooting a shooting target is simultaneously transmitted from a video distribution device to user devices used by each of a plurality of users, and the video of the live video signal is displayed on a display means provided in the user devices, The live video signal is generated based on the result of shooting the shooting target by an imaging means provided in an unmanned moving body moving in the space near the shooting target, The unmanned moving body moves based on a movement control signal generated based on the result of an operator other than the plurality of users operating an operator device. It is configured as follows.

[0014] In the first aspect of the present invention, as the unmanned moving body, an unmanned vehicle or a walking robot that moves on the ground near the imaging target, an unmanned aircraft that flies in the air near the imaging target, etc. can be used.

[0015] Further, as the display means provided in the user device, a flat display panel such as a liquid crystal display or an organic EL (ElectroLuminescence) display, or a head mounted display (hereinafter abbreviated as "HMD") can be used.

[0016] Furthermore, as the imaging means provided in the unmanned moving body, by using a compound eye camera having a plurality of imaging units, etc., the live video signal generated and transmitted by the unmanned moving body can be made into a three-dimensional (hereinafter abbreviated as "3D") video signal. In that case, while using a stereoscopic HMD as the display means provided in the user device, and by providing the user device with a "sensor that detects the movement of the user's head and eyes", it is possible to configure it to project "a video of a visual field range adapted to the movement of the user's head and eyes" from the left and right projection units of the stereoscopic HMD. As a result, it is possible to provide the user with an immersive visual experience (hereinafter, unless otherwise specified, when "HMD" is described, it means "stereoscopic HMD", and when the display means of the user device is an HMD, it is assumed that a 3D video signal that transmits both the video projected from the left projection unit of the HMD and the video projected from the right projection unit is transmitted to the user device).

[0017] Also, in order to achieve the above object, a second aspect of the present invention related to a live video distribution method is In the live video distribution method of the first aspect, All or some of the plurality of users are located at a place remote from the unmanned moving body, The video distribution device receives a live video signal from the unmanned moving body, performs necessary processing on the live video signal, and then transmits it to the user device used by the user located at the remote place via a public network. configured as follows.

[0018] In the first or second aspect of the present invention, the video distribution device may be configured to directly receive a live video signal from an unmanned mobile body, or may receive the signal via a relay device such as a Wi-Fi router and / or a public network.

[0019] In the second aspect of the present invention, the Internet can be used as the public network. When the live video signal is transmitted via the Internet, a video distribution server of a general-purpose platform such as YouTube (registered trademark), Instagram (registered trademark), or Vimeo (registered trademark) can be used as (a part of) the video distribution device. In this case, the live video signal will be transmitted from the unmanned mobile body to the user device via either of the following paths (1) or (2). (1) Unmanned mobile body → Video distribution device 1 → Internet → Video distribution device 2 (video distribution server of a general-purpose platform) → Internet → User device (2) Unmanned mobile body → Internet → Video distribution device (video distribution server of a general-purpose platform) → Internet → User device Note that communication between each device and between each device and the Internet can be appropriately performed via a relay device such as a Wi-Fi router.

[0020] In the second aspect of the present invention, a network of a mobile communication system can be used as a part of the public network. In that case, the video distribution device and / or the user device perform wireless communication with a mobile communication base station. In particular, by using the network of the fifth-generation mobile communication system (hereinafter abbreviated as "5G"), a high-resolution live video signal can be transmitted to a large number of user devices simultaneously with low latency.

[0021] Also, in order to achieve the above object, a third aspect of the present invention relating to a live video distribution method is In the live video distribution method according to the first or second aspect, the operator operates the operator device while directly visually recognizing the unmanned mobile body. It is configured as follows.

[0022] Also, in order to achieve the above object, a fourth aspect of the present invention relating to a live video distribution method is In the live video distribution method according to the first or second aspect, the operator device directly receives, or receives via a video distribution device and / or a public network, the live video signal generated and transmitted by the unmanned mobile body, and displays the video of the live video signal on the display means provided in the operator device. The operator operates the operator device while viewing the video of the live video signal displayed on the display means provided in the operator device. It is configured as follows.

[0023] In the fourth aspect of the present invention, as the display means provided in the operator device, a flat display panel or an HMD can be used. In particular, when using an HMD, it can be configured to be equipped with a sensor for detecting the movement of the user's head and eyes, and project "a video within a visual field range adapted to the movement of the user's head and eyes" from the left and right projection parts of the HMD.

[0024] Also, in the fourth aspect of the present invention, when the operator device receives a live video signal via a public network, a network of a mobile communication system can be used as a part of the public network. And in that case, the operator device performs wireless communication with a mobile communication base station. At that time, in particular, by using a 5G network, a high-resolution live video signal can be transmitted to the operator device with low latency.

[0025] Also, in order to achieve the above object, a fifth aspect of the present invention relating to a live video distribution method is In the live video distribution method according to any one of the first to fourth modes, the operator is located at a place remote from the unmanned moving body, the video distribution device receives a movement control signal from the operator device, performs necessary processing on the movement control signal, and then transmits it to the unmanned moving body via a public network. It is configured as described above.

[0026] In the fifth mode of the present invention, the network of the mobile communication system can be used as part of the public network. And in that case, the video distribution device and / or the operator device perform wireless communication with the mobile communication base station. In particular, by using the 5G network, the movement control signal can be transmitted to the video distribution device with low latency.

[0027] Also, in order to achieve the above object, the sixth mode of the present invention related to the live video distribution method is In the live video distribution method according to any one of the first to fifth modes, there are a plurality of unmanned moving bodies, the plurality of unmanned moving bodies simultaneously photograph a substantially same photographing object from different viewpoints by photographing means provided respectively, and "a plurality of live video signals with different display ranges" generated based on the results are simultaneously transmitted from the video distribution device to the user device. It is configured as described above.

[0028] Note that the meanings of "viewpoint", "line of sight", "imaging range", and "viewpoint (center of the imaging range)" of the imaging means as used in this specification and the like are as shown in FIG. 30.

[0029] Also, in the sixth aspect of the present invention, each of the user devices can be configured to receive only one live video signal out of the plurality of live video signals with different display ranges. Also, each of the user devices can be configured to receive a plurality of live video signals out of the plurality of live video signals with different display ranges simultaneously. And in the latter case, the display means of the user device can be configured to display the videos of the received plurality of live video signals in parallel, or to display only the video of one live video signal out of the received plurality of live video signals. And in the last case, a configuration can be adopted in which the user can switch the video of the live video signal to be displayed by operating the user device.

[0030] Also, in the sixth aspect of the present invention, one operator may operate one unmanned moving body, or one operator may operate a plurality of unmanned moving bodies. And in the latter case, by adding an autonomous movement function to the unmanned moving body, the risk of collision or fall can be reduced.

[0031] Also, to achieve the above object, the seventh aspect of the present invention relating to a live video distribution method is in the live video distribution method of the sixth aspect, each of the plurality of users selects "which unmanned moving body's imaging means among the plurality of unmanned moving bodies the live video signal generated based on the result of imaging by the user device they use receives", and the display means provided in the user device displays the video of the selected live video signal. configured as such.

[0032] Note that in the sixth aspect of the present invention, the user can be configured to select in advance before receiving the live video signal. Also, a configuration can be adopted in which during the reception of the live video signal generated based on the result of imaging by the imaging means of a certain unmanned moving body, the reception is switched to the live video signal generated based on the result of imaging by the imaging means of another unmanned moving body.

[0033] Also, in order to achieve the above object, an eighth aspect of the present invention relating to a live video distribution method is In the live video distribution method according to any one of the first to seventh aspects, There are a plurality of the unmanned moving bodies, The plurality of unmanned moving bodies are deployed in a plurality of regions remote from each other, Each of the plurality of users selects "which live video signal generated based on the result of shooting by the imaging means of the unmanned moving body deployed in which region among the plurality of regions is received by the user device used by the user itself", and the display means provided in the user device displays the video of the selected live video signal. It is configured as described above.

[0034] Also, in order to achieve the above object, a ninth aspect of the present invention relating to a live video distribution method is In the live video distribution method according to any one of the first to eighth aspects, The imaging means provided in the unmanned moving body shoots a wide-angle video including a panoramic video, an all-round video, and an omnidirectional video, The display means of the user device cuts out and displays a video within a predetermined display range (hereinafter abbreviated as "predetermined display range video") from the wide-angle video. It is configured as described above.

[0035] Here, the "panoramic video" means "a video that can be panoramically viewed 360 degrees in the horizontal direction", the "all-round video" is also called the "hemispherical video", and the "all-round video" means "a video with a viewing angle (solid angle) of 180 degrees that covers the upper half of the field of view on a dome screen like a planetarium". The "omnidirectional video" means "a video with a viewing angle (solid angle) of 360 degrees that covers the entire field of view from front to back, left to right, above the head to the feet". The "wide-angle video" means a video that spreads over a wider range than the normal field of view, such as a "panoramic video", an "all-round (hemispherical) video", and an "omnidirectional video".

[0036] In the ninth aspect of the present invention, as the display means provided in the user device, a flat display panel or an HMD can be used. Further, the extraction of the video within a predetermined display range from the wide-angle video may be performed either by the video distribution device or by the user device. In particular, when an HMD is used as the display means provided in the user device, the user device is provided with a "sensor for detecting the movement of the user's head and eyes", and can be performed by any of the following methods (1) and (2). (1) Transmit a wide-angle video signal from the video distribution device to the user device, and in the user device, based on "the result of the sensor detecting the movement of the user's head and eyes", cut out "the video within the visual field range adapted to the movement of the user's head and eyes", and project it from the left and right projection units of the HMD (2) Transmit a "signal based on the result of the sensor detecting the movement of the user's head and eyes" from the user device to the video distribution device, and in the video distribution device, generate a video signal for transmitting "the video within the visual field range adapted to the movement of the user's head and eyes" from the wide-angle video signal, and transmit the video signal to the user device. In the user device, project the video based on the received video signal from the left and right projection units of the HMD

[0037] Further, in order to achieve the above object, the tenth aspect of the present invention relating to the live video distribution method is in the live video distribution method according to any one of the first to ninth aspects, collect the live operator's voice signal by collecting the voice emitted by the operator using the operator device or a person near the operator, and transmit it from the video distribution device directly or via a public network to the plurality of user devices at the same time, and simultaneously emit the live voice by the sound emitting means of the plurality of user devices. It is configured as follows.

[0038] ​​In addition, in the tenth aspect of the present invention, the "sound collection means for collecting the sound emitted by the operator using the operator device or a person near the operator" can be configured as a component provided in the operator device, a component provided in the video distribution device, or a configuration provided independently of those devices. Further, as the sound collection means, a microphone can be used. In particular, in a configuration where the operator device includes the sound collection means, when it includes display means such as a flat display panel or an HMD, the microphone can be integrated with those display means.

[0039] Also, as the sound playback means provided in the user device, a microphone can be used. In particular, it can be configured to be integrated with display means such as a flat display panel or an HMD provided in the user device.

[0040] Also, in order to achieve the above object, an eleventh aspect of the present invention relating to a live video distribution method is In the live video distribution method of the tenth aspect, all or some of the plurality of user devices have sound collection means for collecting the sound emitted by the user using the user device, A superimposed audio signal obtained by superimposing all or part of the live user audio signal based on the result of sound collection by the sound collection means of the user device and the live operator etc. audio signal is transmitted from the video distribution device to the plurality of user devices simultaneously, either directly or via a public network, and the sound playback means of the plurality of user devices simultaneously plays back the live audio. It is configured as follows.

[0041] In the eleventh aspect of the present invention, for example, when there are four users, namely A to D, who use the user device, the video distribution device may be configured to generate the following four types of superimposed audio signals and transmit the superimposed audio signals A, B, C, and D to each of the users A, B, C, and D (hereinafter, the "live audio signal based on the result of sound collection by the sound collection means of the operator device", the "live audio signal based on the result of sound collection by the sound collection means of the user device used by user A", etc. are abbreviated as "operator audio signal", "user A audio signal", etc., respectively). Superimposed audio signal A... An audio signal obtained by superimposing "operator audio signal + user B audio signal + user C audio signal + user D audio signal" Superimposed audio signal B... An audio signal obtained by superimposing "operator audio signal + user A audio signal + user C audio signal + user D audio signal" Superimposed audio signal C... An audio signal obtained by superimposing "operator audio signal + user A audio signal + user B audio signal + user D audio signal" Superimposed audio signal D... An audio signal obtained by superimposing "operator audio signal + user A audio signal + user B audio signal + user C audio signal" With such a configuration, each of the users A to D can avoid the unpleasant feeling associated with "hearing the voice they themselves emit as external audio".

[0042] Also, in order to achieve the above object, a twelfth aspect of the present invention relating to a video distribution device used in a live video distribution method is A video distribution device used in the live video distribution method according to any one of the first to eleventh aspects, Receives a live video signal from the unmanned moving body, performs necessary processing on the live video signal, and then transmits it simultaneously to the plurality of user devices directly or via a public network. It is configured as described above.

[0043] Also, in order to achieve the above object, a 13th aspect of the present invention relating to a video distribution apparatus used in a live video distribution method is In the video distribution apparatus according to the 12th aspect, receiving a movement control signal from the operator device, performing necessary processing on the movement control signal, and then transmitting it directly or via a public network to the unmanned moving body. It is configured as described above.

[0044] Also, in order to achieve the above object, a 14th aspect of the present invention relating to a video distribution apparatus used in a live video distribution method is In the video distribution apparatus according to the 12th or 13th aspect, A video distribution apparatus used in the live video distribution method according to any one of the 4th to 11th aspects, receiving a live video signal from the unmanned moving body, performing necessary processing on the live video signal, and then transmitting it directly or via a public network to the operator device. It is configured as described above.

[0045] Also, in order to achieve the above object, a 15th aspect of the present invention relating to a video distribution apparatus used in a live video distribution method is In the video distribution apparatus according to the 12th or 14th aspect, A video distribution apparatus used in the live video distribution method according to any one of the 5th to 11th aspects, receiving a movement control signal from the operator device, performing necessary processing on the movement control signal, and then transmitting it via a public network to the unmanned moving body It is configured as described above.

[0046] Also, in order to achieve the above object, a 16th aspect of the present invention relating to a video distribution apparatus used in a live video distribution method is In the video distribution apparatus according to any one of the 12th to 15th aspects, A video distribution apparatus used in the live video distribution method according to any one of the 6th to 11th aspects, Receiving "live video signals with different imaging ranges" from each of the plurality of unmanned mobile bodies, performing necessary processing on each of the "live video signals with different imaging ranges", and then transmitting them simultaneously to the plurality of user devices directly or via a public network.

[0047] Also, in order to achieve the above object, a 17th aspect of the present invention related to a video distribution device used in a live video distribution method is In the video distribution device according to any one of the 12th to 16th aspects, A video distribution device used in the live video distribution method according to any one of the 7th to 11th aspects, Receiving an unmanned mobile body selection signal based on the selection result of the user using the user device from the user device, Based on the unmanned mobile body selection signal, determining to which user device to transmit the live video signals received from the plurality of unmanned mobile bodies. It is configured as described above.

[0048] Also, in order to achieve the above object, an 18th aspect of the present invention related to a video distribution device used in a live video distribution method is In the video distribution device according to any one of the 12th to 17th aspects, A video distribution device used in the live video distribution method according to any one of the 8th to 11th aspects, Receiving a region selection signal based on the selection result of the user using the user device from the user device, Based on the region selection signal, determining to which user device to transmit the live video signals received from the plurality of unmanned mobile bodies. It is configured as described above.

[0049] Also, in order to achieve the above object, a 19th aspect of the present invention related to a video distribution device used in a live video distribution method is In the video distribution device according to any one of the 12th to 18th aspects, A video distribution apparatus used in a live video distribution method according to any one of the ninth to eleventh aspects, receives a wide-angle video signal from the unmanned mobile body, performs necessary processing on the wide-angle video signal, and then transmits it directly or via a public network to the plurality of user devices simultaneously. It is configured as described above.

[0050] Also, in order to achieve the above object, a twentieth aspect of the present invention relating to a video distribution apparatus used in a live video distribution method is In the video distribution apparatus according to the nineteenth aspect, a predetermined range video signal for transmitting a video of a predetermined display range cut out from the wide-angle video is generated, and the predetermined range video signal is transmitted to the user device. It is configured as described above.

[0051] In the twentieth aspect of the present invention, the transmission of the predetermined display range video signal from the video distribution apparatus to the user device can be performed by any one of the following methods (1), (2), and (3). (1) The user device is equipped with a "sensor for detecting the movement of the user's head and eyes", and then a signal based on "the result detected by the sensor" is sent from the user device to the video distribution apparatus, and in the video distribution apparatus, based on the received signal, a "field of view range adapted to the movement of the user's head and eyes" is set, and a video signal of a display range corresponding to the field of view range is generated to generate the predetermined display range video signal, and the video signal is transmitted to the user device. (2) Based on a display range setting signal generated based on the result of the user's operation of the user device, the display range is determined, a video signal of the determined display range is generated, and the predetermined display range video signal is transmitted to the user device. (3) The display range is determined regardless of the movement of the user's head and eyes or the user's operation while using the user device, a video signal of the determined display range is generated, and the predetermined display range video signal is generated.​​​​ Transmit a signal to the user device In particular, when using an HMD as the display means provided in the user device, it is preferable to adopt the method of (1). In this case, the video distribution device transmits a 3D video signal composed of video signals projected from each of the left and right projection units of the HMD.

[0052] Further, in order to achieve the above object, a 21st aspect of the present invention related to a video distribution device used in a live video distribution method is In the video distribution device according to any one of the 12th to 20th aspects A video distribution device used in the live video distribution method according to the 10th or 11th aspect, The live operator voice signal is transmitted to the plurality of user devices simultaneously, either directly or via a public network. configured as such.

[0053] Further, in order to achieve the above object, a 22nd aspect of the present invention related to a video distribution device used in a live video distribution method is In the video distribution device according to the 21st aspect A video distribution device used in the live video distribution method according to the 11th aspect, Receiving a live user voice signal from all or part of the plurality of user devices, generating a superimposed voice signal by superimposing all or part of the live user voice signal and the live operator voice signal, and directly or via a public network Transmit the superimposed voice signal to the plurality of user devices simultaneously. configured as such.

[0054] Further, in order to achieve the above object, a 23rd aspect of the present invention related to a video distribution device used in a live video distribution method is In the video distribution device according to any one of the 12th to 22nd aspects Start recording the video data transmitted by the live video signal almost simultaneously with the start of transmission of the live video signal, and save the video data as a video data file almost simultaneously with the end of transmission of the live video signal. configured as described above.

[0055] Also, in order to achieve the above object, a 24th aspect of the present invention relating to a video archive device is saving the video data file generated by the video distribution device according to the 23rd aspect. configured as described above.

[0056] In the 24th aspect of the present invention, the video distribution device and the video archive device can be integrally configured.

Advantages of the Invention

[0057] According to the live video distribution method of the present invention and the video distribution device used in the live video distribution method of this aspect, only the operator who uses the operator's device operates the unmanned moving body, and other users can enjoy the video of the live video signal based on the result of the imaging means of the unmanned moving body capturing the imaging target without operating the unmanned moving body. Therefore, even users who do not have the skills or qualifications to remotely control an unmanned moving body can enjoy the video of the live video signal transmitted from the unmanned moving body moving within the space near the imaging target.

[0058] Furthermore, according to the live video distribution method of the present invention and the video distribution device used in the live video distribution method of this aspect, since a plurality of users enjoy "the video of the live video signal based on the result of the imaging means of the unmanned moving body remotely controlled by one operator capturing the imaging target", compared with the conventional live video distribution method in which each user remotely controls an unmanned moving body, the number of unmanned moving bodies required is less, which is economical.

[0059] Moreover, particularly according to the live video distribution method of the second aspect of the present invention and the video distribution apparatus used in the live video distribution method of this aspect, since the transmission of the live video signal from the unmanned mobile body to the user device is performed via a public network such as the Internet, the user can enjoy the video of the live video signal captured by the unmanned mobile body even when in a place where the wireless signal from the unmanned mobile body does not directly reach. For this reason, for example, the user can enjoy the video of the live video signal of the autumn foliage in Arashiyama, Kyoto while staying in Tokyo, or enjoy the video of the live video signal of wild animals in Africa.

[0060] Moreover, particularly according to the live video distribution method of the third aspect of the present invention and the video distribution apparatus used in the live video distribution method of this aspect, since the operator operates the operator device while directly visually recognizing the unmanned mobile body, the risk that the unmanned mobile body collides with the imaging target or another unmanned mobile body and the imaging target or the unmanned mobile bodies of itself and others are damaged can be minimized. Therefore, it is suitable for the case of distributing the video of the live video signal captured of a precious relic or the video of the live video signal captured of precious exhibits in a museum or art museum.

[0061] Moreover, particularly according to the live video distribution method of the fourth aspect of the present invention and the video distribution apparatus used in the live video distribution method of this aspect, the operator causes the video of the live video signal captured by the unmanned mobile body to be displayed on the display means provided in the operator device used by the operator himself / herself, and operates the operator device while watching the video of the live video signal, so that the unmanned mobile body can be moved to a place where it cannot be sufficiently visually recognized with the naked eye.

[0062] Moreover, particularly according to the live video distribution method of the fifth aspect of the present invention and the video distribution apparatus used in the live video distribution method of this aspect, since the transmission of the movement control signal from the operator device to the unmanned moving body is performed via a public network such as the Internet, the operator can operate the unmanned moving body even from a remote location where a radio signal cannot directly reach the unmanned moving body. For this reason, for example, the operator can fly an unmanned aircraft over Arashiyama in Kyoto or over Africa while staying in Tokyo, and on the other hand, the user can enjoy the video of the live video signal of autumn foliage and wild animals while staying in New York.

[0063] Moreover, particularly according to the live video distribution method of the sixth aspect of the present invention and the video distribution apparatus used in the live video distribution method of this aspect, for example, by simultaneously photographing the launched fireworks at a certain time with the imaging means provided in a plurality of unmanned moving bodies located at different positions, the user can enjoy the launched fireworks as "video seen from above", "video seen from below", "video seen from the side", etc., simultaneously or while switching.

[0064] Moreover, particularly according to the live video distribution method of the seventh aspect of the present invention and the video distribution apparatus used in the live video distribution method of this aspect, the user can, for example, select a suitable unmanned moving body for himself / herself based on the profiles (gender, age, experience, etc.) of each operator who operates each of the plurality of unmanned moving bodies and the movement plans (movement routes and movement times) preset for each of the plurality of unmanned moving bodies, and enjoy the video of the live video signal photographed by the selected unmanned moving body.

[0065] Moreover, particularly according to the live video distribution method of the eighth aspect of the present invention and the video distribution apparatus used in the live video distribution method of this aspect, by deploying unmanned mobile bodies at various tourist destinations at home and abroad, users can freely select a tourist destination and enjoy the live video of that tourist destination. Therefore, for example, while staying in Tokyo, a user can enjoy the live video of the Parthenon in Greece and then enjoy the live video of the pyramids in Egypt.

[0066] Furthermore, particularly according to the live video distribution method of the ninth aspect of the present invention and the video distribution apparatus used in the live video distribution method of this aspect, by the user moving their head or eyes or operating the user device, it becomes possible to cut out and display "video within a visual field range adapted to the movement of the user's head or eyes" or to display "video within a display range that the user wants to display on the display means of the user device".

[0067] Among these, particularly according to the video distribution apparatus of the twentieth aspect of the present invention, it is possible to switch and transmit a wide-angle video signal and a video signal within a predetermined range according to the user device. For example, for a user device with a display means being a flat display, a video signal within a predetermined range can be transmitted, and for a user device with a display means being an HMD, an omnidirectional 3D video signal or a full-surround 3D video signal can be transmitted. Also, when the live video distribution method of the present invention is used commercially, only a video signal within a predetermined range can be transmitted to the user device used by a user who uses a free service, and an omnidirectional video signal or a full-surround video signal can be transmitted to the user device used by a user who uses a paid service.

[0068] Moreover, particularly according to the live video distribution method of the tenth aspect of the present invention and the video distribution apparatus used in the live video distribution method of this aspect, the voices of the operator or a navigator different from the operator can be emitted from the sound-emitting means of the user device used by each user. Therefore, even if the operator or navigator is in a location remote from each user, they can provide real-time commentary on the object being photographed by the unmanned mobile body in their own voice, just like a tour conductor in a group tour.

[0069] Moreover, particularly according to the live video distribution method of the eleventh aspect of the present invention and the video distribution apparatus used in the live video distribution method of this aspect, the sound obtained by superimposing the voices of users enjoying the video of the live video signal based on the result of the imaging means of the same unmanned mobile body photographing the object to be photographed can be emitted from the sound-emitting means of the user device used by each user. Therefore, even if each user is in a location remote from each other, they can discuss their impressions of the object being photographed by the unmanned mobile body as if they were participating in the same group tour.

[0070] Moreover, particularly according to the video distribution apparatus of the twenty-third aspect of the present invention and the video archive apparatus that stores the video data file generated by the video distribution apparatus of this aspect, since the video data transmitted by the live video signal is recorded as a video data file, users who were unable to enjoy the video of the live video signal in real time during the live distribution can also enjoy the recorded video by playing back the video data file.

Brief Description of the Drawings

[0071]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Embodiments for Carrying Out the Invention

[0072] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the present invention is not limited to such embodiments, and various modifications are possible within the scope of its technical idea. It is not limited to such embodiments, and various changes are possible within the scope of its technical idea.

Examples

[0073] FIG. 1 is a block diagram for explaining the configuration and functions of a live video distribution system that realizes the live video distribution method according to Embodiment 1 of the present invention, and the functions of a video distribution device used in the live video distribution method.

[0074] The live video distribution system of this embodiment includes an aerial drone 1, a pilot's HMD_21 used by a pilot, a drone control controller 22 used by the pilot (hereinafter, the pilot's HMD_21 and the drone control controller 22 are collectively referred to as the "pilot device 2"), a video distribution device 3, and user A's HMD_4A to user D's HMD_4D used by users A to D.

[0075] Note that the pilot's HMD_21 and user A's HMD_4A to user D's HMD_4D all have a microphone and a microphone speaker integrally configured. Further, the drone control controller 22 has wireless communication means, and the pilot's HMD_21 and the drone control controller 22 are wired-connected by a communication cable. In such a configuration, the pilot's HMD_21 transmits and receives signals to and from the aerial drone 1 and the video distribution device 3 via the wireless communication means of the drone control controller 22. On the other hand, user A's HMD_4A to user D's HMD_4D themselves have wireless communication means, and transmit and receive signals to and from the video distribution device 3 by their own wireless communication means.

[0076] Also, in this embodiment, the number of users using the live video distribution system is four users A to D, but the number of users can be increased to any number in principle.

[0077] The aerial drone 1 is a multi-rotor unmanned aerial vehicle, and by controlling the rotational speed of each rotor blade, it can float above an area where a shooting target exists, such as a tourist destination, and perform movements such as straight movement, turning movement, and rotational movement.

[0078] In addition, the aerial photography drone 1 is equipped with two omnidirectional cameras that cover a 360-degree field of view (solid angle) as imaging means, and photographs the shooting target within the area from above. Further, an omnidirectional 3D live video signal based on the results imaged by the two omnidirectional cameras is generated, and the omnidirectional 3D live video signal is wirelessly transmitted to the drone operation controller 22.

[0079] The drone operation controller 22 wired-transmits the received omnidirectional 3D live video signal to the operator's HMD_21 via a communication cable, while wirelessly transmitting it to the video distribution device 3.

[0080] The operator's HMD_21 is equipped with sensors that detect the movements of the operator's head and eyes. The operator's HMD_21 projects the video of the live video signal from the left and right projection units based on the received omnidirectional 3D live video signal and the detection results of the sensors. As a result, the operator can view the video within a predetermined display range of the omnidirectional 3D live video signal corresponding to the movements of their own head and eyes.

[0081] The operator operates the drone operation controller 22 while viewing the predetermined display range of the omnidirectional 3D live video signal projected from the left and right projection units of the operator's HMD_21. The drone operation controller 22 is a prop-type joystick controller, which generates a flight control signal based on the operator's lever operation and dial operation, and wirelessly transmits the flight control signal to the aerial photography drone 1. The rotation speed of the rotors of the aerial photography drone 1 is controlled based on the received flight control signal, and the aerial photography drone 1 performs flight as intended by the operator.

[0082] On the other hand, each of the HMD_4A for user A to the HMD_4D for user D is also equipped with sensors that detect the movements of the heads and eyes of users A to D. The HMD_4A for user A to the HMD_4D for user D generate a signal for setting the display range of the video projected from the left and right projection units (hereinafter abbreviated as "display range setting signal") based on the detection results of their respective sensors, and wirelessly transmit the display range setting signal to the video distribution device 3.

[0083] Based on the display range setting signals received from each of the HMDs_4A for User A to HMDs_4D for User D, the video distribution device 3 generates a 3D live video signal to be transmitted to each of the HMDs_4A for User A to HMDs_4D from the omnidirectional 3D live video signal received from the drone control controller 22, and wirelessly transmits the 3D live video signal to each of the HMDs_4A for User A to HMDs_4D.

[0084] Each of the HMDs_4A for User A to HMDs_4D projects an image from the left and right projection units based on the 3D live video signal received from the video distribution device 3. As a result, Users A to D can view a predetermined display range image of the 3D live video signal corresponding to the movement of their own heads and eyeballs. In this way, Users A to D can enjoy the predetermined display range of the omnidirectional 3D live video signal of the object photographed by the omnidirectional camera of the aerial photography drone 1 without performing any operation themselves.

[0085] In this embodiment, the video distribution device 3 is configured to cut out a predetermined display range image based on the display range setting signals received from each of the HMDs_4A for User A to HMDs_4D. However, an omnidirectional 3D video signal is transmitted from the video distribution device 3, and in the HMDs_4A for User A to HMDs_4D, a configuration may be adopted in which a predetermined display range image is cut out based on the received omnidirectional 3D live video signal and the detection result of the sensor.

[0086] Also, while viewing the image of the omnidirectional 3D live video signal projected from the left and right projection units of the operator's HMD_21, the operator verbally explains the object photographed by the omnidirectional camera of the aerial photography drone 1. The voice uttered by the operator is collected by a microphone integrally provided in the operator's HMD_21 and wirelessly transmitted as a voice signal to the video distribution device 3 via the drone control controller 22.

[0087] In this embodiment, the subject photographed by the operator himself / herself is described. However, it is also possible to provide a navigator separately from the operator and have the navigator give explanations. In that case, the navigator wears an HMD with a microphone similar to the operator's HMD_21 on the head, and while looking at the video of the live video signal projected from the left and right projection units on the HMD, gives oral explanations. The voice emitted by the navigator is collected by a microphone integrally provided on the HMD, and is wirelessly transmitted as an audio signal to the video distribution device 3 via the drone control controller 22.

[0088] Also, each of users A to D gives their impressions about the subject photographed by the omnidirectional camera of the aerial drone 1 while looking at the video within a predetermined display range of the 3D live video signal projected from the left and right projection units of the HMD_4A for user A to the HMD_4D for user D. The voice emitted by each of users A to D is collected by a microphone integrally provided on the HMD_4A for user A to the HMD_4D for user D, and is wirelessly transmitted as an audio signal to the video distribution device 3.

[0089] Based on the audio signals generated and transmitted by the microphones integrally provided on the operator's HMD_21 and the HMD_4A for user A to the HMD_4D for user D, the video distribution device 3 generates the following four types of superimposed audio signals, and transmits the superimposed audio signals A to D to each of the HMD_4A for user A to the HMD_4D for user D, respectively (hereinafter, "the audio signal generated and transmitted by the microphone of the operator's HMD_21", "the audio signal generated and transmitted by the microphone of the HMD_4A for user A", etc. are abbreviated as "operator audio signal", "user A audio signal", etc., respectively). Superimposed audio signal A... An audio signal obtained by superimposing "operator audio signal + user B audio signal + user C audio signal + user D audio signal" Superimposed audio signal B... An audio signal obtained by superimposing "operator audio signal + user A audio signal + user C audio signal + user D audio signal" Superimposed audio signal C... "Operator audio signal + user A audio signal + user B audio signal + user An audio signal with a "D audio signal" superimposed Superimposed audio signal D... "Pilot audio signal + User A audio signal + User B audio signal + User An audio signal with a "C audio signal" superimposed

[0090] Also, the video distribution device 3 generates the following superimposed audio signal 0 and transmits it to the pilot's HMD_21. Superimposed audio signal 0... "User A audio signal + User B audio signal + User C audio signal + User An audio signal with a "D audio signal" superimposed The pilot's HMD_21 receives the superimposed audio signal 0 from the video distribution device 3 via the drone control controller 22.

[0091] On the other hand, each of the micro speakers integrally provided in the pilot's HMD_21 and the user A's HMD_4A to the user D's HMD_4D emits sound based on the received superimposed audio signal. As a result, each of the pilot and users A to D can hear the sound with the voices of others other than themselves superimposed. In particular, users A to D can hear the explanations given orally by the pilot (or navigator) and the impressions dictated by their own users. Also, since the pilot can hear the impressions dictated by users A to D, he / she can adjust the flight of the aerial photography drone 1 or supplement the explanation according to the impressions.

[0092] Also, when the video distribution device 3 starts transmitting the 3D live video signal and the superimposed audio signal 0, it writes the video data and audio data transmitted by those signals as temporary data to the storage means provided in itself. Also, when the transmission of the 3D live video signal and the superimposed audio signal 0 is completed, it creates a video / audio data file in a format such as MP4 from the temporary data and saves it in the storage means.

[0093] In this embodiment, the HMD_21 for the operator is configured to transmit and receive omnidirectional 3D video signals and audio signals via the drone control controller 22. However, the HMD_21 for the operator can also be directly configured to transmit and receive signals to and from the aerial drone 1 and the video distribution device 3 after having wireless communication means.

Example

[0094] Figure 2 is a block diagram for explaining the configuration and functions of a live video distribution system that realizes the live video distribution method according to Embodiment 2 of the present invention, and the functions of the video distribution device used in the live video distribution method.

[0095] The functions of the components of the live video distribution system of this embodiment (aerial drone 1, HMD_21 for the operator, drone control controller 22, video distribution device 3, HMD_4A for user A to HMD_4D for user D) are the same as those of Embodiment 1, except for the points described in the following paragraphs.

[0096] In this embodiment, the transmission of the omnidirectional 3D live video signal from the aerial drone 1 to the video distribution device 3 is performed by direct wireless transmission, rather than via the drone control controller 22.

Example

[0097] Figure 3 is a block diagram for explaining the configuration and functions of a live video distribution system that realizes the live video distribution method according to Embodiment 3 of the present invention, and the functions of the video distribution device used in the live video distribution method.

[0098] The functions of the components of the live video distribution system of this embodiment (aerial drone 1, HMD_21 for the operator, drone control controller 22, video distribution device 3, HMD_4A for user A to HMD_4D for user D) are the same as those of Embodiment 2, except for the points described up to paragraph

[0101] .

[0099] In this embodiment, the signal transmission and reception between the aerial drone 1 and the drone control controller 22 are not performed by direct wireless communication, but through the video distribution device 3.

[0100] That is, the video distribution device 3 wirelessly transmits the omnidirectional 3D live video signal received from the aerial drone 1 to the drone control controller 22.

[0101] Also, the drone control controller 22 generates a flight control signal based on the operator's lever operation or dial operation, and wirelessly transmits the flight control signal to the video distribution device 3. The video distribution device 3 wirelessly transmits the flight control signal received from the drone control controller 22 to the aerial drone 1 after performing necessary processing. The rotation speed of the rotor of the aerial drone 1 is controlled based on the received flight control signal, and the aerial drone 1 performs flight as intended by the operator.

Example

[0102] FIG. 4 is a block diagram for explaining the configuration and functions of a live video distribution system that realizes the live video distribution method according to Embodiment 4 of the present invention, and the functions of the video distribution device and the video archive device used in the live video distribution method.

[0103] The live video distribution system of this embodiment includes, in addition to the aerial drone 1, the operator's HMD_21, the drone control controller 22, the video distribution device 3, and the HMD_4A for user A to the HMD_4D for user D, a video archive device 8. The functions of these components are the same as those in Embodiment 1 except for the points explained up to paragraph

[0107] .

[0104] In this embodiment, the signal transmission and reception between the drone control controller 22 and the video distribution device 3 are not performed by direct wireless communication, but through the public network.

[0105] That is, the drone control controller 22 transmits the omnidirectional 3D live video signal received from the aerial photography drone 1 to the 5G base station 5 which is a base station of the 5G network, and the omnidirectional 3D live video signal is transmitted to the video distribution device 3 via the 5G network (not shown in the figure) and the Internet 6.

[0106] In addition, the drone control controller 22 transmits the audio signal generated based on the result of sound collection by the microphone integrally configured in the operator's HMD_21 to the 5G base station 5, and the audio signal is transmitted to the video distribution device 3 via the Internet 6, the 5G network, and the Internet 6. On the other hand, the superimposed audio signal 0 generated by the video distribution device 3 is transmitted to the drone control controller 22 through the route of Internet 6 → 5G network → 5G base station 5, and finally transmitted to the operator's HMD_21.

[0107] In addition, when the video distribution device 3 starts transmitting the 3D live video signal and the superimposed audio signal 0, it writes the video data and audio data transmitted by these signals into the storage means provided in itself as temporary data almost simultaneously. Also, when the transmission of the 3D live video signal and the superimposed audio signal 0 is completed, a video and audio data file in a format such as MP4 is created from the temporary data, and the video data file is transmitted to the video archive device 8. The video archive device 8 stores the transmitted video data file.

[0108] In this embodiment, the drone control controller 22 is configured to have a 5G communication function. However, a 5G communication unit (such as a 5G-compatible Wi-Fi router) may be separately provided, and the drone control controller 22 may be configured to communicate with the 5G communication unit by wire or wirelessly.

[0109] In addition, in this embodiment, the video archive device 8 is provided as a device separate from the video distribution device 3. However, the video distribution device 3 itself may be configured to store the video data file (that is, the video distribution device 3 and the video archive device 8 are integrally configured).

[0110] In addition, in Example 6 and the example described in the previous paragraph, the video archive device 8 is configured to store the video and audio data files transmitted from the video distribution device 3. However, the aerial drone 1 is provided with a video data recording device, and the video captured by the imaging means is recorded in the video data recording device. After the aerial drone 1 returns from flight, the video data is recovered as a video data file, and the video data file can be stored in the video archive device 8. By adopting such a configuration, high-quality video can be stored without being restricted by the capacity of wireless communication, and the high-quality video of the stored video data file can be distributed to the user device.

Example

[0111] FIG. 5 is a block diagram for explaining the configuration and functions of a live video distribution system that realizes the live video distribution method according to Example 5 of the present invention, and the functions of the video distribution device and the video archive device used in the live video distribution method.

[0112] The functions of the components of the live video distribution system in this example (aerial drone 1, operator's HMD_21, drone control controller 22, video distribution device 3, user A's HMD_4A to user D's HMD_4D, video archive device 8) are the same as those in Example 4 except for the points described in the following paragraph.

[0113] In this example, the transmission of the omnidirectional 3D live video signal from the aerial drone 1 to the video distribution device 3 is performed without passing through the drone control controller 22. That is, the aerial drone 1 transmits the omnidirectional 3D live video signal to the 5G base station 5, which is a base station of the 5G network, and the omnidirectional 3D live video signal is transmitted to the video distribution device 3 via the 5G network (not shown in the figure) and the Internet 6.

Example

[0114] FIG. 6 is a block diagram for explaining the configuration and functions of a live video distribution system that realizes the live video distribution method according to Embodiment 6 of the present invention, and the functions of a video distribution apparatus and a video archive apparatus used in the live video distribution method.

[0115] The functions of the components of the live video distribution system in this embodiment (aerial drone 1, operator's HMD_21, drone control controller 22, video distribution apparatus 3, user A's HMD_4A to user D's HMD_4D, video archive apparatus 8) are the same as those in Embodiment 4, except for the points explained up to paragraph

[0119] .

[0116] In this embodiment, the transmission and reception of signals between the video distribution apparatus 3 and user A's HMD_4A to user D's HMD_4D are not performed by direct wireless communication, but via a public network.

[0117] That is, each of user A's HMD_4A to user D's HMD_4D transmits a display range setting signal generated based on the detection results of sensors that detect the movements of the heads and eyeballs of users A to D to the video distribution apparatus 3 via the Internet 6 and each of ISP servers A_7A to ISP servers D_7D.

[0118] The video distribution apparatus 3 transmits a 3D live video signal generated based on the display range setting signals received from each of user A's HMD_4A to user D's HMD_4D and the omnidirectional 3D live video signal received from the aerial drone 1 to each of user A's HMD_4A to user D's HMD_4D via the Internet 6 and each of ISP servers A_7A to ISP servers D_7D.

[0119] In addition, each of the HMDs 4A for User A to 4D for User D transmits an audio signal generated based on the result of sound collection by an integrally configured microphone to the video distribution device 3 via the Internet 6 and each of the ISP servers 7A for ISP server A to 7D for ISP server D. On the other hand, the superimposed audio signals A to D generated by the video distribution device 3 are transmitted to each of the HMDs 4A for User A to 4D for User D via the Internet 6 and each of the ISP servers 7A for ISP server A to 7D for ISP server D.

Example

[0120] FIG. 7 is a block diagram for explaining the configuration and functions of a live video distribution system that realizes the live video distribution method according to Example 7 of the present invention, and the functions of the video distribution device and the video archive device used in the live video distribution method.

[0121] The functions of the components of the live video distribution system of this example (aerial drone 1, HMD 21 for the operator, drone control controller 22, video distribution device 3, HMDs 4A for User A to 4D for User D, video archive device 8) are the same as those of Example 5 except for the points described in the following paragraphs.

[0122] In this example, the transmission and reception of signals between the video distribution device 3 and the HMDs 4A for User A to 4D for User D are not performed by direct wireless communication, but via a public network.

Example

[0123] FIG. 8 is a block diagram for explaining the configuration and functions of a live video distribution system that realizes the live video distribution method according to Example 8 of the present invention, and the functions of the video distribution device and the video archive device used in the live video distribution method.

[0124] The functions of the components of the live video distribution system of this embodiment (aerial drone 1, operator's HMD_21, drone control controller 22, video distribution device 3, user A's HMD_4A to user D's HMD_4D, video archive device 8) are the same as those of Embodiment 6, except for the points described up to paragraph

[0129] .

[0125] In this embodiment, the user A device to the user D device are composed of user A's smartphone_41A to user D's smartphone_41D in addition to user A's HMD_4A to user D's HMD_4D, and user A's HMD_4A to user D's HMD_4D are each wired-connected to user A's smartphone_41A to user D's smartphone_41D via a communication cable.

[0126] And user A's smartphone_41A to user D's smartphone_41D have 5G communication functions. Each of user A's HMD_4A to user D's HMD_4D does not transmit and receive signals with each of ISP server A_7A to ISP server D_7D, but via user A's smartphone_41A to user D's smartphone_41D, transmits and receives signals with each of 5G base station A_5A to 5G base station C_5C and 5G base station D_5D.

[0127] That is, each of user A's HMD_4A to user D's HMD_4D transmits a display range setting signal generated based on the detection results of sensors that detect the movements of the heads and eyeballs of users A to D to the video distribution device 3 via each of user A's smartphone_41A to user D's smartphone_41D, each of 5G base station A_5A to 5G base station D_5D, and the 5G network and the Internet 6.

[0128] The video distribution device 3 transmits the 3D live video signal generated based on the display range setting signals received from each of the HMDs 4A for user A to 4D for user D and the omnidirectional 3D live video signal received from the aerial drone 1, via the Internet 6, the 5G network, and each of the 5G base stations A_5A to D_5D and the smartphones 41A for user A to 41D for user D, to each of the HMDs 4A for user A to 4D for user D.

[0129] Also, each of the HMDs 4A for user A to 4D for user D transmits the audio signal generated based on the result of sound collection by the integrally configured microphone, via each of the smartphones 41A for user A to 41D for user D, each of the 5G base stations A_5A to D_5D, the 5G network, and the Internet 6, to the video distribution device 3. On the other hand, the superimposed audio signals A to D generated by the video distribution device 3 are transmitted to the HMDs 4A for user A to 4E for user D via the Internet 6, the 5G network, and each of the 5G base stations A_5A to D_5D and the smartphones 41A for user A to 41D for user D.

Example

[0130] FIG. 9 is a block diagram for explaining the configuration and functions of a live video distribution system that realizes the live video distribution method according to Example 9 of the present invention, and the functions of the video distribution device and the video archive device used in the live video distribution method.

[0131] The functions of the components of the live video distribution system in this example (aerial drone 1, operator's HMD 21, drone control controller 22, video distribution device 3, HMDs 4A for user A to 4D for user D, video archive device 8) are the same as those in Example 7, except for the points explained up to paragraph

[0133] .

[0132] In this embodiment, the devices for User A to User D are composed of smartphones for User A to User D (Smartphone_41A to Smartphone_41D) in addition to the HMDs for User A to User D (HMD_4A to HMD_4D). The HMDs for User A to User D (HMD_4A to HMD_4D) are each wired-connected to the smartphones for User A to User D (Smartphone_41A to Smartphone_41D) via communication cables.

[0133] And the smartphones for User A to User D (Smartphone_41A to Smartphone_41D) have 5G communication functions. Instead of directly transmitting and receiving signals between each of the HMDs for User A to User D (HMD_4A to HMD_4D) and each of the ISP servers A to D (ISP Server A_7A to ISP Server D_7D), signals are transmitted and received between each of the HMDs for User A to User D (HMD_4A to HMD_4D) and each of the 5G base stations A to C (5G Base Station A_5A to 5G Base Station C_5C) and 5G Base Station D_5D via the smartphones for User A to User D (Smartphone_41A to Smartphone_41D).

Embodiment

[0134] FIG. 10 is a block diagram for explaining the configuration and functions of a live video distribution system that realizes the live video distribution method according to Embodiment 10 of the present invention, and the functions of the video distribution device and the video archive device used in the live video distribution method.

[0135] The functions of the components of the live video distribution system in this embodiment (aerial photography drone 1, HMD for operator 21, drone control controller 22, video distribution device 3, HMDs for User A to User D (HMD_4A to HMD_4D), video archive device 8) are the same as those in Embodiment 5 except for the points explained up to paragraph

[0139] .

[0136] In this embodiment, the signal transmission and reception between the aerial photography drone 1 and the drone control controller 22 are not performed by direct wireless communication, but via the video distribution device 3 and the public network. Also, the audio signal transmission and reception between the HMD for operator 21 and the video distribution device 3 are performed by direct wireless communication without passing through the public network.

[0137] That is, the video distribution device 3 wirelessly transmits the omnidirectional 3D live video signal received from the aerial drone 1 via the 5G base station 5 → 5G network → Internet 6 to the drone control controller 22.

[0138] Also, the drone control controller 22 generates a flight control signal based on the operator's lever operation or dial operation, and wirelessly transmits the flight control signal to the video distribution device 3. The video distribution device 3 performs necessary processing on the flight control signal received from the drone control controller 22, and finally transmits it from the 5G base station 5 to the aerial drone 1 via the Internet 6 and the 5G network. The rotation speed of the rotor of the aerial drone 1 is controlled based on the received flight control signal, and the aerial drone 1 performs flight as intended by the operator.

[0139] Also, the drone control controller 22 wirelessly transmits the audio signal generated based on the result of sound collection by the microphone integrally configured in the operator's HMD_21 to the video distribution device 3. On the other hand, the superimposed audio signal 0 generated by the video distribution device 3 is transmitted to the operator's HMD_21 via the drone control controller 22.

Example

[0140] FIG. 11 is a block diagram for explaining the configuration and functions of a live video distribution system that realizes the live video distribution method according to Embodiment 11 of the present invention, and the functions of the video distribution device and the video archive device used in the live video distribution method.

[0141] The functions of the components of the live video distribution system in this embodiment (aerial drone 1, operator's HMD_21, drone control controller 22, video distribution device 3, user A's HMD_4A to user D's HMD_4D, video archive device 8) are the same as those in Embodiment 10 except for the points described in the following paragraphs.

[0142] In this embodiment, the aerial drone 1 does not perform signal transmission and reception with only one 5G base station 5, but performs signal transmission and reception with the 5G base station I_5I to the 5G base station III_5III in a cellular manner. In this embodiment, there are three 5G base stations, namely the 5G base station I_5I to the 5G base station III_5III, but in principle, the number of 5G base stations can be increased as many as possible.

[0143] In this embodiment, with this configuration, the aerial drone 1 is not limited to the range where it can perform wireless communication with only one 5G base station, but can fly anywhere as long as it is within the range where cellular communication is possible. Therefore, it can photograph imaging targets spreading over a wide area.

Embodiment

[0144] FIG. 12 is a block diagram for explaining the configuration and functions of a live video distribution system that realizes the live video distribution method according to Embodiment 12 of the present invention, and the functions of the video distribution device and the video archive device used in the live video distribution method.

[0145] The functions of the components of the live video distribution system in this embodiment (the aerial drone 1, the operator's HMD_21, the drone control controller 22, the video distribution device 3, the user A's HMD_4A to the user D's HMD_4D, the video archive device 8) are the same as those in Embodiment 11 except for the points explained up to paragraph

[0149] .

[0146] In this embodiment, the signal transmission and reception between the drone control controller 22 and the video distribution device 3 are not performed by direct wireless communication, but are performed via a public network.

[0147] That is, the video distribution device 3 transmits the omnidirectional 3D live video signal received from the aerial drone 1 via the 5G base station I_5I to the 5G base station III_5III, the 5G network, and the Internet 6 to the drone control controller 22 from the 5G base station 50 via the Internet 6 and the 5G network.

[0148] In addition, the drone control controller 22 transmits a flight control signal generated based on the operator's lever operation or dial operation to the 5G base station 50, and the flight control signal is transmitted to the video distribution device 3 via the 5G network and the Internet 6.

[0149] In addition, the operator's HMD_21 transmits an audio signal generated based on the result of sound collection by the integrally configured microphone to the 5G base station 50 via the drone control controller 22, and the audio signal is transmitted to the video distribution device 3 via the 5G network and the Internet 6. On the other hand, the superimposed audio signal 0 generated by the video distribution device 3 is finally transmitted from the 5G base station 50 to the operator's HMD_21 via the Internet 6 and the 5G network.

Example

[0150] FIG. 13 is a block diagram for explaining the configuration and functions of a live video distribution system and a local system that realize the live video distribution method according to Embodiment 13 of the present invention, and the functions of the video distribution device and the video archive device used in the live video distribution method.

[0151] The functions of the components of the live video distribution system in this embodiment (aerial photography drone (1)_11, aerial photography drone (2)_12, HMD_211 for operator (1), HMD_212 for operator (2), drone control controller (1)_221, drone control controller (2)_222, video distribution device 3, HMD_4A for user A to HMD_4C for user C, notebook PC_4D for user D, video archive device 8, local system 9) are the same as those in Embodiment 1 except for the points explained up to paragraph

[0161] .

[0152] The live video distribution system of this embodiment includes an aerial drone (1)_11 and an aerial drone (2)_12 as components. The aerial drone (1)_11 flies based on a flight control signal generated and transmitted based on the operation of the drone controller (1)_221 by the operator (1). The aerial drone (2)_12 flies based on a flight control signal generated and transmitted based on the operation of the drone controller (2)_222 by the operator (2). Also, the HMD_211 for the operator (1) and the HMD212 for the operator (2) are each equipped with sensors for detecting the movements of the heads and eyes of the operator (1) and the operator (2). The HMD_211 for the operator (1) projects the video of the live video signal from the left and right projection units based on the omnidirectional 3D live video signal received from the aerial drone (1)_11 and the detection result of the sensor. The HMD_211 for the operator (2) projects the video of the live video signal from the left and right projection units based on the omnidirectional 3D live video signal received from the aerial drone (2)_12 and the detection result of the sensor.

[0153] In this embodiment, there are two operators, operator (1) and operator (2), who operate the aerial drones. However, in principle, the number of operators can be increased to any number. Also, hereinafter, "aerial drone (1)_11, aerial drone (2)_12, HMD_211 for operator (1), HMD_212 for operator (2), drone controller (1)_221 for drone operation, drone controller (2)_222 for drone operation, video distribution device 3" are collectively referred to as the on-site system 9.

[0154] On the other hand, in this embodiment, there are four users, users A to C and user D, who use the live video distribution system. Among them, only user D uses a notebook PC_4D for user D as a user device instead of an HMD.

[0155] Then, User A to User C and User D select which of the live video signals captured by Aerial Drone (1)_11 and Aerial Drone (2)_12 to receive, based on the profiles of Operator (1) and Operator (2), and the information on the flight plans (flight routes and flight paths) of Aerial Drone (1)_11 and Aerial Drone (2)_12, which are projected in the virtual space on HMDs_4A for User A to HMDs_4C for User C or displayed on the display of Notebook PC_4D for User D. In the following, it is described assuming that User A to User C select Aerial Drone (1)_11 and User D selects Aerial Drone (2)_12.

[0156] The selection described in the previous paragraph is made by User A to User C operating an operation means different from HMDs_4A for User A to HMDs_4C for User C (not shown in the figure), and by User D operating the input means (keyboard and pointing device) of Notebook PC_4D for User D, and a UAV selection signal based on the selection result is transmitted to Video Distribution Device 3.

[0157] Based on the received UAV selection signal, Video Distribution Device 3 associates Aerial Drone (1)_11 and Aerial Drone (2)_12 with HMDs_4A for User A to HMDs_4C for User C and Notebook PC_4D for User D. As a result of this association, a 3D live video signal generated from the omnidirectional 3D live video signal received from Aerial Drone (1)_11 is transmitted to HMDs_4A for User A to HMDs_4C for User C, and a two-dimensional live video signal generated from the omnidirectional 3D live video signal received from Aerial Drone (2)_12 is transmitted to Notebook PC_4D for User D.

[0158] Also, in this embodiment, the signal transmission and reception between Video Distribution Device 3 and HMDs_4A for User A to HMDs_4C for User C and Notebook PC_4D for User D are not performed by direct wireless communication, but via a public network.

[0159] That is, each of the HMDs 4A for User A to 4C for User C transmits a display range setting signal generated based on the detection results of sensors that detect the movements of the heads and eyes of Users A to C to the video distribution device 3 via the Internet 6 and each of the ISP servers 7A for ISP server A to 7C for ISP server C. Also, the notebook PC 4D for User D transmits a display range setting signal generated by User D operating the input means of the notebook PC 4D for User D to the video distribution device 3 via the Internet 6 and the ISP server 7D for ISP server D.

[0160] The video distribution device 3 transmits a 3D live video signal generated based on the display range setting signal received from each of the HMDs 4A for User A to 4C for User C and the omnidirectional 3D live video signal received from the aerial drone (1)_11 to each of the HMDs 4A for User A to 4C for User C via the Internet 6 and each of the ISP servers 7A for ISP server A to 7C for ISP server C. Also, the video distribution device 3 transmits a two-dimensional live video signal generated based on the display range setting signal received from the notebook PC 4D for User D and the omnidirectional 3D live video signal received from the aerial drone (2)_12 to the notebook PC 4D for User D via the Internet 6 and the ISP server 7D for ISP server D.

[0161] Also, each of the HMDs 4A for User A to 4C for User C and the notebook PC 4D for User D transmits an audio signal generated based on the result of sound collection by an integrally configured microphone to the video distribution device 3 via each of the ISP servers 7A for ISP server A to 7D for ISP server D and the Internet 6. On the other hand, the superimposed audio signals A to D generated by the video distribution device 3 are transmitted to each of the HMDs 4A for User A to 4C for User C and the notebook PC 4D for User D via the Internet 6 and the ISP servers 7A for ISP server A to 7D for ISP server D.

Example

[0162] FIG. 14 is a block diagram for explaining the configuration and functions of a live video distribution system and a local system that implement the live video distribution method according to Embodiment 14 of the present invention, and the functions of a video distribution device and a video archive device used in the live video distribution method.

[0163] The functions of the components of the live video distribution system in this embodiment (aerial drone (1)_11, aerial drone (2)_12, HMD for operator (1)_211, HMD for operator (2)_212, drone control controller (1)_221, drone control controller (2)_222, video distribution device 3, HMD for user A_4A to HMD for user C_4C, notebook PC for user D_4D, video archive device 8, local system 9) are the same as those in Embodiment 13 except for the points described in the following paragraphs.

[0164] In this embodiment, the transmission of the omnidirectional 3D live video signal from the aerial drone (1)_11 and the aerial drone (2)_12 to the video distribution device 3 is performed by direct wireless transmission instead of via the drone control controller (1)_221 and the drone control controller (2)_222.

Embodiment

[0165] FIG. 15 is a block diagram for explaining the configuration and functions of a live video distribution system and a local system that implement the live video distribution method according to Embodiment 15 of the present invention, and the functions of a video distribution device and a video archive device used in the live video distribution method.

[0166] The functions of the components of the live video distribution system in this embodiment (aerial drone (1)_11, aerial drone (2)_12, HMD for operator (1)_211, HMD for operator (2)_212, drone control controller (1)_221, drone control controller (2)_222, video distribution device 3, HMD for user A_4A to HMD for user C_4C, notebook PC for user D_4D, video archive device 8, local system 9) are the same as those in Embodiment 14 except for the points described up to paragraph

[0169] .

[0167] In this embodiment, the transmission and reception of signals between the aerial drone (1)_11 and the drone controller (1)_221 for drone operation, and between the aerial drone (2)_12 and the drone controller (2)_222 for drone operation are not performed by direct wireless communication, but are performed via the video distribution device 3.

[0168] That is, the video distribution device 3 wirelessly transmits the omnidirectional 3D live video signals received from the aerial drone (1)_11 and the aerial drone (2)_12 to the drone controller (1)_221 for drone operation and the drone controller (2)_222 for drone operation, respectively.

[0169] Also, the drone controller (1)_221 generates a flight control signal based on the lever operation and dial operation of the operator (1), and wirelessly transmits the flight control signal to the video distribution device 3. The video distribution device 3 wirelessly transmits the flight control signal received from the drone controller (1)_221 to the aerial drone (1)_11 after performing necessary processing. The rotation speed of the rotor of the aerial drone (1)_11 is controlled based on the received flight control signal, and the aerial drone (1)_11 performs flight as intended by the operator (1). On the other hand, the drone controller (2)_222 generates a flight control signal based on the lever operation and dial operation of the operator (2), and wirelessly transmits the flight control signal to the video distribution device 3. The video distribution device 3 wirelessly transmits the flight control signal received from the drone controller (2)_222 to the aerial drone (2)_12 after performing necessary processing. The rotation speed of the rotor of the aerial drone (2)_12 is controlled based on the received flight control signal, and the aerial drone (2)_12 performs flight as intended by the operator (2).

Embodiment

[0170] FIG. 16 is a block diagram for explaining the configuration and functions of a live video distribution system and a local system that implement the live video distribution method according to Embodiment 16 of the present invention, and the functions of the video distribution device and the video archive device used in the live video distribution method.

[0171] The functions of the components of the live video distribution system of this embodiment (aerial photography drone (1)_11, aerial photography drone (2)_12, HMD for operator (1)_211, HMD for operator (2)_212, drone control controller (1)_221, drone control controller (2)_222, video distribution device 3, HMD for user A_4A to HMD for user C_4C, notebook PC for user D_4D, smartphone for user A_41A to smartphone for user D_41D, video archive device 8, local system 9) are the same as those of Embodiment 13 except for the points described up to paragraph

[0176] .

[0172] In this embodiment, the devices for users A to C are composed of smartphones for user A_41A to smartphones for user C_41C in addition to HMDs for user A_4A to HMDs for user C_4C, and the device for user D is composed of smartphone for user D_41D in addition to notebook PC for user D_4D. HMDs for user A_4A to HMDs for user C_4C and notebook PC for user D_4D are each wired-connected to smartphones for user A_41A to smartphones for user D_41D by communication cables.

[0173] And smartphones for user A_41A to smartphones for user D_41D have 5G communication functions. Each of HMDs for user A_4A to HMDs for user C_4C and notebook PC for user D_4D does not transmit and receive signals to and from ISP servers A_7A to ISP servers C_7C and ISP server D_7D respectively, but transmits and receives signals to and from 5G base stations A_5A to 5G base stations C_5C and 5G base station D_5D via smartphones for user A_41A to smartphones for user D_41D.

[0174] That is, each of the HMDs 4A for User A to 4C for User C transmits a display range setting signal generated based on the detection results of sensors that detect the movements of the heads and eyes of Users A to C to the respective smartphones 41A for User A to 41C for User C, the respective 5G base stations A_5A to C_5C, and the video distribution device 3 via the 5G network and the Internet 6. Also, the notebook PC 4D for User D transmits a display range setting signal generated by User D operating the input means of the notebook PC 4D for User D to the video distribution device 3 via the smartphone 41D for User D, the 5G base station D_5D, and the 5G network and the Internet 6.

[0175] The video distribution device 3 transmits a 3D live video signal generated based on the display range setting signal received from each of the HMDs 4A for User A to 4C for User C and the omnidirectional 3D live video signal received from the aerial drone (1)_11 to each of the HMDs 4A for User A to 4C for User C via the Internet 6, the 5G network, and the respective 5G base stations A_5A to C_5C and the respective smartphones 41A for User A to 41C for User C. Also, the video distribution device 3 transmits a two-dimensional live video signal generated based on the display range setting signal received from the notebook PC 4D for User D and the omnidirectional 3D live video signal received from the aerial drone (2)_12 to the notebook PC 4D for User D via the Internet 6, the 5G network, the 5G base station D_5D, and the smartphone 41D for User D.

[0176] Also, each of the HMDs 4A for User A to 4C for User C and the notebook PC 4D for User D transmits an audio signal generated based on the result of sound collection by an integrally configured microphone to each of the smartphones 41A for User A to 41D for User D, each of the 5G base stations A_5A to D_5D, and the video distribution device 3 via the 5G network and the Internet 6. On the other hand, the superimposed audio signals A to D generated by the video distribution device 3 are transmitted to each of the HMDs 4A for User A to 4C for User C and the notebook PC 4D for User D via the Internet 6, the 5G network, each of the 5G base stations A_5A to D_5D, and each of the smartphones 41A for User A to 41D for User D.

Example

[0177] Figure 17 is a block diagram for explaining the configuration and functions of a live video distribution system and a local system that implement the live video distribution method according to Embodiment 17 of the present invention, and the functions of the video distribution device and the video archive device used in the live video distribution method.

[0178] The functions of the components of the live video distribution system in this embodiment (aerial drone (1)_11, aerial drone (2)_12, HMD 211 for operator (1), HMD 212 for operator (2), drone control controller (1)_221, drone control controller (2)_222, video distribution device 3, HMDs 4A for User A to 4C for User C, notebook PC 4D for User D, smartphones 41A for User A to 41D for User D, video archive device 8, local system 9) are the same as those in Embodiment 14 except for the points explained up to paragraph

[0180] .

[0179] In this embodiment, the devices for User A to User C are composed of the HMDs for User A to User C (HMD_4A to HMD_4C) and the smartphones for User A to User C (smartphone_41A to smartphone_41C). The device for User D is composed of the notebook PC for User D (notebook PC_4D) and the smartphone for User D (smartphone_41D). The HMDs for User A to User C (HMD_4A to HMD_4C) and the notebook PC for User D (notebook PC_4D) are each connected by a communication cable to the smartphones for User A to User D (smartphone_41A to smartphone_41D).

[0180] And the smartphones for User A to User D (smartphone_41A to smartphone_41D) have 5G communication functions. Each of the HMDs for User A to User C (HMD_4A to HMD_4C) and the notebook PC for User D (notebook PC_4D) does not transmit and receive signals directly with the ISP servers A to C (ISP server A_7A to ISP server C_7C) and ISP server D (ISP server D_7D), but instead transmits and receives signals with the 5G base stations A to C (5G base station A_5A to 5G base station C_5C) and 5G base station D (5G base station D_5D) via the smartphones for User A to User D (smartphone_41A to smartphone_41D).

Example

[0181] Figure 18 is a block diagram for explaining the configuration and functions of a live video distribution system and a local system that implement the live video distribution method according to Embodiment 18 of the present invention, and the functions of a video distribution device and a video archive device used in the live video distribution method.

[0182] The functions of the components of the live video distribution system of this embodiment (aerial drone (1)_11, aerial drone (2)_12, HMD for operator (1)_211, HMD for operator (2)_212, drone control controller (1)_221, drone control controller (2)_222, video distribution device 3, HMD for user A_4A to HMD for user C_4C, notebook PC for user D_4D, smartphone for user A_41A to smartphone for user D_41D, video archive device 8, local system 9) are the same as those of Embodiment 15, except for the points described up to paragraph

[0184] .

[0183] In this embodiment, the devices for users A to C are composed of smartphones for user A_41A to smartphones for user C_41C in addition to HMDs for user A_4A to HMDs for user C_4C, and the device for user D is composed of smartphone for user D_41D in addition to notebook PC for user D_4D. Each of HMDs for user A_4A to HMDs for user C_4C and notebook PC for user D_4D is wired-connected to smartphones for user A_41A to smartphones for user D_41D via communication cables, respectively.

[0184] And smartphones for user A_41A to smartphones for user D_41D have 5G communication functions. Each of HMDs for user A_4A to HMDs for user C_4C and notebook PC for user D_4D does not transmit and receive signals to and from ISP server A_7A to ISP server C_7C and ISP server D_7D, respectively, but transmits and receives signals to and from each of 5G base station A_5A to 5G base station C_5C and 5G base station D_5D via smartphones for user A_41A to smartphones for user D_41D.

Embodiment

[0185] FIG. 19 is a block diagram for explaining the configuration and functions of a live video distribution system that realizes the live video distribution method according to Embodiment 19 of the present invention, and the functions of the video distribution device and the video archive device used in the live video distribution method.

[0186] The functions of the components of the live video distribution system of this embodiment (aerial photography drone (1)_11, aerial photography drone (2)_12, HMD for operator (1)_211, HMD for operator (2)_212, drone control controller (1)_221, drone control controller (2)_222, video distribution device 3, HMD for user A_4A to HMD for user C_4C, notebook PC for user D_4D, video archive device 8) are the same as those of Embodiment 15 except for the points described up to paragraph

[0189] .

[0187] In this embodiment, the transmission and reception of signals between the aerial photography drones (1)_11 and (2)_12 and the video distribution device 3 are not performed by direct wireless communication, but via a public network.

[0188] That is, the aerial photography drones (1)_11 and (2)_12 transmit the omnidirectional 3D live video signal generated based on the results captured by the two omnidirectional cameras to the 5G base station 5, which is a base station of the 5G network, and the omnidirectional 3D live video signal is transmitted to the video distribution device 3 via the 5G network (not shown in the figure) and the Internet 6.

[0189] Also, the video distribution device 3 performs necessary processing on the flight control signal received from the drone control controller (1)_221 and finally transmits it from the 5G base station 5 to the aerial photography drone (1)_11 via the Internet 6 and the 5G network. The rotation speed of the rotor of the aerial photography drone (1)_11 is controlled based on the received flight control signal, and the aerial photography drone 1 flies as intended by the operator. Also, the video distribution device 3 performs necessary processing on the flight control signal received from the drone control controller (2)_222 and finally transmits it from the 5G base station 5 to the aerial photography drone (2)_12 via the Internet 6 and the 5G network. The rotation speed of the rotor of the aerial photography drone (2)_12 is controlled based on the received flight control signal, and the aerial photography drone (2)_12 flies as intended by the operator.

Embodiment

[0190] FIG. 20 is a block diagram for explaining the configuration and functions of a live video distribution system that realizes the live video distribution method according to Embodiment 20 of the present invention, and the functions of a video distribution device and a video archive device used in the live video distribution method.

[0191] The functions of the components of the live video distribution system of this embodiment (aerial drone (1)_11, aerial drone (2)_12, HMD for operator (1)_211, HMD for operator (2)_212, drone control controller (1)_221, drone control controller (2)_222, video distribution device 3, HMD for user A_4A to HMD for user C_4C, notebook PC for user D_4D) are the same as those of Embodiment 19 except for the points to be described in the following paragraphs.

[0192] In this embodiment, the aerial drones (1)_11 and (2)_12 do not transmit and receive signals only with a single 5G base station 5, but perform signal transmission and reception with 5G base stations I_5I to III_5III in a cellular manner. In this embodiment, there are three 5G base stations, namely 5G base stations I_5I to III_5III, but in principle, the number of 5G base stations can be increased arbitrarily.

Embodiment

[0193] FIG. 21 is a block diagram for explaining the configuration and functions of a live video distribution system that realizes the live video distribution method according to Embodiment 21 of the present invention, and the functions of a video distribution device and a video archive device used in the live video distribution method.

[0194] The functions of the components of the live video distribution system of this embodiment (aerial drone (1)_11, aerial drone (2)_12, HMD for operator (1)_211, HMD for operator (2)_212, drone control controller (1)_221, drone control controller (2)_222, video distribution device 3, HMD for user A_4A to HMD for user C_4C, notebook PC for user D_4D, video archive device 8) are the same as those of Embodiment 20 except for the points described up to paragraph

[0199] .

[0195] In this embodiment, the transmission and reception of signals between the devices 21 for operator (1) and the devices 22 for operator (2) and the video distribution device 3 are not performed by direct wireless communication, but through a public network.

[0196] That is, the HMD_211 for operator (1) transmits a display range setting signal generated based on the detection results of sensors that detect the movements of the head and eyes of operator (1) to the 5G base station 50, and the display range setting signal is transmitted to the video distribution device 3 via the 5G network and the Internet 6. On the other hand, the HMD_212 for operator (2) transmits a display range setting signal generated based on the detection results of sensors that detect the movements of the head and eyes of operator (2) to the 5G base station 50, and the display range setting signal is transmitted to the video distribution device 3 via the 5G network and the Internet 6.

[0197] The video distribution device 3 generates a 3D live video signal to be transmitted to the HMD_211 for the operator (1) from the display range setting signal received from the HMD_211 for the operator (1) and the omnidirectional 3D live video signal received from the aerial drone (1)_11. The 3D live video signal is finally transmitted from the 5G base station 50 to the HMD_211 for the operator (1) via the Internet 6 and the 5G network. Also, the video distribution device 3 generates a 3D live video signal to be transmitted to the HMD_212 for the operator (2) from the display range setting signal received from the HMD_212 for the operator (2) and the omnidirectional 3D live video signal received from the aerial drone (2)_12. The 3D live video signal is finally transmitted from the 5G base station 50 to the HMD_212 for the operator (2) via the Internet 6 and the 5G network.

[0198] Also, the drone control controller (1)_221 transmits a flight control signal generated based on the operator's lever operation or dial operation to the 5G base station 50, and the flight control signal is transmitted to the video distribution device 3 via the 5G network and the Internet 6. On the other hand, the drone control controller (2)_222 transmits a flight control signal generated based on the operator's lever operation or dial operation to the 5G base station 50, and the flight control signal is transmitted to the video distribution device 3 via the 5G network and the Internet 6.

[0199] Also, the HMD_211 for the operator (1) and the HMD_212 for the operator (2) transmit an audio signal generated based on the result of sound collection by a microphone integrally configured to the 5G base station 50, and the audio signal is transmitted to the video distribution device 3 via the 5G network and the Internet 6. On the other hand, the superimposed audio signal 0 generated by the video distribution device 3 is finally transmitted from the 5G base station 50 to the HMD_211 for the operator (1) and the HMD_212 for the operator (2) via the Internet 6 and the 5G network.

Example

[0200] FIG. 22 is a block diagram for explaining the configuration and functions of a live video distribution system that realizes the live video distribution method according to Embodiment 22 of the present invention, and the functions of a video distribution device and a video archive device used in the live video distribution method.

[0201] The functions of the components of the live video distribution system in this embodiment (aerial drone (1)_11, aerial drone (2)_12, HMD for operator (1)_211, HMD for operator (2)_212, drone control controller (1)_221, drone control controller (2)_222, video distribution device 3, HMD for user A_4A to HMD for user C_4C, notebook PC for user D_4D, video archive device 8) are the same as those in Embodiment 21 except for the points explained up to paragraph

[0203] .

[0202] In this embodiment, the devices for users A to C are composed of smartphones for user A_41A to smartphones for user C_41C in addition to HMDs for user A_4A to HMDs for user C_4C, and the device for user D is composed of a notebook PC for user D_4D in addition to a smartphone for user D_41D. The HMDs for user A_4A to HMDs for user C_4C and the notebook PC for user D_4D are each wired-connected to the smartphones for user A_41A to smartphones for user D_41D via a communication cable.

[0203] And, the smartphones for user A_41A to smartphones for user D_41D have a 5G communication function. Each of the HMDs for user A_4A to HMDs for user C_4C and the notebook PC for user D_4D does not transmit and receive signals directly with the ISP servers A_7A to ISP servers C_7C and ISP server D_7D respectively, but transmits and receives signals with each of the 5G base stations A_5A to 5G base stations C_5C and 5G base station D_5D via the smartphones for user A_41A to smartphones for user D_41D.

Embodiment

[0204] FIG. 23 is a block diagram for explaining the configuration and functions of a live video distribution system that realizes the live video distribution method according to Embodiment 23 of the present invention, and the functions of a video distribution device and a video archive device used in the live video distribution method.

[0205] In this embodiment, the live video distribution method of the present invention is applied to the distribution of the video of the live video signal of exhibits (including artworks and museum items) exhibited in an exhibition hall (including art museums and museums).

[0206] In this embodiment, there is only one operator. Therefore, there is only one aerial photography drone, which is the aerial photography drone 1, and the operator's device also consists of only a set of the operator's HMD_21 and the drone control controller 22. Also, the functions of the HMD_4A for User A to the HMD_4C for User C and the notebook PC_4D for User D are the same as those in Embodiment 18, except that each of Users A to C and User D cannot select from a plurality of aerial photography drones and can only enjoy the video of the live video signal of the object photographed by the omnidirectional camera of the aerial photography drone 1.

[0207] In this embodiment, the exhibition hall is divided into the following three areas. Exhibition area... An area where exhibits are exhibited and entry by people viewing the exhibits is prohibited area Moving area... An area where people viewing the exhibits can move Backyard... An area other than the exhibition area and the moving area

[0208] The operator is located in the backyard and operates the drone control controller 22 while viewing the video of the live video signal projected from the left and right projection parts of the operator's HMD_21. The operator's device (the operator's HMD_21 and the drone control controller 22) and the video distribution device 3 perform signal transmission and reception in the same manner as in Embodiment 2.

[0209] On one side, a drone control antenna 23 is installed within the movement area. The drone control antenna 23 is connected to the video distribution device 3 by wire. The drone control antenna 23 mediates the transmission and reception of signals between the aerial photography drone 1 and the video distribution device 3. As a result, the aerial photography drone 1 and the video distribution device 3 can transmit and receive signals via the drone control antenna 23, similar to Example 2.

[0210] In addition, the aerial photography drone 1 is provided with an approach determination means for determining the approach to the exhibition area based on the results captured by two omnidirectional cameras. Thus, contact with the exhibit can be avoided.

Example

[0211] FIG. 24 is a block diagram for explaining the configuration and functions of a live video distribution system and a local system that realize the live video distribution method according to Example 24 of the present invention, and the functions of the video distribution device and the video archive device used in the live video distribution method.

[0212] The functions of the components of the live video distribution system in this example (aerial photography drone α1_1α1, local systems α_9α to ω_9ω, video distribution devices α_3α to ω_3ω, HMD_4A for user A to HMD_4C for user C, notebook PC_4D for user D, smartphones_41A for user A to smartphones_41D for user D, video archive device 8, local system 9) are the same as those in Example 16 except for the points explained up to paragraph

[0225] .

[0213] The live video distribution system of this example includes local systems α_9α to ω_9ω as components, and these local systems are deployed in geographically remote areas from each other. In addition, each of the local systems α_9α to ω_9ω is provided with video distribution devices α_3α to ω_3ω, and a plurality of operators who operate the aerial photography drone using operator devices are deployed.

[0214] In addition, the signal exchange between the video distribution devices α_3α to ω_3ω and the HMDs 4A for user A to 4C for user C and the notebook PC 4D for user D is performed via the Internet 6 and the video archive device 8.

[0215] Hereinafter, the live video distribution method in this embodiment will be described with reference to FIGS. 25 and 26, taking user D who uses the notebook PC 4D for user D and the smartphone 41D for user D as an example.

[0216] User D operates the input means of the notebook PC 4D for user D and inputs an ID and a password. The input information is transmitted to the video archive device 8 via the smartphone 41D for user D, the 5G base station D_5D, and the 5G network and the Internet 6. As a result of receiving authentication, the notebook PC 4D for user D can access the video archive device 8.

[0217] On the display of the notebook PC 4D for user D that has accessed the video archive device 8, the live video selection screen shown in FIG. 21 is displayed. When user D operates the input means of the notebook PC 4D for user D and inputs the "distribution start time" and the "shooting area", the input information is transmitted to the video archive device 8 via the smartphone 41D for user D, the 5G base station D_5D, and the 5G network and the Internet 6. The video archive device 8 creates a list of corresponding live videos based on the flight plans of all aerial drones belonging to the local systems α_9α to ω_9ω, and transmits it to the notebook PC 4D for user D via the Internet 6, the 5G network, and the 5G base station D_5D and the smartphone 41D for user D. On the display of the notebook PC 4D for user D that has received the list, a list of corresponding live videos is displayed as shown in FIG. 6.

[0218] When user D operates the input means of the notebook PC_4D for user D and selects a live video that is already being distributed from the list, the screen transitions to the video display screen. On the other hand, when a live video whose distribution will start after that point in time is selected, the screen transitions to the standby screen, and when the streaming distribution of the selected live video starts, the screen transitions to the video display screen.

[0219] Hereinafter, until paragraph

[0225] , it will be described assuming that user D selects "Shiretoko drifting ice tour (2)", and the aerial photography drone α1_1α1 operated by operator α1 deployed in the local system α_9α shoots the live video.

[0220] When the scheduled distribution start time (17:50) arrives, the aerial photography drone α1_1α1 rotates each rotor based on the flight control signal transmitted from the drone controller operated by operator α1, ascends, and immediately moves to the sky above Shiretoko, and at the same time starts transmitting the "omnidirectional 3D live video signal that has shot the drifting ice that is the shooting target" and the "live audio signal that has collected the surrounding sound" to the video distribution device α_3α.

[0221] Also, while transmitting the omnidirectional 3D live video signal, the aerial photography drone α1_1α1 continuously generates a shooting position signal based on the measurement result of the GPS sensor it is equipped with, and wirelessly transmits it to the video distribution device α_3α together with the omnidirectional 3D live video signal.

[0222] The video distribution device α_3α converts the received omnidirectional 3D live video signal into a two-dimensional live video signal, and together with the received live audio signal, transmits it to the video archive device 8 via the Internet 6. At the same time, the received omnidirectional 3D live video signal and live audio signal are temporarily written as data in the storage means it is equipped with. At that time, the reception time of the omnidirectional 3D live video signal and live audio signal is taken as the shooting time, and after specifying the shooting position based on the shooting position signal transmitted from the aerial photography drone α1_1α1, it is written as video and audio data with the shooting time and shooting position.

[0223] Also, when the shooting by the aerial drone α1_1α1 is completed and the signal transmission ends, the video distribution device α_3α creates a video / audio data file from the temporary data and transmits the video / audio data file to the video archive device 8. The video archive device 8 stores the transmitted video / audio data file. Further, the video archive device 8 holds a list of the video / audio data files it stores, and updates the list of the video / audio data files each time it receives a data file.

[0224] On the other hand, the notebook PC_4D for User D starts receiving a two-dimensional live video signal and a live audio signal from the video archive device 8 via the Internet 6 and the 5G network, and the 5G base station D_5D and the smartphone_41D for User D, and switches the display screen of the display to a video display screen. In the video display screen, User D can display the live video in the display range that he / she wants to view by operating the input means of the notebook PC_4D for User D. Also, the audio based on the received live audio signal is played from the microphone speaker of the notebook PC_4D for User D.

[0225] The video ("Shiretoko drift ice tour (2)") taken by the aerial drone α1_1α1 can also be live-streamed to user devices used by users other than User D. The video archive device 8 counts the number of user devices that are currently live-streaming the video and the total number of user devices that have been distributed during the live-streaming distribution time. Among them, the number of user devices that are currently live-streaming is displayed in the "number of viewers" in FIG. 22, and the total number of user devices that have been distributed during the live-streaming distribution time is added as data to the video / audio data file stored in the video archive device 8.

[0226] Note that in this embodiment, the functions of the aerial drone, the HMD for the operator, the controller for drone operation, and the video distribution devices α_3α to ω_3ω that constitute the local system α_9α to ω_9ω are the same as those of the aerial drones (1)_11, aerial drones (2)_12, the HMD_211 for the operator (1), the HMD_212 for the operator (2), the controller (1)_221 for drone operation, the controller (2)_222 for drone operation, and the video distribution device 3 that constitute Embodiment 16. However, they can also be the same as those in Embodiment 17 or Embodiment 18.

[0227] Also, in this embodiment, the video archive device 8 has a function of transmitting the live video signal and the live audio signal to the user device. However, it is also possible to assign the function of transmitting the live video signal, the live audio signal, and the user device to a separately provided video distribution device.

Embodiment

[0228] FIG. 27 is a block diagram for explaining the configuration and functions of the live video distribution system and the local system that realize the live video distribution method according to Embodiment 25 of the present invention, and the functions of the video distribution device and the video archive device used in the live video distribution method.

[0229] The functions of the components of the live video distribution system in this embodiment (aerial drones (1)_11, aerial drones (2)_12, aerial drones (3)_13, video distribution device 3, HMD_4A for user A to HMD_4C for user C, notebook PC_4D for user D, smartphones_41A for user A to smartphones_41D for user D, video archive device 8, local system 9) are the same as those in Embodiment 16 except for the points explained up to paragraph

[0237] .

[0230] In this embodiment, three aerial drones (1)_11 to (3)_13 are deployed in the local system 9. Each of the aerial drones (1)_11 to (3)_13 has one CCD camera module, and the CCD camera module is installed on a three-way pan-tilt head. Operators (1) to (3) who operate each of the aerial drones (1)_11 to (3)_13 can rotate the three-way pan-tilt head around three rotation axes by operating drone control controllers (1)_221 to (3)_223 (not shown in the figure). Accordingly, tilting and panning of the CCD camera module in the front-rear and left-right directions can be realized, and images within a desired imaging range can be captured.

[0231] In this way, the imaging means mounted on each aerial drone captures the same building, which is the imaging target, from different viewpoints. The viewpoints (indicated by ★ in the figure) and lines of sight of the imaging means of each of the aerial drones (1)_11 to (3)_13 are shown in FIG. 28.

[0232] Each of the aerial drones (1)_11 to (3)_13 generates a two-dimensional video signal based on the result captured by the CCD camera module, and wirelessly transmits the two-dimensional video signal to the video distribution device 3 via the drone control controllers (1)_221 to (3)_223.

[0233] In addition, the video distribution device 3 simultaneously transmits the two-dimensional video signals received from each of the aerial drones (1)_11 to (3)_13 to the smartphones _41A for user A to _41D for user D via the Internet 6 and the 5G network.

[0234] Hereinafter, the live video distribution method in this embodiment will be described with reference to FIG. 29, taking user D who uses the notebook PC _4D for user D and the smartphone _41D for user D as an example.

[0235] On the display of the notebook PC_4D for User D, the following three types of live videos received via the smartphone_41D for User D can be displayed in parallel. Live Video 1... A live video based on the results captured by the aerial drone (1)_11 Live Video 2... A live video based on the results captured by the aerial drone (2)_12 Live Video 3... A live video based on the results captured by the aerial drone (3)_13

[0236] Also, User D can display only one of the live videos from Live Video 1 to Live Video 3 in full screen on the display of the notebook PC_4D for User D.

[0237] The switching between the above parallel display and full screen display can be realized by User D operating the input means of the notebook PC_4D for User D.

[0238] In this embodiment, the building is configured to be photographed from different viewpoints using a plurality of aerial drones. However, for example, it can also be configured to photograph a concert or sports event held at one venue from different viewpoints using a plurality of aerial drones.

[0239] Also, as in Embodiment 24, it is also possible to configure to simultaneously receive different live video signals from a plurality of on-site systems deployed in geographically remote areas, display the corresponding live videos in parallel, or select one of the live videos and display it in full screen.

Industrial Applicability

[0240] The present invention can be used in industries that manufacture unmanned mobile bodies such as unmanned aircraft (aerial photography drones, etc.), unmanned vehicles, and walking robots, as well as their peripheral devices and accessory devices, industries that manufacture video distribution devices, industries that manufacture user devices (especially video display means such as HMDs), and industries that manufacture video archive devices. It can also be used in industries that provide services related to observation using live video distribution methods (especially tourism services).

Explanation of Signs

[0241] 1....Aerial photography drone 11....Aerial photography drone (1) 12....Aerial photography drone (2) 13....Aerial photography drone (3) 1α1....Aerial photography drone α1 21....HMD for operator 211....HMD for operator (1) 212....HMD for operator (2) 22....Controller for drone operation 221....Controller for drone operation (1) 222....Controller for drone operation (2) 23....Antenna for drone operation 3....Video distribution device 3α....Video distribution device α 3β....Video distribution device β 3γ....Video distribution device γ 3ω....Video distribution device ω 4A....HMD for user A 4B....HMD for user B 4C....HMD for user C 4D....Notebook PC for user D 41A....Smartphone for user A 41B....Smartphone for user B 41C....Smartphone for user C 41D....Smartphone for user D 5....5G base station 5I....5G base station I 5II ······ 5G Base Station II 5III ······ 5G Base Station III 50 ······ 5G Base Station 0 5A ······ 5G Base Station A 5B ······ 5G Base Station B 5C ······ 5G Base Station C 5D ······ 5G Base Station D 6 ······ Internet 7A ······ ISP Server A 7B ······ ISP Server B 7C ······ ISP Server C 7D ······ ISP Server D 8 ······ Video Archive Device 9 ······ Local System 9α ······ Local System α 9β ······ Local System β 9γ ······ Local System γ 9ω ······ Local System ω

Claims

1. A live video distribution method for simultaneously transmitting live video signals of a target object from a video distribution device to user devices used by a plurality of users, and displaying video of the live video signals on display means of the user devices, comprising: the live video signal is generated based on a result of an image of the subject captured by an image capturing means provided on an unmanned moving body moving in a space near the subject, The unmanned moving body moves based on a movement control signal generated based on a result of an operator other than the plurality of users operating an operator device. A live video distribution method comprising:

2. All or some of the plurality of users are located in a location remote from the unmanned vehicle; the video distribution device receives a live video signal from the unmanned moving object, performs necessary processing on the live video signal, and transmits the live video signal to a user device used by a user located at the remote location via a public network; The live video distribution method according to claim 1 .

3. There are a plurality of the unmanned moving bodies, The plurality of unmanned moving bodies simultaneously photograph substantially the same object from different viewpoints using the photographing means provided therein, A plurality of live video signals having different display ranges are generated based on the results of the analysis and are then simultaneously transmitted from the video distribution device to the user device.

3. The live video distribution method according to claim 1 or 2.

4. Each of the plurality of users selects "the user device that the user uses will receive a live video signal generated based on the result of imaging by an imaging means of which of the plurality of unmanned moving bodies," and displays an image of the selected live video signal on a display means provided in the user device. The live video distribution method according to claim 3 .

5. There are a plurality of the unmanned moving bodies, The plurality of unmanned vehicles are deployed in a plurality of areas remote from each other, Each of the plurality of users selects "in which of the plurality of regions the user device that he / she uses will receive a live video signal generated based on the result of imaging by an imaging means of an unmanned moving body deployed in that region," and displays an image of the selected live video signal on a display means provided in the user device.

5. The live video distribution method according to claim 1,

6. The imaging means included in the unmanned moving body captures wide-angle images including panoramic images, omnidirectional images, and spherical images; a display means for displaying an image of a predetermined display range (hereinafter, abbreviated as "predetermined display range image") cut out from the wide-angle image; The live video distribution method according to any one of claims 1 to 5.

7. A live pilot voice signal, which is a collection of voices emitted by a pilot using the pilot device or a person in the vicinity of the pilot, is transmitted from a video distribution device directly or via a public network to the plurality of user devices simultaneously, and the live voices are simultaneously emitted by the sound emission means of the plurality of user devices. The live video distribution method according to any one of claims 1 to 6.

8. All or some of the plurality of user devices have a sound collection means for collecting a sound uttered by a user who uses the user device, A superimposed audio signal obtained by superimposing all or a part of the live user audio signal based on the results of the sound collection by the sound collection means of the user device and the live pilot etc. audio signal is simultaneously transmitted from a video distribution device directly or via a public network to the multiple user devices, and the live audio is simultaneously emitted by the sound emission means of the multiple user devices. The live video distribution method according to claim 7 .

9. A video distribution device used in the live video distribution method according to any one of claims 1 to 8, receiving live video signals from the unmanned moving object, performing necessary processing on the live video signals, and transmitting the live video signals simultaneously to the plurality of user devices directly or via a public network; A video distribution device comprising:

10. A video distribution device used in the live video distribution method according to any one of claims 3 to 8, receiving "live video signals having different imaging ranges" from each of the plurality of unmanned moving bodies, performing necessary processing on each of the "live video signals having different imaging ranges", and then transmitting the signals to the plurality of user devices simultaneously, directly or via a public network; A video distribution device comprising:

11. A video distribution device used in the live video distribution method according to any one of claims 4 to 8, receiving, from the user device, an unmanned vehicle selection signal based on a selection result of a user using the user device; determining, based on the unmanned vehicle selection signal, to which user device the live video signals received from the plurality of unmanned vehicles are to be transmitted; 11. The video distribution device according to claim 9 or 10.

12. A video distribution device used in the live video distribution method according to any one of claims 5 to 8, receiving, from the user device, a region selection signal based on a selection result of a user using the user device; determining to which user device the live video signals received from the plurality of unmanned moving objects are to be transmitted based on the region selection signal; 12. The video distribution device according to claim 9, wherein the video distribution device is a video distribution device that includes a video distribution unit.

13. A video distribution device used in the live video distribution method according to any one of claims 6 to 8, receiving a wide-angle image signal from the unmanned vehicle, performing necessary processing on the wide-angle image signal, and transmitting the wide-angle image signal directly or via a public network to the plurality of user devices simultaneously; 13. The video distribution device according to claim 9, wherein the video distribution device is a video distribution device that includes a video distribution unit.

14. generating a predetermined range image signal transmitting a predetermined display range image extracted from the wide-angle image, and transmitting the predetermined range image signal to a user device; 14. The video distribution device according to claim 13.

15. A video distribution device used in the live video distribution method according to claim 7 or 8, transmitting the live pilot or other audio signals simultaneously to the plurality of user devices, either directly or via a public network; 15. The video distribution device according to claim 10, wherein the video distribution device is a video distribution device that includes a video distribution unit.

16. A video distribution device used in the live video distribution method according to claim 8, receiving live user voice signals from all or a portion of the plurality of user devices, generating a superimposed voice signal by superimposing all or a portion of the live user voice signals and the live pilot etc. voice signal, and transmitting the superimposed voice signal simultaneously to the plurality of user devices directly or via a public network; 16. The video distribution device according to claim 15.

17. starting recording of the video data transmitted by the live video signal substantially simultaneously with the start of transmission of the live video signal, and storing the video data as a video data file substantially simultaneously with the end of transmission of the live video signal; 17. The video distribution device according to claim 10, wherein the video distribution device is a video distribution device that includes a video distribution unit.

18. A video archive device that stores a video data file generated by the video distribution device according to claim 17.

Images