Performance event execution method and relay device used in the performance event execution method

Abstract

To provide a performance event execution method which enables a player to perceive in real time reactions of a remote viewer located in a remote place from an event venue and enables the remote viewer to perceive in real time reactions of other remote viewers.SOLUTION: Each of remote viewers possesses a terminal device for remote viewer. An avatar device generates a live video signal and a live sound signal on the condition that a plurality of avatar devices associated with the respective remote viewers are installed in an event venue. The terminal device for remote viewer displays a live video on the basis of the live video signal and emits live sound on the basis of the live sound signal, whereas the terminal devices for remote viewers generate signals based on a result of sensing operation and / or motions of remote viewers who possess the terminal devices for the remote viewers. The avatar device configures at least one of exercises, vibrations, light emission, sound emission, and image display on the basis of the remote viewers' operation / motion signals.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a performance event implementation method in which a large number of spectators watch "a performance given by players at an event venue." [Background technology]

[0002] In order to prevent the spread of the new coronavirus disease (COVID-19), performance events such as sporting events and concerts, where a large number of spectators watch "performances (matches or performances) by players (athletes or musicians) at an event venue," have been canceled or postponed.

[0003] In an effort to alleviate this situation, performance events are being organised as games or performances without spectators, with these being broadcast live on television and the internet.

[0004] However, in cases where spectators of a performance event do not go to the event venue (a "stadium" or "arena" in the case of a sporting event, or a "concert hall" in the case of a concert), but instead watch from a location remote from the event venue, such as a match or performance without spectators, the players are unable to perceive in real time the reactions of spectators (hereinafter referred to as "remote spectators") who are located remote from the event venue (such as "applause," "cheers," "cheering," "jeers," and "booing" in the case of a sporting event, and "applause" and "cheers" in the case of a concert), and remote spectators are unable to perceive the reactions of other remote spectators in real time, resulting in a lack of excitement for the performance event.

[0005] In addition, in normal performance events, event organizers can collect admission fees from spectators to the event venue (a "stadium" or "arena" in the case of a sporting event, or a "concert hall" in the case of a concert).However, in the case of games or performances without spectators, they cannot collect admission fees from remote spectators, which means they are unable to make a profit from holding the event, let alone cover the necessary costs. Summary of the Invention [Problem to be solved by the invention]

[0006] The present invention has been made in consideration of the above circumstances, and has as its object to provide a method for implementing a performance event that enables players to perceive the reactions of remote spectators in real time, and also enables remote spectators to perceive the reactions of other remote spectators in real time.

[0007] Another object of the present invention is to provide a method for implementing a performance event that enables an event organizer to reliably receive compensation from remote spectators for viewing the event. [Means for solving the problem]

[0008] In order to achieve the above object, a first aspect of the present invention relates to a performance event implementation method, A performance event implementation method in which a large number of spectators watch a "performance performed by a player at an event venue," comprising: The majority of the spectators are located at a location remote from the venue of the event; Each of the viewers located at a location remote from the venue of the event (hereinafter abbreviated as "remote viewers") has a remote viewer terminal device; A plurality of avatar devices corresponding to each of the remote viewers are installed at the event venue, The avatar device generates a live video signal based on the results of imaging by its own imaging means and generates a live audio signal based on the results of sound collection by its own sound collection means, and the remote viewer terminal device displays live video on its own display means based on the live video signal generated by the avatar device associated with the remote viewer who owns it, and emits live audio from its own sound emission means based on the live audio signal generated by the avatar device associated with the remote viewer who owns it, The remote viewer terminal device generates a signal (hereinafter abbreviated as "remote viewer operation / movement signal") based on the result of sensing the operation and / or movement (including vocalization) of the remote viewer who possesses the remote viewer terminal device, and the avatar device realizes at least one of movement, vibration, light emission, sound emission, and image display based on the remote viewer operation / movement signal generated by the remote viewer terminal device possessed by the remote viewer with which it is associated. It was configured as follows.

[0009] In order to achieve the above object, a second aspect of the present invention relates to a performance event implementation method, A performance event implementation method in which a large number of spectators watch a "performance performed by a player at an event venue," comprising: The majority of the spectators are located at a location remote from the venue of the event; Each of the viewers located at a location remote from the venue of the event (hereinafter abbreviated as "remote viewers") has a remote viewer terminal device; A plurality of avatar devices corresponding to each of the remote viewers are installed at the event venue, The avatar device generates a live video signal based on the results of imaging by its own imaging means and generates a live audio signal based on the results of sound collection by its own sound collection means, and the remote viewer terminal device displays live video on its own display means based on the live video signal generated by the avatar device associated with the remote viewer who owns it, and emits live audio from its own sound emission means based on the live audio signal generated by the avatar device associated with the remote viewer who owns it, the remote viewer terminal device generates an image signal (hereinafter abbreviated as "remote viewer external appearance image signal") based on the result of photographing the external appearance of the remote viewer who owns the remote viewer terminal device, and the avatar device displays the external appearance image of the remote viewer based on the remote viewer external appearance image signal generated by the remote viewer terminal device owned by the remote viewer with which the avatar device corresponds; It was configured as follows.

[0010] In order to achieve the above object, a third aspect of the present invention relates to a relay device used in a performance event implementation method, A relay device for use in the performance event implementation method of the first or second aspect, comprising: receiving the live video signal and the live audio signal from the avatar device, performing necessary processing on the live video signal and the live audio signal, and transmitting the signals via a public network to a remote viewer terminal device associated with the avatar device that is the source of the signals; receiving a remote viewer operation / motion signal and / or a remote viewer external image signal from the remote viewer terminal device via a public network, carrying out necessary processing on the remote viewer operation / motion signal and / or the remote viewer external image signal, and transmitting the signal to an avatar device associated with the remote viewer terminal device that is the source of the signal; It was configured as follows. Effect of the Invention

[0011] According to the first embodiment of the performance event implementation method of the present invention and the relay device used in the performance event implementation method, the remote spectators can clap, cheer, and send support while enjoying the live video displayed on the display means of the remote spectator terminal device owned by the remote spectator and the live sound emitted from the sound emitting means. They can also "jeer" and boo. The remote spectator terminal device owned by the remote spectator detects these actions (including vocalization) and generates a signal based on the detected result. The remote spectator can express his / her own emotions such as sympathy, antipathy, and excitement by operating the buttons and levers of the remote spectator terminal device owned by the remote spectator while enjoying the live video and live sound. The remote spectator terminal device detects these operations and generates a signal based on the detected result. Meanwhile, each of the multiple avatar devices installed at the event venue realizes at least one of movement, vibration, light emission, sound emission, and image display based on the remote spectator operation / action signal generated by the remote spectator terminal device associated with the remote spectator. As a result, the player can perceive the reactions of the remote spectators by observing the movements, vibrations, light emission, sound emission, and image display of the multiple avatar devices installed at the event venue. Also, the remote spectators can perceive the reactions of other remote spectators by viewing and listening to the live video and live audio of the movements, vibrations, light emission, sound emission, and image display of the multiple avatar devices installed at the event venue through the display means and sound emission means of their own remote spectator terminal devices.

[0012] According to the second embodiment of the performance event implementation method and the relay device used in the performance event implementation method, the remote spectator can express facial expressions such as laughter and anger and move his / her body while enjoying the live video displayed on the display means of the remote spectator terminal device owned by the remote spectator and the live sound emitted from the sound emitting means. The remote spectator terminal device owned by the remote spectator captures an appearance image including these facial expressions and movements, and generates a remote spectator appearance image signal based on the captured result. Meanwhile, each of the multiple avatar devices installed at the event venue displays the appearance image of the remote spectator based on the remote spectator appearance image signal generated by the remote spectator terminal device associated with the remote spectator. As a result, the player can perceive the reaction of the remote spectator by observing the appearance images of the remote spectators displayed on the multiple avatar devices installed at the event venue. The remote spectator can perceive the reaction of other remote spectators by viewing the appearance images of the remote spectators displayed on the multiple avatar devices installed at the event venue through the display means of the remote spectator terminal device owned by the remote spectator.

[0013] Furthermore, according to the performance event implementation method of the present invention and the relay device used in the performance event implementation method, it is possible to transmit live video signals and live image signals only to remote spectator terminal devices owned by remote spectators that are associated with the multiple avatar devices installed at the event venue. Therefore, by limiting the association of the avatar devices to only remote spectator terminal devices owned by remote spectators who have paid or promised to pay the viewing fee in advance, it is possible to reliably obtain compensation for viewing the performance event from the remote spectators. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a block diagram for explaining the configuration and functions of a system for realizing a performance event implementation method according to a first embodiment of the present invention, and the functions of a relay device used in the performance event implementation method. [Diagram 2] 1 is an external view illustrating the configuration and functions of a remote spectator terminal device used in a performance event implementation method according to a first embodiment of the present invention. FIG. [Diagram 3] 1 is an external view illustrating the configuration and function of an avatar device used in a performance event implementation method according to a first embodiment of the present invention. FIG. [Figure 4] 1A to 1C are external views illustrating the movement of an avatar device used in a performance event implementation method in accordance with a first embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The present invention is not limited to the embodiments, and various modifications are possible within the scope of the technical concept. EXAMPLES

[0016] FIG. 1 is a block diagram for explaining the configuration and functions of a system for realizing a performance event implementation method according to a first embodiment of the present invention, and the functions of a relay device used in the performance event implementation method, and in particular, is a block diagram for explaining a situation in which remote spectators A to X who are located remotely from the event venue are watching a "concert of a three-member female idol group being held at an outdoor event venue." Fig. 2 is an external view for explaining the configuration and functions of a terminal device for remote viewers used in the performance event implementation method according to the first embodiment of the present invention. Fig. 3 is an external view for explaining the configuration and functions of an avatar device used in the performance event implementation method according to the first embodiment of the present invention.

[0017] The system for realizing the performance event implementation method of this embodiment is composed of avatar devices A_1A through X_1X, a relay device 2, and terminal devices _4A through _4X for remote viewer A through remote viewer X used by each of the remote viewers A through X, and the relay device 2 and terminal devices _4A through _4X for remote viewer A through remote viewer X are connected to the Internet.

[0018] Each of the avatar devices A_1A to X_1X is composed of a drone 11, an imaging means 12, and a tablet 13, and the imaging means 12 and the tablet 13 are fixed to the drone 11.

[0019] The drone 11 is a multicopter type unmanned aerial vehicle, and by controlling the rotation speed of each rotor, it can hover above a specific area such as a tourist spot and perform straight-line motion, circular motion, and rotational motion.

[0020] The imaging means 12 also includes two omnidirectional cameras that cover a viewing angle (solid angle) of 360 degrees, and captures the performance (in this embodiment, "singing and dancing") of the players (in this embodiment, a "three-member female idol group") and other scenes at the event venue. Furthermore, the imaging means 1 generates a omnidirectional 3D live video signal based on the results captured by the two omnidirectional cameras, and wirelessly transmits the omnidirectional 3D live video signal to the relay device 2.

[0021] The tablet 13 also includes a display panel for displaying images, a speaker for emitting sound, and a microphone for collecting sounds from the event venue, including sounds (in this embodiment, "singing voices") emitted by the players (in this embodiment, "a three-person female idol group"). The tablet 13 also generates a live audio signal based on the results of imaging by the microphone, and wirelessly transmits the live audio signal to the relay device 2.

[0022] Before the performance event is held, the relay device 2 accepts applications from remote spectators to view the event, and associates the remote spectators with avatar devices based on the applications. In this embodiment, the avatar devices A_1A through X_1X correspond one-to-one to the remote spectators A through X, respectively, but it is also acceptable for the "number of remote spectators who have applied to view the event" to be less than the "number of avatar devices installed at the event venue" or for multiple remote spectators to share one avatar device.

[0023] The relay device 2 performs necessary processing on the spherical 3D live video signal received from the imaging means 12 of the avatar device A_1A and the live audio signal received from the tablet 13 of the avatar device A_1A, and then transmits the signals to terminal device _4A for remote viewer A via the Internet 3. Similarly, the relay device 2 transmits signals received from avatar devices B_1B to X_1X to terminal device _4B for remote viewer B to terminal device _4X for remote viewer X.

[0024] On the other hand, each of the terminal devices _4A for remote viewer A to _4X for remote viewer X is composed of an HMD (Head Mounted Display) 41, headphones 42, a controller 43, a Web camera 44, and a signal processing and communication unit 45, and the HMD (Head Mounted Display) 41, headphones 42, controller 43, Web camera 44, and signal processing and communication unit 45 transmit and receive signals between them.

[0025] Meanwhile, the HMD 41 is equipped with sensors that detect the movements of the head and eyes of the remote viewer wearing the HMD 41, and the sensors transmit signals based on the detection results of the respective sensors to the signal processing and communication unit 45. The signal processing and communication unit 45 sets the display ranges of the images to be projected from the left and right projection units, generates a 3D live video signal from the omnidirectional 3D live video signal received from the relay device 2 via the Internet 3 based on the display range setting results, and transmits the 3D live video signal to the HMD 41. The HMD 41 projects images from the left and right projection units based on the 3D live video signal received from the signal processing and communication unit 45. As a result, each of the remote viewers A to X can view the "live video of the performance event" in a field of view corresponding to the movement of their own head and eyes.

[0026] The signal processing and communication unit 45 performs necessary processing on the live audio signal received from the relay device 2 via the Internet 3 and transmits it to the headphones 42. The headphones 42 emit sound based on the live audio signal received from the signal processing and communication unit 45. As a result, each of the remote viewers A to X can hear the "live audio of the performance event."

[0027] While watching the live video and listening to the live audio, each of the remote viewers A to X operates a controller 43. The controller 43 is a radio-controlled joystick controller, which generates a flight control signal based on a lever or dial operation by the pilot and transmits the flight control signal to a signal processing and communication unit 45.

[0028] The signal processing and communication unit 45 performs necessary processing on the flight control signal received from the controller 43 and then transmits it to the relay device 2 via the Internet 3 .

[0029] The relay device 2 performs the necessary processing on the flight control signal received from the signal processing and communication unit 45 via the Internet 3, and then transmits it to the drone 11 of the avatar device (any of the avatar devices A_1A to X_1X) associated with a remote viewer who owns a terminal device for remote viewer (any of terminal device _4A for remote viewer A to terminal device _4X for remote viewer X) that includes the signal processing and communication unit 45 that is the source of the flight control signal.

[0030] The rotation speed of the rotor of drone 11 is controlled based on the received flight control signal, and as a result, drone 11 flies as intended by the remote pilot with which the avatar device including drone 11 (any of avatar devices A_1A to X_1X) is associated.

[0031] On the other hand, the Web camera 44 captures an external image of the remote viewer, generates a video signal based on the captured image, and transmits the video signal to the signal processing and communication unit 45. The Web camera 44 also includes a microphone that collects sounds such as voices and applause from the remote viewer, generates an audio signal based on the collected sounds, and transmits the video signal to the signal processing and communication unit 45.

[0032] The signal processing and communication unit 45 performs necessary processing on the video signal and audio signal received from the controller 43 , and then transmits them to the relay device 2 via the Internet 3 .

[0033] The relay device 2 performs the necessary processing on the video signal and audio signal received from the signal processing and communication unit 45 via the Internet 3, and then transmits the video signal and audio signal to the tablet 13 of the avatar device (any of the avatar devices A_1A to X_1X) associated with a remote viewer who owns a terminal device for remote viewer (any of terminal device _4A for remote viewer A to terminal device _4X for remote viewer X) that includes the signal processing and communication unit 45 that is the source of the video signal and audio signal.

[0034] Based on the received video signal, tablet 13 displays on its display panel the external image of the remote operator with whom the avatar device including itself (any of avatar devices A_1A to X_1X) is associated, and based on the received audio signal, emits on its display panel the sound, such as the voice or applause, made by the remote operator with whom the avatar device including itself (any of avatar devices A_1A to X_1X) is associated.

[0035] On the other hand, the signal processing and communication unit 45 determines the psychological state (excitement level, types of joy, anger, sadness, happiness, etc.) of a remote viewer who owns a terminal device for remote viewer (any of terminal device _4A for remote viewer A to terminal device _4X for remote viewer X) that includes the signal processing and communication unit 45 based on a signal based on the ``movement of the head and eye balls of the remote viewer'' received from the sensor of the HMD 41 and / or a video signal and / or an audio signal received from the Web camera 44, and generates an excitement level signal and a joy, anger, sadness, or happiness signal based on the determination result, and transmits the excitement level signal and the joy, anger, sadness, or happiness signal to the relay device 2 via the Internet 3.

[0036] The relay device 2 performs the necessary processing on the excitement signal and the joy, anger, sadness, and happiness signals received from the signal processing and communication unit 45 via the Internet 3, and then transmits them to the tablet 13 of the avatar device (any of the avatar devices A_1A to X_1X) associated with a remote viewer who owns a terminal device for remote viewer (any of terminal device _4A for remote viewer A to terminal device _4X for remote viewer X) that includes the signal processing and communication unit 45 that is the source of the excitement signal and the joy, anger, sadness, and happiness signals.

[0037] The tablet 13 changes the luminance of the display panel and the color tone of the image displayed based on the received excitement signal and the joy, anger, sadness, and happiness signal. That is, when the intensity of the received excitement signal is high, the luminance of the display panel is increased, and when the intensity of the received excitement signal is low, the luminance of the display panel is decreased. In addition, when the joy, anger, sadness, and happiness signal corresponding to a positive emotion such as joy or pleasure is received, the image displayed on the display panel is displayed in warm colors, and when the joy, anger, sadness, and happiness signal corresponding to a positive emotion such as anger or sadness is received, the image is displayed in cool colors.

[0038] In addition, the relay device 2 performs the necessary processing on the excitement level signal received from the signal processing and communication unit 45 via the Internet 3, and then transmits the excitement level signal to the drone 11 of the avatar device (any of the avatar devices A_1A to X_1X) associated with a remote viewer who owns a terminal device for remote viewer (any of terminal device _4A for remote viewer A to terminal device _4X for remote viewer X) that includes the signal processing and communication unit 45 that is the source of the excitement level signal.

[0039] Based on the received excitement level signal, the drone 11 performs one of the movements of "axial rotation vibration", "up-down vibration", and "left-right vibration" or a combination of these movements as shown in Fig. 4. In this case, if the intensity of the received excitement level signal is high, the amplitude of the vibration is increased, and if the intensity of the received excitement level signal is low, the amplitude of the vibration is decreased or the vibration (movement) is stopped. [Industrial Applicability]

[0040] The present invention can be used in industries that provide services related to performance events such as sports, concerts, etc. Also, the present invention can be used in industries that manufacture servers and the like used as relay devices. [Explanation of symbols]

[0041] 1A‥‥‥Duplicate device A 1B‥‥‥Duplicate device B 1C‥‥‥Duplicate device C 1D‥‥‥Duplicate device D 1E‥‥‥Duplicate device E 1X‥‥‥Duplicate device X 11. Drone 12. Imaging means 13. Tablet 2. Relay device 3. Internet 4A: Terminal device for remote viewer A 4B: Terminal device for remote viewer B 4C...Terminal device for remote viewer C 4D...Terminal device for remote viewers D 4E: Terminal device for remote viewer E 4X...Terminal device for remote viewer X 41. HMD 42. Headphones 43. Controller 44. Web camera 45. Signal processing and communication unit

Claims

1. A performance event implementation method in which a large number of spectators watch a "performance performed by a player at an event venue," comprising: The majority of the spectators are located at a location remote from the venue of the event; Each of the viewers located at a location remote from the venue of the event (hereinafter abbreviated as "remote viewers") has a remote viewer terminal device; A plurality of avatar devices corresponding one-to-one to each of the remote viewers and a relay device are installed at the event venue, The avatar device generates a live video signal based on the results of imaging by its own imaging means and generates a live audio signal based on the results of audio collection by its own audio collection means, and the remote viewer terminal device displays live video on its own display means based on the live video signal generated by the avatar device associated with the remote viewer who owns it, and emits live audio from its own audio emission means based on the live audio signal generated by the avatar device associated with the remote viewer who owns it, the remote viewer terminal device generates a signal (hereinafter abbreviated as "remote viewer operation / action signal") based on the result of sensing the operation and / or movement (including vocalization) of the remote viewer who possesses the remote viewer terminal device, and the avatar device realizes at least one of movement, vibration, light emission, sound emission, and image display based on the remote viewer operation / action signal generated by the remote viewer terminal device possessed by the remote viewer with which the avatar device corresponds; the avatar device associates the remote viewer terminal device with the avatar device, receiving the live video signal and live audio signal from the avatar device, performing necessary processing on the live video signal and live audio signal, and transmitting them via a public network to a remote viewer terminal device associated with the avatar device that originated the signals; and receiving a remote viewer operation / motion signal from the remote viewer terminal device via the public network, performing necessary processing on the remote viewer operation / motion signal, and transmitting it to the avatar device associated with the remote viewer terminal device that originated the signals. A method for implementing a performance event.

2. A performance event implementation method in which a large number of spectators watch a "performance performed by a player at an event venue," comprising: The majority of the spectators are located at a location remote from the venue of the event; Each of the viewers located at a location remote from the venue of the event (hereinafter abbreviated as "remote viewers") has a remote viewer terminal device; A plurality of avatar devices corresponding one-to-one to each of the remote viewers and a relay device are installed at the event venue, The avatar device generates a live video signal based on the results of imaging by its own imaging means and generates a live audio signal based on the results of audio collection by its own audio collection means, and the remote viewer terminal device displays live video on its own display means based on the live video signal generated by the avatar device associated with the remote viewer who owns it, and emits live audio from its own audio emission means based on the live audio signal generated by the avatar device associated with the remote viewer who owns it, the remote viewer terminal device generates an image signal (hereinafter abbreviated as "remote viewer external appearance image signal") based on the result of photographing the external appearance of the remote viewer who owns the remote viewer terminal device, and the avatar device displays an external appearance image of the remote viewer based on the remote viewer external appearance image signal generated by the remote viewer terminal device owned by the remote viewer with which the avatar device corresponds, the avatar device associates the remote viewer terminal device with the avatar device, receiving the live video signal and live audio signal from the avatar device, performing necessary processing on the live video signal and live audio signal, and transmitting them via a public network to a remote viewer terminal device associated with the avatar device that originated the signals; and receiving a remote viewer external image signal from the remote viewer terminal device via the public network, performing necessary processing on the remote viewer external image signal, and transmitting it to the avatar device associated with the remote viewer terminal device that originated the signals. A method for implementing a performance event.

3. A relay device used in the performance event implementation method according to claim 1 or 2, comprising: Correlating the remote viewer terminal device with the avatar device; receiving the live video signal and the live audio signal from the avatar device, performing necessary processing on the live video signal and the live audio signal, and transmitting the signals via a public network to a remote viewer terminal device associated with the avatar device that is the source of the signals; receiving a remote viewer operation / motion signal and / or a remote viewer external appearance image signal from the remote viewer terminal device via a public network, performing necessary processing on the remote viewer operation / motion signal and / or the remote viewer external appearance image signal, and transmitting the signal to an avatar device associated with the remote viewer terminal device that is the source of the signal; A relay device comprising:

Images