Reaction presentation control device and method

Abstract

In one aspect of the present invention, when data related to an event to be executed in a virtual space is distributed to a terminal of a participant, information indicating the action state of the participant is acquired during the distribution of the data related to the event; and, on the basis of the acquired information indicating the action state of the participant, a support preparation behavior indicating a sign of a support behavior of the participant is determined. Then, according to the determination result of the support preparation behavior, data indicating a reaction, to the participant, of a performer who appears in the event is generated, and the generated data indicating the reaction is distributed to the terminal of the participant after the end of the support behavior of the participant.

Description

Reaction presentation control device and method

[0001] One aspect of the present invention relates to a reaction presentation control device and method used in a system for holding a live event in a virtual space, for example.

[0002] In recent years, the number of live events using virtual spaces has been increasing. One of the joys of live events in real space is the real-time interaction between performers and participants, and attempts are being made to realize this interaction in live events in virtual space.

[0003] For example, there is a method in which the cheering behavior of participants is predicted in advance in the event script, and the performers react to the cheering behavior in time with the cheering behavior. However, some participants do not necessarily perform the cheering behavior according to the event script. In particular, people who are participating in an event for the first time may not be familiar with the cheering behavior at the event, and it is extremely unnatural to present the performers' reactions to such participants.

[0004] Other proposed techniques include preparing multiple reaction patterns and presenting them while switching between them, and dynamically generating reactions. For example, Non-Patent Document 1 describes a technique for seamlessly changing objects by morphing in computer animation.

[0005] "An Overview of Computer Animation and Its Uses | Unity," [Retrieved May 2, 2024], Internet <URL: https: / / unity.com / ja / solutions / computer-animation-explained>

[0006] However, morphing creates a seamless change between two objects by inserting interpolation frames between key frames. Therefore, if the amount of change between the two objects is large, it takes time to calculate this change, and the number of frames to be inserted increases, so it takes a sufficient amount of time to seamlessly change the objects, which creates the problem of impairing the immediacy of the reaction.

[0007] The present invention has been made in light of the above circumstances, and aims to provide a technique that enables reaction data to be presented smoothly without impairing immediacy.

[0008] In order to solve the above problems, one aspect of the reaction presentation control device or method of the present invention, when distributing data related to an event held in a virtual space to the terminals of the participants, acquires information representing the motion states of the participants while the data related to the event is being distributed, and determines cheer preparation behavior indicating signs of cheering behavior by the participants based on the acquired information representing the motion states of the participants. Then, in accordance with the determination result of the cheer preparation behavior, generates data representing the reactions of performers appearing in the event toward the participants, and distributes the generated data representing the reactions to the participants' terminals after the participants have finished cheering behavior.

[0009] According to one aspect of the present invention, it is possible to present data representing the reaction of the performer to the participants at an appropriate timing after the completion of the cheering action of the participants. In other words, it is possible to present the reaction of the performer to the cheering action of the participants to the participants at a natural timing without losing immediacy.

[0010] That is, according to one aspect of the present invention, it is possible to provide a technology that allows reaction data to be presented smoothly without impairing immediacy.

[0011] Fig. 1 is a diagram showing an example of a virtual live event distribution system according to an embodiment of the present invention. Fig. 2 is a block diagram showing an example of the hardware configuration of a reaction presentation control device provided in the virtual live event distribution system shown in Fig. 1. Fig. 3 is a block diagram showing an example of the software configuration of the reaction presentation control device provided in the virtual live event distribution system shown in Fig. 1. Fig. 4 is a flowchart showing an example of the processing procedure and processing content of reaction presentation control executed by a control unit of the reaction presentation control device shown in Fig. 3. Fig. 5 is a timing chart for explaining an example of the reaction presentation control operation shown in Fig. 4. Fig. 6 is a diagram showing an example of a determination table used to determine cheering preparation behavior of a participant participating in a live event.

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0013] [One Embodiment] (Configuration Example) (1) System FIG. 1 is a diagram showing an example of a virtual live event distribution system according to one embodiment of the present invention.

[0014] A virtual live event distribution system according to one embodiment includes a reaction presentation control device SV, which enables communication of information data between the reaction presentation control device SV and multiple user terminals UT1 to UTn used by participants via a network NW.

[0015] The user terminals UT1 to UTn are configured, for example, by personal computers and equipped with a browser function for receiving event data. The user terminals UT1 to UTn also have multiple types of sensors, such as a camera, microphone, and vital signs sensor, for detecting the operating status of participants while they are viewing the event data.

[0016] The camera is used to capture images of the participants' faces or limb movements. The microphone is used to collect the participants' voices. The vital sensors are used to measure vital data such as the participants' heart rate, sweating, and breathing rate. The user terminals UT1 to UTn may be, for example, mobile terminals such as smartphones or tablet-type information terminals, or television receivers with built-in browser functionality, and the sensors may be either built into the user terminals UT1 to UTn or externally attached.

[0017] (2) Reaction Presentation Control Device SV FIGS. 2 and 3 are block diagrams showing examples of the hardware and software configurations of the reaction presentation control device SV, respectively.

[0018] The reaction presentation control device SV is composed of a server computer located on the web or the cloud, for example. The reaction presentation control device SV includes a control unit 1 having a hardware processor, and a storage unit including a program storage unit 2 and a data storage unit 3, and a communication I / F unit 4 connected to the control unit 1 via a bus 5.

[0019] The communication I / F unit 4 transmits and receives information data to and from the user terminals UT1 to UTn in accordance with a communication protocol defined by the network NW. In one embodiment, the communication I / F unit 4 distributes event data to the user terminals UT1 to UTn and receives sensing data transmitted from the user terminals UT1 to UTn.

[0020] The program storage unit 2 is, for example, a combination of a non-volatile memory such as a hard disk drive (HDD) or a solid state drive (SSD) as a storage medium that can be written to and read from at any time, and a non-volatile memory such as a read only memory (ROM), and stores application programs necessary for executing various processes related to one embodiment of the present invention, in addition to middleware such as an operating system (OS).

[0021] The data storage unit 3 is a combination of a non-volatile memory such as an HDD or SSD that can be written to and read from at any time, and a volatile memory such as a RAM (Random Access Memory), and its storage area includes an event data storage unit 31, a judgment table storage unit 32, a participant position information storage unit 33, and a reaction basic data storage unit 34. The storage area also includes a buffer area for temporarily storing data when the control unit 1 executes various processes.

[0022] Event data for live distribution, for example, created in advance, is stored in the event data storage unit 31. The event data includes, for example, time-series data of motion capture generated based on the movements of performers in an event held in real space, and sound data.

[0023] The determination table storage unit 32 stores threshold values ​​for determining whether a movement corresponds to cheering preparation, in association with changes in multiple types of movement of the participant detected by the sensor.

[0024] The participant position information storage unit 33 stores the position information of participants in the virtual space during the event distribution.

[0025] The reaction basic data storage unit 34 stores a plurality of types of reaction basic data used to generate the reaction data of the performers.

[0026] The control unit 1 has the processing functions necessary to realize one embodiment of the present invention, including an event data distribution processing unit 11, a participant action information acquisition processing unit 12, a cheering action determination processing unit 13, a participant position information management processing unit 14, a reaction possibility determination processing unit 15, and a reaction data generation processing unit 16.

[0027] Each of the processing units 11 to 16 is realized by causing a hardware processor in the control unit 1 to execute an application program stored in the program storage unit 2. Note that some or all of the processing units 11 to 16 may be realized using hardware such as an LSI (Large Scale Integration) or an ASIC (Application Specific Integrated Circuit).

[0028] The event data distribution processing unit 11 distributes event data stored in the event data storage unit 31 from the communication I / F unit 4 via the network NW to user terminals UT1 to UTn used by participants who have registered to participate in a live event held in a virtual space during the event period.

[0029] The participant motion information acquisition processing unit 12 receives sensing data representing the motion status of participants detected by the group of sensors while participating in the live event from the user terminals UT1 to UTn via the network NW through the communication I / F unit 4.

[0030] The cheering behavior determination processor 13 determines whether or not the acquired change in the motion state of the participant corresponds to cheering preparation, based on a determination threshold defined in a determination table stored in the determination table storage unit 32. At the same time, the cheering behavior determination processor 13 determines the type of cheering behavior and the duration or predicted end time of the cheering behavior based on the change in the motion state of the participant.

[0031] The participant position information management processing unit 14 manages the position information of each participant within the virtual space where the live event is taking place, and stores the position information in the participant position information storage unit 33 in association with the participant's identification information (participant ID).

[0032] When the cheering action determination processing unit 13 determines that a participant has begun cheering preparation actions, the reaction possibility determination processing unit 15 determines whether the performer is able to react to the participant's cheering action based on the possibility in terms of the performance defined by the event data and consistency with the performer's current acting actions.

[0033] When the reaction possibility determination processing unit 15 determines that the performer is capable of reacting, the reaction data generation processing unit 16 generates reaction data based on the basic reaction data stored in the basic reaction data storage unit 34, taking into consideration the behavioral state of the performer immediately before the reaction, the type and duration of the cheering behavior of the participants, and the position information of the participants in the virtual space, and presents the generated reaction data to the participants. Note that an example of the reaction data generation process will be described in the operation example.

[0034] (Example of Operation) Next, an example of operation of the reaction presentation control device SV configured as above will be described.

[0035] FIG. 4 is a flowchart illustrating an example of the procedure and content of the reaction presentation control process executed by the control unit 1 of the reaction presentation control device SV.

[0036] Figure 5 is a diagram showing the timing of a series of control operations from the start of distribution of event data to the presentation of the performer's reaction in response to the participant's cheering behavior, where T0 represents the period for determining the cheering preparation behavior that is a precursor to the participant's cheering behavior, T1 represents the duration of the participant's cheering behavior, and T2 represents the period for presenting the performer's reaction.

[0037] (1) Distribution of Event Data When the control unit 1 of the reaction presentation control device SV determines in step S10 that the preset event start time has arrived, in step S11, under the control of the event data distribution processing unit 11, the control unit 1 reads the event data of the corresponding live event from the event data storage unit 31. Then, the event data distribution processing unit 11 simultaneously distributes the read event data from the communication I / F unit 4 via the network NW to the user terminals UT1 to UTn of the participants.

[0038] The event data includes time-series motion capture data (hereinafter referred to as motion capture data) that represents the movements of performers in the virtual space, as well as corresponding audio data, etc. Therefore, participants can participate in a live event being held in the virtual space by viewing the motion capture data and audio data, etc., on their own user terminals UT1 to UTn. Note that, in addition to the motion capture data of the performers, avatars that reflect the movements of each participant acquired from the user terminals UT1 to UTn may also be displayed in the virtual space.

[0039] (2) Determining participants' cheering behavior During the distribution of the event data, the participants' behavior status is detected by sensors in the user terminals UT1 to UTn, and the detected sensor data is transmitted from the user terminals UT1 to UTn to the reaction presentation control device SV.

[0040] In one embodiment, a camera, a microphone, and a vital sensor are used as sensors to detect the movement status of participants while they are viewing event data. These sensors detect, for example, video data showing the participants' facial or limb movements, audio data showing the participants' cheers, and vital data such as the participants' heart rate, sweating, and breathing rate, and the detected data is transmitted as sensing data from the user terminals UT1 to UTn to the reaction display control device SV.

[0041] In response to this, in step S12, the control unit 1 of the reaction presentation control device SV, under the control of the participant action information acquisition processing unit 12, receives the sensing data transmitted from the user terminals UT1 to UTn via the communication I / F unit 4. The received sensing data is temporarily stored in a buffer memory in the data storage unit 3 for the next cheering action determination process.

[0042] Next, in step S13, under the control of the cheering behavior determination processing unit 13, the control unit 1 of the reaction presentation control device SV determines the cheering behavior of the participants based on the received sensing data as follows.

[0043] That is, the cheering behavior determination processing unit 13 first detects changes in the motion state of each participant from the sensing data, then reads out threshold values ​​corresponding to the types of motion state of the participant from the determination table stored in the determination table storage unit 32, and compares the detected changes in the motion state with the threshold values ​​for each type to determine whether the motion state of the participant corresponds to cheering preparation.

[0044] 6 shows an example of the determination table. The determination table stores threshold values ​​in advance for determining changes in the intake volume, mouth opening, and arm height, which indicate the type of motion state of the participant. The threshold values ​​may be values ​​for determining absolute values ​​in addition to values ​​for determining the amount of change.

[0045] The cheering behavior determination processor 13 determines whether the amount of change in the amount of intake per unit time, the amount of change in the amount of mouth opening, and the amount of change in the height of the arms, which are represented by the participant's sensing data, are equal to or greater than the corresponding thresholds. If at least one of the amounts of change in the amount of intake, the amount of mouth opening, and the height of the arms is equal to or greater than the threshold, the cheering behavior determination processor 13 determines that the participant has started cheering preparation behavior.

[0046] In addition, the cheering behavior determination processing unit 13 may determine that a participant has started cheering preparation behavior when two or more of the changes in the above-mentioned intake volume, mouth opening volume, and arm height become equal to or greater than a threshold value.

[0047] In addition to the above-mentioned cheering preparation behavior determination process, the cheering behavior determination processing unit 13 continues to monitor the participants' movements using sensing data, and based on the movement pattern, estimates the type of cheering behavior being performed by the participant and the duration or predicted end time of the cheering behavior.

[0048] Incidentally, the cheering behavior of one participant is often triggered by the cheering behavior of surrounding participants. Therefore, when determining the cheering preparation behavior of the participant to be determined, the cheering behavior determination processing unit 13 also refers to the movements of nearby participants in the vicinity to determine the type of cheering preparation behavior and cheering behavior of the participant to be determined.

[0049] For example, the cheering behavior determination processing unit 13 acquires sensing data representing the movement of the participant to be determined, as well as sensing data representing the movement of at least one nearby participant in the vicinity, from the participant movement information acquisition processing unit 12. Then, the cheering behavior determination processing unit 13 determines the cheering preparation behavior of the participant to be determined by taking into account the movement states of the surrounding nearby participants in addition to the movement state of the acquired participant to be determined.

[0050] More specifically, even if cheer preparation behavior of the participant to be judged is not detected, if cheer preparation behavior of the nearby participants around the participant to be judged is detected, the participant to be judged is predicted to start cheering behavior. Furthermore, the type of cheering behavior of the nearby participants around the participant to be judged is used as the type of cheering behavior of the participant to be judged.

[0051] (3) Managing Participant Positions In step S14, the control unit 1 of the reaction presentation control device SV manages the positions of each participant participating in the event in the virtual space under the control of the participant position information management processing unit 14. For example, the positions of participants in the virtual space are managed as the position coordinates of the participant's avatar in the virtual space. Note that the positions of participants may be managed as relative positions to the positions of performers in the virtual space, or may be managed as the two-dimensional or three-dimensional effective field of view range of the participant in the virtual space.

[0052] (4) Determination of whether a reaction is possible The control unit 1 of the reaction presentation control device SV then, under the control of the reaction possibility determination processing unit 15, determines in step S15 whether the performer is in a state where they can react to the participant who has been determined to have started the cheer preparation action, as follows:

[0053] That is, the reaction possibility determination processing unit 15 determines whether or not the performer is currently in a state where he or she can react from the perspective of the performance of the event, based on the event data being distributed, or whether or not he or she is in a state where he or she can react, based on the consistency with the performer's current acting movements.

[0054] These determinations can be made by, for example, adding marker information indicating the period during which the performer can react to the event data in advance and referencing this marker information. Note that the method for determining whether a reaction is possible is not limited to the above method, and any method can be used, such as a method based on the amount of movement in motion capture.

[0055] (5) Generation of performer reaction data When the reaction possibility determination processing unit 15 determines that the performer is capable of reacting, in step S16, the control unit 1 of the reaction presentation control device SV generates the performer's reaction data in response to the participants' cheers under the control of the reaction data generation processing unit 16 as follows:

[0056] That is, the reaction data generation processing unit 16 first selectively reads out reaction basic data corresponding to the type of cheering action from the reaction basic data storage unit 34 based on the type of cheering action determined by the cheering action determination processing unit 13.

[0057] The reaction data generation processing unit 16 then performs morphing to change the motion capture data immediately before the reaction of the performer in the event data into reaction data based on the basic reaction data.

[0058] For example, by inserting multiple interpolation frames between the motion capture frame immediately before the actor's reaction and the frame of the basic reaction data, reaction data is generated so that the actor's movement changes smoothly into a reaction movement. At this time, based on the position information of the participants stored in the participant position information storage unit 33, the actor's animation image in the reaction data is changed so that the actor's face or body faces the participant.

[0059] Next, in step S17, the reaction data generation processing unit 16 adjusts the presentation start timing of the generated reaction data to an appropriate timing after the completion of the cheering action of the participant, based on the cheering duration T1 or the predicted end time determined by the cheering action determination processing unit 13. Then, the reaction data is presented to the participants by distributing it from the communication I / F unit 4 to the user terminals UT1 to UTn of the participants for a period T2 from the adjusted presentation timing.

[0060] (6) End of reaction presentation control In step S18, the control unit 1 of the reaction presentation control device SV determines whether the distribution of the event data has ended. If the result of this determination is that the distribution of the event data is still in progress, the control unit 1 returns to step S11 and executes again the series of processing procedures related to the reaction presentation control in steps S11 to S18 described above. Thereafter, the control unit 1 similarly repeatedly executes the reaction presentation control in steps S11 to S18 described above while the event data is being distributed. On the other hand, when the distribution of the event data has ended, the control unit 1 ends the processing and returns to the standby state.

[0061] (Effects) As described above, in one embodiment, sensing data representing the motion status of a participant is acquired during distribution of event data, and a cheering preparation action of the participant is determined based on changes in the motion status of the participant represented by the acquired sensing data. Then, when a cheering preparation action is detected, from that point on, a process of generating actor reaction data to the participant is started in accordance with the actor's immediately preceding motion, the type of cheering action of the participant, and the position of the participant in the virtual space, and the generated reaction data is inserted into the event data and distributed at an appropriate time after the participant's cheering action has ended.

[0062] Therefore, it is possible to present the performer's reaction data to the participants at an appropriate timing after the participants have finished cheering. In other words, the performer's reaction to the participants' cheering actions can be presented to the participants at a natural timing without losing immediacy.

[0063] Furthermore, when generating reaction data, the orientation of the performer's face or body image is changed based on the positional information of the participants in the virtual space so that the performer's gaze is directed in the direction of the participants who cheered them on. This maintains a high level of interactivity between the participants and the performers, making it possible to increase participants' satisfaction with the event.

[0064] Furthermore, in addition to the behavioral state of the participant being judged, the behavioral state of nearby participants in the vicinity is also referenced, and the cheer preparation behavior and type of cheer of the participant being judged is inferred from the cheering behavior of the nearby participants. This makes it possible to present reaction data at an appropriate time even to participants who are not accustomed to cheering behavior, such as those attending an event for the first time.

[0065] [Other Embodiments] (1) In one embodiment, the movement state of a participant is determined based on at least one of breathing volume, mouth opening, and arm height, but it may also be determined by combining other factors such as the participant's whole-body movements, changes in voice due to cheering, heart rate, amount of sweating, blinking of eyelids, etc.

[0066] For example, when a participant starts cheering, their inhalation volume and heart rate generally increase, while their blinking tends to decrease. Therefore, by focusing on these contradictory characteristics and detecting the occurrence of both characteristics within the same period as a sign of cheering, accurate judgment can be made.

[0067] Furthermore, the types of cheering behaviors tend to correspond to the genre of the event, so a determination table for determining signs of cheering behaviors for each event may be prepared in advance, and each time an event is held, a corresponding determination table may be selected to determine the types of cheering preparation behaviors and cheering behaviors of participants.

[0068] (2) In addition, the processing procedures and contents of the process for determining whether or not a reaction is possible and the process for generating reaction data, the structure of the reaction data, and the timing of presentation to participants can be modified in various ways without departing from the spirit of this invention.

[0069] Although the embodiments of the present invention have been described in detail above, the above description is merely an example of the present invention in every respect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. In other words, when implementing the present invention, specific configurations according to the embodiments may be appropriately adopted.

[0070] In short, this invention is not limited to the above-described embodiments, and in the implementation stage, the components can be modified and embodied without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

[0071] SV...reaction presentation control device UT1 to UTn...user terminal 1...control unit 2...program storage unit 3...data storage unit 4...communication I / F unit 5...bus 11...event data distribution processing unit 12...participant action information acquisition processing unit 13...support action determination processing unit 14...participant position information management processing unit 15...reaction possibility determination processing unit 16...reaction data generation processing unit 31...event data storage unit 32...determination table storage unit 33...participant position information storage unit 34...reaction basic data storage unit

Claims

1. A reaction presentation control device that distributes data related to an event held in a virtual space to a participant's terminal, comprising: a first processing unit that acquires information representing the participant's motion state while the data related to the event is being distributed; a second processing unit that determines a cheer preparation behavior that indicates a sign of the participant's cheering behavior based on the acquired information representing the participant's motion state; and a third processing unit that generates data representing the reaction of performers appearing in the event to the participant according to the determination result of the cheer preparation behavior, and distributes the generated data representing the reaction to the participant's terminal after the participant has finished cheering.

2. The reaction presentation control device described in claim 1, wherein the first processing unit acquires information representing the motion status of nearby participants around the participant in addition to information representing the motion status of the participant, and the second processing unit estimates the cheering preparation behavior of the participant by referring to the information representing the motion status of the nearby participants.

3. A reaction presentation control device as described in claim 1, further comprising a fourth processing unit that manages information representing the positional relationship of the participants relative to the performer in the virtual space, wherein the third processing unit, when generating data representing the reaction, changes the image of the performer so that the performer's line of sight is directed toward the participant in the virtual space based on the information representing the positional relationship of the participants relative to the performer.

4. A reaction presentation control method in which an information processing device executes an operation of distributing data related to an event held in a virtual space to a participant's terminal, the reaction presentation control method comprising the steps of: acquiring information representing the participant's motion status while the data related to the event is being distributed; determining a cheering preparation behavior of the participant that indicates an indication of cheering behavior based on the acquired information representing the participant's motion status; and generating data representing the reaction of performers appearing in the event toward the participant according to the determination result of the cheering preparation behavior, and distributing the generated data representing the reaction to the participant's terminal after the participant has finished cheering behavior.

Images