Virtual space generation device, virtual space generation method, and program

Abstract

A virtual space generation device according to one aspect of the present invention comprises a determination unit, a performance information generation unit, and an avatar motion information generation unit. The determination unit determines whether or not a user is performing a predetermined action on the basis of user motion information indicating the motion of the user in the real world. When the user is not performing a predetermined action, the performance information generation unit generates first performance information for changing the motion of a third-party avatar other than a user avatar, which is the avatar of the user in a virtual space. The avatar motion information generation unit generates first performance motion information related to the third-party avatar on the basis of the first performance information.

Description

Virtual space generation device, virtual space generation method, and program

[0001] The present invention relates to a virtual space generation device, a virtual space generation method, and a program.

[0002] Conventionally, events such as virtual reality (VR) live events and remote live events have been held via the web. Since users participate in such events from their homes, they may lack the emotional excitement and sense of unity that users experience compared to events where participants are actually present on-site. To create such excitement and a sense of unity, it is important to generate an interaction loop, for example.

[0003] An interaction loop is a repeated three-step process: one is motivated to take action, one performs the action, and the other person responds to that action. For example, at an event you actually attend in person, various actions such as performers calling out to the audience, the audience cheering in response to the call, and performers providing fan service to the cheering audience members lead to repeated interactions between audience members and between the audience and performers, making it easy for an interaction loop to occur. In this way, the occurrence of an interaction loop is thought to contribute to improving the user's live experience.

[0004] On the other hand, in events held via the web, users participate in a virtual space. Because such users are not physically present at the event, technical issues may prevent the various actions mentioned above from being reflected. In such cases, no interaction is formed, and therefore no interaction loop can occur.

[0005] Yuka Yoshida and Yoshiaki Miyashita, "Sharing a sense of unity on video by superimposing body movements," Information Processing Society of Japan, Interaction 2012

[0006] Prior art, such as the technology disclosed in Non-Patent Document 1, attempts to enhance the sense of unity by superimposing the physical movements of audience members onto live video footage. However, these prior art technologies do not take into account interactions between audience members and between audience members and performers during the event. As such, these prior art technologies may not create an interaction loop, which can detract from the user's live experience in the virtual space.

[0007] The present invention aims to provide a technology that contributes to improving a user's live experience in a virtual space.

[0008] A virtual space generation device according to one aspect of the present invention includes a determination unit, a performance information generation unit, and an avatar movement information generation unit. The determination unit determines whether a user is performing a predetermined action based on user movement information indicating the user's movement in reality. If the user is not performing the predetermined action, the performance information generation unit generates first performance information that changes the movement of another avatar other than the user avatar that is the user's avatar in the virtual space. The avatar movement information generation unit generates the first performance information for the other avatar based on the first performance information.

[0009] According to the present invention, a technique is provided that contributes to improving a user's live experience in a virtual space.

[0010] FIG. 1 is an explanatory diagram illustrating an example of an overview of a VR live performance using a VR system according to an embodiment. FIG. 2 is a block diagram illustrating an example of a specific configuration of a VR system according to an embodiment. FIG. 3 is a block diagram illustrating an example of a hardware configuration of the virtual space generation device of FIG. 2. FIG. 4 is a block diagram illustrating an example of a functional configuration of the virtual space generation device of FIG. 3. FIG. 5 is a block diagram illustrating an example of a functional configuration of an avatar action determination unit of FIG. 4. FIG. 6 is a flowchart illustrating an example of an avatar action determination process in the virtual space generation device according to an embodiment. FIG. 7 is a table illustrating a first specific example of trigger effect information according to an embodiment. FIG. 8 is a table illustrating a second specific example of trigger effect information according to an embodiment. FIG. 9 is a diagram illustrating an example of execution of interrupt action data according to an embodiment. FIG. 10 is a table illustrating a specific example of reinforcement effect information according to an embodiment. FIG. 11 is a table illustrating a first specific example of reward effect information according to an embodiment. FIG. 12 is a table illustrating a second specific example of reward effect information according to an embodiment. FIG. 13 is a diagram illustrating a visual effect related to other spectator avatars in a modified example of the embodiment.

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0012] 1 is an explanatory diagram showing an example of an overview of a VR live event using a VR system according to an embodiment. The VR system 1 in FIG. 1 includes a head-mounted display (HMD) and a controller C. A user U can experience a VR live event taking place in a virtual space VS through the HMD they wear.

[0013] The virtual space VS includes, for example, a user avatar UA, which is an avatar of a user U as an audience member, other audience avatars OAA, which are avatars of audience members other than the user avatar, and performer avatars PA, which are avatars of performers. The user avatar UA resembles a penlight with a stick-shaped light portion. Similarly, the other audience avatars OAA also resemble penlights. The performer avatar PA is a CG (Computer Graphics) character.

[0014] By moving the controller C held by the user U in real space, the user U can move the user avatar UA in the virtual space VS in synchronization with the movement of the controller C. As shown in Fig. 1 , the movement RM of the controller C is linked to the movement VM of the user avatar UA.

[0015] In a VR live performance, other audience avatars OAA, for example, perform movements that mimic the movements of penlights in real space in time with the movements of the performer avatar PA and the music. Also, as described above, the user avatar UA is synchronized with the movements of the controller C operated by the user U. Therefore, the user U experiencing the VR live performance can realize in the virtual space the actions of surrounding audience members waving penlights to cheer on the performers, just as they would at a real live concert venue.

[0016] Furthermore, in the VR system 1 according to the embodiment, the behavior of at least one of the user avatar UA, other audience avatars OAA, and performer avatar PA can be changed according to the movement of the controller C operated by the user U.

[0017] The outline of the VR live using the VR system according to the embodiment has been described above. Next, the configuration of the VR system according to the embodiment will be described.

[0018] 2 is a block diagram showing an example of a specific configuration of a VR system according to an embodiment. The VR system 1 in FIG. 2 includes a virtual space generation device 10, a sensor 20, a distribution server 30, and a display 40.

[0019] The virtual space generation device 10 is, for example, an information processing device such as a computer. The virtual space generation device 10 is configured to generate a virtual space to be provided to a user in a VR live performance.

[0020] The sensor 20 includes, for example, a position measurement sensor such as an acceleration sensor and a gyro sensor. Specifically, the position measurement sensor serving as the sensor 20 is attached to a controller held by the user. The sensor information measured by the position measurement sensor is used by the virtual space generation device 10 to analyze, for example, the three-dimensional position of the user's wrist. The sensor information related to the position measurement sensor may also be referred to as motion data.

[0021] The sensor 20 may include a microphone that captures the user's voice. Specifically, the sensor information captured by the microphone serving as the sensor 20 may be used by the virtual space generating device 10 to analyze, for example, the volume of the user's cheers. The sensor information related to the microphone may also be referred to as voice data.

[0022] The distribution server 30 is, for example, a server managed by an administrator who distributes the VR live. The distribution server 30 distributes distribution information related to the VR live to the virtual space generation device 10 via the network NW.

[0023] The display 40 is a display device for providing a virtual space to the user. The display 40 is, for example, an HMD. When the display 40 is an HMD, the virtual space generating device 10 may be configured integrally with the display 40. Furthermore, when the display 40 is an HMD, a part of the sensor 20 may be further attached to the HMD. In this case, the sensor information is information regarding the relative positional relationship between the sensor 20 attached to the HMD and the controller.

[0024] The configuration of the VR system according to the embodiment has been described above. Next, the hardware configuration of the virtual space generation device included in the VR system will be described.

[0025] Fig. 3 is a block diagram showing an example of the hardware configuration of the virtual space generation device of Fig. 2. The virtual space generation device 10 includes a control circuit 110, a storage 120, a communication module 130, an interface 140, and a drive 150.

[0026] The control circuit 110 is a circuit that controls the overall components of the virtual space generating device 10. The control circuit 110 includes a CPU (Central Processing Unit), RAM (Random Access Memory), and ROM (Read Only Memory). The ROM of the control circuit 110 stores programs used in various processes in the virtual space generating device 10. The CPU of the control circuit 110 controls the entire virtual space generating device 10 in accordance with the programs stored in the ROM of the control circuit 110. The RAM of the control circuit 110 is used as a working area for the CPU of the control circuit 110.

[0027] The storage 120 is configured by, for example, a hard disk drive (HDD), a solid state drive (SSD), a flash memory, etc. The storage 120 stores information used in various processes in the virtual space generating device 10.

[0028] The communication module 130 is a circuit used to send and receive data between the sensor 20, the distribution server 30, and the display 40.

[0029] The interface 140 is an interface used for communication between the user U and the control circuit 110. The interface 140 includes, for example, an input device and an output device. The input device includes an audio microphone, a touch panel, and operation buttons. The output device includes a speaker and a display. When the virtual space generating device 10 is configured integrally with the display 40, the display included in the output device may be the same as the display 40.

[0030] The drive 150 is a device for reading software stored in a storage medium 151. The drive 150 includes, for example, a CD (Compact Disk) drive or a DVD (Digital Versatile Disk) drive.

[0031] The storage medium 151 is a medium that stores software electrically, magnetically, optically, mechanically, or chemically. The storage medium 151 may store programs for executing various processes in the virtual space generation device 10.

[0032] The storage medium 151 may be a USB (Universal Serial Bus) memory. When the storage medium 151 is a USB memory, the storage medium 151 is connected to, for example, the communication module 130. In this case, the virtual space generation device 10 does not need to be provided with the drive 150.

[0033] The hardware configuration of the virtual space generation device included in the VR system has been described above. Next, the functional configuration of the virtual space generation device will be described.

[0034] Fig. 4 is a block diagram showing an example of the functional configuration of the virtual space generation device 10 of Fig. 3. The virtual space generation device 10 of Fig. 4 includes a sensor information acquisition unit 210, a user movement analysis unit 220, a distribution information acquisition unit 230, an avatar movement determination unit 240, and a virtual space generation unit 250.

[0035] 4, the CPU of the control circuit 110 in FIG. 3 loads a program stored in the ROM of the control circuit 110 or in the storage medium 151 into the RAM of the control circuit 110. The CPU of the control circuit 110 then interprets and executes the program loaded into the RAM of the control circuit 110. As a result, the virtual space generation device 10 functions as a computer including a sensor information acquisition unit 210, a user movement analysis unit 220, a distribution information acquisition unit 230, an avatar movement determination unit 240, and a virtual space generation unit 250.

[0036] The sensor information acquisition unit 210 acquires sensor information measured by the sensor 20. The sensor information includes, for example, information measured in response to the user's actual movements of the controller in accordance with the movements of the performer avatar and the music during the VR live performance. The sensor information acquisition unit 210 outputs the acquired sensor information to the user movement analysis unit 220. The acquisition and output of sensor information by the sensor information acquisition unit 210 is periodically repeated at predetermined times.

[0037] The user motion analysis unit 220 receives sensor information from the sensor information acquisition unit 210. The user motion analysis unit 220 analyzes the user's real-world motion based on the sensor information. Specifically, the user motion analysis unit 220 calculates time-dependent changes in the three-dimensional position of the user's wrist by analyzing the sensor information. The user motion analysis unit 220 outputs user motion information indicating the user's real-world motion to the avatar motion determination unit 240. The user motion information includes the three-dimensional coordinates and rotation information of the user's wrist.

[0038] The three-dimensional coordinate of the user's wrist may be defined arbitrarily. For example, the X-axis of the three-dimensional coordinate is the direction of the short side of the controller to which the sensor 20 is attached, and is defined as the axis that passes through the user's body when the controller is held correctly. The Y-axis of the three-dimensional coordinate is the direction of the other short side of the controller, and is defined as the axis that passes from one side of the user's shoulder to the other when the controller is held correctly. The X-axis of the three-dimensional coordinate is defined as the direction of the long side of the controller. The rotation information above represents the tilt of the controller, and is expressed as Euler angles or quaternions.

[0039] The distribution information acquisition unit 230 acquires distribution information related to the VR live broadcast distributed from the distribution server 30. The distribution information includes, for example, information on other spectator avatars (other spectator avatar information), information on performer avatars (performer avatar information), and music information. The other spectator avatar information includes, for example, information on the basic movements (basic movements) of other spectator avatars (basic movement information). Basic movements are movements that are not affected by user movements during the VR live broadcast. The performer avatar information includes, for example, the basic movement information of the performer avatar. The distribution information acquisition unit 230 outputs the distribution information to the avatar movement determination unit 240 and the virtual space generation unit 250.

[0040] The avatar movement determination unit 240 receives user movement information from the user movement analysis unit 220 and distribution information from the distribution information acquisition unit 230. The avatar movement determination unit 240 determines the movements of the user avatar, other audience avatars, and performer avatars based on the user movement information and distribution information. The avatar movement determination unit 240 outputs information on the movements of these avatars (avatar movement information) to the virtual space generation unit 250. A specific configuration of the avatar movement determination unit 240 will be described below with reference to FIG. 5.

[0041] Fig. 5 is a block diagram showing an example of the functional configuration of the avatar movement determination unit 240 in Fig. 4. The avatar movement determination unit 240 in Fig. 5 includes a user state determination unit 310 (determination unit), a performance information generation unit 320, and an avatar movement information generation unit 330.

[0042] The user state determination unit 310 receives user movement information from the user movement analysis unit 220 and receives distribution information from the distribution information acquisition unit 230. The user state determination unit 310 determines whether or not the user is performing a predetermined action based on the user movement information. The user state determination unit 310 outputs the determination result regarding whether or not the user is performing the predetermined action to the effect information generation unit 320. The user state determination unit 310 may specify a section for determining the user state based on music information included in the distribution information.

[0043] The predetermined behaviors include, for example, cheering behaviors and reinforced cheering behaviors. A cheering behavior is, for example, a behavior in which the user's behavior exceeds a first threshold. An reinforced cheering behavior is, for example, a behavior in which the user's behavior exceeds a second threshold that is greater than the first threshold. The cheering behavior and the reinforced cheering behavior may be referred to as a first behavior and a second behavior, respectively.

[0044] The above determination results include, for example, information on whether a supportive action can be taken and information on whether an enhanced supportive action is required. The information on whether a supportive action can be taken is information on whether a supportive action is being taken. The information on whether an enhanced supportive action is required is information on whether an enhanced supportive action needs to be taken.

[0045] The effect information generation unit 320 receives the determination result from the user state determination unit 310. Triggered by the determination result, the effect information generation unit 320 generates effect information for an avatar according to a predetermined action of the user. The effect information generation unit 320 outputs the generated effect information to the avatar action information generation unit 330. Note that the effect information generation unit 320 may select an avatar to which the effect information is to be applied based on music information included in the distribution information.

[0046] The performance information is information for changing the avatar's behavior. The performance information includes, for example, a behavior parameter that indicates the rate at which the avatar's behavior is changed and interrupt behavior data that causes the avatar to perform a predetermined behavior midway through a basic behavior. Furthermore, for example, the performance information may include a facial expression parameter related to the avatar's facial expression and a volume parameter related to the volume of cheers and the like. The behavior parameter, facial expression parameter, and volume parameter may be used independently or in combination.

[0047] The types of effect information include, for example, inducement effect information, reward effect information, and reinforcement effect information. The inducement effect information is effect information that changes the behavior of other avatars in order to induce cheering behavior in the user. Other avatars are a general term for avatars other than the user avatar, and include at least one of other spectator avatars and performer avatars. The reward effect information is effect information that changes the behavior of other avatars depending on the level of the user's cheering behavior. The reinforcement effect information is effect information that changes the behavior of the user avatar in order to reinforce the user's cheering behavior.

[0048] Specifically, the effect information generation unit 320 includes an inducement effect information generation unit 321, a reward effect information generation unit 322, and a reinforcement effect information generation unit 323. Which of these units generates the effect information is determined based on, for example, the content of the determination result.

[0049] The trigger effect information generating unit 321 generates trigger effect information for inducing cheering behavior, for example, in response to a determination result that cheering behavior is not being performed. The trigger effect information generating unit 321 outputs the generated trigger effect information to at least one of the other spectator avatar movement information generating unit 331 and the performer avatar movement information generating unit 332.

[0050] The reward effect information generation unit 322 generates reward effect information according to the degree of the cheering behavior, for example, triggered by a determination result in which some determination has been made regarding both the cheering behavior and the reinforced cheering behavior. Specifically, the reward effect information generation unit 322 compares the user's behavior deemed to be a cheering behavior with a threshold (≧second threshold), and generates effect information for a large reward (large reward) if the cheering behavior exceeds the threshold, and generates effect information for a small reward (small reward) if the cheering behavior does not exceed the threshold. The reward effect information generation unit 322 outputs the generated reward effect information to at least one of the other spectator avatar movement information generation unit 331 and the performer avatar movement information generation unit 332.

[0051] The reinforcement effect information generation unit 323 generates reinforcement effect information for strengthening the cheering behavior, for example, in response to a determination result that the cheering behavior needs to be strengthened. The reinforcement effect information generation unit 323 outputs the generated reinforcement effect information to the user avatar action information generation unit 333.

[0052] The avatar movement information generation unit 330 receives the performance information from the performance information generation unit 320. The avatar movement information generation unit 330 generates information on movements of the avatar (avatar movement information) based on the performance information. The avatar movement information generation unit 330 outputs the generated avatar movement information to the virtual space generation unit 250.

[0053] Specifically, the avatar movement information generation unit 330 includes an other-audience avatar movement information generation unit 331, a performer avatar movement information generation unit 332, and a user avatar movement information generation unit 333. Which of these units generates the avatar movement information depends on the content of the production information.

[0054] When the other spectator avatar movement information generation unit 331 receives induced performance information from the induced performance information generation unit 321, it generates induced performance action information for other spectator avatars based on the induced performance information. Furthermore, when the other spectator avatar movement information generation unit 331 receives reward performance information from the reward performance information generation unit, it generates reward performance action information for other spectator avatars based on the reward performance information. The other spectator avatar movement information generation unit 331 outputs the induced performance action information for other spectator avatars or the reward performance action information for other spectator avatars to the virtual space generation unit 250 as other spectator avatar movement information.

[0055] When the performer avatar movement information generation unit 332 receives the induced performance information from the induced performance information generation unit 321, it generates induced performance movement information for the performer avatar based on the induced performance information. Furthermore, when the performer avatar movement information generation unit 332 receives the reward performance information from the reward performance information generation unit 322, it generates reward performance movement information for the performer avatar based on the reward performance information. The performer avatar movement information generation unit 332 outputs the induced performance movement information for the performer avatar or the reward performance movement information for the performer avatar to the virtual space generation unit 250 as performer avatar movement information.

[0056] The user avatar action information generation unit 333 receives the reinforcement effect information from the reinforcement effect information generation unit 323. The user avatar action information generation unit 333 generates reinforcement effect action information for the user avatar based on the reinforcement effect information. The user avatar action information generation unit 333 outputs the reinforcement effect action information for the user avatar to the virtual space generation unit 250 as user avatar action information.

[0057] The virtual space generation unit 250 receives distribution information from the distribution information acquisition unit 230 and avatar movement information from the avatar movement determination unit 240. Based on the distribution information and the avatar movement information, the virtual space generation unit 250 generates a virtual space in which the movements of at least one of the user avatar, other audience avatars, and performer avatars are changed. The virtual space generation unit 250 outputs the generated virtual space to the display 40.

[0058] The functional configuration of the virtual space generation device has been described above. Next, the operation of the virtual space generation device will be described.

[0059] [Operation] Figure 6 is a flowchart showing an example of avatar movement determination processing in a virtual space generation device according to an embodiment. The avatar movement determination processing shown in the flowchart in Figure 6 is performed repeatedly at arbitrary time intervals, for example, during a VR live performance. The arbitrary time interval may be determined, for example, by a fixed period of time, a scene (such as an MC) set during the VR live performance, the music of the VR live performance, and a predetermined section (musical passage) within the music. Below, an example of performing the avatar movement determination processing at a predetermined section within the music performance will be described.

[0060] (Step S110) When a predetermined segment starts, the user state determination unit 310 determines whether or not the user is performing a cheering behavior. If it is determined that the user is performing a cheering behavior, the process proceeds to step S140. If it is determined that the user is not performing a cheering behavior, the process proceeds to step S120.

[0061] Specifically, if the user's movement does not exceed the first threshold for a predetermined length of time after the start of a predetermined segment, the user state determination unit 310 determines that the user is not performing a cheering behavior. Also, if the user's movement exceeds the first threshold without a predetermined length of time after the start of a predetermined segment, the user state determination unit 310 determines that the user is performing a cheering behavior.

[0062] (Step S120) After it is determined that the cheering behavior is not being performed, the induced effect information generation unit 321 generates induced effect information for inducing the cheering behavior. A first specific example and a second specific example of the induced effect information will be described below with reference to FIGS. 7 and 8, respectively.

[0063] 7 is a table for explaining a first specific example of trigger effect information in an embodiment. The first specific example uses movement parameters and the like as trigger effect information. Table 700 in FIG. 7 associates target avatars and movement parameters, etc., for Example 1, which relates to other audience avatars, and Example 2, which relates to performer avatars.

[0064] In Example 1, the action parameter "×1.3" is associated with the target "other spectator avatars around the user avatar." The range of the target may be set arbitrarily in advance, or may be set according to the user's action. The action parameter for other spectator avatars is, for example, a magnification by which the swing width of the penlight is changed from the basic action. Furthermore, the value of the action parameter may be set arbitrarily in advance, or may be set according to the user's action.

[0065] In Example 2, the target "performer avatar" is associated with a movement parameter "×1.3" and an expression parameter "smile." The movement parameter for the performer avatar is, for example, a magnification factor by which the choreography of the character is changed from the basic movement. The value of the movement parameter may be set arbitrarily in advance or may be set according to the user's movement.

[0066] As described above, in Example 1, it is believed that by exaggerating the movements of at least one of the other spectator avatars and the performer avatars, it is possible to encourage users who are not proactive in cheering to cheer.

[0067] 8 is a table for explaining a second specific example of trigger effect information in an embodiment. The second specific example uses interruption action data as trigger effect information. Table 800 in FIG. 8 associates target avatars and interruption action data for Example 1, which relates to other spectator avatars, and Example 2, which relates to performer avatars.

[0068] In Example 1, the target "other spectator avatars around the user avatar" is associated with interrupt action data "perform cheering action all at once." This interrupt action data sets a predetermined action different from the basic action for other spectator avatars within a predetermined range.

[0069] In Example 2, the target "performer avatar" is associated with interruption action data "make eye contact with the user." This interruption action data sets a predetermined action for the performer avatar that is different from the basic action.

[0070] As described above, in specific example 2, for users who are not proactive in cheering, it is thought that the cheering of users can be encouraged by having at least one of the other spectator avatars and the performer avatar perform an action different from the basic action (for example, an action that evokes cheering).

[0071] In each of the above-described specific examples 1 and 2, the triggered performance information includes at least one of examples 1 and 2. In other words, the triggered performance information is set for at least one of other audience avatars and performer avatars. The avatars that are the targets of the triggered performance information may be arbitrarily set in advance.

[0072] Furthermore, the above-described specific example 1 and specific example 2 may be combined with each other. For example, the induced effect information may include Example 1 of specific example 1 and Example 2 of specific example 2. For example, the induced effect information may include Example 2 of specific example 1 and Example 1 of specific example 2.

[0073] (Step S130) After the triggering effect information is generated, the avatar action information generating unit 330 generates triggering effect action information for the other avatar based on the triggering effect information. After step S130, the process proceeds to step S140.

[0074] Specifically, when the triggering performance information targets other spectator avatars, the other spectator avatar movement information generation unit 331 generates triggering performance action information for the other spectator avatars based on the triggering performance information. On the other hand, when the triggering performance information targets a performer avatar, the performer avatar movement information generation unit 332 generates triggering performance action information for the performer avatar based on the triggering performance information. Furthermore, when the triggering performance information targets other avatars (other spectator avatars and performer avatars), the performer avatar movement information generation unit 332 generates triggering performance action information for the other spectator avatars and the performer avatar based on the triggering performance information. An execution example of interruption action data related to a performer avatar will be described below with reference to FIG. 9.

[0075] 9 is a diagram illustrating an example of execution of interrupt action data in an embodiment. In FIG. 9, basic action data 900A, interrupt action data 910, and basic action data 900B are shown in chronological order on a timeline. Also shown in FIG. 9 is a virtual space VS1 in which the interrupt action data 910 is being executed. The basic action data 900A and basic action data 900B are continuous data, and will hereinafter be referred to simply as basic action data 900.

[0076] 9 , for example, at time t0, virtual space generation device 10 displays a virtual space based on basic action data 900. Next, at time t1, when virtual space generation device 10 acquires performance information including interruption action data, virtual space generation device 10 switches the virtual space based on basic action data 900 that has been displayed up until now to virtual space VS1 based on interruption action data 910. Virtual space VS1 includes the user avatar, other spectator avatars, and performer avatar PA1 based on interruption action data 910. Next, at time t2, when the playback time of interruption action data 910 ends, virtual space generation device 10 displays a virtual space based on a continuation of basic action data 900 that has been displayed up to time t1.

[0077] (Step S140) After it is determined in step S110 that the user is performing a cheering behavior, or after the triggering performance operation information for the other person's avatar is generated in step S130, the user state determination unit 310 determines whether or not the cheering behavior needs to be strengthened. If it is determined that the cheering behavior needs to be strengthened, the process proceeds to step S150. If it is determined that the cheering behavior does not need to be strengthened, the process proceeds to step S170.

[0078] Specifically, the user state determination unit 310 determines that the supportive behavior needs to be strengthened if the user's behavior regarded as the supportive behavior does not exceed a second threshold value that is greater than the first threshold value. Furthermore, the user state determination unit 310 determines that the supportive behavior does not need to be strengthened if the user's behavior regarded as the supportive behavior exceeds the second threshold value.

[0079] (Step S150) After it is determined that the cheering behavior needs to be strengthened, the strengthening effect information generation unit 323 generates strengthening effect information for strengthening the cheering behavior. A specific example of the strengthening effect information will be described below with reference to FIG.

[0080] 10 is a table illustrating a specific example of the reinforcement effect information in the embodiment. In the table 1000 in FIG. 10, the action parameters and the volume parameters are associated with each of Example 1, which relates to all users, and Example 2, which takes into account the characteristics of each user.

[0081] In Example 1, the target "all users" is associated with an action parameter "x1.3" and a volume parameter "x1.3". These action parameters and volume parameters relate to the user avatar. The action parameter relating to the user avatar is, for example, a magnification for changing the swing width of a penlight. The volume parameter relating to the user avatar is, for example, a magnification for changing the volume of the user's voice data. Furthermore, the values ​​of these parameters may be arbitrarily set in advance, or may be set according to the user's action.

[0082] As described above, in the first embodiment, it is considered that the user can feel a sense of accomplishment by exaggerating the movements of the user avatar, and therefore it is considered that it is possible to support the user in continuing the cheering behavior.

[0083] In Example 2, a motion parameter and a volume parameter are associated with each of two different targets. The two targets are a “user who is active in cheering” and a “user who is passive in cheering,” respectively, and are assumed to be input in advance into the virtual space generating device 10.

[0084] Specifically, in Example 2, the target "user who is active in cheering" is associated with the action parameter "x0.8" and the volume parameter "x0.8". Also, the target "user who is passive in cheering" is associated with the action parameter "x1.5" and the volume parameter "x1.5".

[0085] As described above, in Example 2, it is believed that for users who are proactive in cheering, the actions of the user avatar are intentionally downsized, thereby encouraging the users to further cheer. On the other hand, for users who are reluctant to cheer, the actions of the user avatar are exaggerated, which may enable the users to feel a sufficient sense of accomplishment even if their cheering actions are small. Therefore, in either case, it is believed that it is possible to support the users to continue cheering.

[0086] (Step S160) After the reinforcement effect information is generated, the user avatar action information generation unit 333 generates reinforcement effect action information for the user avatar based on the reinforcement effect information. After step S160, the process proceeds to step S170.

[0087] (Step S170) After it is determined in step S140 that reinforcement of the cheering behavior is not necessary, or after a reinforcement effect action for the user avatar is generated in step S160, the reward effect information generation unit 322 generates reward effect information according to the level of the cheering behavior. A first specific example and a second specific example of the reward effect information will be described below with reference to FIGS. 11 and 12, respectively.

[0088] 11 is a table illustrating a first specific example of reward effect information in an embodiment. The first specific example uses movement parameters and the like as reward effect information. Table 1100 in FIG. 11 associates target avatars and movement parameters, etc., with the magnitude of rewards for Example 1, which relates to a performer avatar, and Example 2, which relates to other audience avatars.

[0089] In Example 1, a "large" reward is associated with a movement parameter of "x2.0" and a facial expression parameter of "smile" for the target "performer avatar." A "small" reward is associated with a movement parameter of "x1.5" and a facial expression parameter of "no change" for the target "performer avatar." The values ​​of the movement parameters may be arbitrarily set in advance or may be set according to the user's movements.

[0090] In Example 2, in the case of a "large" reward, the target "all other spectator avatars" is associated with a movement parameter of "x 1.5". In addition, in the case of a "small" reward, the target "half of other spectator avatars" is associated with a movement parameter of "x 1.5". The proportion of other spectator avatars that are the target may be arbitrarily set in advance, or may be set in accordance with the user's movement. In addition, the value of the movement parameter may be arbitrarily set in advance, or may be set in accordance with the user's movement.

[0091] As described above, in Example 1, it is conceivable that the user can feel a sense of accomplishment by exaggerating the actions of at least one of the other spectator avatars and the performer avatar depending on the level of the user's cheering behavior.

[0092] 12 is a table illustrating a second specific example of reward effect information in an embodiment. The second specific example uses interruption action data as reward effect information. Table 1200 in FIG. 12 associates target avatars and interruption action data according to the magnitude of rewards for Example 1, which relates to a performer avatar, and Example 2, which relates to other audience avatars.

[0093] In Example 1, in the case of a "large" reward, the target "performer avatar" is associated with the interruption action data "turn face toward the user and smile at the user." In addition, in the case of a "small" reward, the target "performer avatar" is associated with the interruption action data "turn face toward the user."

[0094] In Example 2, when the reward is "large," the interruption action data "cheer vigorously" is associated with the target "all other spectator avatars." When the reward is "small," the interruption action data "cheer vigorously" is associated with the target "half of the other spectator avatars."

[0095] As described above, in specific example 2, depending on the level of the user's cheering behavior, at least one of the other spectator avatars and the performer avatar may perform an action different from the basic action (for example, an action that makes the user want to continue cheering), which may give the user a sense of accomplishment.

[0096] In each of the above-described specific examples 1 and 2, the reward performance information includes at least one of examples 1 and 2. In other words, the reward performance information is set for at least one of the performer avatar and the other audience avatar. The avatar that is the target of the reward performance information may be set arbitrarily in advance.

[0097] Furthermore, the above-described specific example 1 and specific example 2 may be combined with each other. For example, the reward presentation information may include Example 1 of specific example 1 and Example 2 of specific example 2. For example, the reward presentation information may include Example 2 of specific example 1 and Example 1 of specific example 2.

[0098] (Step S180) After the reward effect information is generated, the avatar action information generation unit 330 generates reward effect action information for the other person's avatar based on the reward effect information.

[0099] Specifically, when the reward performance information targets other spectator avatars, the other spectator avatar movement information generation unit 331 generates reward performance movement information for the other spectator avatars based on the reward performance information. On the other hand, when the reward performance information targets a performer avatar, the performer avatar movement information generation unit 332 generates reward performance movement information for the performer avatar based on the reward performance information. Furthermore, when the reward performance information targets other avatars (other spectator avatars and performer avatars), the performer avatar movement information generation unit 332 generates reward performance movement information for the other spectator avatars and the performer avatar based on the reward performance information.

[0100] (Step S190) After the reward performance action information is generated, the avatar action determination unit 240 determines whether the predetermined section has ended. If it is determined that the predetermined section has not ended, the process returns to step S110. If it is determined that the predetermined section has ended, the avatar action determination process of FIG. 6 for this predetermined section ends.

[0101] To summarize, the avatar movement determination process in Figure 6 determines whether a cheering action and an enhanced cheering action are necessary based on the user's movements, and changes the movements of other avatars and the user avatar based on the results of each determination. This series of processes creates various interactions between the user and other spectators, or between the user and performers, depending on the user's movements. By creating such various interactions, the virtual space generation device that executes the avatar movement determination process can encourage the creation and continuation of a natural interaction loop involving the user's movements.

[0102] [Effect] According to the above embodiment, the virtual space generation device determines whether or not the user is performing a predetermined action based on user action information indicating the user's actions in reality, and if the user is not performing the predetermined action, generates first performance information that changes the actions of other avatars other than the user avatar, which is the user's avatar in the virtual space, and generates first performance action information for the other avatar based on the first performance information.

[0103] Therefore, the virtual space generating device can change the behavior of other avatars in accordance with the user's behavior, thereby forming interactions between the user and other avatars, thereby contributing to improving the user's live experience in the virtual space.

[0104] The predetermined behavior may include a first behavior in which the user's movement exceeds a first threshold, and the virtual space generating device may determine whether or not the user is performing the first behavior.

[0105] This allows the virtual space generating device to determine, for example, whether or not the user is performing a cheering action as a first action.

[0106] Furthermore, when a user is performing a first action, the virtual space generation device may generate second performance information different from the first performance information that changes the behavior of another avatar in accordance with the first action, and generate second performance action information for the other avatar based on the second performance information.

[0107] This allows the virtual space generating device to generate, for example, reward performance information according to the user's cheering behavior, and generate reward performance action information for another person's avatar based on the reward performance information.

[0108] In addition, the specified behavior may further include a second behavior in which the user's behavior exceeds a second threshold that is greater than the first threshold, and the virtual space generation device may determine whether the user is performing the second behavior, and if the user is not performing the second behavior, generate a third performance behavior that changes the behavior of the user avatar in accordance with the user's behavior, and generate third performance behavior information for the user avatar based on the third performance information.

[0109] As a result, the virtual space generation device can, for example, determine whether or not to have the user perform an enhanced cheering behavior as a second behavior, and if an enhanced cheering behavior is necessary, can support the user's cheering behavior by changing the behavior of the user avatar in accordance with the user's behavior.

[0110] Furthermore, the other person's avatar may be at least one of an other audience member's avatar, which is an avatar of another audience member, and a performer's avatar, which is an avatar of a performer.

[0111] This allows the virtual space generating device to take into consideration changes in the movements of various avatars.

[0112] Furthermore, the first performance information may be a first action parameter that indicates the rate at which the avatar's action is changed or first interrupt action data that causes a predetermined action to be performed, the second performance information may be a second action parameter that has a different value from the first action parameter or second interrupt action data that causes an action different from the first interrupt action data, and the third performance information may be a third action parameter that has a different value from the first action parameter and the second action parameter.

[0113] This allows the virtual space generating device to use different action parameters or interrupt action data depending on various situations.

[0114] Although the above embodiment is based on the assumption that the user is wearing an HMD, the present invention is not limited to this. For example, the display may be a two-dimensional display such as a television screen, which is configured with a two-dimensional flat or curved surface.

[0115] In the above embodiment, the motion data as sensor information is information measured regarding the movement of the wrist corresponding to the movement of the controller, but this is not limited to this. The sensor information may be information measured regarding the movement of various body parts other than the wrist, such as the arms, head, legs, and waist.

[0116] In the above embodiment, the sensor information is information acquired regarding the position of the controller, but this is not limited to this. For example, the sensor may be an optical sensor used for motion capture, which optically measures the user's movements. When such an optical sensor is used, the sensor information may be information acquired regarding the position of the user's hand.

[0117] In the above embodiment, the voice data as the sensor information is time-series voice data, but is not limited to this. For example, the voice data may be voice recognition data in text format obtained by performing voice recognition processing on the time-series voice data.

[0118] In the above embodiment, motion data and voice data are given as examples of sensor information, but the sensor information is not limited to these. The sensor information may also include data on temperature, airflow, humidity, smell, and the like.

[0119] In the above embodiment, the user avatar and other spectator avatars are objects resembling penlights, but this is not limiting. For example, at least one of the user avatar and other spectator avatars may be a CG character. Furthermore, CG characters, including performer avatars, may be replaced with composite images of live-action footage in some cases.

[0120] In the above embodiment, the movements of the other spectator avatars have been described using the movements of penlights, but this is not limiting. For example, if the other spectator avatars are represented by CG characters, the movements of the other spectator avatars may be the movements of the characters. Furthermore, when the other spectator avatars make a sound (hereinafter referred to as cheering), the movements of the other spectator avatars may be accompanied by visual effects that visually indicate that they are cheering. Visual effects related to the other spectator avatars will be described below with reference to FIG. 13 .

[0121] FIG. 13 is a diagram illustrating a visual effect related to another spectator avatar in a modified example of the embodiment. FIG. 13 shows a virtual space VS2 viewed by a user through an HMD. The virtual space VS2 includes another spectator avatar OAA2 and a performer avatar PA2. The other spectator avatar OAA2 is represented by a CG character. A visual effect EF is rendered above the head of the other spectator avatar OAA2. The visual effect EF indicates that the other spectator avatar OAA2 is cheering for the performer avatar PA2. The rendering of the visual effect EF allows the user to identify the other spectator avatar who is cheering.

[0122] When a visual effect is accompanied by the action of another spectator avatar, the size of the rendered visual effect or the number of visual effects may be changed based on the triggering effect information for the other spectator avatar. For example, as in Example 1 of table 700 in FIG. 7, if the action parameter "×1.3" is associated with the target "other spectator avatars around the user avatar," the action parameter "×1.3" may be applied to the visual effect. The same applies to the reward effect information for the other spectator avatar.

[0123] The visual effects described above are not limited to visual representations of cheers. For example, the visual effects may be particles that appear in response to the movements of an avatar. When particles are appearing as a visual effect, the size and quantity of the particles may be changed in response to the movement parameters of the performance information.

[0124] The above visual effects are not limited to those for other audience avatars. They may also be applied to user avatars and performer avatars. In particular, the appearance of particles, which cannot be experienced in real space, is believed to further enhance the live experience for users that is unique to virtual space.

[0125] In the above embodiment, the virtual space generation device changes the behavior of other avatars in response to the user's actions to enhance the user's live experience in the virtual space. However, this is not limiting. For example, the virtual space generation device may provide feedback on the behavior of other avatars to the user. As feedback, for example, the controller held by the user may be vibrated when another spectator avatar cheers. The vibration of the controller allows the user to haptically experience the air in the entire venue vibrating due to the cheers (i.e., excitement). Another example of feedback may be an audio effect that sounds like the sound of the wind from a penlight being waved by another spectator avatar.

[0126] In the above embodiment, the operation parameter is a magnification factor for the operation value, but is not limited to this. For example, the operation parameter may be a numerical value for adding or subtracting a predetermined value.

[0127] In the above embodiment, the user's actions are primarily assumed to be motion data, but are not limited to this. The user's actions may be at least one of audio data and motion data. For example, for audio data representing the user's actions, a threshold value related to the volume is used to determine the user's predetermined action. For motion data representing the user's actions, a threshold value related to at least one of the magnitude and speed of the motion is used to determine the user's predetermined action. When multiple comparison targets are used as described above to determine the user's predetermined action, a final determination result may be output based on a combination of the respective determination results.

[0128] In the above embodiment, the case where the program that executes the avatar movement determination process is executed in the virtual space generation device 10 has been described, but this is not limiting. For example, the program that executes the avatar movement determination process may be executed by computational resources on the cloud.

[0129] In the above embodiment, the flowchart in FIG. 6 is an example. The order of the steps in the flowchart described in the embodiment may be changed to the extent possible, or other steps may be added. For example, after step S130, the process may be changed to return to step S110, or the process may proceed to step S170. Furthermore, for example, a step in which the virtual space generation unit 250 generates a virtual space may be added after each of steps S130, S160, and S180.

[0130] The present invention is not limited to the above-described embodiments, and various modifications can be made in the implementation stage without departing from the spirit of the invention. Furthermore, the embodiments may be implemented in appropriate combinations, in which case the combined effects can be obtained. Furthermore, the above-described embodiments include various inventions, and various inventions can be extracted by combining selected components from the disclosed components. For example, if the problem can be solved and the effects can be obtained even if some components are removed from all the components shown in the embodiments, the configuration from which these components are removed can be extracted as an invention.

[0131] DESCRIPTION OF SYMBOLS 1...VR SYSTEM 10...VIRTUAL SPACE GENERATION DEVICE 20...SENSOR 30...DISTRIBUTING SERVER 40...DISPLAY 110...CONTROL CIRCUIT 120...STORAGE 130...COMMUNICATION MODULE 140...INTERFACE 150...DRIVE 151...STORAGE MEDIUM 210...SENSOR INFORMATION ACQUISITION UNIT 220...USER BEHAVIOR ANALYSIS UNIT 230...DISTRIBUTING INFORMATION ACQUISITION UNIT 240...AVATAR BEHAVIOR DECISION UNIT 250...VIRTUAL SPACE GENERATION UNIT 310...USER STATUS DETERMINATION UNIT 320...PERFORMANCE INFORMATION GENERATION UNIT 321...INDUCTION BEHAVIOR GENERATION INFORMATION GENERATION UNIT 322...REWARD BEHAVIOR GENERATION INFORMATION GENERATION UNIT 323...REINFORCEMENT BEHAVIOR GENERATION INFORMATION GENERATION UNIT 330...AVATAR BEHAVIOR ... 900, 900A, 900B...Basic operation data 910...Operation data C...Controller EF...Visual effect NW...Network OAA, OAA2...Other audience avatars PA, PA1, PA2...Performer avatars RM, VM...Operation t0, t1, t2...Time U...User UA...User avatar VS, VS1, VS2...Virtual space

Claims

1. A virtual space generation device comprising: a determination unit that determines whether a user is performing a predetermined action based on user action information that indicates the user's action in reality; a performance information generation unit that generates first performance information that changes the action of another avatar other than a user avatar that is the user's avatar in virtual space if the user is not performing the predetermined action; and an avatar action information generation unit that generates first performance information for the other avatar based on the first performance information.

2. The virtual space generating device according to claim 1, wherein the predetermined behavior includes a first behavior in which the user's movement exceeds a first threshold, and the determination unit determines whether the user is performing the first behavior.

3. The virtual space generating device according to claim 2, wherein the performance information generating unit generates second performance information different from the first performance information that changes the behavior of the other person's avatar in accordance with the first behavior when the user is performing the first behavior, and the avatar behavior information generating unit generates second performance information for the other person's avatar based on the second performance information.

4. The virtual space generation device according to claim 3, wherein the predetermined behavior further includes a second behavior in which the user's behavior exceeds a second threshold that is greater than the first threshold, the determination unit determines whether the user is performing the second behavior, the performance information generation unit generates third performance information that changes the behavior of the user avatar in accordance with the user's behavior if the user is not performing the second behavior, and the avatar behavior information generation unit generates third performance information for the user avatar based on the third performance information.

5. The virtual space generating device according to claim 4, wherein the other person's avatar is at least one of an other spectator avatar that is an avatar of another spectator and an actor avatar that is an avatar of a performer.

6. A virtual space generating device as described in claim 4, wherein the first performance information is a first action parameter that indicates a rate at which the avatar's action is changed or first interrupt action data that causes a predetermined action to be performed, the second performance information is a second action parameter that has a value different from that of the first action parameter or second interrupt action data that performs an action different from that of the first interrupt action data, and the third performance information is a third action parameter that has a value different from that of the first action parameter and the second action parameter.

7. A virtual space generation method comprising: determining whether a user is performing a predetermined action based on user action information indicating the user's action in reality; if the user is not performing the predetermined action, generating first performance information that changes the action of another avatar other than a user avatar that is the user's avatar in the virtual space; and generating first performance action information for the other avatar based on the first performance information.

8. A program for causing a computer to function as each unit of the virtual space generating device according to any one of claims 1 to 6.

Images