Image generating device, image generating method, and program

Abstract

This image generation device is for use in a live distribution system that distributes, in real time, music performed by a performer via a communication network to terminal devices of a plurality of viewers. The image generation device has: an acquisition unit that acquires action information according to actions of viewers who view a distribution; and an image generation unit (104) that generates an image in which avatar groups obtained by grouping avatars of a plurality of viewers disposed in a virtual space according to the musical performance are caused to perform an action on the basis of the action information.

Description

[Technical field]

[0001] The present invention relates to an image generating device and an image generating method. ,program Regarding. [Background technology]

[0002] There is a system that delivers live video footage of singing or musical performance (see, for example, Patent Document 1). In this system, performers such as singers and musicians perform in separate locations. Each performance location is equipped with a camera. The center synthesizes the footage captured by each camera and delivers it to receiving terminals as a distribution video. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2008-131379 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in an actual live performance, the performer performs while watching the reactions of the audience in the audience. Therefore, it is better to be able to watch the reactions of the viewers while performing in a live broadcast. Some viewers also want to know how to enjoy the music being played and want to know how other viewers are reacting. Therefore, it is desirable to be able to share the reactions of the viewers of the live broadcast.

[0005] The present invention has been made in consideration of the above circumstances, and has an object to share viewer reactions to live streaming. [Means for solving the problem]

[0006] One aspect of the present invention is an image generating device used in a live distribution system that distributes a piece of music performed by a performer to multiple viewers' terminal devices in real time via a communication network, and includes an acquisition unit that acquires motion information corresponding to the motions of the viewers watching the distribution, and an image generating unit that generates an image that causes a group of avatars, in which the avatars of the multiple viewers are divided into groups and placed in a virtual space corresponding to the performance, to move based on the motion information, and the image generating unit causes the avatars to move in accordance with the timing at which a specific part included in the distributed piece arrives, based on the motion information. The image generation unit is an image generation device that detects the number of viewers belonging to the group whose action information corresponding to the action to be counted is obtained from their terminal devices based on the acquired action information, and if the detected number is equal to or greater than a reference value, generates an image in which the action to be counted is performed as the action of a group of avatars belonging to the group. Moreover, one aspect of the present invention is an image generating device used in a live streaming system in which a song performed by a performer is distributed in real time to terminal devices of a plurality of viewers via a communication network, the image generating device having an acquisition unit that acquires motion information corresponding to the motion of the viewer watching the distribution, and an image generating unit that generates an image that causes a group of avatars, in which the avatars of the plurality of viewers are divided into groups and placed in a virtual space corresponding to the performance, to move based on the motion information, the image generating unit causes the avatars to move based on the motion information in accordance with the timing at which a predetermined part included in the song to be distributed arrives, the acquisition unit acquires motion information corresponding to the motion of the performer throwing an item, the image generating unit estimates the drop point of the item in the virtual space based on the motion information corresponding to the motion of throwing the item, and identifies an avatar that can receive the item in the virtual space in accordance with the estimated position as a special avatar.

[0007] Another aspect of the present invention is an image generating method executed by a computer for use in a live distribution system that distributes a piece of music performed by a performer to terminal devices of a plurality of viewers in real time via a communication network, the method comprising: acquiring motion information corresponding to motions of viewers viewing the distribution; and generating an image for moving a group of avatars, in which avatars of the plurality of viewers are divided into groups and placed in a virtual space corresponding to the performance, based on the motion information; the generating of the image includes causing the avatars to move in accordance with the timing at which a predetermined part included in the distributed piece of music arrives, based on the motion information. and detecting the number of viewers belonging to the group whose terminal devices have motion information corresponding to the motions to be counted based on the acquired motion information, and generating an image in which the motions to be counted are performed as motions of avatars belonging to the group when the detected number is equal to or greater than a reference value. An image generation method. Another aspect of the present invention is a program executed by a computer used in a live distribution system that distributes a piece of music performed by a performer to terminal devices of a plurality of viewers in real time via a communication network, the program acquiring motion information corresponding to the motions of the viewers viewing the distribution, and generating an image that causes a group of avatars, in which avatars of the plurality of viewers are divided into groups and placed in a virtual space corresponding to the performance, to move based on the motion information, the generating of the image including causing the avatars to move in accordance with the timing of the arrival of a predetermined part included in the distributed piece of music, based on the motion information. and, based on the acquired motion information, detects the number of motion information corresponding to the motion to be counted among the viewers belonging to the group obtained from the terminal devices of the viewers belonging to the group, and if the detected number is equal to or greater than a reference value, generates an image in which the motion to be counted is performed as a motion of the avatars belonging to the group. . Moreover, one aspect of the present invention is an image generation method executed by a computer for use in a live distribution system in which a song performed by a performer is distributed in real time to terminal devices of a plurality of viewers via a communication network, the image generation method comprising the steps of: acquiring motion information corresponding to the motions of the viewers watching the distribution; generating an image in which a group of avatars of a plurality of viewers who are placed in a virtual space corresponding to the performance are caused to move based on the motion information; acquiring motion information corresponding to the motion of the performer throwing an item; and generating the image, the image generation method including causing the motion to occur based on the motion information in accordance with the timing at which a predetermined part included in the song to be distributed arrives; and estimating a drop point in the virtual space where the item will fall based on the motion information corresponding to the motion of throwing the item; and identifying an avatar that is capable of receiving the item in the virtual space in accordance with the estimated position as a special avatar. Furthermore, one aspect of the present invention is a program executed by a computer used in a live distribution system that distributes a song performed by a performer to terminal devices of a plurality of viewers in real time via a communication network, the program acquiring motion information corresponding to the motions of the viewers watching the distribution, generating an image that causes a group of avatars, in which the avatars of a plurality of viewers are divided into groups and placed in a virtual space corresponding to the performance, to move based on the motion information, acquiring motion information corresponding to the motion of the performer throwing an item and generating the image includes causing the motion to occur based on the motion information in accordance with the timing at which a predetermined part contained in the song to be distributed arrives, and estimating a point at which the item will fall in the virtual space based on the motion information corresponding to the motion of throwing the item, and identifying an avatar that can receive the item in the virtual space according to the estimated position as a special avatar. Effect of the Invention

[0008] You can share viewers' reactions during live broadcasts. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a schematic block diagram showing the configuration of a live distribution system 1. [Diagram 2] FIG. 1 is a schematic functional block diagram showing the configuration of a live streaming device 10. [Diagram 3] FIG. 2 is a sequence diagram illustrating the flow of processing in the live distribution system 1. [Figure 4] FIG. 13 is a diagram showing an example of an image displayed on a display screen of a viewer's terminal device. [Diagram 5] FIG. 13 is a diagram showing an example of an image displayed on a display screen of a performer's terminal device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] An image generating apparatus according to an embodiment of the present invention will now be described with reference to the drawings. FIG. 1 is a schematic block diagram showing the configuration of a live distribution system 1 using an image generation device according to an embodiment of the present invention. The live streaming system 1 includes a live streaming device 10, a designer terminal 20, a group of performer devices P1, a group of performer devices P2, a group of viewer devices A1, and a group of viewer devices A2, which are connected via a network N so as to be able to communicate with each other.

[0011] The live distribution device 10 distributes (live distributes) content corresponding to a live performance by a performer to a viewer's terminal in real time. Live streaming device 10 can live stream a single piece of music, whether it is performed by performers gathering at one live venue or performing different parts of the same piece of music at different live venues. When performers perform at different live venues, live streaming device 10 synthesizes performance data obtained from the performer devices installed at each live venue, and transmits the synthesized data to the viewer's device as live streaming data. The live venue may be any place where performance is possible, such as a home, a studio, a live venue, etc.

[0012] The performer device group P1 and the performer device group P2 are used by performers appearing in a live performance. Here, an example is described in which a performer using the performer device group P1 and a performer using the performer device group P2 perform one song at different live venues. Note that one song may be performed at one live venue instead of multiple live venues. In that case, one performer device group is used. Here, a case is described in which there are two performer device groups, but if there are three or more performance locations, a performer device group may be provided at each performance location. For example, if there are different performance parts such as vocals, guitar, bass, drums, keyboard, etc., they can be performed from different performance locations using different performance device groups.

[0013] The performer devices group P1 includes a terminal device P11, a sound collecting device P12, and a camera P13. The terminal device P11 is communicatively connected to a sound collecting device P12 and a camera P13, and is also communicatively connected to a network N. The terminal device P11 is equipped with various input devices such as a mouse and a keyboard, or a touch panel, and is also equipped with a display device. The terminal device P11 is, for example, a computer.

[0014] The sound collection device P12 collects sound and outputs a sound signal corresponding to the collected sound to the terminal device P11. The sound collection device P12 may have any one of the functions of a sound sensor that collects the performance sound output from a musical instrument, an input device that inputs a sound signal output from an electronic musical instrument, and a microphone that collects the singing sound of a performer. Here, one sound collection device P12 is connected to the terminal device P11, but multiple sound collection devices may be connected. For example, when a performer plays a musical instrument while singing, a sound collection device as a microphone and a sound collection device that collects the sound of the musical instrument can be used. The camera P13 captures images of performers using the performer devices P1, and outputs the captured image data to the terminal device P11. The captured image data is, for example, video data.

[0015] The performer device group P2 includes a terminal device P21, a sound collecting device P22, and a camera P23. The terminal device P21 has the same functions as the terminal device P11, the sound collecting device P22 has the same functions as the sound collecting device P12, and the camera P23 has the same functions as the camera P13, so their explanations are omitted.

[0016] The designer terminal 20 is used by a designer in charge of directing content related to live distribution. The designer terminal 20 inputs setting information for moving avatars to the live distribution device 10. The setting information can be at least one of the following: the design of the venue, the movement patterns of avatars according to music, and group assignments for dividing the audience seats into groups.

[0017] The viewer device group A1 and the viewer device group A2 are each used by viewers who watch live broadcasts. The viewer device group A1 and the viewer device group A2 are each used by different viewers. The viewer device group A1 includes a terminal device A11 and a motion sensor A12. The terminal device A11 includes various input devices such as a mouse and a keyboard, or a touch panel, and a display device. The terminal device A11 is connected to the motion sensor A12 so as to be able to communicate with each other, and is also connected to the network N so as to be able to communicate with each other. The terminal device P11 may be, for example, a computer, a smartphone, a tablet, or any other device.

[0018] Terminal device A11 receives an image signal from live distribution device 10 and displays the image signal on a display screen. When displaying the image signal, terminal device A11 can change the viewing position in the virtual space in response to operational input from the viewer. Terminal device P11 generates three-dimensional information of the virtual space showing a live venue in the virtual space based on the image signal, and generates an image signal displaying three-dimensional information representing the live venue as seen from the specified viewing position. Terminal device P11 displays this generated image signal on a display screen.

[0019] The motion sensor A12 detects the motion of a viewer using the terminal device group A1, generates motion information according to the detected motion, and outputs the motion information to the outside. The motion sensor A12 is communicably connected to the terminal device A11, and outputs the motion information to the terminal device A11.

[0020] The motion sensor A12 captures an image of the viewer, detects the viewer's posture based on the captured image, and detects the viewer's movement from the detection result. The detected movement may be, for example, at least one of whether the viewer is standing up, whether the viewer is raising their arms, whether the viewer is swaying their body from side to side, whether the viewer is waving their hands from side to side, etc. In this case, the motion sensor A12 is attached near the display device of the terminal device A11, and the capture direction is adjusted in advance so that the viewer watching the live broadcast is within the capture range.

[0021] When detecting the viewer's movements based on an image, the motion sensor A12 can detect the body part moved by the viewer, the direction of the movement, the speed of the movement, etc., generate movement information including these, and output it to the terminal device. The motion sensor A12 may also generate movement information indicating whether or not a specific body part has moved. In this case, the motion sensor A12 can represent the movement information by a binary value indicating whether or not the movement has occurred. Therefore, the movement information can be represented by a small amount of information (low-dimensional information), and the load on the transmission process of transmitting the movement information from the viewer's terminal device to the live distribution device 10 can be reduced, and the load on the network N can also be reduced.

[0022] The motion sensor A12 may be an acceleration sensor, a gyro sensor, or the like. In this case, the motion sensor A12 is attached to the viewer's body or held by the viewer's hand. Even when an acceleration sensor or a gyro sensor is used, the motion sensor A12 may be configured to express the motion information of whether or not the viewer has moved by a binary value. The motion sensor A12 may have a function of connecting to the network N. In this case, the motion sensor A12 can transmit the operation information to the network N without the terminal device A11, instead of transmitting the operation information to the network N via the terminal device A11. Regarding whether the terminal device A11 and the motion sensor A12 are used by the same viewer, the same user may be identified by logging in using the same login ID and password when viewing a live broadcast. In addition, the user may be registered by inputting the individual identification number of the motion sensor A12 in the terminal device A11.

[0023] The viewer device group A2 includes a terminal device A21 and a motion sensor A22. The terminal device A21 has the same function as the terminal device A11, and the motion sensor A22 has the same function as the motion sensor A12, so a description thereof will be omitted.

[0024] FIG. 2 is a schematic functional block diagram showing the configuration of live streaming device 10. The live distribution device 10 includes a communication unit 101, a memory unit 102, a motion determination unit 130, an image generation unit 104, a sound processing unit 105, a synchronization processing unit 106, a CPU (Central Processing Unit) 107, and a destination information output unit 108. The communication unit 101 is connected to a network N and communicates with other devices via the network N. For example, the communication unit 101 has a function as an acquisition unit that acquires, from a terminal device of a viewer, motion information corresponding to the motion of the viewer viewing a live broadcast.

[0025] The storage unit 102 stores various types of data. For example, the storage unit 102 includes a venue data storage unit 1021 and an avatar storage unit 1022 . The venue data storage unit 1021 stores venue data that represents a live performance venue in a virtual space. The venue data may be three-dimensional data that represents the live performance venue in a three-dimensional space.

[0026] The avatar storage unit 1022 stores image data representing avatars to be placed at a live venue in the virtual space. The avatars may have the same design for each viewer, or may have different designs for at least some of the viewers. When an avatar is designed according to the viewer, the avatar storage unit 1022 stores avatar data representing the design of the avatar for each viewer (user). Here, at a goods store installed at the live venue in the virtual space, it is possible to purchase items such as avatar shapes, clothes and accessories that can be worn by the avatar, using electronic money or the like. Viewers can purchase avatar shapes and items according to their preferences at the goods store and set them for their own avatars.

[0027] The storage unit 102 is a storage medium, such as a hard disk drive (HDD), a flash memory, an electrically erasable programmable read only memory (EEPROM), a random access read / write memory (RAM), a read only memory (ROM), or any combination of these storage media. For example, a non-volatile memory can be used for the storage unit 102.

[0028] The motion determination unit 103 determines which motion the viewer is performing based on the motion information transmitted from the viewer's terminal device. The motion of the viewer may be, for example, at least one of whether the viewer is standing up, whether the viewer is raising his / her arms, whether the viewer is performing a motion of shaking his / her body from side to side, whether the viewer is performing a motion of waving his / her hands from side to side, and the like.

[0029] The image generating unit 104 generates an image signal according to the music played by the performers. The image generating unit 104 includes a stage synthesis unit 1041 and an audience synthesis unit 1042. The stage synthesis unit 1041 synthesizes image data of the performers performing the song at a position on the stage in the live concert venue in the virtual space indicated by the venue data. The audience seat synthesis unit 1042 synthesizes avatars corresponding to the viewers at the positions of the audience seats in the live concert venue in the virtual space.

[0030] The image generation unit 104 generates an image signal in which the stage synthesis unit 1041 synthesizes the image of the performers in the live venue in the virtual space, and the audience seat synthesis unit 1042 synthesizes the avatars of the viewers in the audience seats in the live venue in the virtual space. The image generation unit 104 transmits the generated image signal to the viewers' terminal devices (e.g., terminal device A11, terminal device A21) via the communication unit 101 and the network N.

[0031] The audience synthesis unit 1042 of the image generation unit 104 generates an image in which a group of avatars arranged in a virtual space to be live-streamed are moved based on the motion information obtained from the motion determination unit 103. The group of avatars is formed by dividing the avatars of multiple viewers into groups.

[0032] When generating an image for moving a group of avatars, the audience synthesis unit 1042 may move the group of avatars to which the avatar corresponding to the viewer belongs based on the movement information. For example, the position of a seat in a live venue in a virtual space is assigned to each viewer as seat data. This seat may be an individual seat assigned to one viewer, or an area capable of accommodating multiple viewers may be assigned. When the audience synthesis unit 1042 detects the movement information of a viewer, it determines the seat to which the viewer belongs based on the seat data. The audience synthesis unit 1042 identifies a group of viewers including the determined seat position. The audience synthesis unit 1042 reflects the movement information of the viewer in the movements of multiple avatars in the identified group.

[0033] Here, the audience synthesis unit 1042 may move the avatars of the entire group based on the movement information of one viewer belonging to the group. In this case, the avatars of the entire group can be moved according to the movement of the representative viewer of the group. Furthermore, the audience synthesis unit 1042 may analyze the tendency of the behavior of each viewer belonging to a group based on the behavior information of the multiple viewers belonging to the group, and move the avatars belonging to the group according to the analysis results. For example, the number of viewers in a certain group who jumped during a song is detected, and if the number of people who jumped is equal to or greater than a reference value (e.g., more than half), each avatar belonging to the group is moved (e.g., jumped). In this way, the avatars can be moved according to the tendency of the viewers' behavior for each group.

[0034] In this embodiment, the image generation unit 104 is described as being installed inside the live streaming device 10, but it may also be installed as an independent device separate from the live streaming device 10 and communicate with the live streaming device 10.

[0035] The sound processor 105 generates a sound signal according to the music played by the performer. The image generating unit 104 includes a mixer 1051 and a performance synchronization unit 1052 . The mixer 1051 synthesizes sound signals to be mixed among sound signals obtained from each performer device group. For example, the mixer 1051 inputs a sound signal of an instrument (e.g., guitar) played by an actor of the performer device group P1, a singing voice of an actor of the performer device group P1, and a sound signal of an instrument (e.g., bass) played by an actor of the performer device group P2, and generates a sound signal (sound signal of an accompaniment part) by mixing the sound signal of the instrument (e.g., guitar) played by an actor of the performer device group P1 and the sound signal of the instrument (e.g., bass) played by an actor of the performer device group P2. In this case, the mixer 1051 outputs two types of sound signals, the sound signal of the singing voice of the actor of the performer device group P1 and the sound signal of the accompaniment part.

[0036] The performance synchronization unit 1052 synchronizes the performance signals obtained from the performer devices of each part that performs one song. For example, the performance synchronization unit 1052 synchronizes the performance signal of the singing voice performed by the performer of the performer device group P1, the performance signal of the musical instrument performed by the performer of the performer device group P1, and the performance signal of the musical instrument performed by the performer of the performer device group P2.

[0037] The synchronization processor 106 synchronizes the image signal generated by the image generator 104 with the sound signal generated by the sound processor 105 .

[0038] The CPU 107 controls each unit within the live distribution device 10. The delivery destination information output unit 108 outputs delivery destination information for an item based on the personal information of the viewer corresponding to the special avatar.

[0039] At least one of the operation determination unit 103, the image generation unit 104, the sound processing unit 105, the synchronization processing unit 106, and the destination information output unit 108 may be realized by executing a computer program in a processing device such as the CPU 107, or may be realized by a dedicated electronic circuit.

[0040] Next, the operation of the above-mentioned live distribution system 1 will be described. FIG. 3 is a sequence diagram illustrating the flow of processing in the live distribution system 1. The designer terminal 20 transmits setting information to the live distribution device 10 in response to an operational input from the designer (Step S101). The live distribution device 10 receives the setting information transmitted from the designer terminal 20, and stores it in the storage unit 102 (Step S102). When the live streaming start time arrives, live streaming device 10 starts live streaming based on the setting information stored in memory unit 102 (step S103). Here, live streaming device 10 receives image signals and performance signals transmitted from performer device group P1 and performer device group P2, respectively, and distributes the image signals synthesized with the image signals to the live venue in the virtual space, and the performance signals. In response to the operation input from the viewer, the terminal device A11 transmits a request for live streaming to the live streaming device 10 together with a purchase request for a procedure to purchase an electronic ticket for viewing the live streaming (step S104). In response to this purchase request, the live streaming device 10 assigns the electronic ticket to the viewer of the terminal device A11 and a seat in the live venue in the virtual space, allowing the viewer to view the live streaming. Here, the electronic ticket may be purchased by reservation in advance, or may be sold at the stage of viewing the live streaming. In response to a request for live distribution from the terminal device A11, the live distribution device 10 synchronizes the image signal and the performance signal and transmits them to the terminal device A11 (step S105), thereby starting live distribution to the terminal device A11.

[0041] After the performance has started, a performance signal is transmitted from terminal device P11 (step S106), and when a performance signal is transmitted from terminal device P12 (step S107), the live streaming device 10 receives the performance signal. The live streaming device 10 broadcasts the received performance signal to the terminal devices of the viewers. Here, the live streaming device 10 can also broadcast the performance signal to each of the performers' terminal devices. When the performers' terminal devices receive the performance signal from the live streaming device 10, they output the performance signal to speakers, headphones, etc. As a result, the performance signal is output to the outside as sound. This allows the performers to perform their own performance while listening to the performance sounds of performers performing in other locations.

[0042] In response to operational input from the viewer, the terminal device A21 transmits a request for live distribution to the live distribution device 10 (Step S108). In response to a request for live distribution from the terminal device A21, the live distribution device 10 transmits an image signal and a performance signal to the terminal device A21 (step S109), thereby starting live distribution to the terminal device A21.

[0043] In addition to playing the music, the performers may also call out to the audience through a microphone, saying things like, "Look behind, can you hear me?" In this case, the audience members sitting at the back of the virtual live venue can respond by raising their arms or other body movements.

[0044] When the motion sensor A12 detects that the viewer has raised his / her arm, it outputs motion information according to the detection result to the terminal device A11. The terminal device A11 transmits the motion information obtained from the motion sensor A12 together with identification information of the terminal device A11 to the live distribution device 10 (Step S110). When the motion sensor A22 detects that the viewer has raised his / her arm, it outputs motion information according to the detection result to the terminal device A21. The terminal device A21 transmits the motion information obtained from the motion sensor A22 together with the identification information of the terminal device A21 to the live distribution device 10 (step S111).

[0045] Upon receiving the motion information from the terminal device A11 and the terminal device A21, the motion determination unit 130 of the live streaming device 10 determines the motion of the viewer based on each piece of motion information. Here, the motion determination unit 130 determines that the motion represented by each piece of motion information is a motion of raising an arm (Step S112).

[0046] The image generating unit 104 counts the motion information for each group of audience seats based on the determination result of the motion determining unit 103. For example, when the front row (seats near the stage) of the entire audience seats are assigned as the first group and the back row (seats far from the stage) are assigned as the second group, the image generating unit 104 counts the number of motion information pieces indicating arm raising received from the terminal devices of the audience members belonging to the first group, and determines whether the count result is equal to or greater than a reference value. Here, since the performer's call is directed to the back row, most of the audience members assigned to the front row seats do not raise their arms, and therefore the count does not exceed the reference value. In this case, the image generating unit 104 does not perform the motion of raising the arms for the avatars belonging to the first group. Here, even if several of the audience members belonging to the first group perform the motion of raising their arms, it is possible to prevent any of the avatars arranged at the position corresponding to the first group in the live venue in the virtual space from raising their arms.

[0047] On the other hand, if the performer's appeal is directed to the back row and many of the viewers assigned to the back row seats are excited by the music, each of the viewers will raise their arms. In this case, the number of pieces of motion information in which viewers belonging to the second group have raised their arms exceeds the reference value. In this case, image generating unit 104 causes the avatars belonging to the second group to perform the motion of raising their arms. In this way, an image of each avatar belonging to the second group raising their arm is synthesized (step S113).

[0048] When the images are combined, the image generating unit 104 transmits image signals indicating that the avatars belonging to the second group have raised their arms to the terminal devices of the viewers and the terminal devices of the performers via the communication unit 101 and the network N (step S114). As a result, an image is displayed on the display screen of each of the terminal devices of the viewers (for example, terminal device A11 and terminal device A21) showing that the avatars arranged in the front row have their arms lowered and that the avatars arranged in the back row have their arms raised. Similarly, an image is displayed on the display screen of the terminal device of the performers (terminal device A11 and terminal device A21) showing that the avatars arranged in the front row have their arms lowered and that the avatars arranged in the back row have their arms raised.

[0049] FIG. 4 is a diagram showing an example of an image displayed on the display screen of a viewer's terminal device. An image of the stage viewed from the audience seating side is displayed on the viewer's terminal device. An image 410 showing performers performing on a stage located in front is displayed on a display screen 400 of a viewer's terminal device, and an image 420 showing the audience seats is displayed in front of the stage. In the audience seats, the seats in the area close to the stage (front row side) are assigned as a first group 421, and the seats in the area away from the stage (back row side) are assigned as a second group 422. As shown in this figure, viewers in the front row can see on the screen that the avatars in the front row have not raised their arms, and can therefore understand that if they had not raised their arms, they would have been able to react in the same way as the other viewers. Even if they have raised their arms, the avatars corresponding to them are not raising their arms, and so they can understand that their actions were different from those of the people around them. Furthermore, the actions of their own avatars can be displayed as if they are making the same actions as the viewers around them.

[0050] Furthermore, viewers in the back row can see on the screen that the avatars in the back row are raising their arms, and can therefore understand that if they had raised their arms, they would have reacted in the same way as the other viewers. Even if they did not raise their arms, the avatar corresponding to them is displayed with their arms raised, and so they can understand that their actions were different from those of the people around them. Furthermore, the actions of their own avatars can be displayed as if they were making the same actions as the viewers around them.

[0051] FIG. 5 is a diagram showing an example of an image displayed on the display screen of the performer's terminal device. An image of the audience seen from the stage side is displayed on the performer's terminal device. That is, the image generating unit 104 may have a function of generating an image signal of the stage seen from the audience side, or may have a function of generating an image signal of the audience seen from the stage side. On the display screen 500 of the performer's terminal device, an image 510 showing the state on stage is displayed in the foreground, and an image 520 showing the state of the audience seats is displayed behind the stage. In the audience seats, the seats in the area close to the stage (front row side) are assigned as a first group 521, and the seats in the area away from the stage (back row side) are assigned as a second group 522. The performer can see on the screen that the avatars in the front row (first group) are not raising their arms, but the avatars in the back row (second group) are raising their arms, so he can understand that the audience in the back row have responded to his call. Here, if the back row audience members' arms are displayed by individual avatars, rather than by group, there will be variation in whether they have raised their arms. In this case, even if a certain number of people in the back row seats have raised their arms, the audience members' reactions to the performance will not necessarily be fully felt. In this embodiment, if the number of back row audience members who have raised their arms is equal to or greater than a reference value, the avatars belonging to the back row will each raise their arms, so the audience members can feel a sense of satisfaction in the audience members' reactions to the performance, and can enjoy the performance even more. In addition, the audience members' reactions to the performance are displayed by group, so the audience members can feel a sense of unity through the song. In particular, in live distribution, it is difficult for a performer to grasp the audience's reaction to his / her performance, since the performer is not performing in front of the actual audience. During live distribution, text data representing comments may be sent from the terminal device based on the viewer's operation input, and displayed on the performer's screen. In this case, the audience's reaction can be expressed by a character string, but since the audience's reaction felt by the performer in an actual live venue is cheers and body movements, even if the character string is displayed on the screen, the expression of the audience's reaction is different from that of an actual live venue. Therefore, it is difficult for the performer to grasp the audience's reaction. According to this embodiment, the avatars are made to move in groups according to the audience's movements, so the performer can feel as if he / she is watching the audience's movements in an actual live venue.

[0052] In the embodiment described above, the image generating unit 104 displays an image according to whether or not the arm is raised, but the movement of the avatar can be displayed in other display modes. For example, when a song with a slow rhythm is played, a viewer may listen to the song while swaying his / her body from side to side to the rhythm. In this case, each motion sensor detects the movement of swaying the body from side to side and outputs it as movement information. The viewer's terminal device transmits this movement information to the live distribution device 10.

[0053] Based on this motion information, the image generating unit 104 generates an image of the avatars for each group shaking their bodies from side to side, and delivers the image to the terminal device. By looking at this image, the performers can understand whether the audience is following the rhythm and enjoying the song. Also, because the audience moves to the rhythm as a group, they can feel a sense of unity in each area of ​​the live venue. By watching the movements around their seat, viewers can see the rhythm to which other viewers are swaying. This allows them to sway their bodies to the same rhythm as other viewers, and share the enjoyment of the music.

[0054] Furthermore, the image generating unit 104 may extract a frequency according to the movement, and generate an image in which the avatars move according to the extracted frequency. When extracting the frequency, the image generating unit 104 performs a Fourier transform on the movement information to calculate the signal strength for each frequency included in the movement information. Then, the image generating unit 104 extracts any frequency from the entire frequency band according to the obtained signal strength. The image generating unit 104 may extract the frequency with the highest signal strength. When a viewer is swaying his / her body from side to side, a frequency may be extracted from the periodic swaying motion, and the avatars may be made to sway from side to side in accordance with that frequency. When the group of avatars is made to move according to the frequency, rather than simply reproducing the viewer's movements, characteristics of periodic movements can be extracted from the viewer's movements and the group of avatars can be made to move according to those characteristics.

[0055] Furthermore, the image generating unit 104 may calculate a signal strength corresponding to the movement based on the movement information, and generate an image in which the avatars move according to the calculated signal strength. When extracting the signal strength, the image generating unit 104 performs a Fourier transform on the movement information to calculate the signal strength for each frequency included in the movement information. Then, one of the signal strengths calculated for each frequency is extracted. For example, the image generating unit 104 may extract the largest signal strength.

[0056] Furthermore, the image generating unit 104 may display an avatar that moves in accordance with the viewer's movements together with the group of avatars. Specifically, the image generating unit 104 displays the viewer's own avatar together with the group of avatars that move in the same way in group units. In this case, the image generating unit 104 moves the viewer's own avatar in accordance with the movement information obtained from the viewer. As a result, on the viewer's terminal device, an avatar that moves in accordance with the viewer's movements is displayed together with the group of avatars that move in group units. Therefore, the viewer can identify which avatar is his / her own in the live venue by finding the avatar that moves in accordance with his / her movements from among the multiple avatars arranged in the live venue in the virtual space.

[0057] Alternatively, an image displaying an avatar that moves in accordance with the viewer's own movement together with a group of avatars moving in groups may be generated by the image generating unit 104, or may be generated in the viewer's terminal device. For example, the live streaming device 10 distributes images of the group of avatars moving in groups to the viewer's terminal device. The viewer's terminal device displays an image signal on a display screen based on the image signal distributed from the live streaming device 10, and displays an avatar corresponding to the viewer by synthesizing it with the image signal. This allows the avatar corresponding to the viewer to be synthesized in the viewer's terminal device without being synthesized in the live streaming device 10. This reduces the processing load of synthesizing images in the live streaming device 10.

[0058] Furthermore, the image generating unit 104 may perform an action according to the timing of the arrival of a specific part included in the song to be live-streamed, based on the action information. When a song is played at an actual live venue, the audience may enjoy the song by making the same action, such as jumping all at once, when a specific part of the song arrives. Some songs are known to perform a customary action according to the timing of the specific part of the song. When it is known in advance that such a song will be played in live streaming, the timing of the specific part of the song and the action pattern for moving the avatar are associated with each other and stored in advance in the storage unit 102. When it is determined that the number of viewers who performed a specific action is equal to or greater than a reference value based on the action information collected when the specific part arrives after the song starts to be played, the image generating unit 104 causes the avatars to move according to the action pattern of the avatars associated with the timing of the song.

[0059] Here, it is possible to determine in advance what kind of action the avatars will take according to the action information when a specific part of the song arrives. For this reason, for example, the designer can consider how to produce the images in the live distribution, and register the results of the consideration in advance in the storage unit 102 of the live distribution device 10 from the designer terminal 20. Also, here, a plurality of patterns of movement may be registered in advance, and the designer may input which action to take from the designer terminal 20, taking into consideration the excitement of the viewers watching the live distribution for the song and the atmosphere of the entire venue. This allows the image generation unit 104 to perform an action at the timing when a specific part of the song arrives, according to the action pattern instructed from the designer terminal 20 out of several action patterns.

[0060] Also, when the genre of the music is classical, in an actual concert hall, the audience may listen to the music without moving their bodies much during the performance, and may give a standing ovation after the performance ends. In such a case, if the audience stands up or claps before the performance ends, the avatars may stand up or clap in the middle of the music. In this case, the performer may want the audience to enjoy the music without making any movements during the performance. In such a case, the timing when the music ends and the movements for the standing ovation are stored in advance in the storage unit 102, and when the image generating unit 104 detects that the music has ended, it moves the avatars based on the movement information. This allows the performer to grasp the audience's reaction after the performance ends. Also, even if the audience stands up or claps shortly before the music ends, the avatars can be made to move according to the timing when the music ends, without moving in the middle of the music, in relation to other audiences and performers.

[0061] Also, the image generating unit 104 may be configured to perform an operation according to the genre of the music to be live-delivered. For example, if the music being played is jazz, visitors at the actual venue may enjoy the music by swaying their bodies to the rhythm of the music. If the music being played is classical, visitors may listen to the music without moving their bodies much during the song, and clap their hands after the performance has ended. If the music being played is pop, visitors may enjoy the music by clapping their hands or raising their hands and swaying them from side to side while listening to the music. In this way, the way in which a song is enjoyed may differ depending on the genre of the song.

[0062] The storage unit 102 stores the genre of the music and the movement patterns of the avatars in association with each other. The image generation unit 104 determines the genre of the music to be played, and reads out a movement pattern corresponding to the determined genre of the music from the storage unit 102. The image generation unit 104 may cause the avatars to move in response to the obtained movement information according to the read movement pattern. When the songs to be played in the live distribution are decided in advance, the storage unit 102 stores genre information indicating the genre for each live performance or for each song. The image generating unit 104 may determine the genre of the songs to be played by reading out this genre information.

[0063] In the above-described embodiment, the avatars may be made to move in the same manner as the movements represented by the movement information obtained from the viewer, or may be made to move in a different manner. For example, the storage unit 102 stores in advance movement patterns of the avatars according to songs and parts. When there is a change in the movement based on the movement information, or when a specific movement is performed, the image generating unit 104 may read out the movement pattern from the storage unit 102 and make the avatars move according to the movement pattern. In this way, when making the avatars jump, the viewer can make the avatars move in a different manner from the detected movement by making simple movements such as raising their hands or moving them left and right, without actually jumping.

[0064] The communication unit 101 may also acquire motion information corresponding to the motions of the performers in the performer equipment group P1. For example, a motion sensor may be provided in one of the performer equipment groups. Here, a case will be described in which a motion sensor is provided in the performer equipment group P1. The motion sensor generates motion information corresponding to the motions of the performers and outputs it to the terminal device P11. The terminal device P11 transmits the motion information obtained from the motion sensor to the live streaming device 10. The live streaming device 10 generates an image signal based on the motion information transmitted from the terminal device P11. For example, in a real live concert venue, performers may give gifts to attendees by throwing picks, towels, or balls from the stage into the audience. It would be preferable to realize such a performance in a live broadcast in order to create a more live-like atmosphere.

[0065] The image generating unit 104 detects the movement of the performer throwing the object, and estimates the falling position of the object in the virtual space according to the movement. The image generating unit 104 can estimate the trajectory and falling position of the thrown object by calculating the throwing direction and the throwing movement speed based on the movement information obtained from the performer. The image generating unit 104 draws an image of the object along the obtained trajectory, and distributes it to each terminal device of the viewers. As a result, the terminal device of the viewers can confirm on the display screen that the object has been thrown by the performer into the audience, and the trajectory of the thrown object.

[0066] When an object falls near the viewer's position in the virtual space, the viewer makes a gesture to receive the object, such as raising his or her hand. In response to this gesture, the motion sensor detects the object and transmits the detected object to the live distribution device 10. The live distribution device 10 determines avatars within a certain range including the point where the item will fall as one group, generates images that move the avatars based on the movement information of the viewers corresponding to that area, synthesizes the images with the images in the virtual space, and distributes them. This allows an image corresponding to the performer's action of throwing the item to be distributed, and the reaction in the venue corresponding to the action of throwing the item to be shared with the audience and the performers. The image generating unit 104 specifies an avatar that can receive the item in the virtual space according to the estimated drop position as a special avatar. For example, the image generating unit 104 determines the avatar that is closest to the drop position as the special avatar. Here, the image generating unit 104 may select one avatar from among the avatars that are located within a certain range and that have performed the action of receiving the item as the special avatar. The live distribution system 1 includes a destination information output unit 108. The destination information output unit 108 may output destination information for an item based on personal information of a viewer corresponding to a special avatar. The destination information includes, for example, the address and name of the viewer. The destination information may further include a telephone number. The storage unit 102 may store, for example, personal information at the time of user registration. An operator who operates the live distribution device 10 receives the item from the performer. Then, the operator ships the item based on the personal information output from the live distribution device 10. This allows the viewer to actually receive the item and feel special joy. Here, an actual item may be shipped to the viewer, or an item for decorating the avatar may be given to the viewer. When an item is given, the viewer to whom the item is given can have the item displayed in a virtual space with the item attached to his or her avatar. This allows the viewer to share the fact that the item has been obtained with other users and performers.

[0067] In the above embodiment, the image generating unit 104 generates an image in which the avatars move. However, the image generating unit 104 may transmit information indicating that the avatars are to move to a terminal device of a performer or a terminal device of a viewer via the communication unit 101 and the network N. In this case, the terminal device may generate an image in which the avatars move based on the information indicating that the avatars are to move, and display the image on a display screen. This reduces the load of image processing on the live distribution device 10, and also reduces the amount of information transmitted to the terminal device via the network N. This allows the viewer's reaction to be displayed on the display screen even if a small amount of information is transmitted.

[0068] In addition, a program for implementing the functions of the processing unit in Fig. 1 may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read into a computer system and executed to perform construction management. Note that the term "computer system" here includes hardware such as the OS and peripheral devices.

[0069] In addition, if a WWW system is used, the "computer system" also includes the homepage providing environment (or display environment). In addition, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. Furthermore, the term "computer-readable recording medium" refers to storage devices that hold programs for a certain period of time, such as volatile memory in a computer system that is a server or client. The above-mentioned program may be for implementing part of the above-mentioned functions, or may be capable of implementing the above-mentioned functions in combination with a program already recorded in the computer system. The above-mentioned program may be stored in a specified server, and the program may be distributed (downloaded, etc.) via a communication line in response to a request from another device.

[0070] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and designs that do not deviate from the gist of the present invention are also included. [Explanation of symbols]

[0071] 1···Live distribution system, 10···Live distribution device, 20···Designer terminal, 101···Communication unit, 102···Storage unit, 103···Motion determination unit, 104···Image generation unit, 105···Sound processing unit, 106···Synchronization processing unit, 107···CPU, 108···Destination information output unit, 130···Motion determination unit, 1021···Venue data storage unit, 1022···Avatar storage unit, 1041···Stage synthesis unit, 1042···Audience seat synthesis unit, 1051···Mixer, 1052···Performance synchronization unit

Claims

1. An image generating device used in a live distribution system that distributes a piece of music performed by a performer to multiple viewers' terminal devices in real time via a communication network, An acquisition unit that acquires motion information corresponding to a motion of a viewer viewing the distribution; an image generating section that generates an image for moving a group of avatars, which are groups of avatars of a plurality of viewers arranged in a virtual space according to the performance, based on the motion information; having The image generating unit includes: performing an action according to a timing when a predetermined part included in the song to be distributed arrives based on the action information; The image generating unit includes: Based on the acquired motion information, the number of motion information corresponding to the motion to be counted among the viewers belonging to the group that is obtained from the terminal devices of the viewers belonging to the group is detected, and if the detected number is equal to or greater than a reference value, an image is generated in which the motion to be counted is performed as a motion of the avatars belonging to the group. Image generating device.

2. The image generating unit displays, on a display screen of the terminal device of the viewer, an avatar corresponding to the viewer and avatars around the seat of the avatar corresponding to the viewer so that they appear to be performing the same action.

2. The image generating device of claim 1.

3. The image generating unit includes: Based on the motion information, a frequency corresponding to the motion is extracted, and an image in which the group of avatars moves according to the extracted frequency is generated.

3. The image generating device according to claim 1 or 2.

4. The image generating unit includes: Calculating a signal strength corresponding to the movement based on the movement information, and generating an image in which the group of avatars moves according to the calculated signal strength. The image generating device according to any one of claims 1 to 3.

5. The image generating unit includes: Displaying both an avatar that moves in response to the viewer's actions and a group of avatars that move in the same way in groups 5. An image generating device according to claim 1.

6. The image generating unit includes: Operate according to the genre of the music to be distributed 6. An image generating device according to any one of claims 1 to 5.

7. The acquisition unit acquires motion information corresponding to a motion of the performer throwing an item, The image generation unit estimates a drop point of the item in the virtual space based on motion information corresponding to the motion of throwing the item, and identifies an avatar that can receive the item in the virtual space according to the estimated position as a special avatar.

7. An image generating device according to any one of claims 1 to 6.

8. The image generating device performs the distribution to a terminal device of the viewer and a terminal device used by the performer, The image generating unit generates an image of an audience seat viewed from a stage where the performers are performing, the image being for moving the avatars divided into groups in the audience seat, and displays the image of the audience seat viewed from the stage on a terminal device used by the performers.

2. The image generating device according to claim 1.

9. An image generating device used in a live distribution system that distributes a piece of music performed by a performer to multiple viewers' terminal devices in real time via a communication network, An acquisition unit that acquires motion information corresponding to a motion of a viewer viewing the distribution; an image generating section that generates an image for moving a group of avatars, which are groups of avatars of a plurality of viewers arranged in a virtual space according to the performance, based on the motion information; having The image generating unit includes: performing an action according to a timing when a predetermined part included in the song to be distributed arrives based on the action information; The acquisition unit acquires motion information corresponding to a motion of the performer throwing an item, The image generation unit estimates a drop point of the item in the virtual space based on motion information corresponding to the motion of throwing the item, and identifies an avatar that can receive the item in the virtual space according to the estimated position as a special avatar. Image generating device.

10. The display device further includes a destination information output unit that outputs destination information of the item based on personal information of the viewer corresponding to the special avatar.

10. An image generating device according to claim 7 or claim 9.

11. 1 is a computer-executed image generating method for use in a live distribution system that distributes a piece of music performed by a performer to multiple viewers' terminal devices in real time via a communication network, Obtaining motion information according to the motion of a viewer viewing the distribution; generating an image of an avatar group in which the avatars of the plurality of viewers are divided into groups and placed in a virtual space according to the performance, and moving the avatar group based on the movement information; generating the image and performing an action according to a timing when a predetermined part included in the song to be distributed arrives based on the action information; Based on the acquired motion information, the number of motion information corresponding to the motion to be counted among the viewers belonging to the group that is obtained from the terminal devices of the viewers belonging to the group is detected, and if the detected number is equal to or greater than a reference value, an image is generated in which the motion to be counted is performed as a motion of the avatars belonging to the group. The image generating method includes:

12. A program executed by a computer for use in a live distribution system that distributes a piece of music performed by a performer to multiple viewers' terminal devices in real time via a communication network, Obtaining motion information according to the motion of a viewer viewing the distribution; generating an image of an avatar group in which the avatars of the plurality of viewers are divided into groups and placed in a virtual space according to the performance, and moving the avatar group based on the movement information; generating the image and performing an action according to a timing when a predetermined part included in the song to be distributed arrives based on the action information; Based on the acquired motion information, the number of motion information corresponding to the motion to be counted among the viewers belonging to the group that is obtained from the terminal devices of the viewers belonging to the group is detected, and if the detected number is equal to or greater than a reference value, an image is generated in which the motion to be counted is performed as a motion of the avatars belonging to the group. A program that includes:

13. 1 is a computer-executed image generating method for use in a live distribution system that distributes a piece of music performed by a performer to multiple viewers' terminal devices in real time via a communication network, Obtaining motion information according to the motion of a viewer viewing the distribution; generating an image of an avatar group in which the avatars of the plurality of viewers are divided into groups and placed in a virtual space according to the performance, and moving the avatar group based on the movement information; acquiring motion information corresponding to the motion of the performer throwing an object; generating the image and performing an action according to a timing when a predetermined part included in the song to be distributed arrives based on the action information; and estimating a drop point of the item in the virtual space based on motion information corresponding to the motion of throwing the item, and specifying an avatar that can receive the item in the virtual space according to the estimated drop point as a special avatar. Image generation method.

14. A program executed by a computer for use in a live distribution system that distributes a piece of music performed by a performer to multiple viewers' terminal devices in real time via a communication network, Obtaining motion information according to the motion of a viewer viewing the distribution; generating an image of an avatar group in which the avatars of the plurality of viewers are divided into groups and placed in a virtual space according to the performance, and moving the avatar group based on the movement information; acquiring motion information corresponding to the motion of the performer throwing an object; generating the image and performing an action according to a timing when a predetermined part included in the song to be distributed arrives based on the action information; and estimating a drop point of the item in the virtual space based on motion information corresponding to the motion of throwing the item, and specifying an avatar that can receive the item in the virtual space according to the estimated drop point as a special avatar. program.

Images