Program and computer system

Abstract

To provide a new technique for assisting in a user pertaining to distribution of animations using an on-the-spot character.SOLUTION: A computer system generates an on-the-sport animation 12 obtained by combining an animation of an on-the-sport character 7 for broadcasting a play animation 10 of game play on the spot with the play animation 10. The computer system also analyzes a play situation of game play. Based on the analysis result, the on-the-sport character 7 is controlled automatically. An animation of the on-the-spot character 7 included in the on-the-spot animation 12 becomes an animation based on the analysis result in which the on-the-spot character 7 is controlled automatically.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a program for enabling a computer to realize a function for generating moving images. [Background technology]

[0002] In recent years, so-called "YouTubers" have become known, who edit gameplay videos and add other audio (for example, commentary, commentary, and audio of chats with friends) to create commentary videos and distribute them on video distribution sites. Related technologies are also known (for example, see Patent Document 1). In recent years, a form of video commentary made by fictional characters called "Vtubers" instead of YouTubers has also become known. Hereinafter, such fictional characters will be referred to as "commentary characters." [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2017-182603 A Summary of the Invention [Problem to be solved by the invention]

[0004] Generally, for a YouTuber to distribute a live commentary video, they must go through the following steps: preparing a gameplay video, editing the video, adding a live commentary voice (usually the YouTuber's user voice), and uploading the video to a distribution site. Furthermore, to have a live commentary character appear in the live commentary video, it is necessary to set the character's movements as if it were a live announcer or a sports program personality, generate a video of the character, and synthesize this into the gameplay video. Setting the character's movements and preparing the video of the character require a certain amount of equipment and time, so many users who wanted to try being a VTuber were unable to get started.

[0005] In addition, the action settings of the commentary character and the preparation of the video of the commentary character also placed restrictions on the format of the live video distribution. Specifically, in the case of live video distribution in which the YouTuber himself plays the game as a player and provides commentary in real time, the YouTuber must simultaneously play three roles: playing the game, providing commentary or commentary, and operating the commentary character. For this reason, it was difficult to realize the act of distributing games in genres in which the gameplay cannot be stopped even for a moment (for example, action games, shooting games, flight simulators, etc.) in a live format using a commentary character.

[0006] The present invention has been devised with the objective of providing a new technique for assisting users involved in video distribution using commentary characters. [Means for solving the problem]

[0007] A first invention for solving the above-mentioned problems is a program for causing a computer system to function as a commentary moving image generating means (e.g., server system 1100 of FIG. 1, commentary moving image generating server 1102, server processing unit 200s of FIG. 9, commentary moving image generation control unit 228, image generating unit 292s, step S16 of FIG. 19) that generates a moving image of a commentary character who provides commentary on a game play video and a commentary moving image that combines the game play video, an analysis means (e.g., server system 1100 of FIG. 1, commentary moving image generating server 1102, server processing unit 200s of FIG. 9, analysis unit 224, step S34 of FIG. 19) that analyzes the play situation of the game play, and a commentary character control means (e.g., server system 1100 of FIG. 1, commentary moving image generating server 1102, server processing unit 200s of FIG. 9, commentary character control unit 230, steps S60 and S62 of FIG. 20) that controls the commentary character based on the analysis results of the analysis means.

[0008] The term "computer system" as used herein refers not only to a single computer, but also to a system consisting of multiple computers working together.

[0009] According to the first invention, the play situation of the gameplay displayed as a gameplay video can be automatically analyzed, and the commentary character can be automatically controlled based on the analysis result. Therefore, the user does not need to create the commentary character's movements one by one and prepare a video in which the commentary character is controlled. In addition, since the need for user operations related to the movement control of the commentary character can be significantly reduced, the user can easily generate a live-style commentary video using a commentary character by simply inputting commentary or commentary audio even while playing the game. A new technology can be provided to assist users in video distribution using a commentary character.

[0010] The second invention is a program of the first invention, which causes the computer system to function as a voice acquisition control means (e.g., server system 1100 in Figure 1, live video generation server 1102, server processing unit 200s in Figure 9, voice acquisition control unit 222, step S12 in Figure 19) that controls the acquisition of a user's voice, and the live character control means has a voice response control means that performs voice response control to control the movement of the live character based on the voice acquired by the voice acquisition control means.

[0011] According to the second invention, the behavior of the commentary character can be automatically controlled in response to the user's voice. For example, when a user's voice indicating surprise is acquired, it is possible to automatically control the commentary character to perform a surprised behavior. This adds a live feel to the behavior of the commentary character, making the commentary video richer in expression than when voice response control is not performed. Also, from the user's perspective, since the commentary character can be controlled by voice, the work involved in creating the commentary video and distributing the commentary video in a live format becomes easier.

[0012] A third invention is a second invention program, in which the live video generation means generates the live video by using a video of the gameplay being performed by the user as the play video, and the audio acquisition control means controls the acquisition of audio of the user while playing the game.

[0013] According to the third invention, the user can input voice while playing and control the commentary character with that voice, which makes it easier to create a commentary video of the gameplay video.

[0014] A fourth invention is a program of the third invention, in which the voice response control means controls the movement of the commentary character based on the voice acquired by the voice acquisition control means when the play situation satisfies a voice reception situation condition indicating that the voice response control is acceptable.

[0015] According to the fourth aspect of the present invention, voice input is permitted only when the play situation satisfies the voice reception situation condition, and the commentary character can be made to act according to the voice input at that time. In other words, it is possible to appropriately determine the timing at which the commentary character's action can be controlled by voice according to the play situation.

[0016] A fifth invention is the program of any one of the second to fourth inventions, in which the live video generation means generates the live video using the user's voice acquired by the voice acquisition control means as live voice.

[0017] According to the fifth invention, a live video can be generated in which the user's voice is used as the live voice.

[0018] The sixth invention is a program of the first invention, which causes the computer system to function as a comment acquisition control means (e.g., comment acquisition control unit 229 in FIG. 23) that controls the acquisition of user comments, and the live commentary character control means has a comment response control means (e.g., comment response control unit 236 in FIG. 23) that performs comment response control to control the behavior of the live commentary character based on the comments acquired by the comment acquisition control means.

[0019] According to the sixth invention, it becomes possible to control the commentator character based on comments input by the user.

[0020] A seventh invention is a program of the sixth invention, wherein the live video generation means generates the live video by using a video of the gameplay being performed by the user as the play video, and the comment acquisition control means controls the acquisition of comments made by the user while playing the game.

[0021] According to the seventh invention, the user can input comments while playing the game and can control the commentator character with the comments.

[0022] An eighth invention is a program of the seventh invention, in which the comment response control means controls the action of the commentary character based on the comment acquired by the comment acquisition control means when the play situation satisfies a comment reception situation condition indicating that the comment response control is acceptable.

[0023] According to the eighth aspect of the present invention, it is possible to appropriately determine the timing at which the action of the commentator character can be controlled using a comment, depending on the playing situation.

[0024] A ninth invention is a program of any of the first to eighth inventions for causing the computer system to function as a distribution control means (e.g., server system 1100 in FIG. 1, distribution server 1103, server processing unit 200s in FIG. 9, distribution control unit 240, step S16 in FIG. 19) that controls the distribution of the live video or controls the distribution of the live video by uploading data of the live video to a predetermined distribution system.

[0025] According to the ninth aspect of the present invention, live video can be distributed.

[0026] A tenth invention is a program of any of the first to ninth inventions for causing the computer system to function as a layout setting means (e.g., server system 1100 in FIG. 1, live video generation server 1102, server processing unit 200s in FIG. 9, layout setting unit 226, step S18 in FIG. 19) that sets the layout of the gameplay video and the commentary character in the live video.

[0027] According to the tenth aspect of the present invention, it becomes possible to set the relative positions and relative sizes of the gameplay animation and the commentary character in the commentary animation, that is, to set the so-called screen layout.

[0028] An eleventh invention is a program of any of the first to tenth inventions, in which the commentary character control means has a movement control pattern selection control means (e.g., server system 1100 of Figure 1, live video generation server 1102, server processing unit 200s of Figure 9, movement control pattern selection control unit 234, step S58 of Figure 20) that presents selection candidates for the movement control pattern of the commentary character to a user and controls the movement of the commentary character based on the movement control pattern of the selected selection candidate.

[0029] According to the eleventh aspect of the present invention, the user can select from the selection candidates what he / she thinks is appropriate as the commentary character's behavior. For example, even if the commentary character should perform a surprised behavior, the nuance of the surprised behavior differs depending on whether the character is genuinely surprised or just pretending to be surprised. Depending on which behavior the commentary character performs, the character's personality changes, so the user can manage the character's personality settings in detail.

[0030] A twelfth invention is a program of any of the first to eleventh inventions, which causes the computer system to function as a movement control pattern setting means (e.g., server system 1100 in Figure 1, live video generation server 1102, server processing unit 200s in Figure 9, movement control pattern setting unit 220 in Figure 19) that sets a movement control pattern of the commentary character based on a user's operational input, and the commentary character control means controls the movement of the commentary character based on the movement control pattern set by the movement control pattern setting means.

[0031] According to the twelfth aspect of the present invention, the user can set the movement control pattern of the commentary character. Therefore, the commentary character can be made to perform an original movement that is not included in the existing movement control patterns prepared by the manufacturer or administrator.

[0032] A thirteenth invention is the program according to any one of the first to twelfth inventions, wherein the commentary character control means controls and changes an appearance of the commentary character.

[0033] According to the thirteenth aspect of the present invention, the appearance of the commentary character can be changed together with the change in the movement control of the commentary character. For example, it is possible to change the appearance of the commentary character by making the commentary character act as if surprised and by making the character's face turn pale.

[0034] A fourteenth invention is the program according to any one of the first to thirteenth inventions, wherein the analysis means performs image analysis on the play video and analyzes the play situation based on a result of the image analysis.

[0035] According to the fourteenth aspect of the present invention, the playing situation can be analyzed by performing image analysis on the play video.

[0036] A fifteenth invention is the program according to any one of the first to fourteenth inventions, wherein the analysis means analyzes operation input information of the game play, and analyzes the play situation based on the analysis result.

[0037] According to the fifteenth aspect, the playing situation can be analyzed based on the operation input information of the game play.

[0038] A 16th invention is a program described in any one of claims 1 to 15, wherein the live video generation means generates the live video by changing the playback speed of the video of the live character and / or the gameplay video based on the analysis results of the analysis means.

[0039] There is a video production technique in which impressive or shocking scenes are played in slow motion to more effectively convey the impression or impact. There is also a video production technique in which scenes that are necessary for the passage of time or explaining a situation but are not important from the viewer's perspective are fast-forwarded to maintain the tempo of the story. According to the sixteenth aspect of the present invention, such video expression techniques can be automatically applied to live commentary videos, making it possible to generate live commentary videos with rich dramatic effects.

[0040] A seventeenth invention is a computer system comprising: a video of a commentary character providing commentary on game play video; a commentary video generation means for generating a commentary video by combining the game play video; an analysis means for analyzing a play situation of the game; and a commentary character control means for controlling the commentary character based on an analysis result of the analysis means.

[0041] According to the seventeenth aspect, a computer system can be realized which can provide the same effects as the first aspect. [Brief description of the drawings]

[0042] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a live video generation and distribution system. [Diagram 2] FIG. 4 is a front view showing a configuration example of a user terminal. [Diagram 3] FIG. 1 is a diagram for explaining an overview of live video generation. [Figure 4] FIG. 13 is a diagram for explaining analysis of a play situation. [Diagram 5] 11A and 11B are diagrams for explaining the operation control of a commentator character based on analysis results. [Figure 6] 13 is a diagram for explaining how to handle a case in which multiple action control patterns of commentary characters are assigned to one play situation. [Figure 7] FIG. 11 is a diagram for explaining user settings of an operation control pattern. [Figure 8] 11A and 11B are diagrams for explaining control of changing the playback speed of a gameplay video when a play-by-play video is created by adding commentary to an existing gameplay video. [Figure 9] FIG. 2 is a functional block diagram showing an example of the functional configuration of the server system according to the first embodiment. [Figure 10] FIG. 4 is a diagram showing an example of programs and data stored in a server storage unit in the first embodiment. [Figure 11] FIG. 4 is a diagram showing an example of a data configuration of motion control pattern definition data. [Figure 12] 5 is a diagram showing an example of the data structure of playback speed change pattern definition data; [Figure 13] FIG. 4 is a diagram showing an example of the data configuration of game management data. [Figure 14] FIG. 4 is a diagram showing an example of the data configuration of live management data. [Figure 15] FIG. 4 is a diagram showing an example of a data configuration of user management data. [Figure 16] FIG. 13 is a diagram showing an example of the data configuration of play video registration data. [Figure 17] FIG. 4 is a diagram showing an example of the data structure of live video registration data. [Figure 18] FIG. 4 is a functional block diagram showing an example of a functional configuration of a user terminal. [Figure 19] 5 is a flowchart for explaining a process flow in the server system of the first embodiment. [Figure 20] Flowchart continued from Figure 19. [Figure 21] FIG. 11 is a functional block diagram showing an example of the functional configuration of a user terminal according to the second embodiment. [Figure 22] FIG. 11 is a diagram showing an example of programs and data stored in a terminal storage unit according to the second embodiment. [Diagram 23] FIG. 11 is a functional block diagram showing an example of the functional configuration of a server system according to a modified example of the first embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0043] An example of an embodiment of the present invention will be described below, but it goes without saying that the form to which the present invention can be applied is not limited to the following embodiment.

[0044] [First embodiment] FIG. 1 is a diagram showing an example of the configuration of a live moving image generating and distributing system according to the present embodiment. The live video generation and distribution system 1000 is a computer system that provides a service of generating live video and distributing live video, and includes a server system 1100 and a plurality of user terminals 1500 (1500a, 1500b, ...) that are connected to each other via a network 9 so that data communication is possible.

[0045] The network 9 refers to a communication path that allows data communication. In other words, the network 9 includes a dedicated line (dedicated cable) for direct connection, a LAN (Local Area Network) using Ethernet (registered trademark), etc., as well as a telephone communication network, a cable network, the Internet, and other communication networks, and the communication method may be either wired or wireless.

[0046] The server system 1100 is a computer system having a game server 1101, a live video generation server 1102, and a distribution server 1103.

[0047] The game server 1101 is a server system and a computer system that manages the progress of a game to execute an online game on a user terminal 1500 (1500a) used by a user 2 (2a) (hereinafter referred to as a "player") who is a registered user of the live video generation and distribution system 1000 and a player of the game. The game server 1101 has, for example, a keyboard 1106, a touch panel 1108, and a storage 1140, and a control board 1150 is mounted on the main unit.

[0048] The control board 1150 is equipped with various microprocessors such as a CPU (Central Processing Unit) 1151, a GPU (Graphics Processing Unit), and a DSP (Digital Signal Processor), various IC memories 1152 such as a VRAM, RAM, and ROM, and a communication device 1153. Note that a part or the whole of the control board 1150 may be realized by an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or an SoC (System on a Chip).

[0049] The game server 1101 realizes a user management function related to user registration and the like, and a function of providing data necessary for a registered user, user 2 (2a), to play a game on a user terminal 1500 (1500a), by the control board 1150 performing calculations based on a predetermined program and data. Note that, although the present embodiment will be described assuming that there is one user 2 (2a), there may be multiple users.

[0050] The live video generation server 1102 can be realized by the same hardware as the game server 1101, and is a computer system that handles processes related to the generation of live video. In this embodiment, data of play video is acquired from the game server 1101, and a live video is generated by synthesizing a video of a commentary character set by a user who is a registered user of the live video generation and distribution system 1000 and who will be a commentator (hereinafter referred to as a "commentary user"), who performs commentary, with the play video. In this embodiment, an example is given of live-format live video distribution in which a user 2 (2a) also serves as a commentator while playing a game, so that in this embodiment, the user 2 (2a) is also a commentary user.

[0051] The distribution server 1103 can be realized by the same hardware as the game server 1101, and is responsible for accepting the provision (so-called posting) of video data (in this embodiment, gameplay videos and live video), managing the video data, and distributing the video data. In this embodiment, the video data of the live video generated by the live video generation server 1102 is sequentially acquired, and distributed to the user terminals 1500 (1500c, 1500d, ...) of one or more viewers 3 (3a, 3b, ...).

[0052] In addition, although each is depicted as a separate server device in Fig. 1, a single server device may be configured to realize the three server functions. In that case, the server system 1100 may be configured to include multiple blade servers each sharing each function, and connected to each other via an internal bus so that data communication is possible. Furthermore, the installation location of the hardware constituting the server system 1100 does not matter. A configuration in which multiple independent servers installed in separate locations communicate data via the network 9 to function as the server system 1100 as a whole may also be used.

[0053] The user terminal 1500 (1500a, 1500c, ...) is a computer system that is used individually by the user 2 (2a, 2c, ...) to play games, generate live videos, and watch gameplay videos and live videos, and is an electronic device (electronic device) that can access the server system 1100 via the network 9. Functionally speaking, the user terminal 1500 is a game device, a live video editing device, and a video viewing device. The user terminal 1500 in this embodiment is a device called a smartphone, but may also be a portable game device, a tablet computer, a personal computer, or the like.

[0054] FIG. 2 is a front view showing an example of the configuration of a user terminal 1500 in this embodiment. The user terminal 1500 includes a directional input key 1502, a button switch 1504, a touch panel 1506 that functions as an image display device and a contact position input device, a speaker 1510, an internal battery 1509, a microphone 1512, an image sensor unit 1520, a control board 1550, and a memory card reader 1542 that can read and write data from a memory card 1540 that is a computer-readable storage medium. In addition, a power button, a volume control button, and the like (not shown) are provided. In addition, an IC card reader that can read and write data contactlessly from an IC card type credit card or prepaid card that can be used to pay for playing games, etc. may be provided.

[0055] The control board 1550 includes: 1) Various microprocessors such as CPU1551, GPU, DSP, etc. 2) Various IC memories such as VRAM, RAM, ROM, etc. 1552, 3) a wireless communication module 1553 for wirelessly communicating with a mobile phone base station or a wireless LAN base station connected to the network 9; 4) Equipped with an interface circuit 1557, etc.

[0056] The interface circuit 1557 includes a driver circuit for the touch panel 1506, a circuit for receiving signals from the directional input keys 1502 and the button switches 1504, an output amplifier circuit for outputting audio signals to the speaker 1510, an input signal generation circuit for generating an audio signal collected by the microphone 1512, a circuit for inputting image data of an image captured by the image sensor unit 1520, and a signal input / output circuit for the memory card reader 1542.

[0057] These elements mounted on the control board 1550 are electrically connected via a bus circuit or the like, and are connected so as to be able to read and write data and send and receive signals. Note that a part or all of the control board 1550 may be configured with an ASIC, FPGA, or SoC. The control board 1550 stores in an IC memory 1552 programs and various data for realizing the functions of the user terminal of the game of this embodiment.

[0058] In this embodiment, the user terminal 1500 is configured to download programs and various setting data from the server system 1100, but may be configured to read them from a storage medium such as a memory card 1540 obtained separately.

[0059] FIG. 3 is a diagram for explaining an overview of generation of a live video in this embodiment. In this embodiment, the user 2 (2a) serves both as a game player and a commentary user using the commentary character 7. The user 2 (2a) then performs the task of live-streaming a commentary video 12 that combines a gameplay video 10 with a video of the commentary character 7 commenting on the gameplay (video of the commentary character 7).

[0060] On the user terminal 1500 (1500a) of the user 2 (2a), the game is executed by communicating with the game server 1101, and the game screen is displayed. The continuous live display of this game screen becomes the gameplay video 10 in this embodiment. Of course, if the distribution format is not the live format, the game screen can be separately saved in a video format and used as the gameplay video 10.

[0061] The genre of the game is not important. The example of FIG. 3 shows an action game in which a player character 4 operated by a user 2 (2a) as a player fights an enemy character 6 operated by another user or a computer. In the game of this embodiment, for example, when an enemy is encountered on the battlefield, a battle with a time limit starts, and the side that is defeated within the time limit or the side that has suffered the most damage when the time limit has elapsed becomes the loser of that round. Then, the side that first wins a predetermined number of times becomes the winner of the battle.

[0062] In this embodiment, the game server 1101 controls the progress of the game and: 1) Recording the history of operation inputs received from the user terminal 1500 (1500a); 2) Recording the history of various parameter values ​​for controlling the progress of the game (hereinafter referred to as progress control parameter values); 3) Recording the video data of 10 gameplay videos, 4) recording voice data of various user voices uttered by the user 2 (2a) during game play collected by the microphone 1512 of the user terminal 1500 (1500a); and Do the following.

[0063] The live commentary moving image generating server 1102 stores therein registration information of the user 2 (2a) as a live commentary user and setting information of the live commentary character 7 prepared in advance in association with each other.

[0064] When a specific live video generation operation is input on the user terminal 1500 (1500a), the live video generation server 1102 starts acquiring various data, such as gameplay video 10 (a live game screen in this embodiment), operation input information history, progress control parameter value history, and user voice, from the game server 1101. Hereinafter, these will be collectively referred to as "play information."

[0065] Then, the live video generation server 1102 analyzes the game play situation based on the play information. The "play situation" here refers to the progress of the game, and corresponds to the content of the game play that is the subject of live commentary. For example, "The player is in the lead," "The player's character is about to be defeated with one more hit," "A rare item was obtained," "It's a close game," "Less than one minute left," etc.

[0066] Then, based on the analysis results of the play situation, the live video generation server 1102 searches for a movement control pattern that defines a movement corresponding to the analysis results at each moment from among the movement control patterns of the live commentary character 7 prepared in advance, and controls the movement of the live commentary character 7 using the searched movement control pattern.

[0067] The state of the commentary character 7 in action (video of the commentary character 7) is synthesized with the gameplay video 10 to form the screen of the commentary video 12. In addition, the game sound included in the gameplay video 10 is synthesized with the user sound that becomes the commentary sound, and the commentary video 12 is generated by adding sound. Data of the live video 12 is sequentially transmitted from the live video generation server 1102 to the distribution server 1103, and the distribution server 1103 distributes the live video 12 in a live format.

[0068] In this embodiment, the live commentary video 12 is distributed in a live format, so that control of the game play progress, storage of the gameplay video 10, control of the movement of the commentary character 7, generation of the live commentary video 12, and control of distribution of the live commentary video 12 are performed simultaneously in parallel.

[0069] Previously, in order to distribute a live play video, it was necessary to perform the tasks of recording the play video, editing the video, adding live commentary audio, and uploading it to a distribution site one by one. In addition, in order to have a live commentary character appear in the live play video, the live commentator had to set the actions of the live commentary character as if the character were a live announcer or a sports program personality, create the video, and then composite it.

[0070] However, in this embodiment, the commentary character 7 is automatically controlled, so the user 2 (2a), who is both a player and a commentary user, only needs to operate the game while making a commentary or commentary sound. It is possible to easily distribute a live commentary video 12 as a VTuber without having to go through a number of steps as in the past.

[0071] In addition, user 2 (2a), who is both a player and a commentary user, can operate the game while emitting commentary or explanation audio, making it easy to deliver commentary video in a live format, especially for game genres in which playing operations cannot be stopped even for a moment (e.g., action games, shooting games, flight simulators, etc.).

[0072] FIG. 4 is a diagram for explaining the analysis of the play situation in this embodiment. In this embodiment, the play situation is analyzed using four methods. As a first technique, the live video generation server 1102 performs image analysis of the gameplay video 10. For example, the image analysis dictionary data 512 defines a dictionary image to be compared with a capture image of the gameplay video 10 and a play situation to be recognized when the dictionary image matches. The live video generation server 1102 executes a process of determining image matching between the capture image and the dictionary image, and regards the play situation indicated by the image analysis dictionary data 512 of the matching dictionary image as the analysis result of the technique.

[0073] Of course, the image analysis method is not limited to this and can be appropriately selected. For example, to recognize the state of the player character 4 by image analysis, image analysis dictionary data 512 for analyzing the hit point gauge of the player character 4 is prepared. Then, when it is recognized that the hit points are lower than a reference value, the play situation of "player character is in trouble" may be determined as the analysis result of the method. Also, for example, an image of the performance display when the enemy character 6 is defeated may be searched from the capture image, and if the corresponding image is included, the play situation of "defeat the enemy" may be determined as the analysis result of the method.

[0074] As a second technique, the live video generation server 1102 compares the progress control parameter value with the dictionary data 514 for progress control parameter values, and determines as the analysis result of the technique a play situation that is pre-associated with the matching dictionary data 514 for progress control parameter values. For example, when a recognition criterion is satisfied that indicates that the value by which the number of wins of the player exceeds the number of wins of the opponent has reached a predetermined disadvantage criterion value, the play situation of "the player is at a disadvantage" is determined as the analysis result of the technique.

[0075] As a third method, the operation input information history of the user 2 (2a) playing the game is compared with the operation input information dictionary data 516, and the play situation that is preliminarily associated with the matching dictionary data is taken as the analysis result in this method. For example, if the operation input information regarding the enemy character 6 is "execution of a special move," the play situation associated therewith, such as "enemy executes a special move" or "player is in danger," may be taken as the analysis result in this method.

[0076] As a fourth technique, the live video generation server 1102 compares the user's voice with the voice recognition dictionary data 518, and determines the playing situation that is previously associated with the matching dictionary data as the analysis result in the technique. For example, when keywords such as "bad," "bad," and "failed" are recognized by voice, the playing situation that is associated with them, such as "the player made a mistake" and "is the player in trouble!?", may be determined as the analysis result in the technique.

[0077] The fourth technique may be applied only when the play situation satisfies a predetermined voice reception situation condition. The "voice reception situation condition" referred to here is, for example, when an event occurs. The "event" referred to here may be acquisition of a rare item, defeat of the player character 4, defeat of the enemy character 6, the volume of the user's voice exceeding a reference value, or the like, and may be set appropriately. Of course, there is no need to limit the application scene of the fourth technique based on the voice reception situation condition.

[0078] FIG. 5 is a diagram for explaining the action control of the commentator character based on the analysis result. A plurality of pieces of movement control pattern definition data 520 for each type of movement are stored in advance in the live commentary video generation server 1102. Each piece of movement control pattern definition data 520 includes a type of movement 521, application requirements 522 indicating the conditions under which the movement of the data definition data should be applied to the movement control of the live commentary character, motion data 523 for realizing the movement, effect data 524 for realizing a performance display executed together with the movement, and appearance change data 525 for realizing an appearance change applied together with the movement. Of course, data other than these may also be included as appropriate.

[0079] The live commentary animation generation server 1102 searches for the movement control pattern definition data 520 having the application requirement 522 that matches the analysis result. Then, the live commentary character 7 is controlled in movement according to the motion data 523 set in the searched movement control pattern definition data 520, a performance display is performed according to the effect data 524, and the appearance of the live commentary character 7 is changed according to the appearance change data 525.

[0080] 5 shows that the analysis result has revealed that the player character 4 is in a situation where "one more hit is needed to defeat it." The motion control pattern definition data 520 has definition data in which application requirements 522 are set in correspondence with such a situation where "one more hit is needed to defeat it." In accordance with the motion data 523, the commentator character 7 makes a motion of holding his head in frustration, and a sweat object comically showing the impatience is added and displayed in accordance with effect data 524, and the appearance is changed to white eyes and an open mouth in shock in accordance with appearance change data 525.

[0081] By setting in detail in the action control pattern definition data 520 the play situations predicted to occur under the game rules of the target game title and defining appropriate actions for the commentary character 7, the commentary character 7 will automatically take appropriate actions to comment on the play situation unfolding in the gameplay video 10. The user 2 (2a), who is both the player and the commentary user, controls the actions of the commentary character 7 by performing game play operation inputs with his / her own hands and providing commentary and explanations with his / her own voice, so that a high-quality commentary video 12 can be easily created without the user 2 (2a) having to perform operation inputs just for controlling the actions of the commentary character 7 every time.

[0082] FIG. 6 is a diagram for explaining how to handle a case in which a plurality of movement control patterns of the commentator character 7 are assigned to one play situation. Even in the same play situation, there are cases where a plurality of different actions are performed. For example, when the commentary character 7 performs a surprised action in response to a certain play situation, the surprised action may be a genuine surprised action or a formal surprised action for other purposes such as to elicit laughter. Of course, there may be other different surprised actions. Depending on the surprised action performed at that time, the character's characterization changes, and the overall flow of the presentation of the commentary video 12 is determined. For the user 2 (2a) who is the commentary user, this is an important choice.

[0083] In this embodiment, when multiple definition data suitable for the current play situation are found, the live commentary generating server 1102 presents them to the user 2 (2a) as selection candidates and accepts a selection operation. Then, the live commentary character 7's movement is controlled according to the selected definition data.

[0084] 6, two selection candidates 30 (30a, 30b) are searched for as candidates for the action, and the live commentary video generation server 1102 displays, on the user terminal 1500 (1500a) of the live commentary user, an action monitor image 32 showing the current action of the live commentary character 7 and selection operation icons 33 (33a, 33b) corresponding to the selection candidates 30 (30a, 30b), as overlays on the game screen W6 (or a live commentary operation screen not shown). Then, the action control pattern corresponding to any of the selection operation icons 33 touched by the user 2 (2a) is applied as the action of the live commentary character 7.

[0085] Thus, in this embodiment, even if multiple action control patterns are assigned to one play situation, the user 2 (2a) can easily make important choices regarding the commentary character 7 and commentary video 12 with just a few operations.

[0086] In addition, in a configuration in which the user terminal 1500 is equipped with a three-axis acceleration sensor, the selection may be input by shaking or tilting the user terminal 1500, instead of selecting by touching the selection operation icon 33. In this case, fingertip operations, which are important for game play, do not need to be used for selecting a motion control pattern, which is preferable because it improves usability.

[0087] FIG. 7 is a diagram for explaining user settings of the operation control pattern. The movement control patterns of the commentator character 7 are basically prepared by the manufacturer or administrator of the server system 1100, but can also be created and registered by the user himself.

[0088] When a user setting operation of a predetermined action control pattern is detected in the user terminal 1500, the live video generation server 1102 displays a action control pattern setting screen W7 on the user terminal 1500. In the setting screen, an operation input for calling up a function for setting a new action control pattern or editing an existing pattern is possible.

[0089] In the new setting, a plurality of setting method selection icons 40 (40a, 40b, . . . ) are displayed. In this embodiment, user 2 can choose to load a motion file that he or she has created separately, activate the motion capture function to create the basis for motion data, or create a new motion by combining pre-prepared motion elements.

[0090] 7 shows an example in which a motion capture function is activated to create motion data material. User 2 (2a) activates the motion capture function on user terminal 1500 (1500a) and fixes it in front of a green screen 44 or the like on a tripod 42. User terminal 1500 is oriented so that the camera of image sensor unit 1520 faces green screen 44.

[0091] The user 2 (2a) stands in front of the green screen 44, performs the action that he or she wishes to set toward the user terminal 1500 (1500a), and has the user terminal 1500 (1500a) take a picture of the action.

[0092] When the user terminal 1500 (1500a) transmits the captured video to the live video generation server 1102, the live video generation server 1102 recognizes the joint structure of the human body from the captured video, and based on the recognized movement of the human body joints, creates motion data 46 for enabling the live character 7 to reproduce the same movement, and transmits this to the user terminal 1500 (1500a). Then, the user terminal 1500 (1500a) displays a motion editing screen W8 that accepts editing operations for the received motion data 46.

[0093] The user 2 (2a) cuts out unnecessary movement parts of the motion through an editing operation on the motion editing screen W8 and sets an appropriate data name. When the user terminal 1500 detects a predetermined upload operation, it transmits the edited motion data 46 together with the user account to the live animation generation server 1102. Then, the live animation generation server 1102 registers and stores the uploaded movement control pattern definition data 520 in association with the received user account.

[0094] By using such a user setting service of movement control patterns, the user 2 (2a) can impart originality to the commentator character 7 in terms of movements.

[0095] The process of generating motion data from a captured image may be configured to be performed by the user terminal 1500. Furthermore, the motion data 46 may be generated by other methods (for example, by using 3DCG generation software).

[0096] Up to this point, the description has been given on the premise of live distribution of the play-by-play video 12 in this embodiment, but in the play-by-play video generation and distribution system 1000, the user 2 (2a) can input commentary or commentary audio to an already recorded and registered gameplay video 10 (hereinafter, referred to as an existing gameplay video 10), while automatically controlling the commentary character 7 to create the play-by-play video 12.

[0097] FIG. 8 is a diagram for explaining a technique for changing and controlling the playback speed of a gameplay video 10 when creating a play-by-play video 12 by adding commentary to an existing gameplay video 10. In FIG. When creating a commentary video 12 using an existing gameplay video 10, if a keyword that changes a specified playback speed is detected in voice recognition of a user's voice, the commentary video generation server 1102 changes the playback speed of the gameplay video 10 that is the subject of commentary.

[0098] Specifically, as shown in Fig. 8(1), in a play situation where a production technique of making the scene appear more slowly to enhance the impression is effective, if a keyword predicted to be uttered by the user (for example, examples in Fig. 8(1) are "Get a rare (item)!" or "Defeat the boss!") is detected by user voice recognition, the playback speed of the existing gameplay video 10 to be commented on is changed to slow playback for a certain period of time or until a specified release condition is met. However, the movements of the commentary character 7 are maintained at standard speed.

[0099] Also, as shown in Fig. 8(2), when a voice keyword indicating that the playing situation is one in which a production technique of fast-forwarding scenes that are necessary for explaining the passage of time and the situation but tend to be boring for the viewer of the video to maintain the tempo of the progression is effective (for example, the examples in Fig. 8(2) are "Let's skip through it quickly" and "Let's let it flow") is detected by user voice recognition, the playback speed of the existing gameplay video 10 to be commented on is changed to fast-forward playback (meaning playback at a speed faster than 1x) for a certain period of time or until a specified release condition is met. However, the movements of the commentary character 7 are maintained at standard speed.

[0100] In this way, according to this embodiment, video expression techniques that make full use of slow playback and fast-forward playback can be automatically applied to the live commentary video 12, making it possible to generate a live commentary video with rich dramatic effect.

[0101] The change in the playback speed may include reverse playback, repeated playback using the reverse playback, etc., as appropriate. In addition, the speed of the movement of the commentary character 7 may be changed, or the playback speed of the video of the commentary character 7 may be changed, in accordance with the change in the playback speed of the play-by-play video 10. The change in the playback speed may also be applied to the distribution of the play-by-play video 12 in a live format, if a replay video can be generated on the spot.

[0102] FIG. 9 is a functional block diagram showing an example of the functional configuration of the server system 1100. The server system 1100 includes an operation input unit 100s, a server processing unit 200s, a sound output unit 390s, an image display unit 392s, a communication unit 394s, and a server storage unit 500s.

[0103] The operation input unit 100s is a means for inputting various operations for managing the server, and corresponds to the keyboard 1106 in FIG.

[0104] The server processing unit 200s is realized by electronic components such as processors serving as arithmetic circuits, such as a CPU, GPU, ASIC, and FPGA, as well as IC memories, and controls input and output of data between each functional unit including the operation input unit 100s and the server storage unit 500s.The server processing unit 200s performs various types of arithmetic processing based on predetermined programs and data, operation input signals from the operation input unit 100s, data received from the user terminal 1500, etc., and performs integrated control of the operation of the server system 1100.

[0105] The server processing unit 200s includes a user management unit 210, a game management unit 212, a motion control pattern setting unit 220, a voice acquisition control unit 222, an analysis unit 224, a layout setting unit 226, a live video generation control unit 228, a live character control unit 230, a delivery control unit 240, a timing unit 280s, a sound generation unit 290s, an image generation unit 292s, and a communication control unit 294s. Of course, other functional units may also be included as appropriate.

[0106] In this embodiment, the user management unit 210 and the game management unit 212 correspond to the functions of the game server 1101. Also, the action control pattern setting unit 220, the voice acquisition control unit 222, the analysis unit 224, the layout setting unit 226, the live video production control unit 228, and the live video character control unit 230 correspond to the functions of the live video production server 1102. Also, the distribution control unit 240 corresponds to the function of the distribution server 1103. Of course, depending on the hardware configuration of the server system 1100, these may be realized by one server device, and the division of roles between the functions and the hardware is not limited to this.

[0107] The user management unit 210 performs processing related to user registration procedures and stores and manages information on each user associated with a user account.

[0108] The game management unit 212 performs various processes related to game execution, such as controlling the progress of the game.

[0109] The movement control pattern setting unit 220 sets a movement control pattern of the commentator character 7 based on an operational input by the user.

[0110] The voice acquisition control unit 222 performs control to acquire the voice of the user while playing a game.

[0111] The analysis unit 224 analyzes the play situation of the game play. Specifically, the analysis unit 224 performs image analysis on the gameplay video 10 and analyzes the play situation based on the image analysis results. The analysis unit 224 can also analyze operation input information of the game play and analyze the play situation based on the analysis results.

[0112] The layout setting unit 226 sets the layout of the gameplay video 10 and the commentary character 7 in the commentary video 12. The "layout" here refers to the setting of how to display both the gameplay video 10 and the commentary character 7, such as the relative positions and relative sizes of the gameplay video 10 and the commentary character 7 within the screen display of the commentary video 12. Of course, the elements displayed within the game screen of the commentary video 12 can include display elements other than the gameplay video 10 and the commentary character 7, and in that case, the settings for those display elements can also be included in the layout settings.

[0113] The commentary video generation control unit 228 controls the generation of the commentary video 12, which is a combination of the video of the commentary character 7 commenting on the gameplay video 10 of the game play and the gameplay video 10. The commentary video generation control unit 228 functions as a commentary video generating means together with an image generating unit 292s described later. Specifically, the commentary video generation control unit 228 generates the commentary video 12 by using the video of the game play being performed by the user as the gameplay video 10 to be commented on, and using the user's voice acquired by the voice acquisition control unit 222 as the commentary voice.

[0114] Furthermore, the commentary video generation control unit 228 can generate a commentary video by changing the playback speed of the video of the commentary character 7 and / or the gameplay video 10 based on the analysis result of the analysis unit 224.

[0115] The commentary character control unit 230 controls the commentary character 7 based on the analysis result of the analysis unit 224. Specifically, it controls the movement of the commentary character 7, controls changes to the appearance, and controls the display of the performance display that is displayed in conjunction with the movement control of the commentary character 7. The commentary character control unit 230 has a voice response control unit 232 and a movement control pattern selection control unit 234.

[0116] When the play situation satisfies a voice reception situation condition indicating that the voice response control is acceptable, the voice response control unit 232 performs voice response control to control the movement of the commentary character 7 based on the voice acquired by the voice acquisition control unit 222.

[0117] The movement control pattern selection control unit 234 presents selection candidates for the movement control pattern of the commentator character 7 to the user, and controls the movement of the commentator character 7 based on the movement control pattern of the selected selection candidate.

[0118] The distribution control unit 240 controls the distribution of the live video 12.

[0119] The timekeeping unit 280s uses a system clock to keep track of the current date and time, time limit, and the like.

[0120] The sound generation unit 290s is realized by executing an IC or software that generates or decodes audio data, and generates or decodes audio data such as operation sounds and BGM related to the system management and video distribution of the server system 1100. Then, the audio signal related to the system management is output to the sound output unit 390s.

[0121] 1, the sound output unit 390s corresponds to a speaker (not shown) provided in the main device or the touch panel 1108.

[0122] The image generating unit 292s generates images, synthesizes images, and outputs image signals for displaying them on the image display unit 392s. In this embodiment, the image generating unit 292s is responsible for part of the function of generating images related to system management of the server system 1100, game images (or data for displaying game images on the user terminal 1500), live video 12, and the like.

[0123] The communication control unit 294s executes data processing related to data communication, and realizes data exchange with an external device via the communication unit 394s.

[0124] The communication unit 394s realizes communication by connecting to the network 9. For example, it is realized by a wireless communication device, a modem, a TA (terminal adapter), a jack for a wired communication cable, a control circuit, etc. In the example of FIG. 1, it corresponds to the communication device 1153.

[0125] The server storage unit 500s stores programs and various data for implementing various functions for the server processing unit 200s to comprehensively control the server system 1100. It is also used as a working area for the server processing unit 200s, and temporarily stores the results of calculations executed by the server processing unit 200s according to various programs. This function is implemented by, for example, IC memory such as RAM or ROM, magnetic disks such as hard disks, optical disks such as CD-ROMs or DVDs, online storage, etc. In the example of FIG. 1, this corresponds to storage media such as IC memory 1152 and hard disks mounted on the main unit, and storage 1140.

[0126] 10 is a diagram showing examples of programs and data stored in the server storage unit 500s in this embodiment. The server storage unit 500s in this embodiment stores a server program 501, a distribution client program 503, game initial setting data 510, image analysis dictionary data 512, progress control parameter value dictionary data 514, operation input information dictionary data 516, voice recognition dictionary data 518, motion control pattern definition data 520, and playback speed change pattern definition data 540.

[0127] The server storage unit 500s also stores, as sequentially generated and managed data, game management data 550, commentary management data 570, user management data 600, play video registration data 700, commentary video registration data 730, and current date and time 800. The server storage unit 500s can also store other programs and data (such as timers, counters, and various flags) as appropriate.

[0128] The server program 501 is a program that is read and executed by the server processing unit 200s to realize the functions of the user management unit 210 to the distribution control unit 240. In this embodiment, the server system 1100 is composed of three hardware elements, namely, a game server 1101, a live video generation server 1102, and a distribution server 1103, and therefore stores server programs for realizing the functions assumed by each of these elements.

[0129] The delivery client program 503 is an original client program provided to the user terminal 1500. In this embodiment, one delivery client program 503 includes a delivery game client program 503a for executing an online game on the user terminal 1500, and a delivery commentary client program 503b for causing the user terminal 1500 to function as a terminal for generating commentary actions. These may be delivered to the user terminal 1500 separately.

[0130] The game initial setting data 510 includes various initial setting data necessary for executing an online game. In the example of Fig. 10, only one game initial setting data 510 is illustrated, but if the server system 1100 provides multiple game titles, the game initial setting data 510 is stored for each game title.

[0131] Image analysis dictionary data 512, progress control parameter value dictionary data 514, operation input information dictionary data 516, and voice recognition dictionary data 518 are dictionary data that the analysis unit 224 refers to in order to analyze the play situation (see FIG. 4).

[0132] The motion control pattern definition data 520 is prepared for each motion control pattern and stores various data that defines the pattern. As shown in Fig. 11, one motion control pattern definition data 520 includes a unique motion type 521, application requirements 522, motion data 523, effect data 524, appearance change data 525, a user account 526, and permitted use range 527 (see Fig. 5).

[0133] The application requirement 522 is described by combining one condition or multiple conditions with AND or OR. Examples of conditions include: 1) A character type condition 522a which is a condition regarding the type of the commentary character 7; 2) Game title condition 522b, which is a condition regarding the game title being played in the gameplay video 10; 3) Image analysis result conditions 522c, which are conditions on analysis results based on image analysis; 4) progress control conditions 522d, which are conditions on the analysis results based on the progress control parameter values; 5) Operation input conditions 522e, which are conditions regarding the analysis results based on the operation input information; 6) Speech recognition result conditions 522f, which are conditions on analysis results based on speech recognition; Of course, other conditions may also be included as appropriate.

[0134] The setter account 526 indicates the person who set the definition data. Basically, the value indicates the manufacturer or administrator of the server system 1100, but for settings made by a user, the user account of that person is set.

[0135] The permitted use range 527 indicates the range within which the use of the definition data is permitted. For example, if it is set to unlimited, anyone can use the definition data at any time. If a specific user account is set, the users who can use the data are limited. The setting of the permitted use range 527 is included in one of the setting items in the user settings of the motion control pattern.

[0136] 10, the playback speed change pattern definition data 540 is prepared for each pattern of changing the playback speed of the gameplay video 10 displayed in the live-action video 12, and stores various related data. As shown in Fig. 12, one playback speed change pattern definition data 540 includes a unique playback speed change pattern ID 541, an application requirement 542, an application playback speed 543, and an application end condition 544. Of course, data other than these may also be included as appropriate.

[0137] The application requirement 542 indicates a condition that must be satisfied in order for the definition data to be applied. The application requirement 542 is described by combining a single condition or multiple conditions with AND or OR, similar to the application requirement 522 of the motion control pattern definition data 520.

[0138] The application end condition 544 specifies a condition for ending the change in the playback speed and returning it to the standard speed. For example, the application of the definition data and the elapsed time from the start of the change in the playback speed, the detection of a predetermined operation input, the detection of a predetermined keyword by voice recognition, etc. can be set.

[0139] Returning to Fig. 10, game management data 550 is prepared for each game played on the user terminal 1500, and stores various data for managing the execution of the game. One piece of game management data 550 includes, for example, as shown in Fig. 13, a unique game management ID 551, a player account 552, terminal access information 553, a latest progress control parameter value 554, and play information 560. Of course, data other than these may also be included as appropriate.

[0140] The play information 560 includes play video data 561, progress control parameter value history data 562 which stores the latest progress control parameter values ​​554 at each time in chronological order, operation input history data 563 which stores operation input information in chronological order, and user voice recording data 564.

[0141] Returning to Fig. 10, commentary management data 570 is prepared for each generation of commentary video 12, and stores various data related to the generation. One commentary management data 570 includes, for example, as shown in Fig. 14, commentary target ID 571 indicating the target of commentary, applied layout type 572 indicating the layout of the current screen of commentary video 12, commentary character control data 573, commentary video data 574, and analysis result history data 580. Of course, data other than these may also be included as appropriate.

[0142] As the commentary target ID 571, the game management ID 551 (see FIG. 13) of the game play to be commented on or the gameplay video ID 701 (see FIG. 16) uniquely set to the existing gameplay video 10 is set.

[0143] The applied layout type 572 is set to a predetermined initial setting value when the live broadcast starts, but is changed in response to input of a predetermined layout change operation.

[0144] Commentary character control data 573 stores various data necessary for controlling the movement and display of commentary character 7. For example, control data for motion data, control data for performance display, control data for appearance texture and color table, position information, posture information, various flags and counters, etc. are stored.

[0145] Analysis result history data 580 stores analysis results of the play situation in chronological order. In this embodiment, four methods are used to analyze the play situation, and therefore, the corresponding data includes image analysis result history data 581, analysis result history data 582 based on progress control parameter values, analysis result history data 583 based on operation input information, and voice recognition result history data 584.

[0146] Returning to Fig. 10, user management data 600 is prepared for each registered user of commentary moving image generation and distribution system 1000, and stores various data related to the user. One user management data 600 stores, for example, a user account 601, terminal access information 602, commentary character setting data 603, character type 604 indicating the type of commentary character 7, and self-set action type list 605, as shown in Fig. 15. Of course, data other than these may also be stored as appropriate.

[0147] The commentary character setting data 603 is created for each commentary character 7 used by the user, and stores data such as the character model, standard texture data, a list of added exterior accessory items, a color table, and the like.

[0148] The self-set operation type list 605 is a list of operation types 521 (see FIG. 11) of the operation control pattern definition data 520 set by the user using the user setting function of the operation control pattern.

[0149] Returning to Fig. 10, gameplay video registration data 700 is prepared for each existing gameplay video 10. One gameplay video registration data 700 stores, for example, as shown in Fig. 16, a unique gameplay video ID 701, a providing user account 702 indicating the poster / provider of the gameplay video, and play information 710 of the game play when the gameplay video was recorded. Of course, data other than these may also be stored as appropriate.

[0150] The play information 710 is a copy of the play information 560 (see FIG. 13) at the time of registration, and includes play video data 711, progress control parameter value history data 712, operation input history data 713, and user voice recording data 714.

[0151] Returning to Fig. 10, the live commentary registration data 730 is, so to speak, live commentary registration data, which is created for each live commentary video 12 generated. The registered live commentary video 12 becomes a target for distribution at any time in response to a request from the viewer 3. One live commentary registration data 730 stores, for example, a unique live commentary ID 731, a providing user account 732, and live commentary video data 733, as shown in Fig. 17. Of course, data other than these may also be stored as appropriate.

[0152] 18 is a functional block diagram showing an example of a functional configuration of a user terminal 1500 in this embodiment. The user terminal 1500 includes an operation input unit 100, a voice input unit 102, a device processing unit 200, a sound output unit 390, an image display unit 392, a communication unit 394, and a device storage unit 500.

[0153] The operation input unit 100 outputs operation input signals corresponding to various operation inputs made by the player to the device processing unit 200. For example, this can be realized by a push switch, a joystick, a touch pad, a track ball, an acceleration sensor, a gyro, a CCD module, etc. Examples of this include the direction input key 1502, the button switch 1504, the touch panel 1506, and the image sensor unit 1520 in Fig. 2.

[0154] The voice input unit 102 collects voice uttered by the user (user voice) and environmental sounds, and outputs a voice signal to the device processing unit 200. In the example of FIG.

[0155] The device processing unit 200 is realized by, for example, a microprocessor such as a CPU or GPU, or electronic components such as an IC memory, and controls input and output of data between each functional unit including the operation input unit 100 and the device storage unit 500. Then, it executes various arithmetic processes based on predetermined programs and data, an operation input signal from the operation input unit 100, and various data received from the server system 1100, to control the operation of the user terminal 1500. This corresponds to the control board 1550 in FIG. 2. And, the device processing unit 200 in this embodiment includes a user terminal arithmetic unit 260, a timer unit 280, a sound generation unit 290, and a communication control unit 294.

[0156] The user terminal computing unit 260 includes an operation signal transmission control unit 261 and an image display control unit 262 .

[0157] The operation signal transmission control unit 261 executes a process for transmitting various data and requests to the server system 1100 in response to an operation performed on the operation input unit 100. In this embodiment, the operation signal transmission control unit 261 also controls the generation of voice data of the user's voice collected by the voice input unit 102 and the external transmission.

[0158] The image display control unit 262 performs control for displaying a game screen, etc., based on various data received from the server system 1100. In this embodiment, the server system 1100 is configured to generate a game space image (an image of the game space photographed by a virtual camera; for example, a 3DCG image, etc.), but the user terminal 1500 may also generate the game space image. In that case, the image display control unit 262 includes control of objects arranged in a virtual three-dimensional space for generating the 3DCG, for example.

[0159] In addition, in relation to the generation of the commentary video 12, in this embodiment, the server system 1100 generates a video of the commentary character 7 and synthesizes it with the gameplay video 10 to generate the commentary video 12, but if these generation and synthesis processes are performed on the user terminal 1500 side, the image display control unit 262 will be responsible for some or all of these.

[0160] Then, the image display control unit 262 outputs the image signal to the image display unit 392. The image display unit 392 displays various game images and various screens related to the generation of the live video 12 based on the image signal input from the image display control unit 262. For example, it can be realized by an image display device such as a flat panel display, a projector, or a head mounted display. In this embodiment, this corresponds to the touch panel 1506 in FIG. 2.

[0161] The sound generation unit 290 is realized by, for example, a digital signal processor (DSP), a processor such as a voice synthesis IC, an audio codec capable of playing audio files, etc., and generates sound signals of sound effects and background music related to the game, various operation sounds, and operation sounds and sound effects related to the generation of the live video 12, and outputs them to the sound output unit 390.

[0162] The sound output unit 390 is realized by a device that outputs sound such as sound effects and background music based on the sound signal input from the sound generation unit 290. This corresponds to the speaker 1510 in FIG.

[0163] The communication control unit 294 executes data processing related to data communication and realizes data exchange with an external device via the communication unit 394. The communication unit 394 realizes communication by connecting to the network 9. For example, this is realized by a wireless communication device, a modem, a TA (terminal adapter), a jack for a wired communication cable, a control circuit, etc., and corresponds to the wireless communication module 1553 in FIG.

[0164] The terminal storage unit 500 stores programs for implementing various functions for the terminal processing unit 200 to comprehensively control the user terminal 1500, various data, etc. It is also used as a working area for the terminal processing unit 200, and temporarily stores the results of calculations executed by the terminal processing unit 200 according to various programs, input data input from the operation input unit 100, etc. These functions are implemented by, for example, IC memories such as RAM and ROM, magnetic disks such as hard disks, optical disks such as CD-ROMs and DVDs, etc. The IC memory 1552 and memory card 1540 mounted on the control board 1550 in FIG. 2 correspond to this. A configuration using online storage is also possible.

[0165] The terminal storage unit 500 of this embodiment stores a client program 502. Of course, other programs and data can also be stored as appropriate.

[0166] The client program 502 is application software that the device processing unit 200 reads and executes to realize the functions of the user terminal computing unit 260. In this embodiment, the client program 502 is a copy of a distribution client program 503 (see FIG. 10) provided by the server system 1100. The client program 502 may be a dedicated client program, or may be configured by a web browser program and a plug-in that realizes interactive image display.

[0167] Next, the operation of the live video creation and distribution system 1000 of this embodiment will be described. 19 and 20 are flowcharts for explaining the flow of processing in the server system 1100. The flow of processing explained here is realized by the server processing unit 200s of the server system 1100 executing the server program 501. It is assumed that the client program 502 is executed in the user terminal 1500, the game can be played, and a function related to generation of an execution moving image is available.

[0168] As shown in FIG. 19, the server system 1100 first accepts a user setting of an action control pattern (step S8; see FIG. 7).

[0169] Next, the server system 1100 creates and initializes live commentary management data 570 (step S10; see FIG. 14). Live commentary target ID 571 is set to a predetermined value meaning live distribution, and applied layout type 572 is set to a predetermined initial value. Live commentary character control data 573, live commentary video data 574, and analysis result history data 580 are not stored at this stage.

[0170] Next, the server system 1100 starts acquiring user voice data collected by the user terminal 1500 (step S12). Specifically, the live video generation server 1102 starts generating and transmitting voice data of the user voice collected by the microphone 1512 to the user terminal 1500. The user terminal 1500 starts generating the user's voice data, and starts controlling the sequential transmission to the live video generation server 1102.

[0171] Next, the server system 1100 starts controlling the commentary character 7 (step S14), starts generating and recording the commentary video 12, and if the commentary video 12 is to be distributed in a live format, starts the distribution here (step S16).

[0172] At the stage of step S14, since no motion control based on the motion control pattern definition data 520 is performed, the commentary character 7 is controlled to take a pause motion in a standby state. Then, from step S14 onwards, the commentary character control data 573 of the commentary management data 570 is updated one by one, and from step S16 onwards, the commentary video data 574 is also updated one by one. At this point, since game play has not started, the gameplay video 10, which is a live video, is not yet displayed in the commentary video 12. In terms of the flow of the commentary video 12, this is the time period in which the user 2 (2a), who is the commentary user, greets the viewers and talks about the outline of the commentary video 12 that is about to begin.

[0173] Next, the server system 1100 starts layout change control when detecting an input of a layout change operation of the live video 12 at the user terminal 1500 (step S18). After that, the user 2 (2a) can arbitrarily change the screen layout even during the live broadcast.

[0174] Next, when the server system 1100 detects input of a predetermined game start operation at the user terminal 1500, it performs preparation control for game play (step S30). Specifically, the game server 1101 creates and initializes game management data 550 (see FIG. 13). It accepts the selection and setting of the player character 4 and the selection of the play stage, constructs a game space in a virtual three-dimensional space, and arranges objects such as the player character 4 and enemy character 6. It also initializes the latest progress control parameter value 554.

[0175] Also, at this stage, the server system 1100 sets the gameplay video 10 of the game play as the commentary target. That is, the game server 1101 provides the game management ID 551 to the commentary video generating server 1102, and the game management ID 551 is set as the commentary target ID 571 (see FIG. 14).

[0176] Then, after completing the preparation, the server system 1100 starts controlling the game progress and recording the play information 560 (step S32), and starts analyzing the play situation (step S34). Specifically, the game server 1101 starts transmitting the play information 560 to the live video generation server 1102. The live video generation server 1102 starts receiving the play information 560. Then, the live video generation server 1102 starts sequentially analyzing the play situation based on the received play information 560, and accumulates the analysis results in the analysis result history data 580 (see FIG. 14).

[0177] In addition, in the case of distribution of a non-live commentary video 12, for example, when the commentary subject is an existing gameplay video 10, step S30 accepts the selection setting of the existing gameplay video 10 to be commented on, and step S32 starts playback of the existing gameplay video 10 and reference to the gameplay information.

[0178] If the analysis result of the play situation, which is determined one by one, satisfies the application requirement 542 of any of the playback speed change pattern definition data 540 (see FIG. 12) (YES in step S40), the server system 1100 starts changing the playback speed according to the definition data (step S42). The started change in the playback speed is continued until the application end condition 544 of the definition data is satisfied.

[0179] Turning now to FIG. 20, if the analysis results of the latest play situation at that time satisfy the voice reception situation conditions (YES in step S50) and there is movement control pattern definition data 520 (see FIG. 11) for which the application requirements 522 are satisfied (YES in step S52), the server system 1100 sets the movement control pattern of the corresponding definition data as a selection candidate (step S54).

[0180] If there are multiple selection candidates (multiple in step S56), the server system 1100 displays the selection operation icons 33 for each selection candidate on the user terminal 1500 and presents them to the user (step S58; see FIG. 6). Then, the server system 1100 starts the movement control of the commentator character 7 with the movement control pattern of the movement control pattern definition data 520 of the selection candidate selected by the user (step S60).

[0181] If there is a single selection candidate (single in step S56), the server system 1100 starts controlling the movement of the commentary character 7 with the movement control pattern of the movement control pattern definition data 520 that is the selection candidate (step S62).

[0182] Then, when the action control starts, the server system 1100 starts changing the appearance associated with the action control pattern (step S64), and starts applying the effect (step S66).

[0183] When the reproduction of the action control pattern that has been applied is completed, the server system 1100 controls the commentator character 7 to take a pause action in a standby state. Also, when the reproduction of the action control pattern is completed, the appearance change is completed, the appearance is returned to the standard appearance, and the application of the effect is also completed.

[0184] The server system 1100 repeats steps S40 to S66 until a predetermined operation to end the live commentary is detected in the user terminal 1500 (NO in step S70). If an operation to end the live commentary is detected (YES in step S70), the server system 1100 registers the current live commentary video 12 again, and ends the distribution here if the live commentary video 12 is being distributed in a live format (step S74), and ends the series of processes.

[0185] As described above, according to this embodiment, a new user assistance technology for video distribution using a commentary character can be provided. That is, the play situation of the game play displayed as a gameplay video can be automatically analyzed, and the commentary character can be automatically controlled based on the analysis result. Therefore, the user does not need to prepare a video in which the commentary character is controlled by creating each and every action of the commentary character. In addition, since the need for user operations related to the action control of the commentary character can be significantly reduced, the user can easily generate a live-style commentary video using the commentary character 7 by simply inputting the voice of the commentary or explanation while playing the game.

[0186] Second Embodiment Next, a second embodiment to which the present invention is applied will be described. In the following, differences from the first embodiment will be mainly described, and the same components as those in the first embodiment will be given the same reference numerals as those in the first embodiment, and duplicated explanations will be omitted.

[0187] This embodiment differs from the first embodiment in that, among the processes executed by the server system 1100 in the first embodiment, processes related to game management and generation of live video, except for distribution, are executed by a user terminal 1500.

[0188] 21 is a functional block diagram showing an example of the functional configuration of the user terminal 1500B in this embodiment. Compared to the user terminal 1500 in the first embodiment, the user terminal 1500B has a game management unit 212B corresponding to the game management unit 212 in the first embodiment instead of the user terminal calculation unit 260, and executes various processes for executing and controlling the game in the user terminal 1500B. In addition, the user terminal 1500B has a motion control pattern setting unit 220, a voice acquisition control unit 222, an analysis unit 224, a layout setting unit 226, a live video generation control unit 228, a live character control unit 230, and a distribution control unit 240B.

[0189] The distribution control unit 240B controls distribution of the live video 12 by uploading the generated data of the live video 12 to a predetermined distribution system (in this embodiment, the distribution server 1103).

[0190] 22 is a diagram showing an example of programs and data stored in the terminal storage unit 500 in this embodiment. The terminal storage unit 500 in this embodiment stores a terminal program 504. The terminal program 504 is a program for causing the terminal processing unit 200 to realize the functions of the game management unit 212B, the action control pattern setting unit 220, the voice acquisition control unit 222, the analysis unit 224, the layout setting unit 226, the live video generation control unit 228, and the live character control unit 230. The original of the terminal program 504 may be stored in the server system 1100 as a distribution terminal program and downloaded as necessary.

[0191] The terminal storage unit 500 of this embodiment also stores game initial setting data 510, image analysis dictionary data 512, progress control parameter value dictionary data 514, operation input information dictionary data 516, voice recognition dictionary data 518, motion control pattern definition data 520, playback speed change pattern definition data 540, game management data 550, and commentary management data 570. Originals of these various data may be stored in the server system 1100 and downloaded as necessary.

[0192] The processing flow executed by the user terminal 1500B in this embodiment is basically the same as that executed by the server system 1100 in the first embodiment (see Figures 19 to 20). In accordance with the processing related to game management and generation of play-by-play video being executed by the user terminal 1500B, the execution subject of each step may be appropriately replaced with the user terminal 1500B.

[0193] When the user terminal 1500B executes a process that requires referring to or changing the user management data 600, it appropriately requests the server system 1100 to provide or verify the necessary data, or to carry out the process itself.

[0194] [Modifications] Although the embodiment to which the present invention is applied has been described above, the forms to which the present invention can be applied are not limited to the above-described forms, and constituent elements can be added, omitted, or modified as appropriate.

[0195] (Variation 1) In the first embodiment, an example is given in which processing related to game management and generation of play-by-play video is executed by server system 1100, and in the second embodiment, an example is given in which processing related to game management and generation of play-by-play video is executed by user terminal 1500B, but the entity executing processing related to game management and generation of play-by-play video is not limited to the above embodiments.

[0196] For example, it is possible to configure the server system 1100 to handle the processing related to game management, and the user terminal 1500 to handle the processing related to the generation of live video. That is, it is possible to appropriately set the allocation of which hardware will execute the game management unit 212, the action control pattern setting unit 220, the voice acquisition control unit 222, the analysis unit 224, the layout setting unit 226, the live video generation control unit 228, and the live character control unit 230.

[0197] (Variation 2) In addition, in the above embodiment, the movement control pattern to be applied to the commentary character 7 is selected based on all of the analysis results of each of the four analysis methods in analyzing the play situation, but it is also possible to configure the four analysis results to be selected and used to select the movement control pattern.

[0198] For example, the live video generation server 1102 may determine the final analysis result of the play situation at each time based on the analysis results of each of these four methods. Specifically, when an analysis result is output by any one of the methods, the analysis result may be set as the final analysis result, and when analysis results are output by multiple methods, the analysis result by the method with the highest adoption priority, which is preset for each method, may be set as the final analysis result. Of course, the final analysis result may be selected by majority vote.

[0199] (Variation 3) Moreover, an element for controlling a commentary character based on comments input by the player 2a during play can be added to the above embodiment. Specifically, for example, if the first embodiment is used as a base, a comment acquisition control unit 229 that performs control for acquiring comments from the player, the user 2 (2a), is added as shown in Fig. 23, and a comment response control unit 236 that performs comment response control for controlling the behavior of the commentary character 7 based on the comments acquired by the comment acquisition control unit 222 is added to the commentary character control unit 230.

[0200] When the operation input unit 110 includes a keyboard, the comment acquisition control unit 229 acquires a comment by inputting text from the keyboard. Alternatively, a text input unit (such as a software keyboard) may be displayed within the game screen to acquire a comment. Alternatively, the voice acquired by the voice acquisition control unit 222 may be used as the voice comment as it is, or text obtained by performing voice recognition processing may be used as the text comment.

[0201] The comment response control unit 236 may use the fourth analysis method by the analysis unit 224. That is, the comment response control may be realized by comparing the voice of the voice comment acquired by the voice acquisition control unit 222 (user voice) with the voice recognition dictionary data 518, taking the play situation previously associated with the matching dictionary data as the analysis result in the method, and controlling the movement of the commentator character 7 based on the analysis result. When the comment response control is acquired as a text comment, the acquired text may be regarded as a word obtained as a result of voice recognition, and a movement control pattern based on the voice recognition result condition 522f (see FIG. 11) may be selected to control the movement of the commentator character 7, thereby realizing the comment response control.

[0202] Furthermore, a comment acceptance status condition indicating that the play situation is one in which comment response control is acceptable may be set by reusing the voice acceptance status condition in the fourth method for analyzing the play situation, and when the comment acceptance status condition is satisfied, the behavior of the commentary character may be controlled based on the comment acquired by the comment acquisition control unit 229.

[0203] Of course, even if the second embodiment is used as a base, an element for controlling a commentary character based on comments input by the player 2a during play can be added in a similar manner. [Explanation of symbols]

[0204] 2. User 7...Commentary character 10...Play video 12...Live video 30...Candidates 100...Operation input section 102...Audio input section 200s…Server processing section 212…Game Management Department 220...Motion control pattern setting unit 222...Voice acquisition control unit 224…Analysis Department 226…Layout setting section 228...Live video generation control unit 229...Comment acquisition control unit 230…Commentary character control unit 232...Voice response control unit 234...Operation control pattern selection control unit 236...Comment response control unit 240...Distribution control unit 500s…Server storage 501...Server program 502 - Client program 510... Game initial setting data 512...Image analysis dictionary data 514...Dictionary data for progress control parameter values 516... Dictionary data for operation input information 518…Speech recognition dictionary data 520...Motion control pattern definition data 522…Applicable requirements 523...Motion data 525…Appearance change data 540...Playback speed change pattern definition data 550...Game management data 560...Play information 561...Play video data 562...Progress control parameter value history data 563...Operation input history data 564...User voice recording data 570…Live management data 571…Live commentary target ID 572…Applicable layout type 573... Live character control data 574...Live video data 580...Analysis result history data 581...Image analysis result history data 582...Progress control parameter value standard analysis result history data 583...Operation input information criteria analysis result history data 584…Speech recognition result history data 603...Commentary character setting data 1000…Live video production and distribution system 1100...Server system 1101...Game server 1102…Live video generation server 1103...Distribution server 1500: User terminal W6…Game screen W7…Motion control pattern setting screen W8: Motion editing screen

Claims

1. A computer system, a commentary video generating means for generating a commentary video by combining a video of a commentary character who provides commentary on the game play video of the game, and the game play video; An analysis means for analyzing a playing situation of the game play; commentary character control means for controlling the commentary character to automatically perform a preset action in correspondence with a play situation predicted based on an analysis result of the analysis means; A program to function as a

2. A computer system, a commentary video generating means for generating a commentary video by combining a video of a commentary character that provides commentary on a gameplay video of a game in which a commentary user and an opponent user compete against each other by operating each other's characters, and the gameplay video; an analysis means for analyzing a playing situation of the game by analyzing operation input information of the opponent user; a commentary character control means for controlling the commentary character based on an analysis result of the analysis means; A program to function as a

3. A distribution control means for controlling the distribution of the live video or controlling the distribution of the live video by uploading data of the live video to a predetermined distribution system; 3. The program according to claim 1 or 2, for causing the computer system to function as a

4. a layout setting means for setting a layout of the gameplay video and the commentary character in the commentary video; The program according to any one of claims 1 to 3, for causing the computer system to function as a

5. The commentary character control means changes and controls the appearance of the commentary character. The program according to any one of claims 1 to 4.

6. the play-by-play video generation means changes a playback speed of the play-by-play video based on the analysis result of the analysis means to generate the play-by-play video; The program according to any one of claims 1 to 5.

7. a commentary video generating means for generating a commentary video by combining a video of a commentary character who comments on a game play video of a game, and the game play video; An analysis means for analyzing a playing situation of the game play; commentary character control means for controlling the commentary character to automatically perform a preset action corresponding to a play situation predicted based on an analysis result of the analysis means; A computer system comprising:

8. a commentary video generating means for generating a commentary video by combining a video of a commentary character that provides commentary on a gameplay video of a game in which a commentary user and an opponent user compete against each other by operating each other's characters, and the gameplay video; an analysis means for analyzing a playing situation of the game by analyzing operation input information of the opponent user; a commentary character control means for controlling the commentary character based on an analysis result of the analysis means; A computer system comprising:

Images