Computer program, server device, terminal device, and method

Abstract

A computer program, a server device, a terminal device, and a method are provided that enable an avatar to be more appropriately expressed. [Solution] A computer program according to one embodiment is a computer program that is executed by one or more processors to acquire information relating to one or more of a predetermined action of an avatar performed by a broadcaster, the facial expression of the avatar, the complexion of the avatar, the voice of the avatar, or a predetermined situation occurring regarding the avatar within a virtual space, determine whether the acquired information satisfies predetermined conditions, and if it determines that the predetermined conditions are satisfied, generate information to change the facial expression or pose of the avatar related to the broadcaster that was generated based on the facial expression or pose of the broadcaster to a predetermined facial expression or pose that corresponds to the predetermined conditions, and display the avatar with the predetermined facial expression or pose applied.

Description

[Technical field]

[0001] The technology disclosed in this application relates to a computer program, a server device, a terminal device, and a method. [Background technology]

[0002] In recent years, technology using avatars played by broadcasters has been developed. In this type of technology, viewers can view the avatars played by broadcasters via a broadcasting service.

[0003] Also, in relation to services using avatars, a service called "Custom Cast" is known as a service that uses a technology to control the facial expressions and movements of an avatar object based on the movements of a performer, etc. (Non-Patent Document 1). In this service, a performer pre-assigns one of a large number of prepared facial expressions and movements to each of a number of flick directions on a smartphone screen, and by flicking the smartphone screen in the direction corresponding to the desired facial expression or movement, the performer can cause the avatar object displayed in the video to express that facial expression or movement. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] "Custom Cast", [online], Custom Cast Inc., [Retrieved December 10, 2019], Internet (URL: https: / / customcast.jp / ) Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the technology disclosed in Non-Patent Document 1, the performer must flick the smartphone screen while speaking, and it is difficult for the performer to perform the flick operation.

[0006] The above-mentioned non-patent document 1 is incorporated herein in its entirety by reference.

[0007] Some embodiments disclosed in the present application provide a computer program, a server device, a terminal device, and a method that can more appropriately represent an avatar. [Means for solving the problem]

[0008] A computer program according to one embodiment, when executed by one or more processors, acquires information relating to one or more of a predetermined movement of an avatar performed by a distributor, a facial expression of the avatar, a complexion of the avatar, a voice of the avatar, or a predetermined situation occurring regarding the avatar within a virtual space, determines whether the acquired information satisfies a predetermined condition, and if it is determined that the predetermined condition related to the acquired information is satisfied, generates information for changing the facial expression or pose of the avatar related to the distributor that was generated based on the facial expression or pose related to the distributor to a predetermined facial expression or pose that corresponds to the predetermined condition, and displays the avatar with the predetermined facial expression or pose applied.

[0009] A computer program in one embodiment, when executed by one or more processors, acquires information relating to one or more of a predetermined movement of an avatar performed by a broadcaster, a facial expression of the avatar, a complexion of the avatar, a voice of the avatar, or a predetermined situation occurring regarding the avatar in a virtual space, determines whether the acquired information satisfies a predetermined condition, and if it determines that the predetermined condition is satisfied, generates information for changing the facial expression or pose of the avatar of the broadcaster that was generated based on the facial expression or pose of the broadcaster to a predetermined facial expression or pose that corresponds to the predetermined condition, and transmits the information for changing the predetermined facial expression or pose.

[0010] A server device according to one embodiment includes one or more processors, and executes computer-readable instructions to acquire information relating to one or more of a predetermined movement of an avatar performed by a broadcaster, a facial expression of the avatar, a complexion of the avatar, a voice of the avatar, or a predetermined situation occurring regarding the avatar in a virtual space, determine whether the acquired information satisfies predetermined conditions, and if it is determined that the predetermined conditions are satisfied, generate information for changing the facial expression or pose of the avatar of the broadcaster that was generated based on the facial expression or pose of the broadcaster to a predetermined facial expression or pose that corresponds to the predetermined conditions, and transmit the information for changing the predetermined facial expression or pose.

[0011] A method according to one embodiment is a method executed by one or more processors executing computer readable instructions, which acquires information relating to one or more of a predetermined movement of an avatar performed by a broadcaster, a facial expression of the avatar, a complexion of the avatar, a voice of the avatar, or a predetermined situation occurring regarding the avatar in a virtual space, determines whether the acquired information satisfies a predetermined condition, and if it is determined that the predetermined condition related to the acquired information is satisfied, generates information for changing the facial expression or pose of the avatar related to the broadcaster that was generated based on the facial expression or pose related to the broadcaster to a predetermined facial expression or pose corresponding to the predetermined condition, and displays the avatar with the predetermined facial expression or pose applied.

[0012] A method according to one embodiment is a method executed by one or more processors executing computer-readable instructions, which acquires information relating to one or more of a predetermined movement of an avatar performed by a broadcaster, a facial expression of the avatar, a complexion of the avatar, a voice of the avatar, or a predetermined situation occurring regarding the avatar in a virtual space, determines whether the acquired information satisfies a predetermined condition, and if it is determined that the predetermined condition is satisfied, generates information for changing the facial expression or pose of the avatar of the broadcaster that was generated based on the facial expression or pose of the broadcaster to a predetermined facial expression or pose that corresponds to the predetermined condition, and transmits the information for changing the predetermined facial expression or pose.

[0013] A viewer terminal device according to one embodiment includes one or more processors, and executes computer-readable instructions to acquire information relating to one or more of a predetermined action of an avatar performed by a distributor, a facial expression of the avatar, a complexion of the avatar, a voice of the avatar, or a predetermined situation occurring regarding the avatar, within a virtual space, determine whether the acquired information satisfies predetermined conditions, and if it is determined that the predetermined conditions related to the acquired information are satisfied, generate information for changing the facial expression or pose of the avatar related to the distributor that was generated based on the facial expression or pose related to the distributor to a predetermined facial expression or pose that corresponds to the predetermined conditions, and display the avatar with the predetermined facial expression or pose applied.

[0014] A broadcaster terminal device according to one embodiment includes one or more processors, and executes computer-readable instructions to acquire information relating to one or more of a predetermined movement of an avatar performed by a broadcaster within a virtual space, a facial expression of the avatar, a complexion of the avatar, a voice of the avatar, or a predetermined situation occurring regarding the avatar, determine whether the acquired information satisfies predetermined conditions, and if it is determined that the predetermined conditions are satisfied, generate information for changing the facial expression or pose of the avatar of the broadcaster that was generated based on the facial expression or pose of the broadcaster to a predetermined facial expression or pose that corresponds to the predetermined conditions, and transmit the information for changing the predetermined facial expression or pose. [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 is a block diagram showing an example of a configuration of a system according to an embodiment. [Diagram 2] FIG. 2 is a block diagram illustrating an example of a hardware configuration of the server device 20 (terminal device 30) illustrated in FIG. [Diagram 3] FIG. 3 is a block diagram illustrating an example of the functions of a system according to an embodiment. [Figure 4] FIG. 4 is an example of data held by the system according to an embodiment. [Diagram 5] FIG. 5 is an example of data held by the system according to an embodiment. [Figure 6] FIG. 6 is an example of a flow of processing performed by a system according to an embodiment. [Figure 7] FIG. 7 is an example of a flow of processing performed by a system according to an embodiment. [Figure 8] FIG. 8 is an example of a flow of processing performed by a system according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Hereinafter, various embodiments of the present invention will be described with reference to the attached drawings. Please note that components depicted in one drawing may be omitted in another drawing for convenience of explanation. Furthermore, please note that the attached drawings disclose an embodiment of the present invention, but are not necessarily drawn to scale. In addition, in this document, the subject indicated by the term application may be software or a program, which is an instruction for a computer and may be combined to obtain a result.

[0017] 1. System Example FIG. 1 is a block diagram showing an example of a system configuration according to an embodiment. As shown in FIG. 1, the system 1 may include one or more server devices 20 connected to a communication network 10, and one or more terminal devices 30 connected to the communication network 10. In FIG. 1, three server devices 20A to 20C are illustrated as examples of the server devices 20, and three terminal devices 30A to 30C are illustrated as examples of the terminal devices 30. However, one or more server devices 20 other than these may be connected to the communication network 10 as the server devices 20, and one or more terminal devices 30 other than these may be connected to the communication network 10 as the terminal devices 30. In the present application, the term "system" may include both a server and a terminal device, or may refer to only a server or only a terminal device. That is, the system may be in any of the following forms: only a server, only a terminal device, or both a server and a terminal device.

[0018] The system may be an information processing device on a cloud. The system may be a virtual information processing device and may be logically configured as one information processing device. The owner and the administrator of the system may be different persons.

[0019] The communication network 10 may be, but is not limited to, a mobile phone network, a wireless LAN, a landline telephone network, the Internet, an intranet, Ethernet (registered trademark), and / or a combination thereof. In the communication network 10, various data disclosed in the present application document may be communicated.

[0020] The server device 20 executes the installed specific application to acquire information relating to one or more of a predetermined action of an avatar performed by a distributor, a facial expression of the avatar, a complexion of the avatar, a voice of the avatar, or a predetermined situation occurring regarding the avatar, in a virtual space, determines whether or not the acquired information satisfies a predetermined condition, and if it determines that the predetermined condition is satisfied, generates information for changing the facial expression or pose of the avatar related to the distributor, which has been generated based on the facial expression or pose related to the distributor, to a predetermined facial expression or pose corresponding to the predetermined condition. It may be possible to execute operations such as:

[0021] Alternatively, the terminal device 30 may be able to execute an installed web browser or a specific application to acquire information relating to one or more of a specified action of an avatar performed by a broadcaster, the facial expression of the avatar, the complexion of the avatar, the voice of the avatar, or a specified situation occurring regarding the avatar, within a virtual space, determine whether the acquired information satisfies specified conditions, and if it determines that the specified conditions are satisfied, generate information for changing the facial expression or pose of the avatar of the broadcaster that was generated based on the facial expression or pose of the broadcaster to a specified facial expression or pose that corresponds to the specified conditions, by executing the installed web browser or a specific application.

[0022] The terminal device 30 is any terminal device capable of performing such operations, and may be, but is not limited to, a smartphone, a tablet, a mobile phone (feature phone), and / or a personal computer.

[0023] 2. Hardware configuration of each device Next, an example of a hardware configuration of the server device 20 will be described.

[0024] 2-1. Hardware configuration of server device 20 An example of the hardware configuration of the server device 20 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing a schematic example of the hardware configuration of the server device 20 (terminal device 30) shown in Fig. 1 (note that in Fig. 2, the reference symbols in parentheses are those described in relation to each terminal device 30, as will be described later).

[0025] 2, the server device 20 may mainly include a calculation device 21, a main memory device 22, and an input / output interface device 23. The server device 20 may further include an input device 24 and an auxiliary output device 26. These devices may be connected to each other by a data bus and / or a control bus.

[0026] The arithmetic device 21 performs calculations using instructions and data stored in the main memory device 22, and stores the results of the calculations in the main memory device 22. Furthermore, the arithmetic device 21 can control an input device 24, an auxiliary memory device 25, an output device 26, and the like via an input / output interface device 23. The server device 20 may include one or more arithmetic devices 21. The arithmetic device 21 may include one or more central processing units (CPUs), microprocessors, and / or graphics processing units (GPUs).

[0027] The main memory device 22 has a storage function and stores commands and data received from the input device 24, the auxiliary memory device 25, and the communication network 10 (such as the server device 20) via the input / output interface device 23, as well as the results of calculations by the calculation device 21. The main memory device 22 can include, but is not limited to, a RAM (random access memory), a ROM (read only memory), and / or a flash memory.

[0028] The auxiliary storage device 25 is a storage device. It may store instructions and data (computer programs) constituting the specific application, web browser, etc., and may be controlled by the arithmetic device 21 to load these instructions and data (computer programs) into the main storage device 22 via the input / output interface device 23. The auxiliary storage device 25 may be, but is not limited to, a magnetic disk device and / or an optical disk device, a file server, etc.

[0029] The input device 24 is a device for inputting data from the outside, and may be a touch panel, a button, a keyboard, a mouse, and / or a sensor.

[0030] The output device 26 may include, but is not limited to, a display device, a touch panel, and / or a printer device, etc. Also, the input device 24 and the output device 26 may be integrated together.

[0031] In such a hardware configuration, the calculation device 21 sequentially loads instructions and data (computer program) constituting a specific application stored in the auxiliary storage device 25 into the main storage device 22, and calculates the loaded instructions and data, thereby controlling the output device 26 via the input / output interface device 23, and / or transmitting and receiving various information to and from other devices (such as the server device 20 and other terminal devices 30) via the input / output interface device 23 and the communication network 10.

[0032] By having such a configuration and executing a specific application installed, the server device 20 may be able to execute a part or all of the following operations (including various operations described in detail later): acquiring information related to one or more of a predetermined action of an avatar performed by a distributor, a facial expression of the avatar, a facial color of the avatar, a voice of the avatar, or a predetermined situation occurring regarding the avatar in a virtual space defined below; determining whether the acquired information satisfies a predetermined condition; and, if it is determined that the predetermined condition is satisfied, generating information for changing the facial expression or pose of the avatar related to the distributor that was generated based on the facial expression or pose of the distributor to a predetermined facial expression or pose corresponding to the predetermined condition. Moreover, such operations may be executed by a user giving an instruction to a system of an example of the invention disclosed in the present application documents using the input device 24 or the input device 34 related to the terminal device 30 described later. Moreover, when the program is executed on the computing device 21, the program may be displayed by the output device 26 of the server device 20 as a system used by the user, or may be configured to be displayed on the output device 36 of the terminal device 30 as a system used by the user.

[0033] 2-2. Hardware configuration of terminal device 30 An example of the hardware configuration of the terminal device 30 will also be described with reference to Fig. 2. The hardware configuration of each terminal device 30 may be the same as that of each of the above-mentioned server devices 20. Therefore, the reference characters for the components of each terminal device 30 are shown in parentheses in Fig. 2.

[0034] 2, each terminal device 30 can mainly include a calculation device 31, a main memory device 32, an input / output interface device 33, an input device 34, an auxiliary memory device 35, and an output device 36. These devices are connected to each other by a data bus and / or a control bus.

[0035] The arithmetic device 31, the main memory device 32, the input / output interface device 33, the input device 34, the auxiliary memory device 35, and the output device 36 may be substantially the same as or different from the arithmetic device 21, the main memory device 22, the input / output interface device 23, the input device 24, the auxiliary memory device 25, and the output device 26 included in each of the above-mentioned server devices 20. The arithmetic device and the memory devices may have different capacities and capabilities. The input device 34 and the output device 36 will be described in detail below.

[0036] In such a hardware configuration, the calculation device 31 sequentially loads instructions and data (computer programs) constituting a specific application stored in the auxiliary storage device 35 into the main storage device 32, and by calculating the loaded instructions and data, it is possible to control the output device 36 via the input / output interface device 33, and also to send and receive various information between the output device 36 and other devices (e.g., each server device 20, etc.) via the input / output interface device 33 and the communication network 10.

[0037] The terminal device 30 may be equipped with such a configuration, and may be capable of performing some or all of the following operations (including various operations described in detail later): acquiring information related to one or more of a predetermined action of an avatar performed by a distributor, a facial expression of the avatar, a facial color of the avatar, a voice of the avatar, or a predetermined situation occurring regarding the avatar in a virtual space; determining whether the acquired information satisfies a predetermined condition; and, if it is determined that the predetermined condition is satisfied, generating information for changing the facial expression or pose of the avatar related to the distributor, which was generated based on the facial expression or pose of the distributor, to a predetermined facial expression or pose corresponding to the predetermined condition. Also, by executing an installed web browser, the terminal device 30 may receive and display a web page from the server device 20 to perform the same operation. Also, such operations may be performed by a user giving an instruction to the system of the example of the invention disclosed in the present application document using the input device 34. Also, when the program is executed on the computing device 31, the configuration may be such that the program is displayed on the output device 36 of the terminal device 30 as a system used by the user.

[0038] In the present application, the terminal device may be classified into two types of terminal devices, a "distributor terminal" and a "viewer terminal". These may be names given to the roles of the terminals, and one terminal device may be a viewer terminal, a distributor terminal, or a viewer terminal and a distributor terminal at the same time. The distributor terminal may be a terminal used by a distributor who appears as a subject such as an avatar in the distributed content. On the other hand, the viewer terminal may be a terminal used mainly by a viewer who views the distributed content. The viewer may also have a certain influence on the distributed content by a gifting act of providing a gift to a subject such as an avatar, such that a gift appears in the distributed content. Similarly, the viewer may have a certain influence on the distributed content in the form of a voice, a comment, or the like. The distributed content may be a video, a video, or an animation, and it is sufficient that a subject such as an avatar played by the distributor appears in some of these. The above description is merely an example of application of one embodiment of the present invention, and each embodiment of the present invention may be applied to a virtual space in which a subject without such a premise appears.

[0039] As described above, the viewer terminal may be configured to allow voice input, comment input, gifting, and the like. For example, the user terminal may have, as the input device 34, a keyboard, a touch panel, or the like that can input comments and acquire comments as text information. The user terminal may also have a function of generating certain comments by the computing device 34. These acquired comments may be transmitted to the communication network 10 via the input / output interface device 23 in the same manner.

[0040] The comment according to the present invention may be a comment automatically generated by the system in addition to a comment input by a user terminal. Such a comment may be transmitted to the user terminal and displayed on the user terminal.

[0041] The terminal may also include a microphone capable of acquiring sound as the input device 34. Human voice may be acquired by the microphone as the input device 34 and treated as sound, or may be acquired by the microphone as the input device 34 and then converted into text by speech recognition by the arithmetic device 31 and treated as a comment. The user terminal may generate sound by the arithmetic device 31. The acquired sound or comment may be transmitted to the communication network 10 via the input / output interface device 23 in the same manner.

[0042] On the other hand, the broadcaster terminal may include a motion sensor capable of detecting human movement as the input device 34. The motion sensor may be attached to the broadcaster's body and have a function capable of detecting the movement of each part of the body. The parts of the body may include various parts such as the head, face, upper arms, forearms, hands, thighs, lower legs, feet, and trunk, but are not limited to some or all of these, and may be attached at a position where the movement of the body can be detected. Such a sensor may track the facial expression and pose of the broadcaster and generate them as information. The motion sensor may have an acceleration sensor, a gyro sensor, a geomagnetic sensor, etc. The acquired information may be transmitted to the communication network 10 via the input / output interface device 23 in the same manner.

[0043] The distributor terminal may also include a camera capable of capturing video as the input device 34. The camera may capture the distributor's head, face, eyes, a part of the body, the whole body, etc. The camera may be capable of detecting the direction of the face when capturing the head or face, of detecting facial expressions when capturing the face, of detecting the line of sight when capturing the eyes, and of detecting body movement together with or instead of the motion sensor when capturing a part or the whole body. Information acquired about the face direction, line of sight, facial expressions, and body movement based on these image information may be transmitted to the communication network 10 via the input / output interface device 23 in the same manner.

[0044] Moreover, the output device 36 in the viewer terminal and the distributor terminal may be a display device or a sound generating device. The display device may be various devices capable of displaying, such as a liquid crystal display, a plasma display, an organic EL display, a micro LED display, a head mounted display, etc., but is not limited to these. The sound generating device may be a speaker, a headphone, an earphone, etc. Information output by the output device 36 may be information generated by the arithmetic device 31, or may be information received from the communication network 10.

[0045] 3. Functions of each device Next, an example of the functions of each of the server device 20 and the terminal device 30 will be described with reference to Fig. 3. Fig. 3 is a block diagram that illustrates an example of the functions of the system illustrated in Fig. 1. As illustrated in Fig. 3, the example system may include an acquisition unit that acquires information that can be used to generate an object, and a generation unit that generates an object using the acquired information.

[0046] 2.1. Acquisition part The acquisition unit has a function of acquiring information that can be used to determine whether or not a predetermined condition, which will be described later, applies.

[0047] The information acquired by the acquisition unit may include information relating to one or more of the following within the virtual space: a specific action of an avatar performed by the distributor, the facial expression of the avatar, the complexion of the avatar, the voice of the avatar, or a specific situation occurring regarding the avatar (sometimes referred to as "specific actions, etc." in the present application documents).

[0048] The specific contents of the information acquired by the acquisition unit will be described in the section on the condition application unit. Note that Fig. 4 shows an example of part of the information acquired by the acquisition unit.

[0049] Furthermore, when a gift is provided to a specific avatar, the acquisition unit may acquire information that identifies the avatar to which the gift was provided.

[0050] The information processing device from which the acquisition unit acquires information may be the same as or different from the information processing device having the function of the acquisition unit. The term "acquire" may be a higher-level concept for receiving information from another information processing device, for reading information from a storage device, or for receiving information from another program via a storage device.

[0051] 2.2. Condition application part The condition application unit has a function of determining whether the acquired information satisfies a specified condition, and if it is determined that the specified condition related to the acquired information is satisfied, generating information to change the facial expression or pose of the avatar related to the broadcaster that was generated based on the facial expression or pose related to the broadcaster to a specified facial expression or pose that corresponds to the specified condition.

[0052] Here, generating information for changing the facial expression or pose of the avatar of the distributor, which has been generated based on the facial expression or pose of the distributor, to a predetermined facial expression or pose corresponding to the predetermined condition may mean generating only information for changing the facial expression of the avatar of the distributor, which has been generated based on the facial expression of the distributor, to a predetermined facial expression corresponding to the predetermined condition, when the predetermined condition is satisfied, or generating only information for changing the pose of the avatar of the distributor, which has been generated based on the pose of the distributor, to a predetermined pose corresponding to the predetermined condition, or generating information for changing the facial expression and pose of the avatar of the distributor, which have been generated based on the facial expression and pose of the distributor, to a predetermined facial expression and pose corresponding to the predetermined condition. In the present application, the predetermined facial expression and / or pose specified in response to the predetermined condition may be referred to as a predetermined facial expression, etc., and the facial expression and / or pose may be referred to as a facial expression, etc.

[0053] The predetermined facial expression may be an expression displayed using predetermined parameters of facial expressions, or parameters of facial expressions based on the distributor's facial expression corrected based on a predetermined rule, or an expression displayed by setting a predetermined shape or movement, or a pose displayed using predetermined pose parameters, or parameters of poses based on the distributor's pose corrected based on a predetermined rule, or a pose displayed by setting a predetermined shape or movement. Here, the predetermined parameters of facial expressions indicate a plurality of parameters that affect the movement of each part of the face, and one facial expression may be displayed by setting each of the plurality of parameters to a predetermined value. Such one or more parameters may be parameters that affect the movement of each part of the face, and each parameter may be for a specific part of the face, or may be for the entire face (or a part of the face) but the shape of the entire face (or a part of the face) may differ for each parameter. The latter is a technique in which a plurality of objects with the same area but different shapes are prepared, and one facial expression is expressed by combining a part or a plurality of them. An example of the latter is blendshape, in which multiple different shapes of the whole face are prepared in advance and a single facial expression is generated by synthesizing the multiple shapes. In this technology, a special facial expression may be made by preparing a predetermined shape in advance in addition to normal facial elements. For example, if a shape showing vertical lines on the face is prepared in advance, such a vertical line on the face can be produced, and if an element for producing a cross-eyed expression is prepared in advance, such a cross-eyed expression can be produced. In this case, since the parameters are set in advance, the facial expression is reproduced regardless of the broadcaster's facial expression, etc., and there is an advantage that an easy-to-understand facial expression can always be reproduced when appropriate parameters indicating the facial expression are selected. In this case, the predetermined facial expression may be set differently depending on the avatar. In other words, the predetermined facial expression applied when the same predetermined condition is satisfied for avatar A and avatar B may be different for avatar A and avatar B.In this case, the value may be set for each avatar, or may be set according to the type of avatar.

[0054] On the other hand, if the predetermined facial expression is a parameter of a facial expression based on the facial expression of the distributor, the predetermined facial expression may be a facial expression based on the facial expression of the distributor. For example, assume that the distributor has a surprised expression at some gift. In this case, the facial expression of the avatar is set to a facial expression corresponding to surprise by one or more parameters in response to the detection of the facial expression of the distributor, but the value of the multiple parameters related to surprise is increased or decreased by a predetermined value as a predetermined facial expression corresponding to a predetermined condition, so that the surprised facial expression is emphasized. While the above-mentioned technology is absolute in that one or more parameters are determined as a predetermined facial expression, the present technology may use a parameter relative to information obtained from the current facial expression of the distributor. Therefore, even when the same avatar played by the same distributor is applied with the same predetermined conditions, the parameters of the facial expression based on the facial expression of the distributor at that time may be the same or different, so that the facial expression according to such different parameters may become the predetermined facial expression. Therefore, it is possible to provide more variation in facial expressions.

[0055] Furthermore, the facial expression displayed by setting a predetermined shape or movement may be one in which a predetermined shape, such as a mask, is itself the specified facial expression.

[0056] Although the above describes predetermined facial expressions, the corresponding techniques may be used for predetermined poses as well.

[0057] Furthermore, the predetermined facial expression, etc. may be a facial expression, etc. associated with a predetermined action, etc. to be performed. In this case, there is an advantage that the viewer can easily understand the reason why the predetermined facial expression, etc. is expressed by associating it with the predetermined action, etc.

[0058] Furthermore, the predetermined facial expression, etc. may act on parameters for all parts constituting the facial expression, etc., or may act on some of the parameters. For example, the predetermined facial expression, etc. may act only on parameters for the mouth, and parameters for parts other than the mouth may be parameters linked to the facial expression of the broadcaster and based on information corresponding to the facial movement of the broadcaster from the broadcaster terminal. Furthermore, the predetermined facial expression, etc. may include the appearance of the avatar. For example, the predetermined facial expression, etc. may include a performance in which the avatar is charred.

[0059] In addition, the predetermined facial expression, etc. may change in conjunction with the direction of the predetermined motion, etc. In the virtual space, the broadcaster may not know where the object, etc. described below appears. In this case, the broadcaster may not notice the location where the object, etc. appears, and as a result, the avatar related to the broadcaster may not be able to react to the object, etc. For example, the broadcaster may not notice when a ghost character appears or a basin falls from above. Therefore, there is an advantage that the predetermined facial expression, etc. related to the avatar changes in conjunction with the direction in which the predetermined motion, etc. is displayed, so that the viewer can also become more interested in the avatar's behavior. Here, there may be various modes in which the facial expression, etc. changes in conjunction with the direction of the predetermined motion, etc. For example, the predetermined facial expression, etc. set may be directed toward the direction in which the ghost character has come into contact or the direction in which the zombie character has approached. When the predetermined facial expression, etc. is directed, a part of the avatar's body may change in conjunction with the avatar, or multiple parts of the avatar may change in conjunction with the avatar. For example, when a certain facial expression is made, the entire body may be turned in that direction, only the head may be turned in that direction, or only a part of the face such as the eyes may be turned in that direction.

[0060] The predetermined pose may be various poses. The pose may be only a part of the avatar's body, or may be the entire avatar's body. For example, it may be a pose of only the head, such as a head tilt pose, a pose of only the hands, including an effect of the hands being enlarged or red and swollen, a pose of only the upper body, such as a fist pump, or a pose of the whole body, such as a pose with a gun held up or a pose of jumping with the whole body.

[0061] The specified condition may use information relating to one or more of a specified action of an avatar performed by the broadcaster, a facial expression of the avatar, a complexion of the avatar, a voice of the avatar, or a specified situation occurring regarding the avatar.

[0062] The predetermined action of the avatar performed by the distributor may be a predetermined action that is perceived by the avatar with the five senses if the avatar is a living being. The predetermined action of the avatar may include an action of looking at a predetermined object. For example, when the avatar is surprised at seeing something, a surprised expression may be set as the predetermined expression, etc. In this case, an object that is natural for the avatar to have a surprised expression may be set in advance, and when such an object is on an extension of the line of sight, it may be determined that the avatar has seen such an object. That is, when the predetermined action includes an action of looking at the predetermined object, the predetermined condition may include moving the line of sight of the avatar so that the predetermined object is in the line of sight of the avatar, or the predetermined object being present on the line of sight of the avatar. In the former case, when the avatar changes its line of sight by moving its viewing direction on its own initiative, if a surprising object is located on an extension of the line of sight, a surprised expression may be set as the predetermined expression, etc. In the latter case, a predetermined object may be present, and when the avatar is looking in a predetermined direction, a predetermined object may be generated and displayed in that direction, etc.

[0063] In the above, the object that brings the avatar a feeling of surprise has been described, but it may be an object that brings about other feelings or sensations, not limited to the feeling of surprise. For example, it may be an object that causes a certain feeling or sensation when a person sees the object, such as an object that brings about a feeling of fear, such as a ghost, an object that brings about a feeling of like, such as a steak, or an object that brings about a feeling of dislike, such as natto. In addition, as for the predetermined facial expression when seeing such an object, a predetermined facial expression showing a face that a person generally makes when seeing such an object may be set. The above-mentioned object that brings about a feeling of fear, an object that brings about a feeling of like, and an object that brings about a feeling of dislike may be a frightening facial expression, a smiling facial expression, a disgusting facial expression, or the like.

[0064] In the above, the object to be viewed has been described as an object, but the object to be viewed may be another character or another avatar. That is, when the predetermined object is another avatar or character, the predetermined action of the avatar may include the line of sight of the avatar meeting the line of sight of the predetermined object. In this way, when the line of sight of the avatar meets with another character or another avatar, a predetermined facial expression may be set. In addition, when the lines of sight of the avatars meet, a predetermined facial expression may be set only for one avatar, or a predetermined facial expression may be set for both avatars. In the latter case, for example, a facial expression of excitement in which the lines of sight meet and both avatars turn red, etc., may be considered. In addition, in the latter case where a predetermined facial expression is set for both avatars, the set facial expression may be the same predetermined facial expression, etc., or may be different predetermined facial expressions, etc. The reason for such different predetermined facial expressions, etc. may be a predetermined facial expression, etc. according to the type to which the avatar belongs, a predetermined facial expression, etc. according to the preferences or characteristics of the avatar, or a predetermined facial expression, etc. selected probabilistically. As an example of a case where the avatar meets the gaze of another character, when the gaze of the avatar meets the gaze of a ghost character, the avatar's face may be set to turn pale as a predetermined facial expression of the avatar, etc. Satisfaction of such a predetermined condition may be determined by determining whether or not there is an intersection between the gaze direction of the avatar and an object such as a ghost character.

[0065] In the above, the object to be viewed has been described as an object, but the object to be viewed may be some kind of event. The scale and object of the event may be various and are not limited. For example, the event may be a scary situation such as a swarm of crickets rushing at you, being swept away by the water of a flooded river, or being hit by a gust of wind, or it may be a painful situation such as another character or avatar falling or bleeding. Even in the case of such an event, the avatar who has seen the event may be set to a corresponding predetermined facial expression, etc.

[0066] Furthermore, the predetermined action of the avatar may include an action of touching a predetermined target. When the predetermined action includes an action of touching the predetermined target, the predetermined condition may include a distance between a position related to the avatar and a position related to the predetermined target being within a predetermined distance. Here, the position related to the avatar may be any one of a position of the avatar in a virtual space, a part that activates the predetermined facial expression, a predetermined part of the avatar, or a position of a collider of the avatar.

[0067] Here, the action of contacting refers to a state in which the avatar makes contact, whether the avatar is proactive or passive, but does not refer to a state in which a third party makes contact. A state in which a third party makes contact, whether proactive or passive, is included in a "predetermined situation" that occurs with respect to the avatar. Note that "passive" includes, for example, a case in which contact occurs as a result of the objective movement of the avatar or third party, regardless of the intention of the avatar or third party.

[0068] As an example of an avatar's contact action, for example, when the avatar touches a hot object, the avatar may make a surprised expression, or may make a predetermined expression in which the touched part is swollen due to the heat, etc. More specifically, for example, when the avatar touches something hot with its hand, the avatar may make a predetermined expression in which the hand grows larger and becomes red and swollen, etc.

[0069] The predetermined action of the avatar includes the eating and drinking action of a predetermined target. Here, the predetermined eating and drinking action may be predetermined as the eating and drinking action of the avatar, and may be detected by an ID indicating such eating and drinking action. The ID indicating such eating and drinking action may be an ID given to a specific action predetermined as an eating and drinking action, or may be an ID given to a predetermined action based on one or more motion data. In the present invention, the technical significance of incorporating the eating and drinking action as a predetermined action is that a predetermined facial expression, etc. is set by combining it with other information described later. For example, even if it is the same eating and drinking action (e.g., the action of eating meat), when a certain avatar A eats meat, a predetermined facial expression, etc. expressing a refined way of eating is set, while when another avatar B eats meat, a predetermined facial expression, etc. expressing a wild way of eating is set. Here, factors that result in different performances between avatar A and avatar B include, for example, that avatar A and avatar B are different types of avatars, or that even if the avatars are the same type, predetermined facial expressions, etc. expressing different ways of eating are set according to the set tastes, preferences, etc. The type of the avatar may be the personality set when the avatar is configured, or may be determined from the parts and clothing equipped on the avatar at the time a specified condition is determined.

[0070] The predetermined action of the avatar may include a predetermined gesture in response to a smell or odor. Examples of the predetermined gesture include a gesture of sniffing the nose, a gesture of moving the air in front of the nose from side to side with the hand, and the like. Such gestures may be detected, for example, by using an optical sensor, an inertial sensor, or the like to detect the movement of the nose or the movement of the right or left hand. That is, the predetermined condition may be detection of the predetermined gesture of the avatar. Examples of the corresponding predetermined facial expressions include a predetermined facial expression that creates a sense of ecstasy and a predetermined facial expression that shows agony. This has the advantage that the action performed by the distributor when he or she senses a certain odor or smell allows the viewer to more clearly understand the significance of the action. In addition, by sharing specific gestures corresponding to such odors or odors in advance between the distributor and the system, the predetermined facial expression corresponding to such gestures can be more effectively created.

[0071] The predetermined motion of the avatar may include a predetermined gesture for listening to a sound. The predetermined gesture for listening to a sound may be, for example, a gesture of placing a hand on an ear or a gesture of moving one ear in a certain direction. Predetermined facial expressions corresponding to such gestures may include a gesture of placing a hand on an ear, in which the hand becomes larger, and a gesture of moving one ear in a certain direction, in which the ear becomes larger and moves.

[0072] The predetermined condition including the facial expression of the avatar played by the distributor may be, for example, a predetermined facial expression when the facial expression of the avatar is one or more specific parameters. In addition, even if the facial expression is not always a predetermined facial expression, it may be a predetermined facial expression when other predetermined conditions are satisfied. For example, even if a predetermined facial expression is not normally set as a surprised facial expression of the distributor, a predetermined facial expression may be set when the distributor shows a surprised facial expression when a specific event occurs.

[0073] The facial color of the avatar played by the distributor may include changes in facial color. For example, when the facial color of the avatar becomes different from normal, the facial expression of the avatar may be set to a different predetermined facial expression in accordance with the changed facial color.

[0074] The voice of the avatar played by the distributor may include a change in tone quality of the avatar's voice by more than a predetermined range, or a change in volume of the avatar's voice by more than a predetermined amount. For example, the avatar's voice may have a different voice quality from normal voices due to nervousness, anger, etc. In the case of such a voice, a predetermined facial expression or the like corresponding to the voice may be set.

[0075] The predetermined situation occurring with respect to the avatar may include the occurrence of a predetermined sound in the virtual space. The predetermined sound may be, for example, a loud sound. In response to such a loud sound, the predetermined facial expression may be a surprised facial expression or a pose that expresses surprise with the whole body, such as jumping up.

[0076] The predetermined situation occurring with respect to the avatar may include a distance between a position related to a predetermined object and a position related to the avatar falling within a predetermined range. Here, the position related to the predetermined object may be, for example, a location that is the center of gravity of the predetermined object, a location that is the center of the predetermined object, a center of a circumscribing circle of the predetermined object, a center of an inscribing circle of the predetermined object, a center of a circumscribing circle on a display of the predetermined object, a center of an inscribing circle on a display of the predetermined object, or the like. The position related to the predetermined object may also be an intersection point of a tangent of a straight-line distance between a circle or polygon related to the predetermined object and a position related to the avatar with the circle or polygon. The position related to the predetermined object may be determined by setting a collider for the predetermined object and also for the avatar, and using the collider to determine whether or not the predetermined object and the avatar are in contact with each other. The position related to the avatar may be as described above. If the position related to the avatar is the position of a collider, it may be determined that the predetermined object and the avatar are in contact with each other.

[0077] The predetermined object may be any of the various objects described above. For example, when a bucket falls from above and touches the head, the eyes may be set to be crossed out (X) as the predetermined facial expression, etc. When lightning strikes (touches) the avatar, the whole body may be set to be charred as the predetermined facial expression, etc.

[0078] Also, instead of the predetermined object, other things may be used, for example, characters or other avatars. In the case of a character, for example, the predetermined situation occurring with respect to the avatar may include a situation in which the distance between the position of the zombie character and the position of the avatar is within a predetermined range. In this case, when the distance is within the predetermined distance, vertical lines may be added to the face of the avatar. Also, the predetermined facial expression, etc. may change according to the extent of the distance. For example, when the distance changes to a first distance and a second distance shorter than the first distance, the predetermined facial expression, etc. at the first distance may be different from the predetermined facial expression, etc. at the second distance. For example, the predetermined facial expression, etc. at the first distance may be 10 vertical lines in the facial expression, whereas the predetermined facial expression, etc. at the second distance may be 20 vertical lines in the facial expression and the width of the vertical lines is doubled. Conversely to the above, the predetermined condition may include a situation in which the position of the avatar and the position of the character or avatar are outside the range of the predetermined distance. In this case, a situation is shown in which the avatar or character moves relatively in a direction away from the avatar, but for example, an expression such as tears may be set as the predetermined expression of the avatar.

[0079] For example, the predetermined situation occurring with respect to the avatar may include a case where a predetermined relationship occurs between a first avatar and a second avatar. For example, when the first avatar strokes the head of the second avatar, the eyes of the second avatar may be set to hearts as a predetermined facial expression, etc. In this case, a collider may be set for each of the first avatar and the second avatar, and the collision may be judged by the collision judgment.

[0080] In addition, the predetermined situation occurring with respect to the avatar may include a predetermined combination of a predetermined object and a predetermined pose of the avatar. Here, the predetermined object may be an object, a predetermined situation, or a predetermined event. The object may include a gift. For example, when a pistol gift is attached as an object and the avatar takes a shooting pose, a brave expression may be set as the predetermined expression. Here, the shooting pose may be determined by setting a collider from the center position of the avatar and determining the collision of each position of the body such as the right hand, left hand, right foot, left foot, and head. Here, the positions of the right hand, left hand, right foot, left foot, head, etc. of each distributor in the distributor may be detected by an optical sensor or an inertial sensor. In addition, the predetermined expression, etc. set in such a case may be suggested by the combination of the given gifts. For example, if a frog shield, a frog matter, and a frog cloak are used, the face may look like a frog.

[0081] In addition, among the multiple conditions according to the present invention, multiple conditions may be satisfied. For example, when the avatar and the ghost character make eye contact and the distance between the position of the avatar and the position of the ghost character is within a predetermined distance range, etc. can be mentioned. In such a case, when the predetermined expressions, etc. corresponding to the conditions of the eye contact and the distance being within the range are different, processing when multiple predetermined conditions are satisfied may be defined. For example, when a first predetermined condition is satisfied and a second predetermined condition is satisfied within a predetermined time, a predetermined expression or pose corresponding to the case where a predetermined condition with a higher priority is satisfied among the first predetermined condition or the second predetermined condition may be applied. Here, the priority may be associated with each predetermined condition and stored, or may be set as a rule when multiple predetermined conditions are satisfied. For example, when there are predetermined conditions such as A1, A2, A3, A4, etc., it is sufficient to be able to select a predetermined condition with priority after comparing each predetermined condition. The function using such priority has an advantage that even when a plurality of predetermined conditions are satisfied, especially when a plurality of predetermined conditions are satisfied almost simultaneously, it can be processed appropriately. Moreover, such priority may be a probability. For example, when the probability is determined by a pseudorandom function or the like, and when a first predetermined condition and a second predetermined condition are satisfied, if the value of the pseudorandom function is a predetermined value, a predetermined facial expression or the like corresponding to the first predetermined condition is applied, and if the value is another predetermined value, a predetermined facial expression or the like corresponding to the second predetermined condition is applied. In this case, the variation of the production increases, and there is a possibility that it can attract the interest of the viewer more.

[0082] In addition, in the determination of a plurality of conditions that can be satisfied, a time lag may occur between the satisfaction of the predetermined condition and the presentation of the predetermined facial expression, etc., since the determination is made after waiting for the passage of a predetermined time, if the predetermined time is set to a certain time or more. Therefore, when the above-mentioned plurality of conditions are satisfied without setting such a priority, the predetermined facial expression, etc. corresponding to the predetermined condition that is satisfied first in time may be set. Here, the term "first in time" may mean that the processing unit determines that the predetermined condition is satisfied first, even when the predetermined conditions are satisfied substantially at the same time as a result of a network or information processing. In this case, there is an advantage that the predetermined facial expression, etc. corresponding to the predetermined condition that is satisfied first can be set without waiting for the passage of a predetermined time. In addition, in such a case, the predetermined facial expression, etc. corresponding to the predetermined condition that is satisfied later may or may not be set. In the case of setting, the predetermined facial expression, etc. corresponding to the predetermined condition that is satisfied first is overwritten, but there is an advantage that the predetermined facial expression, etc. corresponding to the predetermined condition that is satisfied later can also be presented to the viewer. On the other hand, in the case of not setting, there is an advantage that the viewer can fully enjoy the predetermined facial expression, etc. that was satisfied first. In this way, when both the first and second predetermined conditions are satisfied, a predetermined facial expression, etc. may be set according to the timing when the first predetermined condition is satisfied and the timing when the second predetermined condition is satisfied. For example, when the time between the first timing when the first predetermined condition is satisfied and the second timing when the second predetermined condition is satisfied is shorter than a predetermined time, only the first predetermined facial expression, etc. corresponding to the first predetermined condition may be produced, and the second predetermined facial expression, etc. corresponding to the second predetermined condition may not be produced. In this case, there is an advantage that the viewer can enjoy the production of the first predetermined facial expression, etc. without changing it midway. On the other hand, when the time between the first timing and the second timing is longer than a predetermined time, a predetermined facial expression, etc. corresponding to the second predetermined condition may be set at the timing when the second predetermined condition is satisfied. In this case, there is an advantage that the viewer can enjoy both the production of the first predetermined facial expression, etc. and the second predetermined facial expression, etc.

[0083] The above-mentioned predetermined time for determining the time between the first timing and the second timing may be set according to a set of predetermined conditions that are satisfied. For example, it is assumed that the first predetermined facial expression, etc. is produced when the first predetermined condition is satisfied, the second predetermined facial expression, etc. is produced when the second predetermined condition is satisfied, and the third predetermined facial expression, etc. is produced when the third predetermined condition is satisfied. It is also assumed that the first predetermined condition is satisfied at the first timing, the second predetermined condition is satisfied at the second timing, and the third predetermined condition is satisfied at the third timing. At this time, it is assumed that the second predetermined facial expression, etc. is not produced when the time between the first timing and the second timing is shorter than the 12th predetermined time, and the second predetermined facial expression, etc. is produced when it is equal to or longer than the 12th predetermined time. It is also assumed that the third predetermined facial expression, etc. is not produced when the time between the first timing and the third timing is shorter than the 13th predetermined time, and the third predetermined facial expression, etc. is produced when it is equal to or longer than the 13th predetermined time. At this time, the 12th and 13th predetermined times may be the same or different. If they are the same, it means that the presentation time of the first predetermined facial expression, etc. will not change even if the second predetermined condition is satisfied or the third predetermined condition is satisfied after the first predetermined condition is satisfied. On the other hand, if they are different, the presentation time of the first predetermined facial expression, etc. will change between the case where the second predetermined condition is satisfied and the case where the third predetermined condition is satisfied after the first predetermined condition is satisfied. In this case, the variety of presentations will increase, and there is a possibility that it will be possible to attract the interest of the viewer more.

[0084] In the above, the priority is associated with a predetermined condition or an example of a rule for the predetermined condition to be satisfied is described, but the priority may also be associated with an object, an event, a situation, etc. within the predetermined condition. For example, in the case where a gift is used under a predetermined condition, the gift and the priority may be associated.

[0085] The priority of an object, event, or situation (sometimes referred to as an object, etc. in the present application) related to a specified condition may be generated using the size at which the object, etc. is displayed, the location at which the object, etc. appears, the timing at which the object, etc. appears, etc.

[0086] For example, regarding the size of an object, if a second object is larger than a first object, a higher priority may be set for the second object than for the first object. In this case, the larger object attracts more attention from the viewer, so giving priority to the larger object is considered to be in line with the viewer's wishes. On the other hand, the opposite may be set. Setting the opposite has the advantage of creating an element of surprise.

[0087] In addition to the size of the object, the price may be used if the object has a price, or the consumption points may be used if the object has consumption points. For example, if the price of the second object is higher than the price of the first object, a higher priority may be set for the second object than the first object. In this case, the higher price attracts more attention from viewers, so it is considered that giving priority to the object is in line with the intention of the viewer. On the other hand, the opposite may be set. If the opposite is set, it has the advantage of creating a sense of surprise. Similarly, for consumption points, for example, if the consumption points of the second object are higher than the consumption points of the first object, the priority may be set for the second object than the first object. In this case, the higher consumption points attract more attention from viewers, so it is considered that giving priority to the object is in line with the intention of the viewer. On the other hand, the opposite may be set. If the opposite is set, it has the advantage of creating a sense of surprise.

[0088] As for the location in the virtual space where an object or the like appears, for example, an object or the like that appears in the center of the virtual space may be set to have a higher priority than an object or the like that appears on the left or right edge of the virtual space. When the priority of an object or the like that appears in a conspicuous place in the virtual space is set to be high, the viewer can include the object or the like that appears in the conspicuous place in his or her field of vision, which has the advantage that the viewer can easily understand the meaning of a certain facial expression or the like.

[0089] In addition, the priority of the object may be static information that is set in advance for the object, or may be dynamic information that is set only when the object is provided. The latter may be information that is set according to the position where the object appears. For example, the priority may be set according to the relationship between the position where the object appears in the virtual space and the position of the avatar. In this case, the distance itself may play the role of the priority, or the distance may be used as one piece of information for setting the priority. For example, in a case where the distance between the avatar and the position related to the first object that appeared is a first distance, and the distance between the avatar and the position related to the second object that appeared is a second distance, if the first distance is longer than the second distance, the priority of the first object may be set higher than the second object. In addition, if the first distance between the center of the screen and the position related to the first object is shorter than the second distance between the center of the screen and the position related to the second object, the priority of the first object may be set higher than the second object since the first object appears closer to the center of the screen. When the priority of the object or the like is dynamically set, there is an advantage that a gift to which a predetermined expression or the like is set can be selected more appropriately and dynamically.

[0090] Also, as information related to the avatar, the type to which the avatar belongs may be used. For example, the predetermined condition may be one of a plurality of types to which the avatar belongs, and the type to which the avatar belongs may be used. This may be such that, when certain information is acquired, a predetermined facial expression, etc. is obtained in a first avatar type, but the predetermined facial expression, etc. is not obtained in a second avatar type. For example, a certain type of avatar may be set to have a timid personality, and when the distance between the ghost character and the avatar is within the range of the first distance, the avatar may be more likely to have a scared facial expression. Also, a certain type of avatar may be set to have a cheerful personality, and the condition for a facial expression including a smile as the predetermined facial expression, etc. may be set to be looser than other avatars. In this case, the avatar may be provided with a personality characteristic such as timid, cheerful, bullish, or competitive, and the satisfaction of the predetermined condition may be determined using the personality characteristic. The personality may be set according to the type to which the avatar belongs, or may be set for each avatar separately from the type to which the avatar belongs.

[0091] The predetermined condition may include a probability. That is, the predetermined condition may include a pseudorandom number function, etc., and a predetermined facial expression, etc. may be generated when the value of such a function satisfies the predetermined condition. When such a probability is included, the predetermined facial expression, etc. is not always generated, so that the viewer's interest can be continuously attracted. Also, for example, when the probability value is lower than a predetermined value, if a rare facial expression is generated, the viewer's interest can be attracted in that the viewer can see a valuable facial expression.

[0092] Furthermore, the satisfaction of the predetermined condition may be determined using the timing of display of each object, such as information on the start of display of the object, information on a change in the display of the object, and / or information on the end of display of the object. In this case, the facial expression and pose of the avatar change in response to the display of the object, such as the start, change, and end of the display of the object, so that the viewer can easily understand the meaning of the change in the facial expression and pose of the avatar by associating it with the situation of the object.

[0093] Furthermore, information that an object has as an attribute may be used as the predetermined condition. The attribute of the object may or may not include characteristics that the object has, such as the type of the object, the level of the object, etc. Furthermore, a predetermined condition may be set so that the predetermined condition is satisfied according to a particular feature that the object has, and a corresponding predetermined facial expression may be set.

[0094] The predetermined condition may also stipulate the order or timing of the acquired information.

[0095] For example, with regard to a predetermined facial expression when a certain condition is satisfied, a predetermined facial expression A may be set if a bat character appears on the screen within a predetermined time after a ghost character appears on the screen, and a predetermined facial expression B may be set if a ghost character appears on the screen within a predetermined time after a bat character appears on the screen, and so on.

[0096] The condition application unit may set the facial expression, etc., specified by the satisfaction of a predetermined condition as the facial expression, etc., of the avatar to which the gift is provided. More specifically, the information indicating the facial expression, etc., specified by the satisfaction of a predetermined condition may be an ID indicating the specified facial expression, etc., or one or more parameters of the facial expression indicating the specified facial expression, etc., and the facial expression, etc. of the avatar may be set using the information indicating the specified facial expression, etc., so that the specified facial expression, etc., can be set as the facial expression of the avatar. After the information of the ID or parameters is specified or generated by the condition application unit, the information may be transmitted to a server or terminal for generating a video or animation, and may be used for a predetermined period of time. Here, the condition application unit may store information indicating the specified facial expression, etc., in association with each predetermined condition, and may be configured to specify information indicating the corresponding specified facial expression, etc., when each predetermined condition is satisfied. Note that after the specified facial expression, etc., applied by the satisfaction of a predetermined condition, is set for the avatar for a predetermined period of time, the original avatar's behavior may be restored. That is, since the avatar is performed by the distributor, after a predetermined period of time, the facial expression and pose of the avatar may be based on the facial expression and pose information of the distributor. Here, based on the facial expression and pose information of the distributor may mean that the movement of the avatar is determined by information on the facial expression and pose of the distributor detected by an optical sensor or an inertial sensor.

[0097] In such processing, when a first information processing device on which the condition application unit is executed is the same as a second information processing device which performs processing including setting the facial expression, etc. of the avatar, the condition application unit may process the setting of the facial expression, etc. of the avatar using the facial expression identified by the condition application unit upon satisfaction of a specified condition.

[0098] In addition, when a first information processing device on which the condition application unit is executed is different from a second information processing device which performs processing including setting the facial expression of the avatar, the first information processing device may transmit the facial expression, etc. identified by the condition application unit upon satisfaction of a specified condition to the second information processing device, and the second information processing device may process the specified facial expression, etc. to be set as the facial expression of the avatar.

[0099] Here, when the first information processing device is a server and the second information processing device is a server, information indicating the specified facial expression, etc. may be transmitted from the server of the first information processing device to the server of the second information processing device. Also, when the first information processing device is a server and the second information processing device is a viewer terminal, information indicating the specified facial expression, etc. may be transmitted from the server of the first information processing device to the viewer terminal of the second information processing device. Also, when the first information processing device is a server and the second information processing device is a distributor terminal, information indicating the specified facial expression, etc. may be transmitted from the server of the first information processing device to the distributor terminal of the second information processing device.

[0100] Similarly, when the first information processing device is a viewer terminal and the second information processing device is a server, information indicating the specified facial expression, etc. may be transmitted from the viewer terminal of the first information processing device to the server of the second information processing device. When the first information processing device is a viewer terminal and the second information processing device is a viewer terminal, information indicating the specified facial expression, etc. may be transmitted from the viewer terminal of the first information processing device to the viewer terminal of the second information processing device. When the first information processing device is a viewer terminal and the second information processing device is a distributor terminal, information indicating the specified facial expression, etc. may be transmitted from the viewer terminal of the first information processing device to the distributor terminal of the second information processing device.

[0101] Similarly, when the first information processing device is a broadcaster terminal and the second information processing device is a server, information indicating the specified facial expression, etc. may be transmitted from the broadcaster terminal of the first information processing device to the server of the second information processing device. Also, when the first information processing device is a broadcaster terminal and the second information processing device is a broadcaster terminal, information indicating the specified facial expression, etc. may be transmitted from the broadcaster terminal of the first information processing device to the broadcaster terminal of the second information processing device. Also, when the first information processing device is a broadcaster terminal and the second information processing device is a viewer terminal, information indicating the specified facial expression, etc. may be transmitted from the broadcaster terminal of the first information processing device to the viewer terminal of the second information processing device.

[0102] Although the above describes an aspect in which the condition application section is executed in a single information processing device, the condition application section may be distributed among a plurality of information processing devices for processing.

[0103] Furthermore, the condition application unit may set the predetermined facial expression, etc., using the timing at which the predetermined facial expression, etc. is set. The timing may be, for example, as described above, the timing when the avatar and the avatar or character make eye contact, the timing when the avatar comes into contact with an object, etc. In this case, information including information indicating the predetermined facial expression, etc., may be transmitted to an information processing device on which the information is displayed, and displayed. The timing deviation in this case may include a time lag in communication or processing.

[0104] On the other hand, the timing at which the predetermined facial expression or the like is set after the predetermined condition is satisfied may be set separately. For example, the predetermined facial expression or the like may be set two seconds after the predetermined condition is set. As one example, the predetermined surprised facial expression may be set two seconds after a situation occurs that surprises the avatar, which is an intentionally delayed timing.

[0105] Fig. 5 is an example showing the relationship between predetermined conditions and predetermined facial expressions, etc. By using the acquired information as the predetermined conditions, the corresponding predetermined facial expressions, etc. are identified.

[0106] As described above, when a certain condition is satisfied, the function of the condition application unit allows a corresponding predetermined facial expression, etc. to be realized, which has the advantage of allowing a variety of facial expressions and poses of the avatar in the virtual space. Even if the performer can specify the facial expression of the avatar by himself / herself using a tablet or the like, when the performer wears a gift by an operator in studio distribution, the performer may not notice that the gift has been worn. In such a case, the performer may not have to perform any operation to apply the facial expression or pose corresponding to the gift to the avatar. In other words, the information for the predetermined facial expression or pose may be generated without any operation other than providing the gift from the distributor.

[0107] Furthermore, as described above, the function of the condition application unit makes it possible to automatically realize a corresponding specified facial expression, etc., when a specified condition is satisfied, which has the advantage that the viewer can enjoy the reaction that results in the avatar's facial expression changing without any operation from the viewer's terminal.

[0108] 4. Embodiment 4-1.Embodiment 1 The first embodiment is an example in which the condition application unit operates in the viewer terminal. In this case, the acquisition unit may operate in the viewer terminal. This example will be described with reference to FIG. 6. Since the condition application unit operates in the viewer terminal, it may be possible to change the expression of the avatar for each viewer terminal by including information based on the viewer terminal as a predetermined condition.

[0109] Step 1 The acquisition unit acquires information on a predetermined action, etc. (Step 001). Such information on the predetermined action, etc. may be information inputted in a viewer terminal on which the condition application unit operates, or information inputted in another viewer terminal or information generated in a server as a performance may be acquired by the viewer terminal on which the condition application unit operates via the server, etc.

[0110] Step 2 Next, the condition application unit determines the predetermined condition using at least the information of the predetermined motion, etc., and if it determines that the predetermined condition is satisfied, it identifies the facial expression, etc. corresponding to the satisfied predetermined condition (step 002). For this purpose, the condition application unit may have information indicating at least some of the predetermined conditions and the predetermined facial expression, etc. when such conditions are satisfied.

[0111] Step 3 The specified facial expression, etc. is set in the viewer terminal as the facial expression, etc. associated with the avatar used to satisfy the predetermined condition (step 003). This has the advantage that the viewer can watch the reaction of the avatar's facial expression, etc. changing in response to the predetermined action, etc.

[0112] Step 4 After a predetermined facial expression of the avatar is displayed for a predetermined period of time on each viewer terminal, the facial expression of the avatar returns to the facial expression based on the facial expression of the distributor associated with the original avatar (step 004).

[0113] 4-2.Embodiment 2 The second embodiment is an example in which the condition application unit operates in the server. In this case, the acquisition unit may operate in the server. Note that this example will be described with reference to FIG. 7 on the premise that animation is generated on the viewer terminal.

[0114] Step 1 The acquisition unit acquires information on a predetermined action or the like (step 001).

[0115] Step 2 Next, the condition application unit determines the predetermined condition using at least information on the predetermined movement, etc., and if it determines that the predetermined condition is satisfied, identifies the facial expression, etc. that corresponds to the satisfied predetermined condition (step 002).

[0116] Step 3 The server transmits information indicating the identified facial expression, etc. to each viewer terminal viewing the distribution of the avatar (step 003).

[0117] Step 4 In each viewer terminal, the facial expression of the avatar is set using the information indicating the facial expression etc. acquired from the server (step 004). Here, in each viewer terminal, an animation may be generated using the set facial expression etc. of the avatar.

[0118] Step 5 In each viewer terminal, the facial expression of the avatar using the information indicating the facial expression obtained from the server is displayed for a predetermined period of time, and then the facial expression is set to the facial expression based on the facial expression of the original distributor associated with the avatar (step 005). When an animation is generated, the generated facial expression is displayed for a predetermined period of time in each animation, and thereafter, the facial expression obtained by tracking based on the facial expression of the distributor associated with the avatar may be used as the facial expression of the avatar.

[0119] 4-3.Embodiment 3 The third embodiment is an example in which the condition application unit operates in the server. In this case, the acquisition unit may operate in the server. Note that this example will be described with reference to FIG. 8 on the premise that a video including an avatar's facial expression is generated on the server.

[0120] Step 1 The acquisition unit acquires information on a predetermined action or the like (step 001).

[0121] Step 2 Next, the condition application unit determines the predetermined condition using at least information on the predetermined movement, etc., and if it determines that the predetermined condition is satisfied, identifies the facial expression, etc. that corresponds to the satisfied predetermined condition (step 002).

[0122] Step 3 The server generates a video for the distribution of the avatar using information indicating the specified facial expression, etc. (step 003). Here, when the server generating the video for the distribution of the avatar and the server executing the condition application unit are the same server, the server executing the condition application unit may generate the video for the distribution of the avatar. On the other hand, when the server generating the video for the distribution of the avatar and the server executing the condition application unit are different servers, information indicating the specified facial expression, etc. may be transmitted from the server executing the condition application unit to the server generating the video, and the information indicating the specified facial expression, etc. may be used in the server generating the video. Note that when an animation is generated in the server, an animation for the distribution of the avatar may be generated together with or instead of the video, and in this case, the animation may be used together with or instead of the video thereafter.

[0123] Step 4 An image of an avatar generated using the specified facial expression based on information indicating the specified facial expression, etc. is transmitted to each viewer terminal (step 004).

[0124] Step 5 At each viewer terminal, an image using information indicating facial expressions, etc. obtained from the server is displayed for a predetermined period of time, and then an image of an avatar based on facial expressions, etc. based on the facial expressions, etc. of the distributor associated with the original avatar is displayed (step 005).

[0125] 4-4. Various implementations The computer program according to the first aspect is "a computer program which, when executed by one or more processors, acquires information relating to one or more of a predetermined movement of an avatar performed by a distributor, a facial expression of the avatar, a complexion of the avatar, a voice of the avatar, or a predetermined situation occurring regarding the avatar within a virtual space; determines whether the acquired information satisfies a predetermined condition; and if it determines that the predetermined condition related to the acquired information is satisfied, generates information for changing the facial expression or pose of the avatar related to the distributor that was generated based on the facial expression or pose related to the distributor to a predetermined facial expression or pose corresponding to the predetermined condition; and displays the avatar with the predetermined facial expression or pose applied."

[0126] The computer program according to the second aspect is the same as in the first aspect above, wherein "the specified movement of the avatar is a pre-determined movement that is perceived by the avatar through the five senses if the avatar is a living being."

[0127] The computer program according to a third aspect is the computer program according to the first or second aspect, wherein "the predetermined action of the avatar includes an action of looking at a predetermined target."

[0128] A computer program according to a fourth aspect is one in any one of the first to third aspects above, in which "when the specified target is another avatar or character, the specified action of the avatar includes meeting the line of sight of the avatar with the line of sight of the specified target."

[0129] A computer program according to a fifth aspect is one in any one of the first to fourth aspects above, in which "when the specified action includes an action of looking at the specified target, the specified condition includes moving the line of sight of the avatar so that the specified target is in the line of sight of the avatar, or the specified target being present in the line of sight of the avatar."

[0130] The computer program according to a sixth aspect is the computer program according to any one of the first to fifth aspects, wherein "the predetermined action of the avatar includes an action of contacting a predetermined object."

[0131] A computer program according to a seventh aspect is any one of the first to sixth aspects above, in which "if the specified action includes an action of contacting the specified object, the specified condition includes that the distance between a position relating to the avatar and a position relating to the specified object is within a specified distance."

[0132] The computer program according to the eighth aspect is any one of the first to seventh aspects above, wherein "the position associated with the avatar is either the position of the avatar in a virtual space, a part of the avatar that produces the predetermined facial expression, a predetermined part of the avatar, or the position of a collider for the avatar."

[0133] The computer program according to a ninth aspect is the computer program according to any one of the first to eighth aspects, wherein "the predetermined action of the avatar includes an eating or drinking action of a predetermined subject."

[0134] The computer program according to a tenth aspect is the computer program according to any one of the first to ninth aspects, wherein "the predetermined action of the avatar includes a smell or a predetermined gesture in response to a smell."

[0135] A computer program according to an eleventh aspect is one in any one of the first to tenth aspects, in which "the specified action of the avatar includes a specified gesture for listening to or / listening to sound."

[0136] The computer program according to a twelfth aspect is the computer program according to any one of the first to eleventh aspects, wherein "the predetermined condition uses detection of the predetermined gesture of the avatar."

[0137] A computer program according to a thirteenth aspect is any one of the first to twelfth aspects above, wherein "the specified situation occurring with respect to the avatar includes the occurrence of a specified sound within the virtual space."

[0138] A computer program according to a 14th aspect is any one of the first to 13th aspects above, in which "the specified situation occurring with respect to the avatar includes the distance between a position relating to a specified object and a position relating to the avatar being within a specified range."

[0139] A computer program according to a 15th aspect is one in any one of the first to fourteenth aspects above, in which "the specified situation occurring with respect to the avatar includes a specified combination of a specified object and a specified pose of the avatar."

[0140] The computer program according to a sixteenth aspect is the computer program according to any one of the first to fifteenth aspects, wherein "the facial color of the avatar includes a change in facial color."

[0141] The computer program according to the 17th aspect is any one of the first to sixteenth aspects above, wherein "the voice of the avatar includes the tone quality of the avatar's voice changing by more than a predetermined range, or the volume of the avatar's voice changing by more than a predetermined amount."

[0142] The computer program according to the 18th aspect is, in any one of the 1st to 17th aspects above, "applying a predetermined facial expression or pose corresponding to when a first predetermined condition is satisfied and a second predetermined condition is satisfied within a predetermined time period, and when the first predetermined condition or the second predetermined condition, whichever is set as the higher priority condition, is satisfied."

[0143] The computer program according to the 19th aspect is any one of the first to 18th aspects above, wherein "information for achieving the specified facial expression or pose is generated without obtaining information for operating the avatar from the distributor terminal."

[0144] The computer program according to the twentieth aspect is "a computer program which, when executed by one or more processors, acquires information relating to one or more of a predetermined movement of an avatar performed by a distributor, a facial expression of the avatar, a complexion of the avatar, a voice of the avatar, or a predetermined situation occurring regarding the avatar in a virtual space, determines whether the acquired information satisfies a predetermined condition, and if it determines that the predetermined condition is satisfied, generates information for changing the facial expression or pose of the avatar of the distributor that was generated based on the facial expression or pose of the distributor to a predetermined facial expression or pose that corresponds to the predetermined condition, and transmits the information for changing the predetermined facial expression or pose."

[0145] The computer program according to a 21st aspect is the same as the 20th aspect above, wherein "the specified movement of the avatar is a pre-determined movement that is perceived by the avatar through the five senses if the avatar is a living being."

[0146] The server device according to the 22nd aspect is equipped with one or more processors, which execute computer-readable instructions to acquire information relating to one or more of a predetermined movement of an avatar performed by a distributor, a facial expression of the avatar, a complexion of the avatar, a voice of the avatar, or a predetermined situation occurring regarding the avatar, within a virtual space, determine whether the acquired information satisfies predetermined conditions, and if it is determined that the predetermined conditions are satisfied, generate information for changing the facial expression or pose of the avatar of the distributor that was generated based on the facial expression or pose of the distributor to a predetermined facial expression or pose that corresponds to the predetermined conditions, and transmit the information for changing the facial expression or pose.

[0147] The server device according to a 23rd aspect is the same as the 22nd aspect, wherein "the specified movement of the avatar is a predefined movement that is perceived by the avatar through the five senses if the avatar is a living being."

[0148] The method according to the 24th aspect is "a method executed by one or more processors that execute computer-readable instructions, which acquires information relating to one or more of a predetermined movement of an avatar performed by a distributor, a facial expression of the avatar, a complexion of the avatar, a voice of the avatar, or a predetermined situation occurring regarding the avatar in a virtual space, determines whether the acquired information satisfies a predetermined condition, and if it is determined that the predetermined condition related to the acquired information is satisfied, generates information for changing the facial expression or pose of the avatar related to the distributor that was generated based on the facial expression or pose related to the distributor to a predetermined facial expression or pose corresponding to the predetermined condition, and displays the avatar with the predetermined facial expression or pose applied."

[0149] The method according to the 25th aspect is the same as the 24th aspect, except that "if the avatar is a living being, the predetermined movement of the avatar is a predefined movement that is perceived by the avatar through the five senses."

[0150] The method according to the 26th aspect is "a method executed by one or more processors that execute computer-readable instructions, which acquires information relating to one or more of a predetermined movement of an avatar performed by a distributor, a facial expression of the avatar, a complexion of the avatar, a voice of the avatar, or a predetermined situation occurring regarding the avatar in a virtual space, determines whether the acquired information satisfies a predetermined condition, and if it is determined that the predetermined condition is satisfied, generates information for changing the facial expression or pose of the avatar of the distributor that was generated based on the facial expression or pose of the distributor to a predetermined facial expression or pose that corresponds to the predetermined condition, and transmits the information for changing the facial expression or pose."

[0151] A method according to a 27th aspect is the same as the 26th aspect, except that "if the avatar is a living being, the predetermined movement of the avatar is a predefined movement that is perceived by the avatar through the five senses."

[0152] The viewer terminal device according to the 28th aspect is equipped with one or more processors, which execute computer-readable instructions to acquire information relating to one or more of a predetermined movement of an avatar performed by a distributor, a facial expression of the avatar, a complexion of the avatar, a voice of the avatar, or a predetermined situation occurring regarding the avatar, within a virtual space, determine whether the acquired information satisfies predetermined conditions, and if it is determined that the predetermined conditions related to the acquired information are satisfied, generate information for changing the facial expression or pose of the avatar related to the distributor that was generated based on the facial expression or pose related to the distributor to a predetermined facial expression or pose corresponding to the predetermined conditions, and display the avatar to which the predetermined facial expression or pose has been applied.

[0153] The viewer terminal device according to the 29th aspect is the same as the 28th aspect above, except that "the specified movement of the avatar is a pre-determined movement that is perceived by the avatar through the five senses if the avatar is a living being."

[0154] The distributor terminal device according to the 30th aspect is "equipped with one or more processors, which execute computer-readable instructions to acquire information relating to one or more of a predetermined movement of an avatar performed by the distributor, a facial expression of the avatar, a complexion of the avatar, a voice of the avatar, or a predetermined situation occurring regarding the avatar in a virtual space, determine whether the acquired information satisfies predetermined conditions, and if it is determined that the predetermined conditions are satisfied, generate information for changing the facial expression or pose of the avatar related to the distributor that was generated based on the facial expression or pose related to the distributor to a predetermined facial expression or pose that corresponds to the predetermined conditions, and transmit the information for changing the predetermined facial expression or pose."

[0155] The distributor terminal device according to the 31st aspect is the same as the 30th aspect above, in which "the specified movement of the avatar is a pre-determined movement that is perceived by the avatar through the five senses if the avatar is a living being."

[0156] The system and information processing described in the present application may be implemented by software, hardware, or a combination of these, and a part or all of the system and information processing may be implemented as computer programs and executed by various computers, and these computer programs may be stored in storage media. These programs may also be stored in non-transient or temporary storage media.

[0157] It goes without saying that what is described in this application document is not limited to what is described in this application document, but can be applied to various examples within the scope of various technical ideas having various technical advantages and configurations described in this application document. [Explanation of symbols]

[0158] 1 System 10. Communications Network 20(20A~20C) Server equipment 30(30A~30C) Terminal device 21(31) Arithmetic device 22(32) Main memory 23(33) Input / Output Interface 24(34) Input Devices 25(35) Auxiliary storage device 26(36) Output Device 41 Acquisition Department 42 Condition application part

Claims

1. one or more processors, A first means for acquiring a facial expression of a distributor; A second means for setting a facial expression corresponding to the facial expression of the distributor acquired by the first means to an avatar corresponding to the distributor in a virtual space; and a third means for, when a situation that satisfies any of a plurality of conditions occurs for the avatar in the virtual space, replacing the facial expression of the avatar set by the second means with a facial expression of the avatar that corresponds to the satisfied condition, and after a predetermined period of time has elapsed since the facial expression was set, setting for the avatar a facial expression corresponding to the facial expression of the distributor obtained by the first means.

2. The program of claim 1, wherein the third means, when a situation occurs in the virtual space for the avatar that satisfies any of a plurality of conditions and occurs regardless of the will of the avatar or a third party, sets the facial expression of the avatar corresponding to the satisfied condition in place of the facial expression of the avatar set by the second means, and after a predetermined period of time has elapsed since the facial expression was set, sets the facial expression of the avatar to one corresponding to the facial expression of the distributor obtained by the first means.

3. 3. The program according to claim 1, wherein a situation that satisfies a first condition of the plurality of conditions is that the avatar comes into contact with a predetermined object, character, or another avatar in the virtual space.

4. a situation that satisfies a second condition among the plurality of conditions is that the avatar comes into contact with a first object, a first character, or a first avatar in the virtual space; a situation that satisfies a third condition among the plurality of conditions is that the avatar comes into contact with a second object different from the first object, a second character different from the first character, or a second avatar different from the first avatar in the virtual space; The program according to claim 1 , wherein a facial expression of the avatar corresponding to the second condition is different from a facial expression of the avatar corresponding to the third condition.

5. The program according to claim 1 or 2, wherein a fourth condition among the plurality of conditions is satisfied when the distance between the position of the avatar and the position of a specified object, character or avatar in the virtual space is within or outside a specified range.

6. a fifth condition among the plurality of conditions being satisfied when, in the virtual space, a distance between a position of the avatar and a position of a first object, a first character, or a first avatar is within a first range or outside a predetermined range; a situation in which a sixth condition among the plurality of conditions is satisfied is that a distance between a position of the avatar and a position of a second object different from the first object, a second character different from the first character, or a second avatar different from the first avatar, in the virtual space, is within a second range different from the first range or outside a predetermined range; The program according to claim 1 , wherein a facial expression of the avatar corresponding to the fifth condition is different from a facial expression of the avatar corresponding to the sixth condition.

7. The seventh condition among the above conditions is satisfied in the following situation: When a predetermined relationship occurs between the avatar and another avatar in the virtual space, In the virtual space, a pose of the avatar and a predetermined object are combined in a predetermined manner; and The program according to claim 1 , further comprising at least one of: a predetermined sound being generated in the virtual space.

8. A first means for acquiring a facial expression of a distributor; A second means for setting a facial expression corresponding to the facial expression of the distributor acquired by the first means to an avatar corresponding to the distributor in a virtual space; and a third means for, when a situation that satisfies any of a plurality of conditions occurs for the avatar in the virtual space, setting for the avatar a facial expression for the avatar that corresponds to the satisfied condition instead of the facial expression for the avatar set by the second means, and after a predetermined period of time has elapsed since the setting, setting for the avatar a facial expression corresponding to the facial expression of the distributor obtained by the first means.

9. A step of a first means acquiring a facial expression of a distributor; A step in which a second means sets a facial expression corresponding to the facial expression of the distributor acquired by the first means to an avatar corresponding to the distributor in a virtual space; a third means, when a situation occurs in the virtual space for the avatar that satisfies any of a plurality of conditions, setting, in the avatar's facial expression set by the second means, a facial expression for the avatar that corresponds to the satisfied condition, and after a predetermined period of time has elapsed since the setting, setting, in the avatar, a facial expression corresponding to the facial expression of the distributor obtained by the first means.

Images